LaSCal: Label-Shift Calibration without target labels
Authors: Teodora Popordanoska
Gorjan Radevski
Tinne Tuytelaars
Matthew B. Blaschko
Venue: NeurIPS 2024 poster (2024)
Keywords: uncertainty calibration
calibration error estimation
label-shift
domain adaptation
Primary area: deep_learning_architectures
TL;DR: We propose a novel strategy for unsupervised calibration under label shift, utilizing importance re-weighting of the labeled source distribution.

Abstract: When machine learning systems face dataset shift, model calibration plays a pivotal role in ensuring their reliability.
Calibration error (CE) provides insights into the alignment between the predicted confidence scores and the classifier accuracy.
While prior works have delved into the implications of dataset shift on calibration, existing CE estimators either (i) assume access to labeled data from the target domain, often unavailable in practice, or (ii) are derived under a covariate shift assumption.
In this work we propose a novel, label-free, consistent CE estimator under label shift. Label shift is characterized by changes in the marginal label distribution p(Y), with a constant conditional p(X|Y) distribution between the source and target. We introduce a novel calibration method, called LaSCal, which uses the estimator in conjunction with a post-hoc calibration strategy, to perform unsupervised calibration on the target distribution. Our thorough empirical analysis demonstrates the effectiveness and reliability of the proposed approach across different modalities, model architectures and label shift intensities.

LaSCal: Label-Shift Calibration without target labels
Teodora Popordanoska'
Gorjan Radevski"
Tinne Tuytelaars
Matthew B. Blaschko
ESAT-PSI, KU Leuven
f first Arsstname.. Lastname@kuleuver. be
Abstract
When machine learning systems face dataset shift, model calibration plays pivotal role in ensuring their reliability. Calibration error (CE) provides insights into the alignment between the predicted confidence scores and the classifier accuracy. While prior works have delved into the implications dataset shift on calibration, existing CE estimators either (i) assume access to labeled data from the target domain, often unavailable assumption. In this work we propose novel, label-free, consistent CE estimator under label shift. Label shift is characterized by changes the marginal label distribution p(Y), with constant conditional p(X distribution between the source and target. We introduce a novel calibration method, called LaSCal, which the estimator conjunction with a post-hoc calibration strategy, perform unsupervised calibration on the target distribution. Our thorough empirical analysis demonstrates the effectiveness and reliability of the proposed approach across different modalities, model architectures and label shift intensities.
practice, or (ii) are derived under covariate shift
Introduction
Reliable uncertainty estimation is crucial for predictive models, particularly in safety-critical appli- cations, where decisions based on predictions have significant consequences [Amodei et al. 20166] Kompa et al. 2021]. The calibration error (CE) [Naeini et al.) 2015 Guo et al. [017]] Vaicenavicius al. 2019 measures the discrepancy between predicted probabilities and observed class frequencies, indicating the reliability of the model's predictions. When a calibrated model predicts an 80% chance of flu, we expect 80 out of 100 patients with similar symptoms have the flu. Estimating CE and addressing miscalibration typically requires i.i.d. labeled held-out data. However, real-world settings often violate these assumptions: (i) the source (train) may differ from the target (test) distribution, known as dataset shift [Quiñooner Candela et al. 22099] leading to false sense of confidence in the model and suboptimal decision-making Park et al. 2020]; and (ii) obtaining labeled target data for CE estimation is often unrealistic or prohibitively expensive, e.g., in medical diagnostics during disease outbreaks acquiring labeled patient data is needed, but costly. Thus, traditional post-hoc -oc calibration methods (e.g., temperature scaling [Guo et al., 2017] or isotonic regression [Zadrozny and Elkan 2002)) ) which rely on labeled calibration sets in an i.i.d. setting, are not directly applicable.
The two most common types of dataset shift are: (i) covariate shift, where the feature distribution changes between the source and target domains, denoted as (X) * Pt (X), but the conditional label distribution remains the same, i.c., ps (Y Pt (Y| and (ii) label shift, where the label distribution differs, that is (Y) o pxYY), while the conditional feature distribution remains the same, i.c., Ps (X (Y MMorenoTTorre et al. 2012 Quifionero Candela et al. 2009
" "Equal contribution.
"Label shift corresponds to anti-causal learning: predicting cause Y from effects X (Schölkopf et al. 2012 e.g., during pneumonia outbreak p(Y) (flu) rises but the symptoms p(X (Y) (cough given flu) remain the same.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table 1: Properties of related calibration methods. LaSCal is accuracy preserving, and relies on consistent CE estimator designed for unsupervised calibration under label-shift.
Consistent estimatio under lahel shift assumption
Calibration method
Label shift
target labels
Accuracy preserving
TempScal (Source) Guo ct al. 2017
x
CPCS Park et al. 2020
x
TransCal Wang et 2020
x
HcadToTaal Chen and Su 2023
LaSCal (Ours)
Previous methods, such as TransCal [Wang al.) 2020 CPCS PPak et al. 2020]. and Tail [Chen and Su 2023). address calibration under dataset shift, but are specifically designed around the covariate shift assumption. Notably, while Tail also assumes a change in the label distribution between the source and target domains (i.e., (Y) ) & p.(Y)) by using long-tailed source data and balanced target data, it only partially addresses the label shift scenario. As result, how effectively estimate and maintain calibration under label shift assumption - especially in the absence of target domain labels remains an open question
To address this gap, we derive a novel CE estimator of a model facing label shift, which allows us to reliably estimate CE without requiring labeled target data. Compared to prior work (see T it is the only label-free, consistent CE estimator under the label shift assumption. We build on ideas from unsupervised domain adaptation, and employ importance weighting to estimate the degree of shift in the target distribution. We utilize current state-of-the-art methods, e.g., ELSA Tian et al. (2023), RLLS Azizzadenesheli al. (2019], etc., which yield per-class importance weights account for the label shift. Note that in contrast to our work, these methods focus only on the predictive performance of model, neglecting the model's calibration altogether.
perserving, post-hoc calibration method, called LasCal (Label-Shift Calibration), which utilizes the proposed CE estimator as a loss function. We conduct experiments across a variety of datasets, models, weight estimators, intensities of shift on the target distribution, and imbalance factors of the source distribution validate its performance. Our results demonstrate that LasCal effectively performs unsupervised calibration on the target domain, yielding better calibrated models, compared to traditional i.idd. calibration methods, calibration methods designed for covariate shift [Chen and Su 2023 Park et al. 2020 Wang et al. 2020 and label shift adaptation methods that rely calibration using labeled validation (source) set Alexandari et al. 2020 Wen et al. [0244..
Furthermore, we propose novel,
To summarize, we make the following contributions:
I We derive the first label-free, consistent calibration error estimator under label shift ($3):
We propose post-hoc calibration method, LaSCal, which outperforms existing methods in unsupervised calibration tasks in the presence of label shift ($4.1):
We analyze the properties of LaSCal, and demonstrate its robustness across various datasets, modalities, model architectures and domain-shift scenarios (4..2).
Our codebase is released at the following repository:
2 Related Work
Estimating CE is a challenging task as requires estimating an expectation conditioned on continuous random variable: E Y (X)], where X is the input, Y is a one-hot label, and f is probabilistic model. The CE is often estimated using binning [Zdrrzzy and Elkan, 2001 Naeini et al. 2015). i.e.. in the binary setting, the unit interval [o, 1] is split into intervals (bins) of either equal width Nguyen and O Connor 2015) or equal mass (adaptive binning) /Vaicenavicius et al.
We design a CE estimator under the label shift assumption, which involves scenarios where both the source and target label distributions may be imbalanced, In contrast, Taid operates under a covariate shift assumption (Chen and Su 2023 Section 3.).). and is applied only in specific type of label shift.

---

2019].
Calibration of multi-class model is often quantified via expected CE (ECE) Naeini et al.
2015], used to assess the so-called top-label (or confidence) calibration [Guo et al. 2017 which
only considers the confidence of the predicted class. flaas-wiss calibration [Kull et al. 2019 is
stronger notion, requiring calibrated scores for each class: fx (X) is compared with E Yx J& (X) for each class k. Canonical calibration |Vaicenavicius et al. 2019 [Popordanoska et al. 2022 is the strictest notion, requiring the whole probability vector to be calibrated, i.c. f(X) should match E [Y| (X)]. In this work, we focus binary and class-wise CE, estimated using adaptive binning.
Numerous calibration methods address neural network miscalibration, falling into two categories: post-hoc and trainable strategies. Post-hoc methods adjust the output scores using held-out calibration set. One of the earliest approaches in binary classification is Platt scaling [Platt| [999). J. which has been extended to multi-class setting via matrix. vector and temperature scaling [Guo et al. 2017]. Other
approaches include isotonic regression (Zadrozny and Hikan 2002], ensemble temperature scaling (Zhang et al. 2020), Beta (Kull et al., 2017] and Dirichlet calibration [Kull et al. 2019]. Trainable methods incorporate a calibration objective alongside the classification loss [Kumar et al.) 2018
Mukhoti et al. 2020 Popordanoska et al., 2022]. All of these strategies focus supervised setting, and do not account for dataset shift. Recent studies [Ovadia et al. 2019 Karandikar et al. |02 have shown that models calibrated with traditional i.i.d. calibration methods lose their calibration under dataset shift. Subsequently, several works address calibration under covariate shift assumption: CPCS (Park et al. 2020). TransCal [Wang et al. 2020), and HeadToTail [Chen and Su 2023), J, which calibrate on the target domain without labels. In contrast, we focus on the label shift setting. Our work introduces a general calibration eπror estimator under label shift, usable as training objective in post-hoc and trainable calibration methods. Importantly, post-hoe calibration is done on the unlabeled target data, enhancing performance and reliability compared to standard source data calibration.
Label shift, also known as prior probability shift, [Lipton et al. 2018 Azizzadeneshelli et al., 2019 Alexandari et al.] 2020] is often intertwined with the broader concept of unsupervised domain adaptation Kouw and Loog 202T]. Several different methods address label shift: importance re-
weighting [Lipton et 22018 Azizzadenesheli et al. 2019 SSerens et al., 2002 Tian et al. 2022)
kernel mean matching (KMM) [Zhang et al. 2013 J. and generative adversarial training [Guo et al. 2020). There two popular importance re- weighitgt approaches: one based maximizing the likelihood function and the other based on inverting a confusion matrix. [Sacrens et al. [2002] propose an Expectation Maximization (EM) procedure estimate the class priors shift between the source and target distributions. Importantly, EM does not require retraining or hyperparameter tuning. However, it assumes calibrated predictions, which modern neural networks often lack [Guo et al. (017]. To
address this, hybrid methods combining calibration techniques and domain adaptation methods have been proposed. Alexandari et al. (0200) propose Bias-Corrected Temperature Scaling (BCTS) alongside E.M. Lipton et al. 2018 propose Black-Box Shift Learning (BBSL), which estimates the re-weighting coefficients even if the model is poorly calibrated. As an improvement over BBSL, Azizzadenesheli et al. 2019) propose technique with statistical guarantees: Regularized Learning under Label Shifts (RLLS). They introduce a regularization hyperparameter, addressing the high estimation error of the importance weights in the low target sample regime. Both BBSL and RLLS estimate importance weights from a confusion matrix of held-out validation set. Both methods cope with label shift when the classifier is miscalibrated, but require model retraining with the importance weights. Recently. Tian et al. 20023 propose moment- motthngg framework (Tian et al., 2023) to address label shift, named Efficient Label Shift Adaptation (ELSA). Wen et al. [2024] propose an algorithm called Class Probability Matching with Calibrated Networks (CPMCN), which improves the computational efficiency and empirically outperforms existing methods. Importantly, the goal of these works is to improve the classifier's predictive performance on the label-shifted domain without addressing its calibration. While some methods [Alexandari et al. 20200 Wenn et al. 2024] include a calibration step on the labeled validation (source) data to obtain importance weights, they do not calibrate the models on the target domain. In contrast, we propose approach for target domain calibration without relying on labeled target data. In the absence of target labels, we leverage importance weight estimators to re-weigh the source data.
3 Methods
We consider a classification setting where X E X Rd the input, and Y E y'= 10 is the one-hot encoded target, with d as the feature space dimensionality, and & the number of classes. The
3

---

data consists of: labeled source data (044.)))211 and unlabeled target data {a; 184m The notation P.(.) and (-).) denotes distributions on the source and target domain, respectively. The support on the target domain is a subset of y, i.e., the target data does not contain new classes. We use capital letters for unbounded random variables, and lower case letters with subscripts for elements of the data sample. Note that we may still treat elements of the data sample as random variables.
Consider a probabilistic classifier f:: X + A where A is a (k - 1)-dimensional probability simplex over & classes, and let Z (Xx) denote the predicted probability distribution for input X. We focus on class-wise calibration error [Kull et al. 2019 Kumar et al. 2019 Gruber and Buettner 2022 given by: CW (1) k BI = 11 Zo) - Zcl"),
where Y. denotes the eh entry in the one-hot label, and Ze denotes the tt class confidence score. Since the CE is defined w.r.t. the data distribution, the model's calibration decreases under domain shift, also empirically shown by Ovadia et al. [2019), Karandikar al. (202T]. However, those works estimate CE on the target shifted data using labels, often unavailable in practice. Thus, we derive an Lp classwise CE estimator for label-free target distribution exhibiting label shift.
Calibration error estimator under label shift. We consider a label shift: (Y) + PY) and P.( Y) Px(X Y). We assume the target distribution absolutely continuous w.r.t the source; i.e. for every Y E y with pr (Y) > O, we require pYY > 0 [Lipton et al. 2018 4 Assuming access to ni labeled source samples, and m unlabeled target samples, we aim find an estimator:
m++
CWCE,())* = k mm
Ep. Yo
2go] 250
where the expectations are taken w.r.. the target, and Zÿe denotes the tt entry of the vector 2z-
The main challenge is estimating the conditional expectation Ep,, |Y Zje] without labels from the target distribution. To re-weigh the source label distribution, we use importance weights w (wi,..., :.))...,,))))) w where wi Pc (Y, 1)/p.(Y, 1). which we estimate using unsupervised domain
adaption methods (Tian et al. 2023 Alexandari et al., 2020 Azizzadenesheli et al. 2019 Lipton
al. 2018. Then, for the conditional expectation, we have:
Pr(Y. = Yo: Za = 2r)
Pr = 2e Y. = 1)p, =
EY Zo = xe] =
Σ Ye
(3)
Dr =zo)
Pe = ko)
(0.1)
ps Ie.
Ye = 1)pa = 1)we 22
Twe Dim kiZe = Zc, 2ic)Bic
(4)
pr ze)
mtn 1m
x<Le Zo; Zio)
where w. Pc PY, We is its empirical estimate, and k is any consistent kemnel over its domain [Silverman 1986). Under the label shift assumption, we estimate (z( using P. P.(Z) YY because P(X[Y) remains constant, with Z = f(X) and f being fixed model. The weights @ are estimated for each Y y using labeled source and unlabeled target data, along with the model f. The error rate of the conditional expectation estimator in Equation ( is determined by the maximum error rate of its components: the weight W. and the ratio estimator. Empirically, we use the RLLS estimator, with an eиror rate: O(n 112 + m -1/2) (Azizzadenesheli et al. 2019 Lemma (same as the ratio
estimator (Scott and Wu 1981 Theorem I)).
Proposition 3.1 Given a kernel k consistent over its domain pointwise consistent estimator of E. [Ye Zel, that is:
Silverman 1986 Ep. [Yo Ze] is
We Ei K6 (Z Zic)
Pz 2e Y. = (Y 1)
plim TI.m m0
(5)
I Em (Z,, Zie)
(Z = 2x)
'The support of the target label distribution should be contained within the source label distribution support.

---

Proof sketch. The proof structure follows [Popordanoska et al. 2022, Proposition 3.2], which demonstrates the pointwise consistency of the ratio estimator. Since the weight estimator is also consistent Azizzzdeneshelii et al., 2019) Lemma 1], by the same argument for the product of two convergent sequences of random variables (Proposition 3.2), the conditional expectation estimator also pointwise consistent.
Plugging Equation (4) back into Equation ( for CE under label shift we get:
P
m+n
L
1= K KZjjss Zie)Bic
CWCE
2ze
(6)
1o 2ge+ Zic)
The estimator has values E 0. 2) Since the ratio is pointwise consistent by Proposition 3.1 and following Poorrinnoooooaaskaaaaa et al. [2022] Proposition 3.5], the CE estimator consistent for any consistent kernel. Depending on the kernel. the estimator can be differentiable and integrated into post-hoc and trainable calibration methods 5 We use binning kernel, returning 1 when &ie and Zje fall in the
same bin, and 0 otherwise. The binning estimator yields consistency under well known conditions on the number of bins as a function of the number of observations [Lugosi and Nobel 1996].
Unsupervised calibration under label shift. The CE estimator can be integrated in any post-hoc calibration method, e.g., temperature scaling. In the supervised i.i.d. setting, the optimal temperature I" is obtained by minimizing the cross entropy loss. In the label shift setting, we propose using LaSCal to find 7 by minimizing the classwise calibration error obtained by the estimator in Equation (6.. In particular, let Ij denote the logits corresponding to 2j, and (-)) the softmax function. We find the optimal temperature T as:
T arg min T
1
Σ j=n+1
Ep ((/TTT/(
k
4 Experiments and Discussion
Datasets. To assess the performance of LaSCal for calibrating models facing label shift, we periment using natural image datasets [Krizhesky et al. 22099) as well as datasets derived from real-world scenarios KKoh et al. 2021). In particular, we use the CIFAR-10/100 Long Tail (LT) datasets [Cao et al. 2019 which are simulated from CIFAR (Krizhevsky et al. 2009) with an imbalance factor (IF) defined
ratio of the number of samples in the most and least prevalent class. We additionally use Wilds [Koh et al. 2021] with different modalities: Camelyon17 [Bandi al. 2018] and iWildCam [Beery et al., 2021] with images, and Amazon (Ni et al., 2019 with text. Camelyon17 consists of histopathologicau images of patient lymph node sections with potential metastatic breast cancer. The labels denote whether the central region contains a tumor (binary). iWildCam consists of images from animal traps in the wild, while the labels are different animal species. The Amazon dataset contains review text samples paired with 1-out-of-5 star ratings as labels. Please refer to Appendix A.] for details about the datasets.
Metrics. Unless stated otherwise, we report L2 calibration error (CE) 100 [Kumar al., 2019]. fix the number of bins to 15, and use adaptive binning strategy Vaicenavicius et al.) 2019)). In multi-class settings, we report the sum of per-class CE. We perform bootstrap procedure, i.e., repeatedly resampling with replacement and estimating CE on each subset, and we report the mean and standard deviation of the estimates. In the reliability diagrams, we report L1 top-label CE (ECE).
Models. For the experiments we conduct on CIFAR-10/109 we use ResNet (He et al. 2016 models initialized from scratch with different depths (20, 32, 56, 110). For experiments iWildCam
report results with a standard ResNet-50 (He et al. 2016), two ViT-large transformer- based models Dosovitskiy al., 2020) (with an image resolution of 224 or 384), and Swin-Large (Liu al. 2021 (all pre-trained on ImageNet). For experiments on Amazon, we use pre-trained transformer- based models: BERT (Devlin al. 20188. RoBERTa [Liu et al. 2019). DistillBert (Sanh al. 2019 and DistillRoBERTa (Sanh et al. 2019]. For the experiments on Camelyon17 17, we use a Res pre-trained on ImageNet. Please refer to Appendix A.2 for implementation details.
Popordanoska et al. 2022 and Zhang
2020 proposed Dirichlet and Triweight kernels, respectively.
5

---

4.1 Calibration under label shift
We compare the performance of LaSCal as a method for post-hoe calibration of model trained on source distribution against several (state-of-the-art) baselines. We compare against: (i) Uncal: uncalibrated model trained on source data; (ii) TempScal calibrated model using temperature scaling on source data; (iii) CPCS (Park et al.] 2020]. TransCal [Wang et al., 2020), and HeadToTTail [hen and Sul [2023]: calibrated models using adapted versions of TempScal, derived under covariate shift assumption. Taid additionally assumes long-tailed source data, and balanced target data. (iv) EM-BCTS [Alexandari et al.) 2020] and CPMCN (Wen et al.) 2024]: methods for label shift adaptation, where calibration step is performed weights. Note that TempScal relies only on labeled source data, while the other baselines also incorporate unlabeled target data. Additionally, in Appendix [A.3] we include other common, post-hoc, source-domain calibration methods: vector scaling (VectScal) [Guo et al. 2017]. an ensemble method designed to improve the expressivity of TempScal, abbreviated as EnsT EnsTempSal (Zhang et al. 2020). and one- vereuusuu--lll isotonic regression (IROvA) (Zadrozny and Hikan 2002].
the source data prior to obtaining the importance
We train ResNet models on the 10FAR-10/100 LT variants, and use IF 10, i... the least frequent class is subsampled to 10% of the original size, while the target is balanced. The iWildCam and Amazon datasets have i.i.d. validation set, serving our source distribution, and an i.i.d. test set, to which we apply label shift and use it as our target distribution. We use the i.i.d. test set to ensure that the input distribution p(X) remains the same. On iWildCam, from the target dataset. On both iWildCam and Amazon, we obtain a uniform target distribution by subsampling each class, based on the frequency of the least frequent class.
select the 20 most frequent classes
Performance of LaSCal across various modalities, datasets and models. In Table 2 we report CE on the label-shifted (balanced) target domain before and after calibration with various post-hoc methods. We observe that LaSCal either achieves a lower macro-averaged CE across models compared to other methods, or performs on par with the top-performing method, irrespective of the input modality. Compared to the second best method EM-BCTS where the calibration is performed on the labeled source data, the proposed LaSCal is explicitly derived for unsupervised calibration a label-shifee target distribution. Notably, LaSCal significantly outperforms other baselines on CIFAR-100 where around 50% of the classes contain less than 30 source data points (see Figure 5lin Appendix A.T). This highlights LaSCal's effectiveness even in low data regime In Appendix A.3]] we report accuracy, additional experiments using other IFs on CIFAR-10/100 and provide results for the scenario where the source is balanced and the target is long-tailed, as commonly studied in related works (Tian et al.,
2023 Alexandari et al. 2020 Lipton et al. 2018 Azizzadenesheli et al.) 2019].
Performance of LaSCal compared to temperature scaling using labels. In Fig. [ (Left), we evaluate how closely LaSCal (without labels) approaches the performance of temperature scaling applied on the target distribution using labels, referred to as TempScal (Target), which serves as a competitive baseline. For comparison, also include temperature scaling applied on the source distribution, referred to as TempScal (Source), representing a lower reference point for the method's performance. Throughout these experiments, we keep the input distribution p(X) fixed. Across different models on iWildCam and Amazon, we observe that LaSCal performs favorably relative to TempScaa (Source) and closes the gap with TempScal (Target), demonstrating its effectiveness in unsupervised calibration.
Label shift with changing input distribution. We consider an alternative setting where both the label p(Y) and input p(X) distributions change, which is common in real-world applications" We investigate this scenario by using the outoo--distributin (OOD) test sets of Amazon and iWildCam (p(X) changes), to which we apply label shift, and use them our target distribution. The iWildCam OOD test set contains images of camera traps from locations absent from the source distribution, with variation in illumination, camera angle, background, vegetation, etc. [Beery et al., 2021]. The Amazon OOD test set contains reviews from users outside of the source distribution. In Fig. | (Right), we report the CE after post-hoc calibration using temperature scaling on the source distribution (TempScal), using HeadTTo Tai 8 and LaSCal. We observe that both Head and LaSCal signifi-
'We noticed that the optimal temperature obtained hy LaSCal for CIFAR-100 100 considerably higher than related methods. We hypothesize the discrepancy arises from the optimization process.
Foor example, in medical diagnosis, the model might be facing label shift because of pandemic, while also dealing with patient data from different demographies between training and testing.
*We chose Toail because it is explicitly designed to address the setting where both the input and label distributions change between source and target domains.

---

Table 2: CE on label-shifted target domain before and after calibration with various post-hoc methods. LaSCal performs unsupervised calibration by minimizing CE on the unlabeled target distribution and either outperforms, or performs competitively with the other methods in scenarios.
Uncal
TempScal
CPCS
TransCal
HeadToTail
EM BCTS
CPMCN
LaSCal
CIFAR-IO-LY (IF=10)
ResNet-20
8.87
1.55
1.01
5.05
2.
3,77
3.79
4. 4.44
ResNet-32
10.45=0.41
5.03g0.24
5.18
6.59
4.87=0.15
4.90
5.19
4.81 4.81 10.17
ResNet-56
11.25
4.82
5.10
6.96
4.75.0.14
2.11
4.42
4.11
ResNet-110
11.8 10.89
6.12
6.
7.51 7.51a0.00
4.78
4.40
4.43
4. 4.
Macro average
10.62
4,88
5,00
6.53
4.78
4.39
4.46
4.63
CIFAR-100-LT (IF
10)
ResNet-20
65.00_0
24.61
24.12
48.15
19.93
25.01a00m
25.02
5.62.00.6
ResNet-32
71.16:0:2
28. 29,2920..4
24.844
67.50
28.37
26. 17
21.76g0.20
5.80 20.07
ResNet-56
72.24
20.71
20.03
59.27 20.28
29.72+n.27
26.33
24.53
5.88=0.07
ResNet-111
72.80
31.55g0.25
26.5
60.52
31.58
28. 28.22
26. 26.40
6.19
Macro average
70.47
28.54
25.13
56.37
27.40
26.43
25.20
5.87
Amazon Reviews
RoBERTa
11.44
4.91g031
4.20
4.36
4.36
2.72.0.55
1.36
3.66
DistillRoEERTa
17.82.0.48
5.21
3.60
7.75
2.90
2.13
2.81 2.81=0.23
2.72+0.22
BERT
27.33
7.7520.55
4.34 59
16.98
3.62
3.95
9.32-..54
3.72
DistilBERT
22.18
6.54
3.94
11. 11.80
3.43
3.41
5.48
3.40
Macro average
19.19
6.10
4.02
10.25
3.58
3.05
4.74
3.38
iWilCCam
ResNer50
18.44_0.34
16.38_0.6:
11.52
13.81 +0.48
15.53 0.
15.84
19.43
13.07:0.45
Swin-Large
22.07
17.39 os
17.57 (577..00
16.42.0.an
16.55g0.57
16.8 16.81
18.03
15.43 20.54
ViT-Large
17.94-0.71
16. 16.78
20.2140.50
13.61.40.4
16.53
21.83
19.33
13.07
Vit-Large (384)
18.99
18. 18.7820.80
21.92g0:90
14.81.0.52
17.92
19.78
20.74.0.72
17.27 20.66
Macro average
19.36
17.33
17.81
14.67
16.63
19.31
19.38
14.71
Label shift with fixed input distribution
Label shift with changing input distribution
TampScal (Source) Tempscal (Tbrgat) LaSCal
TempScal (Source) HeadToTail LaSCal
RoBERT:
DistilRoBERTa
Swin-Large
DistilRoBERTB
Swin-Large
1.13
3.01
1.01
1.00
1.11
0.71
0.95
0.95
BERT
1.050 94
0.51
0.9T1.06
VIT-Lage 1384)
BERT
1:64
0.75
Vi--Large (384)
0.62
0.82
0.96
0.86
1.26
0.93
4.50
DistilBERT
VIT-Large
DistilBERT
ViFLaage
ResNet50
ResNet50
Figure 1: Left. Comparison of LaSCal with temperature scaling on the source distribution (TempScal Source) or target distribution (TempScal Target), using labels. Right. CE after post-hoc calibration on iWildCam and Amazon when the target distribution exhibits both label and covariate shift w.r.t. the source. We report CE normalized by the number of classes (5 for Amazon and 20 for iWildCam) for illustration purposes. Lower numbers are better.
cantly outperform TempScal across all models. Furthermore, despite the more challenging setting, we observe that LaSCal performs par outperforms the TaidToTail method.
Top-label calibration. While classwise calibration is central to our analysis, top-label calibration remains a popular approach in related works. To gain insights about this notion of calibration, we present reliability diagrams in Fig- 2 for DistillRoBERTa trained on Amazon, allowing us to visually assess the calibration quality across confidence levels. We report CE of (a) an uncalibrated model, (b) after applying temperature scaling on the source domain, (c) after label-shift adaption using EMBSCTS, and (d) after applying temperature scaling using LaSCal. The blue bars indicate
7

---

1.3-
13-
140
01
38-
03-
Uap
0.1-
Lop
Assimay
34-
06-
06-
0.1-
94-
04-
0:4
02
92
02
02
BCFF-19
19.05
ECE=3.47
ECE=3
BC2288
100
311
00.
10
0D
041
Confidence
Confideece
Cunfiderce
Confidence
(a) Before calibration
(b) TempScal (Source)
(c) EM-BCTS
(d) LaSCal
Figure 2: Reliability diagrams on Amazon using DistillRoBERTa before and after calibration. Temp- Scal (Source) and EM-BCTS perform IID calibration on the source distribution. LaSCal calibrates the model on the unlabeled target distribution. We report J,1 top-label CE in the bottom right corner.
the accuracy per bin, and the red bars represent the gap of each bin to perfect calibration, i.e., the difference between accuracy and confidence for a given bin (darker shades signify under-confidence, while brighter red colors denote confidiiie... We observe that LaSCal provides better calibration (i.e.. lower ECE) compared to other baselines, as also confirmed by the reported L1 top-label CE in the bottom right corner of each plot. Furthermore, note that while the classwise CE values of EM-BCTS are better than LaSCal (Table 2). the diagrams reveal that top-label confidence scores obtained with LaSCal are favorable compared to EM- EM-BCTS in most bins.
Building on this, we adapt our approach to top-label calibration, and we include the adapted estimator, along with empirical comparisons with competing calibration methods in Appendix BB] The results on Amazon and iWildCam further validate the effectiveness of LaSCal, which continues to outperform other methods, demonstrating its superior calibration capabilities in the top-label setting.
4.2 Empirical analysis of the estimator's properties
Robustness analysis. Using the Camelyon17 7 dataset, we conduct a series of experiments to assess the impact of various factors on the performance of the CE estimator, and report the results in Fig. B We partition the original training set as the source distribution, and we use the i.i.d. validation set to form a target set with varying label distribution shifts. We chose this dataset because (i) the application is both realistic and safety-critical; (ii) the dataset is balanced across source and target, enabling us to alter both distributions as per the setting we are trying to verify; (iii) the problem is binary, allowing us to study the estimator properties on a simple problem. For all experiments, we use a ResNet-50 50 model pre-trained on ImageNet, subsequently fine-tuned on the Camelyon17 dataset.
Across the experiments, we construct the train and validation sets, sampled from the source dis- tribution, by keeping all negative samples and sampling a portion of the positives. Unless stated otherwise, we report results by sampling 20% of the positive samples for training: i.c., 5 : ratio of negative to positive points. We compare the estimated CE values (without labels) to the ground truth (with labels), across different experimental scenarios, designed to assess the impact of a change in the data distribution and the sample size on the CE estimation. To account for the variability in data resampling, we average the results across 10 iterations. For each iteration, we apply bootstrap sampling and compute the mean and variance of the estimated CE. Finally, we report the overall mean and standard deviation (depicted as shaded region in the plot) across all iterations.
Impact of data distribution changes. In Fig. 3a investigate the effect of increasing the label shift intensity of the target distribution. We impose a constraint such that the size of the source and target distribution is the same (n = m). and we systematically shift the target by modifying the ratio of negative to positive samples: 5 : 1, 1,..., 1 I,...,I 1 4. Therefore, in the most favorable scenario : 1). the source and target distribution are the same (no label shift), while in the extreme 1 4, we have 4 times as many positive samples in the target data (which could occur, e.g., during a disease outbreak). We observe that the estimated CE closely follows the ground truth, even in the most extreme case. Furthermore, we observe that the variance increases with the intensity of the shift, indicating greater uncertainty and reducing the confidence one should have in the CE estimates. In Fig. 3bb we analyze the effect of changing the ratio of source to target samples (n m). while keeping the total number of points (n + m) constant. The source distribution has a 5 1 ratio of negative to positive points, while the target is balanced. We observe that the estimator achieves
8

---

Without labels
With labels
4,00
Without labels
With labels
8,00
Wilhout lahels
With labels
3.00
2.00
2.00
4.00
1.00
1.00
2.00
0.00
0.00
0.00
5:1
2:1
1:1
1:3
1:4
5:1
4:1
3:1
2:1
1:1 1:2 1:3
1:4
2,000
4,000
8,000
10,000
Ratios of negative to positive points
Ratios souree larget points
Sample
(a) Effect of label shift
(b) Impact of n m
(c) Influence of sample size
Figure Robustness analysis. We report mean and standard deviation over multiple iterations of resampling the data. 5 I denotes "no shift", and the severity increases from left to right.
LaSCal generalizes well to wide range of shifts, ratios of source to target samples, and sample sizes.
optimal performance when the source and target data sizes are equal (n m). As the ratio increases, the estimated values begin diverge slightly from the ground truth. However, the true values lie within the estimator's standard deviation in most cases, demonstrating that even in more extreme settings 5x more source than target samples our estimator yields reliable CE estimates.
Impacl of data size. In Fig. 3c we constrain that n = m, and we incrementally vary the sample size from 1, 000 10, 000 samples. practice, low data regimes are common where annotated data is costly to obtain. We observe that the CE estimates deviate from the ground truth the most when using the fewest data points (i.c., 1000), and improve the data quantity increases. Furthermore, our estimator has a positive bias w.r.t. ground truth in the small data regime, indicating that the estimator tends be conservative in that setting. Such tendency preferable in this context, as prevents mistakenly reporting good performance of a model on the basis of not enough samples. In essence, our method errs on the side of caution, ensuring reliability even in data-constrained environments.
Method analysis. We further investigate (i) how different weight estimation methods influence the estimator's effectiveness, and (ii) what extent our CE estimates (without labels) deviate from the values obtained using labeled target data.
Impact of the weight estimation method. Our estimator relies on the availability of per-class impor- tance weights, which can be obtained using domain adaptation methods, such as ELSA Tian et al.) 2023). RLLS [Arizzadenesheli al. 2019] and BBSL. (Lipton et al.) 2018). In Fig. (Left), compare the performance of these methods when integrated in our estimator. We observe that RLLS emerges the most favorable compared to the others, providing reasonable importance weights in all settings. See Appendix A.4 for experiments involving other weight estimators.
Performance evaluation. In Fig. 4 (Right), we investigate the CE measured by our estimator, compared to the ground-truth CE (obtained using labels from the target domain). Additionally, we report the CE on the source domain reference point. We observe that the calibration error increases from the source to target domain when the model is facing label shift. Importantly, our estimator (CE,) effectively closes the gap to the ground-truth (CE,), consistently yielding accurate CE estimates across models on Amazon and iWildCam.
5 Discussion and Conclusion
In this work, we addressed the problem of estimating CE of an unlabeled target distribution under label shift. We observe that prior state-of-the-ar art methods Wang et al. [2020), Chen and Su 2023 only address CE estimation under the covariate shift assumption: p. (X) + Pt (X) and ( XX
Xx); while, to the best of our knowledge, we propose the first CE estimator under the label shift assumption: ps (Y) + (Y) and p. (X]Y) = (Y)). We demonstrated that it yields CE estimates closely reflecting the ground truth. Furthermore, we showcase that our estimator can be successfully used as a post-hoe calibration method LaSCal for unsupervised model calibration on target distribution. Overall, our experiments indicate that LaSCal can effectively minimize the CE across

---

LasCal with various weight estimation methods
LasCal calibration estimation without labels
RoBERTa
EE (ELSA)
RoBERTa
CE2 (RLLS)
DistilRuBERTe
c (ER--CCTS)
DistilRuBERTA
Swin
3.36
3.36
2.02
1.03
8.73
BERT
1.06
ViT-large (384)
BERT
1.000.29
Virlarge (384)
1.06
DistilBEER ERT
Vi-large
DEtilBERT
Mi-large
ResNet50
Reshet
Figure 4: Left. Impact of the weight estimation method estimator. Right. Our estimator (CE,) effectively closes the gap the ground truth (CE,). We report CE normalized by the number of classes (5 for Amazon and 20 for iWildCam) for illustration purposes.
different intensities of label shift and various modalities. Finally, we analyze the properties of the estimator and contribute towards nuanced understanding of its strengths and weaknesses.
Limitations. First, LaSCal is specifically designed to address label-shift, however, other types of dataset-shift are equally important, e.g., covariate shift [Shimodaira] 2000]. Note that our experiments in $4.1 shed light on the scenario where the models encounter both label shift and shift in p(X), ), and we observed favorable results compared to prior work. Nevertheless, we consider designing consistent CE estimators under covariate shift crucial direction for future work. Second, estimator we propose is dependent on how well the importance weights reflect the ground truth between the classes, which we obtain from current methods (Tian al.) 2023 Azizzadenesheli et 2019 Alexandari al. 2020). Expectedly, we inherit some limitations of such methods: if certain classes are under- represented, the importance weights could be unreliable. However, our experiments Appendix A..4 showcase that our estimator consistently improves as the weights become more accurate. Third, the estimator requires a sufficient number of data samples, e.g., 4000 samples in Figure [c to accurately estimate the calibration error. In severely data-scarce settings, this requirement may limit potential applications. However, the error rate of our estimator is (n 1n/2 m 1/2). which is the same as the weight estimation methods (see Azizzaaennesheli et al. [2019] Lemma I] for the RLLS method, and Garg al. 22200) page 8, top paragraph] for the EM-BCTS method). Therefore, the data requirement is not unique to our method, but rather common across all weight estimation-base approaches.
Broader impact. Our proposed approach effectively reduces CE under label shift, and allows for more comprehensive and realistic evaluation of model calibration. We consider the ethical risks be essentially the same as for any probabilistic classifier. Overall, we consider this paper significant step toward improving the model's robustness and reliability, crucial for safety-critical applications.
Acknowledgments
This research received funding from the Research Foundation Flanders (FWO) through project num- ber S00142IN, the Flemish Government under the "Onderzoeksprogramma Artificille Intelligentic (AI) Vlaanderen" programme, and KULeuven Methusalem.
10

---

References
Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias-corrected calibration is hard-to-beat label shift adaptation. In International Conference on Machine Learning, pages 222-232. PMLR, 2020.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, 2016.
Kamyar Azizzadenesheli. Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. arXiv preprint arXiv:1903.097. 2019.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, al. From detection of individual metastases to classification of lymph node status the patient level: the camelyon17 challenge. IEEE transactions on medical imaging, 38(2):550-560, 2018.
Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. The iwildcam 2021 competition dataset. arXiv preprint arXiv:2105.03494, 2021.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-avaae margin loss. Advances in neural information processing systems, 32, 2019.
Jiahao Chen and Bing Su. Transfer knowledge from head tail: Uncertainty calibration under long-tailed distribution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19978-19987, 2023.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /8 18.0048805 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Mauthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 119299 2020.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances in Neural Information Processing Systems, 33:3290-3300, 2020.
Sebastian Gruber and Florian Buettner. Better uncertainty calibration via proper scores for classifica- tion and beyond. Advances in Neural Information Processing Systems, 35:8618-8632. 2022.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.
Jiaxian Guo, Mingming Gong, Tongliang Liu, Kun Zhang, and Dacheng Tao. LTF: A label trans- formation framework for correcting label shift. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pages 3843-3853, PMLR, 13-18 Jul 2020. URL httpps://proceedings. mlr. press/viii mttass/vv1//gee
https///19ggggoo...oo guo20d. html
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael Curtis Mozer, and Becca Roclofs. Soft calibration objectives for neural networks. In Neural Information Processing Systems, 2021. htLpstts:///pt semanticscholar Irr/CorpusID 236772324
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of mnldhe-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.

---

Benjamin Kompa, Jasper Snoek, and Andrew L Beam. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digital Medicine, 4(1):4, 2021.
Wouter M. Kouw and Marco Loog. A review of domain adaptation without target labels. JEEE Transactions on Pattern Analysis and Machine Intelligence, 43(3):766-788)) 2021. ISSN 0162-8828. 29949442.
doi:
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 623-631. PMLR, 20-22 Apr 2017.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. Advances in neural information processing systems, 32, 2019.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 3792-3803, 2019.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2805-2814, 14, PMLR, 10-15 Jul 2018.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pages 3122-3130. PMLR, 2018.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907 11692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022. 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.
arXiv preprint
17Xiv: 2017.
Gábor Lugosi and Andrew Nobel. Consistency of data-driven histogram methods for density estimation and classification. The Annals of Statistics, 24(2))687 - 706, 1996. doi: 10.1214/a06 1032894460. httphttss///o...ot
1214 /014//0//002289440
Jose G. Moreno Torres, Troy Raeder, Rocío Alaiz Modriggz Nitesh V. Chawla, and Francisco Herrera. A unifying view dataset shift in classification. Pattern Recognition, 45(1):521- 530, 2012. ISSN 0031-3203. doi: http:///hhiinggglloooooooooooop
2011.00.09000000.000 06.019. URL https
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 08:11288--12999 2020.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 2901-2907, 2015.
Khanh Nguyen and Brendan O'Connor Posterior calibration and exploratory analysis for natural language processing models. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1587-1598, 2015.
12

---

Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeles reviews and fine-graine ane-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNIPP-JCNIPP,, pages 188-197, 2019.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Sangdon Park, Osbert Bastani, James Weimer, and Insup Lee. Calibrated prediction with covariate shift via unsupervised domain adaptation. In International Conference on Artificial Intelligence and Stafistics, pages 3219-3229. PMLR, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large-Margin Classifiers, pages 61-74. MIT Press, 1999.
Teodora Popordanoska, Raphael Sayer, and Matthew B. Blaschko. consistent and differentiable lp canonical calibration error estimator. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
Joaquin Quiñionero Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil Lawrence, editors. Dataset shift in machine learning. MIT Press, 2009.
Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of classifier to new a priori probabilities: A simple procedure. Neural Computation, 14:21-41, 2002. URL https::/appi casernanticccolar....mm org/ IorpusID: 18254013
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.1 108, 2019.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In International Conference on Machine Learning, 2012.
Alastair Scott and Chien-Fu Wu. On the asymptotic distribution of ratio and regression estimators.
Journal of the American Statistical Association, 76(373):98-102, 1981. ISSN 01621459.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of Statistical Planning and Inference, 90:227-244, 10 2000. doi: (0/////0887777000000000000 15-4.
B. W. Silverman. Density Estimation for Statistics and Data Analysis. Chapman & Hall, London, 1986.
Qinglong Tian, Xin Zhang, and Jiwei Zhao. Elsa: Efficient label shift adaptation through the lens of semiparametric models. In International Conference Machine Learning, pages 34120-300 341422.. PMLR, 2023.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3459-3467. PMLR, 2019.
Ximei Wang, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable calibration with lower bias and variance in domain adaptation. Advances in Neural Information Processing Systems, 33:19212-19223 2020.
Hongwei Wen, Annika Betken, and Hanyuan Hang. Class probability matching with calibrated networks for label shift adaption. The Twelfth International Conference on Learning Represen- tations, 2024.
13

---

Ross Wightman et al. Pytorch image models, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv: 2019.
B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability estimates. Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, 2002.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning. ICML '01, page 609-616, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558607781.
Jize Zhang, Bhavya Kailkhura, and T. Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration deep learning. In International Conference on Machine Learning, 2020.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International conference on machine learning, pages 819-827. PMLR, 2013.
14

---

Supplementary Material
LaSCal: Label-Shift Calibration without target labels
The Supplementary material is organized as follows:
Details about the datasets use for the experiments A.1
Implementation details A.2)
Additional experiments with LaSCal (AA.3)
The proposed estimator with different importance weight estimators AA.4)
A
Experiments
In this section, we include more details about the used datasets and training procedures. We also report additional experiments to evaluate the performance of our proposed method using different importance weight estimators.
A.1 Details about the datasets
We report statistics for all datasets in Table
CIFAR-10/100 KKrizhessky et al. 2009 Cao et al. 2019]. Using the CIFAR datasets we examine two different types of label shift. In Task the source distribution is long-tailed, whereas the target is uniform. In Task B the source distribution is uniform, and the target is long-tailed. The CIFAR-10/100 Long-Taie -aig datasets are simulated from CIFAR-T0/100 1001000 respectively, with different imbalance factors (IF). The IF controls the ratio between the number of samples in the most frequent and the least frequent class. For example, an imbalance factor of 10 indicates that the least frequent class appears 10 times less than the most frequent one. In Figure we show the number of target images per class on the long-tailed CIFAR-10/100 with imbalance factors ranging from 1.25 to 100.
In Task A we keep the target distribution unchanged (i.e., balanced across classes), and we resample the source distribution with various IF. In the main paper. we presented setting with source IF 10 in Table Additional results using different imbalance factors induced on the source distribution are given Tables 11
16 In Task the models are trained on the original (balanced) CIFAR datasets, and in Table T0 in Appendix A.3 we report the performance of our CE estimator on label-shifted target distribution with various imbalance factors.
Table 3: Statistics for all datasets used in the paper. Note that report the original number of classes
and samples of the datasets we use.
Dataset
Modality Num. classes
Train samples
Val samples
Test samples
CIFAR-10
Images
40,000
10,000
10,000
CIFAR-100
Images
100
40,000
10,000
10,000
Camelyon17
Images
2
302436
33560
iWildCam
Images
182
129809
7314
8154
Amazon
Text
245502
46950
46950
Camelyon17 [Bandi et al. 2018 consists of x 96 whole-side images (WSI) of breast-cancer metastases in lymph node sections collected from hospitals in the Netherlands. In each WSI, the tumor regions are anmotated manually by pathologists. The labels indicate whether the central 32 x 32 region contains tumor. As Camelyon17 contains only training and validation set cot-drawn from the same (source) distribution-bow of which are balanced across the positive and negative class, perform the following: (i) We use the validation set testing dataset, which convert to desired target distribution by resampling the positive class; (ii) From the training dataset, we allocate validation dataset with the same size the testing dataset. Then,
subsample the validation dataset the same way as we subsample the training dataset, so that both are effectively drawn from the same (source) distribution (e.g., used in the ablation studies in Section 4.3).
iWildCam BBerry al. 2021 consists images obtained from animal camera traps (i.e., heat or motion-activaten static cameras placed in the wild) which are set in countries in different parts of
15

---

Number of imiags class
1,000
Number of images per class
100
800
80
600
60
400
1F=1.0
40
1F=1.0
1F=1.25
IF=1,25
200
1F=2.0
20
1F=2.0
1F=10.0
1F=10.0
1F=100.0
1F=100.0
4
8
20
40
60
80
100
Class index
Class index
(a) CIFAR-10
(b) CIFAR-100
Figure 5: Number of target samples per class simulated long-tail CIFAR-10/100 datasets with
different imbalance factors (IF).
the world. The label of each image is one of the 182 animal species. The training and validation set features the same long-tail distribution among classes, As testing data from the target distribution, we use the original test dataset and we perform the following: (i) we keep only the 20 most frequent classes; (ii) we resample the dataset such that classes follow a uniform distribution (thus representing the target label-shifted distribution), with an imposed minimal frequency of 84 samples per testing class. Therefore, the number of samples in the test dataset (target distribution) is 1680, while the validation set (source distribution) contains 6003 samples.
Amazon Ni et al. 2019 consists of texts which represent user reviews, while the label 1-out-of-5 score of the review. The training and validation set (the source distribution) follows the same long-tail distribution among classes. As the testing dataset is also long-tail, we resample each class based on the frequency of the least frequent class, yielding representing the target, where each class appears 527 times. Therefore, the number of samples in the test dataset (target distribution) is 2860.
set following uniform distribution of classes,
A.2 Implementation details
conduct experiments on consumer-grade GPUs, that is, all experiments can be conducted on a single Nvidia 3090. We use Py PyTorch [Paszke et al. 2019) for all deep-learning-basee -based implementations. Below we provide further information about the training procedure for each of the datasets, along with implementation details of the weight estimators use.
CIFAR-10/100 (Long-Tail), We keep the same training procedure for both CIFAR-10/100 and their loig-tail variants. Namely, we train all models with stochastic gradient descent (SGD) for 200 epochs, with a peak learning rate of 0.1, linearly warmed up for the first 10% of the training, and then decreased to 0.0 until the end. We apply weight decay of 0.0005, and clip the gradients when their norm exceeds 5.0. During training, we augment the images by applying random horizontal flips.
WILDS datasets (Camelyon1 17, iWildCam and Amazon). We keep the same training procedure across all WILDS datasets, with the only difference across datasets being the data augmentation and the used models. Namely, we train all models with Adam W Loshchilov and Hutter (2017) for 10 epochs, with a peak learning rate of 0.0005, linearly warmed up for the first 10% of the training and then decreased to 0.0 until the end. We apply weight decay of 0,001, and clip the gradients when their norm exceeds 5.0. During training, for models trained on Amazon we do not apply any data augmentation on the input text, while for models trained on Camelyon17 and iWildCam we obtain random crop of the image with size 224 224, perform horizontal flipping, and apply color jitter with parameters: brightness=00.6. 1.4), contrast=(0.6, 1.4), saturation=(O.6. 1.4). During testing, we obtain a single crop with size 224 224 from the center of the image. For all ImageNet pre-trained models we use Timm [Wightman et al. 2019) (Camelyon17 and iWildCam), while for all pre-trained language models, we use HuggingFace Face transformers (Wolf et al. 2019]. On Camelyon17 and iWildCam, we train a diverse set of transformer moeedsodels which pre- pre-trained on ImageNet:
16

---

ResNet50 [e et al., 2016 ViT- Large and ViT-L with input resolution of 384 Dosovitskiy et al.
2020]. and LaineLarge [ et al. 202T]. On Amazon. we train different transformer- based language
models: BERT thert-base-uncasee [Devlin et al. 2018), D-BERT
[Sanh
et al.
2019. , RoBERTa (roberta-base,
[Lu et al. 2019]. and D-RoBERTa ddistirobeertabbass |Sanh
et al.
2019
Weight estimators. Our proposed method relies on estimating importance weights using techniques from the unsupervised domain adaptation literature. Most of the weight estimators (RLLS (Azizz zadenesheli al. 2019]. BBSL [Lipton et al., 2018 EM- EM-BCTS [Saerens et al., 2002 that we use are implemented in ht https: //gtthu. com/ jundaj elab/ abstention For the ELSA (Tian et al., 2023] method, we used the original implementation provided by the authors. In some of our settings, we detected issues with the weight estimation methods, prompting us to set a minimal value of the confidence scores to 1 x 10-15 for EM-BCTS, order to get reasonable estimate of the weights in most cases. We also encountered issues with the BBSL method on the iWildCam dataset, due to the source distribution containing O-frequency classes. RLLS consistently delivered the most accurate and stable weight estimations, thus, we report our main results using this method. Note that in certain experiments, some of the importance weight estimation methods (e.g., BBSL) yield poor estimates, resulting in abnormal values for the calibration error. However, addressing these issues is beyond the scope of this paper, as they are specific to the weight estimation methods, and not with our CE estimator.
10-3 for ELSA, and x 10-2 2 for BBSI, in
A.3 Additional experiments with LaSCal as a post-hoc calibration method
In Table report additional performance evaluation of LaSCal compared to other post-hoc calibration strategies on CIFAR-10/100, where the model is trained on long-tail source distribution obtained with various imbalance factors (IF). We report CE on the balanced target distribution. The missing values of the Taall method are due to singular matrix error encountered when running the method, using the original code of the paper.
Table 4: CE on label-shifted target domain before and after calibration with various post-hoc methods.
Model
Uncal
TempScal
VectScal
Fir TempScal
IROvA
CPCS
TransCal
HeadToTail
LaSCal
CIFAR-10-LT
55)
R2
8.38_
4.71gare
1.
4.96 in
5.55 so
4.90gg 20
4.672017
4.601g0.1
1011
ReaNot-32
10.3840.25
5.11
5.31
5. 5.78
6,2846.2 20
5. 5.6320.11
5.99
1.6820.14
1 1669-.15
Re5Net-56
11.864001
5.47.0.17
5.1720
6.5346. ar
5.60 16
7.19
4.76
4.74
ResNet-110
13.19
5.691a
5.77
5.33_oas
6.80
5.70go.a0
9.37
188 m.33
CIFAR-100-LT (IP
ResNer-20
26 2...
27. 20
27. 20
31. 31.14g022
26.26
47 47.48:a25
21.60
5.97=
R2
71.32+0.0
28.75
28.84 28.8440
29.54.0.00
31.08+3.24
27 27.66_0.25
57.39 57.392n224
28.6740
6.19-..0e
ResNet-56
73.07 ao
33.18 28
30
32.57 20
29.03g00.
61. 13
33.1222 07
6.47_0.00
ResNet-110
73.99
35.23 28.00.2
30.71
31.29 10.27
32,15
29.58 00.28
62 22.93_a07
6.61
CIFAR-IO-LT (IF
ResNer-20
9100-=0.14
5.55g0.1
5.48
5.96
6.30
5.87
5.28
4.80
R2sNet-32
11.0246.4
6.29
6.05 14
6.85 12
7.22 a1
6.61.o.11
6.58 14
5.39_ 13
ResNet-56
13.80
6.81 10
6.71
9.52 9.52 po.is
7.14
9,1810m
5.43 .0.11
ResNet-110
13.03
7.23
7.19
6.84gam
8.28
g.
9,28
6. 6..4
CFFAR-100-LT
ResNet-20
27.35
32.12-2.50
32.82+0
33.10.0.29
37.73
32.23-0.29
42.40an029
27.9840.0
6.64_0.0e
R2sNet-32
70.5346m
33.85_0.37
34 66.60
24.44g0.31
37.08
32.8 12.83
56.77g006
29.74g0 36
6.90_o.07
RcsNet-56
71.42
37.188
36.31
36.92 10.25
37.47
$3.52
63.96 63.661a47
37.1010
7.44
ResNet-110
75.50g0.17
41.3720.7
96
41.02g0.32
10.02
36.00 36.06g0.33
06 06.23g024
41.3220
7.550.00
In Table we report additional performance evaluation of LaSCal against traditional, i.i.d, post-hoc calibration methods. Note that VectScal, EnsTempScal, and IROvA are not specifically designed for label-shift scenarios and rely solely on labeled source data. In contrast, LaSCal is tailored for situations where label shift occurs, leveraging both the labeled source data and unlabeled target data to perform calibration. This enables LaSCal to adapt to changes in the class distribution between the source and target domains, providing better calibration under such conditions, reflected in the reported results.
In Figure and Figure 7 we show reliability diagrams for CIFAR-10 using ResNet-20, and Amazon using DiST respectively, before and after calibration. Similar to the results presented in the main text, we observe that LaSCal obtains lowest ECE.
17

---

Table 5: CE on label-shifted target domain before and after calibration with various post-hoe methods.
Model
Uncal
VectScal
EnsTempScal
IROvA
LaSCal
CIFAR-TO-LT (1F=10)
ResNet-20
8.87 20.38
5.30
4.72 20.20
5.19
1.44
ResNet-32
10.45
5.13
5.43
5.78
4.81 1.
ResNet-56
11.25
5.14
4.69
6.01 6.01 #0.25
4.57 4.57 15
ResNet-110
11.8910.38
1.20
1..7
5.8910.29
4.70
Macro average
10.62
5.19
4.95
5.72
4.63
CIFAR-100-L77 (IF
10)
ResNet-20
65.66.0.28
25.91
25.37 20.20
28.75
5.62
ResNet-32
71.1610.24
26. 26.1440.22
27.8 27.84
27. 27.7640.24
5.80
ResNet-56
72.21
26.67 10.25
28.9510.22
28.52.0.21
5.88
ResNet-1110
72.8010.21
28.3610.28
30.96
29.9610.11
6.19
Macro average
70.47
26.77
28.28
28.75
5.87
Amazon Reviews
RoBERTa
11.44
3.48
4.86
4.88
3.66
DistillRoEERTa
17.82=0.98
3.8440.41
5.27=0.41
4.80
2.72
BERT
20.33
8.47+0.52
7.74+0.50
7. 7.02+0.45
3. 3.62
DistillBERT
22.18
7.36+0.47
6.52
6.40+0.46
3.40=
Macro average
19.19
6.79
6.10
5.78
3.38
WWilCam
ResNet50
18.44
21.10
16.61 58
16.07
13.07:0.45
Swin-Large
22.07
20.89
17.36
16.499.0.69
10..51
ViT-Large
17.94 17.94
20.96
17.07 10.69
16.87 10.86
13.07 10.50
Vit-Large (384)
18.99
21.6411.07
18.69
16.49
17.27 10.68
Macro average
19.36
21.15
17.43
16.48
14.71
Pofest Calination
Perlct Lathrastee
Perfect Calbratio
Petfect Caltbratio
Oktpal
Ourpet
Cap
0.1
0.
02
02
201
0.0
04
0..
04
00
Coufideace
Confidence
Coufidence
Confdence
(a) Before calibration
(b) TempScal (Source)
(c) Taid
(d) LaSCal
Figure 6: Reliabiliy diagrams on CIFAR-10 using ResNet-20 before and after calibration.
1.0
10-
18-
1.0
236 Slap
08
68
0.8
06-
0.6
06
0.5
04-
0.4
02-
n2
BCB-23.31
ECE=4.9 1.97
ECE-3.99
BCE=3.63
30
0.
0.0
02
9/
0.5
03
03
02
0.4
0.0
02
60
0.8
Confidence
Confidence
Confidence
Confidence
(a) Before calibration
(b) TempScal (Source)
(c) EM-BCTS
(d) LaSCal
Figure 7: Reliability diagrams Amazon using DostillRoBERTa before and after calibration.
As some of the post-hoc calibration strategies are not accuracy-preserving. in Tables to 8 we report accuracy for the uncalibrated model, as well as for vector scaling, and isotonic regression. We observe that the change in accuracy is negligible in most cases.
18

---

Table 6: Accuracy of CIFAR-10/100 before and after calibration with non eecuracy-perserving
calibration methods. The error bars represent 95% CI.
CIFAR-IO-LT (IF=10)
CIFAR-100-LT (IF=10)
Uncal
VectScal
IROvA
Uncal
VectScal
IROvA
ResNet-20
28.84
77 77.34 77.18
44.81 44.87
44.0040.97
43.91 10.97
ResNet-32
80.7440.77
80.7440.77
80.61 10.77
47.73
46.55.1..8
46. 46.68.10.8
ResNet-56
81.71
81.09
81. 81.2540.77
47 17.1840.90
46. 15,7764..00
46, 46.3040.00
ResNet- 110
81.29
81.2220.77
81.0940.77
49.78+0.98
49.22 49.22+0.98
49. 49.35
Table 7: Accuracy on iWildCam.
Table 8: Accuracy on Amazon.
Uncal
VectScal
IROvA
Uncal
VectScal
IROvA
ResNer50
68.3312.22 10
68.69
RoBERTa DistillRoBERTa
56,2611.82
55.00
54.4811.8
Swin-Large
67. 07.0222.25
69.52.2.20
56.9921.8
35.63
50.91
ViT-Large
66, 26.66+222 25
63 90
64.52=2.29
BERT DistilBBERT
53.3611.88
12.48
52.13
Vit-Large (384)
69.6412
66.6112.26
67. 22.23
55.03
53.99
53.64
In Table 9 we show the full comparison of LaSCal and other post-hoc calibration methods, in setting where additionally to the label shift, the input distribution changes. Across most settings, our method achieves best second best calibration error.
Table 9: Label shift with changing input distribution p(X)
Model
Uncal
TempScul
EncTempSaal
IROVA
CPCS
TranCCal
HeadTooaail
LaSCal (TS)
Amazon Reviews
RoBERTa
11.91
5.02.m2s
5.49
19.7
0.96
1.16
1.38 18.8...3
1.38
3.58
DiCEI
19.30
5.50
6.1lgoni
5.50
5.00
3.37
8,30
2.35 ia
2.44
BERT
28. 2..00...s
19
8.4220
8.08
6.90
3.87
17.7780.50
3.12 17
3.3420.20
DiSUIBERT
25. 25.4120.50
7.50
8 8,20g
7.48.0.3s
2.07
4.16g0.00
14. 14.00a0.00
3.19:
3.36.2..1
Macro average
21.18
3,89
tWildCaw
ResNe150
17.28..0.s
16. 18.69,.. .0.3e
17. 17.25, 1o.34
3.75
13. 13.51....0
15.35
16.13 10.35
Swin Large
26.75
18. 18.84400
22 22.26.004
18. 50.00
4.85
38. 65
17 17.4920.34
17.20
13.86
Vit-large
16.0610.35
15.08
14.48_oas
15.08
14.03_oa
22.43.6.5
11.961n
11.76
Vit-Large (384)
20.03
19.0220a7
19. 19.18
19.05
4.30 es
21.00 40
29 20.35
17.7240.55
15:91 20.30
Mocro everage
21.07
17.54
18.15
17.55
6.86
29.68
14.56
16.09
14/11
Additionally, several related works Tian al. 2023 Alexandari al. 2020 Lipton al. (0188] Azizzadenesheli et al. [2019) predominantly focus on an alternative type of label shift, where the source balanced (i.e., the classes have equal frequency), while the target follows long-tail distribution. We report results for such settings in Table 10 where we induce label shift on the target data with imbalance factors with magnitudes: 10 and TO0. The accuracy remains unaffected by the induced label shift. As before, we perform experiments with ResNet models of varying depths to verify that our findings generalize across models of different complexities. The results on both datasets reveal that the estimator yields reliable CE values in the absence of labeled target data, irrespective the IF intensity.
A.4 Efffet of different importance weight estimators
In this section, we report additional experiments to assess the effectiveness of our proposed approach using various importance weight estimation methods: RLLS, ELSA, EM-BCTS, and BBSL.
In Tables II I6 we report accuracy, ground truth CE (using labels) and estimated CE using different CIFAR-I0/1001
importance weight estimators. The models are trained to different IF imposed on the source distribution, and we report CE with different Lp norms: L1 or L2. For all experiments, the target distribution is uniform. The subscripts and denote the source and target distributions, respectively.
Each table corresponds
2). we observe a comparable performance across weight estimators. However, under more pronounced label shift: Table II (IF 10), in several settings encounter issues with ELSA, EM-BCTS and BBSL
When encountering less severe label shift: Table 12 (IF 5) and Table 13 (IF
19

---

Table 10: Performance evaluation of our estimator on a label-shifted target distribution using different imbalance factors, with models trained on balanced CIFAR- 10100000000 Across both shifts, LaSCal (CF,,) yields accurate estimates compared to the ground truth (with labels), and effectively handles even the severe case with = 100. The error bars represent 95% CI for Acc, and average standard deviation across classes for CE.
IF=10.0
IF=100.0
Model
Acc,
CE₂
CE2
CE
CE₁
CE₁
CIFAR-10
ResNet-20
87 20.65
8. 8.7
a.95
8.70
10.77=0.50
12.13+v.72
ResNet-32
88.47 60.63
10.7
9.36+0..2
9.71 9.71
10.57=0.41
10,88
ResNet-56
88, 20.633...2
10.02...14
8.47
8.82:10.48
10,10=0.55 0..55
10.78.0.53
ResNet-110
90.00
13.74
10.42.0.62
10.36
10.45
10.83
CIFAR-100
ResNet-20
61.19....8
59.15
57 20.42
57.54=0.48
53.83_0.78
53.38
ResNet-32
62. 60.71
67.2440.24
65.1644.46
65.77=0.42
61.10_0.s0
60.20
ResNet-56
65.07 40.93
74.59
72.58 4..35
73.55.0.45
68.67 0.50
69.51
ResNet-110
65.9640.93
76.0440.18
73.83
74.27 #0.38
67.72 0.75
68.1540.53
methods, resulting in abnormal CE values. In contrast, the RLLS method yields stable and reliable values across all settings. The CE estimates obtained using RLLS often closely align with those of the CE estimator that utilizes ground truth weights, denoted wt.
Table 11: Comparison of different importance weight estimators. The source is obtained with an IF 10. We measure L2 CE. The abnormal values CIFAR-100-LT with BBSL and ELSA are due to issues with the weight estimators in this setting.
Model
Ace,
Acc.
CE,
CE2
CBe(w")
(2 RLLS
(P,(2) ELSA
(F,(3)
BM-BCTS
(R,
BBSL.
CIPAR-I0-LT
ResNet-20
83.10.1.1
78.24g0.)
8.19
8.88
9.28
8.93
9.06g0.1
8.57
2.04
ResNet-32
85.4821.0
80.743077
9.18
10.45ga40
11.75:0.00
12. 12.16g0as
12.03:m.30
11.20
12.19
ResNet-56
85.38=105 81.7149.6
9.87
11.24a0a2
11.71:024 11.67aas
62-0..4
11.19:0.40
11.77+0.00
ResNei-110
84.9421.10
31.29
10.06
11.95
12.28g0a5
12.21g0.3
12. 12.18g0.33
11.6940.47
12:26:40
CIPAR-100-LT
ResNet-20
52.2421.07
44.81
61.28
65.67=024
65.63-02
63.9720as
96.61 96.61
63.09
(34078.52214624
ResNet-32
53.48
47.73go.s
65.99
71.24gn06
71.42
70.03g0:4
83.0520.31
69.08
464.12
ResNet-56
14.21
17.18
68.1210a7
12121
72.461a16
71.33gaas
1048.32mm.0
69.89 10:21
12147. 22147.40jm0 107055 30s.5
ResNet-110
56.58
19.784ams
68.8720.45
72.87
73.03
72.11
77.34 77.34
70.7240.15
82,88
Table 12: Comparison of different importance weight estimators. The source is obtained with an
IF = 5. We measure L2 CE.
Model
Ace,
CE
CE2
(i,(w))
(G) RLLS
(E,(#) ELSA
(G, EM-BCTS
CE
BBSL
CIFAR-I0-LT
RexNet-20
84.7720.99
82.77
7.21 20.23
8.40
8.93
8.80
9.07
8.75
8.92
ResNet-32
86.68gaas
83.4710.78
8.03
10.36
10.40ga.3
10. 10.44.0.24
10.4] 20
10.17
10.47
RexNet-56
86.03
84.174n.72
9.50
11.8240.00
11.7440.2
11.6820.28
11.7740
11.582o:2
11.73
ResNet-110
86.44gome
85.04g0.70
9.99
13.17
13.52
13.54g0:30
13.58ga00
13.35go.35
13. 13.47
C10AR-100-LT'
ResNet-20
52.92.1.39
50.39
62.491n.32
65. 19
65. 66.161
65.74g0:3
67.02
64.192007
28.5620
ResNet-32
55.52
50.61g0.0s
68.51
71.24g0.25
71.35+a.18
72.05
73.16.0.ur
70.17:0.19
74.01=0
ResNet-56
56.5941.38
53.3340.95
70.2740.a2
73.1620.14
73.1740.5
73.6020.28
75.48
71.9620.18
75.6220.30
ResNet-110
57.26
54.16gguss
71.39
73.85
74.10
74.05
74.95
72. 12.45
76.00
20

---

Table 13: Comparison of different importance weight estimators. The source is obtained with an IF 2. We measure Ly CE.
Model
Ace,
Acc,
CR2
CV2
CFr(w")
(R,(5)
CRf(a) ELSA
(R,(
(R,(2)
RLLS
EM-BCTS
BBSL.
CIFAR-IO-LF
ResNet-20
86.7 86.74
85.7820.68
5.27
9.05
9. 9.48
9.
9.59
9.55
9.54
ResNet-32
86,94
86.8220.05
a.59
11.00
11.142021
11.03
11.01 11:0
11.00g0.2
ResNet-56
88.4140.74
37.93
13.80 7.84
13 13.85
13.9440.39
13.94
13. 13.93
13 13.9320.a5
ResNet-110
88.33gu.74
20.0
13.00
13.83 13.83
13.99
13.93
13.7920.51
14.0120.47
CIFAR-100-LT"
ResNet-20
56,81+1.15
56.71
60.11
61.43
61.17 61.17
61. 61.76go.18 61.80+o.18
61.15+0.20
123.7121.2
ResNet-32
58 58.43+1.14
58.59
70.48 20.46
71.2440.15 20
21:33
70.2220.22
71.6620.16
ResNet-56
60.12+1.13
59.9620
23.93
71.4920.7
24.3020.19
71.86
74.07 a1.71
74.8620:2
ResNet-110
62.88
61 20.3
74.9320.2
75.45
20.30
75.74
75.8840.15
75.07=0.10
76.68
Table 14: Comparison of different importance weight estimators. The source is obtained with an IF 10. We measure L1 CE.
Model
Acc,
Acez
CE,
CF2²
(i,(w")
(B(( RLLS
(E(() ELSA
(E.(2)
(E,(2)
EM-BCTS
BBSI.
CIFAR-IO-LT
ResNet-20
13.10
28.24
31.56
35.87 ds
35.49
34. 24.00
34.90 20.50
25
35.04 20.04
R2sNet-32
85 85.48_1.05
80.74.0.77
32.77
37.58.g0
30.21 30.21
39. 39.48
39.35
40.53g0.77
39 39.46..0.0
56
85.38_1.0
81.71
13.831g15
38.01 35
18.533
39.385n
39.022 20.2
ResNet-110
119
81.29
31.2111.z
38.97 1oTioss
39. 19.66
39.68 39.6810.s
39.4610.u
10.81 just
39.681..8
CIFAR-100-LT
ResNet-20 20
62.24_1.07
44.81
145:74g0.54
157 267.00
156. 73.0.30
147. 16
198. =8.77
150.78
5083.67
ResNet-32
53.48_1.77
47.73220s
151.01.0.00
162.27 162.27
60.
154.11
157.72-n.. 045.44
400.142a.60
ResNet-56
54.21.157
47.18
151.08g0:57
163.37
164.01
155. 15
509 1.44
15% 69
1504 11:00
ReaNct-110
56.58
19.78guse
153.41gu.50
163. 163.44gma
163.65gu.av
156. 95..1
169. 1322
158. 98 28.33
175.28 10.19
Table 15: Comparison of different importance weight estimators. The source is obtained with an IF = 5. We measure L1 CE.
Model
Acc,
Acc;
CE2
CE2
(E,(w))
(F,(2)
( ELSA
CF,
CF, (a)
RLLS
EM-BCTS
BBSL
CIFAR-10-LT
ReaNet-20
$1.77
82.77
28. 28.66_o.54
29.991..47 .7
31.26 $1.26__oa
31.75
32 22.25 1o.s6
32.60) 1
31. 86
RexNet-32
86.68_083
83.47-0.73
28.84.077
32 28.m
33 33.20120007
33 20
33.883 s5
23.226...00
ReaNet-56
56.03 nas
81.17
32. 32.7111.a5
36.7610.8
36.73 36.71_n.s
35.96 10.54
36 12.3.
36.67 10.57
35. 96 10.57
ResNet-110
86.4420.44
85.04_a.70
33. 33.1021.54
37.97aas6
38 20.92
35.15.0.00
39. 39.03
39.21.g074
CIFAR-F00-LT
ResNet-20
52.92_1.39
50.39_0.58
149.05
155.00=0.31
154.74.0.10
153.35=0.18
157. 157.8320.16
152.64g0.33
ResNet-32
65.52_1.39
50.64_o.ss
156.641..00
161.56.0.3 37
161.79 161.79
160.31
163.52 163.52g..16
159.33_0.10
164 (64.22.1
R6
56.50-1.58
53.33-00.8
157.25
163.20ga.25 25
163.5540.1s
161.8820.77
166.54.0.1s
160.8420.16
165.55
ResNet-110 10
57.26_1.34 54.16_0..s
157.
163.7660.16
164.03ge.11
162.17ga.1
164.27
160.80_0.30
165 165.0331..0
Table 16: Comparison of importance weight estimators. The source is obtained with an IF 2. We measure L CE.
Model
Acc.
Acca
CE2
CE2
(w")
(E2(2) RLLS
CE,(C) ELSA
(E.(2)
CE, (a)
EM-BCTS
CIFAR-10-ILT
ResNet-20
86.74207s
85.78-0.68
22.602o.04
28. 28.866g0.0 40
29. 2..5
30. 10.7720 se
30 20.66an.s
30.28
30. 30.2622..0
ResNett32 32
86.94_0.78
23.62=o.72 86.82-a..6
32.73+0.1
33 =o.22
33.0420.54
32.79.2.11
32.84g0
33.06
R6xNet-56
88.41
88.41_074
87.93_064
26. 26.15go.0s
36.6612771
36.76_o.85
37.15 90
36.992 sp
37.18g074
37.01g0.66
ResNet-110
88.33 _8.33_037 n.74
87.51 nos3
26. 26.75_o.07
36.80
37. 25. 38.23 [o
37.83 11.m
37.37
38.15 18.10_1..0
CIFAR-100-LT
ResNett-0
56.81=1.1
56.71
146. 174.777
148.09 148.09
147 48.24
147.7 147.77
148.02 148.02.a..9
146 146.962n.22
181.7746.22
ReaNot-32
58. 1.14
58.59 uus7
158. 10
158. 12
139.20 139.201qo2
159.41gu.1
159.73..aa
158. 158.42.m.a
160. 16
ResNet-56
00 00.42-1.13
59.96-n.50
161 20
162.96=n.15
163. 1240.15
163. 1820.1
163.21
162.60am.14
163. 163.194c..0
ResNet-110
62.88_1.12
61.90=a.5
162.7240.1b
163.9620.08
164.01
164.0620.14
164.18go.11
163.30=a.12
163.98
21

---

B Top-label calibration
In the main paper, we focus on classwise calibration error, as it provides more comprehensive measure of calibration by assessing the alignment of the model's confidence across all classes, rather than just the highest prediction. However, top-label calibration is widely used in the literature, so we demonstrate here how our estimator can be extended to handle this form of calibration.
For top-label calibration, to the top prediction. As before, the class labels are represented as one-hot encoded variables Y E (el...., 0 C A where & is the one-hot vector corresponding class The top-label 1 calibration eπror is Kumar et al. 2019 Kull et al. 2019 Gruber and Buettner 2022]:
focus the maximum score Q max (X))) which corresponds
TC),(f)
||| = Carg max f(X) | [Q] - -01"
(8)
We aim find estimator of the form:
m++n
TCP,
Σ J=n++
[ = Carg max f(x)) qz]
(9)
m
where the expectations are taken w.r.f. the target, and zj denotes the top-label prediction for input Tz. For the estimator of the conditional expectation we compute:
Ep [] Carg max f(x)) Q = E == Eies, wx((Q 464) (y: arg max f(x))
E (QQ q- 4))
(10)
where Se is the subset of samples where the true label is c, and is an indicator function returning if the predicted label matches the true label for sample T;, and 0 otherwise.
Table 17: Amazon experiments
Model
Uncal
TempScal
Taad'ooTail
RM-BCTS
CPMCN
LaSCal
RoBERTa
6.03
1.14
0.93
0.40
0.37
0.29
BEETA
10.73.10.63
1.83
0.58 12
0.43
0.66
0.42
BERT
17.32.0.67
3.71 20.35
0.73
0.75
2.19
0.61 10.12
DistilBBERT
13.5940.72
2.22-0.25
0.33
0.37
1.72
0.30
Macro-average
11.4240.39
2.23
0.6420.08
0.49
1.24
0.41
Table 18: iWildCam experiments
Model
Uncal
TempScal
HeadToTail
EM-BCTS
CPMCN
LaSCal
ResNet50
3.21
1.66
1.06
0.64
2.40
0.74
Swin-Large
5.92
1.88
1.07
1.48
1.17
1.1720.26
ViT-Large
2.4340.41
1.53
1.48+0.26
3.32+0.43
2.34
0.62
ViT-Large (384)
2.69
1.964
1.85
1.93
2.18
0.81
Macro-average
3.56a0.41
1.7710.26
1.37.0.27
1.84
2.02
0.84.10.12
The results presented Tables 17 and 18 show that the observations remain the same as in the main
paper: (i) LaSCal significantly reduces the CE of all models across datasets; (ii) LaSCal outperforms the baselines. achieving state-of-the-ane results on the datasets and settings we experiment with.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state our contributions: a novel CE estimator, a post-hoc method for unsupervised calibration, and extensive empirical validation. The assumptions and scope (label shift scenario) properly defined.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in a separate paragraph in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification: To the best our knowledge, assumptions are provided. The pointwise consisteney results are supported by sketch proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The method, datasets, metrics, and models described in sufficient detail in the main text. Appendix A.1 and A.2 contain more details about the datasets and the implementation. The code released at: https://github. httpss//ggttbb...c/pppppraaaaoop
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
24

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
All datasets use are publicly available, and the code is released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips. catpp.cccpubblle wwtpppppc/gguupaeecceceoeooocoee dutodddde///ooeeummiiiieee Ctt///oooemmmssiieeeeee sww.bbmiiiiioooccccccc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The main text specifies enough details to understand the results. The full detailed information about datasets, hyperparameters and other implementation details is given in Appendix A.1 and A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The main results (Table and Figure 3) report mean and standard deviation values obtained using bootstrap (repeatedly resampling with replacement and estimating CE on each subset).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In appendix A.2 we provide information about the resources used in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/neurrpssccc///////// cc http://wwpppppppopoii/ccaaaaaaaa
Answer: [Yes]
Justification: To the best of our knowledge, the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impact and ethical risks are discussed in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper will not release data or models with risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We appropriately cite the creators of all datasets, models, and code packages we use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "kDvdZc6izY", "reviewer": "Reviewer_vmtH", "summary": "This paper proposes a consistent estimator of class-wise expected calibration error (class-wise ECE) for unsupervised domain adaptation under label shift assumption, i.e., the class proportion of the source $p_s(y)$ and target distribution $p_t(y)$ differs while the class-conditional probability $p(X|y)$ remains the same. In this problem, estimating class-wise ECE is not straightforward as target domain data is unlabeled. The proposed method LaSCal suggests estimating importance weight using existing methods and then incorporate it in the validation objective. Once we have validation objective, one can use a simple post-hoc method such as temperature scaling to tune the temperature  to minimize validation class-wise ECE objective. Experiments show that the proposed consistent estimator performs better than baselines that were not designed for label shift scenario.", "questions": "1. Is it possible to extend the proposed techniques for tuning different objectives rather than class-wise ECE, e.g., expected calibration error (ECE). I feel that it might also be useful to look into this setting or discuss it in future work. Intuitively, I think it should be possible without much modification from the proposed method. Given the proposed method relies on the accuracy of importance weights, analyzing different objectives under these constraints can also give some insights for reliable confidence learning in this setting.\n2. In (7), it seems we only use target data for the objective. We only used source data for importance weight estimation. I was wondering if it is also possible to consider incorporate source data in this objective as well to improve the performance. What do you think?", "limitations": "The discussions of limitations and future work are appropriate.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method has strong theoretical justification because it is a consistent estimator of the target classwise-ECE.\n2. Experiments clearly showed that the proposed method is effective.\n3. Not only does the proposed method show the best performance in experiments, but several different configurations of the proposed methods were also analyzed (e.g., using different importance weight methods, sensitivity analysis of different ratio of positive/negative, source/target, sample size).", "weaknesses": "1. Given existing techniques for learning under label shift, I might be wrong to find the proposed method is quite straightforward without much difficulty because the estimator is based on the well-known importance weighting method. Having said that, conducting experiments and analyses is still very important and useful as label shift learning is one important kind of dataset shift in unsupervised domain adaptation.\n2. Only classwise-ECE metric is considered for the objective function, although there are many different metrics for calibration error.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback and future work suggestions. We performed further experiments which serve as an interesting addition to our paper. Regarding the questions/concerns:\n\n1. **Proposed method is perceived as straightforward**: While our method builds upon the well-known importance weighting technique, our contribution extends beyond the straightforward application of this method.\n\n- We derive the first consistent calibration error estimator under label shift, without using target labels. Furthermore, the estimator has a known error rate $O(n^{-1/2} + m^{-1/2})$ [L140]. Depending on the choice of kernel, the estimator can be differentiable and integrated as an objective in both post-hoc and trainable calibration methods. To the best of our knowledge, no other estimator exists for target calibration error under label shift with the same properties as ours.\n\n- Based on this estimator, we introduce an accuracy-preserving recalibration method. While some of the papers addressing label shift (*CPMCN, Wen et al. [1]; EM-BCTS, Alexandari et al. [2]*), do incorporate a calibration step, it is performed on validation (source) data, whereas our method performs unsupervised calibration on the target domain.\n\n- We provide a thorough analysis of our method’s properties, demonstrating its robustness across datasets, modalities and severity of shift. We benchmark different weight estimation methods and identify the most robust method for our task. Note that although several weight estimation techniques exist for the label shift scenario, none of them extend their application beyond improving the predictive performance of the classifier. \n\n2. **Alternative objective functions and/or metrics:** Indeed, our method (LaSCal) naturally extends to the standard ECE. In an additional experiment, we adapted our label-free estimator under label shift to estimate ECE and used that as an objective for temperature scaling. The table below reports ECE (with labels) for several baselines: IID temperature scaling, HeadToTail (as representative for methods derived under covariate shift assumption) and the two new baselines proposed by reviewer **Lv5P**, which include most recent works (*CPMCN [1] @ ICLR 2024*) on calibrating models facing label shift. Please see our general response for reasons why we focus on classwise ECE. \n\n### Amazon experiments\n\n| Model  | Uncal | TempScale  | HeadToTail      | EM-BCTS      | CPMCN | LaSCal             |\n|------|--------|-----------|------------|----------|-----------|---------|\n| RoBERTa            | $6.03_{\\pm 0.50}$  | $1.14_{\\pm 0.16}$  | $0.93_{\\pm 0.17}$  | $0.40_{\\pm 0.11}$  | $0.37_{\\pm 0.10}$    | $0.29_{\\pm 0.08}$  |\n| DistillRoBERTa | $10.73_{\\pm 0.63}$ | $1.83_{\\pm 0.25}$  | $0.58_{\\pm 0.12}$  | $0.43_{\\pm 0.09}$  | $0.66_{\\pm 0.14}$    | $0.42_{\\pm 0.11}$  |\n| BERT            | $17.32_{\\pm 0.67}$ | $3.71_{\\pm 0.35}$  | $0.73_{\\pm 0.13}$  | $0.75_{\\pm 0.15}$  | $2.19_{\\pm 0.21}$    | $0.61_{\\pm 0.12}$  |\n| DistillBERT    | $13.59_{\\pm 0.72}$ | $2.22_{\\pm 0.25}$  | $0.33_{\\pm 0.10}$  | $0.37_{\\pm 0.09}$  | $1.72_{\\pm 0.25}$    | $0.30_{\\pm 0.09}$  |\n| **Macro-average**  | $11.42_{\\pm 0.59}$ | $2.23_{\\pm 0.17}$| $0.64_{\\pm 0.09}$  | $0.49_{\\pm 0.07}$ | $1.24_{\\pm 0.19}$ | **0.41$_{\\pm 0.09}$** |\n\n### iWildCam experiments\n\n| Model | Uncal | TempScale| HeadToTail | EM-BCTS | CPMCN | LaSCal |\n|-------|---------|-------|-------------|-----------|---------------|---------|\n| ResNet50         | $3.21_{\\pm 0.40}$  | $1.66_{\\pm 0.27}$  | $1.06_{\\pm 0.24}$  | $0.64_{\\pm 0.15}$  | $2.40_{\\pm 0.34}$    | $0.74_{\\pm 0.17}$  |\n| Swin-Large       | $5.92_{\\pm 0.57}$  | $1.88_{\\pm 0.28}$  | $1.07_{\\pm 0.25}$  | $1.48_{\\pm 0.27}$  | $1.17_{\\pm 0.28}$    | $1.17_{\\pm 0.26}$  |\n| ViT-Large        | $2.43_{\\pm 0.41}$  | $1.59_{\\pm 0.33}$  | $1.48_{\\pm 0.26}$  | $3.32_{\\pm 0.43}$  | $2.34_{\\pm 0.33}$    | $0.62_{\\pm 0.14}$  |\n| Vit-Large (384)    | $2.69_{\\pm 0.40}$  | $1.96_{\\pm 0.33}$  | $1.85_{\\pm 0.34}$  | $1.93_{\\pm 0.34}$  | $2.18_{\\pm 0.37}$    | $0.81_{\\pm 0.16}$  |\n| **Macro-average** | $3.56_{\\pm 0.41}$ | $1.77_{\\pm 0.26}$ | $1.37_{\\pm 0.27}$  | $1.84_{\\pm 0.21}$ | $2.02_{\\pm 0.23}$ | **0.84$_{\\pm 0.12}$** |\n\nFrom the results we see that the observations remain the same as in the main paper: (1) LaSCal significantly reduces the CE of all models across datasets; (2) LaSCal outperforms the baselines, achieving state-of-the-art results on the datasets and settings we experiment with. We will add a summary of these results in the camera-ready version of the paper (and the full experiments in Appendix).\n\n3. **Adding source data to the objective**\n\nIncorporating source data could be useful in the scenario where $p_s(X) = p_t(X)$ (please note that this is NOT implied by the label shift definition). We tested this approach empirically by measuring CE on the target distribution of CIFAR-10 using 3 different estimators: (1) Ground truth (with labels); (2) Our estimator (no labels, no source samples); (3) Our estimator + source samples (no labels, with source samples). The results (reported in the Table below) demonstrate that this approach leads to less precise estimates, i.e., yields higher estimated values compared to ground truth (computed with labels).\n\n| Model      | Ground truth      | Our estimator     | Our estimator + source samples  |\n|---------|----------------|--------------------|--------------------|\n| ResNet-20  | $8.95_{\\pm 0.36}$  | $9.08_{\\pm 0.53}$  | $11.55_{\\pm 0.52}$ |\n| ResNet-32  | $10.46_{\\pm 0.25}$ | $11.97_{\\pm 0.71}$ | $15.37_{\\pm 0.66}$ |\n| ResNet-56  | $11.18_{\\pm 0.29}$ | $11.73_{\\pm 0.48}$ | $15.06_{\\pm 0.50}$ |\n| ResNet-110 | $11.92_{\\pm 0.40}$ | $12.20_{\\pm 0.39}$ | $15.27_{\\pm 0.59}$ |\n\nWe also incorporated the estimator including source samples as an objective function for post-hoc calibration (with the temperature scaling method), and we observed no improvement in the results compared to those reported in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xul5ajnDxP", "reviewer": "Reviewer_Mi4F", "summary": "This work considers the problem of model calibration under label shift with label-free data. The work obtains the unsupervised calibration error through the kernel-based estimation of the TARGET data with the SOURCE data and applies it to the TS calibration method. The problem's has some novelty and is technically feasible.", "questions": "1. What are the potential causes of label shift?\n2. Does label shift have an impact on model calibration? Label shift between training and validation sets does not seem to be reflected by calibration. Can the authors give some explanation for the relationship between label drift and model calibration? In my opinion, calibration methods for unsupervised data are the focus of this work.\n3. Does the result obtained from calibrating the model with a validation set that has data shift negatively affect the results of the training set?\n4. See weaknesses.", "limitations": "see Weaknesses and Questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The article considers the problem of model calibration in unsupervised and label shift for unlabeled data. The problem seems to be interesting and has some novelty.\n2. The article provides a solution to the problem of model calibration under unlabeled data by estimating the calibration error for unlabeled data through a kernel method, which is meaningful.", "weaknesses": "The other methods in the comparison experiments, such as TS, use the source data. Such comparison experiment seems to be not rigorous. In comparison experiments, the validation sets for different comparison methods should remain the same.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review and feedback on our paper. We provide an explanation to the questions below:\n\n1. **Some of the baselines use source data for calibration.**\n\nWhile it is true that compared to i.i.d calibration methods, unsupervised calibration methods (LaSCal, HeadToTail, TransCal, CPCS) have access to samples from the unlabeled target distribution in addition to the labeled validation set (from source), the reasons we included these methods are two-fold:\n\n1. To showcase that traditional i.i.d. calibration methods fall short in the case of label shift and highlight the necessity for designing calibration methods specifically for this scenario.\n\n2. To be consistent with the papers on calibration under covariate shift i.e., HeadToTail, TransCal and CPCS, which also include TS, Vector Scaling, Ensemble TS and IROvA.\n\nIn the revised manuscript, we propose to replace two of these baselines (Vector scaling and ETS) with the baselines proposed by Reviewer **Lv5P** in Table 2, and move the VectScal and ETS to Appendix. We will also group the methods or add markers, to make this difference more visible in the tables. \n\n2. **What are the potential causes of label shift?**\n\nSome of the potential causes of label shift:\n\n- Change in population demographics: e.g. a model trained on mainly adult population (Hospital A) is deployed in a hospital that primarily serves children (Hospital B).\n- Unusual events like disease outbreaks or natural disasters: e.g. during a pneumonia outbreak, $p(Y)$ (e.g. flu) might rise, but the symptoms of the disease $p(X| Y)$ (e.g. cough given flu) do not change.\n- Geographic changes: e.g. consider wildlife classification model for identifying animal species based on camera trap images (as in our experiments), trained on images from one region with a particular animal species distribution, and deployed in a different region with a different animal speciries distribution.\n- Seasonal changes: A model designed to predict seasonal allergies might face label shift if it is trained using data collected during spring, when pollen levels are high and hence allergies are more frequent, and then applied in autumn when allergies are less frequent\n\n3. **Does label shift have an impact on model calibration? Label shift between training and validation sets does not seem to be reflected by calibration.**\n\nModel calibration is defined with respect to a data distribution. When that data distribution changes (e.g., when facing label shift), the predicted probabilities of a model trained on the source distribution may no longer reflect the true empirical frequencies of the target distribution, leading to poor calibration. For instance, if a model trained on a skewed dataset predicts a high probability for a rare class, this probability might not accurately reflect the true frequency of that class (conditioned on that probability) in the new distribution where that class is more frequent.\n\nTables 10-16 in the Appendix confirm that label shift does have an impact on model calibration. For example, in Table 16 we can observe a notable increase of the calibration error from source data (CE_s) to the label-shifted target data (CE_t), particularly prominent for the Amazon dataset. Please note that in our experiments, the training and validation sets (CE_s) are sampled from the source distribution, while the test set (CE_t) is sampled from the label-shifted target distribution.\n\n4. **Does the result obtained from calibrating the model with a validation set that has data shift negatively affect the results of the training set?**\n\nOur proposed post-hoc calibration method is accuracy-preserving, as it performs temperature scaling by optimizing the CE estimator. Since the logits are scaled with a single parameter, temperature scaling maintains the original order of the predictions, ensuring that the predicted class remains unchanged. Therefore, calibration on label-shifted data does not negatively affect the predictive performance of the classifier on the training (source) set.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c3WW6D99fD", "reviewer": "Reviewer_Lv5P", "summary": "This paper proposes a novel calibration error estimator under label shifts (without ground truth). They use this estimator to apply standard calibration techniques such as temperature scaling on the unlabelled target set, allowing to calibrated the model for the target domain without needing access to a labelled set on this domain. Experiments are conducted for different strength of label shifts on various imaging and text datasets.", "questions": "My main point of concern is the missing label shift recalibration baselines, please see weaknesses for details. These missing baselines are the main reason behind my rating. If the authors are able to include these baselines in Table 2 and show superiority of the proposed method over existing label shift recalibration methods, I would be willing to increase my score.", "limitations": "See weaknesses", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* [Clarity] The paper is very well-written, experiments and methods are clearly presented, figures and tables are of high quality. \n* [Relevance] Estimation of calibration error under shift is an important problem, still largely under studied. \n* [Experiments] Experiments are presented on multiple datasets, with statistical errors measures and contains several ablations on the main components of the proposed method. In particular ablations on the density ratio estimator shows that the method is quite robust to the particular choice of estimator.", "weaknesses": "**Pre-rebuttal review:**\n\nMajor:\n* **Missing very important baselines for model calibration under label shift** (table 2). The authors have included multiple baselines to compare their method against, unfortunately their choice of baselines is inappropriate for the type of shift they are studying here. The problem tackled in this work is very clearly stated at the beginning of the paper, the author tackle calibration under _label shift_. However, the baselines they chose to compare against in table 2 are baselines that were designed at improving calibration under _covariate shift_ (TransCal, HeadToTail as the author state themselves in the related work section), which are two very different problems! It is unfair to compare a method specifically designed for improving calibration under label shift with methods designed to tackle covariate shift. There exists methods that are designed to recalibrate probabilities under label shift in the literature but these were completely omitted in the authors' analysis. There is no clear reason why the authors did not include these baselines, even more so since they directly use part of these works in their own method. To cite just a few missing baselines:\n    * The recalibration method from Alexandari et al. In their work the authors proposes to estimate the density ratio $w= p_t(y) / p_t(y)$ via an EM algorithm and then they propose to _recalibrate the outputs_ via $\\hat{p}(y=i|x) = \\frac{w_i \\cdot p(y=i|x)}{\\sum_j w_j \\cdot p(y=j|x)}$, where $w_i$ is the estimated density ratio for class i and $p(y=i|x)$ is the i-th output of the classifier for sample $x$. See section 2.2 of that paper. The authors then use these recalibrated probabilities to get updated classification predictions.\n    * Similar Wen et al, in their work ‘Class Probability Matching with Calibrated Networks for Label Shift Adaption’, proposed another density ratio estimator and recalibrated the probabilities in a similar way as Alexandari et al, see section 4 of that paper. Using their re-calibrated probabilities, they achieve SOTA results in terms of accuracy under shifts. \n    * **These density-ratio based recalibration baselines are the current go-to approach for re-calibration of model outputs under label shifts and should be included in the paper for a fair comparison with relevant baselines.** At least one of them needs to be included in the paper to be able to claim SOTA results in calibration under label shift. \n\nMinor:\n* **Calibration error estimator is data hungry**: in Fig 3.c. we can see that the method is very data hungry and needs at least 4000 samples to be able to perform the calibration error accurately. This should be highlighted and discussed in the limitations section, as this may hinder some practical applications.\n* **Effect of recalibration step on calibration error estimation?**: I would be curious to see if the proposed calibration error estimator is able to estimate the true CE with the same precision after the proposed recalibration step compared to before. I would expect that since the error estimation is based on the estimated density ratio the estimator would be overestimating true CE after output recalibration (since recalibration would close the domain gap). Do the authors have any insights on this?\n\n\n**Update after rebuttal**: \n\nThe authors have successfully addressed my concerns during the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback and suggestions. We implemented these baselines and performed additional experiments, which we will include in the camera-ready version. We address each question below:\n\n1. **Add at least one more relevant baseline for model calibration under label shift**\n\nWe compared our method both with CPMCN [1] and EM-BCTS [2], on CIFAR-10/100, Amazon, and iWildCam.\n\n### CIFAR10-LT\n\n| Model| EM-BCTS| CPMCN| LaSCal |\n|-------|---------|-------|-------|\n| ResNet20  | $3.77_{\\pm 0.12}$  | $3.79_{\\pm 0.11}$   | $4.40_{\\pm 0.15}$  |\n| ResNet32  | $4.99_{\\pm 0.24}$  | $5.19_{\\pm 0.21}$   | $4.78_{\\pm 0.16}$  |\n| ResNet56  | $4.41_{\\pm 0.11}$  | $4.42_{\\pm 0.12}$   | $4.57_{\\pm 0.16}$  |\n| ResNet110 | $4.40_{\\pm 0.12}$  | $4.43_{\\pm 0.13}$   | $4.70_{\\pm 0.16}$  |\n| **Macro-average** | **4.39$_{\\pm 0.22}$** | $4.46_{\\pm 0.20}$ | $4.61_{\\pm 0.16}$ |\n\n### CIFAR-100-LT\n\n| Model | EM-BCTS | CPMCN| LaSCal|\n|--------|------|------|----------|\n| ResNet20  | $25.01_{\\pm 0.21}$  | $25.02_{\\pm 0.24}$   | $5.61_{\\pm 0.08}$  |\n| ResNet32  | $26.17_{\\pm 0.21}$  | $24.76_{\\pm 0.20}$   | $5.79_{\\pm 0.08}$  |\n| ResNet56  | $26.33_{\\pm 0.23}$  | $24.53_{\\pm 0.22}$   | $5.89_{\\pm 0.07}$  |\n| ResNet110 | $28.22_{\\pm 0.24}$  | $26.49_{\\pm 0.22}$   | $6.19_{\\pm 0.07}$  |\n| **Macro-average** | $26.43_{\\pm 0.25}$ | $25.20_{\\pm 0.22}$ | **5.87$_{\\pm 0.08}$** |\n\n### Amazon\n\n| Model  | EM-BCTS | CPMCN    | LaSCal|\n|----|----|----|----|\n| RoBERTa          | $2.72_{\\pm 0.35}$  | $1.36_{\\pm 0.17}$   | $3.64_{\\pm 0.32}$  |\n| DistillRoBERTa   | $2.13_{\\pm 0.28}$  | $2.81_{\\pm 0.23}$   | $2.71_{\\pm 0.25}$  |\n| BERT             | $3.95_{\\pm 0.40}$  | $9.32_{\\pm 0.54}$   | $3.74_{\\pm 0.39}$  |\n| DistillBERT      | $3.41_{\\pm 0.36}$  | $5.48_{\\pm 0.34}$   | $3.40_{\\pm 0.28}$  |\n| **Macro-average**| **3.05$_{\\pm 0.36}$** | $4.74_{\\pm 0.42}$ | $3.37_{\\pm 0.31}$|\n\n### iWildCam\n\n| Model | EM-BCTS| CPMCN | LaSCal |\n|----|---|---|----|\n| ResNet50         | $15.84_{\\pm 0.57}$  | $19.43_{\\pm 0.69}$   | $13.01_{\\pm 0.45}$ |\n| Swin-Large       | $16.81_{\\pm 0.63}$  | $18.03_{\\pm 0.62}$   | $15.19_{\\pm 0.48}$ |\n| ViT-Large        | $24.83_{\\pm 1.31}$  | $19.33_{\\pm 0.80}$   | $13.00_{\\pm 0.43}$ |\n| ViT-Large (384)  | $19.78_{\\pm 0.73}$  | $20.74_{\\pm 0.72}$   | $16.58_{\\pm 0.69}$ |\n| **Macro-average**| $19.31_{\\pm 0.76}$      | $19.38_{\\pm 0.75}$         | **14.45$_{\\pm 0.52}$**          |\n\nOverall, on all datasets LaSCal either outperforms or performs competitively with both CPMCN [1] and EM-BCTS [2], as per the macro-average CE across models (note for CIFAR-10 and Amazon the error bars overlap for LaSCal and EM-BCTS). LaSCal's gains are prominent for datasets which feature a large(r) number of classes (100 in CIFAR-100 and 20 in iWildCam). We hypothesize this is due to the increased complexity of the optimization process associated with higher-dimensional spaces, which our approach seems to handle more effectively. Aside from the empirical gains, our method has the following advantages compared to these methods: (1) is based on a consistent estimator of target calibration error; (2) it enables unsupervised calibration on the target distribution, whereas CPMCN [1] and EM-BCTS [2] perform the calibration step on a labeled validation set (from source).\n\n2. **Calibration error estimator is perceived as data-hungry**\n\nWhile it is true that the estimator requires a sufficient number of data samples (4000 samples in Fig. 3c) to accurately estimate the calibration error (CE), we do not believe that this should be perceived as data-hungry. We do agree that in severely data-scarce settings, this requirement may limit potential applications, and we will discuss this in the revised manuscript.\nHowever, please note that the error rate of our estimator is $O(n^{-1/2} + m^{-1/2})$, which is the same as the weight estimation methods (see Lemma1 for the RLLS estimator in *Azizzadenesheli et al.* [6], and top paragraph on page 8 in *Garg et al.* [4] for the EM-BCTS method from *Alexandari* et al. [2]). Therefore, the data requirement is not unique to our method, but rather is common across all weight estimation-based approaches. Further, Fig. 3c also shows that our estimator has a positive bias in scenarios with limited data. This characteristic is preferable as it prevents the false impression that a model is well-calibrated due to insufficient sample size. In essence, our method errs on the side of caution, ensuring reliability even in data-constrained environments.\n\n3. **Effect of recalibration step on calibration error estimation?**\n\nThank you for the interesting question. The label shift gap is not affected by the recalibration step, and therefore the obtained weights from the weight estimation methods remain the same before and after calibration. Having said that, we hypothesize that the precision of our estimator would remain the same. To gain insights, we performed additional experiments on CIFAR-10, comparing the precision of estimating CE using our estimator, compared to ground truth (with labels) before and after the re-calibration step with LaSCal. The empirical results we obtained seem to confirm our hypothesis.\n\n| Model | Uncal | LaSCal |\n|-----|-----|---|\n| ResNet20 (w/ labels) | $9.01_{\\pm 0.34}$  | $4.43_{\\pm 0.16}$  |\n| ResNet20 (w/o labels) | $9.15_{\\pm 0.50}$ | $4.48_{\\pm 0.25}$  |\n| **Absolute Difference** | **0.14**| **0.05**|\n| ResNet32 (w/ labels)  | $10.41_{\\pm 0.44}$ | $4.76_{\\pm 0.13}$  |\n| ResNet32 (w/o labels) | $11.94_{\\pm 0.49}$ | $6.05_{\\pm 0.41}$  |\n| **Absolute Difference** | **1.53**  | **1.29** |\n| ResNet56 (w/ labels) | $11.18_{\\pm 0.23}$ | $4.56_{\\pm 0.14}$  |\n| ResNet56 (w/o labels)| $11.63_{\\pm 0.32}$ | $4.99_{\\pm 0.17}$  |\n| **Absolute Difference** | **0.45**| **0.43** |\n| ResNet110 (w/ labels)   | $11.86_{\\pm 0.26}$ | $4.71_{\\pm 0.14}$ |\n| ResNet110 (w/o labels)  | $12.15_{\\pm 0.29}$ | $4.98_{\\pm 0.18}$ |\n| **Absolute Difference** | **0.29**| **0.27** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G9lbQYrEQp", "reviewer": "Reviewer_z7nx", "summary": "The paper addresses the problem of confidence calibration under label shift given unlabeled samples from the target domain. The first step is estimating the label distribution of the target domain. Then a calibration parameter is computed using the source domain samples  where each sample is reweighted according to its label probability ratio in the source and the target domain.", "questions": "Can you clearly state the novelty compared to covariate shift methods?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method makes sense and yields good calibration results.", "weaknesses": "My main concern is a lack of novelty. The standard calibration methods in case of domain shift (CPCS, TransCal) are based on importance weighting of a labeled sample (x,y) from the source domain according to its similarity to the target domain p_t(x)/p_s(x). Here the proposed method applies the same principle to the label shift problem.  Given a labeled sample (x,y) from the source domain, it is reweighted according to the ration  p_t(y)/p_s(y).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Can you clearly state the novelty compared to covariate shift methods?**\n\nPlease see our general response for the perceived lack of novelty. To summarize:\n\n- This paper introduces the first label-free, consistent estimator for target calibration error under label shift.  Estimating calibration error under covariate and label-shift are two very different problems.\n- The re-weighting strategy in label shift is based on the label distribution ratio, which is fundamentally different from the feature distribution ratio used in covariate shift methods.\n- The paper provides both a strong theoretical justification and a thorough empirical analysis, demonstrating the effectiveness of our approach, and showcasing that existing covariate shift methods are suboptimal under label shift (see Table 2). This result emphasizes the necessity for designing recalibration methods tailored to label shift conditions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank the reviewers for the constructive feedback on our paper. We appreciate the provided insights and are pleased to see the recognition of several strengths in our work. Below, we summarize the main strengths and weaknesses highlighted by the reviewers, and address them accordingly.

### Strenghts

- **Theoretical justification and novelty:** Our method has been acknowledged for its strong theoretical foundation as a consistent estimator of the target classwise-ECE (**vmtH**). Additionally, the novelty of addressing unsupervised model calibration under label shift was noted as a meaningful and interesting problem (**Mi4F**).
- **Comprehensive experiments**: The comprehensiveness and clarity of our experiments are well-received by most reviewers. Reviewer **vmtH** noted that our experiments clearly showed the effectiveness and superior performance of our method, while reviewer **Lv5P** praised the robustness of our method. Additionally, both reviewers (**vmtH** and **Lv5P**) compliment our clear presentation and analysis which includes multiple datasets, statistical error measures, and thorough ablations.
- **Clarity and presentation:** The clarity and quality of writing, figures, and tables were praised (**Lv5P**), highlighting that our paper effectively communicates the methodology and findings.

### Weaknesses and response

- **Perceived lack of novelty**: Reviewer **z7nx** has concerns that our method, based on importance weighting, could be lacking novelty compared to covariate shift methods.


We would like to emphasize that our work introduces the first label-free, **consistent** estimator of target calibration error under **label shift**. Estimating calibration error under covariate and label shift are two very different problems (also pointed out by reviewer **Lv5P**). Our estimator has a known error rate, and is differentiable when used with a differentiable kernel (*Popordanoska et al. [5]*). **To the best of our knowledge, no other estimator exists for target calibration error under label shift with the same properties as ours**. Furthermore, while importance weighting is a common technique for addressing distribution shifts, the re-weighting in label shift is based on the label distribution ratio $p_t(y) / p_s(y)$, which is fundamentally different from the feature distribution ratio $p_t(x)/p_s(x)$ used in covariate shift methods. Our paper provides both a strong theoretical justification (reviewer **vmtH**) and a comprehensive empirical analysis demonstrating the robustness and clear advantage over covariate shift methods in these conditions. Specifically, Table 2 in the main paper shows that existing methods derived under covariate shift assumption (HeadToTail, CPCS, TransCal) are suboptimal under label shift, which further highlights the need for designing CE estimators and recalibration methods tailored to this type of shift.

- **New baselines**: Reviewer **Lv5P** suggested including two relevant baselines designed for label shift scenarios.

We conducted experiments with the proposed baselines (CPMCN [1], EM-BCTS [2]) and observed that for all dataset / model combinations, LasCal either significantly outperforms or remains competitive with them; therefore these new experiments do not affect the overall conclusions drawn from our paper. Please note that these baselines perform a calibration step on a labeled validation (source) data prior to obtaining the importance weights, whereas our approach allows for unsupervised calibration on the target distribution. See the response to reviewer **Lv5P** for details.

- **Choice of metric**: Reviewer **vmtH** asked if we can extend the proposed technique to different objectives rather than classwise- ECE, e.g. expected calibration error ECE.

We focus on classwise calibration error because it is a stronger notion of calibration compared to ECE (i.e., CWCE = 0 implies ECE = 0, but not the other way around; see Theorem 3.1 in Gruber and Buettner [3]). Note that, we already report ECE in the reliability diagrams in Figure 2 (and Fig. 6 in Appendix), which show that optimizing our CWCE estimator leads to superior performance also in terms of ECE compared to competing methods. As per the reviewer’s suggestion, we adapted our estimator to estimate target ECE without target labels, and performed experiments for the Wilds datasets (iWildCam and Amazon) using ECE as a calibration objective. Overall, our insights and conclusions remain the same as with the original submission. Please see the response to **vmtH** for details and experiments.


*[1] Wen et. al., 2024 @ ICRL: Class Probability Matching with Calibrated Networks for Label Shift Adaption*

*[2] Alexandari et. al., 2020 @ ICML: Maximum Likelihood with Bias-Corrected Calibration is Hard-To-Beat at Label Shift Adaptation*

*[3] Gruber and Buettner 2022 @ NeurIPS: Better Uncertainty Calibration via Proper Scores for Classification and Beyond*

*[4] Garg et al., 2020 @ NeurIPS:  A Unified View of Label Shift Estimation*

*[5] Popordanoska et al., 2022 @ NeurIPS: A consistent and differentiable Lp canonical calibration error estimator*

*[6] Azizzadenesheli et al., 2019 @ ICLR: Regularized Learning for Domain Adaptation under Label Shifts*

Meta-review: This paper introduces a novel calibration error estimator designed for unsupervised domain adaptation under label shift, where class proportions differ between source and target domains, but class-conditional probabilities remain consistent. The proposed method, LaSCal, estimates class-wise Expected Calibration Error (ECE) without needing labeled data from the target domain. LaSCal achieves this by estimating importance weights and incorporating them into a validation objective, which is then minimized using post-hoc techniques like temperature scaling. Experiments demonstrate that LaSCal outperforms existing baselines not tailored for label shift scenarios.

The proposed method is supported by strong theoretical foundations and its effectiveness is thoroughly validated through empirical experiments. Additionally, the authors have effectively addressed most of the concerns raised by the reviewers, leading to a positive consensus. Therefore, I also recommend accepting this paper.
Decision: Accept (poster)