Beyond One-Hot Labels: Semantic Mixing for Model Calibration
Authors: Haoyang Luo
Linwei Tao
Minjing Dong
Chang Xu
Venue: ICML 2025 poster (2025)
Keywords: model calibration
semantic mixing
diffusion models
Primary area: deep_learning->robustness
TL;DR: We introduce Calibration-aware Semantic Mixing (CSM), a data augmentation framework that uses diffusion models to generate training samples with ground-truth confidence, improving model calibration.

Abstract: Model calibration seeks to ensure that models produce confidence scores that accurately reflect the true likelihood of their predictions being correct. However, existing calibration approaches are fundamentally tied to datasets of one-hot labels implicitly assuming full certainty in all the annotations. Such datasets are effective for classification but provides insufficient knowledge of uncertainty for model calibration, necessitating the curation of datasets with numerically rich ground-truth confidence values. However, due to the scarcity of uncertain visual examples, such samples are not easily available as real datasets. In this paper, we introduce calibration-aware data augmentation to create synthetic datasets of diverse samples and their ground-truth uncertainty. Specifically, we present **Calibration-aware Semantic Mixing (CSM)**, a novel framework that generates training samples with mixed class characteristics and annotates them with distinct confidence scores via diffusion models. Based on this framework, we propose calibrated reannotation to tackle the misalignment between the annotated confidence score and the mixing ratio during the diffusion reverse process. Besides, we explore the loss functions that better fit the new data representation paradigm. Experimental results demonstrate that CSM achieves superior calibration compared to the state-of-the-art calibration approaches. Our code is [available here](https://github.com/E-Galois/CSM).

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
Haoyang Luo Linwei Tao 2 Minjing Dong'
Chang Xu
Abstract
Model calibration seeks to ensure that models produce confidence scores that accurately reflect the true likelihood of their predictions being cor- rect. However, existing calibration approaches are fundamentally tied to datasets of one-hot labels implicitly assuming full certainty all the annota- tions. Such datasets effective for classification but provides insufficient knowledge of uncertainty for model calibration, necessitating the curation of datasets with numerically rich ground- confidence values. However, due to the scarcity of uncertain visual examples, such samples are not easily available as real datasets. In this paper, introduce calibration-aware data augmentation to create synthetic datasets of diverse samples and their ground-truth uncertainty. Specifically, we present Calibration-aware Semantic Mixing (CSM), a novel framework that generates train- ing samples with mixed class characteristics and annotates them with distinct confidence scores via diffusion models. Based on this framework. we propose calibrated reannotation to tackle the misalignment between the annotated confidence score and the mixing ratio during the diffusion reverse process. Besides, we explore the loss functions that better fit the new data representa- tion paradigm. Experimental results demonstrate that CSM achieves superior calibration compared to the state-of-the-art calibration approaches. Our code is available here.
Source
Augmentations
Target
Mixup
Cat
Horsc
Bear
Cat
Ours
Cairn
Ostrich
Terrier
i
1.00
0.75
0.5
0.35
0,00
Figure I. Differences between Mixup and the Proposed Aug- mentation. Samples with intermediate mixing coefficients exhibit simple overlapping pattern for Mixup. The proposed augmenta- tion mixes semantics while preserving the object completeness, displaying enhanced image fidelity.
DNNs can achieve exceptional accuracy, they could be un- reliable in real-world scenarios since the predictions are always over-confident (Mukhoti et al.. 2020; Wei et al., 2022). This issue could mislead security-sensitive applica- tions, such as autonomous systems, surveillance applica- tions, medical diagnostics, elc. Model calibration (Vaice- navicius et al., 2019; Tao al., 2023b), the process which refines the models' predicted confidence to be consistent with the actual probability of correct prediction, stands as the key to reliable vision models.
Various calibration techniques have been introduced, in- cluding post-hoc calibration (Platt et al., 1999; Guo et al., 2017: Tao al., 2025), loss function designs (Mukhoti et al., 2020; Tao et al., 2023a), and regularization methods (Kumar et al., 2018; Krishnan & Tickoo, 2020). However, current methods rely on one-hot labeled data, which inaccurately assume uniform uncertainty across samples for confidence estimation. For instance, while clear cat image and an am- biguous cat-dog hybrid image demand distinct confidence distributions, one-hot labels enforce identical certainty. This creates critical gap that current datasets lack ground- truth uncertainty annotations necessary to teach models nuanced confidence distinctions. However, real-world collection of such data infeasible due to scalability and ambiguity chal- lenges. This necessitates us to create synthetically generated datasets with diverse ground-trufe confidence annotations.
1. Introduction
Modern deep neural networks (DNNs) have achieved signif- icant effectiveness in various vision tasks including image recognition, retrieval, and object segmentation (Zagoruyko, 2016; Gordo et al., 2016; Kirillov et al., 2023). Although
'Department of Computer Science, City University of Hong Kong "School Computer Science, University of Sydney. Corre- spondence to: Minjing Dong <minjdong eet.tn...utttt..u........
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
To generate samples with ground-truth uncertainty, naive
2. Related Work
solution could be Mixup-based approaches (Yun et al., 2019;
2.1. Model Calibration
Kim et al., 2020; Uddin et al., 2021; Chen et al., 2022: Noh et al., 2023). By augmenting training data with pixel- interpolated samples and soft labels, DNNs are supposed to learn diverse confidence. However, such methods often produce low-fidelity samples due to unrealistic pixel fusion or fragmentary collage, which deviate significantly from the distribution of real-world data shown in Figure 1. Consequently, these approaches struggle to generalize the learned calibration knowledge to real-world scenarios and hardly contribute to the calibration improvement (Wen et al., 2021; Maroñas et al., 2021; Wang et al., 2023a). Data augmentations have also been explored at test time (Hekler et al., 2023) to improve uncertainty estimation. While these post-hoc methods can mitigate over-confidence globally, they often fail to achieve precise calibration because they ignore miscalibration in the training with one-hot samples.
Network calibration (Guo et al., 2017) requires techniques to align model confidence with actual accuracy. Post-hoc methods adjust test-time parameters using validation set to improve calibration. Notable techniques include Temper- ature Scaling (TS) (Guo et al., 2017), Histogram Binning (Zadrozny & Elkan, 2001), Beta calibration (Kull et al., 2017) and its extension to Dirichlet calibration (Kull et al., 2019). While simple, post-hoc methods are sensitive to distributional shifts (Ovadia et al., 2019). Training-time reg- ularization methods involve explicit learning constraints and implicit loss functions. Label Smoothing (L.) (Müller et al., 2019) reduces output entropy by replacing one-hot targets with smoothened labels. Margin-based Label Smoothing (MbLS) (Liu et al., 2022) balances model discriminability and calibration by imposing a margin on logits. Implicit loss functions, such as those optimizing Expected Calibra- tion Error (Karandikar et al., 2021; Kumar et al., 2018), Focal loss (Ross& & Dollár, 2017: Mukhoti et al., 2020), Inverse Focal loss (Wang al., 2021), and Mean Square Error (Hui & Belkin, 2021; Liang et al., 2024), are effective objectives to improve calibration. Data-aware calibration primarily involves perturbation methods (Tao et al., 2024) and Mixup-based methods (Zhang et al., 2018; Yun et al., 2019; Hendrycks et al., 2019), originally designed for im- proved generalization These techniques have been found effective for calibration (Zhang et al., 2018; Pinto et al., 2022). RegMixup (Pinto et al., 2022) uses Mixup ularizer for cross-entropy loss, enhancing both accuracy and uncertainty estimation. Rank (Noh et al., 2023) incorporates ordinal ranking relationships among samples to reduce Mixup label bias. However, recent work (Wang et al., 2023a) highlights the limitation of simple convex combinations in Mixup methods.
For successful calibration-awant data augmentation, realis- tic samples with different soft class posteriors are expected to be constructed, which remains unexplored in the field of calibration. In this paper, we introduce Calibration-aware Semantic Mixing (CSM), data augmentation approach designed for model calibration that generates fighfiiclitty samples with semantic mixing. Specifically, we adopt pre- trained diffusion models to sample different sets of aug- mented images. Within the same set, the generated images are conditioned on the same latent noise but different soft class posteriors. Therefore, objects can be continuously transformed in the pixel layout from the instance of one class to another, which preserves the basic spatial complete- ness of objects to ensure image fidelity.
reg-
Besides these image samples, their corresponding soft labels should be annotated for learning. A naive solution is the mixing ratio during the diffusion reverse process. However, can rarely indicate the class posterior precisely. To tackle this issue, we propose to reannotate the data by leveraging CLIP's visual features which have been jointly trained with massive language supervision (Radford et al., 2021), and are thus able to provide more precise class posteriors. By representing the optimized sample embeddings as interpo- lations of class prototypes, we can identify and eliminate class- spesifsp biases in the estimated mixing coefficients, resulting in more accurate and effective annotations. More- over, we highlight the problem of imbalanced fitting of augmented data when utilizing traditional loss functions for calibration through some theoretical analysis. We reveal that the training of augmented data with 22 loss intrinsically results in balanced learning and better calibration perfor- mance. Extensive experiments on various benchmarks and tasks demonstrate our method's strong effectiveness in ac- curately estimating confidence levels, achieving superior results for data-aware model calibration.
2.2. Generative Data Augmentation
Generative methods, particularly Diffusion Models (DM), have recently gained popularity for data augmentation due to their high fidelity (He et al., 2022). Label-preserving aug- mentations based on DMs have been proposed to enhance image diversity. DA-Fusion (Trabucco et al., 2023) uses the SDEdit (Meng et al., 2021) technique to produce controlled variations of images within the same class. Diff-Mix (Wang et al., 2024) performs image translations by transferring foreground objects to a different class while preserving the background. In contrast, DiffuseMix (Islam et al., 2024) modifies the background with crafted prompts to enhance style diversity. To mix class concepts, MagicMix (Liew et al., 2022) combines SDEdit with specific prompts to gen- erate semantically mixed images. De-DA (Chen et al., 2024) decouples and re-blends class's dependent and independent
2

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
parts to create one-hot or mixed-label augmentations.
3.2. Calibration-aware Data Augmentation
Existing calibration approaches (Guo et al., 2017: Mukhoti et al., 2020; Kumar et al., 2018; Müller et al., 2019) pri- marily focus on post-hoc and training techniques, while the training data for calibration is rarely considered. In fact, the training is always performed on the datasets with an ordinary structure, i.e., the data pair of an image and its corresponding single label in the image classification task. Such annotated data exhibit zero uncertainty which can be ineffective for confidence calibration. To obtain diverse sam- ples of ground-truth uncertainty values, we propose to tackle model calibration via a calibration- aware data augmenta- tion scheme. Specifically, soft-labeled data are defined as { ( ]o)] where is the total number of augmented samples and (7% LK E A can be the non-binary con- fidence value for each class I of sample Te. We expect such data to provide confidence knowledge via soft label To instead of single class y in order calibrate the model.
3. Method
We provide preliminaries of model calibration and diffu- sion models in Section 3.1 and then introduce our proposed framework illustrated in Figure 2, regarding details of se- mantic mixing, reannotation, and learning objective for model calibration in Section 3.2 and Section 3.3.
3.1. Preliminaries
Model Calibration Considering an image classification task with a dataset {(ar,, ys ) where a; is a sample and y/ is its true label, a DNN classifier fo predicts class probabilities p: = {ut} ) ===
soft ma))((( E A with pf representing the predicted probability for class k. Here, A = {p< [0. 1] E Pk = }} represents the K ki-dimensional probability space. The predicted label for sample i is
K
A straightforward calibration scheme generating soft- labeled data is the mixup (Zhang et al., 2018) augmentation, which generates fused samples and labels from random train- ing pairs as ã = Aa; + (1 - >)ar, and q = Xq: + (1-));; where À ~ Beta(a, a) E [0,1] is a mixup coefficient, a > 0 is hyperparameter controlling the interpolation strength (Noh et al., 2023), and q: is the one-hot vector of Vi- However, the direct utilization of mixup cannot con- tribute to the calibration improvement, as empirically found by several studies (Wen et al., 2021; Maroñas et al., 2021; Wang et al., 2023a). We mainly attribute this failure to the fact that mixup performs calibration learning on low-fidelity images due to simple convex combinations, which makes the learned calibration difficult to generalize to real-world data. Therefore, to establish an effective calibration data augmentation, we propose to generate high-fidelity images with semantic mixing, which reflects the shifting process of class posteriors from one category to another.
+ arg max p
(1)
k
The confidence score of a predicted label i is defined as iu = p." Formally, a model is well-calibrated if the predicted confidence matches the true accuracy, i.e.,
P(ù:- 3/ p. p) for p [0, 1]. To measure net- work calibration, the Expected Calibration Error (ECE) is commonly-used metric defined as Eg, [P( y/ (.) p.]. Practically, due to the finite samples available, ECE is ap- proximated as the weighted average of the absolute differ- ence between accuracy and confidence across M bins of grouped confidence values. Formally, ECE is defined as:
M
ECE = M.
Σ m==1
B (Acc(Bm)) - (..)(
(2)
where Acc(Bm) and Conf(Bm) denote the accuracy and average confidence in the m-th bin, and [Bm| is the number of samples in Bm-
We start by expressing the general problem as how to estab- lish a vicinal distribution that can be sampled from, ie.,
Diffusion Models for Image Generation Diffusion mod- els are trained to generate realistic images via a grad- ual denoising process upon Gaussian noises. Their for- ward process gradually adds noise through Markov (t)
H ~ Po(E Iz, 2.)).
s.t. (I,, x.y)) ~ Pdtuat Dy | yo, Vy),
(4)
chain with Gaussian transitions @owd (z=) Iac (e) N(xx (t),
(t) (1 11-mm) n)) where ae!") is the noised vari- able of x0 x, at step and π) is the noise schedule. The objective for training conditional denoising parameters ø is represented as ming E. c <g (x((), vl. where es is the predicted noise. For conditional image generation, the classifier-free free guidance (CFG) (Ho & Salimans, 2022) is adopted to denoise images altering noise prediction as
vn,,!")
where Pdual is the joint distribution for x; and xy and Pe the vicinal distribution to be designed for semantic mixing.
To blend class semantics with soft labels, we propose a dual- conditioned sampling strategy via a conditional diffusion model explicitly optimized to estimate p(æ 1), Le.,
p5( | (4") (oc., po(æ q" )p (5)
(1) .)) (1 + w)Ea (1) :) - we, (x) Ø), (3)
where w denotes the guidance strength.
Specifically, we generate augmented samples ) via dif-
3

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
x(0
Yt
2
22...
Y1
P)
CE
FL L2
11 Q
XO
Calibration aware Data Augmentation
10
Diffusion model
Transfer
X1
CLIP B())
Separable
Distance
e
X2
cΦ cx(x(),c)
pp (Ag
2
X9
Reannotation
Predicted
Figure The framework of the proposed Calibration-awea -aarrrtr Semantic Mixing (CSM). Left: Calibration-aware Data Augmentation exploits pretrained diffusion model ensure fidelity and coherence in generating conceptually mixed augmentations. Right: To further improve the annotation and fitting of class posteriors, a normalized reannotation scheme and the theoretically unbiased L2 loss are adopted achieve balanced confidence calibration.
fusion reverse process as
as the existing loss functions are specifically designed for hard labels. We provide some theoretical analysis of the potential bias induced by popular loss functions used in the calibration field during the training of soft-labeled data. To tackle these issues, we introduce calibrated reannotation and balanced learning to ensure effective calibration learning.
20 (2 | rاt (") aT()),
a² x 1th (T))
2² ~ pa qr, (T)),
(T)) ~(T)~~ ~N(x((7) 0,, (6)
where â E [0, 1]. The augmentation simultaneously con- ditioned on the mixed label q dqx + (1 "0 and the sampled Gaussian noise (T) at diffusion timestep T'. The merits here are 3-fold: 1) the data fusion introduced at diffusion conditioning rather than pixel levels can signifi- cantly improve the image fidelity, which aligns with the real data distributions. 2) æ0, _ and A are correlated by con- ditioning on a common hidden state x(T), which controls the general image layout, maximizes the proximity of the augmentation series, and ensures unified visual composition for calibration learning. 3) conditional diffusion models are implicit class posterior estimators (Li et al., 2023) that can generalize to unseen conditions (Li et al., 2024), i.e., mixed labels in our case. Therefore, q² can be a naive solution to the label of augmented sample 2
Calibrated Reannotation As analyzed, À is inaccurate as class posterior estimation, which motivates us to reannotate these augmented samples with enhanced soft labels. Aim- ing for precise annotation, we assume that there exists a classification opttmal visual encoder E(-) on the dataset and our semantic-mixed augmented samples. In this case, the confidence ratio of the two mixing classes i and j for the same sample ã should be
E(x)
E(T) P.
P.
A
exp
T
1-X
P E(x)
P,
A o
(7)
is the class prototype for class k. is the sigmoid function, À is the reannotated mixing coefficient replacing x, and r is a temperature parameter. Eq. (7) is the theoretically optimal À annotation. However, the assumption of optimal E(-) can be overly strong in practice. Therefore, we first introduce a feature-level X-annotation, Le.dg, to relax the encoded feature as
where Pa -
- Em. E(æ;)
3.3. Calibrated Reannotation and Balanced Learning
While the proposed data augmentation scheme in Sec- tion 3.2 demonstrates various advantages for calibration learning, several potential issues arise when considering the characteristies of semantic mixing data pair (ž, q). One issue lies in the annotation. Given cemantic-mixed image sample T it is necessary to provide an accurate annota- tion of its true class posterior. However, the annotated soft label 7 from diffusion models can hardly be equivalent to the true class posterior since diffusion models are also pretrained on the datasets with hard labels. In fact, we em- pirically found that the generated sample set { where E [0, 1] shows high proximity, which strengthens the lay- out and object coherence due to the joint conditioning on x (T), As shown in Figure 4, may not accurately reflect the true class posterior due to non-linear semantic transitions in generated images. Another issue arises in the training phase,
An) +r, E(²) = A - -1- (1-
8.1. r (P. P,) =
(8)
where Ag serves as the interpolation coefficient and is linearly independent residual term. Here, dg is acquired by projecting F(@) onto the 1-D affine space of P; and P,. Specifically, by taking the inner product of both sides of Equation (8) with P. P.. we can express X as
(E(+) (P. -PP)) (P.-P,)
A
99)

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
Meanwhile, Eq. (7) can be reformulated by substituting Eq. (8) into Eq. (7) as
more closely to the sharper label 91- The network learns without such a bias only when ((p1, P2) 0. These cases
are also intuitively illustrated in Figure 2.
A A= ( (PB - 2 (PP333))
P3)). (10)
It's worth noting that the majority of common loss func- tions are not anfideence-bllnced. For example, ranking- or margin- losses presume poor reliability of Mixup la- bels. Consequently, their learning behavior basically blind to the exact class posteriors, therefore difficult to establish such balances. For CE and Focal losses, their patterns are both empirically and theoretically predictable. Denoting the optimal Pa as Pa and p (a E {1, 2)) for CE and Focal losses, respectively, we can determine the sign of score f3:
See Appendix A.1 for proof of the above Equations. It
is evident that directly adopting Eq. (10) to calculate X
would induce bias due to the unequal class norms in P P3 # 0 and inter-class distances in P, * (P. Pt) for i * and {i.j} # {l:, (}. respectively. Note that Dr is only measured by the affine space of the class prototypes regardless of these biases. We propose to regard these factors as invariant across classes and simplify Eq. (10) as
Proposition 3.2. For 4ō 2 |
@2112 we have B8 =
(p CE FF) = O, while for 46 < llq1 q2 q2112, we have 3g = 3(p P2 < O.
CE CE
o= (AE - ))
(11)
It can be seen that similar augmented pairs can exhibit neg- ative scores, indicating shifted confidence toward sharper class distributions. In contrast, the behavior of the Focal loss is almost the opposite in such cases, producing positive G scores for similar samples:
where s > O is a scaling factor. With Eq. (11), the new an- notations are free from biases inherent in the visual encoder E(.). In practice, we adopt CLIP's visual encoder E(-).
Balanced Learning Previous Mixup-based calibration methods adopt CE, Focal (Mukhoti et al., 2020), or certain ranking-based losses (Noh et al., 2023) for calibration with Mixup samples. We argue that they are not necessarily effective for augmented samples, especially in our case, where samples are required to be confusingly similar via the conditional sampling and possibly almost indistinguishable in the feature or logit space. To reveal the deficiency of common loss objectives for soft-labeled data, we define canfidence-balancee loss as follows.
Proposition 3.3. For IFL = 1 and V6- < 11 T2 we have 3s = B(pf F pf4)> 0.
This imbalanced learning severely degrades the calibration improvement from Mixup samples by dragging the over- all confidence towards certain sides. Aiming at obtaining balanced confidence, we present the following proposition:
Proposition 3.4. Vo 2 0. we have ((p 1.2 .p?2) = 0,
where L2 is the optimal solution adopting the L.2 loss:
Definition 3.1. Consider two soft-labeled data points as- sociated with classes i and j. with sharper and softer la- bels, respectively: qı \ig. + (1 (1-}1)q; \n)q, and q = d2q: + (1 111--22gq, A2)q, where 0.5 S A2 < < L. A loss func- tion C(p. .): A K
1 K p ql12
L12(P, q)
(13)
The formal proofs of Propositions 3.3, and 3.4 are pro- vided in Appendix A.2. The proposition reveals that when two similar samples exceed the model's 'ss discriminability, Ce loss tends to balance the learned labels of both the harder and softer instances, instead of tending to fit specific one of them. Notably, this objective is tailored for learning on soft-labeled data, in contrast to the calibration of one-hot- labeled samples explored in (Liang et al., 2024). In this way, we can formulate our overall objective as a simple combination of the CE and L2 losses, i.e.
x A K + R is considered a confidence- balanced loss if and only if the optimal outputs Pi, P3 for the following optimization problem:
min L(Pi, )1) + C(p2. Fo) P2
s.t. Pp
P2
2 8
(12)
satisfy the condition that the balance function 3()). = [| - 91 2 1| 22 equals zero for all 5 2 and for all possible data pairs. The prediction proximity condi- tion || 22113 < 8 ensures the non-discriminabiiity of correlated Mixup samples.
Loverall Σ LCe(p. + Ciz(Pe qa). (14)
Intuitively, || q ll2 and |2 - 72 1 estimate how well the model fits the two soft labels, respectively. With 3(p), P2) > O, I|p₂ - 22113 is smaller and the model fits the softer label T2 more effectively. In contrast, when 3(1, P2) < O, Ip: q 1|2 is smaller and the model adheres
4. Experiment
We conduct experiments with DNNs of different ar- chitectures, including ResNet-50/101 (He et al., 2016), Wide-ResNNet-2-110 (Zagoruyko, 2016), and DenseNet-12 121
5

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
Table Calibration errors before and after temperature scaling, Results in the brackets are post-temperature results. R50: ResNet-50. R101: ResNet-101, Available results on the same used settings are cited from (Noh et al., 2023).
McTHOO
5:0
R50
2015)
2016)
R50-HH
RIDL
2015:
ACC
ECE1
AECEL
ACC
ECE4
AECEL
.ACCT
ECE.
AECEL
ACCT
BCE
AECEL
|A40T
ECE.
ABCEL
ACCT
BCEEL
ABCE.
CR
15:18
3.756..97 97)
2.941 01)
01.10 06:16
3.310.933 02100-23
3-55/0.85)
01
11:95:120 933
111412250 963
77: 0.4
21-3417-0 631
06.66. 66)
14
7.8711.50 87)
56-01
3.013 50)
03337033
96.75
4.4111.72)
13.29/2.00
12.2812
13.45/2922
1344233004
1.9511935
2.72/2.73
2.10/2.00
04.87
1.2703-15)
1.225502)
91.38
3-390-911
188915 189
76.05
6.236 379
6.5966.75 766
76.41
1.99 18
1.8714.500 58:
3.2112
3.2522.50 513
55.53
11112 513
1.9212.7 (7)
FT.
M8.82
3/421-2571
1.610137)
01.99
3.2701332) 323
3.2311 773
70.41
2.5711
053
2.8811 733
76.13
3. 1017 585
1-2212-50 513
63.08
2.132-013
1.9 94)
58.03
7. 1182 (8)
3.0M22200 (M)
MIXU7
04.76
2.3641.37)
2S:12.00)
03.50
6.37(1115
6. 6.7912.35 33)
78.47
8.0812
14)
8.68 19)
78.74
1.92:3 07)
8.9113.65 135
65.81
1.221 92)
1.561 96)
66.41
2.411 07)
2.454 (.0
F1.50
M4.17
3.4560.831
>7400.95
01:36
3-9303.933
3.0700994
75.21
2.8012 961
2.59/2.50 863
70.81
3.29
3.2511 786
53.35
1921 933
1.8611 9g
04:01
18811 85)
1.81(181)
3.7322.33
0.3003.471 431
1,521
3.5711 66)
CPC
05.00
5.051.89)
50412.60)
05.36
4.761.521
77.23
13.2913.82
27.50
13.3222500
13.25.3.25)
3.4133
3.7413 74)
MiLL
15.25
1.186.110
3.195.18)
05.33
1.361.36
3.23(3.25)
71-92
(.0114.23)
4.1464.14)
77.45
5.4913.000 69)
0.5216.52)
04:34
1.411.94)
1.7511733
50.51
14211 02)
1.8k1.00
01.68
77600 86)
97)
95.83
47500.53)
4.700994)
76.76
5.531 485
5.0811 481
76.33
4 1.3011 565
1. 1.151 97
63.5 16
3051 861
3.0011 81)
63.26
1.551 86)
1.53 1 68)
AnoMMx
01.57
17500 63)
95:00
136 57)
1.3700311)
21.87
18.933
10:891 77)
78.63
11 06.66.700 (0)
1666188 85
96
2.3763 313
65.85
3.793 46)
1.761 67)
FCL
03:44
0.7660.76)
6.700.25)
05.25
0.3:0.35)
1.2111
75.19
3.7012.44
3.74/2.11)
78.98
4.1913.22 23)
4.5813.27 27
62.61
7,511135
7,5111 45)
54.04
6.921 (6)
6.9911.94)
CSW
95.79
0.548.554
0.310.33
95,80
0.510..54
(.310-33)
78.84
1.2811299)
79.37
1.46/1
1.28/
1.201.20)
1,19.1199)
08.28
1.311.30)
1.42/1.42)
ct al., 2023b). We generate 159, 840; 158, 400; and 318, 400 CSM-augmented samples for CIFAR-IO, CIFAR-I00, and Tiny-Imann Imageee,,,,t, respectively. The setup for training and testing in our work follows (Noh et al., 2023), while we also implement our approach based on their public code. Specifically. we conduct 200-epoch training on CIFAR-10 10 and CIFAR-100 100 and 100-epoch training on Tiny-ImageNG using the SGD optimizer of momentum set to 0.9. We adopt multi-step learning rate schedule which decreases from 0.1 to 0.01 and 0,001 at epochs 81 and 121 for CIFAR-10/10 or epochs 40 and 60 for Tiny-ImageNer, respectively. The weight decay is set to 5 x 10 4 We select the scaling hyper- parameter 8 as 4.0 for CIFAR- CIFAR-10/Ting MoTTin-ImmagNNe and 2.3 for CIFAR-100 using their validation sets.
Table 2. Comparison on the EQ-DATA setting on CIFAR-10 with methods that use non-augmented data. Results in parentheses are post-temperature results.
PLSU
Mel.S
CSM
ACCT
96.87
96.77
95.25
1.221.58)
3.4211
3 R3)
1.161.16)
0,761036)
0.9116.41)
ABCR|
3.673.02)
96)
3.18(3.18)
0.78(0.75)
0.68(0.42)
(Huang et al., 2017). We adopt CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and Tiny-ImageNet (Le & Yang, 2015) for calibration performance and out-of-distribution (OOD) robustness comparisons. For augmentation, we adopt the class-conditional elucidating diffusion model (EDM) (Karras et al., 2022b) to generate soft-labeled sam- ples. We select the standard ECE (Guo et al., 2017) and AECE (Ding et al., 2020) protocols our primary evalua- tion metrics for calibration, while following the widely-used area under the receiver operating characteristics (AUROC) protocol (Liu et al., 2020; Pinto et al., 2022) to evaluate the OOD detection performance. Detailed descriptions of our benchmark datasets, evaluation protocols, and compared methods are presented in Appendix B.
4.1. Results
State-of-the-Ar Comparison We provide comprehen- sive comparisons with -tate-of-the-art approaches in Table 1. From the results, we can derive the following key observa- tions: (1) Our CSM outperforms traditional regularization- based methods (e.g., Label Smoothing and FLSD) by large margins via confidence-aware data augmentation, indicating the strong effectiveness of our method to leverage confi- dence knowledge embedded in the learning samples. (2) CSM gives comparable or superior results for ECE and ACEC consistently across all compared benchmarks in- cluding 10, CIFAR-100, and Tiny-ImageNe es- caammeeraaur--scalige results. Such per- formance suggests that the confidence learning driven by data is crucial for model calibration, and our calibration- aware augmentation scheme can reliably generate the re- quired sample-label pairs. (3) Our method also outperforms Mixup-based calibration methods, including the -ffte-of-the- art RankMixup, suggesting that our generated augmenta- tions enable better sample-confidence alignment compared to traditional Mixup outputs in terms of both fidelity and coherence. (4) With the post-hoc temperature scaling (Guo et al., 2017), optimal temperatures in our method are mostly determined as 1.0 while compared methods generally ex- hibit larger T values, further validating that CSM effectively
During training, data-aware calibration methods supplement each example with additional aug- mented samples. To control the training time, we limit the maximum number of auxiliary data associated with each dataset instance as Nong = 3 considering the settings in (Noh et al., 2023). In fact, our method adopts Nang = for CIFAR-10/100 and Naug = 1 for Tiny- Iiny-ema,,Net, display- ing strong data efficiency. To make fully fair comparison with non-augmentation methods, we also show results with a different data setting Eq- DATA: During training. we limit the number of iterations per epoch to be (ie-Nnn)Dablchssee which ensures the total amount of training data per epoch to be consistent across methods.
Training Time
pecially for the
dutasel size
We use our proposed method to gener- ate augmented samples for each dataset based on the code
Training Details
and checkpoints from (Karras et al., 2022a) and (Wang
6

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
Table Comparison on ImageNet using ResNet-50 architec-
Table S. AUROC (%) for robustness evaluation under distribu- tion shifts. Higher values indicate better performance. We use CIFAR-IO 10 and CIFAR- 100 as in-distribution datasets, with each serving as the OOD dataset for the other, alongside an external Tiny-ImageNet dataset.
ture.
RsUMIXUP
KANAMIKUP
(SM/OOU)) 1.32
73.96
7.07
5.34
ABCEI
7.09
5.42
1.35
ID CIFAR-100
CIFAR-ID TINY-EXAGENET
CIFAR-100
OOD DATASET
CIFAR-IA
TINY IMAGENET
Table 4. Comparison on ImageNet using Swin- Wwvin-Transformer-V
TS
86.73
88.06
16,38
80.12
architecture.
LS
FL MIXLF
85.63
75.23
$6. 34.54
76.29
78.37
72.92
77 7..54
81.23
82,07
$4,09
78.46
81.55
METrIiss
CF
1.8
F1.
ELSD
MAL.S
CALS
CSM (OURS)
CPC
85.28
85.15
74.68
76.66
ACC
75.60
15.42
75.50
14.70
17.18
77.10
81.08
MaL.S
ECEL
9.95
1.32
3.19
2.44
1.95
1.61
1.49
REGMIXUP
72
AECEJ
9.91
7.33
3.18
2.37
1.73
1.69
1.86
RANKMIXUP
75.64
CSM
91.82
91.05
79.02
solves the over-confidence issue embedded in ordinary one- hot datasets and the compared learning paradigms.
Table 6. Ablation Study. Variants regarding augmentations, anno- tations, and learning objectives are compared. Results are from CIFAR-100 with ResNet-50.
We also share results on larger datasets or architectures by comparing CSM with representative methods on ImageNet with the ResNet-50 and Swin- Transformer architectures in Table and Table 4, respectively. Our method performs equally or more effectively compared to these methods, especially to the mixup-basee methods.
VARIANTS
ACC1
DCC 1
ADCE L
DCC"
BCT"
w/o AUGMENTED DATA
77.48
12.94
12.94
2.63
2.66
1.6
CSM (1.2) wTO REANNO.
78.92
3.20
5.20
3.33
3:24
CSM (1.2) CLIP LABGLS.
55.60
52.98
52.78
CSM (1.2) W7 ONE-HOT ADG.
79.24
10:84
10.84
2.45
2.41
MIXUP (CB)
78.47
8.68
8:68
2:14
2.19
(FL.
79.64
2.45
2.44
2:45
2.44
1.0
MIxUP (L22
75.88
2.49
2.62
2.49
2.62
1.0
8. CSM (CE)
75.93
2.47
2.14
2.47
2.14
1.0
We also provide qualitative comparisons using Wide- ResNet-26-10 10 (Zagoruyko, 2016) and DenseNet-121 (Huang et al., 2017) as the network in Appendix C, which demonstrate similar properties to those in Table 1. Mean- while, we provide results on the EQ- EQ-DATA setting in Table 2 for comparisons with non-augmentation methods for equal- ized training time. It is observed that our CSM can still achieve competitive or superior performance, particularly in consistently superior posp-rempee ECE and AECE. Comparisons of the estimated training time is provided in Appendix C, where one can observe that the number of augmented samples per batch is the major factor affecting training time.
9. CSM (FL) 10, CSM (L2, OURS)
77.65
2.43
2.30
2.16
2.09
0.9
75.84
1.29
1.63
1.29
1.63
1.0
Ablation Study We compare our CSM with variants in Table 6, testing different augmentation types, soft labels, and loss functions. We can observe that Mixup and CSM variants outperform the standard CE loss, improving both ECE and accuracy, highlighting the importance of data- driven network calibration. Compared with the variant w/o reannotation (or using the generation labels, Var. 2), our reannotated variant produces significantly better calibration performances, validating the accurate confidence estimation in our refined labels. Mcanwhile, using the vanilla CLIP an- notations (Var. 3) yields the worst ACC and pre-temperature calibration errors, primarily due to the noisy information by annotating all classes. Directly adopting class-conditioned augmentations from the diffusion model (Var. 4) can slightly rise the prediction accuracy, as also evidenced by the aug- mented classification literature. However, it fails to improve model calibration due to the lack of soft-labeled samples.
Dit----Dittritttonn Detection To validate the network calibration from different perspectives, we conduct qual- itative comparisons for the out-of-distribution (OOD) de- tection with the proposed CSM. We measure the sample uncertainty by the entropy of the softmax output p. = softmax fo (x;)). Therefore, we report AUROC scores comparisons in Table 5. In the table, our proposed CSM achieves superior and comparable results the CIFAR-10 and CIFAR- 100 datasets, respectively. Such results indicate that our approach learns precise sample confidence levels and is therefore also superior in detecting distributional shifts, verifying the significance of explicit soft-labeled sam- ples in OOD detection.
Among the losses used in CSM (Var. 8-10), the L.2 loss is the only one that reduces ECE, AECE, and post-TS results below 2.0, suggesting it effectively balances learned confi- dence in augmentations. In contrast, CE and FL losses often require temperature adjustments (Var. 5,9), with CE favor- ing sharper labels and FL for softer ones, aligning with our theoretical expectations from Section 3.3. CSM variants, particularly with CE and L2 losses, significantly outper- form traditional Mixup in terms of calibration, verifying the effectiveness of CSM's 's realistic in-domain augmentations.
4.2. Discussions
We discuss the properties of our method with extensive ex- periments below. More results are available in Appendix C.
7

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
W/o Augrented Samples
1.8
k/ Nixup Samples
W/ Prinitive Labels
1.8
Duns
Gap
Gap
Gap
Gap
Expectedd Accuraty
8.88
Accuracy
Expected Accuracy
8-
Accuracy
Expected Accuracy
80.8
Accuracy
Expected Accuracy
8..8
Accuracy
0.6
0.0
0.6
0.0
0.4
0.4
0.4
0.4
E66-11.22
ECE-3.88
ECE-4.15
ECE-1.29
8.8.8
8.8
0.4
8.8.8
8.8.8
0.4
Figure 3. Reliability diagram of various methods before temperature scaling.
confidence values, we compare the reliability diagrams of different methods across Figure 3. We examine CSM alongside three variants: 1) without augmentations, 2) with vanilla Mixup, and 3) using labels from generation. The augmentation-frre variant shows severe over-confidence across the middle-to-high confidence range, while the Mixup variant also exhibits over-confidence, though less methods that calibrate with one-hot labels can show similar over-confidence, high- lighting common issues in confidence estimation. CSM with primitive labels X, however, is both over- and under- confident at higher and lower ranges, respectively, suggest- ing a misalignment between annotations and confidence levels. In contrast, CSM demonstrates precise confidence estimation across both low and high ranges, with only slight errors in the middle, confirming the effectiveness of our Confidence-awaid aware augmentation.
(gen.)
0.00
Butterfly
Apple
A (anno.)
0.36
0.32
0,28
0.31
0.53
0.61
0.76
0.79
Man
Chimp
A (anno.)
0.36
0.44
0.51
0.73
0.74
severely. Regularization-
Wolf
Shark
A (anno.)
Motor-
Lawn- mower
cycle
(anno.)
0.17
0.18
0.18
0.60
0.86
0.87
0.87
0.87
Figure 4. Visualization of generated samples, including aug- mented sample sets, mixing coefficient during generation (gen.), and their reannotated Å (anno.) for soft labels.
We also examine the distribution of top confidence values in Figure 5(c). Without soft labels, predicted confidence peaks at 1 1.00, while Mixup-augmented training peaks at around 0.99. Our CSM produces a more even distribution, effec- tively reducing the over-confidence observed in calibration methods using one-hot labels.
However, applying 1.2 loss the traditional Mixup samples does not yield significantly better results (Var. 5-7). It is noteworthy that CSM uses conditioned dual-sampling of one-hot samples & and ~1, which are correlated consid- cring joint latent variable (!?), while every pair of them in Mixup is sampled independently, which makes the soft- labeled samples much easier be distinguished apart by the model. Therefore, traditional Mixup with 1.2 loss cannot perform noticeably better all metrics.
Calibration through Training To dynamically study the training-time behavior of CSM, we plot the Negative Log- Likelihood (NLL) and Overconfidence Errors (OE) in Fig- ure 5(a)-(b) using exponential moving averages for better visualization. From the plot, we can observe that better clas- sification has already been achieved in early epochs while the strong calibration is not evident until the second adjust- ment of the learning rate. This mainly due the order of network fitting, which prioritizes easy one-hot samples but learns the actual confidence in hard soft labels later. The lower calibration error of Mixup during early epochs also indicates that the larger difficulty in fitting confidence of our augmentations compared to Mixup samples. However, CSM can eventually achieve lower calibration error with a fine fine-grained learning rate for better fitting, validating the effectiveness of our proposed learning scheme.
Augmented Samples and Annotations To analyze the properties of augmented samples, we visualize some case images Fig.
It can be seen that CSM can pro- duce semantically sound intermediate samples compared to Mixup's 's simple overlap. Mcanwhile, these samples exhibit characteristics of both classes with gradual semantic change from one category to another. For images with abrupt tran- sitions (in red numbers) or inaccurate class posteriors, our annotation paradigm yields more precise À values compared to the generation phase Å.
Confidence Value Adjustment To analyze the learned
8

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
Comparison of N.L Across Models
Comparison of OE Across Models
Log Quantity of Confidence Values
W/o augs.
0.0
Wo augs.
Mixup
NExup
30
Log Quantity
8
W/o zugo. Mixup
Durs
23
103 Epoch
125
199
115
105
350
375
253
6.9
8:97
0.16
1:16
1.0
Epoch
Conficence Value
(a)
(b)
Figure 5. Training Characteristics. (a) NLL values of different methods CIFAR-I00 validation split across training epochs. (b) Over-confidence Errors (OE) of different methods on CIFAR- 100 validation split across training epochs. (c) Log Log-quantity of top confidence values CIFAR-100 test split among different methods.
Table 7. Integration with Test-Time Calibration Methods.
OE
14-
ECE
VARIANTS
ECE1
AECE
PIECE(Xionget al., 2023)
Colibrattoon Errors
Calibretton Errers
ECE(PT)
CSM
1.29
1.63
3.16
(ECT(
CSM + TTA
1.39
1.53
3.15
CSM PRUCAL
1.89
1.82
3.11
2.8
0.4
3.-
0.2
grated with test-time augmentations (TTA), the method bal- ances ECE and AECE effectively, achieving an optimized AECE of 1.53 on CIFAR-100. Compared to (Hekler et al., 2023) using test-time sample-wise scaling, CSM employs training-time augmentation with inter-sample augmenta- tions to expand the proximity space, enhancing calibration robustness. Combined with proximity method ProCal, we find that the proximity-infoma ter results, which validates the robustness growth related to proximity from the integration. Despite these growths, the overall calibration improvement is relatively small, in- dicating the effective sample augmentation and debiased proximity learning of our method.
3.3
(b)
Figure 6. Analysis of the hyperparameter [2.0, 6.0). (a) OE and UE w.r.t different s values. (b) ECE/AECE and their post- temperature values went different values.
metric PIECE displays bet-
Scaling Factor To analyze the influence of hyperparam- eter plot various calibration results CIFAR-10 in Figure 6, including Overconfidence Errors (OE) and Under- confidence Errors (UE) (Thulasidasan et al., 2019). Due to the scaling effect of values to the confidence levels, the label distribution can shift towards shaper and softer ones with higher and lower s, respectively. This is evident in Figure 6(a), where OE increases while UE decreases when the labels are shifted towards traditional one-hot annotations by large s, meaning that the model behaves over-confident about predictions. Consequently, ECE and AECE values increase a lot when s leaves a certain range as shown in Fig- ure 6(b). Nevertheless, with post-hoe temperature scaling, the calibration errors can become consistently low regard- less of values, verifying the stability of the proposed CSM.
5. Conclusion
In conclusion, CSM is a novel framework designed bridge the optimization gap between the calibration objective and the data of full certainty. By generating augmented sam- ples with semantic mixing and reannotating them with con- fidence scores via diffusion models, CSM enables a more accurate alignment between model predictions and their true likelihoods. Our exploration balanced loss functions fur- ther enhances the new data representation paradigm, enhanc- ing it an integrated pipeline for superior model calibration. Theoretical and practical evidence validate the effectiveness of CSM for strong sample-label augmentations. Meanwhile, various ablation results demonstrate CSM's balanced learn- ing true confidence levels. The framework's 's ability to acquire meaningful augmentation positions it as a novel baseline for semantic-aware confidence calibration.
Integrating Test-time Methods While CSM introduces training-time augmentations for softened labels, there exists test-time augmentation methods targeting the same goal. Meanwhile, proximity problems have also been investigated in inference time. We conduct experiments to combine these methods with our CSM, as demonstrated Table 7. Inte-

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
Acknowledgements
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778,10 201,2016.
This work was supported in part by the Start-up Grant (No. 9610680) of the City University of Hong Kong, Young Scientist Fund (No. 62406265) of NSFC, and the Aus- tralian Research Council under Projects DP240101848 and FT230100599.
He, R., Sun, S., Yu, X., Xue, C., Zhang, W., Torr, P., Bai, S., and Qi, X. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.0752 2022.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Hekler, A., Brinker, J., and Buettner, Test time augmen- tation meets post-hoc calibration: uncertainty quantifica- tion under real-world conditions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 14856-14864, 2023.
References
Hendrycks, D., Mu, N., Cubuk, D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint 1rXiv:/912.022 2019.
Charoenphakdee, N., Vongkulbhisal, J., Chairatanakul, N., and Sugiyama, M. On focal loss for class-posterins prob- ability estimation: A theoretical perspective. In Proceed- ings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 5202-5211, 2021.
Ho, and Salimans, T. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downsiream Applications, 2022.
Chen, J.-N., Sun, S., He, J., Torr, P. H., Yuille, A., and Bai, S. Transmix: Attend to mix for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12135-12144, 2022.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Chen, R., Wang, Z., Zhang, K.-Y., Wu, S., Sun, J., Wang, S., Yao, T., and Ding, S. Decoupled data augmenta- tion for improving image classification. arXiv preprint arXiv:2411.02592, 2024.
Hui, L. and Belkin, M. Evaluation neural architectures trained with square loss cross-entropy in classification tasks. In International Conference Learning Repre- sentations, 2021.
Cheng, J. and Vasconcelos, N. Calibrating deep neural networks by pairwise constraints. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13778,, 2022.
Islam, K., Zaheer, M. Z., Mahmood, A., and Nandakumar, K. Diffusemix: Label- sabel-preserving data augmentation with diffusion models. In Proceedings of the IEFF/CVF Con- ference Computer Vision and Pattern Recognition, pp. 27621-27630, 2024.
Chou, H.-P., Chang, S.-C., Pan, J.,Y., Wei, W., and Juan, D.- C. Remix: rebalanced mixup. In European Conference Computer Vision, pp. 95 1101 Springer, 2020.
Karandikar, A., Cain, N., Tran, D., Lakshminaraayannn, B., Shlens, J., Mozer, M. C., and Roelofs, B. Soft calibra- tion objectives for neural networks. Advances in Neural Information Processing Systems, 01-29768-29777 2021.
Ding, Y., Liu, J., Xiong, J., and Shi, Y. Revisiting the evaluation uncertainty estimation and its application trade-off. In
to explore model complexity-uncertan Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 4-5, 2020.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion brasuddoo--baedd generative models. Advances in Neural Information Processing Systems, 35: 26565-26577 2022a.
Gordo, A., Almazán, J., Revaud, J., and Larlus, D. Deep image retrieval: Learning global representations for im- age search. Proceedings of the European Conference on Computer Vision, pp. 241-257. Springer, 2016.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35: 26565-265777 2022b.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On cali- bration of modern neural networks. In International Con- ference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Kim, J.-.., Choo, W., and Song, H. O. Puzzle mix: Ex- ploiting saliency and local statistics for optimal mixup.
10

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
In International Conference on Machine Learning, pp. 5275-5285. PMLR, 2020.
Liu, J., Lin, Z.., Padhy, S., Tran, D., Bedrax Weiss, T., and Lakshminarayanan, B. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33:7498-777 -7512, 2020.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L.,, Xiao, T., Whitehead, S., Berg, A. C.,, Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Maroñas, J., Ramos, D., and Paredes, R. On calibration of mixup training for deep neural networks. In Structural, Syntactic, and Statistical Puttern Recognition: Joint IAPR International Workshops, S+ SSPR 2020, Padua, Italy, January 21-22, 2021, Proceedings, pp. 67-76. Springer, 2021.
Krishman. R. and Tickoo, O. Improving model calibration with accuracy versus uncertainty optimization. Advances in Neural Information Processing Systems, 33:18237- 18248, 2020.
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and edit- ing with stochastic differential equations. arXiv preprint arXiv: 108.01073, 2021.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Kull, M., Silva Filho, T., and Flach, P. Beta calibration: well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Artificial Intelligence and Statistics, pp. 623-631. PMLR, 2017.
Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P., and Dokania, P. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 33:15288-1522 2020.
Kull, M., Perello Nicto, M., Kängsepp. M.. Silva Filho, T., Song, H., and Flach, P. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in Neural Information Processing Systems, 32, 2019.
Müller, R., Kornblith, S., and Hinton, G. E. When does label smoothing help? Advances in Neural Information Processing Systems, 32, 2019.
Noh, J., Park, H., Lee, J., and Ham, B. Rankmixup: Ranking- based mixup training for network calibration. In Proceedings of the IEEE//VF International Conference on Computer Vision, pp. 1 1358-1368, 2023.
Kumar, A., Sarawagi, S., and Jain, U. Trainable calibration measures for neural networks from kernel mean embed- dings. In International Conference on Machine Learning, pp. 1805-28144 14. PMLR, 2018.
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan, B., and Snoek, J. Can you trust your model's 's uncertainty? evalu- ating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 32, 2019.
Le, Y. and Yang, X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
Li, A. C., Prabhudesai, M., Duggal, S., Brown, E., and Pathak, D. Your diffusion model secretly zero-shot classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2206-2217, 2023.
Pereyra, G., Tucker, G., Chorowski, J., Kaiser, L., and Hinton, G. Regularizing neural networks by penal- izing confident output distributions. arXiv preprint 2017.
X., Dai, Y., and Qu, Q Understanding generalizability of diffusion models requires rethinking the hidden gaussian structure. Advances in Neural Information Processing Systems, 37:57499-57538, 2024.
Pinto, F., Yang, H., Lim, S. N., Torr, P., and Dokania, P. Us- ing mixup as a regularizer can surprisingly improve accu- racy & out-of-distriution robustness. Advances in Neural Information Processing Systems, 35:14608-1460 2022.
Liang, W. Dong, C., Zheng, L., Li, Z., Zhang, W., and Chen, W. Calibrating deep neural network using euclidean dis- tance. arXiv preprini arXiv:2410. 18321, 2024.
Platt, J., Cain, N., Tran, D., Lakshminarayanan, B., Shlens, J., Mozer, M. C., and Roelofs, B. Probabilistic outputs for support vector machines and comparisons regu- larized likelihood methods. Advances in Large Margin Classifiers, 10(3):61-74, 1999.
Liew, J. H., Yan, H., Zhou, D., and Feng, J. Magicmix: Semantic mixing with diffusion models. arXiv preprint arXiv:2210. 16056, 2022.
Liu, B., Ben Ayed, I., Galdran, A., and Dolz, J. The devil is in the margin: Margin- based label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 80-88, 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. $748-8763. PMIR, 2021.

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
Ross, T.-Y. and Dollár, G. Focal loss for dense object detec- tion. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pp. 2980-2988, 2017.
Wang, Z., Wei, L., Wang, T., Chen, H., Hao, Y., Wang, X., He, X., and Tian, Q. Enhance image classification via inter-class image mixup with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17223-17233, 2024.
Tao, L.,, Dong, M., and Xu, C. Dual focal loss for calibration. In International Conference on Machine Learning, pp- 33833-33849 PMLR, 2023a.
Wei, H., Xie, R., Cheng, H., Feng, I.,, An, B., and Li, Y. Mitigating neural network overconfidence with logit normalization. In International Conference on Machine Learning, pp. 23631-23644. PMLR, 2022.
Tao, L., Zhu, Y., Guo, H., Dong, M., and Xu, C. benchmark study on calibration. arXiv preprint arXiv:2308. / 1138, 2023b.
Wen, Y., Jerfel, G., Muller, R., Dusenberry, M. W., Snoek, J., Lakshminarayanaaa B., and Tran, D. Combining en- sembles and data augmentation can harm your calibration. In International Conference on Learning Representations, 2021.
Tao, L.. Guo, H., Dong, M., and Xu, C. Consistency calibration: Improving uncertainty calibration via con- sistency among perturbed neighbors. arXiv preprint arXiv:2410. 12295, 2024.
Tao, L., Dong. M., and Xu, C. Feature clipping for uncer- tainty calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 1-0449 2025.
Xiong, M., Deng, A., Koh, P. W. W., Ji, S., Xu, J., and Hooi, B. Proximity-informee calibration for deep neural networks. Advances in Neural Information Pro- cessing Systems, 36:68511-68521 2023.
Thulasidasan, S., Chennupati, G., Bilmes, J. A., Bhat- tacharya, T., and Michalak, S. On mixup training: Im- proved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Pro- cessing Systems, 32. 2019.
Xu, Z.,, Chai, Z., and Yuan, C. Towards calibrated model for Long-tailed visual recognition from prior perspective. Advances in Neural Information Processing Systems, 34: 7139-7152, 2021.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong clas- sifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 0023,6032, 2019.
Trabucco, B., Doherty, K., Gurinas, M., and Salakhutdinov, R. Effective data augmentation with diffusion models. arXiv preprint arXiv:230..07944 2023.
Uddin, A. S., Monira, M. S., Shin, W., Chung, T. C., and Bae, S. H. Saliencymix: A saliency guided data augmen- tation strategy for better regularization. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, 2021.
Zadrozny, B. and Elkan, C. Obtaining calibrated probability estimates from decision trees and naive bayesian clas- sifiers. In Proceedings of the Eighteenth International Conference on Machine Learning, pp. 609-616, 2001. 2001.
Vaicenavicius, J.. Widmann, D., Andersson, C., Lindsten, F., Roll, J., and Schön, T. Evaluating model calibration in classification. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1459-3467. PMLR. 2019.
Zagoruyko, S. Wide residual networks. arXiv preprint 1r05v:1605.070 2016
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D.
Mixup: Beyond empirical risk minimization. In Interna- tional Conference Learning Representations, 2018.
Wang, D.-B., Feng, L., and Zhang, M.-L. Rethinking cal- ibration of deep neural networks: Do not be afraid of overconfidence. Advances Neural Information Pro- cessing Systems,
Zhong, Z.,, Cui, J., Liu, S., and Jia, Improving calibra- tion for long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16489-16498, 2021.
1118-88981800 2021.
Wang, D.-B., Li, L.. Zhao, P., Heng, P.-.,, and Zhang, M.-L. On the pitfall of mixup for uncertainty calibration. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7609-7618, 2023a.
Wang, Z., Pang. T., Du, C., Lin, M., Liu, W., and Yan, S. Better diffusion models further improve adversarial train- ing. In International Conference on Machine Learning, pp. 01246-36626.. PMLR, 2023b.
12

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
Appendix
A. Proofs of Equations and Propositions
A.1. Proofs of Equation (10)
As we have assumed in advance that a fully optimized model E(-) should satisfy
a E( P. E P.
(15)
and we can relax the encoded feature and acquire feature-level mixing coefficient Ae through
E a = Xg + (1- (1-Xg) P3 +r, 8.4. r (P, Pj) = 0,
(16)
first rewrite Equation (16) by vector multiplying (P, P;) on both sides
E T P - P,) + Ar) (P- P ( P3),
(17)
which can be reformulated as
(P. P2) (E(@) P,) (P. - - (P: (P:-)))
(B(T) P,) (P.- P,)
(18)
Therefore, RØ can be expressed as
(B( -P;) (PP -P)) λ
(19)
(P: P2)
which is independent of r.
Meanwhile, substituting Equation (16) into Equation (15), we can acquire
A A P.P. (P. +)) Xe (P: (R-P))) )P)))
(P. -PP-PP.
P2) (P.
(P. - (P) PP)))
+
(P.. P2) 2) +
1
(P.
P (P. -P;))
2
(PP: (AE )) +
p pt P. + PP P.T.P. P, -P,))
2
0 ( (P.-P,) PA) 2 + , (P. Pf))
(20)
Therefore, Ay and X can expressed with Equation (19) and Equation (20), respectively.
A.2. Proofs of Proposition 3.2, Proposition 3.3, and Proposition 3.4.
We start our proofs through restating the definition of a mixup-balanced Loss without the tilde hat for simplicity.
Mixup-balanced Loss. We regard loss function C(p. q) pk pk - as mixup-balanced loss iff the optimal pi P5 for
min C(pi, qı) + L(p2, 42) Pt.Pa
s.t. ||pi p2 < 6
(21)
13

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
a root for score function (p1,, P2) P1 4u 21 - Ip q2ll for all à 2 where pk is the K-dimensional probability simplex space. qi (i 1) is the annotated class probability for mixup perturbed samples of the same source image but with different shifting strengths, i.e.,
8; = a. + (1 0; )=',
q. = a + +1 0; 14,
0.5 5 a < 1, 1,2
(22)
where la and l, one-hot labels of the source and target classes and respectively. The above prediction proximity condition |p P2ll2 < 8 is imposed to outline the non-discriminability of correlated mixup samples. Without losing
generality, we always assume 0.5 < 02 < a1 < 1 in the following proofs.
A.3. Proof of Proposition 3.4
The L2 loss is defined as Lr2(p, q) 1 which is mixup-balanced To prove it, we need to solve
min [p: 9i 11 + ||p2 92l12
P1
s.l. P1 Pa l1 < 6.
(23)
a) When 5 2 |1 - 92112, the optimal solution pi = and p = 92 is mixup-balanced.
b) When < |/: - 92ll2. we represent the optima as
{
p \ı + X1 42 rı, p X242 + (1 - A2) r2,
(24)
where (q) 92) = rg (qi Q2) = 0. The objective in Eq. 23 is reformulated as
min
[p 11 + 11 42113
Pi.P2
(q) q:) + r2112 + II(1 (2) 42) + rrll.
(25)
Temporarily, we regard P as vectors in R we'll see the optimal ones always lie in PK With such relaxation, ri can take any vector value in their subspace, but it optimizes Eq. (25) only when r; O, further simplifying Eq. (25) as minx, A2 (1- - 21)² + (1 - >²)?. Substituting Eq. (24) into the proximity condition, we get (A1 + > 1)2 < -
5
ie,
S \
A2 S 1 q2 1
For each valid value of t + X = A, we consider Lagrangian equation
10. - 2
= (1-x) + $(A1 + A2 A),
(26)
where 3 is the Lagrangian multiplier. By setting the derivative w.E.. A; O, we solve for the optimum Af as
0L(A1,A2,8)
2 - 1) +
Oλ:
A;=A;
2(1 Xi) = =2(1 - >1) B
x; = X₂ 2 E 1).
(27)
Therefore, p.s are actually interpolations of q&s which definitely lie in pk We have |pp - 1: = ||p; - 92 11
(1- - - ll 42ll2 by substituting Eq. (27) and (24) into each term, indicating V.A, B(p). Pt) = 0. Hence, the L2 loss is
mixup-balanced.
A.4. Proofs of Proposition 3.2 and Proposition 3.3
CE and Focal Losses are not Mixup-balanced. We prove that 8 < 0 is always true for the cross entropy loss while for Focal loss with some specific 7. 2 is always the case.
14

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
We denote dual-sample loss function P Ei= Ek=1 of Litem (pt), where Cieem (+) is monotonous and ap- proaches infinity when p approaches O. To solve the loss minimization objective
min PPair s.t.
P2ll -pf=11 =
Σ
P2 =1
(28)
PI.Pa
we should first clarify several points before looking into
For every class k A s, E, of
the optimal p is Otherwise, we can carry out 3 types of operations without increasing
P1 p2ll2- Op. 1 is formulated as
p - pt -, p2 22 -1 pf p +r, P2 +r,
(29)
where e E {8,t} and r > O. While this operation is proximity-invarian, reduces the objective function and can finally decrease least one of (pf.pf. to 0 for every pair of them indexed by k. Then, Op. 2 given as
P1 - Pi - P2 82
P2 T
pi + pi +r, PS + 1 +r,
(30)
where k1, 12 & {s,t), 11 * k2. Op. 2 optimizes both the objective and the proximity value, making only one of {pi,22} (denoted as p.) having spare values. Finally, Op. 3 merges these values into p or p
p p + D r,
(31)
where before operation, p < < j) and PS - p.. It's easy to prove the existence of such class and the reduction of both the objective and the proximity value.
With these operations, we can optimize the objective when p * 0 for any class l * s,t. Therefore, 5 = 0 is always true in the optimal pf. ie., p²: + +p!
b) As we have assumed advance that q3 = 02 < ai = of. we can now immediately derive that p S p 1 Otherwise, we can swap oi and Pž to further lower the loss while keeping the proximity constraint unchanged. This helps us derive
PP p2l = (pi p2)² + (p) p2)²
(rf pz)² + (1 - p - 1 pz)
2(r -12)²= A,
A
pi 12" + +V 2
(32)
for a specific value of E [0, For simplicity, we rewrite Eq. (32) as:
p- p +
(33)
where 0, 0. 1
These observations help us formulate and solve the following Lagrangian equation (for arbitrary A):
L(p1, 1,,1.P2.3,.11 3,,,3 82)
= Pair + :(((pp 2211 A) + 31
pt 1) B2 Σ p 1), >
(34)
&
where 3, 81 and 32 are the Lagrangian multipliers. By taking derivatives w.r. pf and set it to 0, we have:
aL. Pair
+28(1 13) Br =
(35)
0p. Op:
15

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
i,j E (1,2), j.
j. and k {8,(}.
It can be seen that
0L 0L
Por
OL
+
P} + + -D)) +B =
0pp 0p;
O
Opp
0L
0L
PC
OC
+ +23, =
0p: pp! Op:
Opk
( OL Pair Pair
A = 2 0pp
(36)
0pp
and
0L
0L
Poir
PC
+
+ 2B(
p -
p +
31
B2
Opt 0pp
ppp
0pp
OL OL
Par
Parr
+ B1 + 32 =0
(37)
0p1 0p
Op1
0ph
By substituting Eq. (36) into Eq. (37) and let k = 8 (k = t yields the same result), we have:
Parr
Parr
1 1/ e/
PC
1
1/8Caair
OLPair
+
0p5 = 0.
0p2 Pair
2
Opi
2
0p2
Par
Pair
Parr
+
0.
(38)
0pi
Opi
0p2
0ph
CE Loss. When 0 2 1/ - q2112. the optimal solution in the case of CE loss is př = qı and p = q2. which is mixup-
balanced. However, when < ||q q2ll2. 112, the CE loss always yield negative 3 scores, indicating its preference for fitting samples close to the source.
We prove this by regarding the left part of Eq. (38) as function of p since have p" p"2 c and p" +p 1. From here on, we discuss the optimal p; and omit the star superscript for simplicity. The left part of Eq. (38) can be rewritten as
h(x) x e 1- 1 1-02
1- 02
(39)
1-x
The root of h(x) in [0.5, 1 - c] is the root pa of Eq. (38), i.e., p-2. Similarly, the root of h(r e) is p"i. Although it is difficult to find a closed form of their roots, we can find the mean of them, i.e., t = (p" p 22/ as the root of h(x c/2). Our initial goal is to find the sign of (,, which is now
B = (pi 01)² + (1 pi -1 (ps a222 (1 pp 1 + x2)
= 2pp - 01)² 2 - 2p3 - o22)2 2(21 01 - (e) + 02 - -1)
(40)
and is closely associated with f. The sign of B is determined by the sign of (a1 ax2) /2 t. It's intractable to solve t directly, but noting the monotonicity of with
n'(x)
+ (1 x -)²
1- 01
+
02
+ - 02 0
(41)
(a e)
x² (1 (1-x)²
for all x E (0, 1 - c). we now have
0n
+ 02
a a1+ 02
- )
< h(t - ; = 0,
t
$
h(
(42)
2
2
2
2
where 4( b( - } equals
Ter(an, 022,)
Q1
1 1- a1
02
- 02
5 +
1
01402
+ 5
01+02
1
01 01402
+
2
2
1 1- al
01
1 1-02
-2
(43)
1 01 02 {
01 + 02 +
1
01
02
01 + 02
Given that 0.5 <= 02 < 0n < and
S < 01
02 (by
|q:
q2
112),,
we can determine the sign of Te (22
16

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
Lemma A.I. (11 a22, e) < O. Therefore, when ó < 11 -2 the CE loss always yields negative scores.
Proof. We denote that
D1 = 2 - 02 0,
D2 = 01 + 02 + 0,
(44)
D3 = 2 - 01 02 +> 0,
D₃ = 01 +02 6 0
Then, have
Tcnlan,
1 - 01 01
02
012
D1
D2
D3
D4
(1-a1 D2 - 01
D2
(1- 02y D4 - D2
D1
D3D4
-a1 + 02 +
01 -12- -E
+
D2
DzD4
1
1
=(11 02 - E)(
DzD4 D2
Dı D2 D2D1
=(01 02 c)(
(45)
D2D2DaD1
where 01 02 0. D2D3D > 0, and D2D2 - D3D1 -e(a + - 1) 0. Therefore, h( A(01002 TCE(a1, 02,6) c) < O. Based on the previous deductions, we can conclude that 33 is always negative when å < (17 92 l
Focal Loss. Due to the complexity of Focal loss, we specifically analyze the case when YFL = 1. The commonly used 78
are larger and the balance scores become even greater empirically for these values. It's worth noting that 2 1q: q 1
doesn't guarantee the Focal loss to yield p. = qi since the Focal loss is not strictly proper (Charoenphakdee et al., 2021).
Nevertheless, its behavior with 8 < ||q 42llž can be sufficiently clear. We mainly focus on these cases in the following proofs.
Similar to CE loss, the sign of B3 is associated with function
((x) =11 a1)
X +
log(1 x - -c) 01
1 (1-x- --E
-log(0 + (s)
1
x +c
+(1 az)
log(1 - x)) 02
1 1 r
log(x))
(46)
1 1-x
Note that g(x) also increases monotonously. Therefore, the sign of B the optimal P is the same as the sign of g( t02 }) which equals
(11aaa) ai) E I- 1og(1- - -E)) a] F 1-F (1=1
1 1-E
TFi(a1,02,)) (11-a
log(E))
E
+(1 (1)
log(1 -F)) 02
1 - F
log (F),
(47)
F
where E= -artagte = an aa--g-e constrained by 02 < F < E < an, and + = 01 + 022-
2
Lemma A.2. TFL(ai, az, E) > 0. Therefore, when 8 < ||q]- -92ll3. the FL loss with 7FL = 1 always yields positive 3 scores.
Proof. We rewrite TFL(a1, as
TFi(an,a224 = p(a2, + p(an; -x); w -
1(w,x) (1 w) 1
log(1 1))
1 W- -
log(w + x²)
(48)
W + x
17

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
where [0.5, 1.0) and (0, min( 2 1 -w)] We show crucial properties of p(w, 2)):
a) u(w, -x) + p(0.5, z² > 0.
Proof. We reformulate the left part of the inequality as
nww,- xx + ((0.5,22)
-(1 1)
w I
log(1 - w + x))
1
1 w +
log(w - x)
1 w +
w
0.5 + x
0.5 1
+ 0.5
lo0g - xt) 0.5
log(0.5 + x))
0.5 x
0.5 + x
(1 w) log(1 + x)) + w log (w r) 0.5 log x²
0.5 log + +x)
w x
1 1 w + x
0.5 +
0.5 x
+ (1 w)
+ 0.5
0.5
(49)
+
0.5
0.5 x
where consider
T log - -) 0.5 log z),
(50)
T2 0.5 10. I) (1 - w) log(1 x).
(51)
find that
0T
0.5
(w 0.5)x
2 0,
(52)
Or
0.5 - x
w - I
(0.5 - 2) 2)( x)
8T,
0.5
1 w
(w 0.5)x
2
(53)
Or
0.5 + x
1 w + x (0.5 + x) w + x)
T3
0= ((w) w log w (1 w) log w) > (T1 T2)
(54)
Because d"(w) 1-2w 0, Ti T3 > Φ(w) 2 min (0.55) p( )) 0. Now that we have considered the sum of =[
logarithmic terms in Eq. (49), proceed inspect the sum of the rest terms:
W
1 w + x
0.5 + x
0.5
(1 w)
w
0.5
0.5
W +
0.5
0.5
0.5
- W + σ
0.5 0.5 +
0.5 I
>(1 (1 w)
0.5 W
+
0.5
0.5
0.5 x
0.5
=F F(w, X)
(55)
where we can find
OF OF
0.5 E
+
(1 W + x)
W
dw
0.5 + r
W œ
(w x
0.5
- +
0.5 + I
W
(w x
w (w x (1
W + x)
0.5
(w x
0.5 + Ø
+ (w x²2
0.5 T
(w x²²
0.5 + x
x
(1
2x
1 +
(w x²)2
0.5 + x
2x 0
+
(56)
(w 2)
0.5
F(w, x) > F(0.5.
(57)
Therefore, Eq. (49) is positive because the sum of all its terms is positive.
18

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
b) x(w,x) > x))
Proof.
pmw, 2) - ((0.5, x)
(1 w) log(1 -w)) x) + u wwog(w (w + x) 0.g z)
00. +
w +
1- 2
0.5
0.5
(1 w)
0.5
0.5
(58)
w I
+
0.5
0.5
where we consider
T3 = (1 - w) log(1 - x) + w log u +x)
+ 0.5 0.5 x)
0.5 10.5 +.),
(59)
" #+ x
1- w T
0.5 +
0.5 x
T4 -1 - w)
w
0.5
+
(60)
1 w
w w
0.5 -
0.5 +
We find that
0T3
1 w
0.5
0.5
1 W x
1 + a
0.5
0.5+ x
0.5(1 (1 - w) - xx1 w) - 0.5(1 - w) + 0.5x
0.5m + wx - 0.5w 0.5x
(1
< (w +
(w (0.5 +
(w - 0.5)
(w 0.5)
[ + (w + x0) x)
>
(61)
T3 > T3 x=0
w log (1 w/ log(1 w) > 0.
(62)
Meanwhile, we can also find
w #
1 1- w I
0.5 + r
0.5
T4 =1= w)
0.5
+ 0.5
0.5 x
0.5 +
W +
0.5 x
0.5 + x
0.5 2
>1 w)
0.5
0.5
0.5 +
0.5 x
0.5
=G -G(w, (. x).
(63)
where we have
aG(w, x²)
1 W x + w I
0.5
(1 - w)
Iw
1 - W T
(1 W x)²
0.5 +
(w x1
x)
0.5
(1 -1-10 W x)²
0.5 + x
w - 2 +
0.5 x
(1 (1- w x)²
0.5 + x
2x 0,
+
(64)
W x))
0.5 +x
which results Ta > G(w, x) > a00.5 = 0.
Therefore, combining a) and b), have p(a,, -z) x(0...z)) 2(a2,)), Le., TFilai, 02, p(az, +
u(a1. 22) > 0.
B. More Details for Experiments
B.I. Compared Baselines
We adopt diverse training-time methods for comparison including the vanilla CE loss. We set hyperparameters for the compared methods following (Noh et al., 2023). Specifically, we compare calibration performance with the traditional
19

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
regularization-based methods including a) ECP (Pereyra et al., 2017) with 0.1 as the coefficient for entropy penalty, b) MMCE (Kumar et al., 2018), c) LSS (Mäller al., 2019) with 0.05, d) mbLS (Liu et al., 2022) with 6 for CIFAR and m 10 for Tiny-ImageNet, e) FL (Ross & Dollár, 2017) with fixed y = 3, f) FLSD (Mukhoti et al., 2020) adopting schedule of FLSD-53 variant, g) CPC (Cheng & Vasconcelos, 2022), and h) FCL (Liang et al., 2024) with 3 and À = 0.5. We also compare our CSM with data-driven calibration methods, i..,, a) Mixup (Zhang et al., 2018) with Beta distribution shape parameter a = 0.2, b) RegMixup (Pinto et al., 2022) with a Beta distribution shape parameter a = 10. <) AugMix (Hendryeks et al., 2019) with a coefficient of 1.0 for JS Divergence term, which yields better accuracy compared to original value of 12, and d) RankMixup (M-NDCG) (Noh et al., 2023) with weight for M-NDCG set as 0.1 and shape parameters 1,2 for CIFARIO/ 10/100 and TinylmageNet, respectively.
B.2. Datasets and Augmented Samples
We adopt CIFAR-10, CIFAR- 100, and Liny-ImageNe as the evaluated datasets. We hold the principle that the ratios of generated sample amount versus training size are fixed be less than certain value across different datasets, e-g., 4.0 in our experiments. Note that the number of augmented samples we use in training is much less than that in Mixup methods. Meanwhile, we fix the size of each sample set generated with the same noise x²" as 8 for all datasets. Here are detailed descriptions of the datasets and augmented samples:
The CIFAR-IO dataset contains 60, 000 RGB images of the size 32 x 32. All images fall into one 10 semantic categories. By default, the dataset is split as 50, 000, 5, 000, and 5, 000 samples for training, validation, and testing. We generate 550 sets for each of the 45 class pairs, producing 198, 000 samples for confidence-aware augmentation.
CIEAR-10:
CIFAR-100: The CIFAR-100 100 dataset consists of 60, 000 32 x 32 color images in 100 classes. The split is similar to CIFAR-10 with 50, 000 for training, 5, 000 for validation, and 5, 000 for testing. We generate 5 sample sets for each distinctive class pair, yielding 198, 400 augmented samples in total.
Tiny-ImageNet: The ImageNet dataset is a subset of the large-scale ImageNet dataset and includes 120, 000 images of a large set of 200 classes. Each image is a downsized ImageNet sample of size 64 x 64. For CSM augmentation, we generate 2 sets for every 19, 900 class pair, producing a total of 318, 400 samples.
The generation process takes much or even less time than the training time. Specifically, it takes less than 4h for single A4000 GPU to generate the amount of all CIFAR-10 or CIFAR- 100 augmented samples, while using less than 8h for the same computing units to generate for Tmng-ema
B.3. Evaluation Metrics
To assess the model calibration, we employ four key metrics: Expected Calibration Error (ECE), Adaptive Expected Calibration Error (AECE), Overconfidence Error (OE), and Underconfidence Error (UE).
ECE approximates the average discrepancy between a model's confidence and accuracy. Practically, is estimated across equally spaced confidence bins. Let Bi,..., By denote M bins partitioning predictions into intervals [0, , ,...., ; For each bin B,, the accuracy and confidence are aomputed as acc(Bm)) (Bm) = |B...] ΣiEB.. I(y; = (t) and conf (Bm)
M M-1 1].
J B.n. 1im E.. P.. respectively, where yı is the true label, iA is the predicted label, and p. is the predicted probability for the winning class. ECE is then defined
M ECE
|mml N
|aco(Bm) Bc)n((Bm))
(65)
where N is the total number of samples. Lower ECE values indicate better calibration.
AECE addresses potential biases from fixed-width binning by constructing bins with adaptive widths to ensure equal sample counts per bin. This approach reduces sensitivity to irregular confidence distributions. The calculation mirrors ECE by using Eq. (65) but uses bins .,.... By where each bin contains approximately N/M samples.
OE isolates cases where the model's confidence exceeds its accuracy compared to ECE, Using the same binning as ECE, OF
20

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
Table Evaluation results with Wide-ResNet-26-10
CIFAR-10 and Imag-
CIFAR-10
Tiny-ImageNet
Method
Wide-ResNet-6-10
ACC1
ECEL
AECEL
ACC1
ECEL
ABCE]
CE
95.80
2.70
2.66
65.18
6.08
6.06
Mixup
96.53
3.14
3.08
66.36
3.77
3.75
MbLS
95.70
1.45
2.78
65,30
2.57
2.32
RegMixup
95.44
4.18
3,99
63.40
3.87
3.93
FCL
95.84
0.92
1.39
64.62
6.85
6.85
RankMixup
95.73
1.62
1.53
65,56
3.83
3.94
CSM
96.09
0.49
0.23
67.81
1.67
1.66
Table 9. Evaluation results with DenseNet- 121 on CIFAR-I0.
Metric
CE
MbLS
Mixup
RegMixup
ICL
RunkMixup
CSM
ACC1
94.68
94.91
95.28
96.23
95.41
94,66
95.71
BCE!
3.37
1.64
2.01
5.60
2,87
0.51
AFCEL
3.31
3.52
2,15
5.39
1.28
2.84
0.19
Table 10. Results on long-tailed datasets.
CIFAR
CIFARIOOFTT
Method
10
100
10
100
ACC1
ECE!
ACC1
ECEL
ACC1
ECEL
ACC1
ECE,
CE
86.39
6.50
70.36
20.53
55,70
22.85
38.32
38.23
Mixup
87.10
6.35
73.06
19:20
58.02
19.69
39.54
32.72
Remix
88.15
6.81
75.36
15.38
59.36
20.17
41.94
33.56
UniMix
89.66
6.00
82.75
12.87
61.25
19.38
45.45
27.
RankMixup
CSM
90.97
1.81
86.22
3.73
62.35
7.45
48.91
16.02
defined as
M
By
OE
max (0, co.f(B... acc (Bc((m)))
(66)
This metric evaluates overconfident predictions, with lower values indicating fewer instances of excessive confidence. Conversely, UE captures scenarios where the model's confidence underestimates its accuracy. It is computed as
Bm max (0, acc(B..)) conf(
UE
(67)
Lower UE values suggest better alignment in underconfident cases.
C. More Experimental Results
Network Architecture We provide results on Wide-ResNet-26-110 10 and DenseNet-122 121 to verify the performance consistency of our method in Table and Table 9.
Long-tailed Datasets We also analyze effectiveness CSM on long-tailed datasets, where the challenges of class imbalance and miscalibration are more severe. Following the commonly-ados 2021), ResNet32 network and CIFAR IEFARRO/I00 10/00-LTT datasets are adopted as the backbone and benchmarks, respectively. As displayed in Table 10, method outperforms the vanilla mixup and existing mixup-based LT approaches (Chou et al., 2020; Xu et al., 2021; Noh et al., 2023) across datasets and imbalance factors (p 10 and p 100) in terms of both ACC and ECE, demonstrating competitive or superior effectiveness without particular designs to learn on long-tailed datasets. These results suggest that our model's authentic augmentations and confidence-awand data are key factors contributing to its superior performance, even in highly imbalanced LT settings. Our model's ' significant improvements in both accuracy and calibration metrics demonstrate robustness in handling long-tailed datasets.
setup in (Xu et al., 2021: Zhong et al.,
21

---

Beyond One-Hot Labels: Semantic Mixing for Model Calibration
79
-10
AECE (8)
77
ACC
ECE
75
74
58
190
150
208
250
300
350
(a)
Figure 7. Analysis of the amount of augmented data: Classification ACC, ECE, and AECE CIFAR- 100 different number of augmented samples relative to the dataset size.
Table 11. Calibration Comparison with Different Number of Augmented Samples per Training Sample.
NAVOS ECE: CIFAR-10
2
0.83
0.54
0.39
ECE: CIFAR-100
2.07
1.29
1.74
Table 12. Comparison of the Estimated Training Time.
METHOD
CE
MBLS
MIXUP
REGMIXUP
RANKMIXUP
OURS
OURS(EQ-DATA)
TRAINING TIME
2.63H
2.65H
3.48H
3.50H
4.30H
4.28H
2.64H
Number of Augmented Samples We plot the calibration results w.r.t. different amount of used augmentations in Figure 7(c). To ensure equal comparison, randomly sample dataset samples and augmented ones with the fixed ratio 1:2 during training regardless of the available amount of augmentations. Surprisingly, the model calibrate well with relatively small proportions of augmentations. Meanwhile, ECE and AECE values jointly minimized with augmentation amount around 200% dataset size, reaching their best value of 1.20 and 1.02 on CIFAR-100. 100. These results effectively validate the consistency and reliability of our method.
Number of Augmented Samples per Training Sample We provide more analyses about the number of augmented samples per training sample (denoted as Naug) with results Table 11. can be observed that adding the number of accompanied augmentations per dataset sample generally improve the final calibration performance. This is because larger number of Naugs can sample more sufficient proximal data for training, better filling the domain space and providing more accurate confidence estimation. However, simply using larger Naugs could also raise the computational overhead and slow down the training.
Estimated Training Time We provide a comparison of the estimated training time Table 12. One can see the number of augmented samples per batch is the main factor influencing the training time. CSM outperforms others in calibration while maintaining reasonable speed. Even with equalized training samples, i.e. EQ-DATA, it achieves competitive calibration performance. Augmented samples need no re-generation across model, objective, or annotation changes, enabling efficient modular study. We run CSM with single RTX A4000 device.
22


Reviews:
[{"review_id": "rrBsIvYBki", "reviewer": "Reviewer_TsGd", "summary": "Model calibration typically assumes full certainty in datasets with one-hot labels, limiting accurate uncertainty estimation. To address this, the paper introduces Calibration-aware Semantic Mixing (CSM), a data augmentation framework that synthetically generates diverse training samples annotated with explicit confidence scores using diffusion models. Additionally, the authors propose a calibrated reannotation method and explore suitable loss functions for this new data paradigm. Experimental results show CSM significantly improves model calibration over existing state-of-the-art methods.\n\n## update after rebuttal\nThank you for the author rebuttal. The major concerns regarding clarifications and additional experiments have been addressed. I will maintain my current rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer TsGd\n\nThank you for your kind suggestions on clarity and experimental thoroughness. Below are our responses:\n\n**Q1-1**: Clarification on the reason that the proposed L2 loss is a balanced loss.\n\n**A1-2**: \nThank you for this nice concern. We need to clarify that there exists a **typo in Proposition 3.4** which makes the conclusion confusing. Proposition 3.4 should have been presented as\n\n- $\\forall \\delta \\ge 0$, we have $\\beta(p^{L2}_1,p^{L2}_2)=0$,\n\nwhich is an equation rather than an inequality for the $\\mathcal{L}_2$ loss's balance function, meaning that when two similar samples exceed the model's discriminability, $\\mathcal{L}_2$ loss tends to balance the learned labels of the harder and softer instances, instead of tending to fit a specific one of them.\n\nNote that the proof of Proposition 3.4 we provided in the Appendix A.3 does prove that $\\beta(p^{L2}_1,p^{L2}_2) = 0$. With such theoretical justification, we also present an empirical evidence in **A1-5** regarding the confidence balance score.\n\nAlso refer to **A3-1**. We will correct this typo in the revised main paper.\n\n**Q1-2**: Missing comparisons with ACLS [1] and CALS [2].\n\n**A1-2**: We compare our result with theirs in the following tables. The results demonstrate the competitive or superior performance of our method compared to the state-of-the-arts. We will cite these compared methods and include these results in the final paper.\n|ResNet-50|CIFAR-10|||\\||Tiny-ImageNet|||\n|-|-|-|-|-|-|-|-|\n|Metrics|ACC|ECE$\\downarrow$|AECE$\\downarrow$|\\||ACC|ECE$\\downarrow$|AECE$\\downarrow$|\n|ACLS|95.40|1.12|2.87|\\||64.84|**1.05**|**1.03**|\n|Ours|**95.79**|**0.54**|**0.33**|\\||**66.99**|1.29|1.19|\n\n|ResNet-50|Tiny-ImageNet|||\\||ImageNet|||\n|-|-|-|-|-|-|-|-|\n|Metrics|ACC|ECE$\\downarrow$|AECE$\\downarrow$|\\||ACC|ECE$\\downarrow$|AECE$\\downarrow$|\n|CALS|65.03|1.54|1.38|\\||76.44|1.46|**1.32**|\n|Ours|**66.99**|**1.29**|**1.19**|\\||**79.87**|**1.32**|1.35|\n\n**Q1-3**: Insufficient validation on ImageNet and Transformers.\n\n**A1-3**: We compare our result with representative methods on ImageNet with the ResNet-50 and Swin-Transformer architectures. Our method performs equally or more effectively compared to these methods, especially to the mixup-based methods. We will include these results in the final paper.\n|ResNet50|ImageNet|||\n|-|-|-|-|\n|Metrics|ACC|ECE$\\downarrow$|AECE$\\downarrow$|\n|CE|73.96|9.10|9.24|\n|Mixup|75.84|7.07|7.09|\n|CRL|73.83|8.47|8.47|\n|MbLS|75.39|4.07|4.14|\n|RegMixup|75.64|5.34|5.42|\n|RankMixup|74.86|3.93|3.92|\n|CALS|76.44|1.46|**1.32**|\n|Ours|**79.87**|**1.32**|1.35|\n\n**Table A:**\n|SwinTransformerV2|ImageNet|||\n|-|-|-|-|\n|Metrics|ACC|ECE$\\downarrow$|AECE$\\downarrow$|\n|CE|75.60|9.95|9.94|\n|LS|75.42|7.32|7.33|\n|FL|75.60|3.19|3.18|\n|FLSD|74.70|2.44|2.37|\n|MbLS|77.18|1.95|1.73|\n|CALS|77.10|1.61|**1.69**|\n|Ours|**81.08**|**1.49**|1.86|\n\n**Q1-4**: Potential reversal of the following CE/FL explanations in ablation (lines 382–384).\n> \"In contrast, CE and FL losses often require temperature adjustments, with CE favoring sharper labels and FL for softer ones, aligning with our theoretical expectations from Section 3.3.\"\n\n**A1-4**: The analysis corresponds to the searched temperature values of Mixup and CSM in Table 4, where CE results sometimes involve a searched temperature larger than 1.0 compared using Mixup (Mixup (**CE**): **T = 1.3**), while FL results produce a searched temperature of **T = 0.9 < 1** compared with our CSM (CSM (**FL**)). These two specific results highlight the nature of CE and FL losses.\n\nAs studied by existing works, a higher post-temperature can indicate the model's over-confidence while a lower one suggests under-confidence of the pre-temperature model. Therefore, with soft labels during training, these phenomena indicate a preference/bias of fitting different samples for the adopted losses, *i.e.*, harder labels (*e.g.*, close to one hot) vs. softer labels (*e.g.*, mixup labels with $\\lambda=0.5$).\n\n**Q1-5**: Confidence-balancing comparison across CE, FL, and L2.  \n\n**A1-5**: We explicitly compute the average balance scores to illustrate the confidence balancing results here:\n|Loss Objectives|CE|Our Loss|FL|\n|-|-|-|-|\n|CIFAR-100|-0.1438|-0.1330|-0.0393|\n|Relative Value|**-0.0108**|**0.0000**|**+0.0937**|\n\nOur loss shows a clear confidence balance between CE and FL, confirming its effectiveness, though empirical values are typically negative due to the rareness of indistinguishable pairs and easier learning of high-confidence samples in practical experiments.\n\n**References**\n\n[1] Park, H., Noh, J., Oh, Y., Baek, D., & Ham, B. (2023). Acls: Adaptive and conditional label smoothing for network calibration. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3936-3945).\n\n[2] Liu, B., Rony, J., Galdran, A., Dolz, J., & Ben Ayed, I. (2023). Class adaptive network calibration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16070-16079).", "claims_and_evidence": "- The motivation and necessity of semantic mixing from the perspective of network calibration are well articulated. Additionally, the drawbacks of existing data-driven methods (mixup-based approaches) are clearly defined.", "methods_and_evaluation_criteria": "- Leveraging conditional diffusion models, specifically via a pre-trained diffusion network, to generate semantically mixed images is technically novel within the context of network calibration. \n- Further innovation is demonstrated through the identification and resolution of limitations associated with generated labels by introducing a calibration-oriented reannotation process.", "theoretical_claims": "Further clarification and verification are needed regarding the balanced loss section. Specifically, more clarification on why the proposed L2 loss functions as a balanced loss would be helpful.", "experimental_designs_or_analyses": "- The experimental results across various networks and datasets demonstrate superior performance compared to existing state-of-the-art methods.\n-  Although generalization capability is emphasized, experimental validation on larger datasets such as ImageNet and different network architectures such as Transformers appears insufficient. Additionally, comparisons with recent state-of-the-art methods like CALS (CVPR 23) and ACLS (ICCV 23) are missing,.\n- The experiment described in Table 2, which compares training times under identical conditions, is commendable, considering the potential increase in training duration due to the diffusion network. The necessity and effectiveness of reannotation are well-demonstrated in Figure 3.\n- In the ablation study (lines 382–384), please confirm if the explanations regarding CE and FL are reversed, particularly concerning temperature.\n- It would also be beneficial to provide a comparison illustrating the degree of confidence-balancing achieved by using CE, FL, and L2 losses.", "relation_to_broader_scientific_literature": "This approach effectively improves not only the network’s calibration capability but also enhances its interpretability and accuracy.", "essential_references_not_discussed": "Recent state-of-the-art methods, such as ACLS: Adaptive and Conditional Label Smoothing for Network Calibration (ICCV 2023) and Class Adaptive Network Calibration (CVPR 2023), have not been included in the reference list. These works offer significant contributions to network calibration and should be considered for inclusion to provide a more comprehensive and up-to-date overview of current methodologies.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fOyAvhBROF", "reviewer": "Reviewer_xs6o", "summary": "This paper presents Calibration-aware Semantic Mixing (CSM), a novel approach to improving model calibration by generating high-quality augmented data with soft labels. Unlike traditional augmentation methods that rely on one-hot labels, CSM leverages diffusion models to create semantically mixed images with confidence scores. The authors introduce a reannotation strategy based on CLIP features and explore different loss functions, demonstrating that L2 loss leads to better calibration. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet show that CSM surpasses existing calibration techniques.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer xs6o\n\nThank you for your positive and insightful feedback! Here are our responses:\n\n**Q2-1**: Additional details and proofs in the supplementary material.\n\n**A2-1**:\nThank you for the kindly comments on the theoretical soundness. The claims made in our paper (including deducted results in Eq.(6)-(10) and Prop. 3.2-3.4) are sufficiently proved in Appendix A. In detail:\n- Eq.(6)-(7) corresponds to Eq.(14)-(15) with \n  - Assumption 1: Classification optimal classifier $\\operatorname{E}(\\cdot)$ ensures the likelihood ratio of different classes;\n  - Assumption 2: Regarding the feature as the affine set elements of class prototypes with an orthogonal deviation.\n- Eq.(8)-(9) is proved by Eq.(15)-(19).\n- The result of Eq.(10) is acquired from Eq.(9) with the class factor invariance assumption.\n- Proposition 3.4 is first proved through Eq.(22)-(26) with the assumptions given in Definition 3.1.\n- To prove Proposition 3.2 and 3.3, we first give a general analysis of the problem in Line 712-791 (or Eq.(27)-(37)), then prove them by Lemma A.1 and Lemma A.2, respectively. Note that the proof for CE is unconditional, while for FL it's proved with assumption that $\\gamma_{FL}=1.0$ and we empirically find FL more imbalanced with larger $\\gamma_{FL}$.\n\nThese detailed descriptions illustrate the overall framework of the theoretical analysis. We will include connective details and key deductions in the main paper.\n\n**Q2-2**: Additional evaluations on other architectures.\n\n**A2-2**: Our method performs equally or more effectively to others with the Swin-Transformer architecture. Please refer to **A1-3 Table A**.\n\n**Q2-3**: More details on efficiency and resource consumption.\n\n**A2-3**: Thank your for your suggestion. For efficiency analysis, we compare explicitly in terms of the training time as follows:\n\n**Table B:**\n|Methods|CE|MbLS|Mixup|RegMixup|RankMixup|Ours|Ours(EQ-DATA)|\n|-|-|-|-|-|-|-|-|\n|Training Time|2.63h|2.65h|3.48h|3.50h|4.30h|4.28h|2.64h|\n\nOne can see the **number of augmented samples per batch** is the **major factor** for training time. CSM outperforms others in ECE/AECE while maintaining reasonable speed. Even with equalized training samples (EQ-DATA, Table 2), it achieves competitive calibration. CSM runs on a single A4000. Augmented samples need no re-generation across model/loss/annotation changes, enabling efficient modular study.\n\nKey details will be added to the main paper.\n\n**Q2-4**: Comparison with [1], a test-time augmentation (TTA) post-hoc calibration method.\n\n**A2-4**: We compare with [1] by evaluating CSM + [1] as follows:\n|Metrics|ECE|AECE|\n|-|-|-|\n|Ours|**1.29**|1.63|\n|Ours+[1]|1.39|**1.53**|\n\nOur integrated method balances ECE and AECE, achieving an optimized AECE of **1.53** on CIFAR-100. Compared to [1] using test-time sample-wise scaling, CSM employs training-time augmentation with inter-sample augmentations to expand the proximity space, enhancing calibration robustness. We will cite [1] and provide full comparisons in the main paper.\n\n**Q2-5**: Computational overhead of diffusion-based augmentation.\n\n**A2-5**: Our analysis in Appendix C shows that CSM requires **few augmented samples** to launch effectively, ensuring low computational costs. Generating augmented sets takes **~4 hours** (CIFAR-10/100) or **~9 hours** (Tiny-ImageNet) on an RTX4090 GPU, comparable to typical training times. Crucially, CSM **eliminates re-generation** when model architectures/parameters change, further enhancing efficiency through its decoupled design. This validates CSM's computational efficiency.\n\n**Q2-6**: Hyperparameter sensitivity analysis.\n\n**A2-6**: We have analyzed parameter $s$ and No. of augmented samples in Appendix C. We analyze $N_{aug}$ (refer to **A4-2 Table C**) and check sensitivity in **A4-2**. Larger $N_{aug}$ generally enhances our method, while within appropriate ranges of other parameters, it yields stable results.\n\n**Q2-7** Possibility to integrate our CSM with [2] to enhance calibration and robustness.\n\n**A2-7**: Thank you for this insightful question. We conduct post-hoc experiments to integrate our method's outputs with [2], acquiring the following result:\n|Errors$\\downarrow$|ECE|MCE|AECE|PIECE|\n|-|-|-|-|-|\n|Ours|**1.29**|**0.21**|**1.62**|3.16|\n|Ours+[2]|1.89|0.73|1.82|**3.11**|\n\nDue to limited time, we simply integrate CSM with [2] without further adjustments. Although a simple combination of them doesn't yield superioir ECE/AECE results, we find that the proximity-informed metric PIECE displays better results, which validates the robustness growth related to proximity from the integration. We will cite [2] for analysis.\n\n**References**\n\n[1] Hekler, A., Brinker, T. J., & Buettner, F. (2023, June). Test time augmentation meets post-hoc calibration: uncertainty quantification under real-world conditions. AAAI.\n\n[2] Xiong, M., Deng, A., Koh, P. W. W., Wu, J., Li, S., Xu, J., & Hooi, B. (2023). Proximity-informed calibration for deep neural networks. NeurIPS.", "claims_and_evidence": "CSM enhances calibration by generating realistic semantically mixed samples, as evidenced by Figure 1.\nReannotating confidence scores improves performance, which is validated through ablation studies.\nL2 loss provides a better balance in learning, leading to improved calibration, supported by both theoretical analysis and empirical results.", "methods_and_evaluation_criteria": "The paper evaluates model calibration using standard metrics, including Expected Calibration Error (ECE) and Adaptive ECE (AECE), across multiple datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet). Additionally, Reliability diagrams and ablation studies are conducted for further comparison.", "theoretical_claims": "The authors provide theoretical justifications for their reannotation strategy and choice of loss function. While the analysis appears rigorous, additional details and proofs in the supplementary material could further strengthen their claims.", "experimental_designs_or_analyses": "The experiments primarily focus on ResNet-based models, and additional evaluations on other architectures (e.g., Transformer-based models) would be beneficial to confirm the generalizability of CSM.\nComputational overhead is not explicitly discussed—more details on efficiency and resource consumption would enhance the paper.", "relation_to_broader_scientific_literature": "The work builds on existing research in model calibration and data augmentation, presenting a novel approach by incorporating diffusion models for calibration-aware augmentations. However, the discussion could benefit from additional comparisons to post-hoc calibration methods.", "essential_references_not_discussed": "The paper should consider discussing and comparing its approach with existing post-hoc calibration methods, particularly [1] Test Time Augmentation Meets Post-hoc Calibration, which is closely related to data augmentation for calibration.", "other_strengths_and_weaknesses": "Strengths:\nNovel method integrating diffusion models for calibration.\nStrong empirical results demonstrating superior performance over existing techniques.\nComprehensive evaluation using standard calibration metrics.\nWeaknesses:\nLimited hyperparameter analysis—the sensitivity of CSM to different configurations is not well explored.\nUnclear computational cost—the efficiency trade-offs of using diffusion models for augmentation should be discussed.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Since semantic mix augmentation effectively fills sparse regions in the data space and improves local data proximity, could this approach be integrated with [1] to enhance not only calibration but also model robustness? Exploring this synergy could yield further improvements in generalization and uncertainty estimation.\n\nReference: [1] Proximity-Informed Calibration for Deep Neural Networks", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hGPtbwM15t", "reviewer": "Reviewer_WPRF", "summary": "This paper introduces a novel framework, Calibration-aware Semantic Mixing (CSM), designed to improve model calibration. The key contribution lies in addressing the limitations of one-hot labeled datasets by proposing a data augmentation technique that leverages semantic mixing to generate diverse samples via diffusion models. The paper also introduces reannotation techniques to enhance confidence annotation accuracy and explores different loss functions to achieve confidence-balanced learning. Experimental results demonstrate that CSM surpasses existing calibration methods, delivering superior performance across multiple benchmarks and tasks.\n\n## update after rebuttal\nMy concerns have been addressed and would like to recommend accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer WPRF\n\nThank you for your encouraging feedback on the clarity, soundness, and comprehensive evaluation of our work. We truly appreciate your thoughtful suggestions for clarity and comprehensive validation. Here are our responses to the suggestions:\n\n**Q3-1**: Clarify Proposition 3.4 (L2 loss vs. CE/FL).\n\n**A3-1**: Thank you for commenting on the clarity issue. We need to clarify that there exists a **typo** in Proposition 3.4 potentially hindered understanding. Proposition 3.4 should have been presented as\n\n- $\\forall \\delta \\ge 0$, we have $\\beta(p^{L2}_1, p^{L2}_2) = 0$,\n\nwhich is an equation rather than an inequality for the $\\mathcal{L}_2$ loss's balance function, meaning that when two similar samples exceed the model's discriminability, $\\mathcal{L}_2$ loss tends to **balance the learned labels** of the harder and softer instances, instead of tending to fit a specific one of them. Note that the proof of Proposition 3.4 we provided in Appendix A.3 does prove that $\\beta(p^{L2}_1,p^{L2}_2) = 0$.\n\nAs proved in Appendix A, easier samples with $\\delta \\ge \\|q^{L2}_1, q^{L2}_2\\|$ would generally have $\\|p^{L2}_i, q^{L2}_i\\| = 0, i=1,2$. Smaller $\\delta$s indicate difficulty for the learned model to separate the outputs for both samples, hence introducing a balancing problem. A theoretically non-zero balancing score $\\beta$ means one of $\\|p^{L2}_i, q^{L2}_i\\|, i=1,2$, is minimized more completely, indicating the imbalanced nature of the objective.\n\nAmong the three, only the L2 loss theoretically zeros the $\\beta$ balancing score, indicating its ability to balance over- and under-confidence of difficult soft-labeled sample pairs, hence being a superior loss for calibration with our augmented samples. In practice, the soft-label distribution may disturb such balance, which can be one of our future study.\n\nWe will correct this typo in the revised main paper.\n\n**Q3-2**: Baseline for directly adopting CLIP labels. (Var. 1)\n\n**Q3-3**: Baseline for diffusive sample augment without semantic mixing. (Var. 2)\n\n**A3-2, 3-3**:\nWe evaluate these two variants and compare them with our proposed CSM as follows:\nCIFAR-10:\n| Variant | ACC | ECE$\\downarrow$ | AECE$\\downarrow$ |\n| - | - | - | - |\n|Var. 1|*92.13*|2.68(0.92)|2.67(0.88)|\n|Var. 2|**96.12**|2.45(0.96)|2.44(1.13)|\n|CSM (Ours)|95.79|**0.54(0.54)**|**0.33(0.33)**|\n\nCIFAR-100:\n| Variant | ACC | ECE$\\downarrow$ | AECE$\\downarrow$ |\n| - | - | - | - |\n|Var. 1|*66.60*|52.78(1.36)|52.78(**1.11**)|\n|Var. 2|**79.24**|10.84(2.48)|10.84(2.41)|\n|CSM (Ours)|78.84|**1.29(1.29)**|**1.63**(1.63)|\n\nFrom these results, we can have these valuable observations:\n1. The vanilla CLIP annotation method yields the worst ACC and pre-temperature calibration errors, primarily due to the noisy information by annotating all classes. Such degradation is significant for CIFAR-100, in which there are more classes so that the noise is severer.\n2. Directly adopting class-conditioned augmentations from the diffusion model can slightly rise the prediction accuracy, as also evidenced by the generative model augmented classification literature. However, as it does not contain soft-labeled samples, Var. 2 fails to improve model calibration.\n\nTherefore, we can conclude that models are effectively calibrated only when adopting the proper data and re-annotation scheme.", "claims_and_evidence": "The claims presented in the paper are well-supported by empirical evidence from the experiments.", "methods_and_evaluation_criteria": "The proposed method is both novel and well-motivated, offering fresh insights into model calibration. The evaluation follows standard practice in this field, employing accuracy, Expected Calibration Error (ECE), and post-temperature scaling as key metrics.", "theoretical_claims": "One concern arises in Proposition 3.4, where the paper claims that L2 loss outperforms cross-entropy (CE) and focal loss. The reasoning behind this claim is unclear and requires further clarification.", "experimental_designs_or_analyses": "The overall experimental design and analysis are well-structured and reasonable. The authors compare the proposed calibration technique against several widely-used calibration algorithms on diverse models and datasets. Ablation studies effectively highlight the contributions of different components.\nHowever, I have two major concerns:\n- Calibrated Reannotation – The authors utilize CLIP’s visual encoder for reannotation, but they do not discuss or compare it with a simple baseline that directly adopts CLIP outputs as labels. Evaluating this baseline would help assess the added benefit of the proposed reannotation approach.\n- Calibration-aware Data Augmentation – The study proposes a semantic mixing strategy for generating calibrated samples using diffusion models. However, a crucial baseline is missing: directly using generated images from diffusion models without semantic mixing. Evaluating this approach would provide a clearer understanding of semantic mixing’s contribution.", "relation_to_broader_scientific_literature": "Addressing model calibration from a data-driven perspective is an interesting and promising direction. The results across different models and datasets suggest strong potential for real-world applications. The proposed algorithm could contribute significantly to the field of trustworthy AI, enhancing model reliability and confidence estimation in diverse applications.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Overall, the paper is well-written and easy to follow. The proposed semantic mixing framework is conceptually sound and coherently presented. The evaluation is comprehensive, and the method demonstrates superior calibration performance compared to existing baselines. However, as noted earlier, some concerns remain regarding the theoretical claims (Proposition 3.4) and the experimental design (missing baselines for reannotation and augmentation methods).", "other_comments_or_suggestions": "None.", "questions_for_authors": "- Could you discuss or evaluate simple baselines for calibrated reannotation, such as directly adopting CLIP outputs as labels?\n- Could you discuss or evaluate simple baselines for calibration-aware data augmentation, such as using generated images without semantic mixing?\n- Could you clarify Proposition 3.4, particularly regarding why L2 loss outperforms cross-entropy and focal loss?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bdjUMjj2wF", "reviewer": "Reviewer_jJxJ", "summary": "This paper propose Calibration-aware Semantic Mixing, a model calibration approach using diffusion-based data augmentation, like “semantic mixup”.Unlike traditional one-hot labeling, CSM generates mixed samples with soft labels with the CLIP. The authors introduce a reannotation technique using CLIP features and investigate the influence of loss functions prove L2 loss is good for enhances calibration.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer jJxJ\n\nThanks for your helpful suggestions! Here’s our response:\n\n**Q4-1**: Training computational cost and memory usage compared to existing methods.\n\n**A4-1**: As also analyzed in **A2-3**, we compare the computational efficiency in terms of the training time in **A2-3 Table B**:\n\nWe can conclude that the **number of augmented samples per batch** is the **major factor** in training time. Our CSM maintain relative efficiency considering the calibration effectiveness. When compared on the EQ-DATA setting in the main paper Table 2, CSM achieves competitive calibration results with an equalized training time.\n\nSuch factor also decides memory usage. CSM uses ~$\\frac{3}{2}$ memory compared to Mixup/RegMixup, while using ~equal memory compared to the RankMixup (depending on setep).\n\nRegarding resource consumption, we generate augmented samples and train our model on one A4000 device. It's worth noting that CSM **needs no re-generation** when switching model/objectives/re-annotation methods, making it more efficient for decoupled study of these modules.\n\nWe will include the key information in the main paper.\n\n**Q4-2**: Hyperparameter sensitivity.  \n\n**A4-2**: \nWe have analyzed hyperparameter $s$ and the number of augmented samples in Appendix C. We provide more analyses about **the number of augmented samples per training sample** (denoted as $N_{aug}$) here.\n**Table C:**\n| $N_{augs}$ | 1 | 2 | 3 |\n| - | - | - | - | \n| ECE: CIFAR-10 | 0.83 | 0.54 | 0.39 |\n| ECE: CIFAR-100 | 2.07 | 1.29 | 1.74 |\n\nIt can be observed that adding the number of accompanied augmentations per dataset sample can generally improve the final calibration performance. This is because a larger number of $N_{augs}$ can sample more sufficient proximal data for training, better filling the domain space and providing more accurate confidence estimation.\n\nFor **sensitivity analysis**, we have the following key observations to enhance our experiments considering **existing results** in Appendix C:\n1. Our proposed CSM is not sensitive to the total number of augmented samples. A relatively smaller quantity can still make CSM effective.\n2. CSM is relatively sensitive to the scaling factor $s$ as it is related to the temperature. Nevertheless, under an appropriate range of $s$, the method can perform consistently well.\n\n**Q4-3**: Evaluation on the transformer architecture.\n\n**A4-3**: Evaluations on the Swin-Transformer architecture verify our method's equal or stronger effectiveness. Please refer to **A1-3 Table A**.\n\n**Q4-4**: Failure cases of CSM.\n\n**A4-4**: While our method can perform well on all the evaluated datasets, we admit there are some cases CSM fails. For instance, although CSM surpass compared methods on ECE/AECE, some post-temperature results on the CIFAR-100 dataset are not comparative to the SOTA methods. Specifically, they exhibit balanced pre- and post-temperature results (searched $T = 1$) with remarkable pre-temperature ECE values but slightly larger calibration errors after temperature scaling. Such phenomenon is also stated in the calibration literature [1], where they found a balance exists in the two results. Considering both results together, our CSM can still achieve satisfactory  model calibration.\n\n**Q4-5**: Will the choice of generative models affect the performance?\n\n**A4-5**: \nThe choice of the generative model does influence the final prediction results. Some existing study [2] has already proved that adopting a better generative model could improve the classification model's robustness. We anticipate that a better generative backbone can achieve superior confidence calibration using our CSM.\nThis aligns with our empirical experiences that using the ordinary Stable Diffusion architecture, we can only achieve suboptimal results as shown in the table:\n| Gen. Model | ACC | ECE$\\downarrow$ | AECE$\\downarrow$ |\n| - | - | - | - |\n|SD [3]|76.87|1.93|1.78|\n|EDM [4] (in our CSM)|**78.84**|**1.29**|**1.63**|\n\nTherefore, we anticipate that a typical GAN with less parameters or lower fidelity compared to diffusion models would yield worse results.\n\n**References**\n\n[1] Wang, D. B., Feng, L., & Zhang, M. L. (2021). Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. Advances in Neural Information Processing Systems, 34, 11809-11820.\n\n[2] Wang, Z., Pang, T., Du, C., Lin, M., Liu, W., & Yan, S. (2023, July). Better diffusion models further improve adversarial training. In International conference on machine learning (pp. 36246-36263). PMLR.\n\n[3] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).\n\n[4] Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35, 26565-26577.", "claims_and_evidence": "1. CSM improves model calibration by introducing semantically meaningful augmentations, validated through ECE and AECE reductions.\n2.  L2 loss leads to balanced learning, demonstrated by both theoretical insights and empirical results.", "methods_and_evaluation_criteria": "1. The paper introduces a diffusion-based augmentation method using L2 loss supported by theoretical evidence.\n2. This paper is evaluate with ECE, AECE on CIFAR-10, CIFAR-100, and Tiny-ImageNet, using ResNet-50/101, Wide-ResNet-26-10 and DenseNet-12", "theoretical_claims": "The authors provide a theoretical justification for their reannotation strategy and choice of loss functions", "experimental_designs_or_analyses": "1. extensive comparisons with other  train time calibration techniques.\n2. AUROC (%) for robustness evaluation under distribution shifts.\n3. Calibration performance with post hoc calibration methods.\n4. Reliability diagrams and ablation studies", "relation_to_broader_scientific_literature": "This paper is highly related to the field of calibration, with the relationship to test time augmentation and label smoothing.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weaknesses: \n1. Limited discussion on computational efficiency\n2. Hyperparameter sensitivity analysis is not well explored.\n3. No Transformer model is included for comparison", "other_comments_or_suggestions": "1. Provide comparisons of computational cost with existing calibration techniques.\n2. Include failure cases where CSM does not improve calibration performance", "questions_for_authors": "1. How does the choice of diffusion model affect the performance of CSM? How about we use other generative models. There are some  generative model doing interpolation , like GANs. Does it helps?\n2. How does CSM compare to Mixup in terms of training efficiency and memory usage?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper got three weak accepts and an accept. Among the important concerns raised by reviewers in the pre-rebuttal stage are:

- limited hyperparameter analyses
- comparison with some recent calibration methods
- clarification on a theoretical claim
- limited discussion on computational efficiency 
- missing experiments with transformer model

All reviewers acknowledged the rebuttal and decided to keep their original (pre-rebuttal) rating of either weak accept or accept. Given that the rebuttal provides adequate responses to reviewer's comments and no major concern remain, the decision is to recommend acceptance of the paper. Authors are recommended to include important reviewer's comments in the final version.
Decision: Accept (poster)