Do not trust what you trust: Miscalibration in Semisupervised Learning
Authors: Shambhavi Mishra
Balamurali Murugesan
Ismail Ben Ayed
Marco Pedersoli
Jose Dolz
Venue: Accepted by TMLR ()

Abstract: State-of-the-art semi-supervised learning (SSL) approaches rely on highly confident predictions to serve as pseudo-labels that guide the training on unlabeled samples. An inherent drawback of this strategy stems from the quality of the uncertainty estimates, as pseudo-labels are filtered only based on their degree of uncertainty, regardless of the correctness of their predictions.  Thus, assessing and enhancing the uncertainty of network predictions is of paramount importance in the pseudo-labeling process. In this work, we empirically demonstrate that SSL methods based on pseudo-labels are significantly miscalibrated, and formally demonstrate the minimization of the min-entropy, a lower bound of the Shannon entropy, as a potential cause for miscalibration. To alleviate this issue, we integrate a simple penalty term, which enforces the logit distances of the predictions on unlabeled samples to remain low, preventing the network predictions to become overconfident. Comprehensive experiments on a variety of SSL image classification benchmarks demonstrate that the proposed solution systematically improves the calibration performance of relevant SSL models, while also enhancing their discriminative power, being an appealing addition to tackle SSL tasks.

Published in Transactions on Machine Learning Research (09/2024)
Do not trust what you trust: Miscalibration in Semi- supervised Learning
Shambhavi Mishra Balamurali Murugesan" 1.2
shambhavi. mishro. @etsmml net bolamuralim. 1993@gmmil. com ismail. benayed gmail. com marco marco. pedersob@@essmm.. ca jose. dolz@etsmlllccm ca
Ismail Ben Ayed Marco Pedersoli
Jose Dolz ..
LIVIA, ETS Montréal, Canada International Laboratory on Learning Systems (ILLS),
McGILL ETS MILA CNRS- Université Paris-Saclay CentraleSupékee, Canada
Reviewed OpenReview: https: openreview. net/ formm? id= 1WgLLYBBN
Abstract
State-of-the-art semi-supervised learning (SSL) approaches rely highly confident predic- tions to serve as pseudo-labels that guide the training on unlabeled samples. An inherent drawback of this strategy stems from the quality of the uncertainty estimates, as pscudo- labels are filtered only based on their degree uncertainly, regardless of the correctness of their predictions. Thus, assessing and enhancing the uncertainty of network predictions of paramount importance in the pscudo-labeling process. In this work, we empirically demonstrate that. SSL methods based on pseudo-labels are significantly miscalibrated, and formally demonstrate the minimization of the min-entropy, lower bound of the Shannon entropy, as potential cause for miscalibration. To alleviate this issue, we integrate simple penalty term, which enforces the logit distances of the predictions on unlabeled samples to remain low, preventing the network predictions to become overconfident. Comprehensive experiments on a varicty of SSL image classification benchmarks demonstrate that the pro- posed solution systematically improves the calibration performance of relevant SSL models, while also enhancing their discriminative power, being an appealing addition to tackle SSL wwt..o///aaaaaaaaaaeeeeeeeeeeeee
tasks. Code:
1 Introduction
Deep learning models have significantly advanced the state-of-the-art across myriad of tasks (Masana et al., 2022; Minace et al., 2021). Nonetheless, their success has been often contingent the availability of large amounts of labeled data. Having access to curated large training datasets, however, is not easy, and often involves a tremendous human labor, particularly in those domains where labeling data samples requires expertise, hindering the progress to address a broader span of real-world problems.
Semi-supervised learning (SSL) (Chapelle et al., 2006) mitigates the need for large labeled datasets by pro- viding means of efficiently leveraging unlabeled samples, which are easier to obtain. This learning paradigm has led to plethora of approaches, which can be mainly categorized into consistency regularization (Bach- man et al., 2014; Laine & Aila, 2017) and psoudo-labeling (Loe et al., 2013; Xic et al., 2020b) methods. Indeed, state-of-the-art SSL approaches (Wang el al., 2023; Zhang et. al., 2021; Sohn el al., 2020; Chen et al., 2023; Zheng et al., 2022) combine both strategies, obtaining promising results. The underlying idea of these approaches follows the low-density and smoothness assumptions SSL (Chapelle & Zien, 2005). In particular, incorporating pseudo-labels from unlabeled data points into the training process aids the deci- sion boundary to lie in low density regions. Furthermore, consistency regularization assumes that the same

---

Published in Transactions on Machine Learning Research (09/2024)
unlabeled data point should yield the same pseudo-label regardless of the perturbations applied, implicitly capturing the underlying data manifold. As the model can produce very uncertain predictions for strongly perturbed samples, these techniques incorporate a threshold, either fixed (Sohn et al., 2020) or adaptive (Wang et al., 2023; Zhang et al., 2021), to only integrate very confident samples in the training loss. Thus, all samples whose predicted probabilities are highly confident are trusted by these methods as supervisory signals for subsequent steps, even when their predictions are wrong-
Despite being a standard practice, recent evidence (Chen et al., 2023) suggests that the amount of incorrect pseudo-labels integrated into the training is not negligible, potentially undermining the optimization process. Hence, given that the generated pscudo-labels play a significant role in the training of SSL models, producing accurate uncertainty estimates is of pivotal importance. Nevertheless, while we have observed a remarkable progress in their discriminative performance, little attention has been paid to studying, and improving, the calibration of SST. approaches. Motivated by these findings, in this work we address the critical yet under- explored issue of miscalibration SSL, particularly for those methods basod on pseudo-labeling. To this end, we select set of relevant and recent. stralegies that build on pseudo-labels and consistency regularization (Sohn et al., 2020; Wang et al., 2023; Zhang et al., 2021) and empirically demonstrate that they are poorly calibrated. Furthermore, we explore the underlying causes of this issue and shed light about the potential reasons that produce overconfident pseudo-labeling SSL models. Last, inspired by these observations we propose a simple solution to tackle miscalibration in these models. Our contributions can be therefore summarized as follows:
1. We empirically demonstrate that state-of-the-art SSL approaches based on pseudo-labels are sig- nificantly miscalibrated. Through our analysis, we formally show that the cause of miscalibration is the minimization of a min-entropy term, specific case of the Rényi family of entropies, on a considerably large proportion of unlabeled samples, which forces the model to yield overconfident predictions. Indeed, the ensuing gradients from this term strongly push the unlabeled samples to be highly confident since the beginming of the training, even their class predictions are incorrect. This results in large logit magnitudes, a phenomenon known to cause miscalibration.
2. Based on our observations, we propose to use a simple solution that refrains the model from pushing unlabeled samples towards very confident regions, improving the calibration of pseudo-labeling SSL methods. More concretely, we add a penalty term on a dominant set of unlabeled samples, which enforces logit distances to remain low, alleviating the miscalibration issue.
3. Through a comprehensive set of experiments, we empirically demonstrate that the proposed approach consistently improves the uncertainty estimates of a set of very relevant and recent state-of-the-at pseudo-labeling SSL approaches on popular benchmarks, in both standard and long-tailed classifi- cation tasks. In addition, in most cases, the proposed solution further improves their discriminative performance.
2 Related work
2.1 Semi-supervised learning
Prevailing semi-supervised learning (SSL) approaches heavily rely on the concept of pseudo-labels (Lee et al., 2013; Shi al., 2018) and consistency regularization (Bachman al., 2014; Laine & Aila, 2017; Sajjadi et al., 2016; Tarvainen & Valpola, 2017: Miyato et al., 2018), where labels are dynamically generated for unlabeled data throughout the training process. Essentially, these methods exploit the role of perturbations, by stochastically perturbing the unlabeled images and enforcing consistency across their predictions. This consistency is achieved by a pseudo-supervised loss, where the predictions over the strong perturbations are supervised by the pseudo-labele obtained from the weak perturbations (Xie et al., 2020a; Wang et al., 2023; Zhang et al., 2021; Chen et al., 2023; Zheng et al., 2022; Yang et al., 2023; Xu et al., 2021; Zheng et al., 2022). This paradigm to use artificial labels facilitates the integration of unlabeled data into the learning process, thereby augmenting the training set and improving the model ability to generalize. To avoid introducing noise in the pseulo-supervision process, these approaches retain given pseudo-label only if the model
2

---

Published in Transactions on Machine Learning Research (09/2024)
assigns high probability to one of the possible classes. This strategy effectively harnesses the information from unlabeled data by leveraging the network confidence in assigning pseudo labels, enabling the model to access the valuable knowledge encapsulated within these unlabeled samples. Thus, the main differences across the different approaches based on pseudo-labels lie on the mechanism introduced to select confident samples. For example, FixMatch (Sohn et al., 2020) and ShrinkMatch (Yang et al., 2023) employ a fixed threshold, whereas Dash (Xu et al., 2021) proposes a dynamically growing threshold. Other approaches, such as FlexMatch (Zhang et al., 2021) and FreeMatch (Wang et al., 2023), integrate class-adaptative thresholds, considering larger amount of unlabeled data which is otherwise ignored, especially at the early stage of the training process.
Limitations of pseudo-labeling SSL from colibration standpoint. Although these methods enhance the discriminative power of deep models, their calibration has been significantly overlooked, lacking of principled strategies to simultanoously improve the classification performance while maintaining the quality of the uncertainty estimates. As pseudo-labeling statc-of-the-art SSL approaches trust highly confident artificial labels derived from unlabeled samples, understanding how this confidence is assigned, and ensure its accuracy, is of paramount importance. In this context, (Rizve et al., 2021) addresses the challenge of selecting reliable pscudo-labels by incorporating uncertainty estimates into the pscudo-labeling process. Very recently, BAM (Loh et al., 2023) studied miscalibration in SSL, and proposed to replace the last layer of a neural network by a Bayesian layer. Nevertheless, the source of miscalibration was not explored, and in-depth empirical results were not reported.
2.2 Calibration
Recent evidence (Guo et al., 2017; Müller et al., 2019; Mukhoti et al., 2020) has shown that deep networks are prone to make overconfident predictions due to miscalibrated output probabilities. This emerges as a byproduct of minimizing the prevalent cross-entropy loss, which occurs when the softmax predictions for all training samples fully match the ground-truth labels, and thus the entropy of output probabilities is encouraged to be zero. To mitigate the miscalibration issue, and to better estimate the predictive uncertainty of deterministic models, two main families of approaches have emerged recently: post-pracessing (Guo al., 2017; Ding et al., 2021; Tomani et al., 2021) and learning (Pereyra et al., 2017; Müller et al., 2019; Mukhoti et al., 2020; Liu et al., 2022; Cheng & Vasconcelos, 2022; Liu et al., 2023; Noh et al., 2023; Murugesan et al., 2023; Larrazabal et al., 2023; Park et al., 2023) approaches. Among the post-processing strategies, Temperature Scaling (TS) (Guo et al., 2017) has been a popular alternative, which manipulates logit outputs monotonely, by applying a single scalar temperature parameter. This idea is further extended in (Ding et al., 2021), where a local TS per pixel is provided by a regression neural network. Nevertheless, despite the simplicity of these methods, learning approaches have arisen more powerful choice, in this scenario the model adapts to calibration requirements alongside its primary learning objectives, optimizing both aspects simultancously. Initial attempts integrated learning objectives that maximize the entropy of the network softmax predictions either explicitly (Pereyra et al., 2017; Larrazabal et al., 2023), or implicitly (Mukhoti et al., 2020; Müller et al., 2019; Cheng & Vasconcelos, 2022). In order to alleviate the non-informative nature of simply maximizing the entropy of the softmax predictions, recent work (Liu et al., 2022; 2023) has presented a generalized incquality constraint, which penalizes logits distances larger than pre-defined margin.
3 Semi-supervised learning and calibration
3.1 Problem statement
In the semi-supervised learning scenario, the training datasct is composed of labeled and unlabeled data points. In this setting, let. D,. = (xx,, be the labeled dataset and Dr = (x,),vv the unlabeled dataset, where Nr and Nu represent the number of labeled and unlabeled samples, respectively, and Nı << Ny. Furthermore, x; E Rd is d-dimensional training sample, with y; E 10 its associated ground truth (only for labeled data points, i.c., x; E D2) that assigns one of the K classes to the sample. The objective is, given batch of labeled and unlabeled samples, to find optimal set of parameters of deterministic
3

---

Published in Transactions on Machine Learning Research (09/2024)
function, e.g., neural network, parameterized by 0, by using compounded loss including labeled and unlabeled term. Given an input image x;, the network will generate vector of logits fo = 1 E TR which can be converted to probabilities with the softmax function.
Supervised loss. The supervised objective is typically formulated as a standard cross-entropy H between the one-hot encoded labels y, and the corresponding softmax predictions p(y x;) [0, 1] of labeled samples:
Ls =
=- E logog((y)x) iED
(1)
Unsupervised loss. Most modern SSL approaches adopt a consistency regularization strategy based on pseudo-labeling for the unsupervised objective. To this end, the same image x; follows a set of weak and strong augmentations, denoted as w(-) and 92(-), respectively. Thus, a pseudo-cross-entropy term on the unlabeled training dataset can be formulated as:
Lu = Σ 8z log paypp(y(xx)))
iEDD
where y, is the one-hot encoding of the arg max of the softmax probabilities for the weak augmented version, To avoid that samples with high uncertainty, and possibly incorrect predictions,
i.e., arg max intervene in the optimization of the term in eq. (2), a common strategy is to retain only discrete pseudo-labels whose largest class probability fall above a predefined threshold (Lee al., 2013; Wang al., 2023; Sohn al., 2020; Zhang al., 2021). This results in the following objective:
Cu --
Σ iEDO
1( 2 T) log (oyloo(pl((())))
(3)
where + is the predefined threshold. Note that all loss terms are normalized by the cardinality each set, which we omit for simplicity.
3.2 Revisiting the calibration of semi-supervised models
We now introduce a scries of observations revealing several intrinsic propertics of semi-supervised methods built up on pseudo-labels generation, which allows us to motivate a calibration technique tailored for pseudo- label based SST.
Obscrvation 1. Semi-supervised learning degrades the calibration performance. An important body of literature on SSL relies pseudo-labeling to leverage the large amount of unlabeled samples. To achieve this, very common strategy is to generate weak augmentations of each unlabeled image, whose predictions serve as supervision for their strong augmentation counterpart, as presented in eq. (3). Pseudo- labeling (Lee et al., 2013) indeed closely related to entropy regularization (Grandvalet & Bengio, 2004), which favors a low-density separation between classes, a commonly assumed prior for semi-supervised learn- ing. While minimizing the entropy of the predictions can actually improve the discriminative performance of neural networks, it inherently favours overconfident predictions, which one of the main causes of mis- calibration. Figure 1 brings empirical evidence about this observation, where can observe that despite bringing performance gains, in terms of accuracy, pseudo-label based SSL methods naturally degrade the calibration properties of supervised baseline trained with few labeled samples.
Observation 2. Pseudo-labeling in SSL indeed minimizes min-entropy on unlabeled points. The use of hard label makes pscudo-labcling closely related to entropy minimization (Grandvalet & Bengio, 2004). Nevertheless, the different transformations that unlabeled images follow in modern SSL methods produce different probability distributions for the weak and strong versions of the same image, where the
4

---

Published in Transactions on Machine Learning Research (09/2024)
Baseline
FixMatch
bapc.oo A.tea
FlexMatch
FrocMatch
Aitaal
Adtal
Adual
Error: 26.31 Emor 21.09
Emron 18:20
Emar 16.0
Accuracy
PCP: 11.66
ECE: 15.04
ECE: 12
ECE 12 18
82
0.3
5.0
02
04
00
Confidence
Confidence
Confidence
Confidence
Observation 1. Reliability plots for a baseline supervised model (trained with eq. (1)) and three representative SSL approaches (trained with eq. (4)) on CIFAR-100. These plots empirically highlight the calibration degradation observed when training with the standard unsupervised loss, despite the gains achieved in discrimination.
Figure 1:
predictions of the former are used to correct the predictions of the later. Note that this is slightly different from the traditional pscudo-labeling approaches, where the same image used for assigning the pscudo-label and updating the predictive model. Thus, albeit they are related, the minimization of entropy cannot be attributed the cause of miscalibration. Motivated by this, the standard SSL unlabeled loss based on pseudo-labels and its effect on network calibration. The common learning objective is composed of two terms: the first one is the standard cross-entropy (CE) on labeled samples (eq. (1)), while the second term is a CE between pseudo-labels obtained from weak angmentations and the predictions their strong augmented counterparts (eq. (3)):
explore in this section the implications of
Ci =
Σ logs; IEDL
Σ DEDO
y: logs
(4)
CE on labeled samples Pseudo-CE De
where p(y(x) is used for simplicity, and the superscripts w and denote weak and strong transfor- mations, respectively. We now split the unlabeled dataset into Dir, which contains the unlabeled samples whose predicted class from weak and strong augmentations are different, i.e., arg max(s,") A arg max(s;) and Dyr, containing the samples whose predicted class from weak and strong augmentations are the same. Thus, we can decompose the right-hand term in eq. (4) into terms, one acting over Dy and one over Dun:
CT
yogs:
Σ iED
t log
st
Σ iEDon
y log
(5)
iEDL
CE on labeled samples Paeudo-CE on Dyr
the above equation, the second term can be seen as a pseudo cross-entropy on Dyr Furthermore, as y is oqual the onc-hot vector from arg max(s;) on samples from Don, the last term equivalent to the min-entropy+:
Cx
Σ y, logs; iED
Σ iED
y logs
log max s k
(6)
iED
CE on labeled samples Pseudo-CE Dur
minn-cntropy on D
As shown in Figure 2 in the case of two-class distribution (p,1 p). min-entropy is lower bound of the Shannon Entropy, which has several implications in network miscalibration. In particular, while both
Pseudo- 80. s5_ max of and oehrwwise.
5

---

Published in Transactions on Machine Learning Research (09/2024)
Shannon Entropy and min-entropy reach their minimum at the vertices of the simplex, i.., when p = 1 (left), the dynamics of the gradients are different (middle). More concretely, in the case of the Shannon entropy, the gradients of low-confidence predictions at the middle of the simplex are small and, therefore, dominated by the other terms at the beginning of training. In contrast, by minimizing the min- entropy, the inacouracies resulting from uncertain predictions are reinforced, i.e., pushed towards the simplex vertices, yielding early errors in the predictions, which are hardly recoverable, and potentially mislcading the training process.
0.7
0.4
0.9
0.2
0.0
Rafio
FixMatch FlexMatch
0.7
0.2
FreeMatch
0.1
Sharnion Entropy
-0.4
Shannon Entropy
Min-Entropy
Min-E Entroyy
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
40000
00000 120000
10,000
2000000
Iteration
Figure 2: Observation 2. (Left) The unsupervised term in pseudo-label SSL is equivalent to a pscudo cross-entropy on a small subset of unlabeled samples and a pseudo-entropy regularization loss that minimizes the min-entropy (a lower bound of the Shannon Entropy) of the predictions from most unlabeled samples. (Middle) Compared to the Shannon Entropy, the min-entropp more aggressive in the gradient dynamics, particularly the beginning of the training, when most predictions uncertain. (Right) Ratio of samples with same hard prediction for weak and strong augmentations that were above the selection threshold of three relevant SSL methods.
Furthermore, we empirically observed that the amount of samples where the arg max of the predictions from weak and strongly augmented versions was the same, i.e., Di", was significantly larger than those with different predictions, i.c., Dir (Figure 2, right). In particular, as shown in this figure, and after some iterations, more than 80% of unlabeled samples included in the training share the same pseudo- label between weak and strong annotations, regardless of the approach analyzed. Thus, based our observations argue that the training of SSL methods based on pseudo-labels is equivalent to having supervised term coupled with pseudo cross-entropy on small subset of unlabeled samples and pseudo- entropy regularization loss that minimizes the min-entropy of the predictions from most unlabeled samples. This mcans that, while implicitly, or explicitly, minimizing the Shannon entropy on the network softmax predictions known cause miscalibration (Guo al., 2017), employing the min-entropp aggravales the problem, which explains why SSL methods based on pseudo-labels are not well calibrated, particularly compared to simple supervised bascline (Figure 1).
Observation 3. Pseudo-label SSL techniques produce highly overlapped logit distributions, with large logit magnitudes and distances. A direct implication of Observation 2 is that softmax predictions in pseudo-labeling SSL methods become highly confident, which translates into larger logit values compared to a supervised baseline. As a result, even when a predicted category is incorrect, the network will still express a high level of certainty its prediction. Take for instance the example shown in Figure 3, which depicts the logit value distributions for the samples belonging class We can observe that, in the case of the supervised baseline, the maximum logit values for incorrect predicted classes is s 7.5 (classes 6 and 7) and around 6.5 (class 8). In contrast, when adding unlabeled samples in the form of pseudo-labels (c.g., in FixMatch (Sohn et al., 2020)), the maximum logit values for incorrect classes increase to nearly 12.5. This means that, while both models yield incorrect predictions, these are highly confident in the SSL methods. Furthermore, the logit range in the supervised baseline goes from around -7 to 10, whereas FrecMatch produces logits in the range from -10 15, which will result highest probability scores for the predicted category. From these observations, we argue that nell-calibrated SSL model should decrease the magnitude of logits associaled with incorrect classes, well their tolal logits range, thereby decreasing

---

Published in Transactions on Machine Learning Research (09/2024)
confidence in erroneous predictions, while simallaneeussy preserving high values in logits corresponding to the larget class.
Target Class: 5
Baseline
Target Class:
FreeMatch
Density
Density
024
Logit Values
Logit Values
Figure Observation These plots depict the Kernel Density Estimation of the logit distributions obtained by (left) the supervised baseline trained with eq. (1) and (right) FreeMatch on STL-10, for the samples belonging to class 5. We can observe that, even for non-target classes (k/5), the logit magnitudes in FreeMatch are larger, which translates to higher overconfidence in both correct and incorrect predictions. We select STT-10 due to its number of classes (10 us. 100 in CIFAR-100).
4
Our solution
Based findings, especially in Obscrvation 3, it becomes evident that effective strategy for addressing miscalibration within the SSL scenario involves controlling the magnitude of predicted logits for unlabeled samples. Furthermore, from Observation 2 we can derive that the underlying mechanism magnifying the miscalibration issue stems from the hidden min-entropy term on the data points in Dv", which represents the majority unlabeled samples. Thus, we resort to an inequality constraint that imposes controllable margin on the logit distances of predictions in samples from Dyv. This constraint, which draws inspiration from (Liu al., 2022), takes the following form d(I) m, where d(I) = (max, (4,) 1e)ssKKK E R represents the vector of logit distances between the winner class and the rest (only for samples in Dv") and K-dimensional vector defining the margin values, with all elements equal to while (Liu et al., 2022) was proposed in the fully supervised learning seenario, in this work we only enforce the constraint on a subset of samples, which is motivated by the observations presented in 3.2. Furthermore, this choice is also supported empirically in the ablation studies presented in the experiments.
R++. Note that
By integrating this incquality constraint on the logit distances, training becomes a constrained problem, whose objective can formally defined as:
minimize LT
subject to d(1) < m R.R.V E Dun.
The above constrained problem in eq. (7) can approximated by penalty-based optimization method, transforming the formulation into an unconstrained problem by using a simple ReLU function:
min C Σ max max(4:5) 44 mm)
(8)
iEDiN
where the second term, i.c., the non-linear ReLU penalty, prevents logit distances from exceeding prede- termined margin m, and À E R, is a blending hyperparameter which controls the contribution of the CE loss and the corresponding penalty. The intuition behind this penalty term is simple. For the winner logits where the distance with the remaining logits is above the margin m, a gradient will be back-propagatod enforce those values decrease. As result, the whole logit magnitudes will decrease, potentially alleviating the miscalibration issue in the set of unlabeled samples LVn, which dominate the SSL training.

---

Published in Transactions on Machine Learning Research (09/2024)
5 Experiments
Datasets. We resort to the recent Unified Semi-supervised Learning Benchmark for Classification (USB) (Wang et al., 2022), which compiles a diverse and challenging benchmark across several datasets. In partic- ular, we focus on three popular datasets: CI (Krizhevsky & Hinton, 2010), which has significant value as a standard for fino-grained image classification due to its wide range of classes and detailed object distinctions; STL-10 (Coales et al., 2011), which widely recognized for its limited sample size and ex- tensive collection of unlabeled data, rendering it challenging scenario of special significance in the context of SSL; and EuroSAT (Helber et al., 2019), containing 10 unique fine-grained categories related to carth observation and satellite imagery analysis, and important challenges such as high variability and imbalance classes. Last, we also conduct further experiments in the long-taiie taigea version of CIFAR-100.
Architectures. We have prioritized Vision Transformers (ViT) over Convolutional Neural Networks (CNNs), for three main reasons related to discriminative performance, quality of uncertainty estimates, generalization and transfer learning capabilities. First, the emergence of ViTs has proven these models to outperform their CNNs counterparts (Raghu al., 2021; Cai al., 2022). Second, from a calibration standpoint, ViTs have also shown to be better calibrated than CNNs (Minderer et al., 2021; Pinto et al., 2021). Hence, due to their superior discriminative and calibration performance, they pose a more challenging scenario to evaluate the effectivenoss of the proposed strategy. And last, the fine-tuning capabilities of ViTs enable effective transfer learning across diverse visual tasks and datasets. This capability is particularly advantageous in scenarios where labeled data is scarce, as it allows leveraging pre-trained representations learned from large-scale datasets, significantly minimizing the amount of training iterations while maintain- ing consistent performance?. More concretely, we employ a ViTSmall (Gani et al., 2022) with a patch size of 2 and an image size of 32 for CIFAR-100 and EuroSAT, in accordance with the standard in USB, and ViT-Small with image size of 96 for STL10.
We use same settings for all models and benchmarks to provide fair comparison.
Training, evaluation protocol and metrics. While several works use different amounts of labeled data in their experiments, we perform due diligence and follow the settings proposed in USB (Wang et al., 2022) for training. For each method and configuration, we perform three runs with different seeds, select the best checkpoint and report their mean and standard deviation, following the literature. We report error rates for the accuracy performance and the expected calibration error (ECE), following the literature in calibration of supervised models. Implementation details are discussed in Appendix.
5.1 Results
Main results. The proposed strategy is model agnostic, and can be integrated on top of any SSL approach based pseudo-labels, enabling substantial flexibility. For the empirical evaluation, we selected three pop- ular and relevant approaches that resort to hard pseudo-labels: FixMatch (Sohn et al., 2020), FlexMatch (Zhang et al., 2021) and FreeMatch (Wang et al., 2023) and assess the impact of adding our simple solution during training. Discriminative performance (table 1): we observe that in 16 out of the 18 different settings, adding the penalty in eq. (8) brings improvement gains compared to the original versions of cach method, which are only trained with eq. (4). Note that these gains are sometimes substantial, improving the original method by up to 4% (e.g., FixMatch in CIFAR-100(200) and EuroSAT or FlexMatch in EuroSAT(20)). Furthermore, the three approaches combined with our penalty achieve very competitive performance com- pared to existing SSL literature, typically yielding saate-of-thheart results. Calibration performance (table 2): Similarly, including the penalty term systematically enhances the calibration performance of the three ana- lyzed approaches, whose improvements are typically significant (up to 6-7% in several cases).
An interesting observation is that, MixMatch (Berthelot et al., 2019), a consistency regularization based approach, yields surprisingly well calibrated models. Indeed, MixMatch has several components that have shown to improve calibration, such as ensembling predictions and MixUp, which may explain the obtained values. Nevertheless, it is noteworthy to mention that the discriminative performance compared to our
TTraining FreeMatch on CIFAR100 with 400 labeled samples goes from 12 days (WideResNet. from scratch) 10 hours (VITSmall) an NVIDIA V100-32G GPU.
8

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Classification performance (error rate (%)). Arrows indicate whether our modified version improves (4) or deteriorates (t) the performance. Best overall performance in bold and best across pscudo- label SSL approaches underlined.
Dataset
C1FAR-100
EuroSAT
STL-10
# Labeled samples
200
400
20
40
100
Only conaistency regulorisation
MixMaachxcuut1e10
37.6842.66
26.84-1.06
18.77+10.40
14.88+2..0 188-+2.00
25.19 +9.006
11.37+1.49
Daah JCML'21
28.51
19.51=1.20
10.05
6.83+3
18.30
8.74-2..3
AdaMatch TCLR/22
19.26 1..83
17.13
12.01
6.07
13.31 13.75
8.14
DefiixMatch tCLR/23
30.4440.82
20.93
14.27
5.42
25.36
25.36
10.97
Pseudo-labeking
FicMatchhw
31.28+11.8
19.42=1.58
+1.88+6.32
6.64+5.03
16.13+2..0
8.06
FixMatch Ours
27.57
18.08
7.19
83.4
5.021
17.95
1.96
FlesMaaschveuntesn
28.27 1.
17.61
7.89
7.13
13.3441.60
FlexMatch 26.19-0.c24
18. 18.14
3.69-0.814
5.00
12.87 +4.82
7.53+1 3.
FreeMatchier. 23.92
16.18g0.39
4.74
4.48go
14.88 20.73
8.83 8.83go.14
ProcMatch
21.36=1.621
16.09 40.s04
4.30 46.
a..0
13.18
+11.0.0.
modified versions of SSL methods strikingly lower, with differences typically going from 10% to 24%, failing to achieve good compromise between accuracy and calibration.
Table 2: Calibration performance (ECE). Arrows indicate whether our modified version improves (L) or deteriorates (f) the performance. Best overall performance in bold, whereas best across pseudo-label SSL approaches underlined.
Dataset
R-FAR-100
EuroSAT
STI-10
# Labeled samples
Only consistency regularization
MixMatchNeaetPa'1e
8.13
7.19
9.27
3.75
3.42
5.89
Dash 10Mt.21
22.23
13.2001.07
7.09 45
4.24
11.2812.33 22:33
2.23
AdaMatch
1.
11.17
8.55
2.66
8.80
4.96
DeFixMatch ICLR'23
24.54
1..80
10.46+8.71
2.90+1.29
12.89
6.62
labeling
FisMatchxx
27.77
13.45 13.45
8.36
4.72g1.45
10:2712.4
5.83
FixMatch Ours
21.56_1.324 1.34
12.12
4.66 49.
3.84
7.83
2.64
FlexMstccxx..tts
21.95++0.77
11.95=0.30
3.42+2.96
4.50+2.00
9.72+1.68
5.85
FlexMasch + Ours
19.74 o..0
11.61 2o.
2.26 2114
3.21
8.89 23.39
4.97
ProeMatcheuuag
18.27 60
11.56
3.4941.39
3.22
10.49
5.24
FreeMatch + Ours
14.86 1.22
10.35+0.08.
2.82+0.81
2.68 +o.ro
3.74 8.74-
3.50
We would like stress that our goal is not to propose novel state-of-the-art SST approach, but shed light about an important issue in prevalent pseudo-labeling based SSL methods, and present simple yet efficient solution that can improve their performance. These results demonstrate that integrating the penalty in eq. (8) during training appears an appealing strategy to improve both accuracy and calibration performance of pseudo-label SSL approaches.
Lastly, we follow (Wang et al., 2022) and employ the Friedman rank (Friedman, 1937; 1940) to fairly compare the performance of different methods across various settings. This metric can be defined as ranky- = m Em rank,, with m being the number of ovaluation scttings (m 12 in our case, metrics x datasets), and rank; the rank of method in the i-th setting. Hence, the lower the rank obtained, the better the method. These rankings, which are depicted in fig. 4, show that our modified versions pro- vide competitive performances considering accuracy (Error), ECE and both (All), with our FreeMatch and FlexMatch versions presenting the best alternatives across overall methods.
On the impact on the logits. We now analyze in more detail the impact of incorporating the constraint during training, particularly on the logit distribution. In fig. 5, we depict the kernel density estimation of the logits distribution, which was negatively affected by SSL methods compared to a fully-supervised bascline (Observation 3). From these figures,
observe two interesting findings that will have positive effect

---

Published in Transactions on Machine Learning Research (09/2024)
Eror
ECE
4.50
7,00
3,50
8.50
8.50
5.67
6.63
3.83
4.83
1.53
All 7:17 7.00
0.42
2:30
5.17
8.17
4.13
41
60
41
11
Methods
Figure Friedman Rank for the methods analyzed in Tables 1 and 2 following (Wang al., 2022): a) MixMatch, b) Dash, c) AdaMatch, d) DeFixMatch, e) Fixmatch, f) FixMatch + Ours, g) FlexMatch, h) FlexMatch + Ours, FreeMatch and j) FreeMatch + Ours.
in calibration: i) the logit magnitude of the incorrect predicted classes decreased, and ūi the whole logit range also decreases. This means that, even for incorrect predictions, the reduced logit values will lead to less confident predictions, which contrasts with the potentially methods. We argue that the reduction of the logit magnitudes is byproduct of the penalty preventing the logit differences to be large. This limits the effect of the min-entropy pushing hard towards the vertex of the simplex, which is minimized with either 0 or softmax predictions.
confidence scores obtained by SSL
Thus, following our hypothesis from Observation 3, we can advocate that our strategy provides well-calibrated SST. methods, as both the total range and logit magnitude of incorrect predictions are significantly decreased.
Target
Baseline
Target
FreeMatch
Target
Ours
Density
Demsity
Logit Values
Logit Values
Logit Values
Figure Impact of the proposed solution in the logits, which plots the Kernel density estimation of the logits distribution (per-class) for target class for the supervised bascline (left), original FrecMatch (middle) and our version (right).
Comparison to other calibration methods. In this section, we compare our approach to relevant calibration approaches in the fully-supervised scenario. In particular, we evaluate the impact of adding label smoothing (T.S) (Szegedy al., 2016) and focal loss (FT.) (Lin al., 2017) to FreeMatch, as it the most recent studied method. Results from these experiments, depicted in fig, 6, confirm that our approach consistently yields the best Anserininative-calibaalii measured by the largest gap between the accuracy (bright yellow) and ECE (dark yellow). Furthermore, we compare to BAM (Loh et al., 2023), up to our knowledge the only concurrent work that tackles calibration work on SSL. This comparison (table 3) demonstrates that our simple solution outperforms the recent BAM in terms of error rate and ECE, emerging as a promising choice.
trade-off across datasets and settings, which can be
Table Discriminative and calibration performance compared BAM (Loh et al., 2023).
Method
CIFAR100 (200)
CIFAR100 (400)
Error
ECE
Error
ECE
FixMatch BAM
28. 46.1.
25. 25.0141143
19.32+0.04
16.75=0.98
FixMatch Ours
27.5721.49
21.56 21.56_1..2
18.42 21.05
12.1211.70
10

---

Published in Transactions on Machine Learning Research (09/2024)
CIFAR-100 (200)
CIFAR-100 (400)
EUROSAT (20)
EUROSAT (40)
STL-10 (40)
(100)
Curo
Ape
ADc
ASc
Aoe
Apc
Ace
Figure 6: Radar plots for different calibration approaches based on FreeMatch. The range of the radar plot for ECE is changed for better visualization (CIFAR-100: [1, 50), EuroSAT: [1, 20), STL-10: [1, 20))
In which samples to impose the penalty? A natural question that arises from our analysis whether constraining the logit distances should also be applied to the samples in Der. We argue that, indeed, it is only beneficial to the samples in Dz... While we formally exposed that the third term in eq. (8) minimizes the min- entropy, the second term has some sort of corrective effect, where pseudo-labels from weak augmentations correct the predictions obtained with the strong augmented versions. As labels are different, enforcing a penalty on the logit distances might indeed have a counterproductive effect. More concretely, trying to satisfy the constraint in samples from Der; where hard predictions are from different. classes, may actually impede the network learn semantically meaningful features that can bring discriminative capabilities the model. To support these arguments empirically, report in table 4 the performance of the three SSL methods FixMatch, FlexMatch and FreeMatch when the constraint is enforced across different scenarios. We can clearly see that, regardless of the configuration or model used, including the penalty on the samples from Dv has a detrimental effect, with discriminative and calibration results suffering a substantial degradation.
Table 4: Impact of the penalty term in the different unlabeled subsets, Due and Du for the three relevant psoudo-labeling SSL methods studied in this work. The proposed approach is shadowed in gray and best result in bold.
CIFAR- 100
(200)
CIFAR-100 (400)
Dyr
Den
Error
ECE
Error
ECE
27.77+1.40
31.28+1.88
15.76
19.73
FixMatch
/
21.56+ 1.92
27.57+1140
13. 13.49-0.s0
18.57
/
19.30=2.22
25.72
13.00
19. 19.2021..3
21.95+0.67
28.27
11.95-0.50
17.61+0.s1
FlexMatch
/
19.74.0.50
26.49
11.61
18.15
20.98g3.05
27.5720.37
13.4 20.01
19.62
23.92
18.27
16.18=0.58
11.56-go.53
FreeMatch
/
21.36+1.62
14.86+1.44
16.09 +0.80
10.35+0.83
27.44:
20.86g1.48
18.82=0.67
12.39.40.7
Can we add our calibration strategy to other methods?. We have evaluated the impact of incorpo- rating the proposed term into relevant SSL approaches that resort to hard pseudo-labels during training. In- deed, our motivation stems from the observations that under this scenario (i.e., the standard pseudo-labeling process in SSL), there is a hidden min-entropy objective that dominates the training. While we have demon- strated that our approach effectively improves both the discrininative and calibration performance of these models, we further assess its impact on approaches that use soft pseudo-labels, i.e., pseudo-labels on weak augmentations are not converted to onc-hot encoded vectors. For these approaches, the reasoning in eq. (5) and eq. (6) does not entirely hold in these cases. More concretely, the recent work SimMatch (Zheng et al., 2022) uses this strategy3, where the direct softmax prediction over the weak augmented samples used as supervisory signal for its strong augmented counterpart. On the other hand, for SoftMatch (Chen et al,, 2023), the weighted thresholding mask prevents overly confident predictions, distinguishing these methods from those relying entirely hard pseudo-labels. The results from this experiment are reported in Table 5, which show that adding the proposed penalty has positive effect favouring better calibrated models even
Note that rreudo-labeling SSL approaches use hard pseudo-label strategy, where the softmax predictions of weak angmentations are transformed into one-hot encoded labels.
11

---

Published in Transactions on Machine Learning Research (09/2024)
in methods that use soft pseudo-labels, particularly as the number of labeled samples increases. Note that, our main findings, observations, and motivations do not entirely apply for soft pseudo-labels approaches, such as SimMatch and Motthatch, we do not make any claim regarding the benefits of our strategy in these approaches. In contrast, we simply show empirically that even in these cases, our method can still bring performance benefits.
Table 5: Comparison to related approaches, SimMatch and Match, which use soft pseudo-labels, instead of hard pseudo-labels (as FixMatch, FlexMatch and FreeMatch.).
Error rate (%)
ECE
CIFAR-100
# Labeled samples
200
200
1000
SimMatch
22.49
19.45
15.46
3.17=0.28
6.54
9.60
SimMatch Ours
23.00+0.orf
19.10+0.51
15.36+0.41.
30.1=0.80"
5.55++.154
9.18-p.174
Match
22.69-1.74
16.4920.83
14.49
17.6441.27
12.09
10.63
SoftMatch Ours
21.72+0.314
16.82
14.53+0.361
13.80+0.224
11.70-0.s04
10.80+0.40.
Long-tailed experiments. table we report error rale and ECE on CIFARI00-LT asses the effect of the proposed calibration when the class population imbalanced. Across all cases both the method accuracy and the ECE improve consistently. It is noteworthy to mention that for all these experiments we use the same margin as in CIFAR-100, without any adaptation to the specific setting.
Table 6: Qualitative performance on long-tailed classification (CIFAR-100-LTT)
Method
10, Ts
-10
10, 7v
10
= 15, 7u.
15
Error
BCE
Error
ECE
Error
ECE
FixMatch
15.05
8.58+0.e2
15.01
10.51
16.454o.11
10.22
FixMatch + Ours
14.73+0.12
7.22+1.20
14.37+0.21
9.42-0.29
15.81
8.44+00.4
FlexMatch
14.79.0.28
7.70+1.03
14.98
9.4920.47
16.1940.3
9.25
FlexMatch Ours
20.23
7.33+1.49
14.649.05
7.44-0.40
16.08-0.44 0.49
8.85
FreeMatch
14.93
8.32 95
15.04_0.00
9.99
16.26 as
8.68
FreeMatch Ours
14.67+0.28
6.74+0.98
14.77+0.18
9.05-0.54
15.87 -0.69
8,53 +0.00
Additional experiments with detailed analysis across methods, comparison to additional approaches and additional settings are reported Appendix.
Scalability and Computational Costs. The proposed approach boils down to simple penalty term, which can be straightforwardly integrated existing methods by modifying the loss function, whose compu- tational cost is negligible. Indeed, our learning approach does not involve architectural changes, avoiding any increase in model complexity. Thus, the strategy proposed in this work further improves the discriminative and calibration performance of existing SSL methods without incurring additional overheads.
6 Conclusion
In this work have raised awareness of the miscalibration problem induced by pseudo-label training, which is one of the most popular approaches for SSL. We demonstrated that the unsupervised loss used by pscudo- label methods is dominated by the min-entropy term, lower bound of the Shannon entropy, and identified it as a potential source of miscalibration. We then proposed a simple solution based on enforcing a fixed margin constraint between the winner class and its contenders. Our solution on popular SSL datascts and methods yields consistent calibration improvements, whereas we also found consistent gains in terms of predictive accuracy, typically outperforming the state-of-the-art for SSL.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Acknowledgments
This work was funded by the Natural Sciences and Engineering Research Council of Canada (NSERC). We also thank Calcul Quebec and Compute Canada.
References
Philip Bachman, Ouais Alsharif, and Doina Procup. Learning with pscudo-ensembles. Advances in neural information processing systems, 27, 2014.
David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to sumi-super learning. Advances in Neural Information Processing Systems, 32, 2019.
Zhaowei Cai, Avinash Ravichandran, Paolo Favaro, Manchen Wang, Davide Modolo, Rahul Bhotika, Zhuowen Tu, and Stefano Soatto. Semi-supervised vision transformers at scale. In Alice H. Oh, Alekh Agarwal, Daniclle Belgrave, and Kyunghyun Cho (cds.), Advances in Neural Information Processing Sys- lems, 2022.
Olivier Chapolle and Alexander Zien. Semi-supervised classification by low density separation. In Interna- tional workshop on artificial intelligence and statistics, pp. 57-64 PMLR, 2005.
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien (eds.). Semi-Superoised Learning. The MIT Press, 2006.
Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learning. In The Eleventh International Conference on Learning Representations, 2023.
Jiacheng Cheng and Nuno Vasconcelos. Calibrating deep neural networks by pairwise constraints. In Pro- ceedings of the LEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13709 13718, 2022.
Adam Coates, Andrew Ng, and Honglak Lec. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-22222 215-223. JMLR Workshop and Conference Proceedings, 2011.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IRFE/CVF Conference on Compuler Vision and Pattern Recognition Workshops, pp. 702 703, 2020.
Zhipeng Ding, Xu Han, Peirong Liu, and Mare Niethammer. Local temperature scaling for probability calibration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6889- 6899, 2021.
Milton Friedman. The of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the american statistical association, 32(200):675 701, 1937.
Milton Friedman. comparison of alternative tests of significance for the problem of rankings. The annals of mathematical statistics, 11(1):86-92 1940.
Hanan Gani, Muzammal Naseer, and Mohammad Yaqub. How train vision transformer on small-scale datasets? In 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022. BMVA Press, 2022.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. TEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7))2217 2226, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. report, Universily of Toromlo, 2010.
Technical
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations, 2017.
Agostina J Larrazabal, César Martínez, Jose Dolz, and Enzo Ferrante. Maximum entropy on erroncous predictions: Improving model calibration for medical image segmentation. In International Conference on Medical Image Computing and Computer-Assise Assisted Intervention, pp. 273-283. Springer, 2023.
Dong- Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on chollenges in representation learning, ICML, volume 3, pp. 896. Atlanta, 2013.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988 2017.
Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the margin: Margin-based label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 80 80-88,77 2022.
Bingyuan Liu, Jérôme Rony, Adrian Galdran, Jose Dolz, and Ismail Ben Ayed. Class adaptive network calibration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 16070- 16079, 2023.
Charlotte Loh, Rumen Dangovski, Shivehander Sudalairaj, Seungwook Han, Ligong Han, Leonid Karlinsky, Marin Soljacic, and Akash Srivastava. Mitigating confirmation bias in semi-supervised learning via efficient bayesian model averaging. Transactions on Machine Learning Research, 2023.
Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Wei- jer. Ilase-incrementll learning: survey and performance evaluation image classification. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, (5(5):5513-5533 2022.
Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(7):3523 3542, 2021.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Infor- mation Processing Systems, 34:15682 15694, 2021.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regular- ization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979 1993, 2018
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 1115588815299 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Balamurali Murugesan, Sukesh Adiga Vasudeva, Bingyuan Liu, Herve Lombaert, Ismail Ben Ayed, and Jose Dolz. Trust your neighbours: Penalty-based constraints for model calibration. In International Conference on Medical Image Computing and Consuuer-Assiseed
Assisted Intervention, pp. 572 581. Springer, 2023.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Jongyoun Noh, Hyekang Park, Junghyup Lee, and Bumsub Ham. Rankmixup: Ranking- based mixup training for network calibration. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1358 1368, 2023.
Hyekang Park, Jongyoun Noh, Youngmin Oh, Donghyeon Baek, and Bumsub Ham. ACLS: Adaptive and conditional label smoothing for network calibration. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 3936-33 3945, 2023.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. In International Conference on Learning Repre- sentations (ICLR), 2017.
Francesco Pinto, Philip Torr, and Puncot Dokania. Are vision transformers always more robust than convolutional neural networks? In NeurIPS 2021 Workshop Distribulion Shifls: Connecting Methods and Applications, 2021.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning, 2021. URL https:
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Advonces in newral information processing systems, 29, 2016.
Weiwei Shi, Yihong Gong, Chris Ding, Zhiheng MaXiaoyu Tao, and Nanning Zheng. Transduetive semi- supervised deep learning using min-max features. In Proceedings of the European Conference Computer Vision (ECCV), pp. 299-23 315, 2018.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with con- sistency and confidence. Advances in Neural Information Processing Systems, 33, 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on compuler vision and pattern recognition, pp. 2818 2826, 2016.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Christian Tomani, Sebastian Gruber, Muhammed Ebrar Erdem, Daniel Cremers, and Florian Bueliner. Post-hoc uncertainty calibration for domain drift scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124-10132, 2021.
Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjic Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, et al. USB: A unified semi-supervised learning benchmark for classífication. Advances in Neural Information Processing Systems, 35:3938 0000338-36,,,, 2022.
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. Freematch: Self-adaptive thresholding for semi-supervised learning. In The Eleventh International Conference on Learning Representations, 2023.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33, 2020a.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10687-10698, 2020b.
Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Ti, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi- supervised learning with dynamic thresholding. International Conference on Machine Learning, pp. 11525-11536. PMLR, 2021.
Lihe Yang, Zhen Zhao, Lei Qi, Yu Qiao, Yinghuan Shi, and Hengshuang Zhao. Shrinking class space for enhanced certainty in semi-supervised learning. In Proceedings of the IEEE/CVE /CVF International Conference on Compuler Vision, pp. 16187-16196, 2023.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Aduances in Newral Information Processing Systems, 34, 2021.
Mingkai Zheng, Shan You, Lang Huang, Fei Wang, Chen Qian, and Chang Xu. Simmatch: Semi-supervised learning with similarity matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 14471 14481, 2022.
A Additional discriminative results
Table 7: Comparison of error rate (%) for different SSL methods across various labeled settings and datasets (CIFAR-100, EuroSAT, and STI-110-f indicates that the results are reported from (Wang et al., 2022). Best results bold, whereas second-best are underlined.
Dataset
CIFAR-100
EuroSAT
STL-10
Labeled samples
100
r-Model Ne |
36.06+0.15
26.32+0.41
21.82+ 1524-1222
12,0942.27
42.76+16.04
19.85+13.02
NewdP3117
35.47
26.0340.31
26.83
15.85
18.6712..6
24.19410.15
VAT TFAMTIS f
31.49+1.55
21.3444..00
18.45+1.47
26.1640.90
10.09+0.94
10.6940.51
MixMatch Ne
37.68
26.8441.06
28.77
14.88
25.1942.0s
11.37:1.49
ReMixMatch 1c1a020f
22.21 2
16.86
5.05.11.05
5.0720.58
13.0813.34
7.21 7:21
UDA N0
28.80
19.0010.79
9.83
6.22
15.58
7.65
CRMatch CCPR/21
25.70
18.03+0.20
13.24411.0
8.35
10.17
CoMatch 1ccv'21
35.08+0..69
25.35-+0.50
5.75++0.48
4.81+1.05
15.12+
9.56+1.35
Dash 1CML'21
28.51
19.5411.20
10.05:18.15
6.83
18.30
8.74
AdaMatch 1CLR22
19.26
17.1340.92
12.01+4.116
6.07
13.31+
8.14+1.45
SimMatch cVPa/22
23.78
17.0640..7
7.66-20.00
5.27
11.77
7.55
SoftMatch 1CUR33
22.67
16.8410..d
5.75
5.90
13.55
7.84
DePixMatch ICLR'23
30.44
20.9341.42
14.2729.05
5.42=2.69
25.36
10.9741.75
FixMatch 2800
11.28
19.73
11.88
6.64
16.13
8.06
FixMatch Ours
27.57
18.5740.17
7.19 24.88
5.02
17.55
7.96
FlexMatch 21
28.27+0.60
17.6140.61
7.89
7.13+1.29
13.34+1.69
8.35+1.24
26.49:2.622
18.1540.47
3.69 81
5.00.20.95
12.87+44.92
7.53+1.32
FreeMatch ICLR/2
23.9242.02
16.18+0.80
4.74+1.77
4.48
14.88+0.72
8.83
FreeMatch + Ours
1..30+1.62
16.09 40..0
1.30+1.48
3.50==0.70
13.18+1.4
8.57+1.05
In the main paper, we included several relevant SSL methods compare both error and calibration perfor- mance. We did not include considerably large set of approaches because need to run three times each method in each setting (so that each evaluated method means 3x6=18 runs, with average of 10 hours per run). Furthermore, we would like to stress again that our goal in this work is not to provide novel state-of-the-art SSL method, but investigale the miscalibration issue of pseudo-labeling SSL, identify the source of the problem, and provide a solution that can enhance the calibration this family of methods. Having said this, the discriminative performance of additional methods is provided in (Wang al., 2022), which is used to complement the discriminative comparison in Table whose results can be found in Table of this appendix.
16

---

Published in Transactions on Machine Learning Research (09/2024)
B
Dataset details
We provide detailed insights into the datasets utilized in our experiments, whose configuration is strongly inspired by the recent USB (A Unified Semi-supervised Learning Benchmark for Classification) benchmark (Wang et al., 2022). For CIFAR-100, renowned benchmark for fine-grained image classification, we consid- ered two label settings: 2 labeled samples and 4 labeled samples per class for each of the 100 classes, in a total of 50,000 training samples and 10,000 samples for testing. Each image in CIFAR- 100 is sized at 32x32 pixels. STI-10,, known for its limited sample size and extensive unlabeled data, offers a unique challenge. We employed two label settings as well: 4 labeled samples and 10 labeled samples per class for all 10 classes, and an additional 100,000 unlabeled samples for training, along with 8,000 samples for test- ing. Each STL-10 image measures 96 pixels. Lastly, EuroSAT, based on Sentinel-2 satellite images, features two label settings: 2 labeled samples per class and 4 samples per class for 10 classes. With total of 16,200 training samples, including labeled and unlabeled images and 5,400 lesting samples, EuroSAT images are sized at 64x64. While literature includes SVHN and CIFAR-10 10 datasets, we do not use them for evaluation, as state-of-the-art SSL mothods alrcady achieve a performance comparable to that of fully supervised training on these datasets. The selected datasets offer diverse challenges and settings, allowing for comprehensive evaluation of semi-supervised learning methods across various domains.
For the long-tailed CIFAR-100 experiments, we investigate two subsets of the dataset, with N1 represent- ing the number of samples in the minority class and M denoting the number of samples in the majority class. The imbalance ratio is controlled by "It and To: where both parameters are set to -10, 10 or 15, ensuring a consistent level of label imbalance across categories. In the first setting, we set the imbalance ratio of labeled samples tV to 10 and the imbalance ratio of unlabeled samples 7h to 10, with the number of labeled samples (N1) set to 150 and the number of unlabeled samples (M) set to 300, In the second setting, we maintain the same label imbalance ratio (n = 10) and number of labeled samples (N1 150), but we set the unlabeled sample imbalance ratio 7/ to -10. Finally, in the third setting, we increase the a to 15 while keeping the Yuu at 15, with N1 and M set to 150 and 300, respectively. This configuration introduces higher degree of label imbalance, with the minority class having 15 times fewer samples than the majority class. We follow the same setup as authors in (Wang et al., 2022) for the classic setting using WideResnet. These scttings allow us to comprehensively evaluate the performance of machine learning models under varying degrees of label imbalance and class distribution scenarios within the CIFAR100-LT dataset.
C Implementation details
Method-dependentt hyperparemeters. In the context of pseudo-labeling methods, while FixMatch and Flex- Match rely on a single threshold hyperparametor for pscudo-label sclection, FrecMatch introduces additional hyperparameters such as a pre-defined threshold т, unlabeled batch ratio μ4, unsupervised loss weight war fairness loss weight wy, and EMA decay X. To avoid unfair comparisons across methods, we use the default values for all these hyperparaameters, which are reported in their respective papers.
Specific-case: BAM. In our comparison with BAM (Loh et al., 2023), encountered difficulties in hyper- parameter tuning. BAM employs the quantile Q over the batch to determine the threshold for pseudo-label selection, which serves as a key hyperparameter. In the cited paper, Q = 0.75 for the CIFAR-100 benchmark and Q = 0.95 for the CIFAR-10 benchmark were chosen. Additionally, BAM incorporates a separate Adam optimizer for the Bayesian Neural Network (BNN) layer, with a fixed learning rate of 0.01. Notably, for the sharpening temperature paramcter in BaM-UDA, BAM opts for = 0.9 as opposed to t = 0.4 utilized in UDA. These hyperparameter selections are specific to both the dalaset and the method employed, thus presenting a challenge when evaluating the method across different datasets. For our experiments, we chose to focus on CIFAR-100 due to the inherent complexity involved in conducting extensive hyperparameter searches, not only concerning the dataset or method, but also regarding the optimizer and learning rate for the introduced BNN layer.
Margin Selection with Limited Fine-tuning: We stress that the majority of cases, we did not extensively fino-tune the margin hyperparameter. Across various experiments, settings and datasets, the chosen margin values remained consistent without significant adjustments. Specifically, for the CIFAR-100 and EuroSAT
17

---

Published in Transactions on Machine Learning Research (09/2024)
datasets, the margins were uniformly set without requiring further refinement, with values of 10 and 8 across methods, respectively. Given its complexity, we needed to perform a hyperparameter search in a few cases in the STL-10 dataset. In particular, most settings employed a margin equal to 10, similar to the CIFAR- 100 dataset. Nevertheless, we observe that, particularly for FrecMatch, this margin did not yield the best performance across the Iwo settings of labeled data (i.e., 40 and 100 labeled samples). After fine-Luning the margin value, we found that for STI-10 (40) and STI-10 (100) FreeMatch worked best with a margin of and 6, respectively. Furthermore, for all the remaining experiments with CIFAR-100, as well as with long-tailed CIFAR-100, we kept the margin fixed (m 10) across experiments and methods.
Training hyperparemeters. Regarding algorithn-indeeendeer hyperparameters, we adhered to the sottings outlined in (Wang et al., 2022). Specifically, the learning rate was set to 5 x 10-4 for CIFAR-100, 10-4 for STL-10, and 5 x 10-5 for EuroSAT. During training, the batch size was fixed at 8, while for evaluation, it was set to 16. Additionally, the layer decay rate varied across datasets: 0.5 for CIFAR-100, 0.95 for STI,-10, and 1.0 for EuroSAT. Weak augmentation techniques employed included random crop and random horizontal flip, while strong augmentation utilized Augment (Cubuk et al., 2020). The cosine annealing scheduler was utilized with total of 204,800 steps and warm-up period of 5,120 steps. Both labeled and unlabeled batch sizes were set to 16. Furthermore, we follow (Wang et al., 2022; Zhang et al., 2021) to report the best mumber of all checkpoints to avoid unfair comparisons caused by different convergence speeds.
D Model choices
In this work we have selected three relevant SSL approaches based on hard pscudo-labels, as our findings are closely related to these approaches. More concretely, we demonstrate empirically our observations in FixMatch (Sohn et al., 2020), FlexMatch (Zhang et al., 2021) and FreeMatch (Wang et al., 2023), they are popular SSL methods, some of them published recently (i.e., FreeMatch has been published in 2023). Thus, for the Table 1 and Table 2, we have shown the effect of adding the proposed term in all the three approaches. Similarly, the impact of our lerm over the three methods is evaluated in Table (i.e., CIF-R-100-LT), as we consider that long-tailed classification represents an important task, and assessing the performance of the three approaches will undoubtedly strengthen the message that our solution can improve both classification and calibration performance across general family of SSL approaches. Regarding the ablation studies to evaluate the impact of several choices, we have used FreeMatch, the most recent approach among the three, in all the ablations (Fig 6 and Table 4), except when comparing to the concurrent work in BAM (Loh et al., 2023). The reason is that authors in (Loh et al., 2023) only consider FixMatch in their experiments, despite BAM being a work published in 2023. Thus, the code is all constrained to the use of FixMatch and CIFAR-100, making it difficult to evaluate with other SSL approaches and datasets. In fact, we attempted to integrate FreeMatch on the BAM framework, but the results obtained were suboptimal. We believe that, the sensitivity to hyperparameters (e.g., just the choice of the backbone for the Bayesian layer in BAM requires different learning rates) could be the main cause for the low performance of BAM in SSL approaches other than FixMatch. Therefore, in order to avoid an unfair comparison (due to the suboptimal performance of BAM in other SSL methods) we decided to compare to it based only on FixMatch, following their original work.
E Numerical values for results in radar plots
For the radar plots shown in the main paper, we present here (Table 8) the individual quantitative results, for more detailed comparison across methods. As a reminder, FreeMatch (Wang et al., 2023) is used as baseline strategy, and the different calibration strategies (i.e., Focal Loss (Lin et al., 2017), Label Smoothing (Szegedy et al., 2016), and ours) are applied over the Di subset.
httpps//guhhubb.cm///dtt///////
18

---

Published in Transactions on Machine Learning Research (09/2024)
Table 8: Numerical values of radar plots depicted in Figure 6 the main paper.
Method
Setting
Error
ECE
CECE
AECE
Baseline
23.92:2.02
18.27 99
0.11g0.04
18 18.97
Focal Loss
28. 055.11.0.70
24.4220.8
0.53
24. 14.42210
CIFAR 100 (200)
Label Smoothing
29.1 29.14=22.45 2:44.6
23.55+2.88
0.5220.04
23. 23.54422.44
Ours
21.36++1.02
14.86+1.48
0.35++0.8
14.81+1.48
Baseline
16.18
11.5610
0.27=2.00
11.5440.05
Focal Loss
CIFAR 100 (400)
20.35g0.33
15.6319.3
0.35
15.66 15.60
Label Smoothing
19.01.11.00
13.03 8.
0.31 0z
13.02
Ours
16.09
10.35
0.26
10.33
Bascline
4.74
3.50
0.82
3.38
Focal Loss
EuroSAT (20)
10.25
8.54-8.29
1.88:21.70
8.52+8.31
Label Smoothing
6.94
3.57 -2.87
0.95=
3.45+2.811
Ours
1.
2.8211100
0.70
2.78
Baseline
4.48-+0.12
3.22-0.66
0.74+0.11
3.15+-0.01
Focal Loss
EaroSAT AT (40)
4.37
3.10=0.10
0.74
3.01 +:0.08
Label Smoothing
15.8443.57
4.43-a3.5
1.02+0.70
4.37+3..3
Ours
3.50
2.63 40.3...0
0.60.2.12
2.58
Baseline
14.88+0.72
10.49+
2.45+0.40
9.40+
Focal Loss
14.73
11.16 1s
2.39=0.61
11.15
STL 10 (10)
Label Smoothing
15.5912.
10.09
2.20
10.0441.00
Ours
13.18
3.7415.64
2.3920.08
8.43_4.90 4.00
Baseline
7.6211.25
5.2421.08
1.1520
5.1911.00
Focal Loss
STL 10 (100)
8.70.1.00
6.07=11.0
1.35g0.31
6.02
Label Smoothing
8.25 2..05
4.9421.79
1.1320.35
1.
Ours
8.57
3.50
0.94
4.00 4.00_1.21
F
Comparing our method to temperature scaling
Applying temperature scaling (P& exp k/r
Σe= exp/ can indeed modify the shape of the softmax distribution,
leading to smoother distributions when T > 1. Nevertheless, compared to the proposed solution, scaling the softmax values during training (before using the softmax probabilities into the cross-entropy loss) presents several drawbacks. 1) No mechanism to control the importance of softened distributions: System- atically scaling the softmax predictions might be suboptimal in several cases: e.g., reduction of the gradient magnitude, loss of sparsity in the predictions, impaired learning of hard examples, even magnifying the miscalibration issuc, strong softmax smoothing applying TS on the softmax values will always introduce a risk of these drawbacks to appear, the use of the proposed constrained term acts as a regularizer of the main objective, whose importance can be con- trolled by a blending hyperparameter X, which introduces more fiexibility during training. 2) Difficulty on finding the optimal scaling factor. Softmax distributions at the beginning of the training (mostly uniform) are very different from those observed typically when the network converges (peaky distributions with one category dominating the softmax vector). Thus, the scaling factor control the logit distance should also evolve during training (c.g., at the beginning, as this could magnily all the problems presented in previous point). For instance, scaling value of 2 does not result to the same logit distances (or their equivalent softmax predictions) on an almost uniform than on a peaky distribution. On the other hand, the proposed penalty acts on the absolute logit values, enfore- ing the same constraint across the training (i.e., just penalize those predictions whose logit distances are longer than the given margin), which casier to adjust. 3) Different behaviour. Based on the previous point, at the beginning of the training, when the logit distribution is more uniform, the penalty term will not apply to most predictions, allowing the cross-entropy, pseudo cross-entropy and min-entropy terms to guide the training. However, it will start to increasingly act regularizer as logit distributions become sharper, controlling the logits magnitudes from becoming larger and larger, but allowing their distances to be large enough lo have discriminative power. On the oer hand, scaling the softmax from the beginning will reduce the softmax values of all the samples systematically, which may impede the network to properly learn. Thus, we can argue that the proposed constrained term offers more control over the bogit/softma distributions than simply scaling the softmax predictions, more flexibility as it offers a mechanism to control the importance of the constraint, and different behaviour, particularly at the beginning of the training.
lead to meaningless uniform distributions. While
might not want to scale the softmax at all,
19

---

Published in Transactions on Machine Learning Research (09/2024)
To quantify the discussion, present our results in the Table 9. We also illustrate the logit distribution of the three methods under consideration (a) FreeMatch, (b) FreeMatch + Ours and (c) FreeMatch with logits scaled by temperature of 2. In these plots, it can be observed that the impact on the logit distribution of our method, and systematically applying temperature scaling to all samples, different, which empirically supports our hypothesis regarding their different behaviour.
Table Comparison of various metrics across different numbers labeled samples.
# Labeled Samples
Temperature
ECE
Error
(T)
2
26.63 26.6319.98
29.53
26.5849.31
27.93 46
200
10
27.46+0.07
28.07
20
27.55+0.92
27.98+1.03
2
1..49
20.43 64
5
18.32+0.90
19.69+1.11
400
10
18.85
19.48
20
18.7819.52
19.10g0 51
Nomalized Deraty.
FreeMatch
Correct Prodictions
Normalized Denaty
FreeMatch Ours
Correct Prodictions Incomrect Predictions
Nommllized sity
Temperature
Correct Predictions
Incorrect Predictians
Incorrect Predictions
Logit Values
Logit
Logit Values
Figure 7: Logit Distribution Plot depicting the distribution of correct predictions incorrect predictions for (a) FreeMatch (b) FreeMatch + Ours and (c) FreeMatch with logits scaled by temperature of 2 on the CIFAR-100 dataset with 200 labeled samples.
G Evaluating our method in classic setting
As stated in the main paper, we advocate for the use of Visual transformers over standard convolutional neural networks due to their multiple advantages. Nevertheless, to show that approach is also applicable to convolutional deep models, we evaluate our method against the baseline FreeMatch as well as UPS (Rizve et al., 2021) with WideResNet-28 as the backbone architecture. The results obtained from comparing our method are reported in the Table 10 below. While UPS was initially evaluated on higher amounts of labeled data, semi-supervised learning has evolved to use much lesser data in recent settings, which echoes more realistic scenarios. Thus, we have evaluated UPS under these newer settings, alongside Hatte-oftthe-art methods. particular, for sccnario where 4000 labeled samples are available, and UPS achioves an error rale of 40.77 (See Table 1 in (Rizve al., 2021)). Thus, it is reasonable to think that its performance is degraded the amount of labeled samples decreases. Having validated the performance of UPS, we casily observe that our method, as well as the bascline FroeMatch, significantly outperform UPS in terms of both error rate, and Expected Calibration Error (ECE).
Table 10: Results on FreeMatch with WideResnet 28 as backbone on CIFAR-100 datasct.
Error rate (%)
ECE
CIPAR- 100
CIFAR- 100
# Labeled samples
400
2500
400
2500
UPS
82.67
48.85
70.43
38.35
FreeMatch
50.17
33.35
36.05
20.63
PreeMatch Ours
50.45
33.88
27.57
16.13
20

---

Published in Transactions on Machine Learning Research (09/2024)
H Detailed comparison between original and our versions
We include additional comparisons between our proposed method and the original SSL approaches.
Convergence analysis. First, as depicted in Fig. 8, we assess the impact of adding our approach to FreeMatch in terms of convergence. Notably, our method, denoted as FreeMatch (Ours), demonstrates superior per- formance terms of accuracy, outperforming the bascline FreeMatch method within the initial 50, 000 iterations. In contrast, the original method reaches convergence around 170, 000 iterations, making it con- siderably slower if one looks at discriminative performance. Additionally, our approach consistently achieves lower values of Expected Calibration Error (ECE) throughout the iterative process, highlighting its potential to enhance the calibration performance. This contrasts with the original FreeMatch, whose predictions seem to be better calibrated at the first iteration, and these are degraded (i.e., ECE increases) with the training. Thus, adding our method exhibits accelcrated convergence rates compared to the original approach, reaching competitive performance levels a fraction of the time.
0.18
Freematch Freematch Ours
0.84
0.16
0.82
Accuracy
0.14
0.80
MW
0.12
0.78
0.10
0.74
Freematch
0.06
Mmm
0.72
Freematch + Ours
0.04
0.70
50000
100000
150000
200000
100000
150000 200000
Iterations
Iterations
Figure 8: Comparison of convergence of Accuracy (left) and ECE (right) over iterations for STL10 dataset with 40 labeled samples.
Logils distribution. We now provide additional insights into the effects of our approach across the logit distribution. More concretely, in fig. 9 and fig. 10 we present the kernel density estimation of the logits distribution for each class the STL10 dataset for FreeMatch and FreeMatch plus our method (denoted Ours). This analysis expands Observation 3 from the main paper, highlighting the impact of SSL on the calibration of the model across diverse range of classes. Specifically, while the original FreeMatch provides larger logit ranges, as well as larger logit values even for incorrect classes, adding our approach limits these logit increases across all the classes. As the logit range decreases, the resulting softmax probabilities for the predicted classes will be smaller, alleviating the problem of overconfident predictions. This particularly important for wrong predictions, we expect well-calibrated model to not be overconfident when incorrect classes are predicted.
Reliabilily plots. Last, depict several reliability plots for both the semi-supervised learning (SSL.) hods that were discussed in the main paper and our modified version, to highlight the calibration improvements brought by our approach (fig. 11 and fig. 12).
21

---

Published in Transactions on Machine Learning Research (09/2024)
Target Class: 0
FreeMatch
Target Class:
Ours
Density
Density
4.0
Logit Values
Logit Values
Target Class: 1
FreeMatch
Target Class:
Ours
Density
Density
Logit Values
Logit Values
Target Class:
FreeMatch
Target Class:
Ours
Density
Density
6:0
Logit Values
Logit Values
Target Class: 3
FreeMatch
Target Class:
Ours
Density
Density
0.M
6:0
Logit Values
Logit Values
Target Class:
FreeMatch
Target Class:
Ours
Density
Density
650
Logit Values
Logit Values
Figure 9: Comparison of convergence of Accuracy (left) and ECE (right) over iterations for STL.10 dataset
with 40 labeled samples.
22

---

Published in Transactions on Machine Learning Research (09/2024)
Target Class:
FreeMatch
Target Class:
Ours
Density
Density
440
Logit Values
Logit Values
Target Class: 6
FreeMatch
Target Class:
Ours
Density
Density
0.M
144
Logit Values
Logit Values
Target Class:
FreeMatch
Target Class:
Ours
Density
Density
0.M
Logit Values
Logit Values
Target Class: 8
FreeMatch
Target Class:
Ours
Density
Density
k.0
Loqit Values
Logit Values
Target Class:
FreeMatch
Target Class:
Ours
Density
Density
Logit Values
Logit Values
Figure 10: Comparison of convergence of Accuracy (left) and RCE (right) over iterations for STL10 dataset
with 40 labeled samples.
23

---

Published in Transactions on Machine Learning Research (09/2024)
FixMatch
FlexMatch
Expected Actual
FreeMatch
Actual
Actual
0.8
Error: 28.68
Error: 28.92
Error: 23.00
Accuracy
ECE: 22.37
Accuracy
0.6
ECE: 22.72
Accuracy
ECE: 17.74
0.0
0.2
0.0
0.6
0.8
1.0
Confidence
Confidence
Confidence
Expected
FixMatch
Expected
FlexMatch
Expected
FreeMatch
Actual
0.6
0.8
0.8
Error: 27.93
Error: 27.10
Error: 17.59
Accuracy
0.6
ECE: 19.87
Accuracy
0.6
ECE:
Accuracy
0.6
ECE: 11.36
0.4
0.4
0.4
0.2
0.2
0.0
0.2
0.4
0.6
1.0
0.0
0.0
0.4
0.6
0.8
1.0
0.6
Confidence
Confidence
Confidence
Figure 11: Reliability plots on CIFAR-100(200) for the original (top) and our (bottom) versions of the three pseudo-label SSL methods selected.
Expected Actual
FixMatch
FlexMatch
Expected Actual
FreeMatch
Actual
0.8
0.8
0.8
Error: 21.09
Error: 16.20
Error: 16.62
Accuracy
0.6
ECE: 15.01
Accuracy
8.6
ECE: 12.39
Accuracy
0.6
ECE: 12.14
0.4
0.4
0.2
Confidence
Confidence
0.0
0.0
0.2
Confidence
0.4
066
0.8
1.0
Expected
FixMatch
Expected
FlexMatch
FreeMatch
Actual
Actual
Actual
Ours
0.8
0.3-
0.8
Error: 17.78
Error: 17.78
Fror:: 16.51
Accuracy
Accuracy
Accuracy
0.2
0.2
0.0
0.0
0.2
0.4
0.6
0.5
1.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
Confidence
Confidence
Confidence
Figure 12: Reliability plots CIFAR-100(400) for the original (top) and our (bottom) versions of the three pseudo-label SSL methods selected.
24


Reviews:
[{"review_id": "YesAhiMxoN", "reviewer": "Reviewer_RqMG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* The considered problem setting is interesting and broadly relevant for the TMLR community.\n* The work is reasonably clear and easy to understand, and the figures are relatively well designed and make for a pleasant read.\n* The proposed method does appear to improve performance on the considered tasks, both in terms of discriminative ability and calibration.\n\n**Weaknesses:**\n\nI have some concerns regrading the proposed solution and some of the claims made in this work.\n* logits are unnormalized, hence one can minimize the distance between them by simply scaling them all down by the same constant. Since the margin hyper-parameter is fixed, you can achieve the same effect just by using a temperature parameter in the softmax without needing to induce a constrained optimization problem.\n* the margin penalty is not differentiable due to the $max_j\\ (\\ell_{i,j})$ term, how do you handle this?\n* appropriately softening the pseudo-label would achieve a similar effect; this can be seen in Table 5, where the effects of the proposed margin constraint are harmful in the few-label setting (200) and marginally helpful in the many label setting (1000) when using soft pseudo-labels (SimMatch).\n* On the claim that semi-supervised learning using pseudo-labels can be approximated by a supervised loss and an entropy minimization term: I am not sure there is evidence to support this claim based on your exposition. Specifically, even if the third summation in equation (6) has more terms than the second term, the second term (consistency) may still provide important gradient signal. I expect that you will probably obtain notably degraded performance if you ran an ablation using only a supervised loss and an entropy minimization regularizer.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies calibration in semi-supervised learning based on pseudo-labels, where a supervised classification loss is combined with a pseudo-label loss on unlabeled samples. The contributions of this work are primarily empirical, with numerical experiments conducted with FixMatch and variants (FlexMatch, FreeMatch, and an ablation on SimMatch) on CIFAR, STL, and EuroSAT using a ViT-Small.\n\n### Observations:\nThis work finds:\n1. The expected calibration error (ECE) of FixMatch and variants is poor compared to a supervised baseline.\n2. Hard pseudo-labels encourage the model to make confident predictions for individual samples; specifically, for images in the mini-batch  with the same argmax class prediction between weakly and strongly augmented views, the cross-entropy simply becomes an entropy minimization term (min-entropy).\n3. More than 80% of the unlabeled samples have the same argmax class prediction between weakly and strongly augmented views at some point early in training; which the authors take to mean that semi-supervised learning using pseudo-labels can be approximated by a supervised loss and entropy minimization term.\n4. The logits produced by FreeMatch are less concentrated and tend to have higher magnitude than those of a supervised learning baseline.\n\n### Proposed solution:\nBased on these observations, this work proposes to regularize FixMatch and related methods with a logit distance penalty, that encourages the pair-wise logit differences between classes to be **smaller** than a pre-specified margin (this is computed on a per-instance basis) to reduce overconfident predictions. Empirically, the proposed regularizer is reported to improve top-1 prediction accuracy and expected calibration error.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* On the claim that semi-supervised learning using pseudo-labels can be approximated by a supervised loss and an entropy minimization term; please consider running an ablation using only the third term in equation (6), while zeroing out the second term.\n* Please clarify how you handle the fact that the margin penalty is not differentiable due to the strict max term.\n* Please include SimMatch in all the tables/figures (not just Table 5) and expand the evaluation to the other datasets you consider.\n* Please conduct ablations using various temperature parameters in the softmax (rather than the proposed penalty constraint).", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fgA2Wp9Zzq", "reviewer": "Reviewer_FYCB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper studied a vital yet often neglected problem: probability calibration in SSL;\n\n2. The insight provided in this paper is very interesting; the authors proposed that the SSL model tends to be more significantly miscalibrated compared to the fully supervised learning and attribute the potential cause as the different entropy-minimization scheme, which appears to be a very novel perspective;\n\n3. The proposed calibration techniques do not require a labeled validation set to calculate ECE, which is very practical under the SSL setting.\n\n**Weakness:**\n\n1. Some highly relevant works on the importance of SSL and probability calibration are not included and discussed [1,2], all of which should be added for comparison in the experiment section.\n\n2. Some claimed contributions overlap with existing works; specifically, the empirical demonstration of SSL is mis-calibrated and is not new, which has already been extensively discussed in existing works [1, 2]. \n\n3. The attribution on the cause of miscalibration in SSL lacks further support; if the consistency loss on unlabeled data is the cause of more severe miscalibration, this should be demonstrated by the disparency of calibration error between standard training with and without consistency loss. \n\n4. As an empirical study, authors should consider adding experiments on more common settings, i.e., using WRN as most of the other works. \n\n5. While I believe this paper is very insightful, I have doubts about the fundamental cause for the more significant miscalibration in SSL. Specifically, the authors believed it is because the loss is minimizing the min-entropy over the unlabeled data. **However, if the pseudo-labels are correct, the min-entropy of $\\mathcal{D}_{U^{''}}$ has no difference to the standard CE loss,** except for the strong data augmentation. In other words, you can equivalently translate the CE on labeled data into the same decomposition, which contradicts the authors' hypothesis that SSL can lead to more severe miscalibration than fully supervised learning.\n\n6. The solution seemed to be straightforwardly borrowed from existing works, undermining this work's technical contributions.\n\n[1] \"In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning.\" ICLR, 2020.\n\n[2] \"On the Importance of Calibration in Semi-supervised Learning.\" arXiv, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the issue of miscalibration in semi-supervised learning (SSL), showing that these models often become overly confident due to the minimization of min-entropy on unlabeled data, which is different from standard fully-supervised training. The authors leverage a margin-based label smoothing technique to mitigate overconfidence during training, which effectively improves the calibration without compromising the discriminative performance of SSL models across various image classification benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Adding discussion that acknowledges the existing studies in this area, highlighting the differences between your research and their work;\n\n2. Adding existing methods for comparison in experiments; \n\n3. Adding experiments or visualizations to illustrate the disparency of ECE for supervised training with and without consistency loss.\n\n4. Adding more experiments for a more common experiment setup;\n\n5. Properly address any other aforementioned issues.", "broader_impact_concerns": "No need for Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NHDS2QDCjS", "reviewer": "Reviewer_2xis", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Innovative Identification of Miscalibration Issues: The paper provides a thorough empirical and theoretical analysis of miscalibration in pseudo-labeling SSL methods, identifying the minimization of the min-entropy term as a key cause of overconfident predictions.\n\n- Effective Solution with Simple Implementation: The proposed penalty term is straightforward to implement and effectively addresses the identified miscalibration issue, making it a practical addition to existing SSL frameworks.\n\n- Comprehensive Experimental Validation: Extensive experiments across multiple benchmarks and tasks demonstrate the robustness and generalizability of the proposed solution, showing improvements in both calibration and accuracy.\n\nWeaknesses:\n- Focus on Hard Pseudo-labels: The primary focus is on SSL methods using hard pseudo-labels. While there is some discussion on methods using soft pseudo-labels, a more detailed analysis and comparison in this area could provide additional insights.\n\n- Scalability and Computational Cost: The paper does not extensively discuss the computational cost and scalability of the proposed method, particularly for very large datasets or models. Including this analysis would help readers understand the practical implications of adopting the solution in real-world scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors focus on addressing miscalibration in state-of-the-art semi-supervised learning (SSL) methods. They empirically demonstrate that these methods are significantly miscalibrated due to the minimization of the min-entropy term, which leads to overconfident and often incorrect predictions. They introduce a simple penalty term that enforces low logit distances in predictions on unlabeled samples, preventing overconfidence and improving calibration. Comprehensive experiments across various benchmarks show that this solution not only enhances calibration performance but also improves overall accuracy. The proposed method outperforms existing calibration techniques and proves effective in both standard and long-tailed classification tasks, offering a robust enhancement to SSL models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please try to have some discussion with the weaknesses parts.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: No further comments beyond those in claims/evidence and audience.
Decision: Accept (poster)