Credal Self-Supervised Learning
Authors: Julian Lienen
Eyke Hüllermeier
Venue: NeurIPS 2021 Poster (2021)
Keywords: semi-supervised learning
self-training
superset learning
credal sets
label relaxation
pseudo-labeling

Abstract: Self-training is an effective approach to semi-supervised learning. The key idea is to let the learner itself iteratively generate "pseudo-supervision" for unlabeled instances based on its current hypothesis. In combination with consistency regularization, pseudo-labeling has shown promising performance in various domains, for example in computer vision. To account for the hypothetical nature of the pseudo-labels, these are commonly provided in the form of probability distributions. Still, one may argue that even a probability distribution represents an excessive level of informedness, as it suggests that the learner precisely knows the ground-truth conditional probabilities. In our approach, we therefore allow the learner to label instances in the form of credal sets, that is, sets of (candidate) probability distributions. Thanks to this increased expressiveness, the learner is able to represent uncertainty and a lack of knowledge in a more flexible and more faithful manner. To learn from weakly labeled data of that kind, we leverage methods that have recently been proposed in the realm of so-called superset learning. In an exhaustive empirical evaluation, we compare our methodology to state-of-the-art self-supervision approaches, showing competitive to superior performance especially in low-label scenarios incorporating a high degree of uncertainty.

Credal Self-Supervised Learning
Julian Lienen Department of Computer Science
Eyke Hüllermeier Institute of Informatics University of Munich (LMU)
Paderborn University Paderborn 33098, Germany
Munich 80538, Germany eyke@ifi 1mu de
julian. www.ennunn......mm de
Abstract
Self-training an effective approach to semi-supervised learning. The key idea is to let the learner itself iteratively generate "hseudo-supervision" for unlabeled instances based on its current hypothesis. In combination with consistency regular- ization, pseudo-labeling has shown promising performance in various domains, for example in computer vision. To account for the hypothetical nature of the pseudo- labels, these are commonly provided in the form of probability distributions. Still, one may argue that even a probability distribution represents an excessive level of informedness, as suggests that the learner precisely knows the ground-truit conditional probabilities. In our approach, we therefore allow the learner to label instances in the form of credal sets, that is, sets of (candidate) probability distri- butions. Thanks to this increased expressiveness, the learner is able to represent uncertainty and lack of knowledge in more flexible and more faithful manner. To learn from weakly labeled data of that kind, we leverage methods that have recently been proposed the realm of so-called superset learning. In an exhaustive empiri- cal evaluation, we compare our methodology to state-of-the-arr self-supervision approaches, showing competitive to superior performance especially low-label scenarios incorporating high degree of uncertainty.
Introduction
Recent progress and practical success in machine learning, especially in deep learning, is largely due to an increased availability of data, However, even if data collection cheap in many domains, labeling the data so to make it amenable to supervised learning algorithms might be costly and often comes with significant effort. As a consequence, many data sets are only partly labeled, i.c. only few instances are labeled while the majority is not. This is the key motivation for semi-supervised learning (SSL) methods 7 which seek to exploit both labeled and unlabeled data simultaneously.
As one simple yet effective methodology, so-called self-training (26] often also referred to as pseudo- labeling, has proven effective in leveraging unlabeled data improve over training solely on labeled data. The key idea is to let the learner itself generate "pseudo-supervision" based on its own current hypothesis. In the case of probabilistic classifiers, such pseudo-targets are usually provided in the form of (perhaps degenerate) probability distributions. Obviously, since pseudo-labels are mere guesses and might be wrong, this comes with the danger of biasing the learning process, a problem commonly known as confirmation bias 46). Therefore, self-training is nowadays typically combined with additional regularization means, such as consistency regularization 40, or additional uncertainty awareness [38]
for unlabeled instances
Labeling an instance with probability distribution on the target space is certainly better than committing to precise target value, for example single class label in classification, as the latter would suggest a level of conviction that warranted. Still, one may argue that even a probability distribution represents an excessive level of informedness. In fact, actually suggests that the learner
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

precisely knows the ground-trutt conditional probability ply æ). In our approach, we therefore allow the learner to label instances in the form of credal sets, that is, sets of (candidate) probability distributions [27]. Thanks to this increased expressiveness, the learner is able to represent uncertainty and a lack of knowledge about the true label (distribution) manner. For example, by assigning the biggest credal set consisting of all probability distributions, it is able to represent complete ignorance- -a a state of knowledge that is arguably less well represented by uniform probability distribution, which could also be interpreted as full certainty about this distribution being the ground truth. Needless to say, this ability is crucial to avoid a confirmation bias and account for the heteroscedastic nature of uncertainty, which varies both spatially (i.e., among different regions in the instance space) and temporally: typically, the learner is less confident in early stages of the training process and becomes more confident toward the end.
more flexible and more faithful
Existing methods are well aware of such problems but handle them in manner that is arguably ad-hoc. The simple yet effective SSL framework Fix [11, for instance, applies a thresholding technique to filter out presumably unreliable pseudo-labels, which often results in unnecessarily delayed optimization, as many instances are considered only lately in the training. Other approaches, such as MixUp [53), also apply mixing strategies to learn in a more cautious manner from pseudo- labels M 4 Moreover, many of these approaches, including FixMatch, rely on the principle of entropy minimization 19] to separate classes well, the self-supervision is generated in the form of rather peaked even degenerate distributions to learn from, which amplifies the problems of confirmation bias and over-confidence (33]
An important implication of credal pseudo-labeling is the need for extending the underlying learning algorithm, which must be able to learn from weak supervision of that kind. To this end, we leverage the principle of generalized risk minimization, which has recently been proposed in the realm of so-called superset learning [22). This approach supports the idea of data disambiguation: The learner is free to (implicitly) choose any distribution inside a credal set that appears to be most plausible in light of the other data (and its own learning bias). Thus, an implicit trade-off between cautious learning and entropy minimization can be realized: Whenever it seems reasonable to produce an extreme distribution, the learner is free but not urged to do so. Effectively, this not only reduces the risk of potential confirmation bias due to misleading over-confident pseudo-labed allows for incorporating all unlabeled instances in the learning process from the beginning without any confidence thresholding, leading to fast and effective semi-supervised learning method.
it also
To prove the effectiveness of this novel type of pseudo-labeling, we proceed from FixMatch as an effective state-of-the-art SSL framework and replace conventional probabilistic pseudo-labeling by credal target set modeling. In an exhaustive empirical evaluation, we study the effects of this change compared to both hard and soft probabilistic target modeling, well as measuring the resulting network calibration of induced models to reflect biases. Our experiments not only show competitive to superior generalization performance, but also better calibrated models while cutting the time to train the models drastically.
2 Related Work
In semi-supervised learning, the goal is to leverage the potential of unlabeled in addition to labeled data to improve learning and generalization. As it constitutes a broad research field with a plethora of approaches, we will focus here classification methods as these most closely related to approach. We refer to and (48] for more comprehensive overviews.
As one of the earliest ideas to incorporate unlabeled data in conventional supervised learning, self- training has shown remarkably effective in various domains, including natural language processing [12] and computer vision 11 [3]. The technique is quite versatile and can be applied for different learning methods, ranging from support vector machines [30] to decision trees (45) and neural
networks [35). It can be considered as the basic training pattern in distillation, such as self-distillation
from a model to be trained itself (3 or within student- sudent-teacchr settings [36] 51]. Recently, this
technique also lifted completely unsupervised learning in computer vision to new level 66 20].
As a common companion of self-training for classification, especially in computer vision, consistency regularization is employed to ensure similar model predictions when facing multiple perturbed versions of the same input k 40,, resulting noise-robustness as similarly achieved by other
(stochastic) ensembling methods such as Dropout [2). Strong augmentation techniques used in this

---

regard, e.g., CTAugment [3] or RandAugment (9) allow one to learn from instances outside of the (hitherto labeled) data distribution and, thus, lead to more accurate models [0]. For supi-supere learning in image classification, the combination of consistency regularization with pseudo-
widely adopted and has proven to be a simple yet effective strategy
25
50 [55]
As pseudo-labeling comprises the risk of biasing the model by wrong predictions, especially when confidence is low, uncertainty awareness has been explicitly considered in the generic supervisen framework UPS to construct more reliable targets (38]. Within their approach, the model uncertainty estimated by common Bayesian sampling techniques, such as MC-Dropout [15] or DropBlock [16], which is then used to sort out unlabeled instances for which the model provides uncertain predictions. Related this, the selection of pseudo-labels based on the model certainty has also been used in specific domains, such as text classification 32 or semantic segmentation [54]
2.1 FixMatch
As already mentioned, FixMatch [4]] combines recent advances in consistency regularization and pseudo-labeling into a simple yet effective state-of-the-art SSL approach. It will serve as a basis for our new SSL method, provides generic framework for fair comparisons between conventional and our credal pseudo-labeling.
In each training iteration, FixMatch considers a batch of B labeled instances B pt)]: C
X' P(Y) and μ.B unlabeled instances B, { } C X, where X' denotes the input feature space, y the set of possible classes, P()/) the set of probability distributions over y and μ 2 1 the multiplicity of unlabeled over labeled instances in each batch. Here, the probabilistic targets in BR are given as degenerate "one-hot" distributions. When aiming to induce probabilistic classifiers of the form p X - PP(Y), FixMatch distinguishes between two forms of augmentation: While A, X' X' describes "strong" augmentations that perturbs the image in drastic manner by combining multiple operations, simple flip-and-shift transformations are captured by "weak" augmentations A.: X - X.
As an iteration-wise loss to determine gradients, a combination of the labeled loss L and unlabeled loss Lu is calculated. For the former, the labeled input instances from Bi are weakly augmented and used conventional cross-entropy loss H P())2 - R. For the latter, the model prediction version of each unlabeled instance æ is used to construct
q := p(AA(()) on a (hard) pseudo-label ğ E P.Y)) when meeting a predefined confidence threshold т. While q is in FixMatch a degenerate probability distribution by default, one could also inject soft probabilities, which, however, turned out to be less effective (cf. (41). The pseudo-label is then compared to strongly-augmented version of the same input image. Hence, the unlabeled loss L, is given by
Cu
Imax 42ax H(G
μB
mEB
3 Credal Self-Supervised Learning
In this section, we introduce our credal self-supervised learning (CSSL) framework for the case of classification, assuming the target to be categorical with values in y (y1...., yK]- Before presenting more technical details, we start with motivation for the credal (set-valued) modeling of target values and a sketch the basic idea of our approach.
3.1 Motivation and Basic Idea
In supervised learning, we generally assume the dependency between instances æ E X' and associated be of stochastic nature. More specifically, one typically assumes "ground-truth" truth" in the form of a conditional probability distribution p' x)) P())). Thus, for every y V. p" (y x) is the probability to observe y as a value for the target variable in the context æ. Ideally, the distribution p" (-1 æ) would be provided as training information to the learner, along with every training instance æ. practice, however, supervision comes in the form of concrete values of the target, i.e., realization degenerate distribution Py assigning probability mass 1 to y (i.e., Py a)) = I and (, a)) = for y' # y) is taken surrogate for p"
observations (outcomes)
) of the random variable Y ~ p²(- ar), and the corresponding
3

---

Obviously, turning the true distribution p== (- subsequently also called "soft label", into a more extreme distribution Py (i.e., single value y), referred to as "hard label", may cause undesirable effects. In fact, it suggests level of determinism that is not warranted and tempts the learner to over-confident predictions that are poorly calibrated, especially when training with losses such as cross-entropy [33]. So-called label smoothing 44 seeks to avoid such effects by replacing hard labels Py with (hypothetical) soft labels ş that are "close" to py-
Coming back to semi-supervised learning, training of the learner and self-supervision can be charac- terized for existing methods such as FixMatch as follows:
The learner is trained on hard labels, which are given for the labeled instances æ and constructed by the learner itself for unlabeled instances Ta.
Construction of (pseudo-)labed is done in two steps: First, the true soft label p = =ppp (-1 a..) is predicted by distribution á p æu), and the latter is tumned into hard label Py afterward, provided ş suggests a sufficient level of certainty (support for y).
This approach can be challenged for several reasons. First, training probabilistic predictors on hard labels comes with the disadvantages mentioned above and tends to bias the learner. Second, pseudo-labels Py constructed by the learner tend to be poor approximations of the ground truth p". In fact, there will always be discrepancy between p" and its prediction p. and this discrepancy is further increased by replacing á with b. Third, leaving some of the instances - those for which the prediction is not reliable enough- - completely unlabeled causes a loss of information. Roughly speaking, while a part of the training data is overly precise[ another part remains unnecessarily imprecise and hence unused. This may slow down the training process and cause other undesirable problems such as
(training is influenced by the order of the unlabeled instances).
To avoid these disadvantages, we propose to use soft instead of hard labels as training information for the learner. More specifically, to account for possible uncertainty about true soft label p", we model information about the target in the form of set Q S PP))) of distributions, in the literature on imprecise probability also called a credal set (27]. Such a "credal label" is supposed to cover the ground truth, i.e., p² E Q, very much like confidence interval in statistics supposed to cover (with high probability) some ground-truth parameter to be estimated.
This approach is appealing, as it enables the learner to model its belief about the ground-truth p" in cautious and faithful manner: Widening credal label Q may weaken the training information but maintains or even increases its validity. Moreover, credal labeling elegantly allows for covering the original training information as special cases: A hard label y provided for a labeled instance æ corresponds to singleton set Qu = {Py], and the lack of any information in the case of an unlabeled instance va is properly captured by taking Q = P())). Starting with this information, the idea is to modify in two directions:
Imprecisiation: To avoid possible disad vantages of hard labels Qy. these can be made less precise through label relaxation 22)], which is extension of the aforementioned label smoothing. Technically, it means that Qy is replaced by a credal set Q containing, in addition to Py itself, distributions p close Py-
Precisiation: The non-informative and maximally imprecise labels Q= PP))) for unlabeled instances F,, are successively (iteration by iteration) "shrunken" and made more precise. This is done by replacing set Q with a smaller subset Q' C Q, provided the exclusion of certain candidate distributions p E Q sufficiently supported by the learner.
3.2 Credal Labeling
Credal sets are commonly assumed to be convex, i.c., p, 9 E Q implics A X)q E Q for all
distributions p, g and X E (0, 1). In our context, arbitrary (convex) credal sets Q C P()") could in principle be used for the purpose of labeling instances. Yet, to facilitate modeling, we restrict ourselves to credal sets induced by so-called possibility distributions [3].
possibility or plausibility measure II is a set-function 2 - [0, 1] that assigns a degree of
plausibility II(Y) to every subset (event) Y C y. Such a measure is induced by a possibility
'To some extent, this can be alleviated through measures such as instance weighing, i.c. by attaching a weight to pseudo labeled instances (38)

---

distribution y - [0. 1] via II) = m (य) for all Y S y. Interpreting degrees of plausibility as upper probabilities, the set of (candidate) probability distributions p (resp. measures P) in accordance with π resp. IT is given by those for which P(Y) I (T()) for all events Y:
Q+ = O==(PERO)IVYE C). P()) = Σ p(v) S
max π (y) = [()) yEY
(1)
MEY
Roughly speaking, for each E y the possibility [ty determines upper bound for p" (v), i.e., the highest probability that is deemed plausible for y. Note that, to guarantee Q+ + D. possibility distributions must be normalized in the sense that maxyey T(y) = 1. In other words, there must be least one outcome E y that is deemed completely plausible.
In our context, this outcome is naturally taken as the label y argmaxy' p(y') with the highest predicted probability. A simple way of modeling then consists of controlling the degree of imprecision (ignorance of the learner) by single parameter E [0, 1]. considering credal sets of the form
2 {p E P()) p(y) 2 1 a-
Thus, Qg consists of all distributions p that allocate probability mass of at least 1 to y and hence most the other labels {v). As important special cases we obtain Q² { i.e. the degenerate distribution that assigns probability to the label y, and Q = P(Y) modeling complete ignorance about the ground truth y".
Needless to say, more sophisticated credal sets could be constructed on the basis of distribution p. for example leveraging the concept of cababbllty-possbbiiyy transformations [14). Yet, to keep the modeling as simple possible, we restrict ourselves sets of the form (2) in this work.
3.3 Learning from Credal Labels
Our approach requires the learner to be able to learn from credal instead of probabilistic or hard labels. To this end, we refer the generic approach so-called superset learning as proposed in [22].
Essentially, this approach is based on minimizing a generalization L the original (probabilistic)
loss L: 2 _> RR. More specifically, the so-called optimistic superset loss [22 also known as
infimum loss
[ compares credal sets with probabilistic predictions as follows:
L (Q. P)
min C(p.
PEQ
For the specific case where credal sets are of the form and the loss L is the Kullback-Leebler divergence DKL, 43 simplifies to
L ("G
= DKL(D"lpp)
i E 2 otherwise
(4)
where
1-
if y =ツ otherwise
8(v')
(5)
Eyn Pla")
is the projection of the prediction و onto the boundary of Q. This loss has been proven to be convex, making its optimization practically feasible [29]
The loss 3 an optimistic generalization the original loss in the sense that it corresponds to the
loss L(p.p) for the most favorable instantiation E Q. This optimism is motivated by the idea of data disambiguation [22) and can be justified theoretically [.. Roughly speaking, minimizing the sum of generalized losses over all training examples (æ,, Q4) comes down to (implicitly) choosing precise probabilistic target inside every credal set, i.c., replacing (x, Q:) by (æ, p;) with p; E Q.. in such way that the original loss (empirical risk) can be made small as possible.
3.4 Credal Self-Supervised Learning Framework
Our idea of credal self-supervised learning (short CSSL) offers a rather generic framework for designing self-supervised learning methods. In this paper, we focus on image classification concrete and practically relevant application, and combine CSSL with the consistency regularization framework provided by FixMatch (cf. Fig.
5

---

Weakly- augmented image
Precisiation based weighted model prediction
Pdata
Target set
q
P
Model
Do
arginn
Fnist
Original image
Pyt
Strongly- augmented image
Model prediction
2
9/3
p
p
5/2
Model
p
5/
Figure Schematic overview of the learning framework for unlabeled instances following 44 Given classes y/ = {y1, 1/2 ys ,, credal target set Q is generated from the prediction on a weakly- augmented version the imput image. As illustrated in the barycentric coordinate system on the right bottom, Q (red shaded region) covers the ground-truth distribution p" (green point). The degenerate distribution Py, assigning probability to y1 corresponds to the red point the top. To calculate the final loss, the model prediction on to Q, whereby its projection onto Q depicted by p".
version of the original image is compared
To describe the algorithm, we again assume batches of labeled B; and unlabeled instances B,, (cf. Section 2..). For the former, we measure the labeled loss Li in terms of the cross-entropy H between observed target distributions augmened features. At this point, one could apply the idea of imprecisiation through label relaxation as motivated in Section 3.1]] However, as this work focuses on the self-supervision part, i.c., on B. rather than B,, stick to hard labels. This also facilitates the interpretation of experimental results later on, as it avoids the mixing of different effects.
(labeled) training instances and the current model predictions
For the unlabeled instances E Bu. we follow the consistency regularization idea of FixMatch and use the predictions on weakly augmented versions of the unlabeled instances reference for the target set construction. More precisely, we take the class y/- argmax,yyy P: (y) of the current model prediction f: = (x:)). as reference to construct instance-wise targets Qo: as specified in 2
rather straightforward approach to determining the uncertainty level ar is to set as 1 (9:). Yet, motivated by the idea of distribution alignment as proposed in [3, suggest weight the predictions P. by the proportion of the class prior p E P()) and a moving average of the last model predictions p E F()",, that a; 9:(y)/ Ever 4 (y') with the weighted (pseudo--probability scores
q:(y) =:(1)) F(y')
ply')
According this way of modeling pseudo-labels in terms of credal sets, the size (imprecision) of a set is direct correspondence with the confidence of the learner. Therefore, this approach leads some sort (implicit) disambiguation: With increasing confidence, the target sets are becoming more precise, successively fostering entropy minimization without imposing overly constrained targets. Also, mentioned before, this approach allows for using all instances for training from the very beginning, without losing any of them due to confidence thresholding. The weighting mechanism based on the class prior ğ and the prediction history P (second factor on the right-hand side of (6)) accounts for the consistency and hence the confidence in particular class prediction. the past predictions, suggesting that its true likelihood might be higher than predicted by the learner, and vice versa in the case where p(y/) & p(y'). this procedure is simple and computationally efficient, it facilitates the applicability compared to computationally demanding uncertainty methods such as MC-Dropout.
p(y') S p(y'), the label y/ is under-representte

---

The proposed target sets are then used within the unlabeled loss Lu according to 4 addition to the labeled loss C.. Hence, the final loss is given by
which is used in
L =
Σ (B
H(P....) + Au
Σ
L2 (Q
P.)
B
B
B.EB
The pseudo-code of the complete algorithm can be found in the appendix.
We conclude this section with few remarks implementation details. Since are building upon Fix.Mttch, we keep the same augmentation policy as suggested by the authors. Thus, we employ CTAugment by default. We refer to the appendix for further ablation studies, including RandAugment as augmentation policy. Moreover, we consider the same optimization algorithm as used before, namely SGD with Nesterov momentum, for which use cosine annealing learning rate schedule 31. Similar to FixMatch,
set the learning rate to 77 cos IGK
7ark where 7) is the initial learning rate, keeping the
k the current training step and K the total number of steps (2 by default). As algorithmic framework the same and do not require any form of confidence thresholding, we can reduce the number of parameters compared to FixMatch, which further facilitates the use of this approach. We also use an exponential moving average of model parameters for our final model, which comes with appealing ensembling effects that typically improve model robustness and has been considered by various recent approaches for un- or semi-supervised learning 6 41).
4 Experiments
To compare our idea of credal pseudo-labeling, we conduct an exhaustive empirical evaluation with common image classification benchmarks. More precisely, we follow the semi-supervised learning evaluation setup as described in [1 and perform experiments on CIFAR-10/- 100-100 [24]. SVHN [34),
and STL-10 10 [8] with varying fractions of labeled instances sampled from the original data sets, also considering label-scarce settings with only few labels per class. For CIFAR- 10, SVHN, and STL-10. we train Wide ResNet-28-22 [52] with 1.49 M parameters, while we consider Wide ResNet-28-8 (23.4 parameters) models for the experiments on CIFAR-100. To guarantee a fair comparison to existing methods related to FixMatch, we keep the hyperparameters the same as in the original experiments. We refer to the appendix for more comprehensive overview of the experimental details. We repeat each run 5 times with different seeds for higher significance and average the results for the model weights of the last 20 epochs as done in
As baselines, we report the results for Mean Teacher [46.. MixMatch 4). UDA (50), ReMixMatch B]] and EnAET 49] as state-of-the semi-supervised learning methods. Moreover, as we directly compete against the probabilistic hard-labeling employed in FixMatch, we compare our approach to this (with CTAugment) and related methods, namely AlphaMatch [18), CoMatch [28]. and ReRankMatch 47]. Since these approaches extend the basic framework of Fix by additional means (e.g., loss augmentations), the direct comparison with FixMatch is maximally fair under these conditions, avoiding side-effects much as possible. In addition, we show the results of the student-teacher method Meta Pseudo Labels (Meta PL) (36..
4.1 Generalization Performance
In the first experiments, we train the aforementioned models on the four benchmark data sets for different numbers of labeled instances to measure the generalization performance of the induced models. The results are provided in Table I
As can be seen, CSSL is especially competitive when the number of labels is small, showing that the implicit uncertainty awareness of set-based target modeling becomes effective. But CSSL also able to provide compelling performance in the case of relatively many labeled instances, although not substantially improving over conventional hard pseudo-labed labeling. For instance, 11FAR-100 and SVHN. Focusing on the comparison to
approximately
on par with state-of-the-art performance FixMatch, credal supervision improves the performance in almost all cases over hard pseudo- labeling with confidence thresholding, providing further evidence for the adequacy of our method.
7

---

Table I: Averaged misclassification rates for 5 different seeds using varying numbers of labeled instances (bold font indicates the best performing method and those within two standard deviations per data set and label number). Approaches using different models, so that the comparison may not
be entirely fair, are marked with *. We also show the results for STL- 10 as reported in [18], using smaller models due to computational resource limitations, which we mark by f.
CIBAR-19 10
CIFAR-100
SVHN
SIL-10"
40 lab.
250 lab.
2000 lnb.
400 lab.
2500 lab.
10000 lab
40 lab.
250 Jab.
1000 Jab.
1000 lab
Moun Teacher
32.32 12.30
9.19 20.19
53.91 20.57
35.85 20.24
3.57 20.11
3,42 $0.07
Mixcaachh
47.54 $11.50
11:05 10.36
6.42 10.10
67.61
61 30
39.04 10.37
28.31 20:31
42.55 1:1-53
1.95 6:121
3.50 10.28
14.84 21:24
UDA
29.05 =5.93
8.82
4.88
59.28 20.8
33.13
24.30
52.63 4:20:1
5.69 22.7
2.46 20.24
13:43 11
ReMixMatch
19.10=9.64
5.44
4.72
44.28
23,03
10.30
2.92
11.58 60.76
7.6 20.34
5.35
3.21 =9.21
AlphuMacch
5.65
4.97
38.74
25,02 +0.27
20:36
2.44 50.32
9.64 10.75
CoMatch
6.91 =1.90
4.91
ReRcniMacch
18.25 -9.44
6.02 11.31
4.40 =0,00
69.62 #1.33
31.75 20.35
22.32 20.05
20:25 +44433
2.44 +000
2.19 +0009
Meta PL"
3,89 10007
1.99 10007
FixMatch (CTAA)
11.39 33.85
5.07 +0.33
4:31 #0.15
49.95
#3.01
28.64
+0.24
23.18
#0.11 11
7.65 #7.65
2.64 #0.64
2.28
+0. 19
10.72 20.6
CSSL. (CTA)
6.50 #0.90
5.48
+0.49
4.43 +0.10
43.43
#1.80
28.39
+1.00
23,25
#0.25
3.67
#2.36
2.18 1:.12
1.99 10.13
10.54 10.71
Table 2: Averaged misclassification rates and expected calibration errors (ECE) using 15 bins for different seeds.
CIFAR-10
SVHN
40 lab.
4000 lab.
40 lab.
1000 lab.
Emr
BCE
Err.
ECE
Err.
ECE
Err.
ECE
FixMatch
11.39 =3.35
0.087
=0.051
4.31 10.15
0.030
10.002
7.65 17.65
0,040
10014
2.28 20.19
0.010
=0.002
FixMatch (DA)
7.73 =1.92
0.048
-0.012
4.64
+0.10
0.027
+0.001
5.21 +2285
0.031
+0.000
2.04
=0.38
0.010
=0.001
LSMatch
8.37
0.038
=0.012
5.60
11.32
0.024
10:007
3.82
11/46
0.086
10.06
2.13 20.11
0.018 00.11
CSSL
6.50 #0.90
0.032 =0,005
4.43 +0.10
0.023 #0.001
3,67 #2.36
0,022
10.029
1.99 =0.13
0,007 =0,001
4.2 Network Calibration
In a second study, we evaluate FixMatch-based models in terms of network calibration, i.e., the quality of the predicted class probabilities. For this purpose, we calculate the expected calibration error (ECE) as done in 21] using discretized probabilities into 15 bins. The calibration errors provide insight into the bias of the models induced by the different learning methods.
Besides the "raw" hard-labeling, we also consider the distribution alignment (DAiavariant of FixMatch (as described in [41). as well as an adaptive variant using label smoothing [44], which we dub LSMatch (see appendix for an algorithmic description). For label smoothing, we use a uniform distribution policy and calculate the distribution mass parameter a in an adaptive manner as we do for CSSL (cf. Section (.44). As a result, LSMatch can be regarded as the natural counterpart of our approach for a more cautious learning using less extreme targets. We refer to [29] for a more thorough analysis of the differences between smoothed and credal labels. Since both methods realize an implicit calibration 29 33 we omit explicit calibration methods that require additional data. Besides, we experiment with an umcertainty-filterrin variant of FixMatch following UPS [38.. for which we provide results in the supplement.
In accordance with the studies in [29], the results provided in Table Dalee[ssa show improved calibration compared to classical probabilistic modeling. As these effects are achieved without requiring an additional calibration data split, provides an appealing method to induce well calibrated and generalizing models. Nevertheless, the margin to the other baselines gets smaller with an increasing number of labeled instances, which is plausible as implies an increased level of certainty.
4.3 Efficiency
In addition to accuracy, a major concemn of learning algorithms is run-time efficiency. In this regard, we already noted that thresholding mechanisms may severely delay the incorporation of unlabeled instances the learning process. For example, for a data set with 226 images, FixMatch requires up to 22 updates to provide the competitive results reported. This not only excludes potential users without access to computational resources meeting these demands, it also consumes a substantial amount of energy 433.
8

---

Table 3: Averaged misclassification rates after 1/8 (CIFAR-10) and 1/32 (SVHN) of the original iterations used for the results in Tab. 1 (bold font indicates the single best performing method).
CIFAR-10
SVHN
40 lab.
4000 lab.
40 lab.
1000 lab.
FixMatch (r 0.0)
18.50 12.92
6.88
#0.11
13.82
13.57
2.73
10.04
FixMatch (T 0.8)
11.99 +2.32
7.08 40.13
3.52 10.44
2.85 +0.08
FixMatch (r 0.95)
14.73 +3.29
8,26 +0.09
5.85 #5.10
3.03 +0.07
LSMatch
11.60 12.6
7.24 #0.21
7.04 13.29
2.76 10.05
CSSL
10.04 +3.32
6.78 10.94
3.50 10.49
2.84 10.06
As described before, CSSL allows for incorporating all instances from the very beginning. To measure the implied effects, i.e., a faster convergence, apart from using more effective optimizers, we train the CIFAR-10 and SVHN for only eighth and thirty-second
models from the former experiment of the original number of epochs, respectively. As we would grant CSSL and LSMatch an unfair advantage compared confidence thresholding in FixMathh, we experiment with different thresholds r E (0, 0.8, (.95] for FixMatch. The (averaged) learning curves are provided in the supplement.
As shown by the results in Table CSSL achieves the best performance in the label-scarce cases, which confirms the adequacy of incorporating all instances in a cautious manner from the very beginning. Likewise, LSMatch shows competitive performance following the same intuition. In contrast, FixMatch with a high confidence threshold shows slow convergence, which can be mitigated by lowering the thresholding. However, FixMatch with 0.0 provides unsatisfying performance as it drastically increases the risk of confirmation biases; 7 = 0.8 seems to be a reasonable trade-off off between convergence speed and validity of the model predictions. Nevertheless, when providing higher numbers of labels, the performance gain in incorporating more instances from early on is fairly limited, making the concern of efficiency arguably less important in such cases.
5 Conclusion
Existing (probabilistic) methods for self-supervision typically commit to single probability dis- tributions pseudo-labels, which, as we argued, represents the uncertainty in such labels only insufficiently and comes with the risk of incorporating an undesirable bias. Therefore, we suggest to allow the learner to use credal sets, i.e., sets of (candidate) distributions, pscudo-labels. In this way, a more faithful representation of the Iearner's (lack of) knowledge about the underlying ground truth distribution can be achieved, and the risk of biases is reduced. By leveraging the principle of generalized risk minimization, we realize an iterative disambiguation process that implements an implicit trade-off between cautious self- celf-supervision and entropy minimization.
In an exhaustive empirical evaluation in the field of image classification, the enhanced expressiveness and uncertainty-awareness. compared to conventional probabilistic self-supervision proved to yield superior generalization performance, especially in the regime of label-scarce ceni-supervised learning. Moreover, the experiments have shown an improved network calibration when trained with credal self-supervision, as well as an increased efficiency when considering small compute budgets for training.
Motivated by these promising results, we plan to further elaborate on the idea of credal target modeling and extend it in various directions. For example, as we considered rather simple target sets so far, thorough investigation of more sophisticated modeling techniques should be conducted. Such techniques may help, for instance, to sift out implausible classes early on, and lead to more precise pseudo-labed without compromising validity. Besides, as already mentioned, our CSSL framework is completely generic and not restricted to applications in image processing. Although we used it to extend FixMatch in this paper, it can extend any other self-training method, too. Elaborating on such extensions is another important aspect of future work.

---

Acknowledgments and Disclosure of Funding
This work was supported by the German Research Foundation (DFG) (Grant No. 420493 178). Moreover, the authors gratefully acknowledge the funding of this project by computing time provided by the Paderborn Center for Parallel Computing (PC2) and the research group of Prof. Dr. Mareo Platzner.
References
[1] Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Pseudo- labeling and confirmation bias in deep remi-supervised learning. In Proc. of the International Joint Conference on Neural Networks, IJCNN, Glasgow, United Kingdom, July 19-24, pages 1-8. IEEE, 2020.
[2] Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Informa- tion Processing Systems, NIPS, Montreal, Quebec, Canada, December 8-13, pages 3365-3373, 2014.
[3] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. ReMixMatch: Semi-supervised learning with distribution matching and aug- mentation anchoring. In Proc. of the 8th International Conference on Learning Representations ICLR, Addis Ababa. Ethiopia, April 20.30. OpenReview.net, 2020.
[4] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. MixMatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems, NeurlPS, Vancouver, BC, Canada, December 14, pages 5050-5060, 2019.
[5] Vivien Cabannes, Alessandro Rudi, and Francis R. Bach. Structured prediction with partial labelling through the infimumm loss. In Proceedings of the 37th International Conference Machine Learning, ICML, virtual, July 13-18, volume 119, pages 1230-1239. PMLR, 2020.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. CoRR, Abs/2104.14294 2021.
[7] Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-Snpervised Learning. The MIT Press, 2006.
[8] Adam Coates, Andrew Y. Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proc. of the 14th International Conference on Artificial Intelligence and Statistics, AISTATS, Fort Lauderdale, FI, USA, April 11-13, volume 15 of JMLR Proceedings, pages 215-223. JMLR.org, 2011.
[9] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. RandAugment: Practical automated data augmentation with reduced search space. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS, virtual, December 6-12, 2020.
[10] Zihang Dai, Zhilin Yang. Fan Yang. William W. Cohen, and Ruslan Salakhutdinov. Good semi- supervised learning that requires bad GAN. In Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems, NIPS, Long Beach, CA, USA, December 4-9, pages 65 1510-6520, 2017.
[11] Carl Doersch, Abhinay Gupta, and Alexci A. Efros. Unsupervised visual representation learning by context prediction. In Proc. of the IEEE International Conference on Computer Vision, ICCV, Santiago, Chile, December 7-13, pages 1422-1430. IEEE Computer Society, 2015.
[12] Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, and Alexis Conneau. Self-training improves pre-training for natural language understanding, CoRR, abs/2010.02194, 2020.
10

---

[13] Didier Dubois and Henri Prade. Possibility theory, probability theory and multiple-valued logics: A clarification. Annals of Mathematics and Artificial Intelligence. 32:35-66 66, 2004.
[14] Didier Dubois, Henri Prade, and Sandra Sandri. On PossibilityPPooubbiity Transformations, pages 103-112. Springer, 1993.
[15] Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian approximation: Representing model uncertainty in deep learning. In Proc. of the 33nd International Conference on Machine Learning, ICML, New York City, NY, USA, June 19-24, volume 48 of JMLR Workshop and Conference Proceedings, pages 1050-10500 1059. JMLR.org, 2016.
[16] Golnaz Ghiasi, Tsung-Yi Yi Lin, and Quoc V. Le. DropBlock: A regularization method for convolutional networks. In Advances Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems, NeurIPS, Montreal, Quebec, Canada, December 3-8, pages 10750-10760, 2018.
[17] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow. Digging into self-supervised monocular depth estimation. In Proc. the IEEE/CVF International Conference on Computer Vision, ICCV, Seoul, Korea (South), October 27 -ovember 2, pages 3827-3837. IEEE, 2019.
[18] Chengyue Gong, Dilin Wang, and Qiang Liu. AlphaMatch: Improving consistency for semi- supervised learning with alpha-divergence. CoRR, abs/2011.11777 2020.
[19] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems 17: Annual Conference on Neural Information Processing Systems, NIPS, Vancouver, BC, Canada, December 13-18, pages 529- 536, 2004.
[20] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent- new approach to self-supervised learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS, virtual, December 6-12, 2020.
[21] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proc. of the 34th International Conference on Machine Learning, ICML, Sydney, NSW, Australia, August 6-11, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR, 2017.
[22] Eyke Hüllermeier and Weiwei Cheng. Superset learning based on generalized loss minimization. In Proc. of the European Conference on Machine Learning and Knowledge Discovery in Databuses, ECML PKDD, Porto, Portugal, September 7-11, Proc. Part II, volume 9285 of LNCS, pages 260-275. Springer, 2015.
[23] Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distillation with progressive refinement of targets. arXiv:2006. 120000 [cs, stat), ftts/2006.12000 2021.
[24] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Canada, 2009.
[25] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Proc. of the 5th International Conference on Learning Representations, ICLR, Toulon, France, April 24-26. OpenReview.net, 2017.
[26] Dong-Hyur Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, International Conference on Machine Learning, ICML, Atlanta, GA, USA, June 16-21, volume 2013.
[27] Isaac Levi. The Enterprise of Knowledge: An Essay on Knowledge, Credal Probability, and Chance. MIT Press, 1983.

---

[28] Junnan Li. Caiming Xiong. and Steven C. H. Hoi. CoMatch: Semi-supervised learning with contrastive graph regularization. CoRR, abs/2011.11183, 2020.
[29] Julian Lienen and Eyke Hüllermeier. From label smoothing to label relaxation. In Proc. of the 35th AAAI Conference on Artificial Intelligence, virtual, February 2-9, 2021.
[30] Julian Lienen and Eyke Hüllermeier. Instance weighting through data imprecisiation. Int. J. Approx. Reason., 134:1- 2021.
[31] Uya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, In Proc. of the 5th International Conference on Learning Representations, ICLR, Toulon, France, April 24-26. OpenReview.net, 2017.
[32] Subhabrata Mukherjee and Ahmed Hassan Awadallah. Uncertainty aawae self-training for few-shot text classification. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS, virtual, December 6-12, 2020.
[33] Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems, NeurIPS, Vancouver, BC, Canada, December 8- 8-14, pages 4696 4705, 2019.
[34] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Sys- tems, NIPS, Granada, Spain, November 12-17, Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[35] Avital Oliver, Augustus Odena, Colin Raffel, Ekin Dogus Cubuk, and Ian J. Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, NeurlPS, Montreal, Quebec, Canada, December 3-8, pages 3239-3223300 -250,, 2018.
[36] Hieu Pham, Zihang Dai, Qizhe Xie, Minh-Thang Luong, and Quoe V. Le. Meta Pseudo Labels. arXiv:2003. 10580 [cs, stat], 1058000000..000, 2021.
[37] Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with Ladder networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems, NIPS, Montreal, Quebec, Canada, December 7-12, pages 3554,3554, 2015.
[38] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh Singh Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. CoRR, abs/2101.063299 2021.
[39] Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. Proc. of the 7th IEEE Workshop on Applications of Computer Vision / IEEE Workshop Motion and Video Computing, WACVMOTION, Breckenridge, CO, USA, January 5-7, pages 29-36. IEEE Computer Society, 2005.
[40] Mehdi Sajaadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems, NIPS, Barcelona, Spain, December 5-10, pages 1163-1171, 2016.
[41] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: Simplifying semi- supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS, virtual, December 6-12, 2020.
[42] Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: simple way to prevent neural networks from overfitting. Mach. Learn. Res., 15(1):1929-195 2014.
12

---

[43] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proc. of the 57th Conference of the Association for Computational Linguistics, ACI, Florence, Italy, July 28 August 2, Volume I: Long Papers, pages 3645-3650. Association for Computational Linguisties, 2019.
[44] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the Inception architecture for computer vision. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR. Las Vegas, NV, USA, June 27-30, pages 2818 2826. IEEE Computer Socicty, 2016.
[45] Jafar Tanha, Maarten van Someren, and Hamideh Afsarmanesh. Semi-supervised self-training for decision tree classifiers. Int. J. Mach. Learn. Cybern., 8(1):355-370, 2017.
[46] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Neight-average consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, NeurlPS, Long Beach, CA, USA, December 4-9, pages 199--1204, 2017.
[47] Trung Quang Tran, Mingu Kang, and Daeyoung Kim. ReRankMatch: cemi-supervised learning with semantics-oriented similarity representation. CoRR,
2021. [48] Jesper E. van Engelen and Holger H. Hoos. A survey semi-supervised learning. Mach. Learn., 109(():373 440, 2020.
[49] Xiao Wang, Daisuke Kihara, Jiebo Luo, and Guo-Jun Qi. EΛAET: A self-trained framework for semi-supervised and supervised learning with ensemble transformations. IEEE Trans. Image Process., 30:1639-1647, 2021.
[50] Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoe Le. Unsupervised data augmentation for consistency training. In Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NewrlPS, virtual, December 6-12, 2020.
[51] Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoc V. Le. Self Self-training with Noisy Student improves ImageNet classification. 191::2191 [cs, stat], abs/1911.04255 2020.
[52] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proc. of the British Machine Vision Conference, BMVC, York, UK, September 19-22, 2016.
[53] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimization. In Proc. of the 6th International Conference on Learning Repre- sentations, ICLR, Vancouver, BC, Canada, April 30 May 3. OpenReview.net, 2018.
[54] Zhedong Zheng and Yi Yang. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. Int. J. Comput. Vis., (29(4):106-111) 2021.
[55] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Time-consistent self-supervision for semi- supervised learning. In Proc. of the 37th International Conference Machine Learning, ICML, virtual, July 13-18, volume 119 of Proceedings of Machine Learning Research, pages 11523-11533. PMLR, 2020.
13


Reviews:
[{"review_id": "wgGTi3nqk_M", "reviewer": "Reviewer_TVWJ", "summary": "For the task of semi-supervised learning, this work proposes the use of credal sets to model uncertainty in pseudo-labels. By proposing simple scheme for generating and learning from these credal sets, the authors show how the FixMatch baseline can be extended and improved in settings where the number of labeled examples is small. Furthermore, they show that this approach can improve the efficiency of this semi-supervised learning approach and reduce calibration errors. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors proposed used of credal sets to model uncertainty in the pseudo-labeling pipeline of semi-supervised learning is original, well-motivated, and demonstrated to be effective. The authors clearly motivate their approach and through description and accompanying pseudo code present the work in a manner than can be replicated and re-implemented. As both semi-supervised learning and uncertainty quantification are important and timely areas of interest within the machine learning community, this work has significance based on its ability to successfully combine important approaches in both areas. \n\nThe paper seems to lack ablative experiments around various aspects of CSSL, such as exponential moving averaging, which would help clarify which portions of the proposed approach contributed to the improved performance the most significantly.\n\nThe paper is well-written, addresses an important shortcoming of existing ssl approaches (efficiency and generalization) a and presents a way to improve upon competitive baselines. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss limitation or societal impacts at all in their work, this is unfortunate as it represents and area where the authors could point to the importance of their work in various real-world settings.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "IrG624hJxkj", "reviewer": "Reviewer_jEBP", "summary": "This work studied the credal sets, i.e., sets of probability distributions, in self-supervised learning. The author claimed that the increased expressiveness can improve performance and reduce biases. The proposed method is evaluated empirically on the image classification task.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\n\nThe use of such credal probability and so-called \"possibility distributions\" seems rare in machine learning. The proposed method for learning from credal labels is based on the minimal value of the KL-divergence between probability in the set and the prediction, which originated from existing work (optimistic superset loss, infimum loss). The author then used credal labels in self-supervised learning with FixMatch. \n\nTo the best of my knowledge, although its components are from existing work (credal label, infimum loss, FixMatch), the application of learning from credal labels in self-supervised learning is new.\n\n---\n\n## Soundness\n\nMy major concern is about the usefulness of the concept of credal sets in this setting. Indeed, hard labels may cause problems in neural network training. The author suggested that soft labels, probability distributions, are still not enough. However, the evidence is not convincing enough.\n\nIn traditional statistical learning, we want to learn a decision function that outputs labels, where both training data and prediction are deterministic (\"hard label\"). Since it's NP-hard to optimize it, we can resort to probabilistic models that output probabilities of labels, where the training data is still deterministic. It is possible because the expectation over the empirical distribution converges to the expectation over the true distribution as the size of the sample increases. There are a lot of attempts to let the model provide more uncertainty measures using, for example, Bayesian methods. The supervision may be still deterministic.\n\nThe direction of this paper is different from the ones above. Here, the uncertainty of the supervision is increased, from labels to probabilities of labels, to sets of probabilities of labels. Meanwhile, if I understand correctly, the output is still a single-point probability. My concern is, is the same effect can be achieved by, for example, regularizations of the weight or the output? Another straightforward approach would be using distributions over distributions of categorical labels, such as Dirichlet distribution, as the format of supervision. What's the advantage of the proposed method using credal labels?\n\n---\n\n## Technical quality\n\nAnother issue is that this method is verified solely empirically. Theoretical characterization is limited or from existing work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author discussed some limitations of this work through experiments in Section 4.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "6yGyW3uVUYx", "reviewer": "Reviewer_qetJ", "summary": "Unlabelled samples are labelled with credal sets (set of probability distributions), better capturing the 'unknowledge' about the label.\n\nThe methodology integrates seamlessly several state-of-the-art ideas to achieve top performance in the experimental evaluation.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and technically sound.\nThe novelty of the paper is enough.\nAlthough the improvements in accuracy are not impressive, the contribution can be of interest to a wide range of NeurIPS participants.\n\nThe ablation study could include an analysis of the importance of the weighting in Eq (6) to the final results. What would be the results with the straightforward alphas?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "0v4n-ONPLFB", "reviewer": "Reviewer_AL5C", "summary": "This paper proposes to modify the pseudo-labeling procedure in semi-supervised learning. Student model predictions are penalized by distance to a credal set of teacher predictions. The size of credal sets is determined heuristically. Experiments demonstrate improvement over baselines mainly in scarce-label scenarios.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The central idea is clear, and the experiments demonstrate improvements over the SOTA method Fixmatch. However, there are a few weaknesses / questions:\n\n1. The techniques seem like empirical hacks and lacks theoretical support. Can the authors demonstrate the advantage of the proposed method on a simple data generative model?\n2. No ablations on the choice of $\\alpha_i$. Equation 6 seems especially ad hoc and requires justification.\n3. An important missing baseline is 'Sinkhorn Label Allocation: Semi-Supervised Classification via Annealed Self-Training' by Tai et al.\n4. It is not clear to me why distributional matching doesn't work, whereas credal self-training does.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the discussion, the reviewers agree that the paper proposes an interesting approach of using credal sets as a generalization of using probability estimates in self-supervision approaches. The resulting loss function is relatively simple and the authors describe one practical way to implement it. 

The paper is still borderline because the experimental results, even though they seem slightly better than the initial FixMatch approach, are not significantly better than other baselines, and additional ablations could be performed regarding some design choices, such as the effect of the weighting in Eq 6 or the exponential moving average of model weights. These ablations would be good to have, but since these come from previous work on the topic and the baselines are included in the main table, I do not feel that these are decisive arguments for rejection, even though they would be good to have. 

Overall, the approach proposed in the paper provides a cleaner way of using pseudo-labeling than the confidence matching, which is illustrated for instance in section on efficiency (learning curves might be better than a fixed number of epochs here), where we see the advantage of not requiring the confidence threshold as additional hyperparameter. In that respect, the approach can be used in practice even if it does not significantly improves performance compared to the optimally tuned competitors.
Decision: Accept (Poster)