Credal Self-Supervised Learning
Year: 2021

Abstract:
Self-training is an effective approach to semi-supervised learning. The key idea is to let the learner itself iteratively generate “pseudo-supervision” for unlabeled instances based on its current hypothesis. In combination with consistency regularization, pseudo-labeling has shown promising performance in various domains, for example in computer vision. To account for the hypothetical nature of the pseudolabels, these are commonly provided in the form of probability distributions. Still, one may argue that even a probability distribution represents an excessive level of informedness, as it suggests that the learner precisely knows the ground-truth conditional probabilities. In our approach, we therefore allow the learner to label instances in the form of credal sets, that is, sets of (candidate) probability distributions. Thanks to this increased expressiveness, the learner is able to represent uncertainty and a lack of knowledge in a more flexible and more faithful manner. To learn from weakly labeled data of that kind, we leverage methods that have recently been proposed in the realm of so-called superset learning. In an exhaustive empirical evaluation, we compare our methodology to state-of-the-art self-supervision approaches, showing competitive to superior performance especially in low-label scenarios incorporating a high degree of uncertainty.

heading: 1 Introduction
text: Recent progress and practical success in machine learning, especially in deep learning, is largely due to an increased availability of data. However, even if data collection is cheap in many domains, labeling the data so as to make it amenable to supervised learning algorithms might be costly and often comes with a significant effort. As a consequence, many data sets are only partly labeled, i.e., only a few instances are labeled while the majority is not. This is the key motivation for semi-supervised learning (SSL) methods [7], which seek to exploit both labeled and unlabeled data simultaneously.
As one simple yet effective methodology, so-called self-training [26], often also referred to as pseudolabeling, has proven effective in leveraging unlabeled data to improve over training solely on labeled data. The key idea is to let the learner itself generate “pseudo-supervision” for unlabeled instances based on its own current hypothesis. In the case of probabilistic classifiers, such pseudo-targets are usually provided in the form of (perhaps degenerate) probability distributions. Obviously, since pseudo-labels are mere guesses and might be wrong, this comes with the danger of biasing the learning process, a problem commonly known as confirmation bias [46]. Therefore, self-training is nowadays typically combined with additional regularization means, such as consistency regularization [2, 40], or additional uncertainty-awareness [38].
Labeling an instance x with a probability distribution on the target space Y is certainly better than committing to a precise target value, for example a single class label in classification, as the latter would suggest a level of conviction that is not warranted. Still, one may argue that even a probability distribution represents an excessive level of informedness. In fact, it actually suggests that the learner
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
precisely knows the ground-truth conditional probability p(y |x). In our approach, we therefore allow the learner to label instances in the form of credal sets, that is, sets of (candidate) probability distributions [27]. Thanks to this increased expressiveness, the learner is able to represent uncertainty and a lack of knowledge about the true label (distribution) in a more flexible and more faithful manner. For example, by assigning the biggest credal set consisting of all probability distributions, it is able to represent complete ignorance — a state of knowledge that is arguably less well represented by a uniform probability distribution, which could also be interpreted as full certainty about this distribution being the ground truth. Needless to say, this ability is crucial to avoid a confirmation bias and account for the heteroscedastic nature of uncertainty, which varies both spatially (i.e., among different regions in the instance space) and temporally: typically, the learner is less confident in early stages of the training process and becomes more confident toward the end.
Existing methods are well aware of such problems but handle them in a manner that is arguably ad-hoc. The simple yet effective SSL framework FixMatch [41], for instance, applies a thresholding technique to filter out presumably unreliable pseudo-labels, which often results in unnecessarily delayed optimization, as many instances are considered only lately in the training. Other approaches, such as MixUp [53], also apply mixing strategies to learn in a more cautious manner from pseudolabels [1, 3, 4]. Moreover, as many of these approaches, including FixMatch, rely on the principle of entropy minimization [19] to separate classes well, the self-supervision is generated in the form of rather peaked or even degenerate distributions to learn from, which amplifies the problems of confirmation bias and over-confidence [33].
An important implication of credal pseudo-labeling is the need for extending the underlying learning algorithm, which must be able to learn from weak supervision of that kind. To this end, we leverage the principle of generalized risk minimization, which has recently been proposed in the realm of so-called superset learning [22]. This approach supports the idea of data disambiguation: The learner is free to (implicitly) choose any distribution inside a credal set that appears to be most plausible in light of the other data (and its own learning bias). Thus, an implicit trade-off between cautious learning and entropy minimization can be realized: Whenever it seems reasonable to produce an extreme distribution, the learner is free but not urged to do so. Effectively, this not only reduces the risk of a potential confirmation bias due to misleading or over-confident pseudo-labels, it also allows for incorporating all unlabeled instances in the learning process from the beginning without any confidence thresholding, leading to a fast and effective semi-supervised learning method.
To prove the effectiveness of this novel type of pseudo-labeling, we proceed from FixMatch as an effective state-of-the-art SSL framework and replace conventional probabilistic pseudo-labeling by a credal target set modeling. In an exhaustive empirical evaluation, we study the effects of this change compared to both hard and soft probabilistic target modeling, as well as measuring the resulting network calibration of induced models to reflect biases. Our experiments not only show competitive to superior generalization performance, but also better calibrated models while cutting the time to train the models drastically.

heading: 2 Related Work
text: In semi-supervised learning, the goal is to leverage the potential of unlabeled in addition to labeled data to improve learning and generalization. As it constitutes a broad research field with a plethora of approaches, we will focus here on classification methods as these are most closely related to our approach. We refer to [7] and [48] for more comprehensive overviews.
As one of the earliest ideas to incorporate unlabeled data in conventional supervised learning, selftraining has shown remarkably effective in various domains, including natural language processing [12] and computer vision [11, 17, 39]. The technique is quite versatile and can be applied for different learning methods, ranging from support vector machines [30] to decision trees [45] and neural networks [35]. It can be considered as the basic training pattern in distillation, such as self-distillation from a model to be trained itself [23] or within student-teacher settings [36, 51]. Recently, this technique also lifted completely unsupervised learning in computer vision to a new level [6, 20].
As a common companion of self-training for classification, especially in computer vision, consistency regularization is employed to ensure similar model predictions when facing multiple perturbed versions of the same input [2, 37, 40], resulting in noise-robustness as similarly achieved by other (stochastic) ensembling methods such as Dropout [42]. Strong augmentation techniques used in this
regard, e.g., CTAugment [3] or RandAugment [9], allow one to learn from instances outside of the (hitherto labeled) data distribution and, thus, lead to more accurate models [10]. For semi-supervised learning in image classification, the combination of consistency regularization with pseudo-labeling is widely adopted and has proven to be a simple yet effective strategy [1, 4, 25, 37, 41, 50, 55].
As pseudo-labeling comprises the risk of biasing the model by wrong predictions, especially when confidence is low, uncertainty awareness has been explicitly considered in the generic self-supervision framework UPS to construct more reliable targets [38]. Within their approach, the model uncertainty is estimated by common Bayesian sampling techniques, such as MC-Dropout [15] or DropBlock [16], which is then used to sort out unlabeled instances for which the model provides uncertain predictions. Related to this, the selection of pseudo-labels based on the model certainty has also been used in specific domains, such as text classification [32] or semantic segmentation [54].

heading: 2.1 FixMatch
text: As already mentioned, FixMatch [41] combines recent advances in consistency regularization and pseudo-labeling into a simple yet effective state-of-the-art SSL approach. It will serve as a basis for our new SSL method, as it provides a generic framework for fair comparisons between conventional and our credal pseudo-labeling.
In each training iteration, FixMatch considers a batch of B labeled instances Bl = {(xi, pi)}Bi=1 ⊂ X × P(Y) and µB unlabeled instances Bu = {xi}µBi=1 ⊂ X , where X denotes the input feature space, Y the set of possible classes, P(Y) the set of probability distributions over Y , and µ ≥ 1 the multiplicity of unlabeled over labeled instances in each batch. Here, the probabilistic targets in Bl are given as degenerate “one-hot” distributions. When aiming to induce probabilistic classifiers of the form p̂ : X −→ P(Y), FixMatch distinguishes between two forms of augmentation: While As : X −→ X describes “strong” augmentations that perturbs the image in a drastic manner by combining multiple operations, simple flip-and-shift transformations are captured by “weak” augmentations Aw : X −→ X . As an iteration-wise loss to determine gradients, a combination of the labeled loss Ll and unlabeled loss Lu is calculated. For the former, the labeled input instances from Bl are weakly augmented and used in a conventional cross-entropy loss H : P(Y)2 −→ R. For the latter, the model prediction q := p̂(Aw(x)) on a weakly-augmented version of each unlabeled instance x is used to construct a (hard) pseudo-label q̃ ∈ P(Y) when meeting a predefined confidence threshold τ . While q̃ is in FixMatch a degenerate probability distribution by default, one could also inject soft probabilities, which, however, turned out to be less effective (cf. [41]). The pseudo-label is then compared to a strongly-augmented version of the same input image. Hence, the unlabeled loss Lu is given by
Lu := 1
µB ∑ x∈Bu Imax q≥τH(q̃, p̂(As(x))) .

heading: 3 Credal Self-Supervised Learning
text: In this section, we introduce our credal self-supervised learning (CSSL) framework for the case of classification, assuming the target to be categorical with values in Y = {y1, . . . , yK}. Before presenting more technical details, we start with a motivation for the credal (set-valued) modeling of target values and a sketch of the basic idea of our approach.

heading: 3.1 Motivation and Basic Idea
text: In supervised learning, we generally assume the dependency between instances x ∈ X and associated observations (outcomes) to be of stochastic nature. More specifically, one typically assumes a “ground-truth” in the form of a conditional probability distribution p∗(· |x) ∈ P(Y). Thus, for every y ∈ Y , p∗(y |x) is the probability to observe y as a value for the target variable in the context x. Ideally, the distribution p∗(· |x) would be provided as training information to the learner, along with every training instance x. In practice, however, supervision comes in the form of concrete values of the target, i.e., a realization y ∈ Y of the random variable Y ∼ p∗(· |x), and the corresponding degenerate distribution py assigning probability mass 1 to y (i.e., py(y |x) = 1 and py(y′ |x) = 0 for y′ 6= y) is taken as a surrogate for p∗.
Obviously, turning the true distribution p∗ = p∗(· |x), subsequently also called a “soft label”, into a more extreme distribution py (i.e., a single value y), referred to as “hard label”, may cause undesirable effects. In fact, it suggests a level of determinism that is not warranted and tempts the learner to over-confident predictions that are poorly calibrated, especially when training with losses such as cross-entropy [33]. So-called label smoothing [44] seeks to avoid such effects by replacing hard labels py with (hypothetical) soft labels p̂ that are “close” to py .
Coming back to semi-supervised learning, training of the learner and self-supervision can be characterized for existing methods such as FixMatch as follows:
• The learner is trained on hard labels, which are given for the labeled instances xl and constructed by the learner itself for unlabeled instances xu. • Construction of (pseudo-)labels is done in two steps: First, the true soft label p∗ = p∗(· |xu) is predicted by a distribution p̂ = p̂(· |xu), and the latter is turned into a hard label p̂y afterward, provided p̂ suggests a sufficient level of certainty (support for y).
This approach can be challenged for several reasons. First, training probabilistic predictors on hard labels comes with the disadvantages mentioned above and tends to bias the learner. Second, pseudo-labels p̂y constructed by the learner tend to be poor approximations of the ground truth p∗. In fact, there will always be a discrepancy between p∗ and its prediction p̂, and this discrepancy is further increased by replacing p̂ with p̂y. Third, leaving some of the instances — those for which the prediction is not reliable enough — completely unlabeled causes a loss of information. Roughly speaking, while a part of the training data is overly precise1, another part remains unnecessarily imprecise and hence unused. This may slow down the training process and cause other undesirable problems such as “path-dependency” (training is influenced by the order of the unlabeled instances).
To avoid these disadvantages, we propose to use soft instead of hard labels as training information for the learner. More specifically, to account for possible uncertainty about a true soft label p∗, we model information about the target in the form of a set Q ⊆ P(Y) of distributions, in the literature on imprecise probability also called a credal set [27]. Such a “credal label” is supposed to cover the ground truth, i.e., p∗ ∈ Q, very much like a confidence interval in statistics is supposed to cover (with high probability) some ground-truth parameter to be estimated.
This approach is appealing, as it enables the learner to model its belief about the ground-truth p∗ in a cautious and faithful manner: Widening a credal label Q may weaken the training information but maintains or even increases its validity. Moreover, credal labeling elegantly allows for covering the original training information as special cases: A hard label y provided for a labeled instance xl corresponds to a singleton set Qy = {py}, and the lack of any information in the case of an unlabeled instance xu is properly captured by taking Q = P(Y). Starting with this information, the idea is to modify it in two directions:
• Imprecisiation: To avoid possible disadvantages of hard labels Qy, these can be made less precise through label relaxation [29], which is an extension of the aforementioned label smoothing. Technically, it means that Qy is replaced by a credal set Q containing, in addition to py itself, distributions p close to py .
• Precisiation: The non-informative and maximally imprecise labels Q = P(Y) for unlabeled instances xu are successively (iteration by iteration) “shrunken” and made more precise. This is done by replacing a set Q with a smaller subset Q′ ⊂ Q, provided the exclusion of certain candidate distributions p ∈ Q is sufficiently supported by the learner.

heading: 3.2 Credal Labeling
text: Credal sets are commonly assumed to be convex, i.e., p, q ∈ Q implies λp+ (1− λ)q ∈ Q for all distributions p, q and λ ∈ (0, 1). In our context, arbitrary (convex) credal sets Q ⊆ P(Y) could in principle be used for the purpose of labeling instances. Yet, to facilitate modeling, we restrict ourselves to credal sets induced by so-called possibility distributions [13].
A possibility or plausibility measure Π is a set-function 2Y −→ [0, 1] that assigns a degree of plausibility Π(Y ) to every subset (event) Y ⊆ Y . Such a measure is induced by a possibility
1To some extent, this can be alleviated through measures such as instance weighing, i.e., by attaching a weight to a pseudo-labeled instances [38].
distribution π : Y −→ [0, 1] via Π(Y ) = maxy∈Y π(y) for all Y ⊆ Y . Interpreting degrees of plausibility as upper probabilities, the set of (candidate) probability distributions p (resp. measures P ) in accordance with π resp. Π is given by those for which P (Y ) ≤ Π(Y ) for all events Y :
Qπ = { p ∈ P(Y) | ∀Y ⊆ Y : P (Y ) = ∑ y∈Y p(y) ≤ max y∈Y π(y) = Π(Y ) } . (1)
Roughly speaking, for each y ∈ Y , the possibility π(y) determines an upper bound for p∗(y), i.e., the highest probability that is deemed plausible for y. Note that, to guarantee Qπ 6= ∅, possibility distributions must be normalized in the sense that maxy∈Y π(y) = 1. In other words, there must be at least one outcome y ∈ Y that is deemed completely plausible. In our context, this outcome is naturally taken as the label y = argmaxy′∈Y p̂(y
′) with the highest predicted probability. A simple way of modeling then consists of controlling the degree of imprecision (ignorance of the learner) by a single parameter α ∈ [0, 1], considering credal sets of the form
Qαy = { p ∈ P(Y) | p(y) ≥ 1− α } . (2)
Thus, Qαy consists of all distributions p that allocate a probability mass of at least 1 − α to y and hence at most α to the other labels Y \{y}. As important special cases we obtain Q0y = {py}, i.e., the degenerate distribution that assigns probability 1 to the label y, and Q1y = P(Y) modeling complete ignorance about the ground truth p∗.
Needless to say, more sophisticated credal sets could be constructed on the basis of a distribution p̂, for example leveraging the concept of probability-possibility transformations [14]. Yet, to keep the modeling as simple as possible, we restrict ourselves to sets of the form (2) in this work.

heading: 3.3 Learning from Credal Labels
text: Our approach requires the learner to be able to learn from credal instead of probabilistic or hard labels. To this end, we refer to the generic approach to so-called superset learning as proposed in [22]. Essentially, this approach is based on minimizing a generalization L∗ of the original (probabilistic) loss L : P(Y)2 −→ R. More specifically, the so-called optimistic superset loss [22], also known as infimum loss [5], compares credal sets with probabilistic predictions as follows:
L∗(Q, p̂) = min p∈Q L(p, p̂) (3)
For the specific case where credal sets are of the form (2) and the loss L is the Kullback-Leibler divergence DKL, (3) simplifies to
L∗(Qαy , p̂) = {
0 if p̂ ∈ Qαy DKL(p r||p̂) otherwise , (4)
where
pr(y′) =
{ 1− α if y′ = y
α · p̂(y ′)∑
y′′ 6=y p̂(y ′′) otherwise
(5)
is the projection of the prediction p̂ onto the boundary of Q. This loss has been proven to be convex, making its optimization practically feasible [29].
The loss (3) is an optimistic generalization of the original loss in the sense that it corresponds to the loss L(p, p̂) for the most favorable instantiation p ∈ Q. This optimism is motivated by the idea of data disambiguation [22] and can be justified theoretically [5]. Roughly speaking, minimizing the sum of generalized losses over all training examples (xi, Qi) comes down to (implicitly) choosing a precise probabilistic target inside every credal set, i.e., replacing (xi, Qi) by (xi, pi) with pi ∈ Qi, in such a way that the original loss (empirical risk) can be made as small as possible.

heading: 3.4 Credal Self-Supervised Learning Framework
text: Our idea of credal self-supervised learning (short CSSL) offers a rather generic framework for designing self-supervised learning methods. In this paper, we focus on image classification as a concrete and practically relevant application, and combine CSSL with the consistency regularization framework provided by FixMatch (cf. Fig. 1).
To describe the algorithm, we again assume batches of labeled Bl and unlabeled instances Bu (cf. Section 2.1). For the former, we measure the labeled loss Ll in terms of the cross-entropy H between the observed target distributions of the (labeled) training instances and the current model predictions p̂ on weakly-augmented features. At this point, one could apply the idea of imprecisiation through label relaxation as motivated in Section 3.1. However, as this work focuses on the self-supervision part, i.e., on Bu rather than Bl, we stick to hard labels. This also facilitates the interpretation of experimental results later on, as it avoids the mixing of different effects.
For the unlabeled instances xi ∈ Bu, we follow the consistency regularization idea of FixMatch and use the predictions on weakly augmented versions of the unlabeled instances as a reference for the target set construction. More precisely, we take the class yi = argmaxy∈Y p̂i(y) of the current model prediction p̂i = p̂(Aw(xi))) as a reference to construct instance-wise targets Qαiyi as specified in (2).
A rather straightforward approach to determining the uncertainty level αi is to set αi = 1− p̂i(yi). Yet, motivated by the idea of distribution alignment as proposed in [3], we suggest to weight the predictions p̂i by the proportion of the class prior p̃ ∈ P(Y) and a moving average of the last model predictions p̄ ∈ P(Y), so that αi = 1− qi(y)/ ∑ y′∈Y qi(y
′) with the weighted (pseudo-)probability scores
qi(y) = p̂i(y ′)× p̃(y
′)
p̄(y′) . (6)
According to this way of modeling pseudo-labels in terms of credal sets, the size (imprecision) of a set is in direct correspondence with the confidence of the learner. Therefore, this approach leads to some sort of (implicit) disambiguation: With increasing confidence, the target sets are becoming more precise, successively fostering entropy minimization without imposing overly constrained targets. Also, as mentioned before, this approach allows for using all instances for training from the very beginning, without losing any of them due to confidence thresholding. The weighting mechanism based on the class prior p̃ and the prediction history p̄ (second factor on the right-hand side of (6)) accounts for the consistency and hence the confidence in a particular class prediction. If p̃(y′) p̄(y′), the label y′ is under-represented in the past predictions, suggesting that its true likelihood might be higher than predicted by the learner, and vice versa in the case where p̃(y′) p̄(y′). As this procedure is simple and computationally efficient, it facilitates the applicability compared to computationally demanding uncertainty methods such as MC-Dropout.
The proposed target sets are then used within the unlabeled loss Lu according to (4), which is used in addition to the labeled loss Ll. Hence, the final loss is given by
L = 1 |Bl| ∑ (xi,pi)∈Bl
H(pi, p̂i)︸ ︷︷ ︸ Ll
+λu 1 |Bu| ∑
xi∈Bu L∗(Qαiyi , p̂i)︸ ︷︷ ︸ Lu . (7)
The pseudo-code of the complete algorithm can be found in the appendix.
We conclude this section with a few remarks on implementation details. Since we are building upon FixMatch, we keep the same augmentation policy as suggested by the authors. Thus, we employ CTAugment by default. We refer to the appendix for further ablation studies, including RandAugment as augmentation policy. Moreover, we consider the same optimization algorithm as used before, namely SGD with Nesterov momentum, for which we use cosine annealing as learning rate schedule [31]. Similar to FixMatch, we set the learning rate to η cos 7πk16K , where η is the initial learning rate, k the current training step and K the total number of steps (220 by default). As we are keeping the algorithmic framework the same and do not require any form of confidence thresholding, we can reduce the number of parameters compared to FixMatch, which further facilitates the use of this approach. We also use an exponential moving average of model parameters for our final model, which comes with appealing ensembling effects that typically improve model robustness and has been considered by various recent approaches for un- or semi-supervised learning [6, 41].

heading: 4 Experiments
text: To compare our idea of credal pseudo-labeling, we conduct an exhaustive empirical evaluation with common image classification benchmarks. More precisely, we follow the semi-supervised learning evaluation setup as described in [41] and perform experiments on CIFAR-10/-100 [24], SVHN [34], and STL-10 [8] with varying fractions of labeled instances sampled from the original data sets, also considering label-scarce settings with only a few labels per class. For CIFAR-10, SVHN, and STL-10, we train a Wide ResNet-28-2 [52] with 1.49 M parameters, while we consider Wide ResNet-28-8 (23.4 M parameters) models for the experiments on CIFAR-100. To guarantee a fair comparison to existing methods related to FixMatch, we keep the hyperparameters the same as in the original experiments. We refer to the appendix for a more comprehensive overview of the experimental details. We repeat each run 5 times with different seeds for a higher significance and average the results for the model weights of the last 20 epochs as done in [41].
As baselines, we report the results for Mean Teacher [46], MixMatch [4], UDA [50], ReMixMatch [3], and EnAET [49] as state-of-the art semi-supervised learning methods. Moreover, as we directly compete against the probabilistic hard-labeling employed in FixMatch, we compare our approach to this (with CTAugment) and related methods, namely AlphaMatch [18], CoMatch [28], and ReRankMatch [47]. Since these approaches extend the basic framework of FixMatch by additional means (e.g., loss augmentations), the direct comparison with FixMatch is maximally fair under these conditions, avoiding side-effects as much as possible. In addition, we show the results of the student-teacher method Meta Pseudo Labels (Meta PL) [36].

heading: 4.1 Generalization Performance
text: In the first experiments, we train the aforementioned models on the four benchmark data sets for different numbers of labeled instances to measure the generalization performance of the induced models. The results are provided in Table 1.
As can be seen, CSSL is especially competitive when the number of labels is small, showing that the implicit uncertainty awareness of set-based target modeling becomes effective. But CSSL is also able to provide compelling performance in the case of relatively many labeled instances, although not substantially improving over conventional hard pseudo-labeling. For instance, it is approximately on par with state-of-the-art performance on CIFAR-100 and SVHN. Focusing on the comparison to FixMatch, credal self-supervision improves the performance in almost all cases over hard pseudolabeling with confidence thresholding, providing further evidence for the adequacy of our method.

heading: 4.2 Network Calibration
text: In a second study, we evaluate FixMatch-based models in terms of network calibration, i.e., the quality of the predicted class probabilities. For this purpose, we calculate the expected calibration error (ECE) as done in [21] using discretized probabilities into 15 bins. The calibration errors provide insight into the bias of the models induced by the different learning methods.
Besides the “raw” hard-labeling, we also consider the distribution alignment (DA)-variant of FixMatch (as described in [41]), as well as an adaptive variant using label smoothing [44], which we dub LSMatch (see appendix for an algorithmic description). For label smoothing, we use a uniform distribution policy and calculate the distribution mass parameter α in an adaptive manner as we do for CSSL (cf. Section 3.4). As a result, LSMatch can be regarded as the natural counterpart of our approach for a more cautious learning using less extreme targets. We refer to [29] for a more thorough analysis of the differences between smoothed and credal labels. Since both methods realize an implicit calibration [29, 33], we omit explicit calibration methods that require additional data. Besides, we experiment with an uncertainty-filtering variant of FixMatch following UPS [38], for which we provide results in the supplement.
In accordance with the studies in [29], the results provided in Table 2 show improved calibration compared to classical probabilistic modeling. As these effects are achieved without requiring an additional calibration data split, it provides an appealing method to induce well calibrated and generalizing models. Nevertheless, the margin to the other baselines gets smaller with an increasing number of labeled instances, which is plausible as it implies an increased level of certainty.

heading: 4.3 Efficiency
text: In addition to accuracy, a major concern of learning algorithms is run-time efficiency. In this regard, we already noted that thresholding mechanisms may severely delay the incorporation of unlabeled instances in the learning process. For example, for a data set with 226 images, FixMatch requires up to 220 updates to provide the competitive results reported. This not only excludes potential users without access to computational resources meeting these demands, it also consumes a substantial amount of energy [43].
As described before, CSSL allows for incorporating all instances from the very beginning. To measure the implied effects, i.e., a faster convergence, apart from using more effective optimizers, we train the models from the former experiment on CIFAR-10 and SVHN for only an eighth and thirty-second of the original number of epochs, respectively. As we would grant CSSL and LSMatch an unfair advantage compared to confidence thresholding in FixMatch, we experiment with different thresholds τ ∈ {0, 0.8, 0.95} for FixMatch. The (averaged) learning curves are provided in the supplement. As shown by the results in Table 3, CSSL achieves the best performance in the label-scarce cases, which confirms the adequacy of incorporating all instances in a cautious manner from the very beginning. Likewise, LSMatch shows competitive performance following the same intuition. In contrast, FixMatch with a high confidence threshold shows slow convergence, which can be mitigated by lowering the thresholding. However, FixMatch with τ = 0.0 provides unsatisfying performance as it drastically increases the risk of confirmation biases; τ = 0.8 seems to be a reasonable trade-off between convergence speed and validity of the model predictions. Nevertheless, when providing higher numbers of labels, the performance gain in incorporating more instances from early on is fairly limited, making the concern of efficiency arguably less important in such cases.

heading: 5 Conclusion
text: Existing (probabilistic) methods for self-supervision typically commit to single probability distributions as pseudo-labels, which, as we argued, represents the uncertainty in such labels only insufficiently and comes with the risk of incorporating an undesirable bias. Therefore, we suggest to allow the learner to use credal sets, i.e., sets of (candidate) distributions, as pseudo-labels. In this way, a more faithful representation of the learner’s (lack of) knowledge about the underlying ground truth distribution can be achieved, and the risk of biases is reduced. By leveraging the principle of generalized risk minimization, we realize an iterative disambiguation process that implements an implicit trade-off between cautious self-supervision and entropy minimization.
In an exhaustive empirical evaluation in the field of image classification, the enhanced expressiveness and uncertainty-awareness compared to conventional probabilistic self-supervision proved to yield superior generalization performance, especially in the regime of label-scarce semi-supervised learning. Moreover, the experiments have shown an improved network calibration when trained with credal self-supervision, as well as an increased efficiency when considering small compute budgets for training.
Motivated by these promising results, we plan to further elaborate on the idea of credal target modeling and extend it in various directions. For example, as we considered rather simple target sets so far, a thorough investigation of more sophisticated modeling techniques should be conducted. Such techniques may help, for instance, to sift out implausible classes early on, and lead to more precise pseudo-labels without compromising validity. Besides, as already mentioned, our CSSL framework is completely generic and not restricted to applications in image processing. Although we used it to extend FixMatch in this paper, it can extend any other self-training method, too. Elaborating on such extensions is another important aspect of future work.
Acknowledgments and Disclosure of Funding
This work was supported by the German Research Foundation (DFG) (Grant No. 420493178). Moreover, the authors gratefully acknowledge the funding of this project by computing time provided by the Paderborn Center for Parallel Computing (PC2) and the research group of Prof. Dr. Marco Platzner.

Review summaries:
This paper proposes to modify the pseudo-labeling procedure in semi-supervised learning. Student model predictions are penalized by distance to a credal set of teacher predictions. The size of credal sets is determined heuristically. Experiments demonstrate improvement over baselines mainly in scarce-label scenarios.

For the task of semi-supervised learning, this work proposes the use of credal sets to model uncertainty in pseudo-labels. By proposing simple scheme for generating and learning from these credal sets, the authors show how the FixMatch baseline can be extended and improved in settings where the number of labeled examples is small. Furthermore, they show that this approach can improve the efficiency of this semi-supervised learning approach and reduce calibration errors. 

This work studied the credal sets, i.e., sets of probability distributions, in self-supervised learning. The author claimed that the increased expressiveness can improve performance and reduce biases. The proposed method is evaluated empirically on the image classification task.