Credal Self-Supervised Learning

1 Introduction . Recent progress and practical success in machine learning , especially in deep learning , is largely due to an increased availability of data . However , even if data collection is cheap in many domains , labeling the data so as to make it amenable to supervised learning algorithms might be costly and often comes with a significant effort . As a consequence , many data sets are only partly labeled , i.e. , only a few instances are labeled while the majority is not . This is the key motivation for semi-supervised learning ( SSL ) methods [ 7 ] , which seek to exploit both labeled and unlabeled data simultaneously . As one simple yet effective methodology , so-called self-training [ 26 ] , often also referred to as pseudolabeling , has proven effective in leveraging unlabeled data to improve over training solely on labeled data . The key idea is to let the learner itself generate “ pseudo-supervision ” for unlabeled instances based on its own current hypothesis . In the case of probabilistic classifiers , such pseudo-targets are usually provided in the form of ( perhaps degenerate ) probability distributions . Obviously , since pseudo-labels are mere guesses and might be wrong , this comes with the danger of biasing the learning process , a problem commonly known as confirmation bias [ 46 ] . Therefore , self-training is nowadays typically combined with additional regularization means , such as consistency regularization [ 2 , 40 ] , or additional uncertainty-awareness [ 38 ] . Labeling an instance x with a probability distribution on the target space Y is certainly better than committing to a precise target value , for example a single class label in classification , as the latter would suggest a level of conviction that is not warranted . Still , one may argue that even a probability distribution represents an excessive level of informedness . In fact , it actually suggests that the learner 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . precisely knows the ground-truth conditional probability p ( y |x ) . In our approach , we therefore allow the learner to label instances in the form of credal sets , that is , sets of ( candidate ) probability distributions [ 27 ] . Thanks to this increased expressiveness , the learner is able to represent uncertainty and a lack of knowledge about the true label ( distribution ) in a more flexible and more faithful manner . For example , by assigning the biggest credal set consisting of all probability distributions , it is able to represent complete ignorance — a state of knowledge that is arguably less well represented by a uniform probability distribution , which could also be interpreted as full certainty about this distribution being the ground truth . Needless to say , this ability is crucial to avoid a confirmation bias and account for the heteroscedastic nature of uncertainty , which varies both spatially ( i.e. , among different regions in the instance space ) and temporally : typically , the learner is less confident in early stages of the training process and becomes more confident toward the end . Existing methods are well aware of such problems but handle them in a manner that is arguably ad-hoc . The simple yet effective SSL framework FixMatch [ 41 ] , for instance , applies a thresholding technique to filter out presumably unreliable pseudo-labels , which often results in unnecessarily delayed optimization , as many instances are considered only lately in the training . Other approaches , such as MixUp [ 53 ] , also apply mixing strategies to learn in a more cautious manner from pseudolabels [ 1 , 3 , 4 ] . Moreover , as many of these approaches , including FixMatch , rely on the principle of entropy minimization [ 19 ] to separate classes well , the self-supervision is generated in the form of rather peaked or even degenerate distributions to learn from , which amplifies the problems of confirmation bias and over-confidence [ 33 ] . An important implication of credal pseudo-labeling is the need for extending the underlying learning algorithm , which must be able to learn from weak supervision of that kind . To this end , we leverage the principle of generalized risk minimization , which has recently been proposed in the realm of so-called superset learning [ 22 ] . This approach supports the idea of data disambiguation : The learner is free to ( implicitly ) choose any distribution inside a credal set that appears to be most plausible in light of the other data ( and its own learning bias ) . Thus , an implicit trade-off between cautious learning and entropy minimization can be realized : Whenever it seems reasonable to produce an extreme distribution , the learner is free but not urged to do so . Effectively , this not only reduces the risk of a potential confirmation bias due to misleading or over-confident pseudo-labels , it also allows for incorporating all unlabeled instances in the learning process from the beginning without any confidence thresholding , leading to a fast and effective semi-supervised learning method . To prove the effectiveness of this novel type of pseudo-labeling , we proceed from FixMatch as an effective state-of-the-art SSL framework and replace conventional probabilistic pseudo-labeling by a credal target set modeling . In an exhaustive empirical evaluation , we study the effects of this change compared to both hard and soft probabilistic target modeling , as well as measuring the resulting network calibration of induced models to reflect biases . Our experiments not only show competitive to superior generalization performance , but also better calibrated models while cutting the time to train the models drastically . 2 Related Work . In semi-supervised learning , the goal is to leverage the potential of unlabeled in addition to labeled data to improve learning and generalization . As it constitutes a broad research field with a plethora of approaches , we will focus here on classification methods as these are most closely related to our approach . We refer to [ 7 ] and [ 48 ] for more comprehensive overviews . As one of the earliest ideas to incorporate unlabeled data in conventional supervised learning , selftraining has shown remarkably effective in various domains , including natural language processing [ 12 ] and computer vision [ 11 , 17 , 39 ] . The technique is quite versatile and can be applied for different learning methods , ranging from support vector machines [ 30 ] to decision trees [ 45 ] and neural networks [ 35 ] . It can be considered as the basic training pattern in distillation , such as self-distillation from a model to be trained itself [ 23 ] or within student-teacher settings [ 36 , 51 ] . Recently , this technique also lifted completely unsupervised learning in computer vision to a new level [ 6 , 20 ] . As a common companion of self-training for classification , especially in computer vision , consistency regularization is employed to ensure similar model predictions when facing multiple perturbed versions of the same input [ 2 , 37 , 40 ] , resulting in noise-robustness as similarly achieved by other ( stochastic ) ensembling methods such as Dropout [ 42 ] . Strong augmentation techniques used in this regard , e.g. , CTAugment [ 3 ] or RandAugment [ 9 ] , allow one to learn from instances outside of the ( hitherto labeled ) data distribution and , thus , lead to more accurate models [ 10 ] . For semi-supervised learning in image classification , the combination of consistency regularization with pseudo-labeling is widely adopted and has proven to be a simple yet effective strategy [ 1 , 4 , 25 , 37 , 41 , 50 , 55 ] . As pseudo-labeling comprises the risk of biasing the model by wrong predictions , especially when confidence is low , uncertainty awareness has been explicitly considered in the generic self-supervision framework UPS to construct more reliable targets [ 38 ] . Within their approach , the model uncertainty is estimated by common Bayesian sampling techniques , such as MC-Dropout [ 15 ] or DropBlock [ 16 ] , which is then used to sort out unlabeled instances for which the model provides uncertain predictions . Related to this , the selection of pseudo-labels based on the model certainty has also been used in specific domains , such as text classification [ 32 ] or semantic segmentation [ 54 ] . 2.1 FixMatch . As already mentioned , FixMatch [ 41 ] combines recent advances in consistency regularization and pseudo-labeling into a simple yet effective state-of-the-art SSL approach . It will serve as a basis for our new SSL method , as it provides a generic framework for fair comparisons between conventional and our credal pseudo-labeling . In each training iteration , FixMatch considers a batch of B labeled instances Bl = { ( xi , pi ) } Bi=1 ⊂ X × P ( Y ) and µB unlabeled instances Bu = { xi } µBi=1 ⊂ X , where X denotes the input feature space , Y the set of possible classes , P ( Y ) the set of probability distributions over Y , and µ ≥ 1 the multiplicity of unlabeled over labeled instances in each batch . Here , the probabilistic targets in Bl are given as degenerate “ one-hot ” distributions . When aiming to induce probabilistic classifiers of the form p̂ : X −→ P ( Y ) , FixMatch distinguishes between two forms of augmentation : While As : X −→ X describes “ strong ” augmentations that perturbs the image in a drastic manner by combining multiple operations , simple flip-and-shift transformations are captured by “ weak ” augmentations Aw : X −→ X . As an iteration-wise loss to determine gradients , a combination of the labeled loss Ll and unlabeled loss Lu is calculated . For the former , the labeled input instances from Bl are weakly augmented and used in a conventional cross-entropy loss H : P ( Y ) 2 −→ R. For the latter , the model prediction q : = p̂ ( Aw ( x ) ) on a weakly-augmented version of each unlabeled instance x is used to construct a ( hard ) pseudo-label q̃ ∈ P ( Y ) when meeting a predefined confidence threshold τ . While q̃ is in FixMatch a degenerate probability distribution by default , one could also inject soft probabilities , which , however , turned out to be less effective ( cf . [ 41 ] ) . The pseudo-label is then compared to a strongly-augmented version of the same input image . Hence , the unlabeled loss Lu is given by Lu : = 1 µB ∑ x∈Bu Imax q≥τH ( q̃ , p̂ ( As ( x ) ) ) . 3 Credal Self-Supervised Learning . In this section , we introduce our credal self-supervised learning ( CSSL ) framework for the case of classification , assuming the target to be categorical with values in Y = { y1 , . . . , yK } . Before presenting more technical details , we start with a motivation for the credal ( set-valued ) modeling of target values and a sketch of the basic idea of our approach . 3.1 Motivation and Basic Idea . In supervised learning , we generally assume the dependency between instances x ∈ X and associated observations ( outcomes ) to be of stochastic nature . More specifically , one typically assumes a “ ground-truth ” in the form of a conditional probability distribution p∗ ( · |x ) ∈ P ( Y ) . Thus , for every y ∈ Y , p∗ ( y |x ) is the probability to observe y as a value for the target variable in the context x . Ideally , the distribution p∗ ( · |x ) would be provided as training information to the learner , along with every training instance x . In practice , however , supervision comes in the form of concrete values of the target , i.e. , a realization y ∈ Y of the random variable Y ∼ p∗ ( · |x ) , and the corresponding degenerate distribution py assigning probability mass 1 to y ( i.e. , py ( y |x ) = 1 and py ( y′ |x ) = 0 for y′ 6= y ) is taken as a surrogate for p∗ . Obviously , turning the true distribution p∗ = p∗ ( · |x ) , subsequently also called a “ soft label ” , into a more extreme distribution py ( i.e. , a single value y ) , referred to as “ hard label ” , may cause undesirable effects . In fact , it suggests a level of determinism that is not warranted and tempts the learner to over-confident predictions that are poorly calibrated , especially when training with losses such as cross-entropy [ 33 ] . So-called label smoothing [ 44 ] seeks to avoid such effects by replacing hard labels py with ( hypothetical ) soft labels p̂ that are “ close ” to py . Coming back to semi-supervised learning , training of the learner and self-supervision can be characterized for existing methods such as FixMatch as follows : • The learner is trained on hard labels , which are given for the labeled instances xl and constructed by the learner itself for unlabeled instances xu . • Construction of ( pseudo- ) labels is done in two steps : First , the true soft label p∗ = p∗ ( · |xu ) is predicted by a distribution p̂ = p̂ ( · |xu ) , and the latter is turned into a hard label p̂y afterward , provided p̂ suggests a sufficient level of certainty ( support for y ) . This approach can be challenged for several reasons . First , training probabilistic predictors on hard labels comes with the disadvantages mentioned above and tends to bias the learner . Second , pseudo-labels p̂y constructed by the learner tend to be poor approximations of the ground truth p∗ . In fact , there will always be a discrepancy between p∗ and its prediction p̂ , and this discrepancy is further increased by replacing p̂ with p̂y . Third , leaving some of the instances — those for which the prediction is not reliable enough — completely unlabeled causes a loss of information . Roughly speaking , while a part of the training data is overly precise1 , another part remains unnecessarily imprecise and hence unused . This may slow down the training process and cause other undesirable problems such as “ path-dependency ” ( training is influenced by the order of the unlabeled instances ) . To avoid these disadvantages , we propose to use soft instead of hard labels as training information for the learner . More specifically , to account for possible uncertainty about a true soft label p∗ , we model information about the target in the form of a set Q ⊆ P ( Y ) of distributions , in the literature on imprecise probability also called a credal set [ 27 ] . Such a “ credal label ” is supposed to cover the ground truth , i.e. , p∗ ∈ Q , very much like a confidence interval in statistics is supposed to cover ( with high probability ) some ground-truth parameter to be estimated . This approach is appealing , as it enables the learner to model its belief about the ground-truth p∗ in a cautious and faithful manner : Widening a credal label Q may weaken the training information but maintains or even increases its validity . Moreover , credal labeling elegantly allows for covering the original training information as special cases : A hard label y provided for a labeled instance xl corresponds to a singleton set Qy = { py } , and the lack of any information in the case of an unlabeled instance xu is properly captured by taking Q = P ( Y ) . Starting with this information , the idea is to modify it in two directions : • Imprecisiation : To avoid possible disadvantages of hard labels Qy , these can be made less precise through label relaxation [ 29 ] , which is an extension of the aforementioned label smoothing . Technically , it means that Qy is replaced by a credal set Q containing , in addition to py itself , distributions p close to py . • Precisiation : The non-informative and maximally imprecise labels Q = P ( Y ) for unlabeled instances xu are successively ( iteration by iteration ) “ shrunken ” and made more precise . This is done by replacing a set Q with a smaller subset Q′ ⊂ Q , provided the exclusion of certain candidate distributions p ∈ Q is sufficiently supported by the learner .

Summary: This paper proposes to modify the pseudo-labeling procedure in semi-supervised learning. Student model predictions are penalized by distance to a credal set of teacher predictions. The size of credal sets is determined heuristically. Experiments demonstrate improvement over baselines mainly in scarce-label scenarios.