Universal Semi-Supervised Learning

1 Introduction . A critical drawback of training a good neural network [ 25 ] is that it typically requires lots of labeled data , which is quite difficult to satisfy due to the unaffordable monetary cost as well as the huge demand for human resources . A popular way to solve such a problem is Semi-Supervised Learning ( SSL ) [ 7 ] which can effectively leverage scarce labeled data and abundant unlabeled data to train an accurate classifier . However , classical SSL [ 14 , 15 , 16 , 17 , 24 , 39 , 41 , 49 ] relies on the closed-set assumption that the labeled and unlabeled data are drawn from the same class distribution and the same feature distribution . To be concrete , the class set of labeled data Cl is equal to that of unlabeled data Cu , and the marginal distribution pl ( x ) of the labeled data is identical to the unlabeled feature distribution pu ( x ) , which is shown in Figure 1 ( a ) . However , in practice , the datasets at hand could greatly violate the above assumption by having both class distribution mismatch and feature distribution mismatch , which is denoted as open-set1 . In this situation , traditional closed-set SSL would suffer from severe performance degradation [ 31 ] . Recent works [ 4 , 9 , 19 , 22 , 29 , 31 , 45 ] have focused on dealing with the class distribution mismatch problem to extend SSL into the wild . For example , Guo et al . ( 2020 ) [ 19 ] considers the subset mismatch situation when the classes in labeled data are only a subset of the classes in unlabeled data , 1Note that there are some other works [ 4 , 45 ] also named as “ open-set SSL ” . However , they all focus on the class distribution mismatch and do not deal with the feature distribution mismatch , so their setting is very different from ours . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . i.e. , Cl ⊂ Cu and Cu \ Cl 6= ∅ ( Figure 1 ( b ) ) . It utilizes a meta learning scheme to down-weight the unlabeled data that do not belong to Cl . Chen et al . ( 2020 ) [ 9 ] deals with the intersectional mismatch problem where the labeled and unlabeled data both contain a shared class set but also hold a private class set , respectively , i.e. , Cl ∩ Cu 6= ∅ , Cl \ ( Cu ∩ Cl ) 6= ∅ and Cu \ ( Cu ∩ Cl ) 6= ∅ ( Figure 1 ( c ) ) . It proposes a self-distillation method to filter out the probable unlabeled private data . However , existing methods require prior knowledge of the relationship between Cl and Cu , which greatly limits their realistic application . When the class relationship is unknown , the potential private data from Dl and Du could both seriously mislead the learning process . Moreover , existing works only consider the class distribution mismatch and totally ignore the feature distribution mismatch problem . The latter problem is also quite common in practice , as when we try to collect a large amount of unlabeled data to aid the model training , the feature distribution of the newly obtained unlabeled data could be heavily influenced by when , where , and how we collect them . As a result , the potential feature distribution difference between labeled and unlabeled data could seriously harm the learning performance . Therefore , it is necessary to design a universal method to solve different scenarios of class distribution mismatch and meanwhile deal with the feature distribution mismatch . In this paper , we propose a new framework dubbed “ Class-shAring data detection and Feature Adaptation ( CAFA ) ” which is a Universal Semi-Supervised Learning ( UniSSL ) method for tackling different situations of open-set problem . Specifically , by considering that the labeled set and unlabeled set are drawn from different domains , we utilize a novel scoring mechanism to identify both the labeled and unlabeled data from the shared classes . The mechanism integrates two cues , namely domain similarity and label prediction shift , which are perfectly tailored for data detection in open-set SSL . Then we employ domain adaptation [ 5 , 6 , 12 , 40 , 43 , 47 ] to match the identified unlabeled data to the same feature distribution of the class-sharing labeled data . After feature adaptation , the value of original unlabeled data can be fully exploited for boosting the learning performance . Moreover , we conduct weighted SSL to take full advantage of the class-sharing data from the open dataset . To sum up , our main contributions are : • We propose a universal framework that can solve different scenarios of open-set SSL without any prior class knowledge . • Our method can fully exploit the value of unlabeled data by mitigating the feature distribution mismatch between labeled data and unlabeled data . • Experiments show that our method outperforms all other baselines in different situations of open-set problems . 2 Universal Semi-Supervised Learning . In our open-set SSL setting , we are given a labeled set Dl = { ( xi , yi ) } li=1 containing l instances xi labeled with { yi } li=1 , and an unlabeled set Du = { xj } u j=1 consisting of u unlabeled instances xj , where l u . The two datasets Dl and Du are drawn from two different feature distributions pl ( xi ) and pu ( xj ) , respectively . We use Cl to denote the labeled class set which contains the classes of labeled data , and employ Cu to represent the unlabeled class set consisting of the classes of unlabeled ( a ) data . Note that in our setting , we do not know the exact relationship between Cl and Cu . Particularly , we use C = Cl ∩ Cu to denote the common class set shared by Dl and Du , and use Cl and Cu to denote the class sets private to labeled data and unlabeled data , respectively . The feature distributions of labeled data with labels in C and Cl are denoted as plC ( xi ) and plCl ( xi ) , respectively , and the feature distributions of unlabeled data that belong to C and Cu are denoted as puC ( xj ) and puCu ( xj ) , respectively . Our goal is to effectively identify the class-sharing data from both Dl and Du , and then eliminate the feature distribution mismatch between the identified labeled and unlabeled data to help train an accurate semi-supervised model in classifying the test data to the classes-of-interests Cl . 2.1 The General Framework of CAFA . As shown in Figure 2 , CAFA contains a feature extractor F , a classifierC , an adversarial discriminator D , and a non-adversarial discriminatorD′ . Given an input instance x , we use F to compute its feature representation z = F ( x ) . Then we employ C to output the label prediction f using z . The nonadversarial discriminator D′ produces a domain similarity score wd , which quantifies the similarity degree of an instance to one distribution . The adversarial discriminator D aims to adversarially adapt the feature distributions of labeled and unlabeled data to the common classes C. The general framework of CAFA can be concisely formulated as : min θF , θC max θD Exi∼plLce ( C ( F ( xi ) ) , yi ) ︸ ︷︷ ︸ supervised fidelity term − γExi∼pl , xj∼puw l · wu · Ladv ( xi , xj ; θF , θD ) ︸ ︷︷ ︸ feature adaptation term + δExj∼puwu · Lssl ( C ( F ( xj ) ) , yj ) ︸ ︷︷ ︸ class-sharing data exploration term , ( 1 ) in which θF , θC , θD are the parameters of F , C , and D , respectively . In Eq . ( 1 ) , the first term is dubbed supervised fidelity term which involves the standard cross-entropy loss Lce ( · ) . The second term is dubbed feature adaptation term which introduces an adversarial learning loss Ladv to conduct feature adaptation on the class-sharing data from Dl and Du . Here the class-sharing data are detected through two scores wl and wu which will be detailed in Section 2.2 . Through such a feature adaptation procedure , our CAFA approach can maximally exploit the unlabeled data and benefit SSL . The third term refers to class-sharing data exploration term which conducts semi-supervised training using a SSL loss Lssl ( · ) to make full use of class-sharing unlabeled data . Here the SSL loss can be any regularizer in existing methods such as consistency regularizer [ 24 , 39 ] or manifold regularizer [ 1 , 13 , 44 ] , and yj is a ∣∣Cl∣∣-dimensional vector denoting the generated pseudo learning target for each unlabeled datum xj , where the notation |·| indicates the size of the corresponding set . The parameters γ and δ are non-negative coefficients that trade off the above three terms . From the general CAFA framework presented above , we can see that the class-sharing data detection , feature adaptation , and semi-supervised training are essential to our approach , and they will be detailed in Sections 2.2 , 2.3 , 2.4 , respectively . 2.2 Class-Sharing Data Detection . Class-sharing data detection aims to correctly distinguish the training data belonging to C from those in Cl ∪ Cu . To achieve this goal , we hope to model two class-sharing scores wl ( · ) and wu ( · ) for labeled and unlabeled data , respectively , which should satisfy the following inequalities [ 43 ] Ex∼plCw l ( x ) > Ex∼pl Cl wl ( x ) , Ex∼puCw u ( x ) > Ex∼pu Cu wu ( x ) . ( 2 ) The above inequities should hold in a large margin for better detection performance . Here we propose to utilize two cues , namely domain similarity wd and label prediction shift ws , to model wl and wu . Domain Similarity has been employed to quantify whether an instance belongs to a specific domain by several methods [ 6 , 43 , 47 ] . They usually train a non-adversarial discriminator D′ to predict the data from pl as 0 and those from pu as 1 by minimizing a cross-entropy loss . The output value wd = D ′ ( F ( x ) ) can be considered as the domain similarity of the input x . Particularly , the input x is likely to be sampled from pu if wd is large , and pl otherwise , formally Ex∼pl Cl wd ( x ) , Ex∼plCwd ( x ) < Ex∼puCuwd ( x ) , Ex∼puCwd ( x ) . ( 3 ) However , such a training strategy lacks exploitation on the middle region between two feature distributions . Consequently , it is prone to overfit to the situation when Ex∼pl Cl wd ( x ) ≈ Ex∼plCwd ( x ) ≈ 0 and Ex∼pu Cu wd ( x ) ≈ Ex∼puCwd ( x ) ≈ 1 , thus making the class-sharing data unrecognizable , as shown in the upper of Figure 3 ( a ) . To solve this problem , we conduct Mixup [ 46 ] to strengthen the relationship between xi ∼ plC and xj ∼ puC to yield discriminative domain similarities . Specifically , given two feature representations zi = F ( xi ) and zj = F ( xj ) of labeled datum xi and unlabeled datum xj , and their corresponding domain labels di = 0 and dj = 1 , respectively , we can generate a mixed feature representation z̃i , j and a mixed domain label d̃i , j as following z̃i , j = λzi + ( 1− λ ) zj , d̃i , j = 1− λ , ( 4 ) in which λ is sampled from a Beta distribution Beta ( α , α ) where α is a hyper-parameter . Then we leverage the mixed feature representations with their domain labels by adding an extra binary cross-entropy term to our domain similarity loss Ldom , which is formulated as Ldom = −Exi∼pl log ( 1−D ′ ( F ( xi ) ) ) − Exj∼pu logD′ ( F ( xj ) ) +Exi∼pl , xj∼pu [ sim ( zi , zj ) · ( − ( 1− λ ) logD ′ ( z̃i , j ) − λ log ( 1−D′ ( z̃i , j ) ) ) ] , ( 5 ) where sim ( zi , zj ) = z > i zj ‖zi‖‖zj‖ denotes the cosine similarity between zi and zj . Based on a reasonable assumption that zi and zj in C are closer to each other in the feature space than those in C l and Cu , such extra term weighted with cosine similarity can focus on interpolating the middle region between the two feature distributions plC and p u C , which helps preventing the aforementioned overfitting problem and making the domain similarity of class-sharing data closer to each other than the private data in Cl and Cu , as shown in the lower panel of Figure 3 ( a ) . Therefore , we can have Ex∼pl Cl wd ( x ) < Ex∼plCwd ( x ) and Ex∼puCwd ( x ) < Ex∼puCuwd ( x ) . By combining Eq . ( 3 ) , we have Ex∼pl Cl wd ( x ) < Ex∼plCwd ( x ) < Ex∼puCwd ( x ) < Ex∼puCuwd ( x ) . ( 6 ) The above inequality would yield a larger margin than traditional training strategy when equipped with the Mixup training . Thereby , our domain similarity can better detect the class-sharing data in the open-set situation . Nevertheless , the domain similarity alone is not sufficient for class-sharing data detection . Therefore , we introduce label prediction shift to enhance the detection performance . Label Prediction Shift indicates the influence imposed by adversarial perturbation on each instance , which can successfully differentiate the class-sharing data from private data , as shown in Figure 3 ( b ) . Formally , given an input instance x , its label prediction can be denoted as f = [ f1 ( x ) , f2 ( x ) , · · · , f|Cl| ( x ) ] > where { fi ( x ) } |Cl| i=1 can be interpreted as the probability that x belongs to class i . Then we apply an adversarial perturbation on x to obtain the perturbed version of input instance x∗ = x− sign ( −∇xLce ( x , maxi∈ { 1 , ··· , |Cl| } fi ( x ) ) ) , where controls the perturbation magnitude . As a result , the adversarial perturbation would decrease the maximum probability of the given input . Then , the label prediction shift can be computed as ws = max i∈Cl fi ( x ) −max i∈Cl fi ( x ∗ ) . ( 7 ) As a result , the computed label prediction shift would satisfy the following inequality : Ex∼pu Cu ws ( x ) < Ex∼puCws ( x ) < Ex∼plCws ( x ) < Ex∼plCl ws ( x ) . ( 8 ) Intuitively , the learning on scarce labeled data is strongly dependent on the supervised cross-entropy loss Lce , while the unlabeled data that are learned with consistency regularization are much more robust against perturbations [ 24 , 30 , 39 ] , thus it is natural to have Ex∼pu Cu ws ( x ) , Ex∼puCws ( x ) < Ex∼pl Cl ws ( x ) , Ex∼plCws ( x ) . Moreover , the abundant unlabeled data have the effect of improving the model generalizability in SSL . In open-set situation , the model generalizability on the class set Cl is greatly limited since there are only scarce labeled private data available , which makes such kind of data vulnerable against perturbations . On the contrary , the model learning on the classes in C is quite sufficient when compared with Cl since there are both labeled and unlabeled data that can be leveraged . Hence , we have Ex∼plCws ( x ) < Ex∼plCl ws ( x ) . Additionally , the unlabeled private data in Cu do not belong to any known classes and completely lie out of any known distribution . As mentioned in [ 27 ] , the adversarial perturbation would have less influence on their maximal label predictions than those data in Cl . Therefore , we can have Ex∼pu Cu ws ( x ) < Ex∼puCws ( x ) . Based on the above explanations , the inequality in Eq . ( 8 ) should hold . To integrate the proposed two cues wd and ws based on Eqs . ( 6 ) and ( 8 ) , we can compute wl and wu through wl ( x ) = wd ( x ) − ws ( x ) , x ∈ Dl , wu ( x ) = ws ( x ) − wd ( x ) , x ∈ Du . ( 9 ) Note that wd and ws are both normalized into interval [ 0 , 1 ] before computation . Through Eq . ( 9 ) , our class-sharing scores can perfectly satisfy Eq . ( 2 ) , hence they are effective for detecting the class-sharing data from both Dl and Du .

Summary: The paper proposes an approach for universal semi-supervised setting that handles both class distribution and feature distribution mismatch. The key idea of the CAFA approach lies in introducing a scoring mechanism that identifies shared classes in the labeled and unlabeled data. The objective function in CAFA consists of supervised cross-entropy loss on labeled data, adversarial loss for feature adaptation between labeled and unlabeled data and class sharing data exploration term on pseudo-labels that are calibrated by weighted softmax function where weights are computed for each class. The approach is validated on three datasets CIFAR-10, Office-31 and VisDA on three tasks: (1) only class distribution mismatch,(2) only feature distribution mismatch, and (3) both class and feature distribution mismatch.