Universal Semi-Supervised Learning

1 Introduction . A critical drawback of training a good neural network [ 25 ] is that it typically requires lots of labeled data , which is quite difficult to satisfy due to the unaffordable monetary cost as well as the huge demand for human resources . A popular way to solve such a problem is Semi-Supervised Learning ( SSL ) [ 7 ] which can effectively leverage scarce labeled data and abundant unlabeled data to train an accurate classifier . However , classical SSL [ 14 , 15 , 16 , 17 , 24 , 39 , 41 , 49 ] relies on the closed-set assumption that the labeled and unlabeled data are drawn from the same class distribution and the same feature distribution . To be concrete , the class set of labeled data Cl is equal to that of unlabeled data Cu , and the marginal distribution pl ( x ) of the labeled data is identical to the unlabeled feature distribution pu ( x ) , which is shown in Figure 1 ( a ) . However , in practice , the datasets at hand could greatly violate the above assumption by having both class distribution mismatch and feature distribution mismatch , which is denoted as open-set1 . In this situation , traditional closed-set SSL would suffer from severe performance degradation [ 31 ] . Recent works [ 4 , 9 , 19 , 22 , 29 , 31 , 45 ] have focused on dealing with the class distribution mismatch problem to extend SSL into the wild . For example , Guo et al . ( 2020 ) [ 19 ] considers the subset mismatch situation when the classes in labeled data are only a subset of the classes in unlabeled data , 1Note that there are some other works [ 4 , 45 ] also named as “ open-set SSL ” . However , they all focus on the class distribution mismatch and do not deal with the feature distribution mismatch , so their setting is very different from ours . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . i.e. , Cl ⊂ Cu and Cu \ Cl 6= ∅ ( Figure 1 ( b ) ) . It utilizes a meta learning scheme to down-weight the unlabeled data that do not belong to Cl . Chen et al . ( 2020 ) [ 9 ] deals with the intersectional mismatch problem where the labeled and unlabeled data both contain a shared class set but also hold a private class set , respectively , i.e. , Cl ∩ Cu 6= ∅ , Cl \ ( Cu ∩ Cl ) 6= ∅ and Cu \ ( Cu ∩ Cl ) 6= ∅ ( Figure 1 ( c ) ) . It proposes a self-distillation method to filter out the probable unlabeled private data . However , existing methods require prior knowledge of the relationship between Cl and Cu , which greatly limits their realistic application . When the class relationship is unknown , the potential private data from Dl and Du could both seriously mislead the learning process . Moreover , existing works only consider the class distribution mismatch and totally ignore the feature distribution mismatch problem . The latter problem is also quite common in practice , as when we try to collect a large amount of unlabeled data to aid the model training , the feature distribution of the newly obtained unlabeled data could be heavily influenced by when , where , and how we collect them . As a result , the potential feature distribution difference between labeled and unlabeled data could seriously harm the learning performance . Therefore , it is necessary to design a universal method to solve different scenarios of class distribution mismatch and meanwhile deal with the feature distribution mismatch . In this paper , we propose a new framework dubbed “ Class-shAring data detection and Feature Adaptation ( CAFA ) ” which is a Universal Semi-Supervised Learning ( UniSSL ) method for tackling different situations of open-set problem . Specifically , by considering that the labeled set and unlabeled set are drawn from different domains , we utilize a novel scoring mechanism to identify both the labeled and unlabeled data from the shared classes . The mechanism integrates two cues , namely domain similarity and label prediction shift , which are perfectly tailored for data detection in open-set SSL . Then we employ domain adaptation [ 5 , 6 , 12 , 40 , 43 , 47 ] to match the identified unlabeled data to the same feature distribution of the class-sharing labeled data . After feature adaptation , the value of original unlabeled data can be fully exploited for boosting the learning performance . Moreover , we conduct weighted SSL to take full advantage of the class-sharing data from the open dataset . To sum up , our main contributions are : • We propose a universal framework that can solve different scenarios of open-set SSL without any prior class knowledge . • Our method can fully exploit the value of unlabeled data by mitigating the feature distribution mismatch between labeled data and unlabeled data . • Experiments show that our method outperforms all other baselines in different situations of open-set problems . 2 Universal Semi-Supervised Learning . In our open-set SSL setting , we are given a labeled set Dl = { ( xi , yi ) } li=1 containing l instances xi labeled with { yi } li=1 , and an unlabeled set Du = { xj } u j=1 consisting of u unlabeled instances xj , where l u . The two datasets Dl and Du are drawn from two different feature distributions pl ( xi ) and pu ( xj ) , respectively . We use Cl to denote the labeled class set which contains the classes of labeled data , and employ Cu to represent the unlabeled class set consisting of the classes of unlabeled ( a ) data . Note that in our setting , we do not know the exact relationship between Cl and Cu . Particularly , we use C = Cl ∩ Cu to denote the common class set shared by Dl and Du , and use Cl and Cu to denote the class sets private to labeled data and unlabeled data , respectively . The feature distributions of labeled data with labels in C and Cl are denoted as plC ( xi ) and plCl ( xi ) , respectively , and the feature distributions of unlabeled data that belong to C and Cu are denoted as puC ( xj ) and puCu ( xj ) , respectively . Our goal is to effectively identify the class-sharing data from both Dl and Du , and then eliminate the feature distribution mismatch between the identified labeled and unlabeled data to help train an accurate semi-supervised model in classifying the test data to the classes-of-interests Cl . 2.1 The General Framework of CAFA . As shown in Figure 2 , CAFA contains a feature extractor F , a classifierC , an adversarial discriminator D , and a non-adversarial discriminatorD′ . Given an input instance x , we use F to compute its feature representation z = F ( x ) . Then we employ C to output the label prediction f using z . The nonadversarial discriminator D′ produces a domain similarity score wd , which quantifies the similarity degree of an instance to one distribution . The adversarial discriminator D aims to adversarially adapt the feature distributions of labeled and unlabeled data to the common classes C. The general framework of CAFA can be concisely formulated as : min θF , θC max θD Exi∼plLce ( C ( F ( xi ) ) , yi ) ︸ ︷︷ ︸ supervised fidelity term − γExi∼pl , xj∼puw l · wu · Ladv ( xi , xj ; θF , θD ) ︸ ︷︷ ︸ feature adaptation term + δExj∼puwu · Lssl ( C ( F ( xj ) ) , yj ) ︸ ︷︷ ︸ class-sharing data exploration term , ( 1 ) in which θF , θC , θD are the parameters of F , C , and D , respectively . In Eq . ( 1 ) , the first term is dubbed supervised fidelity term which involves the standard cross-entropy loss Lce ( · ) . The second term is dubbed feature adaptation term which introduces an adversarial learning loss Ladv to conduct feature adaptation on the class-sharing data from Dl and Du . Here the class-sharing data are detected through two scores wl and wu which will be detailed in Section 2.2 . Through such a feature adaptation procedure , our CAFA approach can maximally exploit the unlabeled data and benefit SSL . The third term refers to class-sharing data exploration term which conducts semi-supervised training using a SSL loss Lssl ( · ) to make full use of class-sharing unlabeled data . Here the SSL loss can be any regularizer in existing methods such as consistency regularizer [ 24 , 39 ] or manifold regularizer [ 1 , 13 , 44 ] , and yj is a ∣∣Cl∣∣-dimensional vector denoting the generated pseudo learning target for each unlabeled datum xj , where the notation |·| indicates the size of the corresponding set . The parameters γ and δ are non-negative coefficients that trade off the above three terms . From the general CAFA framework presented above , we can see that the class-sharing data detection , feature adaptation , and semi-supervised training are essential to our approach , and they will be detailed in Sections 2.2 , 2.3 , 2.4 , respectively . 2.2 Class-Sharing Data Detection . Class-sharing data detection aims to correctly distinguish the training data belonging to C from those in Cl ∪ Cu . To achieve this goal , we hope to model two class-sharing scores wl ( · ) and wu ( · ) for labeled and unlabeled data , respectively , which should satisfy the following inequalities [ 43 ] Ex∼plCw l ( x ) > Ex∼pl Cl wl ( x ) , Ex∼puCw u ( x ) > Ex∼pu Cu wu ( x ) . ( 2 ) The above inequities should hold in a large margin for better detection performance . Here we propose to utilize two cues , namely domain similarity wd and label prediction shift ws , to model wl and wu . Domain Similarity has been employed to quantify whether an instance belongs to a specific domain by several methods [ 6 , 43 , 47 ] . They usually train a non-adversarial discriminator D′ to predict the data from pl as 0 and those from pu as 1 by minimizing a cross-entropy loss . The output value wd = D ′ ( F ( x ) ) can be considered as the domain similarity of the input x . Particularly , the input x is likely to be sampled from pu if wd is large , and pl otherwise , formally Ex∼pl Cl wd ( x ) , Ex∼plCwd ( x ) < Ex∼puCuwd ( x ) , Ex∼puCwd ( x ) . ( 3 ) However , such a training strategy lacks exploitation on the middle region between two feature distributions . Consequently , it is prone to overfit to the situation when Ex∼pl Cl wd ( x ) ≈ Ex∼plCwd ( x ) ≈ 0 and Ex∼pu Cu wd ( x ) ≈ Ex∼puCwd ( x ) ≈ 1 , thus making the class-sharing data unrecognizable , as shown in the upper of Figure 3 ( a ) . To solve this problem , we conduct Mixup [ 46 ] to strengthen the relationship between xi ∼ plC and xj ∼ puC to yield discriminative domain similarities . Specifically , given two feature representations zi = F ( xi ) and zj = F ( xj ) of labeled datum xi and unlabeled datum xj , and their corresponding domain labels di = 0 and dj = 1 , respectively , we can generate a mixed feature representation z̃i , j and a mixed domain label d̃i , j as following z̃i , j = λzi + ( 1− λ ) zj , d̃i , j = 1− λ , ( 4 ) in which λ is sampled from a Beta distribution Beta ( α , α ) where α is a hyper-parameter . Then we leverage the mixed feature representations with their domain labels by adding an extra binary cross-entropy term to our domain similarity loss Ldom , which is formulated as Ldom = −Exi∼pl log ( 1−D ′ ( F ( xi ) ) ) − Exj∼pu logD′ ( F ( xj ) ) +Exi∼pl , xj∼pu [ sim ( zi , zj ) · ( − ( 1− λ ) logD ′ ( z̃i , j ) − λ log ( 1−D′ ( z̃i , j ) ) ) ] , ( 5 ) where sim ( zi , zj ) = z > i zj ‖zi‖‖zj‖ denotes the cosine similarity between zi and zj . Based on a reasonable assumption that zi and zj in C are closer to each other in the feature space than those in C l and Cu , such extra term weighted with cosine similarity can focus on interpolating the middle region between the two feature distributions plC and p u C , which helps preventing the aforementioned overfitting problem and making the domain similarity of class-sharing data closer to each other than the private data in Cl and Cu , as shown in the lower panel of Figure 3 ( a ) . Therefore , we can have Ex∼pl Cl wd ( x ) < Ex∼plCwd ( x ) and Ex∼puCwd ( x ) < Ex∼puCuwd ( x ) . By combining Eq . ( 3 ) , we have Ex∼pl Cl wd ( x ) < Ex∼plCwd ( x ) < Ex∼puCwd ( x ) < Ex∼puCuwd ( x ) . ( 6 ) The above inequality would yield a larger margin than traditional training strategy when equipped with the Mixup training . Thereby , our domain similarity can better detect the class-sharing data in the open-set situation . Nevertheless , the domain similarity alone is not sufficient for class-sharing data detection . Therefore , we introduce label prediction shift to enhance the detection performance . Label Prediction Shift indicates the influence imposed by adversarial perturbation on each instance , which can successfully differentiate the class-sharing data from private data , as shown in Figure 3 ( b ) . Formally , given an input instance x , its label prediction can be denoted as f = [ f1 ( x ) , f2 ( x ) , · · · , f|Cl| ( x ) ] > where { fi ( x ) } |Cl| i=1 can be interpreted as the probability that x belongs to class i . Then we apply an adversarial perturbation on x to obtain the perturbed version of input instance x∗ = x− sign ( −∇xLce ( x , maxi∈ { 1 , ··· , |Cl| } fi ( x ) ) ) , where controls the perturbation magnitude . As a result , the adversarial perturbation would decrease the maximum probability of the given input . Then , the label prediction shift can be computed as ws = max i∈Cl fi ( x ) −max i∈Cl fi ( x ∗ ) . ( 7 ) As a result , the computed label prediction shift would satisfy the following inequality : Ex∼pu Cu ws ( x ) < Ex∼puCws ( x ) < Ex∼plCws ( x ) < Ex∼plCl ws ( x ) . ( 8 ) Intuitively , the learning on scarce labeled data is strongly dependent on the supervised cross-entropy loss Lce , while the unlabeled data that are learned with consistency regularization are much more robust against perturbations [ 24 , 30 , 39 ] , thus it is natural to have Ex∼pu Cu ws ( x ) , Ex∼puCws ( x ) < Ex∼pl Cl ws ( x ) , Ex∼plCws ( x ) . Moreover , the abundant unlabeled data have the effect of improving the model generalizability in SSL . In open-set situation , the model generalizability on the class set Cl is greatly limited since there are only scarce labeled private data available , which makes such kind of data vulnerable against perturbations . On the contrary , the model learning on the classes in C is quite sufficient when compared with Cl since there are both labeled and unlabeled data that can be leveraged . Hence , we have Ex∼plCws ( x ) < Ex∼plCl ws ( x ) . Additionally , the unlabeled private data in Cu do not belong to any known classes and completely lie out of any known distribution . As mentioned in [ 27 ] , the adversarial perturbation would have less influence on their maximal label predictions than those data in Cl . Therefore , we can have Ex∼pu Cu ws ( x ) < Ex∼puCws ( x ) . Based on the above explanations , the inequality in Eq . ( 8 ) should hold . To integrate the proposed two cues wd and ws based on Eqs . ( 6 ) and ( 8 ) , we can compute wl and wu through wl ( x ) = wd ( x ) − ws ( x ) , x ∈ Dl , wu ( x ) = ws ( x ) − wd ( x ) , x ∈ Du . ( 9 ) Note that wd and ws are both normalized into interval [ 0 , 1 ] before computation . Through Eq . ( 9 ) , our class-sharing scores can perfectly satisfy Eq . ( 2 ) , hence they are effective for detecting the class-sharing data from both Dl and Du .

Summary: The paper investigates a realistic situation of open-set SSL, which includes the class distribution mismatch and the feature distribution mismatch. Moreover, it assumes the class relationship between labeled data and unlabeled data is unknown. The proposed method utilizes two scores to detect the data that belong to the common classes, and conduct domain adaptation to solve the feature mismatch problem.