Universal Semi-Supervised Learning
Year: 2021

Abstract:
Universal Semi-Supervised Learning (UniSSL) aims to solve the open-set problem where both the class distribution (i.e., class set) and feature distribution (i.e., feature domain) are different between labeled dataset and unlabeled dataset. Such a problem seriously hinders the realistic landing of classical SSL. Different from the existing SSL methods targeting at the open-set problem that only study one certain scenario of class distribution mismatch and ignore the feature distribution mismatch, we consider a more general case where a mismatch exists in both class and feature distribution. In this case, we propose a “Class-shAring data detection and Feature Adaptation” (CAFA) framework which requires no prior knowledge of the class relationship between the labeled dataset and unlabeled dataset. Particularly, CAFA utilizes a novel scoring strategy to detect the data in the shared class set. Then, it conducts domain adaptation to fully exploit the value of the detected class-sharing data for better semi-supervised consistency training. Exhaustive experiments on several benchmark datasets show the effectiveness of our method in tackling open-set problems.

heading: 1 Introduction
text: A critical drawback of training a good neural network [25] is that it typically requires lots of labeled data, which is quite difficult to satisfy due to the unaffordable monetary cost as well as the huge demand for human resources. A popular way to solve such a problem is Semi-Supervised Learning (SSL) [7] which can effectively leverage scarce labeled data and abundant unlabeled data to train an accurate classifier. However, classical SSL [14, 15, 16, 17, 24, 39, 41, 49] relies on the closed-set assumption that the labeled and unlabeled data are drawn from the same class distribution and the same feature distribution. To be concrete, the class set of labeled data Cl is equal to that of unlabeled data Cu, and the marginal distribution pl(x) of the labeled data is identical to the unlabeled feature distribution pu(x), which is shown in Figure 1 (a). However, in practice, the datasets at hand could greatly violate the above assumption by having both class distribution mismatch and feature distribution mismatch, which is denoted as open-set1. In this situation, traditional closed-set SSL would suffer from severe performance degradation [31].
Recent works [4, 9, 19, 22, 29, 31, 45] have focused on dealing with the class distribution mismatch problem to extend SSL into the wild. For example, Guo et al. (2020) [19] considers the subset mismatch situation when the classes in labeled data are only a subset of the classes in unlabeled data,
1Note that there are some other works [4, 45] also named as “open-set SSL”. However, they all focus on the class distribution mismatch and do not deal with the feature distribution mismatch, so their setting is very different from ours.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
i.e., Cl ⊂ Cu and Cu \ Cl 6= ∅ (Figure 1 (b)). It utilizes a meta learning scheme to down-weight the unlabeled data that do not belong to Cl. Chen et al. (2020) [9] deals with the intersectional mismatch problem where the labeled and unlabeled data both contain a shared class set but also hold a private class set, respectively, i.e., Cl ∩ Cu 6= ∅, Cl \ (Cu ∩ Cl) 6= ∅ and Cu \ (Cu ∩ Cl) 6= ∅ (Figure 1 (c)). It proposes a self-distillation method to filter out the probable unlabeled private data. However, existing methods require prior knowledge of the relationship between Cl and Cu, which greatly limits their realistic application. When the class relationship is unknown, the potential private data from Dl and Du could both seriously mislead the learning process. Moreover, existing works only consider the class distribution mismatch and totally ignore the feature distribution mismatch problem. The latter problem is also quite common in practice, as when we try to collect a large amount of unlabeled data to aid the model training, the feature distribution of the newly obtained unlabeled data could be heavily influenced by when, where, and how we collect them. As a result, the potential feature distribution difference between labeled and unlabeled data could seriously harm the learning performance. Therefore, it is necessary to design a universal method to solve different scenarios of class distribution mismatch and meanwhile deal with the feature distribution mismatch.
In this paper, we propose a new framework dubbed “Class-shAring data detection and Feature Adaptation (CAFA)” which is a Universal Semi-Supervised Learning (UniSSL) method for tackling different situations of open-set problem. Specifically, by considering that the labeled set and unlabeled set are drawn from different domains, we utilize a novel scoring mechanism to identify both the labeled and unlabeled data from the shared classes. The mechanism integrates two cues, namely domain similarity and label prediction shift, which are perfectly tailored for data detection in open-set SSL. Then we employ domain adaptation [5, 6, 12, 40, 43, 47] to match the identified unlabeled data to the same feature distribution of the class-sharing labeled data. After feature adaptation, the value of original unlabeled data can be fully exploited for boosting the learning performance. Moreover, we conduct weighted SSL to take full advantage of the class-sharing data from the open dataset. To sum up, our main contributions are:
• We propose a universal framework that can solve different scenarios of open-set SSL without any prior class knowledge.
• Our method can fully exploit the value of unlabeled data by mitigating the feature distribution mismatch between labeled data and unlabeled data.
• Experiments show that our method outperforms all other baselines in different situations of open-set problems.

heading: 2 Universal Semi-Supervised Learning
text: In our open-set SSL setting, we are given a labeled set Dl = {(xi, yi)}li=1 containing l instances xi labeled with {yi}li=1, and an unlabeled set Du = {xj} u j=1 consisting of u unlabeled instances xj , where l u. The two datasets Dl and Du are drawn from two different feature distributions pl(xi) and pu(xj), respectively. We use Cl to denote the labeled class set which contains the classes of labeled data, and employ Cu to represent the unlabeled class set consisting of the classes of unlabeled
(a)
data. Note that in our setting, we do not know the exact relationship between Cl and Cu. Particularly, we use C = Cl ∩ Cu to denote the common class set shared by Dl and Du, and use Cl and Cu to denote the class sets private to labeled data and unlabeled data, respectively. The feature distributions of labeled data with labels in C and Cl are denoted as plC(xi) and plCl(xi), respectively, and the feature distributions of unlabeled data that belong to C and Cu are denoted as puC(xj) and puCu(xj), respectively.
Our goal is to effectively identify the class-sharing data from both Dl and Du, and then eliminate the feature distribution mismatch between the identified labeled and unlabeled data to help train an accurate semi-supervised model in classifying the test data to the classes-of-interests Cl.

heading: 2.1 The General Framework of CAFA
text: As shown in Figure 2, CAFA contains a feature extractor F , a classifierC, an adversarial discriminator D, and a non-adversarial discriminatorD′. Given an input instance x, we use F to compute its feature representation z = F (x). Then we employ C to output the label prediction f using z. The nonadversarial discriminator D′ produces a domain similarity score wd, which quantifies the similarity degree of an instance to one distribution. The adversarial discriminator D aims to adversarially adapt the feature distributions of labeled and unlabeled data to the common classes C. The general framework of CAFA can be concisely formulated as:
min θF ,θC max θD Exi∼plLce(C(F (xi)), yi)︸ ︷︷ ︸ supervised fidelity term − γExi∼pl,xj∼puw l · wu · Ladv(xi,xj ; θF , θD)︸ ︷︷ ︸ feature adaptation term
+ δExj∼puwu · Lssl(C(F (xj)),yj)︸ ︷︷ ︸ class-sharing data exploration term
, (1)
in which θF , θC , θD are the parameters of F , C, and D, respectively. In Eq. (1), the first term is dubbed supervised fidelity term which involves the standard cross-entropy loss Lce(·). The second term is dubbed feature adaptation term which introduces an adversarial learning loss Ladv to conduct feature adaptation on the class-sharing data from Dl and Du. Here the class-sharing data are detected through two scores wl and wu which will be detailed in Section 2.2. Through such a feature adaptation procedure, our CAFA approach can maximally exploit the unlabeled data and benefit SSL. The third term refers to class-sharing data exploration term which conducts semi-supervised training using a SSL loss Lssl(·) to make full use of class-sharing unlabeled data. Here the SSL loss can be any regularizer in existing methods such as consistency regularizer [24, 39] or manifold regularizer [1, 13, 44], and yj is a
∣∣Cl∣∣-dimensional vector denoting the generated pseudo learning target for each unlabeled datum xj , where the notation |·| indicates the size of the corresponding set. The parameters γ and δ are non-negative coefficients that trade off the above three terms.
From the general CAFA framework presented above, we can see that the class-sharing data detection, feature adaptation, and semi-supervised training are essential to our approach, and they will be detailed in Sections 2.2, 2.3, 2.4, respectively.

heading: 2.2 Class-Sharing Data Detection
text: Class-sharing data detection aims to correctly distinguish the training data belonging to C from those in Cl ∪ Cu. To achieve this goal, we hope to model two class-sharing scores wl(·) and wu(·) for labeled and unlabeled data, respectively, which should satisfy the following inequalities [43]
Ex∼plCw l(x) > Ex∼pl Cl wl(x), Ex∼puCw u(x) > Ex∼pu Cu wu(x).
(2)
The above inequities should hold in a large margin for better detection performance. Here we propose to utilize two cues, namely domain similarity wd and label prediction shift ws, to model wl and wu.
Domain Similarity has been employed to quantify whether an instance belongs to a specific domain by several methods [6, 43, 47]. They usually train a non-adversarial discriminator D′ to predict the data from pl as 0 and those from pu as 1 by minimizing a cross-entropy loss. The output value wd = D
′(F (x)) can be considered as the domain similarity of the input x. Particularly, the input x is likely to be sampled from pu if wd is large, and pl otherwise, formally
Ex∼pl Cl wd(x),Ex∼plCwd(x) < Ex∼puCuwd(x),Ex∼puCwd(x). (3)
However, such a training strategy lacks exploitation on the middle region between two feature distributions. Consequently, it is prone to overfit to the situation when Ex∼pl
Cl wd(x) ≈ Ex∼plCwd(x) ≈ 0
and Ex∼pu Cu wd(x) ≈ Ex∼puCwd(x) ≈ 1, thus making the class-sharing data unrecognizable, as shown in the upper of Figure 3 (a). To solve this problem, we conduct Mixup [46] to strengthen the relationship between xi ∼ plC and xj ∼ puC to yield discriminative domain similarities. Specifically, given two feature representations zi = F (xi) and zj = F (xj) of labeled datum xi and unlabeled datum xj , and their corresponding domain labels di = 0 and dj = 1, respectively, we can generate a mixed feature representation z̃i,j and a mixed domain label d̃i,j as following
z̃i,j = λzi + (1− λ)zj , d̃i,j = 1− λ, (4)
in which λ is sampled from a Beta distribution Beta(α, α) where α is a hyper-parameter. Then we leverage the mixed feature representations with their domain labels by adding an extra binary cross-entropy term to our domain similarity loss Ldom, which is formulated as
Ldom = −Exi∼pl log(1−D ′(F (xi)))− Exj∼pu logD′(F (xj))
+Exi∼pl,xj∼pu [sim(zi, zj) · (−(1− λ) logD ′(z̃i,j)− λ log(1−D′(z̃i,j)))] ,
(5)
where sim(zi, zj) = z>i zj ‖zi‖‖zj‖ denotes the cosine similarity between zi and zj . Based on a reasonable assumption that zi and zj in C are closer to each other in the feature space than those in C l
and Cu, such extra term weighted with cosine similarity can focus on interpolating the middle region between the two feature distributions plC and p u C , which helps preventing the aforementioned overfitting problem and making the domain similarity of class-sharing data closer to each other than the private data in Cl and Cu, as shown in the lower panel of Figure 3 (a). Therefore, we can have Ex∼pl
Cl wd(x) < Ex∼plCwd(x) and Ex∼puCwd(x) < Ex∼puCuwd(x). By combining Eq. (3), we have
Ex∼pl Cl wd(x) < Ex∼plCwd(x) < Ex∼puCwd(x) < Ex∼puCuwd(x). (6)
The above inequality would yield a larger margin than traditional training strategy when equipped with the Mixup training. Thereby, our domain similarity can better detect the class-sharing data in the open-set situation. Nevertheless, the domain similarity alone is not sufficient for class-sharing data detection. Therefore, we introduce label prediction shift to enhance the detection performance.
Label Prediction Shift indicates the influence imposed by adversarial perturbation on each instance, which can successfully differentiate the class-sharing data from private data, as shown in Figure 3 (b). Formally, given an input instance x, its label prediction can be denoted as
f = [ f1(x), f2(x), · · · , f|Cl|(x) ]> where {fi(x)} |Cl| i=1 can be interpreted as the probability that x belongs to class i. Then we apply an adversarial perturbation on x to obtain the perturbed version of input instance x∗ = x− sign(−∇xLce(x,maxi∈{1,··· ,|Cl|} fi(x))), where controls the perturbation magnitude. As a result, the adversarial perturbation would decrease the maximum probability of the given input. Then, the label prediction shift can be computed as
ws = max i∈Cl fi(x)−max i∈Cl fi(x ∗). (7)
As a result, the computed label prediction shift would satisfy the following inequality: Ex∼pu
Cu ws(x) < Ex∼puCws(x) < Ex∼plCws(x) < Ex∼plCl ws(x). (8)
Intuitively, the learning on scarce labeled data is strongly dependent on the supervised cross-entropy loss Lce, while the unlabeled data that are learned with consistency regularization are much more robust against perturbations [24, 30, 39], thus it is natural to have Ex∼pu
Cu ws(x),Ex∼puCws(x) <
Ex∼pl Cl ws(x),Ex∼plCws(x). Moreover, the abundant unlabeled data have the effect of improving the model generalizability in SSL. In open-set situation, the model generalizability on the class set Cl is greatly limited since there are only scarce labeled private data available, which makes such kind of data vulnerable against perturbations. On the contrary, the model learning on the classes in C is quite sufficient when compared with Cl since there are both labeled and unlabeled data that can be leveraged. Hence, we have Ex∼plCws(x) < Ex∼plCl ws(x). Additionally, the unlabeled private data in Cu do not belong to any known classes and completely lie out of any known distribution. As mentioned in [27], the adversarial perturbation would have less influence on their maximal label predictions than those data in Cl. Therefore, we can have Ex∼pu
Cu ws(x) < Ex∼puCws(x). Based on
the above explanations, the inequality in Eq. (8) should hold.
To integrate the proposed two cues wd and ws based on Eqs. (6) and (8), we can compute wl and wu through
wl(x) = wd(x)− ws(x), x ∈ Dl, wu(x) = ws(x)− wd(x), x ∈ Du.
(9)
Note that wd and ws are both normalized into interval [0, 1] before computation. Through Eq. (9), our class-sharing scores can perfectly satisfy Eq. (2), hence they are effective for detecting the class-sharing data from both Dl and Du.

heading: 2.3 Feature Adaptation
text: After detecting the class-sharing data in the above subsection, now we should eliminate the feature distribution mismatch between plC and p u C such that the value of unlabeled data can be properly extracted to aid the subsequent SSL. To this end, we treat the labeled data as target domain (i.e., di = 0,xi ∈ Dl) and the unlabeled data as the source domain (i.e., dj = 1,xj ∈ Du), and conduct adversarial domain adaptation [5, 6, 12, 40, 47] to achieve this goal. Particularly, we apply the classsharing scores wl and wu to the adversarial learning loss Ladv , and train the adversarial discriminator D to distinguish the labeled and unlabeled data. Meanwhile, the feature extractor F is trained to deceive D. The above adversarial process is formulated as the following min-max game:
max θF min θD
Exi∼pl,xj∼puLadv(xi,xj ; θF , θD) = −Exi∼plw l(xi) · log(1−D(F (xi)))
−Exj∼puwu(xj) · logD(F (xj)). (10)
Thanks to the two class-sharing scoreswl andwu, we can successfully mitigate the feature distribution mismatch between plC and p u C without being influenced by the irrelevant distributions p l
Cl and puCu . As
we will show later in the experiments, our feature adaptation can re-discover the value of unlabeled data and boost the SSL performance.

heading: 2.4 Semi-Supervised Training
text: With the aforementioned class-sharing data detection and feature adaptation, we can take full advantage of the open-set information by alleviating the negative impact from both class distribution mismatch and feature distribution mismatch. Then, we should aim at effectively exploring the classsharing unlabeled data, meanwhile weakening the negative impact from private data. Particularly, the private data in Cl could mislead the unlabeled data transferring to the wrong classes, and the unlabeled private data in Cu could be erroneously incorporated into network training, causing further performance degradation. To solve this problem, we propose the following SSL training strategy:
min θF ,θC
wu(x) · Lssl(C(F (x)), ŷ), (11)
where wu(x) is employed to weaken the network learning on unlabeled private data, and ŷ indicates the calibrated pseudo target for each unlabeled datum to mitigate the misleading bias introduced by labeled private data. To calibrate the original biased pseudo target y, we propose to utilize a weighted softmax function. Particularly, we compute the average weight of wl with respect to each class c as
wavgc = 1
l l∑ i=1 I(yi = c) · wl(xi), c ∈ Cl. (12)
Based on Eq. (2), the computed weight wavgc would be large if c is in C, and be small if c is in C l . Then we can calibrate the pseudo target y through
[ŷ]c = wavgc · exp [y]c∑|Cl| i=1 w avg i · exp [y]i , c ∈ Cl, (13)
where the notation [·]c denotes the c-th entry of the input vector. Through such process, the entries of ŷ belonging to Cl would be suppressed, and those belonging to C would be enhanced, which successfully alleviates the bias from the original target y.
To sum up, our general framework can be instantiated by substituting Eq. (10) and Eq. (11) into the feature adaptation term and class-sharing data exploration term in Eq. (1). Later experiments will show that our CAFA framework can effectively tackle different scenarios of the open-set problem without any prior knowledge of the class relationship and achieve encouraging performances.

heading: 3 Related Work

heading: 3.1 Closed-Set SSL
text: Closed-set SSL deals with the problem when Cl = Cu and pl(x) = pu(x). Early closed-set SSL methods such as Entropy Minimization [18] and Pseudo-Label (PL) [26] enforce the networks to make confident predictions on unlabeled data. Later, consistency-based methods such as Π-Model (PI) [24], Temporal Ensembling [24], and Mean Teacher (MT) [39] conduct consistency training between temporally or spatially different models. After that, Virtual Adversarial Training (VAT) [30] computes adversarial perturbations which maximally change the input image data to enhance the model robustness. Recent methods mostly rely on data augmentation to improve the network generalizability. For instance, MixMatch (MM) [3] and ReMixMatch [2] employ the Mixup [46] to augment the training data as well as the label information, which is beneficial to network training. FixMatch (FM) [38] utilizes the label predictions of weakly augmented image data to guide the learning of strongly augmented image data and achieves state-of-the-art performance. However, as mentioned before, closed-set SSL cannot perform satisfactorily in practice. Thereby, open-set SSL is designed to extend SSL into the wild.

heading: 3.2 Open-Set SSL
text: Open-set SSL is a new topic in SSL which aims to tackle the problem when Cl 6= Cu and pl(x) 6= pu(x). Laine & Aila (2016) [24] and Oliver et al. (2018) [31] firstly raised the class distribution mismatch problem in open-set SSL, which is further investigated by many subsequent methods. For example, Uncertainty Aware Self-Distillation (UASD) [9] deals with the intersectional mismatch by proposing a self-distillation method to filter out the probable unlabeled private data. Then Safe Deep Semi-Supervised Learning (DS3L) [19] and Multi-Task Curriculum Framework (MTCF) [45] solve the subset mismatch by employing different weighting strategies to down-weight the unlabeled
private data. Recently, Cao et al. (2021) [4] uses contrastive learning [8, 42] to separate the unlabeled private data from the class-sharing data in the subset mismatch problem. Existing open-set SSL methods made some significant attempts toward a more practical setting. However, when the feature distribution mismatch between labeled and unlabeled data presents, almost all existing methods would fail.

heading: 3.3 Domain Adaptation
text: Domain adaptation aims to transfer knowledge from the labeled source domain to the unlabeled target domain, which is generally divided into four categories: 1) closed-set domain adaptation assumes that the class sets of the source and target data are the same. Existing methods [10, 11, 12, 28, 40, 48] aim to learn class discriminative and domain invariant features from source and target domains; 2) partial domain adaptation considers that the class set of source data is larger than that of target data. Some methods [5, 6, 47] aim to apply class-level weight on each source datum to achieve per-class distribution matching; 3) open-set domain adaptation denotes the situation when the target domain contains private classes. Busto et al. (2017) [32] trains SVMs to identify the target private classes. Saito et al. (2018) [37] employs an extra logit to the classifier to incorporate the unknown target classes. Zhuo et al. (2019) [50] proposes to leverage word vectors to recognize the open domains; and 4) universal domain adaptation requires no prior knowledge of the class relationship between source and target domains. You et al. (2020) [43] utilizes the domain knowledge and the entropy value to find the data in the shared classes. Saito et al. (2020) [36] employs neighborhood clustering [21] to help the feature alignment between the class-sharing source and target data.

heading: 4 Experiments
text: In this section, we first specify the implementation details2 in Section 4.1. Then, to thoroughly validate the proposed CAFA approach, we conduct extensive experiments under different scenarios of open-set SSL by comparing our method with popular closed-set methods as well as several existing open-set methods in Section 4.2. Finally, we present the detailed performance analysis of our method in Section 4.3.

heading: 4.1 Experimental Setup
text: Datasets. We use CIFAR-10 [23], Office-31 [35], and VisDA2017 [33] to evaluation our method. We denote the 10 classes from CIFAR-10 as “0” ∼ “9”, the 31 classes from Office-31 as “0” ∼ “30”, and the 12 classes from VisDA2017 as “0” ∼ “11”. The numbers of labeled instances in CIFAR-10, Office-31, and VisDA2017 are set to 2,400, 100, and 1,800, respectively, and the number of unlabeled instances in CIFAR-10, Office-31, and VisDA2017 are set to 20,000, 400, and 20,000, respectively.
Firstly, we use CIFAR-10 to construct datasets with class distribution mismatch, which contains two scenarios: for subset mismatch, we choose the class from “0” ∼ “5” to form Cl, and the classes from “0” ∼ “8” to form Cu; and for intersectional mismatch, we choose the classes from “0” ∼ “5” to form Cl, and the classes from “3” ∼ “8” to form Cu. Then, we use Office-31 and VisDA2017 to create datasets with both class and feature distribution mismatch. Office-31 dataset contains three domains (A, D, W) and VisDA2017 contains two domains (Simulation, Reality). In the experiments, we choose the labeled data and unlabeled data from different domains to create a feature distribution mismatch. Particularly, in Office-31, we have six combinations of “labeled data domain/unlabeled data domain” including “A/D”, “A/W”, “D/A”, “D/W”, “W/A”, and “W/D”. For subset mismatch, we choose the classes from “0” ∼ “19” as Cl, and the classes from “0” ∼ “29” as Cu; and for intersectional mismatch, we choose the classes from “0” ∼ “19” as Cl, and the classes from “9” ∼ “29” as Cu. In VisDA2017, we choose the reality domain as labeled data domain, and simulation domain for unlabeled data domain. For subset mismatch, we choose the classes from “0” ∼ “8” as Cl, and the classes from “0” ∼ “11” classes to form Cu; and for intersectional mismatch, we choose the classes from “0”∼ “8” as Cl, and the classes from “3”∼ “11” classes to form Cu. For all experiments, we test on the dataset sampled from the same feature and class distribution as the labeled dataset.
Compared Methods. The compared methods include some popular closed-set SSL methods: PI [24], PL [26], MT [39], VAT [30], MM [3], FM [38]; and several existing open-set SSL methods: UASD [9], DS3L [19], and MTCF [45]. Moreover, we train the network only using labeled data to form the “Supervised” baseline.
2More details can be found in the supplementary material.
Implementation Details. We implement all methods in PyTorch and run all experiments on a single Tesla V100 GPU. We use ResNet-50 [20] pre-trained on ImageNet [34] as the backbone network. The batch size is set to 100 for CIFAR-10 dataset and 64 for other datasets. We adopt SGD optimizer with the initial learning rate 3× 10−4. The perturbation magnitude is set to 0.014 and the Beta distribution parameter α is set to 0.75.

heading: 4.2 Open-Set Evaluation
text: In this subsection, we first evaluate our method on CIFAR-10 dataset under class distribution mismatch which includes subset mismatch and intersectional mismatch. Then we testify the effectiveness on Office-31 and VisDA2017 datasets under feature distribution mismatch. Finally, we consider a more complex situation when both class and feature distribution mismatch exists in the open-set to validate the capability of our method in achieving UniSSL.
4.2.1 Only Class Distribution Mismatch Table 1: Averaged test accuracies (%) over three runs on CIFAR-10 with class distribution mismatch. The best results are highlighted in bold.
Method CIFAR-10
Subset Mismatch Intersectional Mismatch
Supervised 76.13 PI [24] 75.02 73.19 PL [26] 75.11 74.71 MT [39] 75.38 74.63 VAT [30] 76.07 75.25 MM [3] 79.08 78.43 FM [38] 80.19 80.01
UASD [9] 77.11 76.30 DS3L [19] 79.78 78.16 MTCF [45] 77.23 76.67
CAFA-PI (ours) 79.36 79.10 CAFA-FM (ours) 83.97 81.28 To testify the effectiveness of our method under class distribution mismatch with no need for a prior relationship between Cl and Cu, we create both subset mismatch and intersectional mismatch in CIFAR-10 dataset. The experimental results are shown in Table 1. We can see that our framework can largely enhance the simple PI method to achieve almost the best performances. Moreover, by using the strongest SSL method FM, our framework surpasses all other compared baselines with a large margin, which validates the ability of our method in handling both scenarios of class distribution mismatch.

heading: 4.2.2 Only Feature Distribution Mismatch
text: Then, we consider the feature distribution mismatch problem where the labeled data and unlabeled data are sampled from different domains when Cl = Cu. The experimental results are shown in Table 2. We can see that in this situation, almost all closed-set and open-set methods show severe performance degradation than the supervised baseline. However, our method using PI as the backbone method outperforms all other methods as well as the supervised baseline, which certifies that by eliminating the feature distribution gap, our feature adaptation is beneficial for boosting the learning performance of SSL.

heading: 4.2.3 Both Class and Feature Distribution Mismatch
text: Finally, we conduct experiments under both feature distribution mismatch and class distribution mismatch. Here the latter also includes two scenarios: subset mismatch and intersectional mismatch. The experimental results are shown in Table 3. We can see that our method using PI again outperforms all other methods and the supervised baseline, and achieves encouraging results in both scenarios. Hence, it validates that the proposed CAFA framework can universally deal with unknown circumstances and tackle both mismatch problems to achieve encouraging results.
Table 3: Averaged test accuracies (%) over three runs on Office-31 and VisDA2017 dataset with feature distribution mismatch and class distribution mismatch. The best results are highlighted in bold. The notation “A/D” denotes that the labeled data are from A domain and unlabeled data are from D domain.
Method Subset Mismatch Intersectional Mismatch
Office-31 VisDA2017 Office-31 VisDA2017 A/D A/W D/A D/W W/A W/D A/D A/W D/A D/W W/A W/D
Supervised 57.07 58.89 58.23 62.89 52.96 54.48 78.29 57.07 58.89 58.23 62.89 52.96 54.48 78.29 PI [24] 45.15 56.97 38.45 66.99 48.34 54.94 17.54 64.09 66.11 66.39 64.79 46.85 52.74 26.83 PL [26] 34.79 46.14 63.67 57.04 61.44 44.84 22.06 56.14 52.09 58.79 47.14 46.05 38.20 32.22 MT [39] 74.89 71.84 69.69 72.75 67.74 62.34 21.35 65.54 68.14 66.19 70.89 59.37 61.57 27.52 VAT [30] 26.19 28.89 49.89 57.24 49.36 41.14 35.56 23.64 27.50 40.04 43.54 23.45 32.66 19.67 MM [3] 53.80 57.06 54.34 49.45 61.41 55.97 70.32 59.78 59.23 62.50 61.41 55.97 47.82 66.34 FM [38] 68.74 69.34 60.64 52.88 63.39 55.62 83.17 66.99 64.12 62.19 65.44 57.93 55.76 85.57
UASD [9] 42.52 38.34 56.54 67.54 44.83 50.78 37.97 45.99 31.14 39.44 71.84 30.84 49.78 21.57 DS3L [19] 48.36 50.54 61.41 65.76 46.19 60.86 69.44 52.17 50.54 48.36 61.08 55.43 49.56 67.17 MTCF [45] 55.97 53.80 55.79 59.78 47.28 51.63 74.48 59.78 55.43 58.15 62.17 53.80 54.34 58.38
CAFA-PI (ours) 81.44 82.49 78.49 77.29 74.13 78.50 88.86 81.57 80.17 78.74 75.19 73.69 72.39 86.30
4.3 Performance Analysis
Figure 4: The probability density curve of wl and wu.
In this subsection, we conduct several analytical experiments using PI method as the backbone on the Office-31 dataset to evaluate our method.
Class-Sharing Scores. To testify the effectiveness of our class-sharing data detection, we plot the probability density curve of our class-sharing scores wl and
wu in Figure 4, we can see that the score distribution is clearly separable between class-sharing data and private data, thus validating our hypothesis in Eq. (2).
Averaged Weight. To validate the performance of our pseudo target calibration, we plot the averaged weight wavgc with respect to each class c in Figure 5 (a). We can see that wavgc is small when c is in Cl, and large when c belongs to C. Therefore, our pseudo target calibration is helpful to mitigate the learning bias for unlabeled data.
Robustness. Finally, we vary the number of private classes from 0 to 10 to verify the robustness of our method against
different levels of class distribution mismatch. Here we set the number of labeled private classes Cl equaling to unlabeled private classes Cu, hence the number of shared classes varies from 20 to 10. As shown in Figure 5 (b), we can see that our method surpasses all compared baselines with a large margin. Moreover, all compared methods show serious performance degradation when the number of private classes increases. However, our CAFA changes much flatter than other approaches, which indicates the robustness of our methods against various levels of class distribution mismatch.

heading: 5 Conclusion and Future Work
text: In this paper, we present a universal framework dubbed “Class-shAring data detection and Feature Adaptation” (CAFA) to solve different scenarios of open-set SSL problems with no need for any prior class knowledge. Particularly, we utilize a novel scoring mechanism that integrates the domain similarity and the label prediction shift to detect the data from the shared classes. Then we mitigate the feature distribution gap between the class-sharing data through domain adaptation to fully exploit the value of unlabeled data. Finally, we conduct semi-supervised training to properly learn consistent predictions between class-sharing labeled data and unlabeled data. Comprehensive experiments show the effectiveness of our CAFA framework on solving various open-set SSL problems. However, the proposed method is computationally expensive: during each round, our method requires an extra backward propagation to obtain the adversarial perturbation. Hence it is necessary to consider both effectiveness and efficiency in future work.

heading: 6 Acknowledgments and Disclosure of Funding
text: This research is supported by NSF of China (Nos: 61973162, U1713208), NSF of Jiangsu Province (No: BZ2021013), the Fundamental Research Funds for the Central Universities (Nos: 30920032202, 30921013114), CCF-Tencent Open Fund (No: RAGR20200101), Hong Kong Scholars Program (No: XJ2019036), RGC Early Career Scheme (No. 22200720), NSFC Young Scientists Fund (No. 62006202), HKBU CSD Departmental Incentive Grant, and “111” Program B13022.

Review summaries:
The paper proposes an approach for universal semi-supervised setting that handles both class distribution and feature distribution mismatch. The key idea of the CAFA approach lies in introducing a scoring mechanism that identifies shared classes in the labeled and unlabeled data. The objective function in CAFA consists of supervised cross-entropy loss on labeled data, adversarial loss for feature adaptation between labeled and unlabeled data and class sharing data exploration term on pseudo-labels that are calibrated by weighted softmax function where weights are computed for each class. The approach is validated on three datasets CIFAR-10, Office-31 and VisDA on three tasks: (1) only class distribution mismatch,(2) only feature distribution mismatch, and (3) both class and feature distribution mismatch.

This paper studies the universal semi-supervised learning, which combines traditional SSL and domain adaptation with the open-set problem. Under this setting, two issues are raised as the technical challenge, i.e., the class distribution shift and feature distribution shift. Accordingly, the CAFA is proposed in this paper, which identifies the class shared data among training and testing domain first, and then employs domain adaptation on these data for semi-supervised learning. Experiment results demonstrate the appilcability of the proposed method on several benchmark datasets.

The paper targets the universal semi-supervised learning problem where the class distribution and the feature distribution both mismatch between the labeled data and the unlabeled data.  The paper proposed a new framework Class-shAring data detection and Feature Adaptation (CAFA) to address the universal semi-supervised learning problem. The key insight is to design a scoring mechanism integrating domain similarity and label prediction shift to identify both the labeled and unlabeled data from the shared classes.  The paper conducts experiments on several datasets for semi-supervised learning and domain adaptation and shows competitive results on the semi-supervised learning setting with both class distribution mismatch and feature distribution mismatch. 

The paper investigates a realistic situation of open-set SSL, which includes the class distribution mismatch and the feature distribution mismatch. Moreover, it assumes the class relationship between labeled data and unlabeled data is unknown. The proposed method utilizes two scores to detect the data that belong to the common classes, and conduct domain adaptation to solve the feature mismatch problem.