Universal Semi-Supervised Learning
Authors: Zhuo Huang
Chao Xue
Bo Han
Jian Yang
Chen Gong
Venue: NeurIPS 2021 Poster (2021)
Keywords: Semi-Supervised Learning

Abstract: Universal Semi-Supervised Learning (UniSSL) aims to solve the open-set problem where both the class distribution (i.e., class set) and feature distribution (i.e., feature domain) are different between labeled dataset and unlabeled dataset. Such a problem seriously hinders the realistic landing of classical SSL. Different from the existing SSL methods targeting at the open-set problem that only study one certain scenario of class distribution mismatch and ignore the feature distribution mismatch, we consider a more general case where a mismatch exists in both class and feature distribution. In this case, we propose a ''Class-shAring data detection and Feature Adaptation'' (CAFA) framework which requires no prior knowledge of the class relationship between the labeled dataset and unlabeled dataset. Particularly, CAFA utilizes a novel scoring strategy to detect the data in the shared class set. Then, it conducts domain adaptation to fully exploit the value of the detected class-sharing data for better semi-supervised consistency training. Exhaustive experiments on several benchmark datasets show the effectiveness of our method in tackling open-set problems.

Universal Semi-Supervised Learning
Zhuo Huang' 1.2.3 Chao Xue Bo Han", Jian Yang!?? 1:2 Chen Gong' 1,2,3
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Ding Bigh-Dinensioal Information of Ministry of Education
2 JJiangsu Key Lab of Image and Video Understanding for Social Security,
School of Computer Science and Engineering, Nanjing University of Science and Technology $JD Explore Academy *Department of Computer Science, Hong Kong Baptist University {hzhuo, cs jyang, chen chen.gong) jujust..................uu edu. cn, xuechaol 90 jad.dddcoo com, bhanml @ (comp. akbu.eddt .hk
Abstract
Universal SemiiSupervisse Learning (UniSSL) aims to solve the open-set problem where both the class distribution (i.e., class set) and feature distribution (i.e., feature domain) are different between labeled dataset and unlabeled dataset. Such a problem seriously hinders the realistic landing of classical SSL. Different from the existing SSL methods targeting at the open-set problem that only study one certain scenario of class distribution mismatch and ignore the feature distribution mismatch, we consider a more general case where mismatch exists in both class and feature distribution. In this case, we propose a "Class-shAring data detection and Feature Adaptation" (CAFA) framework which requires no prior knowledge of the class relationship between the labeled dataset and unlabeled dataset. Particularly. CAFA utilizes a novel scoring strategy detect the data the shared class set. Then, it conducts domain adaptation to fully exploit the value of the detected class-sharing data for better semi-supervised consistency training. Exhaustive experiments on several benchmark datasets show the effectiveness method in tackling open-set problems.
Introduction
A critical drawback of training a good neural network [25] is that it typically requires lots of labeled data, which is quite difficult to satisfy due the unaffordable monetary cost as well as the huge demand for human resources. A popular way to solve such a problem is Semi- Semi-Supervised Learning (SSL) 7 which can effectively leverage scarce labeled data and abundant unlabeled data train an accurate classifier. However, classical SSL. [14] [5 16. 17 24 assumption that the labeled and unlabeled data are drawn from the same class distribution and the same feature distribution. To be concrete, the class set of labeled data C² is equal to that of unlabeled data C", and the marginal distribution p/(x) ) of the labeled data is identical the unlabeled feature distribution p" (x), which is shown in Figure 1 (a). However, in practice, the datasets at hand could greatly violate the above assumption by having both class distribution mismatch and feature distribution mismatch, which is denoted open-se In this situation, traditional closed-set SSL
39
41 49
relies on the closed-set
would suffer from severe performance degradation [1].
Recent works 4 9 19 problem extend SSL into the wild. For example, Guo al. (2020) [9] considers the subset mismatch situation when the classes in labeled data are only subset of the classes in unlabeled data,
29 31 45 have focused on dealing with the class distribution mismatch
Note that there are some other works A 45 also named as "open-set SSL". However. they all focus on the class distribution mismatch and do not deal with the feature distribution mismatch, their setting very different from ours.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

labeled data
unlabeled data
dosed--et
(a)
O
SSL
bike
(b)
bike
mag
(c)
open-se
SSL
bike
mug
Chair
distribttion
(d)
bike
mug
Figure 1: Problem illustration. (a) Closed-set SSL. (b) and (c) depict class distribution mismatch, where (b) describes subset mismatch; and (c) presents intersectional mismatch. (d) Feature distribution mismatch. In this figure, the dashed red box denotes feature distribution mismatch, and the solid green boxes mean the class distribution mismatch.
i.e. C C" and C" \ C' * (Figure 1 (b)). It utilizes meta learning scheme to weight
the unlabeled data that do not belong to C.. Chen et al. (2020) [9] deals with the intersectional mismatch problem where the labeled and unlabeled data both contain a shared class set but also hold private class set, respectively, i.e., C¹ n C Ø, C² (C" n C²) j ø and C" \ (C" n CC) + ø (c)) proposes self-distillation method to filter out the probable unlabeled private data. However, existing methods require prior knowledge of the relationship between C' and C", which greatly limits their realistic application. When the class relationship is unknown, the potential private data from D' and D" could both seriously mislead the learning process. Moreover, existing works only consider the class distribution mismatch and totally ignore the feature distribution mismatch problem. The latter problem is also quite common in practice, when try collect large amount of unlabeled data to aid the model training, the feature distribution of the newly obtained unlabeled data could be heavily influenced by when, where, and how collect them. As result, the potential feature distribution difference between labeled and unlabeled data could seriously harm the learning performance. Therefore, it is necessary design universal method solve different scenarios of class distribution mismatch and meanwhile deal with the feature distribution mismatch.
In this paper, propose Adaptation (CAFA)" which a Universal Semi- Semi-Supervised Learning (UniSSL) method for tackling different situations of open-set problem. Specifically, by considering that the labeled set and unlabeled set are drawn from different domains, we utilize a novel scoring mechanism to identify both the labeled and unlabeled data from the shared classes. The mechanism integrates two cues, namely domain similarity and label prediction shift, which are perfectly tailored for data detection in open-set SSL. Then we employ domain adaptation 6 40
framework dubbed "Class- shAring data detection and Feature
43 47 to match the identified unlabeled data to the same feature distribution of the class-sharing labeled data. After feature adaptation, the value of original unlabeled data can be fully exploited for boosting the learning performance. Moreover, we conduct weighted SSL to take full advantage of the class-sharing data from the open dataset. To sum up, our main contributions are:
We propose universal framework that can solve different scenarios of open-set SSL without any prior class knowledge.
Our method can fully exploit the value of unlabeled data by mitigating the feature distribution mismatch between labeled data and unlabeled data.
Experiments show that our method outperforms all other baselines in different situations of open-set problems.
2 Universal Semi-Supervised Learning
In our open-set SSL setting, we are given a labeled set D ((x,, w)) containing instances x; labeled with {v) |= and unlabeled set n { consisting of unlabeled instances x,- where & u. The two datasets D' and D are drawn from two different feature distributions (xx,) and y² (x))), respectively. We use Cl denote the labeled class set which contains the classes of labeled data, and employ C" to represent the unlabeled class set consisting of the classes of unlabeled

---

D
Sady
F
+
C
f
+
4-W
W
D
conv layer
fc layer
loss
data flow
scoring mechanism
Figure 2: The pipeline of our Class-sAAring data detection and Feature Adaptation (CAFA) approach.
data. Note that in setting, do not know the exact relationship between C' and C". Particularly, we use = n C" to denote the common class set shared by D' and D", and use C' and C" to denote the class private labeled data and unlabeled data, respectively. The feature distributions of labeled data with labels in C and č are denoted as (x;) and P' (x:), respectively, and the feature distributions of unlabeled data that belong to C and C" are denoted as p²(x,) and pe (x,). respectively.
Our goal is effectively identify the class-sharing data from both D and D and then eliminate the feature distribution mismatch between the identified labeled and unlabeled data to help train an accurate semi-supervised model in classifying the test data to the classes-of-interee C².
2.1 The General Framework of CAFA
CAFA contains a feature extractor F, classifier C, an adversarial discriminator D, and non-adversarial discriminator D', Given an input instance x, use F' compute its feature representation z (x)).. Then we employ C' to output the label prediction f using z. The non- adversarial discriminator D' produces a domain similarity score Was which quantifies the similarity degree of an instance to one distribution. The adversarial discriminator D aims to adversarially adapt the feature distributions of labeled and unlabeled data the common classes C.
As shown in Figure 2
The general framework of CAFA can be concisely formulated as:
min max (CC(())))))(( "EE xy w² w
OF De On
supervised fidelity
feature adapiation term
+ S1 31xxx~pp''" La (FF
class-sharing exploration term
in which 0F, Oc. 0D are the parameters of F, C, and D, respectively. In Eq. ( the first term is dubbed supervised fidelity term which involves the standard cross-entropy loss (-) The second term is dubbed feature adaptation term which introduces an adversarial learning loss Cadv to conduct feature adaptation the shassng data from D' and D Here the class-sharing data are detected through two scores w and w" which will be detailed in Section 2.2 Through such a feature adaptation procedure,
CAFA approach can maximally exploit the unlabeled data and benefit SSL. The third term refers to class-sharing data exploration term which conducts semi-supervised training using SSL. loss (.a(()) to make full use of class-sharing unlabeled data. Here the SSL. loss can be any regularizer in existing methods such as consistency regularizer 224 39 or manifold regularizer 13 44], and y, is C' -dimensional vector denoting the generated pseudo learning target for each unlabeled datum Xj. where the notation |-] indicates the size of the corresponding set. The parameters 7 and 8 are nen-negttive coefficients that trade off the above three terms.
From the general CAFA framework presented above, we can see that the class-sharing data detection, feature adaptation, and semi-supervised training are essential to our approach, and they will be
detailed in Sections 2.2 2.3 2.4 respectively.

---

pd
Existing approcchh
label pradiction shift
p
pa
Our approach
(a)
Figure 3: Illustration of our domain similarity and label prediction shift.
2.2 Class-Sharing Data Detection
Class-sharing data detection aims to correctly distinguish the training data belonging to C from those in č' U' To achieve this goal, we hope to model two class-sharing scores w (-) and w"(-) for labeled and unlabeled data, respectively, which should satisfy the following inequalities 43
E Xx w/ (x) > Exwpty (xx,
E Ex-ny wn (x) > Ex-py, wn (xx).
The above inequities should hold in large margin for better detection performance. Here we propose utilize two cues, namely domain similarity wa and label prediction shift w,, model wf and w
Domain Similarity has been employed to quantify whether an instance belongs to a specific domain by several methods (6 43 47 They usually train non-adversarial discriminator D' to predict the data from p' as and those from p" as I by minimizing a cross-entropy loss. The output value wa (F)))) can be considered as the domain similarity of the input Particularly, the input is likely to be sampled from p" if wa is large, and p' otherwise, formally
E Ex-pL (x(x), Exxpp wa(x) < Ex-pq wa(x), Ex-py (a(x).
(3)
However, such training strategy lacks exploitation on the middle region between two feature distri- butions. Consequently, it is prone to overfit to the situation when Ex-p' Wa(x) Ex-nt wa(x)
and Ex wx(x) Ex-ny Wa wx((x) (x) 2 1, thus making the class-sharing data unrecognizable, as shown in the upper of Figure (a). To solve this problem, we conduct Mixup [46] to strengthen the relationship between x; Pc and x, pK to yield discriminative domain similarities. Specifically, given two feature representations z; = F(x;) and 2j = F(x,) labeled datum x; and unlabeled datum x,, and their corresponding domain labels d, and d, 1, respectively, mixed feature representation žig and mixed domain label di, as following
generate
ب.j = Az; + (1 - A)zz: ر, =1 X
in which À is sampled from a Beta distribution Beta(a, a) where a is a hyper-parameter. Then we leverage the mixed feature representations with their domain labels by adding an extra binary cross-entropy term our domain similarity loss Ldom, which is formulated as
Ldom Exx log(1 - D' (x.(()))) - Ex,~~= log (F((x)))
(5)
+E Exx upp x, ~p" [sim (2;, 2,) (-(1 - log (22..) - Alog(1 D'(
where sim(z,, 2,)
x x, 1| ||
denotes the cosine similarity between and zy. Based reasonable
assumption that z; and z, in C are closer to each other in the feature space than those in c¹ and C". such extra term weighted with cosine similarity can focus interpolating the middle region between the two feature distributions P/c and pž. which helps preventing the aforementioned overfitting problem and making the domain similarity of class-sharing data closer each other than the
private data in č and T" as shown in the lower panel of Figure 3 (a). Therefore, we can have Exeup! wx(x) < Exxng wa(x) and SxmpgWaaXx < Ex-pg. (x)... By combining Eq. (3), we have
Expupy wx(x) Expop wa(x) E Ex wx(x) < Exxpg. wa(x).
(6)

---

The above inequality would yield a larger margin than traditional training strategy when equipped with the Mixup training. Thereby, our domain similarity can better detect the class-sharing data in the open-set situation. Nevertheless, the domain similarity alone is not sufficient for class-sharing data detection. Therefore, we introduce label prediction shift to enhance the detection performance.
Label Prediction Shift indicates the influence imposed by adversarial perturbation on each in- stance, which can successfully differentiate the class-sharing data from private data, as shown in Figure 3 (b). Formally, given an input instance x, its label prediction can be denoted as
f = [ fz(x),.... fie (x)] where {f((x)) |c|| can be interpreted as the probability that x belongs to class i. Then we apply an adversarial perturbation on x to obtain the perturbed version of input instance x*
Sxign(-VxCae(x... max (I.)), f((x))), where controls the pertur- bation magnitude. As a result, the adversarial perturbation would decrease the maximum probability of the given input. Then, the label prediction shift can be computed
Wi max fi(x) max fx(x").
iEC!
sec!
As result, the computed label prediction shift would satisfy the following inequality:
Ex-ppt Ws (x) < [Ex-p="'s<> <
Ex~rt wx(x) < E Ex=pf_ (x()).
(8)
Intuitively, the learning scarce labeled data is strongly dependent the supervised cross-entropy
loss Lee, while the unlabeled data that are learned with consistency regularization are much more
robust against perturbations [24 30 39], thus natural to have Ex-pp (x)x), Exxpy (eeenW,,(* (x) < Ex-py (x)x), Ex (x)).. Moreover, the abundant unlabeled data have the effect of improving the model generalizability SSL. In open-set situation, the model generalizability on the class set c' is greatly limited since there are only scarce labeled private data available, which makes such kind of data vulnerable against perturbations. On the contrary, the model learning on the classes is quite sufficient when compared with č since there are both labeled and unlabeled data that can be leveraged. Hence, we have Ex-rt W. (x)x) Ex-ply (x)x). Additionally, the unlabeled private
data in ~ do not belong to any known classes and completely lie out of any known distribution. As mentioned in [27]. the adversarial perturbation would have less influence on their maximal label predictions than those data in C². Therefore, we can have Ex-pg. (x) Exxpp w. (x). Based on the above explanations, the inequality in Eq. (8) should hold.
To integrate the proposed two cues wa and w, based on Eqs. through
(6 and we can compute w and w
w/ (x) = w.(xx) (x)x), x D
w" (x) = wx(x) ) - wa(x), x E D
Note that wa and are both normalized into interval [0, 1] before computation. Through Eq. (9) our class-sharing scores can perfectly satisfy Eq. (2). hence they are effective for detecting the class-sharing data from both D' and "
2.3 Feature Adaptation
After detecting the class-sharing data in the above subsection, now we should eliminate the feature distribution mismatch between p and pa such that the value of unlabeled data can be properly extracted to aid the subsequent SSL. To this end, we treat the labeled data target domain (i.e., d, = 0, X4 D') and the unlabeled data as the source domain (i.e., dj = 1, E D)), and conduct adversarial domain adaptation [5] o [2 40 47 to achieve this goal. Particularly,
apply the class- sharing scores w' and w" to the adversarial learning loss Lade, and train the adversarial discriminator D distinguish the labeled and unlabeled data. Meanwhile, the feature extractor F is trained to deceive D. The above adversarial process is formulated as the following min-max game:
max min Ex pp' ~ Ladv (xi, xj; Op) -xx,pp' w/ (x:)) log(1 D(F(x:))
Or On
(10)
- -xx up w (x,) log D(F(x;)).
Thanks to the two class-sharing scores w² and w", we can successfully mitigate the feature distribution mismatch between P'e and På without being influenced by the irrelevant distributions p²u and pK... As we will show later in the experiments, our feature adaptation can re-discover the value of unlabeled data and boost the SSL performance.
5

---

2.4 Semi-Supervised Training
With the aforementioned class-sharing data detection and feature adaptation, we can take full ad- vantage of the open-set information by alleviating the negative impact from both class distribution mismatch and feature distribution mismatch. Then, we should aim at effectively exploring the class- sharing unlabeled data, meanwhile weakening the negative impact from private data. Particularly,
the private data in c' could mislead the unlabeled data transferring to the wrong classes, and the unlaheled private data in C could be erroneously incorporated into network training, causing further performance degradation. To solve this problem, we propose the following SSL training strategy:
min w" (x) (CFx((()()))
(11)
where (x) is employed to weaken the network learning on unlabeled private data, and y indicates the calibrated pseudo target for each unlabeled datum to mitigate the misleading bias introduced by labeled private data. To calibrate the original biased pscudo target y, we propose to utilize a weighted softmax function. Particularly, we compute the average weight of w' with respect to each class
I(:: - c) w (xi), CEC.
(12)
Based on Eq. (2). the computed weight wang would be large if is in C, and be small if e is in ट. Then we can calibrate the pseudo target y through
w exp lyle
[
ceC'
(13)
E
wi o0g cxp [y];
where the notation H.] denotes the e-th entry of the input vector. Through such process, the entries of ý belonging to C' would be suppressed, and those belonging to C would be enhanced, which successfully alleviates the bias from the original target y.
To sum up, our general framework can be instantiated by substituting Eq. (00) and Eq. into the feature adaptation term and class-sharing data exploration term Eq. Later experiments will show that our CAFA framework can effectively tackle different scenarios of the open-set problem without any prior knowledge of the class relationship and achieve encouraging performances.
Related Work
3.1 Closed-Set SSL
Closed-set SSL deals with the problem when C = C and p'(x) (1xx)>>"xxx (x) Early closed-set SSL
methods such as Entropy Minimization [8] and Pseudo-Label (PI.) [26] enforce the networks to
make confident predictions on unlabeled data. Later, consistency-bases methods such as IT-Model
(PI) 24), Temporal Ensembling (24,, and Mean Teacher (MT) (39) conduct consistency training
between temporally spatially different models. After that, Virtual Adversarial Training (VAT) [30 computes adversarial perturbations which maximally change the input image data to enhance the model robustness. Recent methods mostly rely on data augmentation to improve the network generalizability. For instance, MixMatch (MM) 3 and ReMixMatch (2) employ the Mixup [46) to augment the training data as well as the label information, which is beneficial to network training. FixMatch (FM) [38] utilizes the label predictions of weakly augmented image data to guide the learning of strongly augmented image data and achieves state-of- wi--hhe-n art performance. However, as mentioned before, closed-set SSL cannot perform satisfactorily in practice. Thereby, open-set SSL is designed to extend SSL into the wild.
3.2 Open-Set SSL
Open-set SSL is new topic in SSL which aims to tackle the problem when C¹ + C and (x(x) * pxx)). Laine & Aila (2016) 24) and Oliver et al. (2018) [3T] firstly raised the class distribution mismatch problem in open-set SSL, which is further investigated by many subsequent methods. For example, Uncertainty Aware Seff-Distillation (UASD) 9 deals with the intersectional mismatch by proposing a eelf-distillation method to filter out the probable unlabeled private data. Then Safe Deep Semi-Supervised Learning (DS3L) [19] and Multi- Task Curriculum Framework (MTCF) [45 solve the subset mismatch by employing different weighting strategies to down-weight the unlabeled

---

private data. Recently, Cao et al. (2021) 4 uses contrastive learning 8 42 to separate the unlabeled private data from the class-sharing data in the subset mismatch problem. Existing open-set SSL methods made some significant attempts toward more practical setting. However, when the feature distribution mismatch between labeled and unlabeled data presents, almost all existing methods would fail.
3.3 Domain Adaptation
Domain adaptation aims to transfer knowledge from the labeled source domain to the unlabeled target domain, which is generally divided into four categories: 1) closed-sel domain adaptation assumes that the class sets of the source and target data are the same. Existing methods [10] [T] 12, 28 40 48
aim learn class discriminative and domain invariant features from source and target domains; 2) partial domain adaptation considers that the class set of source data is larger than that of target data. Some methods [5|6]] 47] aim to apply class-level weight on each source datum to achieve per-class distribution matching; 3) open-set domain adaptation denotes the situation when the target domain contains private classes. Busto et al. (2017) [32 trains SVMs to identify the target private classes. Saito et al. (2018) (37) employs extra logit to the classifier to incorporate the unknown target classes. Zhuo et al. (2019) [50] proposes to leverage word vectors to recognize the open domains; and 4) universal domain adaptation requires no prior knowledge of the class relationship between source and target domains. You et al. (2020) 45 utilizes the domain knowledge and the entropy value to find the data in the shared classes. Saito et al. (2020) (36] employs neighborhood clustering (2 to
help the feature alignment between the class-sharing source and target data.
4 Experiments
In this section, we first specify the implementation detail:" in Section 4.1
Then, to thoroughly validate the proposed CAFA approach, we conduct extensive experiments under different scenarios of open-set SSL by comparing our method with popular closed-set methods as well as several existing open-set methods in Section 4.2 Finally, we present the detailed performance analysis of our method
in Section 4.3
4.1 Experimental Setup
Datasets. We use CIFAR-10 23 Office-31 (35] and VisDA2017 33 to evaluation our method.
We denote the 10 classes from CIFAR-10 as "0" "9", the 31 classes from Office-31 as "0" ~ "30", and the 12 classes from VisDA2017 as "0" ~ "11". The numbers of labeled instances in CIFAR-10, Office-31, and VisDA2017 are to 2,400, 100, and 1,800, respectively, and the number of unlabeled instances in CIFAR- 10, Office-31, and VisDA2017 are set to 20,000, 400, and 20,000, respectively.
Firstly, we use CIFAR- to construct datasets with class distribution mismatch, which contains two scenarios: for subset mismatch, we choose the class from "0" ~ "5" to form C², and the classes from "0" ~ "8"" to form C"; and for intersectional mismatch, we choose the classes from "0" ~ "5" to form C², and the classes from "3" ~ "8" to form C". Then, we use Office-31 and VisDA2017 to create datasets with both class and feature distribution mismatch. Office-31 dataset contains three domains (A, D. W) and VisDA2017 contains two domains (Simulation, Reality). In the experiments, we choose the labeled data and unlabeled data from different domains create feature distribution mismatch. Particularly, in Office-31, we have six combinations of "labeled data domain/unlabelle data domain" including "A/D". "A/W", "D/A", "D/W". "W/A", and "W/D". For subset mismatch. we choose the classes from "0" ~ "19" as C². and the classes from "0" ~ "29" as C"; and for intersectional mismatch, we choose the classes from "0" ~ "19" C². and the classes from "9" "29" as C" In 2011 we choose the reality domain as labeled data domain, and simulation domain for unlabeled data domain. For subset mismatch, we choose the classes from "0" "8" as C², and the classes from "0" ~ "11" classes to form C*; and for intersectional mismatch, we choose the classes from "0" ~ "8" C², and the classes from "3" "11" classes to form C For all experiments, we test on the dataset sampled from the same feature and class distribution the labeled dataset.
Compared Methods. The compared methods include some popular closed-set SSL methods: PI (24), PL 26). MT (39], VAT (30), MM (3), FM [38]; and several existing open-set SSL methods: UASD (. DS3L 19) and MTCF [45). Moreover, we train the network only using labeled data to form the "Supervised" baseline.
Mooe details can be found in the supplementary material.
7

---

Table 2: Averaged test accuracies (%) over three runs on Office-31 and VisDA2017 dataset with feature distribution mismatch. The best results are highlighted in bold. The notation "A/D" denotes that the labeled data are from A domain and unlabeled data are from domain.
Method
Office331
VisDA20017
A/D
A/W
DVA
D/W
W/A
W-D
Supervised
57.07
58. 89
58. 23
62.89
52.96
54.48
78.29
49.29
75.71
71.83
68.74
05.94
52:97
33.39
52.89
34.32
18.40
MT 20
69.34
70.49
60.65
65.39
64.40
66.34
20.12
VAI 3V
16.19
31. 11.85
25.04
38.89
35.51
30.32
16.89
23.34
41.45
33.89
31.42
40.09
34,12
67.58
FM
69.77
70.02
60.29
62:50
09.61
85.78
UASD
43.20
50:32
47.22
DS3L
19
$5.97
47.22
53.26
51.08
38.95
52:71
60.28
MICF
45
38.99
42.93
46.19
36.95
40.76
47.28
06.08
CAFA Pl (ouns
81.97
83.57
79.04
76.44
74.59
80.48
91.02
Implementation Details. We implement all methods in PyTorch and run all experiments on single Tesla V100 GPU. We use ResNet-50 (20] pre-trained on ImageNet 34 as the backbone network. The batch size is set to 100 for CIFAR-10 dataset and 64 for other datasets. We adopt SGI) optimizer with the initial learning rate 3 10-4. The perturbation magnitude is set to 0.014 and the Beta distribution parameter is set to 0.75.
4.2 Open-Set Evaluation
In this subsection, we first evaluate our method on CIFAR-10 dataset under class distribution mismatch which includes subset mismatch and intersectional mismatch. Then we testify the effectiveness on Office-31 and VisDA2017 datasets under feature distribution mismatch. Finally, we consider more complex situation when both class and feature distribution mismatch exists in the open-set to validate the capability of our method in achieving UniSSL.
4.2.1 Only Class Distribution Mismatch
Table 1: Averaged test accuracies (%) over three runs CIFAR-10 with class distribution mis- match. The best results are highlighted bold.
To testify the effectiveness of our method under class distribution mismatch with no need for a prior rela- tionship between C and C". we create both subset mismatch and intersectional mismatch in CIFAR- 10 dataset. The experimental results are shown in Table We can see that our framework can largely enhance the simple PI method achieve almost the best per- formances. Moreover, by using the strongest SSL method FM, our framework surpasses all other com- pared baselines with large margin, which validates the ability of our method in handling both scenarios of class distribution mismatch.
CTFAR-10
Method
Subset Mismatch
Intersectioal Mismarch
Supervised
76.13
PL 26
75.11
74.71
MT 39
75.38
74.65 63
VAT
76.07
75 25
79.08
FM
80.19
80.01
UASD
77.11
78.30
DS3L
1
79.78
78.16
MTCP g5
77.23
76.67
CAFA- (oum))
79.38
79.10
CAFA-FM (ours)
81.28
4.2.2 Only Feature Distribution Mismatch
Then, consider the feature distribution mismatch problem where the labeled data and unlabeled data are sampled from different domains when C¹ = C". The experimental results are shown in Table 2 We see that this situation, almost all closed-set and open-set methods show severe performance degradation than the supervised baseline. However, our method using Pf as the backbone method outperforms all other methods as well as the supervised baseline, which certifies that by eliminating the feature distribution gap. our feature adaptation is beneficial for boosting the learning performance of SSL.
4.2.3 Both Class and Feature Distribution Mismatch
Finally, we conduct experiments under both feature distribution mismatch and class distribution mis- match. Here the latter also includes two scenarios: subset mismatch and intersectional mismatch. The experimental results are shown in Table We can see that our method using PI again outperforms all other methods and the supervised baseline, and achieves encouraging results in both scenarios. Hence, supervise
validates that the proposed CAFA framework can universally deal with unknown circumstances and tackle both mismatch problems to achieve encouraging results.
8

---

Table 3: Averaged test accuracics (%) over three runs on Office-31 and VisDA2017 datasct with feature distribution mismatch and class distribution mismatch. The best results are highlighted in bold. The notation "A/D" denotes that the labeled data from A domain and unlabeled data are from D domain.
Subset Mismasch
Incerscctional Mismarch
Method
Office-31
ViaDA3017
Office-31
VidDA2017
A/W
W/A
W/D3
A/W
D/A
W/1D
Superiised
67.U7
288889
62.89
02.99
04.48
78.22
07.07
08.89
02. 125
78.29
P1 [5]
45.15
56.97
38.45
66.99
48.34
64.34
17.54
64.00
66.11
66.39
64.79
46.85
52.74
26.83
PL. 80
34.79
46i.14
63.67
37.04
01.44
44.84
22.06
56.14
02.09
08.79
47.14
46.05
36.20
32.22
50
74.89
71.84
67.74
02.34
21.35
68. 16.144
06.19
09.37
63.67
27.52
VAT
26.19
57.24
49.36
41.14
35.56
23.64
27.50
40.04
43.54
23.45
19.67
MM
63.80
57 .0G
54.34
49.45
61.41
05.97
70.32
59.78
69:23
62.50
61.41
55.97
87.82
06.34
FM 3K
68.74
69.34
60.64
52.58
63.39
55.62
63.17
66.99
64.12
62.19
65.44
37.
56.76
85.37
UASD 3
42.52
38.34
56. 54
67.54
44.83
50.78
37.97
45.99
31.14
39.44
71.84
30.84
49.78
21.57
DSAL 1
60.64
69.44
00.54
48.36
55.43
67.17
MICF
65.97
53.80
53.79
39.78
47.28
51.63
74.48
09.78
05.43
58.15
02:17
53. 53.80
54.34
58.38
CARA-PI (ours) 81.44
82.49
78.49
77.29
74.18
78.50
88.86
81.57
80.17
78.74
75.19
73.69
72.89
86.30
4.3 Performance Analysis
Labede Instances (%)
Date
Unisleted
Labeted Clae-Skuring Data
Unlaheted Instances
Unbileted Chos-Sariig Datn
In this subsection, we conduct several analytical experiments using Pl method
as the backbone on the Office-31 dataset to evaluate our method.
Class-Sharing Scores. To testify the effectiveness of our class-sharing data detection, plot the probability density curve of our class-sharing scores w/ and w" in Figure we can see that the score distribution is clearly separable between class-sharing data and private data, thus validating our hypothesis in Eq. (2)
Laheled Class-Sharing Score w/
Unlabeldd Class-Shuring Soore W
Figure 4: The probability density curve of w' and w".
Averaged Weight. To validate the per- formance of our pseudo target calibra- tion, we plot the averaged weight wang
Avergged Weight werd
E Accuracy
with respect to each class e in Figure [
(a). We can see that wan is small when
CAFA
is in d', and large when belongs to C. Therefore, our pseudo target calibration is helpful mitigate the learning bias for unlabeled data.
#Prtvate Classes (c*) = (CD
(a) The values of w over C¹, We provide the aver- age value over three runs with standard deviation. (b) The test accuracies with respect to different numbers of private classes.
Figure 5:
Robustness. Finally, we vary the num- ber of private classes from to 10 to ver- ify the robustness of method against
different levels of class distribution mismatch. Here we set the number of labeled private classes ج equaling to unlabeled private classes C", hence the number of shared classes varies from 20 to 10. As shown in Figure 5 (b), we can see that our method surpasses all compared baselines with large margin. Moreover, all compared methods show serious performance degradation when the number of private classes increases. However, our CAFA changes much flatter than other approaches, which indicates the robustness of our methods against various levels of class distribution mismatch.
5 Conclusion and Future Work
In this paper, we present a universal framework dubbed "Class-shAring data detection and Feature Adaptation" (CAFA) to solve different scenarios of open-set SSL problems with no need for any prior class knowledge. Particularly, we utilize novel scoring mechanism that integrates the domain similarity and the label prediction shift detect the data from the shared classes. Then we mitigate the feature distribution gap between the class- sharing data through domain adaptation to fully exploit the value of unlabeled data. Finally, we conduct comi-super training to properly learn consistent predictions between class-sharing labeled data and unlabeled data. Comprehensive experiments show the effectiveness of our CAFA framework on solving various open-set SSL problems. However, the proposed method is computationally expensive: during each round, our method requires an extra backward propagation to obtain the adversarial perturbation. Hence necessary to consider both effectiveness and efficiency in future work.

---

6 Acknowledgments and Disclosure of Funding
This research supported by NSF of China (Nos: 61973162. U1713208), NSF Jiangsu Province (No: BZ2021013), the Fundamental Research Funds for the Central Universities (Nos: 30920032202. 30921013114), CCF-Tencent Open Fund (No: RAGR20200101), Hong Kong Scholars Program (No: XJ201 9036). RGC Early Career Scheme (No. 22200720), NSFC Young Scientists Fund (No. 62006202), HKBU CSD Departmental Incentive Grant, and "111 11" Program B13022.
References
[1] Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research (JMLR), 7(Nov):2399-243) 2006.
[2] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv: 1.09785, 2019.
[3] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, pages 5049-5059, 2019.
[4] Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. arXiv preprint
2021.
[5] Zhangjie Cao, Lijia Ma, Mingsheng Long, and Jianmin Wang. Partial adversarial domain adaptation. In ECCV, pages 135-150, 2018.
[6] Zhangjie Cao, Kaichao You, Mingsheng Long, Jianmin Wang, and Qiang Yang. Learning to transfer examples for partial domain adaptation. In CVPR, pages 3985-29944 2019.
[7] Olivier Chapelle. Bernhard Scholkopf, and Alexander Zien. Semi supervvee learning. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 20(3):542- 5422, 2009.
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607. PMLR, 2020.
[9] Yanhei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In AAAI, 2020.
[10] Shuhao Cui, Xuan Jin, Shuhui Wang, Yuan He, and Qingming Huang. Heuristic domain adaptation. In NeurIPS, 2020.
[11] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, pages 1180-1189, PMLR, 2015.
[12] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarie training of neural networks. Journal of Machine Learning Research (JMLR), 17(1):009--000 2030, 2016.
[13] Bo Geng, Dacheng Tao, Chao Xu, Linjun Yang, and Kian-Sheng Hua. Ensemble manifold regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 3266):1227 2012.
[14] Chen Gong, Tongliang Liu, Dacheng Tao, Keren Fu, Enmei Tu, and Jie Yang. Deformed graph laplacian for semisupervised learning. IEEE Transactions on Neural Networks and Learning 2015.
Systems (TNNLS), 26(1):2221111
[15] Chen Gong, Dacheng Tao, Keren Fu, and Jie Yang. Fick's law assisted propagation for semisupervised learning. IEEE Transactions Neural Networks and Learning Systems (TNNLS), 26(9):2148-2122 2014.
[16] Chen Gong, Dacheng Tao, Wei Liu, Stephen Maybank, Meng Fang, Keren Fu, and Jie Yang. Saliency propagation from simple to difficult. In CVPR, pages 2531-2539, 2015.
10

---

[17] Chen Gong, Dacheng Tao, Stephen J Maybank, Wei Liu, Guoliang Kang, and Jie Yang. Multi- modal curriculum learning for sunerviseee image classification. IEEE Transactions on Image Processing (TTP), 25(7):3249-322 -2600 2016.
[18) Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NeurIPS, pages 529-536, 2005.
[19] Lan-Zhe Guo, Zhen- Yu Zhang, Yuan Jiang. Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In ICML, 2020.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[21] Jiabo Huang, Qi Dong, Shaogang Gong, and Xiatian Zhu. Unsupervised deep learning by neighbourhood discovery. In ICML, pages 2849-2858, PMLR, 2019.
[22] Jingchen Ke, Chen Gong, Tongliang Liu, Lin Zhao, Jian Yang, and Dacheng Tao. Laplacian welsch regularization for robust semisupervised learning. IEEE Transactions on Cybernetics (TCYB), 2020.
[23] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[24] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2016.
[25] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436444 2015.
[26] Dong-Hyrn Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop, 2013.
[27] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- distribution image detection neural networks. In ICLR, 2018.
[28] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In NeurIPS, 2018.
[29] Huixiang Luo, Hao Cheng, Yuting Gao, Ke Li, Mengdan Zhang, Fanxu Meng. Xiaowei Guo, Feiyue Huang, and Xing Sun. On the consistency training for open-set semi cemi-supervised learning. 2021.
arXiv preprint
[30] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and cupissuper learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 1979 2018.
[31] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In NeurlPS, pages 3235- 3246, 2018
[32] Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In ICCV, pages 754-763, 2017.
[33] Xingchao Peng, Ben Usman, Neela Kaushik, Dequan Wang, Judy Hoffman, and Kate Saenko. Visda: A synthetic-to-real benchmark for visual domain adaptation. In CVPRW, pages 2021- 2026, 2018.
[34] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)211-252 2015.
[35] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV, pages 213-226. Springer, 2010.

---

[36] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. In NeurIPS, 2020.
[37] Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In ECCV, pages 153-168, 2018.
[38] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang. Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07688, 2020.
[39] Antti Tarvainen and Harri Valpola. Meanteachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurlPS, pages 1195- 1204, 2017.
[40] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, pages 7167-7176, 2017.
[41] Sheng Wan, Shirui Pan, Jian Yang, and Chen Gong. Contrastive and generative graph convolu- tional networks for graph-based sumi-supervised learning. In AAAI, 2020.
[42] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, pages 3733-3742, 2018
[43] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Universal domain adaptation. In CVPR, 2020.
[44] Bing Yu, Jingfeng Wu, Jinwen Ma, and Zhanxing Zhu. Tangent-normal adversarial regulariza- tion for semi-supervised learning. CVPR, pages 10676-10684, 2019.
[45] Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi-supervised learning. In ECCV, 2020.
[46] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2017.
[47] Jing Zhang, Zewei Ding, Wanqing Li, and Philip Ogunbona. Importance weighted adversarial nets for partial domain adaptation. In CVPR, 2018
[48] Qiming Zhang, Jing Zhang, Wei Liu, and Dacheng Tao. Category anchor-guided unsupervised domain adaptation for semantic segmentation. In NeurIPS, pages 433 033-443, 2019.
[49] Dengyong Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In NeurlPS, pages 321-328, 2004.
[50] Junbao Zhuo Shuhui Wang, Shuhao Cui, and Qingmign Huang. Unsupervised open domain recognition by semantic discrepancy minimization. In CVPR, 2019.
12


Reviews:
[{"review_id": "gKb7T6FRnDn", "reviewer": "Reviewer_v3uq", "summary": "This paper studies the universal semi-supervised learning, which combines traditional SSL and domain adaptation with the open-set problem. Under this setting, two issues are raised as the technical challenge, i.e., the class distribution shift and feature distribution shift. Accordingly, the CAFA is proposed in this paper, which identifies the class shared data among training and testing domain first, and then employs domain adaptation on these data for semi-supervised learning. Experiment results demonstrate the appilcability of the proposed method on several benchmark datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThis paper tries to unify SSL and domain adaptation in a single framework. The originality of is OK. But authors seem to miss an important reference which proposes the unsupervised open domain recognition problem:\n\n[1] Unsupervised Open Domain Recognition by Semantic Discrepancy Minimization. In CVPR, 2019.\n\nThe key difference between CAFA and [1] lies in the usage of data that does not belong to the shared class. In CAFA, these data is assumed to be discarded, while in [1] these data are encouraged to be predicted to the categories (only in target domain) with equal probability. Authors should discuss the difference between CAFA and UODTN regarding to the problem setting. \n\nQuality and clarity:\nThe quality and clarity of this paper is good.\n\nSignificance:\nI think this paper has revealed a good point towards the development of SSL. However, the relation among SSL, unsupervised domain adaptation and other related problem setting (e.g., semi-supervised domain adaptation), should be discussed in an comprehensive way.\n\nDetailed comments:\n\na. In CAFA, the authors choose to discard those data that does not belong to the shared class. This point still need to be justifed in a proper manner, since it has been proved that these private data can be used to enhance the prediction diversity as in [1] and [2].\n\n[2] Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations. CVPR, 2020.\n\nAuthors should compare CAFA with these methods on these methods under similar setting.\n\nb. How does the wrongly detected class shared data affect the accuracy?\n\n------\nI have read the author response, which addressed my concerns appropriately. I stick to my original score and vote for acceptance on this paper.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See the detailed comment.", "review_text": "", "extra_scores": {"ethical_concerns": "There seems no ethical concern on this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "dpthTcWzTsC", "reviewer": "Reviewer_3QaJ", "summary": "The paper targets the universal semi-supervised learning problem where the class distribution and the feature distribution both mismatch between the labeled data and the unlabeled data.\n\nThe paper proposed a new framework Class-shAring data detection and Feature Adaptation (CAFA) to address the universal semi-supervised learning problem. The key insight is to design a scoring mechanism integrating domain similarity and label prediction shift to identify both the labeled and unlabeled data from the shared classes.\n\nThe paper conducts experiments on several datasets for semi-supervised learning and domain adaptation and shows competitive results on the semi-supervised learning setting with both class distribution mismatch and feature distribution mismatch.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThe problem setting tackled in this paper is quite practical and general for realistic applications. Since the unlabeled can be collected from any resource, there is potential that the unlabeled data have both different class distribution and feature distributions from the labeled data. Also, the problem setting is much more challenging than prior works with no prior knowledge of class distributions.\n\nThe paper addresses the overfitting problem of domain discriminators by training on mixup samples, which generates more discriminative domain similarities. \n\nLabel prediction shift with the adversarial perturbation can reflect the quality of the prediction.\n\nThe experiments are conducted on both semi-supervised learning dataset and domain adaptation datasets, which can demonstrate the effectiveness of the proposed method on the setting with feature distribution shift.\n\nWeakness:\n\nIn Eqn. (1), the semi-supervised learning loss is performed on all the unlabeled data. As demonstrated in [14,25], if performing semi-supervised learning loss on all the unlabeled data, the performance degraded severely.\n\nNo variance is reported in the experiment results.\n\nOn line 109, both samples from p^u and p^l are labeled as '0'. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No obvious negative social impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Y-voBCyaduS", "reviewer": "Reviewer_3d3x", "summary": "The paper proposes an approach for universal semi-supervised setting that handles both class distribution and feature distribution mismatch. The key idea of the CAFA approach lies in introducing a scoring mechanism that identifies shared classes in the labeled and unlabeled data. The objective function in CAFA consists of supervised cross-entropy loss on labeled data, adversarial loss for feature adaptation between labeled and unlabeled data and class sharing data exploration term on pseudo-labels that are calibrated by weighted softmax function where weights are computed for each class. The approach is validated on three datasets CIFAR-10, Office-31 and VisDA on three tasks: (1) only class distribution mismatch,(2) only feature distribution mismatch, and (3) both class and feature distribution mismatch.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The proposed universal SSL setting is very realistic and there is no method proposed specifically for this setting so far.\n- The way scoring mechanism for labeled and unlabeled data is computed is interesting and provides insights about relationships of weights between shared and private classes of labeled and unlabeled data.\n- Proposed approach significantly outperforms SSL methods.\n\nCons:\n- The proposed approach is very complex and consists of so many design choices that are not clearly motivated and experimentally validated. Each design decision is supported only by an intuitive explanation why it should work. \n- Why is L_ssl weighted by w_u when it does not consist of only private but also shared classes? \n- What is the effect of pseudo-target calibration by w_c^avg?  While there is an experiment showing the average weight of classes in Fig 5, there is no ablation study that shows the effect on the performance.\n- In domain similarity loss authors add an extra binary cross entropy term to help prevent overfitting but mixed feature representations will be generated from both domain-private and non-private classes so it is not clear what is the effect of this besides intuitive explanations that feature representations of shared classes should be closer.\n- There are many parameters that are not explained how they should be scheduled and what is their effect on the performance. For example, why are \\gamma and \\delta in (1) scheduled as exp(-5(1-min(iter/n,1))^2)? Why is n 8000 for \\gamma and 4000 for \n\\delta? How do different values affect performance? Similarly, how to schedule \n the parameter \\lambda (i.e. alpha in Beta distribution)?\n- How is the separation of the private classes in the unlabeled data guaranteed, or this can not be guaranteed at all and the approach can only guarantee to separate them from shared classes?  This should be clearly explained.\n- Where is domain loss in eq (5) utilized? Eq (1) consists of L_{CE}, L_{adv} and L_{ssl}, but it is not clear where domain loss is used.\n- Are the accuracies in Tables 1 and 3 reported only on the shared set of classes? If that is the case, does it mean that the authors report accuracy on only 3 classes for intersectional mismatch on the CIFAR-10?\n- CIFAR-10 is quite a simple dataset, and it would be desirable to report performance on a more challenging CIFAR-100 dataset.\n- Why do the authors consider 9 classes for CIFAR-10 instead of 10 and 30 classes instead of 31 on the Office dataset?\n- In Figure 5b is this experiment done on CIFAR-10 dataset? Why is the performance of other baseline so low in the case when there are no private classes? How are methods evaluated when all classes are private?\n- Lack of baselines. For feature distribution mismatch it is desirable to compare to the state-of-the-art partial and universal domain adaptation methods such as DANCE (Saito et al. Neurips ‘20) and ETN (Cao el al. CVPR ‘19)\n\nOther comments:\n- There is a mistake in the domain similarity paragraph. Non-adversarial discriminator D’ should predict data from p^l as 1 not 0.\n- There is a typo in the number of examples of the Office dataset. I believe it should be 4000.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "KrSlzZcDl1", "reviewer": "Reviewer_A1eA", "summary": "The paper investigates a realistic situation of open-set SSL, which includes the class distribution mismatch and the feature distribution mismatch. Moreover, it assumes the class relationship between labeled data and unlabeled data is unknown. The proposed method utilizes two scores to detect the data that belong to the common classes, and conduct domain adaptation to solve the feature mismatch problem.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contributions are: 1) to the best of my knowledge, this paper is the first work to investigate the domain difference between labeled and unlabeled data in SSL (in both label and feature domains), and it shows that conducting domain adaptation can address such problem; 2) the proposed domain similarity training which performs Mixup is novel; 3) this work shows that the adversarial perturbation can be utilized to perform OOD data detection, which is novel in open-set SSL.\n\nStrengths: \n- The problem setting is quite realistic, and the experimental results are also encouraging. I feel that this work will promote the application of semi-supervised learning to more real-world problems.\n- The writing of this paper is good, and the details of algorithms are clearly stated.\n- The proposed method is a general framework that can be deployed to many existing methods. \n\n\nWeaknesses: \n- The proposed method is a bit complex. Since it conducts both Mixup training and adversarial training in each round, the method could be computationally expensive. \n- Can the proposed method outperform other SSL methods in the closed-set SSL setting? \n- Is there any deeper analysis on why the closed-set SSL methods would fail in such a feature distribution mismatch setting? \n- Since this work is strongly related to domain adaptation (the subset mismatch is related to partial domain adaptation, the intersectional mismatch is related to open-set domain adaptation, and the proposed setting is related to universal domain adaptation), it would be nice to discuss the difference between this problem and domain adaptation. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It seems that the proposed method is a bit complex. Besides, it would be better if the authors can discuss the relationship between the universal semi-supervised learning discussed in this paper and universal domain adaptation.\nOther limitations have been indicated in the above weakness.\n\nI think this work will attract intensive attention as this work studies a situation which is quite close to realistic applications. This work paves the way for applying semi-supervised learning to solving real-world problems.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies universal semi-supervised learning problem. In this setting, class distribution and feature distribution both mismatch between labeled and unlabeled data. The authors propose an algorithm CAFA and demonstrate its effectiveness on several benchmark data sets. The reviewers agree that the paper provide new insight on this problem and the setting is of practical importance. Good paper. I recommend accept.
Decision: Accept (Poster)