Robust Semi-Supervised Learning when Not All Classes have Labels
Authors: Lan-Zhe Guo
Yi-Ge Zhang
Zhi-Fan Wu
Jie-Jing Shao
Yu-Feng Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: semi-supervised learning
novel class discovery
robust

Abstract: Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data. Existing SSL typically requires all classes have labels. However, in many real-world applications, there may exist some classes that are difficult to label or newly occurred classes that cannot be labeled in time, resulting in there are unseen classes in unlabeled data. Unseen classes will be misclassified as seen classes, causing poor classification performance. The performance of seen classes is also harmed by the existence of unseen classes. This limits the practical and wider application of SSL. To address this problem, this paper proposes a new SSL approach that can classify not only seen classes but also unseen classes. Our approach consists of two modules: unseen class classification and learning pace synchronization. Specifically, we first enable the SSL methods to classify unseen classes by exploiting pairwise similarity between examples and then synchronize the learning pace between seen and unseen classes by proposing an adaptive threshold with distribution alignment. Extensive empirical results show our approach achieves significant performance improvement in both seen and unseen classes compared with previous studies.

Robust Semi-Supervised Learning when Not All Classes have Labels
Lan-Zhe Guo", Yi-Ge Zhang", Zhi-Fan Wu, Jie-Jing Shao, Yu-Feng Lit National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China {guolz, zhangyg, wuzf, shaojj liyf))lladdll nju.edu.com edu.
Abstract
Semi-supervised learning (SSL) provides powerful framework for leveraging unlabeled data. Existing SSL typically requires all classes have labels. However, in many real-world applications, there may exist some classes that are difficult to label or newly occurred classes that cannot be labeled in time, resulting in there are unseen classes in unlabeled data. Unseen classes will be misclassified as seen classes, causing poor classification performance. The performance seen classes is also harmed by the existence of unseen classes. This limits the practical and wider application of SSL. To address this problem, this paper proposes a new SSL approach that can classify not only seen classes but also unseen classes. Our approach consists of two modules: unseen class classification and learning pace synchronization. Specifically, we first enable the SSL methods to classify unseen classes by exploiting pairwise similarity between examples and then synchronize the learning pace between seen and unseen classes by proposing an adaptive thresh- old with distribution alignment. Extensive empirical results show our approach achieves significant performance improvement in both seen and unseen classes compared with previous studies.
Introduction
Machine learning, especially deep learning, has achieved great success in various tasks by leveraging sufficient labeled training data [21]. However, for many practical tasks, it can be difficult to attain number of labeled examples due the high cost of the data labeling process the widespread adoption of machine learning techniques.
41 23 which limits
Semi-supervised learning (SSL) 43 provides powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. There has been a rapid development SSL. methods in recent years, such entropy minimization methods 22 g. consistency regularization methods [24] 29 20 32). and holistic methods (30 [3) (3]
It has been reported that in certain cases, such as image classification [30). SSL methods can achieve the performance of purely supervised learning even when a substantial portion of the labels in given dataset have been discarded.
All the positive results of SSL, however, based each of the classes that one wishes to learn, i.e., all training and testing data are from seen classes that are observed in the labeled dataset. However, in many real-world applications, particularly those involving open-environmen scenarios 42 11, such an assumption is difficult to hold. For example, in the product recognition task, thousands of types of products are introduced to the supermarkets every once a while, and it would be expensive to label them all time; in the judicial sentencing task, some sentencing elements naturally scarce, resulting in labeled judgment documents being difficult to obtain for these elements in the training phase.
basic assumption that there are labels for
"Contribute to this work equally 1 'Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Training
Testing
Seen Class
Class
Unseen Class Unlabeled
Seen Class
Class Unlubeled Datu
Labeled Data
Unlabeled Data
Unlabeled
Cat
Cat
Novel
Dog
Dog
Novel 2
Figure I: SSL when not all classes have labels. Training data includes labeled examples from few seen classes as well unlabeled examples from both seen and unseen classes, Testing data includes unlabeled examples from both seen and unseen classes. The goal classify not only seen class examples into accurate classes but also partition unseen class examples into proper clusters.
It is evident that existing SSL methods will misclassify unlabeled examples from unseen classes into seen classes. Even the seen classes, SSL performance can also degrade severely with the presence of unseen classes unlabeled data 25
I0
40 The problem is related to open-set SSL and novel
36 23 aims decrease the negative impact of unseen classes and maintain the performance robustness of seen classes. However, these methods simply detect and drop examples from unseen classes and fail to classify them. NCD [3 38 aims to discover unseen classes automatically. However, they ignore the classification task on seen classes, which results in performance degradation seen classes.
class discovery (NCD) studies. Open-set set SSL 10
An illustration of the problem concerned in this paper is presented in Figure [ It is evident that both existing SSL methods and NCD methods could tackle the problem. This inspires us consider answering the following question in this study:
Can design robust SSL algorithm that can classify both and unseen classes when not all classes have labels in the training data?
To this end, we propose new SSI. method called NACH, which consists of two key modules: unseen class classification and learning pace synchronization. Specifically, we first propose a novel unseen class classification loss that can exploit pairwise similarity to classify similar example pairs into the same class and eliminate noisy pairs based on a similarity filter. We then adopt an adaptive threshold with distribution alignment to alleviate the issue that different learning paces between and unseen classes. Experimental results on CIFAR-10. CIFAR-100. 100, and ImageNet- 100 datasets show that NACH achieves 37.7% improvement in unseen classes compared with SSL methods, and 26.3% improvement in seen classes compared with NCD methods.
Related Work
Semi-Supervised Learning. SSL assumes all training and testing data are from seen classes, no matter whether labeled or unlabeled, and the goal is to classify unlabeled examples into the ground- truth classes. SSL has a long research history [4]. Our paper is mainly related to deep SSL, which introduces SSL techniques to deep neural networks and has achieved significant advancement in recent years. The mainstream of deep SSL can be broadly categorized into entropy minimization methods 122 H. consistency regularization methods 24 20 32] and holistic methods [30 2 M
When not all classes have labels in the training data, these methods will misclassify unlabeled data from unseen classes the seen classes and fail to address the problem concerned in this paper.
Open-Set Superviie Learning. Open-set SSL relaxes assumption of SSL and considers more practical scenario that training data could contain unseen class unlabeled examples. However, they still assume all testing examples are from seen classes, and the goal to decrease the negative impact of unseen class unlabeled data in order to maintain the performance robustness in

---

classes. Many open set SSL methods have been proposed in recent years [10] 6 36 31 28 16 26 I7I. such DS3L [TO], which assign weights unlabeled data based on a bi-level optimization. UASD 6 which filter unlabeled examples based on the prediction consistency, MTC [36, which adopt a multi-task curriculum learning framework to detect unseen classes and classify seen classes simultaneously, T2T [16], which propose novel pross-modal matching strategy to detect unseen classes. OpenMatch (28), which unify FixMatch algorithm with novelty detection based on one-vs-all (OVA) classifiers. However, these methods can still not classify unseen classes
Novel Class Discovery. NCD assumes training data consists of seen class labeled examples and unseen class unlabeled examples, and the goal is to classify both seen and unseen classes during the testing phase. The NCD problem first formally introduced [33] Recently, many NCD methods have been proposed based on a two-step training strategy [ 15 12,
39 (38. i... data embedding learned on the labeled data using a metric learning technique, and then fine-tuned while learning the cluster assignments on the unlabeled data. In contrast to the problem studied in this paper, NCD methods ignore the abundant seen unlabeled examples that are usually easy collect in real-world applications.
3 Preliminary and Background
Give the training data which contains labeled examples Di {(x1, .... (Xn, yn)] and unlabeled examples D, = (x,+1,'' ... Xn+m)- Usually, m >> n. x E R . E yy= ... CL} where D is the feature dimension and C2 the number of seen classes. We total number of classes in unlabeled data, Previous SSL studies assume CL = Cy and NCD assumes Cr n C. 0. In this paper, the number of seen classes Cseen = Cr. n Ct and the number of unseen classes Cunseen = Cu \ Cr.- The goal is to learn classification model s(x;0) from training data. Specifically. the f(x; 8) can be decomposed a representation learning model g(x: 0) R. Rd to learn a oow-dimensional feature and a classification model h(z): Rdd -> R. R.Ceen +uunzeen
Cu to represent the
The training loss of an SSL algorithm usually contains supervised loss L. and unsupervised loss Lu. with trade-off parameter >, 0: L, + >u where L, constructed D) and L,, is constructed on Du. Typically, L. applies the standard cross-entropy loss on labeled examples:
C. Σ H(y,,((x))
where p(x) = (f 0)) indicate the predicted probabilities produced by the model f for the input x, and (.,.) is the cross-entropy function.
Different constructions of the unsupervised loss L, lead to different SSL algorithms. Typically, there are two ways of constructing L,: one is to assign pseudo-labels to formulate "supervised loss" such as the cross-entropy loss, and another one is to optimize regularization that does not depend on labels such as consistency regularization.
Take the FixMatch 30 and UDA 34 for examples, FixMatch adopts the pseudo-label loss which can be written as:
n+m
C.
(max (p(a(x:))) 2) (pxx)(A(x:)))
17
where (xx) and A(x) indicate the weak and strong augmentation, y: = arg maxpp(a(x) represent the pseudo-label for unlabeled example x,, is the confidence threshold for pseudo-label selection, T(-) is the indicator function.
UDA 34 adopts the consistency regularization which can be written as
m C.
Σ 1=
llp(aug(x))) (xaaug(x))))
(3)
where aug(-) and aug'( (-) represents different augmentation strategics.
3

---

Filtere BCE Loss
Lubeled Data
Filher
Seen Class
Leurnal
Siunlarity
sinilarty
UC Loss
Eniropy
Unlabeled Date
distibbrtion
Scen Class
Throbbik
DTA Loss
Prodiction
Selenwe Prediction
Seleeted Ponedo lobel
Unseon Class
Alian
Prolictia of rogmestation
Aligned Preliction
Figure 2: Framework of the proposed NACH algorithm. The unsupervised loss of NACH can be decomposed into Lue and LDTA- The flowchart of how compute Luc and CDTA lies in the green box (top) and blue box (bottom), respectively.
The NACH Method
In this section, we present the proposed NACH method. The overall framework of NACH is illustrated in Figure Specifically, NACH consists of two main parts: unseen class classification, which contains new unsupervised loss to discover unseen classes automatically; ii) learning pace synchronization, which contains adaptive threshold with distribution alignment synchronize the different learning paces between seen and unseen classes. We first provide an overview of the learning objective. The concrete details of the objective are provided in the following.
4.1 NACH: An Overview for Two Problems
Previous SSL methods do not have the ability to classify unseen classes, which leads to number of examples from unseen classes being misclassified into seen classes when not all classes have labels. To address this problem, two main challenges need be considered.
The first one how to automatically classify unseen classes during model training. We propose to cluster unlabeled examples using the pairwise objective [12], and then adjust the results of the clustering using a known prior distribution. Specifically, we adopt the cosine similarity to find the most similar example situation where an example from seen classes and an example from unseen classes are wrongly paired, design similarity based filter to get rid of the appearance of such noisy pairs.
mini-batch for each example the positive pairs. To avoid the mismatched
The second is how to synchronize the different learning paces caused by different learning styles between seen and unseen classes. We propose metric to measure the difference between the learning status of seen unseen classes. Then, we formulate this metric into an adaptive threshold with distribution alignment. By selecting pseudo-labels with the adaptive threshold, the model can adjust the predicted probability adaptively to synchronize the learning differences between seen classes and unseen classes.
Moreover, we adopt the contrastive learning method SimCLR (5] pre-train the backbone network on the whole dataset in an unsupervised fashion in order to learn a good representation.
Overall, the objective of NACH consists of three parts: i) supervised loss L. for labeled data; ii) an unseen classes classification loss Lve by exploiting pairwise similarity to classify unseen classes; iii) unsupervised loss LDTA a by assigning pseudo-labels using the adaptive threshold achieve robust performance on both seen and unseen classes:
[ = L, + L + A2DDTA
(4)
where Xj and X2 are trade-off ceperrparaamters, which are all set in default.

---

4.2 Unseen Class Classification
To enable SSL the ability to classify unseen classes, one way is to adopt the binary cross-entropy (BCE) loss which can effectively exploit the pairwise similarity. The classical BCE loss can be written as:
m+n m++
1
LBC =
8ig log (x.) (xx)
(m + n)2
1=1 1=1
+ +1- 11--spp (4ب log p (x:) p pxx)))
where Sij is measure of the similarity between X; and Xj (e.g., cosine similarity). The first term aims to pull two similar examples closer, while the latter term aims push two dissimilar examples farther apart. Taking advantage of the BCE loss, we can achieve clustering of unseen classes by pairing similar examples.
This BCE loss is commonly adopted in NCD studies. However, in our study, directly using the BCE loss can not effectively classify unseen classes since both seen and unseen classes are the unlabeled data. Due to the different learning paces between seen and unseen classes, two examples belonging to the same unseen class are likely similarity. To address this issue, we neglect the push-apart strategy and adopt the cosine similarity to find the most similar example to pulled together for each example in a batch [.) Moreover, to further improve the pair accuracy, we propose new filter strategy to decrease the wrong pairs that consist of examples from seen and unseen classes.
pushed apart at the early training stage due their low
Specifically, given a batch of B labeled examples {(x),,)) e (1, B}} and batch of μB unlabeled examples {xī, 6 (I..... }BB} where μ determines the relative batch size of labeled and unlabeled data. For labeled examples, we can match pairs based on their ground-truth labels. For unlabeled example x we first find the example X which is the most similar to x according to their representation, then compute the cosine similarity between Xÿ and the batch of labeled examples: (cos(g(3) ( 9xx))) cos(g(X)) g(x))), ... cos(g(X))) 9xx))))))))))))))))))))))))))))) 9(xgg))) We sort the similarity descending order and obtain {di, d2. ... .d). where indicates the i-largest similarity. Consider the fact that if K is an example from unseen classes and x is an example from seen classes, then cos(g) xx)) (xx)) is likely to be less than some values d2, k E {1..... B). Based on this phenomenon, we can set threshold k, and retain the example x only when two examples xÿ and x5 satisfy cos (os(g(())) () (xx)) ) 2 dg. Then the filtered BCE loss (LFBCE) can be obtained as follows:
B log (p(x)) (T(xx,Tpp(())) p(x)))
1
CPBCE =
B
b=1
(6)
JB
1
(ccos(g(x)) (xx) g({E)) 2 de) log (p(xE) p(x)))
pB
Moreover, inspired by (3) regularize the predictive distribution of the pseudo-label to be close to a prior probability distribution P of labels y to prevent the model from classifying all unseen classes into one class and thus hindering the performance of unseen classes classification:
uB
CENT = KL
M 6=1
p(x)
xxxx)))((X))
(7)
B
The definition of Lue can be written as:
Luc LFBCE CENT
(8)
4.3 Learning Paces Synchronization
For the learning of seen classes, the supervision is from the ground-truth labels. However, for unseen classes, the model can only learn from the pairwise objective. This results in the learning pace of unseen classes being slower than seen classes. To maintain robust performance in seen classes, propose to synchronize the learning paces between seen and unseen classes. Previous
5

---

SSL algorithms, e.g,, FixMatch [30]. select pseudo- based on fixed confidence threshold. In this paper, we propose an adaptive threshold synchronize the learning paces adaptively. Different we assign different threshold for seen and unseen classes separately in order to synchronize their learning paces. First, we define metric to measure the difference in learning status between seen and unseen classes. We calculate the maximum classification confidence and the corresponding pscudo-label for each example:
from previous SSL methods with adaptive threshold [5 37 9
p max(p(a(xx))).
yi arg max(p((
Then we define the metric U as follows:
U = (N1 Σ
Neeen x,E2..cmm
p:) (
Nunseen
Σ
Pj
(10)
X,EAunsece
where eeen and Nunseen denote the total number of examples which are classified as seen classes and unseen classes. Xxeen refers to examples with pseudo-label belongs seen classes and Xunseen refers to examples with pseudo-label belongs to unseen classes.
U can effectively assess the degree of learning difference between seen classes and unseen classes. We applied U to adaptively adjust the confidence threshold for unseen classes. For seen classes we assume that the confidence threshold is then for unseen classes, we heuristically set the threshold to JU where 3 the trade-off parameters.
The above operation ensures that more examples predicted as unseen classes can be selected in the model training process. To further exploit the pseudo-label, we adopted an distribution alignment strategy. It should be noted that although CENT already takes into account distribution alignment, that is for all examples. Here we consider distribution alignment based on examples with confidence above the threshold.
Our goal is have the distribution of these above-threshold examples converge to known prior distribution to better learn the unseen classes, so we add distribution alignment as a fine-tuning for logits LDTA- Specifically, define:
P s(xeetect)
Σ Exea
[ 2 p(x(x))
(11)
Σ
3U) p(a(x;))
XyERunseen
We then require P(Xselect) and prior distribution Pιy) to be aligned:
Fals = log P(Xsetect/PP()
(12)
The adjustment factor Fals aims to align the distribution of selected data a prior distribution. We then compute the cross-entropy between prediction on strong augmented examples and the corresponding pseudo-label as the LDT loss:
LITA=
Σ
7. (3.,((()) + Fats)
(13)
X:ERscen EENEEE
where r, is for x; belongs to Xzeen and T for x, belongs to Xamecen. It is important to note that the adaptive thresholds and logit adjustment with distribution alignment factor are complementary: adaptive threshold adjustment designed for preventing the fixed confidence threshold from hindering the learning of unseen classes. Thus, adaptively adjust the threshold so that more examples of unseen classes could be learned by the model, which is beneficial to logit adjustment with distribution alignment. Distribution alignment factor for logit adjustment also makes the model less biased towards seen classes and thus facilitate the learning of unseen classes
5 Experiments
In this section, we give a comprehensive evaluation of NACH. Experimental results and detailed analysis reported to demonstrate the effectiveness of proposal.

---

Table Classification accuracy of compared methods on seen, unseen and all classes. The underline indicates the performance is worse than the baseline SSL methods.
SSL
Open-Set SSL
NCD
Classes
Dataset
Fixmatch
DS3L
CGDL
DTC
RankStats
ORCA
OURS
CIFAR-10
71.5
77.6
72.3
53.9
86.6
88.2
89.5
Seen
CIFAR-100
39.6
55.1
49.3
31.3
36.4
66.9
68.7
ImageNet-100
65.8
71.2
67.3
25.6
47.3
89.1
91.0
Average
59.0
68.0
63.0
36.9
56.8
81.4
83.1
CIFAR-10
50.4
45.3
44.6
39.5
81.0
90.4
92.2
Unseen
CIFAR-100
23.5
23.7
22.5
22.9
28.4
43.0
47.0
ImageNet-100
36.7
32.5
33.8
20.8
28.7
72.1
75.5
Average
36.9
33.9
33.6
27.7
46.0
68.5
71.6
CIFAR-10
49.5
40.2
39.7
38.3
82.9
89.7
91.3
All
CIFAR-100
20.3
24.0
23.5
18.3
23.1
48.1
52.1
ImageNet-
34.9
30.8
31.9
21.3
40.3
77.8
79.6
Average
34.9
31.7
31.7
26.0
48.8
71.9
74.3
5.1 Experimental Setup
Datasets. We evaluate NACH and compared methods on three SSL benchmark datasets CIFAR- 10, CIFAR- 100 [18] and ImageNet (27). Specifically, for the ImageNet dataset. 100 classes are sub- sampled following 33 3]. We first divide classes into 50% seen and 50% unseen classes, then select 50% of seen classes as the labeled data, and the rest as unlabeled data.
Compared Methods. We compare NACH with representative SSL, open-set SSL, and NCD methods. The SSL and open-set SSL methods extended to be applicable to unseen classes in the following way: examples are divided into known classes and unknown classes, we report their performance on classes and apply K-means clustering
class examples obtain clustering results. For SSL, the FixMatch (30 is adopted due to its empirical success and estimate unseen classes based on softmax confidence scores. For open-set SSL, we adopt two representative methods DS3L [10) which tries to assign lower weights to unseen classes unlabeled data, and CGDL [31] which automatically rejects unseen class examples. NCD methods are extended classify seen classes by using the Hungarian algorithm [19] to match some of the discovered classes with classes in the labeled data. Specifically, two NCD methods are employed: DTC [3] and RankStats [12] which have been reported to achieve the tatte-of-theaatd performance on NCD tasks. Moreover, we also compare
NACH with ORCA 3 methods, which consider similar setting with our paper.
All compared methods are implemented based on the pre-trained model using the contrastive learning algorithm SimCLR [5) The only exception is DTC which has its specialized pre-training procedure on labeled data [13]
Implementation Details. For CIFAR datasets, model is trained by using the standard Stochastic Gradient Descent method with a momentum of 0.9 and weight decay of 0.0005. We trained the model for 200 epochs with batch size of 512. Following [], we only update the parameters of the last block ResNet in the second training stage to avoid over-fitting. For the ImageNet dataset, is trained by using standard SGD with a momentum of 0.9 and weight decay of 0.0001. We train the model for 90 epochs with batch size of 512. For all experiments. the cosine annealing learning rate schedule is adopted. All experiments are performed on a single NVIDIA 3090 GPU.
ResNet-18 the backbone model. The
ResNet-50 the backbone model. The model
5.2
Main Results
The mean classification accuracy on CIFAR-10, CIFAR-100, and ImageNet- 100 dataset are provided in T From the results, it can be observed that open-set SSL methods can address the perfor- mance degradation problem on seen classes, but can not classify unseen classes accurately. NCD methods, e.g., RankStats, can improve the unseen classification performance, but suffer performance

---

70
60
Accuracy (%)
Accuracy (%)
50
88
40
86
All OURS
All OURS
Seen
ORCA
DURS Seen.
OURS Seen.
Uneam
OURS Uroczs.
OURSS Uracos.
10
0.5
0.5
Ratio of labeled data
Ratio of labeled data
(a) Classification accuracy on CIFAR-10 10.
(b) Classification accuracy on CIFAR-100.
Figure 3: Performance of NACH and ORCA with different numbers of labeled data.
degradation problem on seen classes, DTC performs even worse than SSL on unseen class classifica- tion. On the contrary, our proposal NACH not only classify unseen classes accurately but also maintain robust performance on seen classes. For example, NACH achieves 24.1% 1% improvement on seen classes and 34.7% on unseen classes compared with the FixMatch methods. Compared with ORCA, NACH also achieves significant performance improvement in both seen and unseen classes.
To further demonstrate the effectiveness of our proposal with varying label sizes, we evaluate the performance of NACH and ORCA with different numbers of labeled data, as shown in Figure The results show that the performance of NACH is always better than ORCA in all cases with significant margin. Morcover, the results also demonstrate that our proposal NACH is robust with label size, for example, with only 10% of labeled examples, the unseen-elass accuracy of our method still reach more than 89,4 and 37.5.
5.3 Detail Analysis
In this subsection, detailed analyzes are shown to help understand the superiority of our proposal, including analyzes of the two modules: Lue and LDTA, and hyper-parameter sensitivity analysis.
Analysis of Unseen-Class Classification Loss: Luc. We first show the effectiveness of the proposed FBCE loss by proposing basic model (BM) with Czeen + Cunseen classification heads and optimize the model by directly minimizing the standard BCE loss and our proposed FBCE loss separately. The comparison results are reported in Figure 4 including the ratio of wrongly selected seen-une unseen pairs and correctly selected unseen-unsee unseen pairs during the model training process. From the results, we can see that the proposed FBCE loss can effectively decrease unseen pairs while selecting more correct unseen-unseen pairs. This demonstrates the effectiveness of proposed filter strategy. We also study the effectiveness the Lue loss, the results are presented in Table 2 From the results, can the performance both seen and unseen classes can be improved by applying the Lvo loss.
ratio of wrongly selected seen-
Table 2: Analysis of Luc: classification accu- racy on CIFAR-100.
Table 3: Analysis of LDTA: classification accu- racy CIFAR-100.
Method
Seen
Unseen
All
Method
Seen
Unseen
All
FixMatch
39.6
23.5
20.3
UC Model
68.2
44.3
50.2
BM LBCE
72.8
28.3
31.5
UC Model + DA
68.6
44.8
50.3
BM LECE + LENT
67.2
44.2
50.7
UC Model DT
69.3
45.6
49.6
BM + Luc (UC Model)
68.2
44.3
50.2
UC Model + DTA
68.7
47.0
52.1
Analysis on Adaptive Threshold: LDTA- We compare the performance between the proposed
adaptive threshold and the statie threshold (e.g., - 0.95 in Fitch)hh), the results are reported in Figure 5 From Figure 5a
see that the prediction confidence between seen and unseen

---

45.
BCE
FBCE
33
Wrong pairs (96)
Correct pairs (%)
30
20
BCE
FBCE
15
125
130
200
25
200
Epuch
Epoch
Wrongly selected seen-unseen pairs
(b) Correctly selected unseen-unsecn pairs
Figure 4: (a) Wrongly selected seen-unst unn-uneeen pairs; (b) Correctly selected unseen-unsed unseen pairs. Both the above results are conducted on CIFAR-100 dataset.
classes are significantly different, thus, it is not proper to adopt a static threshold. Figure 5b and Figure
show show that the proposed adaptive threshold can help select more pseudo-labels for unseen class examples and improve the accuracy of pscudo-label assignment significantly. Results in Table give more clear ablation study demonstrate the effectiveness of our proposed CD A loss.
1.0
175
Adaptinc daessoli
52.5
150
State theabold
Confidence
0.6
Selected unsoom
45.0
claces
100
42.5
0.4
Ulocntaitty
35.0
Adgeiv drodeel Goolald
32.5
25
50
100
125
150
175
200
25
100 Epoch
125
150
I7s 260
Epech
Epoch
(a) Learning difference between
(b) Selected pscudo-labels for un- classes
(c) Accuracy of pseudo-labels for unseen classes
seen and unseen classes
Figure 5: (a) learning difference between seen classes and unseen classes during training; (b) number of pseudo-labels for unseen classes; (c) accuracy of pseudo-labels for unseen classes. All the above results are conducted on CIFAR- 100 dataset.
5.4 Parameter Sensitivity Analysis
Evaluating different i used in LFBCE- The intention of LFBCE is filter mismatched pairs containing examples from seen classes and unseen classes and the hyper-paramen parameter k determines the threshold of the noisy pairs filter. We provide the performance with different l in Figure From the results, we can see that, when A: 2. the proposal achieves the best performance and the performance does not degrade severely with è changes. This demonstrates that our proposal is quite robust. with the selection of k. Moreover, we also study how the wrongly selected pairs and correctly selected pairs changes with different k. The results show that, the filter strategy can decrease the wrong pairs significantly compared with no filter, with the decrease of & the wrong pairs the
all classes,
are decreased. Meanwhile, the filter strategy can also increase the correct pairs. When proposal achieves satisfying balance between the wrongly selected pairs and correctly selected pairs. so that the proposed method achieves the best performance.
Evaluating different 3 used in LDTA. We further analyze the impact of the adaptive threshold hyper-pane prperrparameter 3. /3 controls the margin of confidence threshold between seen and unseen classes. More pseudo-labels for unseen classes will be selected with a larger B. The results show the when the proposal achieves the best result and the performance does not degrade severely with changes. This demonstrate that the proposal is quite robust to the selection of 3.

---

Seco
Seen
Unsee
Wron pais
Ome paina
(N))
All
All
Ablotion ock
Abtation of
(a) Accuracy with different (b) Wrongly selected seen- (c) Correctly selected (d) Accuracy with different hyper-parameter k
parameter
unseen pairs
unseen-unseon pairs
hyper-parameter &
Figure 6: (a) accuracy with different k; (b) wrongly selected pairs with different k; (c) correctly
selected pairs with different k; (d) accuracy with different 33.
Conclusion
In this paper, we tackle an important and practical scenario of SSL., that is, SSL when not all classes have labels. We propose robust SSL. algorithm NACH that consists of an unseen class classification objective that can exploit pairwise similarity and eliminate noisy pairs, and an adaptive threshold with distribution alignment that synchronize the learning paces between and unseen classes. Extensive experiments clearly show the effectiveness of our proposal. The code is available at https://www lamda. nju. edu. cn/code coce_NACe.eaeecCCCCC ashx
How classify unseen classes with no labeled data is an important problem in SSL. Our work puts promising scheme in this direction. One limitation of our scheme is it does not have theoretical guarantees. We will put efforts into this direction in future work, such as giving generalization risk analysis on unseen classes.
Broader Impact
This paper studies the problem of SSL when not all classes have labels, which has been less investigated SSL. We hope this work can attract more future attention explore the robustness of SSL in more practical scenarios and promote SSL in wider applications.
Acknowledgment
This research was supported by the National Key R&D Program of China (2022YFC3340911, the National Science Foundation of China (62176118. 61921006), and the Huawei Cooperation Fund.
References
[1] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. In Proceedings of the 8th International Conference on Learning Representations, 2020.
[2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to remi-supervised learning. In Advances in Neural Information Processing Systems, pages 5050-5060, 2019.
[3] Kai-Dr Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In Proceedings of the 10th International Conference on Learning Representations, 2022.
[4] Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning. MIT Press, 2006.
[5] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference Machine Learning, pages 1597-1607, 2020.
10

---

[6] Yan-Bei Chen, Xia-Tian Zhu, Wei Li, and Shao-Gang Gong. der class distribution mismatch. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, pages 3569-3576, 2020.
learning un-
[7] Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9264-9272, 2021.
[8] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems, pages 529-536, 2004.
[9] Lan-Zhe Guo and Yu-Feng Li. Class-imbalanced semi-supervised learning with adaptive thresholding. In Proceedings of the 39th International Conference on Machine Learning, pages 8082-8094, 2022.
[10] Lan-Zhe Guo, Zhen-Yo Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In Proceedings of the 37th International Conference Machine Learning, pages 3897-3906, 2020.
[11] Lan-Zhe Guo, Zhi Zhou, and Yu-Feng Li. Record: Resource constrained semi-supervised learning under distribution shift. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1636-1644 2020.
[12] Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisser- man. Automatically discovering and learning new visual categories with ranking statistics. In Proceedings of the 8th International Conference on Learning Representations, 2020.
[13] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8401-8409, 2019.
[14] Yen-Chang Hsu, Zhao- Yang Lv. and Zsolt Kira. Learning to cluster in order to transfer across domains and tasks. In Proceedings of the 6th International Conference on Learning Representations, 2018.
[15] Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-class classification without multi-class labels. In Proceedings of the 7th International Conference on Learning Representationss 2019.
[16] Jun-Kai Huang, Chao-Wei Fang, Wei-Kai Chen, Zhen-Hua Chai, Xiao-Lin Wei, Peng-Xu Wei, Liang Lin, and Guan-Bin Li. Trash to treasure: Harvesting ood data with cross-modal matching for open-set superviid learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8310-8319, 2021.
[17] Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. In Advances in Neural Information Processing Systems, 2021.
[18] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, 2009.
[19] Harold W Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83-97, 1955.
[20] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Proceedings of the 5th International Conference on Learning Representations, 2017.
[21] Yann LeCun, Bengio Yoshua, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
[22] Dong-Hyun Lec. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, page 896, 2013.
[23] Yu-Feng Li, Lan-Zhe Guo, and Zhi-Hua Zhou. Towards safe weakly supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1):334-344 2019.
11

---

[24] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi- supervviee learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4 41(8):1979-199) 2019.
[25] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information P Processing Systems, pages 3239-3250, 2018.
[26] Alex Yuxuan Peng, Yun Sing Koh, Patricia Riddle, and Bernhard Pfahringer. Investigating the effect of novel classes in semi-supervised learning. In Proceedings of the 11th Asian Conference on Machine Learning, pages 615-630, 2019.
[27] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhi- Zhi-Heng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252, 2015.
[28] Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set semi-supervised learning with sptn-set consistency regularization. Advances in Neural Information Processing Systems, pages 25956 25967, 2021.
[29] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi eei-suuperviee learning. In Advances in Neural Information Processing Systems, pages 1 1163-1171, 2016.
[30] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and ChunLiang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems, pages 596-608, 2020.
[31] Xin Sun, Zhen-Ning Yang, Chi Zhang, Keck-Voon Ling, and Guo-Hao Peng. Conditional gaussian distribution learning for open set recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13480-13489, 2020.
[32] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, pages 1195-12044 1204, 2017.
[33] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In Proceedings of the European Conference on Computer Vision, pages 268-285, 2020.
[34] Qi-Zhe Xie, Zi-Hang Dai, Eduard H. Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, pages 6256-6268, 2020.
[35] Yi Xu, Lei Shang, Jin-Xing Ye, Qi Qian, Yu-Feng Li, Bai-Gui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In Proceedings of the 38th International Conference on Machine Learning, pages 11525-11536. 2021.
[36] Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi supervipee learning. In Proceedings of the European Conference on Computer Vision, pages 438 454,,,, 2020.
[37] Bo- Go-Wen Zhang, Yi-Dong Wang, Wen-Xin Hou, Hao Wu, Jin-Dong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In Advances in Neural Information Processing Systems, pages
2021.
[38] Zhun Zhong, Enrico Fini, Subhankar Roy, Zhi-Ming Luo, Elisa Ricci, and Nieu Sebe. Neigh- borhood contrastive learning for novel class discovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10867-10875, 2021.
[39] Zhun Zhong, Lin-Chao Zhu, Zhi-Ming Luo, Shao-Zi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories in an open world. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 9462-9470, 2021.
12

---

[40] Zhi Zhou, Lan-Zhe Guo, Zhan-Zha Cheng, Yu-Feng Li, and Shi-Liang Pu. Step: Out-of- distribution detection the presence limited in-distribution labeled data. In Advances in Neural Information Processing Systems, pages 29168-29 2168809800, 2021.
[41] Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1):44-55) 000444-3,,22 2017.
[42] Zhi-Hua Zhou. Open-environment machine learning. National Science Review, 2022.
[43] Xiao-Jin Zhu and Andrew B Goldherg. Introduction to semi-supervised learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 3(1):1-130, 2009.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releusing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "tGapPI2U_r6", "reviewer": "Reviewer_ZFnr", "summary": "This paper proposes a new SSL approach, that can not only classify seen classes but also classify unseen classes automatically. Specifically, the authors assume that the unlabeled data contains classes that have not appeared in the labeled data set. This is a practical scenario and it is meaningful to study this problem. To address it, the authors propose two important techniques: pairwise similarity based unseen class classification and adaptive threshold based pseudo-label selection. Experimental results on three datasets are reported, and the results demonstrate the effectiveness of the method.", "questions": "1. Can the proposal be applied to other SSL methods, e.g., MixMatch, ReMixMatch, etc. \n2. In some cases, the NCD methods perform worse than the baseline SSL method on unseen classes, why?\n3. If we simply replace the fixed threshold in FixMatch with the new proposed threshold, could the performance be improved on standard SSL tasks?\n\n", "limitations": "The authors addressed the limitations of their work.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe problem is novel. This paper focuses on a new problem, i.e., the unlabeled data contains unseen classes. Unlike previous robust SSL methods that discover and reject these examples, this paper tries to classify the unseen classes automatically. It is non-trivial to address this problem.\n2.\tThe proposed method is technically sound. It is a useful technique to adopt a pairwise similarity based objective to discover new patterns. Balance the learning pace between seen and unseen classes also seems reasonable.\n3.\tThe experimental results are convincing. Results on three benchmark datasets show the performance improvement of this proposal. Ablation studies are also convincing to show the usefulness of each technique.\n\nWeakness:\n1.\tIn my view, the unsupervised part is mainly based on the FixMatch algorithm. Can the proposed techniques be applied to other SSL methods?\n2.\tThere are some typos and grammar mistakes. I suggest the author check the paper writing and improve the representation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Responsible Research Practice (e.g., IRB, documentation, research ethics)"]}, "extra_text": {}}, {"review_id": "jOqR2xW1PQl", "reviewer": "Reviewer_yoL8", "summary": "This paper focuses on a new semi-supervised learning problem: not all classes have labels, i.e., there are some classes that have no one labeled example. The problem is important and different from previous SSL studies. The ability of SSL to classify unseen classes is useful in real-world applications since the labels of some classes may be difficult to collect in real tasks. The learning goal is to maximize performance in unseen classes and maintain safety in seen classes. The proposed method contains an unseen class discovery loss that exploits pairwise similarity to classify unseen classes and an adaptive threshold unsupervised loss to balance the learning between seen and unseen classes. The experimental results demonstrate the effectiveness of the proposal.", "questions": "1.\tHow to relax the prior knowledge about the number of unseen classes. In real-world tasks, it may be hard to know in advance.\n\n2.\tHow about the impact of $\\lambda_1$ and $\\lambda_2$? How to set the two parameters in real applications?\n", "limitations": "The authors addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThis paper studies a new problem in SSL, i.e, not all classes have labeled training examples. This is a practical setting and worth studying. In my view, this paper can inspire subsequent researchers and tackle SSL in more realistic scenarios.\n\n2.\tThe proposal is simple and effective. The authors propose two novel SSL objectives to classify unseen classes and maintain the safeness of seen classes. This can be simply integrated with existing SSL methods. Experimental results are convincing to demonstrate the effectiveness of the proposal.\n\n3.\tThe paper is well organized and easy to follow.\n\nWeaknesses:\n1.\tThe proposed method needs to know the number of unseen classes. It may limit the application of the proposal.  \n\n2.\tThe impact of the trade-off hyper-parameters $\\lambda_1$ and $\\lambda_2$ need to be discussed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Responsible Research Practice (e.g., IRB, documentation, research ethics)"]}, "extra_text": {}}, {"review_id": "aIp2wU--O1Z", "reviewer": "Reviewer_42nR", "summary": "This paper studies semi-supervised learning where unlabeled data contains unseen classes. Unlike previous open-set SSL methods and NCD methods, this paper aims to achieve high classification accuracy on both seen and unseen classes. This is an important problem and has not been well-studied. To address it, the authors propose a novel SSL approach called SU-SSL. The proposal exploits pairwise similarity to discover unseen classes and avoid performance degradation on seen classes via an adaptive threshold. Experimental results show that the proposal can achieve both performance improvement on seen and unseen class classification.", "questions": "1.\tHow does the performance vary with different ratios of unlabeled examples belonging to unseen classes?", "limitations": "The authors have provided a broader impact statement.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe problem studied in this paper is important and has not been well addressed. This paper focuses on semi-supervised learning where unlabeled data contains unseen classes. Unlike previous open-set SSL methods that only reject all unseen classes and NCD methods that only classify unseen classes, this paper aims to achieve high performance on both seen classes and unseen classes. This is an interesting problem and can promote SSL to be applied in more practical settings.\n2.\tThe proposed method is clear and technically sound. To maximize performance on unseen classes, the proposal adopts a novel unseen class classification loss that exploits pairwise similarity to classify similar sample pairs into the same classes. To maintain safeness in seen classes, the proposal adopts an adaptive threshold with distribution alignment to alleviate the different learning paces. These two techniques can address the problem effectively.\n3.\tThe experiments are comprehensive. On three commonly used benchmark datasets, the proposal achieves the best performance on both seen, unseen, and all classes. Representative SSL, open-set SSL, and NCD methods are compared. Ablation studies also show the usefulness of the proposed techniques. This clearly demonstrates the effectiveness of the proposal.\n\nWeaknesses:\n1.\tThe representation could be further improved. For example, there are both “unseen classes” and “unseen-classes” in the paper, this should be unified.\n2.\tIt would be better to study the impact of the ratio of unseen classes. For example, how the performance varies with different ratios of unseen classes unlabeled examples. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SYvbQFFAoUB", "reviewer": "Reviewer_jZK2", "summary": "This paper has proposed a new SSL method, Safe Unseen classification Semi-Supervised Learning (SU-SSL), which is able to classify unseen classes automatically while maintaining safeness on seen classes. Two main components are covered in this work: unseen class classification and adaptive threshold. Specifically, a novel unseen class classification objective used to exploit pairwise similarity and eliminate noisy pairs is proposed. Besides, a semi-supervised objective that adopts an adaptive threshold with distribution alignment to improve the performance on both seen and unseen classes is designed. Three commly-used datasets (i.e., CIFAR-10, CIFAR-100 and ImageNet-100) are covered in the experiments.", "questions": "See above.", "limitations": "Could this work be extended to other scenarios and tasks (e.g., semi-supervised object detection, etc.)?", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Paper Strengths:**\n>(1) This paper is easy to follow.\n\n>(2) The structure is clear and most of the figures are easy to understand.\n\n>(3) Some figures are interesting and intuitive.\n\n**Paper Weaknesses:**\n>(1) For one of the main contributions of this work (adaptive threshold), the novelty and motivation is not impressive. Note that the concept of adaptive threshold has been widely studied in existing SSL methods (e.g., [1], [2], etc.) and the solution of this work is relatively straightforward. Although [1] and [2] are covered in related work, it is not enough to just regard them as holistic methods without detailed descriptions. Please highlight the motivation of this contribution point. Besides, more comparison and analysis with related work should be added. \n\n>(2) This work attempts to address unseen classes on the basis of open set semi-supervised learning. This is similar to semi-supervised few-shot learning, which contains base classes (seen classes) and novel classes (unseen classes). It is valuable to explain and analyze for this. In addition, some experiments can be added if necessary.\n\n>(3) In addition to existing 'Analysis on Dynamic Threshold', some ablation studies and analyses about adaptive threshold can be considered to be added. For example, the advantages of dynamic threshold adjustment for specific SSL scenarios (unseen classes), comparison to other recently proposed methods using adaptive threshold, etc. Besides, It is suggested to give the specific threshold changes of each category and the visual results before and after using the adaptive threshold.\n\n>(4) Why use Imagenet-100 instead of directly use the complete Imagenet for large-scale dataset validation experiments?\n\n>(5) Many components are covered in Figure 2. This makes it difficult for me to master the details of the whole framework. More explanations may be helpful.\n\n>(6) Experimental evaluation seems not sufficient. More up-to-date state-of-the-art SSL methods should be compared. Note that only ORCA is recently proposed in Table 1.\n\n**References**\n\n[1] B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki, “Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling,” in NeurIPS 2021.\n\n[2] Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, Rong Jin, \"Dash: Semi-Supervised Learning with Dynamic Thresholding\", in ICML 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inadequate Data and Algorithm Evaluation"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept