Re-Evaluating the Impact of Unseen-Class Unlabeled Data on Semi-Supervised Learning Model

Authors: Rundong He, Yicong Dong, Lan-Zhe Guo, Yilong Yin, Tailin Wu
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Safe Semi-Supervised Learning, Unseen-Class Unlabeled Data

Abstract:
Semi-supervised learning (SSL) effectively leverages unlabeled data and has been proven successful across various fields. Current safe SSL methods believe that unseen classes in unlabeled data harm the performance of SSL models. However, previous methods for assessing the impact of unseen classes on SSL model performance are flawed. They fix the size of the unlabeled dataset and adjust the proportion of unseen classes within the unlabeled data to assess the impact. This process contravenes the principle of controlling variables. Adjusting the proportion of unseen classes in unlabeled data alters the proportion of seen classes, meaning the decreased classification performance of seen classes may not be due to an increase in unseen class samples in the unlabeled data, but rather a decrease in seen class samples. Thus, the prior flawed assessment standard that "unseen classes in unlabeled data can damage SSL model performance" may not always hold true. This paper strictly adheres to the principle of controlling variables, maintaining the proportion of seen classes in unlabeled data while only changing the unseen classes across five critical dimensions, to investigate their impact on SSL models from global robustness and local robustness. Experiments demonstrate that unseen classes in unlabeled data do not necessarily impair the performance of SSL models; in fact, under certain conditions, unseen classes may even enhance them.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729321199002

thread_type:
official_review

<thread_object>
creator: reviewer_wdk8

<content>
summary:
This paper proposes an evaluation framework for safe semi-supervised learning (SSL) with unseen class data. The paper observes that previous works fix the total number of seen and unseen class data, which may not reflect the true influence of unseen class data. Therefore, they fix the number of seen class data and change a number of quantities of unseen class data to investigate their influence. Extensive experiments are performed with much analysis.

strengths:
- The shortcomings of previous evaluation approaches are new to the SSL field.
- Many experiments are performed, providing a solid and comprehensive evaluation for robust SSL.
- The experimental analysis is also very comprehensive.
- The paper is well written and easy to understand.

weaknesses:
- My main concern is the practicality of the framework. The main purpose of the paper is to test the influence of *increasing numbers* of unseen class data on the classification performance on seen class data. The proposed metrics are mainly *one-order statistics* of the classification performance, which show the change in performance with more unseen class data. However, the main interest of SSL with unseen class data in real-world applications may not be focused on one-order statistics. In real-world applications, we only care about the performance of a certain number of unseen class data instead of the performance change with more data, although many papers provide the figures with different numbers of unseen class data. Therefore, although the framework is valid for different sizes of unseen class data, the practicability for real applications may be limited.
- Only CIFAR datasets were used for experiments. Since previous SSL benchmarks provide more datasets from different modalities, such as text or tabular data, it is beneficial to consider more text and tabular datasets.
- The standard deviations of the experimental results should also be included.
- Some typos should be checked. For example, in Eq. (1), it should be argmin. In Line 431, it should be Table 5.

questions:
- Are the experimental results in tables the average accuracies with multiple seeds?
- Is $C_{ib}$ the ratio between unseen and seen data?
- Is each entry in Table 6 the mean of several $C$, i.e. the mean of each row?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729603066100

thread_type:
official_review

<thread_object>
creator: reviewer_cs3q

<content>
summary:
The paper proposes an evaluation framework and then evaluates whether unlabeled samples from unseen classes deteriorate the performance of semi-supervised learning methods on seen classes.

strengths:
**S1.** The motivation of the paper is valid and interesting.

**S2.** The proposed evaluation framework follows the principle of controlling variables by fixing the confounding factor, i.e. the proportion of unlabeled samples from visible classes. So, the proposed evaluation framework overcomes the shortcomings of current evaluation schemes.

**S3.** The influence of unlabeled samples from unseen classes is evaluated from five perspectives and five evaluation indices.

**S4.** Some representative methods are chosen for experiment and the experiment results can provide some guidance for understanding the working mechanism of existing methods and designing new ones.

weaknesses:
In order to more clearly present the findings of the paper, the presentation needs to be further improved, specifically.

**W1.** Since $r$ (or $r_u$) is recorded in the tables in the experiment section, there is a precise definition for $r$. To reduce ambiguity, the formal definition of $r$ should be given based on the symbol system in Section 2. 

**W2.** It is necessary to provide a table or brief description for each indicator, showing its range, interpretation of positive/negative values, and how it relates to model performance.

**W3.** To facilitate readers to intuitively assess the degree of impact and facilitate cross dataset comparisons, the evaluation indicators in formulas (1)-(5) should to be normalized if possible.

**W4.** More experiment details should be recorded, such as the names of seen and unseen classes, the number of labeled samples for each class, and the number of unlabeled samples for each class. If there is not enough space, put it in the appendix.  Furthermore, the authors should provide a link to their code or dataset splits, if possible, to further enhance reproducibility.

**W5.** In addition, some clerical errors need to be corrected, specifically.

    a. In line 072, ”unseen classes in ---> “unseen classes in.

    b. In line 141, Fig.2(b) ---> FIg.2(a).

    c. In line 480-481, the table’s title should be above the table.

    d. In the experiment, how are the integrals in formulas (1) and (2) calculated? It seems that the integral operation should be the accumulation operation.

    e. In line 190, how can we obtain the “the expected accuracy”? In my opinion, it should be “the mean of ....”.

questions:
I hope the authors can clarify the following question.

**Q1.**  As mentioned in line 269-299, there are three experiments (generated by three random seeds) for each kind of dataset construction. So, the experiment results given in the paper are still with high randomness. So, it is difficult to accept these conclusions derived from these experiment results. In order to dispel this doubt, the authors should strengthen the experiment from the following aspects.

    a. The number of datasets should be increased, e.g., 10 or more.

    b. The diversity of datasets should also be increased, rather than being limited to image datasets.

    c. For each dataset construction (labeled and unlabeled samples setting),  more experiments should be carried out, e.g., 10 times or more.

    d. Some statistical significance tests should be introduced.

**NOTE: I am willing to increase my rating to a 6 or 8, if the authors can dispel my doubts through more adequate experiments.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1730373601401

thread_type:
official_review

<thread_object>
creator: reviewer_244r

<content>
summary:
This paper studies the problem of safe semi-supervised learning. It found that the previous works are based on flawed evaluations, which did not adhere to the principle of controlling variables. They fixed the size of unlabeled data but change the proportion of unseen classes, which would also affect the proportion of seen classes. To deal with this problem, the paper proposes a re-evaluation framework to keep the proportion of seen classes unchanged and adjust the proportion of unseen classes. Furthermore, the paper also propose five metrics to comprehensively evaluate the impact of unseen classes. Based on the proposed framework, extensive experiments are performed to evaluate the impact of unseen classes on various SSL models.

strengths:
1. The paper does not provide any new method. It gives an insightful perspective on the previous studies and finds that these methods have issues accurately assessing the impact of unseen classes. The motivation is very clear and seems to be reasonable. 

2. Based on the above motivation, the paper proposes to re-evaluate previous safe SSL models by fixing the proportion of seen classes. Furthermore, the paper proposes five evaluation metrics to access the impact of unseen classes on SSL models.

3. Extensive experiments based on the proposed framework are performed with various safe SSL methods. These results provide a comprehensive and fair comparison among existing methods.

weaknesses:
The main contribution of this paper lies in proposing a more reasonable experimental setup; however, its major issue is the credibility of the experiments. The experiments are conducted solely on CIFAR-10 and CIFAR-100, which makes the results unconvincing. While I understand that past studies has primarily been conducted on these two datasets, given that this paper aims to make innovations from an experimental perspective, it should propose more practical experimental setups and conduct experiments on more realistic datasets.

questions:
There are five evaluation metrics. Which one is the most important?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705729064

thread_type:
official_review

<thread_object>
creator: reviewer_mtt1

<content>
summary:
The paper investigates the impact of unseen classes in unlabeled data on the performance of semi-supervised learning (SSL) models. It challenges the prevailing assumption that unseen classes are detrimental to SSL performance, highlighting flaws in previous assessment methods that altered the proportion of unseen and seen classes simultaneously. This paper adheres to the principle of controlling variables by keeping the proportion of seen classes constant while varying unseen classes across five dimensions. Through rigorous experimentation, the authors demonstrate that unseen classes do not inherently degrade SSL model performance; in some scenarios, they may even enhance it. The findings suggest a reevaluation of the role of unseen classes in SSL, emphasizing that their effects are more nuanced than previously thought.

strengths:
1. This paper effectively identifies a flaw in previous methods for assessing the impact of unseen classes on SSL model performance. By fixing the size of the unlabeled dataset and adjusting the proportion of unseen classes, earlier approaches fail to control for variables appropriately. This adjustment alters the proportion of seen classes, meaning that any decrease in classification performance for seen classes may not be solely due to an increase in unseen class samples, but rather to a decrease in seen class samples. This clarification helps refine our understanding of how unseen classes affect SSL model performance.
2. The experiments presented in this paper are thorough, as the authors control various variables to validate their claims from different perspectives. This comprehensive approach strengthens the credibility of their findings.
3. The writing in this paper is clear and easy to understand, making it accessible to readers.

weaknesses:
1. In the proposed five metrics, some formulas are not clearly defined. For instance, the meaning of \(\hat{ACC}\) in Eq. 1 and \(\bar{ACC}\) in Eq. 2 is not explicitly explained. Providing clearer definitions for these terms would enhance the understanding of the metrics and their implications in the context of the research.
2. In the tables, it would be helpful to use bold text or dashed lines to highlight specific data points, emphasizing the advantages of certain algorithms under particular settings.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732475463741

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank the reviewers for their thorough and constructive comments. We are glad that the reviewers agree that our motivation is clear and interesting (all the reviewers mentioned). Our motivation is new to the SSL field (Reviewer wdK8 mentioned). Reviewers also pointed out that our experimental evaluation is solid and comprehensive (Mtt1, 244R, wdK8), our experimental analysis is very comprehensive (wdK8), providing some guidance for understanding the working mechanism of existing methods and designing new ones (CS3Q), and our presentation is clear (Mtt1, wdK8). 


Based on the reviewers' valuable feedback, we have conducted a number of additional experiments, which hopefully resolve the reviewers’ concerns. In this revised version, we have also updated the manuscript and Appendix, where we highlight modifications with blue color. The major additional experiments and improvements are as follows:

1. We expanded the number of datasets to include more datasets across various modalities, including image, text, and tabular data, to ensure a more comprehensive evaluation.
2. We provided a concise summary of each indicator with their range, interpretation, and significance for evaluating model performance.
3. We have provided clearer definitions for several terms to enhance the understanding of the metrics.
4. We have used bold text to highlight the best-performing results for each metric and setting.
5. We conducted experiments over 10 times to ensure reliability and robustness of the results.
6. We introduced statistical significance test (p-value) to rigorously validate the experimental findings, further enhancing the credibility of the analysis.
7. We have added the standard deviations and the updated results for more comprehensive evaluation of the experimental results.
8. To enhance reproducibility, we have provided an anonymous link to our code at [https://anonymous.4open.science/r/RE-SSL-F034/README.md] and also added more experimental details. Our code will be public once our paper is accepted.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper challenges the assumption that unseen classes in unlabeled data harm the performance of semi-supervised learning (SSL) models, arguing that prior assessments were flawed. The authors propose a re-evaluation of existing safe SSL models by fixing the proportion of seen classes and introduce five evaluation metrics to accurately assess the influence of unseen classes on SSL models, thereby addressing the shortcomings of current evaluation schemes. By controlling variables and adjusting only the proportion of unseen classes, the study demonstrates that unseen classes may not degrade SSL performance and can even enhance it under certain conditions.

The experiments conducted are extensive and thorough, providing a comprehensive analysis of various safe SSL methods, and the clear writing style makes the paper accessible to readers, enhancing the overall credibility of the findings. However, the reviewers also raised concerns about the credibility of the experiments, noting that conducting experiments only on CIFAR-10 and CIFAR-100 limits the scope and reduces the generalizability of the results. 

Based on the overall review, the paper is recommended for acceptance.