Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposed a novel setting for SSL where unlabeled data contains novel categories and it aims to simultaneously train classifiers for seen classes and discover novel classes. This is achieved by introducing pairwise loss and regularizing with prior class distribution. Experiments are carried out on synthesized data splits and demonstrated good results.

main_review: Strength:

This paper proposed a new setting for SSL. Simultaneously training classifier for seen classes and discovering novel classes from unlabeled data is novel.

Comparison with existing methods adapted to the new setting is competitive.

Weakness:

The expected number of novel classes is a strong assumption for open-world SSL. Inferring the number of classes is only briefly mentioned in the paper. I would like to see more details.

The uncertainty in Eq(4) is not explained well. I am wondering why other choices, e.g. entropy, is not selected. Moreover, is the max_k is among the known classes or over all classes? If it is over all classes I do not see how the classifier W_k, k\in Cu is trained.

I do not see any term that can avoid assigning unlabeled data to seen classes. For example Eq(5) can still be minimized if all unlabeled pairs are assigned to the same seen classes.

Regularizing the average posterior w.r.t. a prior distribution is somehow a strong assumption in that the distribution for unseen classes are known in advance.

Pre-training could substantially improve the quality of feature representation. As a result, the pair-wise objective might heavily rely on the pre-training. I think it is worth discussing or providing experiments to validate this point.

summary_of_the_review: This paper provides a new perspective into SSL. But the designs are not fully explained and some remains unjustified, e.g. the prior distribution for all classes and the number of novel classes.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of open-world learning, where the labeled data are from seen classes and unlabeled data are from both seen and novel classes. To address this problem, this paper presents an Uncertainty Based Adaptive Margin method for learning the joint classifier of seen and unseen classes. The proposed method can avoid the model assigning samples of novel classes to the seen classes. Experiments on several datasets show the effectiveness of the proposed method.
main_review: Pros:

+ This paper is well-written and easy to follow. The motivation is clear and the overall organization is good.

+ A new setting is proposed in the community of semi-supervised learning, which jointly considers semi-supervised learning, open-set learning, and novel class discovery. This is a difficult but very practice problem in the real world.

+ A simple but effective approach, Uncertainty Based Adaptive Margin, is proposed to address the introduced setting. It can avoid the model quickly converge the seen classes so that the model can well recognize both seen and unseen classes.

+ Extensive experiments are provided to verify the effectiveness of the proposed method. In addition, this paper makes great contributions to implement existing SSL and novel class discovery methods into open-world learning.

Cons:

- This paper states that the proposed method can estimate the number of unseen classes. However, it seems that it should first estimate a rough number with DTC. How about the results of set a larger number of unseen classes (e.g., 200 for CIFAR100), learn the proposed model and then estimate the number of unseen classes?

- I appreciate that the authors implemented existing SSL and novel class discovery methods on the proposed setting. However, their implementation details are not very clear to me. Please introduce how to reproduce them in the supplementary.

- For the estimation of uncertainty, why not use other methods? Such as the entropy, and MC-Drop [A]? In addition, for the fixed negative margin, does 0.5 produce the best results? If not, the authors should compare the proposed method with different values of the negative margin.

- The proposed pairwise loss and regularization loss are not novel. We can find many semi-supervised learning and clustering methods using these two losses.

- In addition, for the pairwise loss, this paper selects the nearest neighbor as the positive candidate. However, in practice, this could introduce many false-negative pairs when the number of classes is large and the training batch size is small. I think this will be an issue for the proposed method especially when there are many classes. For example, the proposed method may meet problems when learning on the large ImageNet which has 1000 classes. Indeed, I think this is a long-standing problem that our community should consider and this may not be addressed in this paper.

- For self-supervised learning, I would like to see the results of using different self-supervised learning methods, such as SimCLR, MOCOV2, and RotationNet. It is very interesting that [B] found that RotationNet achieves better results than SIMCLR and MOCOV2 for novel class discovery. Therefore, I am curious about that if this phenomenon happens in the proposed setting.

- I think the main contribution is the proposed adaptive margin method. Therefore, this approach can be applied to other novel class discovery methods. How about applying the proposed margin method to RankingStatic?

- I think one important result is missed in the paper. We assume that the samples of novel classes and samples of seen classes are separated in the unlabeled data. That is, for the unlabeled data, we know which samples are from novel classes or seen classes but we do not know the class labels. I think this is could one upper bond for the proposed setting. In addition, under such an assumption, can the proposed method improve the results? Or, we only need simple pairwise loss and regularization loss (also RankingStatics)?

- Another two important experiments are missing. (1) Applying existing open-set methods for the introduced setting. (2) Applying the proposed method on the open-set recognition setting.

- At last, I strongly recommend the authors release the source code during submission. (1) This is the first work that studies the open-world setting. (2) I know that this paper was submitted to several venues and many researchers know this paper/setting. I also found several works that follow this setting and compare this work. Thus, providing the source code could be a good start for this new task.

[A] Y. Gal and Z. Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016
summary_of_the_review: Overall, I like this work especially the proposed setting. Although the novelty is not very strong, this is the first step for open-world learning. Also, there are some concerns and missing experiments that should be addressed during the rebuttal. I think this is a promising and interesting task so that the first work should design a well-considered setting and provide extensive comparisons for the following works.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies a new setting for open-world semi-supervised learning. This setting extends the typical semi-supervised learning by considering unseens classes in the test set. This paper introduces a uncertainty adaptive margin mechanism for this new problem. The results are evaluated on multiple datasets. The results are promising. 
main_review: Overall, this paper is well written. The paper is well motivated. The idea of adaptive margin and estimating intra-class variance using uncertainty is novel to me. 

I have a few questions regarding the experiments:

- In Figure 3 (right), the accuracy of ORCA-ZM drops significant at the 140 epoch. The authors explained that this shows ORCA-ZM is not able to reduce intra-class variance. However, it's not clear why the decay of learning rate will trigger the performance drop.

- I suggest replacing "improvement" in Table 2 with "relative improvement" for the readers convenience.

- The proposed margin term is neat as described in Eq 4. It would be good if the authors could provide some analysis regarding L_S when the margin is Eq (4). For example, how will the margin mitigate the bias?
summary_of_the_review: This paper is very well written and well motivated. I recommend accepting this paper for publication if the authors could address a few issues described above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors define a problem with a more realistic perspective in semi-supervised learning, open-world semi-supervised learning, which considers a situation that unseen classes are included in unlabeled and test datasets. The authors propose a method called ORCA, which tries to solve the problem caused by open-world semi-supervised learning. The main function of the ORCA is balancing intra-class variation between seen and novel classes with 3 kinds of loss terms.

main_review: # Strong points- 
- By defining a new, realistic problem, the authors provide a direction for researchers to move forward with a persuasive and structured explanation.
- This paper contains a wide range of experiments to support the authors' claims. Although there is no clear competitor in the defined problem, authors adopted various methods from relevant fields, with slight appropriate changes. Also, they provide ablation study results to validate the proposed method.

# Weak points
- Compared to the detailed technical explanation of the proposed method, the solid explanation of the statements and intuition is insufficient.
- As the authors define a new problem, open-world SSL, the experimental setting is somewhat arbitrary.

# Questions
- While uncertainty adaptive margin takes a key role in ORCA, a detailed explanation of how it works doesn’t seem to suffice. Readers would not be able to understand with only intuitive explanations and references, so a more detailed explanation is necessary. For example, why eq.3 means the large margin of seen classes in early training epochs?
- In the experiment settings, authors set the ratio of seen/novel classes to 50%. Can it be said that this setting represents a real-world problem?
- Though there are no competitors to open-world SSL, baseline algorithms like FixMatch and DS3L show somewhat low performances than its original paper with conventional SSL settings. For the authors' method to be more convincing, it seems necessary to show that the ORCA does not fall behind the baseline in the existing SSL problem (without any novel class). 

summary_of_the_review: Although the paper has some weaknesses such as comparison in the conventional SSL setting, the paper proposes a new problem that considers a more realistic situation and the proposed method seems to work properly for the designed setting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a new semi-supervised learning problem, open-world semi-supervised learning, where the objective is to recognize samples from seen classes as well as cluster samples from novel classes. The proposed method consists of three components to address the different aspects of this problem:  a supervised CE loss with uncertainty adaptive margin to recognize samples from seen classes, a pairwise objective to cluster samples from novel classes, and a regularization term to avoid degenerate solutions. The effectiveness of the proposed method has been validated on multiple benchmark datasets.
main_review: **Strengths**

- This work introduces a realistic semi-supervised problem that accounts for the presence of samples from novel classes in the unlabeled set. This problem setup is interesting and would make the developed SSL solutions applicable to more practical scenarios. 
- The proposed solution achieves promising results. Even though the technical novelty of different components of the proposed solution is not high, incorporating these multiple components to solve different aspects of the challenging open-world semi-supervised learning problem is novel enough. 
- The paper is well written and easy to follow.
- The ablation is extensive. Besides, the work includes analysis under various challenging conditions like class imbalance, varying percentage of seen and novel classes, etc. 

**Weakness and Concerns**

- The effectiveness of the pairwise objective should greatly depend on the quality of self-supervised features. The proposed method and most of the baselines have utilized SimCLR pretrained weights. An analysis would have been nice to see how the performance of the proposed method changes with different self-supervised pretraining schemes; especially the ones with pretext tasks like RotNet, Jigsaw etc. Besides, for a lot of datasets the self-supervised pretrained weights might not be available; for instance, the single-cell dataset. Therefore, it is crucial to know how the method performs without incorporating the self-supervised pretrained weights on the benchmark datasets: CIFAR-10, CIFAR-100, ImageNet-100.   
- The experiments on the main text used a large portion of labeled data (50%), which is not a very practical setup for semi-supervised learning. However, the appendix includes results with 10% labeled data. Since the SOTA closed-world SSL methods are very label efficient, similar experiments (with 1%, 5%, etc labeled data) could have been conducted. 
- Pseudo-labels for pairwise objective have been generated by finding the most confident pairs. However, for datasets with a large number of classes, more than one sample might not be present in a minibatch if the batch-size is not sufficiently large. Therefore, it would be interesting to know the sensitivity of the proposed method to the batch-size parameter.  
- How were the hyperparameters for the proposed method tuned? Did the authors use a validation set for this purpose? If that is the case, then did that validation set contain labeled samples from novel classes? Because using labeled samples from novel classes to tune hyperparameters would make the proposed method less practical. 
- Comparison with an unsupervised clustering method like SCAN[1] would have been interesting. 

**Questions and Comments**

- The results reported in Table 2, and 5 demonstrates that the accuracy on novel classes is higher than that of seen classes for the CIFAR-10 dataset. Since for all the other methods the seen class performance is higher it would be interesting to know any insight behind this phenomenon.
- In parallel to the zero margin and fixed margin experiments have the authors conducted any experiments by decaying the margin for CE loss with a predefined schedule (linear, sigmoid, cosine, etc)? Even though this approach involves some hyperparameter tuning it would reduce the computation needed to compute the uncertainty adaptive margin.

[1] Gansbeke, et al. SCAN: Learning to Classify Images without Labels. ECCV 2020.
summary_of_the_review: This work introduces a new semi-supervised learning problem that has more real-world applications. Even though the technical novelty of different components of the proposed method is not high, the solution is well motivated and achieves promising performance on multiple datasets. However, some of the points are not convincing and require further clarification to assess the significance of the work. I would appreciate it if the authors address these concerns in their response.
review_title: null

Meta-review: This paper is proposed to address a novel but practical setting that the test set consists of both seen and unseen classes of the training set. To tackle the crucial challenge of distribution mismatch between the inlier and outlier features, the authors proposed a new method named ORCA by grouping similar instances to enlarge the class-wise margin for de-biasing. The experimental results on ImageNet have shown the proposed ORCA has significantly outperformed baselines in both inlier classification and outlier detection. The whole paper is written with clear logic and is easy to follow. Moreover, such a new setting may bring more inspiration to the community.
Decision: Accept (Poster)