Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper considers open-world SSL settings  where the model recognizes previously seen classes, and detects novel classes which are not present in the labeled dataset. The method contains three losses to train a model in this setting: a) supervised loss on labeled data, b) unsupervised loss on unlabeled data from pseudo-labels obtained  from confident pairwise similarities, and c)  regularization term that avoids assigning all the unlabeled samples to the same class. Then the paper evaluates the effectiveness of the  method on CIFAR-10/100 and ImageNet-100 datasets.

The paper is well organized, well-written, and tackle SSL problem in a more realistic setting. Experiments are enough to some extent. However, I feel that the paper just combines several well-known and well-studied techniques for different tasks which intersect with open-world SSL settings. (e.g., self-supervised learning in SSL, clustering for SSL or clustering for transferring knowledge  across domains and tasks, using confident pseudo-labels for training an SSL model, etc). From this perspective, technical novelty of the work is limited, although experiments show good results for open-world SSL setting. 


-First, I think this set-up does not contain all the scenarios for a real-world SSL. For example,  this set-up does not consider covariate shift where the data belonging to the same classes but different image statistics or style (e.g., dogs in natural images, dogs in the painting or sketches images). However, in the real-world scenario, we may have images of the same class but different domains. 


-The paper adds several losses studied in the literature without analyzing if any has a negative effect on the others in the open-world SSL setting.

-Assuming that the number of novel classes |C_u| is known is a bit unrealistic to me in the open-world SSL setting.

-The method ranks the distances, and for each sample generates the pseudo-label for its most similar neighbor. However,  in an open world SSL setting due to the very-limited-label regime, the representation may not be ideal, and  therefore, pairwise-labeling may not be ideal and can possibly propagate the error through the network over the course of training.


-Does the regularization towards uniform distribution consider unbalanced novel classes which is common in open-world SSL? 

-Does the method perform well in cases where the test set contains novel classes that do not appear in the unlabeled set?

Generally, the work has some potential from practical point of view, however, it needs more work to be improved technically.  

review_title: Open-world Semi-supervised Learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros
- The paper explores an interesting semi-supervised learning (SSL) setting in which the unlabeled data contain not only the seen class but also novel classes. The problem is interesting in that it is more practical than the classic SSL setting in the real world and has been seldomly researched. 
- The method includes a self-supervised pretraining step and a finetune step. The finetune step jointly solves the classification task and the clustering task with a unified objective loss. The supervised loss, which is the main contribution of the method, overcomes the imbalance problem caused by BCE with a novel adaptive margin-based loss. The authors validate this loss with empirical results.
- The paper is well-organized and clearly written. As far as I can see, the method is technically sound.

Cons
- The comparison methods (such as pseudo-labeling, DS3L) are not strong enough. It is suggested to compare with more SOTA methods. Since SimCLR is more powerful than RotNet, it is strange to see RankStats get worse performance than the original paper.
- The number of novel classes has to be prefixed.

review_title: The paper proposed a novel method for solving open-world semi-supervised learning problem.

final_score: rating: 6: Marginally above acceptance threshold
review_content: --Paper summary--

The authors propose a method to tackle a new problem setting of semi-supervised learning, called an open-world semi-supervised learning, where the model is required to accurately discriminate known-class data as well as to appropriately discover unknown classes contained in an unlabeled dataset. The objective function to be minimized in the proposed method comprises three terms: unsupervised loss, supervised loss with uncertainty based adaptive margin, and entropy regularization. The experimental results with several datasets have validated the advantage of the proposed method.

--Review summary--

This study is well-motivated and tackles an important problem that would occur in real-world applications. The design of the proposed method seems reasonable, and it works well in the experiments with several datasets. However, some points remain unclear or are not convincing, which makes my score a bit conservative. To appropriately determine my score, I would appreciate if the authors clarify these points in their response.

--Details--

Strength

- The setting of the open-world semi-supervised learning is interesting and should be practically important.
- The paper is well-written and well-organized.

Weakness and concerns

- It is not clear whether the design of the proposed method really leads to the improvement of the performance or not. Since the authors use SimCLR to pretrain the model, the pretrained model has already had a substantially good feature representation. Consequently, if we know the number of classes in the training data, unsupervised learning methods like [R1] (or simple clustering methods) might work well to discriminate all classes. Although the authors adopt a new idea to the supervised loss, it seems unclear whether the supervised loss itself really contributes to good performance. Are there any experimental result on performance sensitivity to \eta?  
[R1] "Learning Discrete Representations via Information Maximizing Self-Augmented Training," ICML 2017.
- The experimental setup is not convincing. Due to the motivation of SSL, the number of unlabeled data should be much larger than that of labeled data, and it is often tens or hundreds times larger in the literature. However, in this paper, it is basically 3~7 times larger in the experiments. Is there any reason why the authors did not much reduce the number of labeled data?
- Two concerns on uncertainty based adaptive margin.
	- How did the authors estimate the class posterior probability in Eq. (4)? Is it just the output of the softmax function?
	- Since this adaptive margin is adopted to improve the accuracy of pseudo-labels, how much it is improved should be reported in Fig. 3.
- How did the authors conduct validation? Due to the existence of unseen classes in unlabeled data, how to conduct validation is not trivial.
- Is it reasonable to call L_BCE unsupervised loss? Since Z_l' is obtained using ground-truth labels, L_BCE cannot be computed in an unsupervised manner.

Minor concerns
- I could not get the reason why the proposed method is called ORCA.


--After receiving authors' response--

I would like to thank the authors for providing additional experimental results and giving clarifications. Since my concerns are almost solved, and I updated my score from 5 to 6. This study would be a great first step to tackle the challenging problem, open-world semi-supervised learning, though there are several remaining issues (e.g., validation is hard to conduct, the number of unknown classes should be known, etc.). I vote for "weak accept."

review_title: This study is well-motivated and tackles an important problem, but some points remain unclear.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: the authors propose Open World Semi-Supervised Learning (ORCA), a semi-supervised method that learns to classify previously seen classes in the labeled data and novel class in the unlabeled data. The proposed method is end to end and achieves significant improvement on ImageNet dataset compared to baseline methods.

Pros:
The paper takes one of the most important issue of semi-supervised learning: recognizing novel classes. For me, the problem itself is real and practical.

The experiments are strong and methods shows significant improvement compared to competing methods.

Cons:
The paper lacks clarity in some sections. The writing could be better and the notations better defined.

Comments:
The unsupervised loss function is not clear to me, I find it difficult to parse equation 1 from the way it's written. Could you please explain the equation better?

It's my understanding that each z_i in the unlabeled set is assigned a class based on the ranking of pairwise distance in the mini batch.  If the mini batch of z_i contains labeled samples, then it could be assigned to the labels example if the images are similar. For example if there's an unlabeled example of a horse, it could be assigned to the labeled donkey class. Wouldn't this be problematic for the novel class prediction if the batches always contain similar examples from different classes? 

Why the choice of cosine distance for the similarity measure? Using cosine similarity means that you do not consider the magnitude of the embeddings. This would make sense for text embeddings but the experiments are on image datasets
review_title: OPEN-WORLD SEMI-SUPERVISED LEARNING

Meta-review: This work addresses a novel and important real-world setting for semi-supervised learning – the open-world problem where unlabeled data may contain novel classes that are not seen in labeled data.  The paper provides an approach by combining three loss functions: a supervised cross-entropy loss, a pairwise cross-entropy loss with adaptive uncertainty margin, and a regularization towards uniform distribution.  

The authors were responsive to reviewers’ comments and have respectively improved their paper by adding experiments, including an ablation study of each component of the objective function, study of the effect regularization on unbalanced class distributions, reporting accuracy on pseudo-labels.  While two reviewers have slightly increased their scores, some concerns still remain.

This is a borderline paper, and after some discussion and calibration, we decided that the work in its current form does not quite meet the bar for acceptance.
Decision: Reject