Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################
Summary:

The paper proposes a new approach for open set semi-supervised learning, where there are unlabeled data from classes not in the labeled data. The paper uses a contrastive representation learning paradigm to learn a feature encoder and a similarity measurement. Then the paper filters outlier samples by the similarity measurement and further utilizes outlier samples with soft labels. The separate BN layers address the distribution shift between in-class and out-class data.

##########################################################################

Reasons for score: 

 
Overall, I vote for accepting. I like the idea of learning representation in an unsupervised way for both labeled and unlabeled data. My major concern is about how to select a reasonable threshold and why the threshold works well (see cons below). Hopefully the authors can address my concern in the rebuttal period. 

########################################################################## 

Pros:

The paper addresses a very interesting and practical problem in semi-supervised learning, where the unlabeled samples may include out-class samples. This problem promotes the practical use of semi-supervised learning in real world applications.

The paper employs contrastive representation learning to learn the encoder and similarity measurement, which preserves the similarity between features of a sample transformed by different transformations. The learned encoder can encode the semantic information into the feature for both labeled and unlabeled data in an unsupervised way. 

The paper filters the out-class samples by the learned similarity measure based on a threshold, which can better filter out out-class samples.

The paper employs different batch normalization layers for the in-class and out-class samples, which avoids the influence of the distribution shift.

The paper is well-written and the claims are clearly clarified.

Experimental results on three semi-supervised learning benchmarks: CIFAR-10, CIFAR-100 and ImageNet, show that the proposed method can detect open-class samples and achieves higher accuracy the previous semi-supervised learning methods. Qualitative results show that the method is not very sensitive to hyper-parameters.

##########################################################################

Cons:

The performance gain of the proposed method is only over closed-set semi-supervised learning methods. The paper does not compare with the state-of-the-art open set semi-supervised learning method (Guo et al., 2020).

The threshold hyper-parameter is the key to filtering out-class samples. The paper decides the threshold by mean minus 2 standard deviation. Could the authors explain why choosing this value? Is there any insight in this threshold or why does this threshold fitful for any dataset?

review_title: Review for the paper

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper considers the problem of semi-supervised learning, where the unlabeled data may include out-of-class samples. To address this task, the paper proposes a method consisting of three steps: (1) detecting out-of-class samples in the unlabeled set, (2) assigning soft-labels to the detected out-of-class samples using class-conditional likelihoods from labeled data, and (3) using auxiliary batch normalization layers  to help mitigate the class distribution mismatch problem. Experiments are conducted on CIFAR-10, CIFAR-100, ImageNet datasets. Results show improvements over competing methods.

Quality: This paper is well written and well organized. I find this paper easy to follow.

Clarify: The preliminaries section clearly describes the setting of semi-supervised learning concerned in this paper and also clearly describes the contrastive representation learning. I like the way that the authors include a preliminaries section before the proposed method section. This allows me to understand which parts of the method are built upon existing approaches and helps me better identify the contributions and new things of this paper.

Originality: While I think the idea of this paper makes sense, I think there are several parts that are highly similar from a recent paper published in ECCV 2020 [a]. While the setting considered in [a] (for metric learning problems) is a bit different from that concerned in this submission (for image classification tasks), the high-level idea (learning representations that can be used to describe unlabeled images with labels different from those in the training set) is very similar. For example, the idea of Equation 5 and Equation 6 in this submission is almost exactly the same as that in Equation 2 of [a]. In addition, the idea of Equation 9 of this submission is very similar to that of Equation 3 of [a]. However, the authors did not acknowledge the similarity with [a] in the submission. This will make readers feel like the idea of Equation 5 and Equation 6 is original in this paper.

[a] Chen et al. Learning to learn in a semi-supervised fashion. In ECCV, 2020. https://arxiv.org/pdf/2008.11203.pdf

Significance: Given that some parts of this paper are highly similar to [a], the significance of this paper is downplayed.

Request for author response: I would like to see how the authors compare their paper with [a] in terms of problem setting, idea of class-wise similarity and representation learning. In particular, I would like to know the pros and cons of this paper compared to [a]. It would be great if the authors can talk about whether the semantics-oriented similarity representation in [a] could be used (and how to use it) to help improve the performance of the proposed method in the setting concerned by the paper.

Rating: Given that there are many parts similar to [a] and they are not acknowledged in the paper, I can only rate 4 for this submission at this point. I will reevaluate this paper after seeing the reviews from the other reviews as well as the author response.
review_title: Review of OpenCoS

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors proposed to address the task of semi-supervised learning (SSL) by contrastive learning techniques, and the proposed techniques can be applied to handle open-set unlabeled data (i.e., the label spaces between label and unlabeled data are partially disjoint). I found the paper clearly written and easy to follow. The review comments are discussed below.

The proposed learning scheme is extended from SimCLR by Chen et al. More precisely, the authors perform unsupervised learning using D_l (w/o observing labels) and D_u using SimCLR techniques. In order to handle out-of-class samples in D_u, the authors present the idea of learning class-wise prototypical representation based on the above contrastive features. Detection of such samples is performed by a simple cosine similarity comparison between each instance in D_u and the prototypes of D_l. The main contribution lies in the last stage, i.e., SSL with auxiliary loss, which trains classifiers for recognizing in-class samples while assigning soft label scores for out-of-sample ones. The use of such soft labels allows the training of such unlabeled and out-of-class samples, which would be the major novelty of this work. (I do not feel that the use of batch norm would be viewed as technical contributions.)

Overall, I feel that most of the technical components come from existing works (e.g., SimCLR from Chen et al. 2020, auxiliary batch norm from Xie et al. 2020.). The use of soft-label assignments has also been proposed by existing works, which makes the overall technical contributions to be marginal. The lack of verification on the soft label assignment would be the concern as well. For example, is "cat" assigned with 0.1*leopard + 0.2*lion + 0.7*tiger? Most importantly, are out-of-class samples assigned with uniformly soft labels (i.e., 1/C)? Out-of-class samples might still exhibit similarity with selected in-class categories, and thus forcing their soft labels to be a uniform distribution might not seem to be practical (if that's the case). Existing vision and learning models have been considering class-similarity based representation, etc. techniques for handling zero-shot or open-dataset learning problems (e.g., Xian et al. PAMI'18 and Scheirer et al. PAMI'12). Based on the above remarks, I feel that the paper is yet above the ICLR standard for acceptance.


review_title: Solid work but marginal novelty/technical contributions

final_score: rating: 4: Ok but not good enough - rejection
review_content: Quality: 
The quality of the paper is below average. The loss function that integrates out-of-class samples is counter intuitive and seems to be chosen based on improved empirical evidence.

Clarity:
The paper reads well.

Originality/Significance:
The originality of the approach is limited. Main ideas are borrowed from SimCLR and applied to SSL. 

Detailed Comments:

The paper uses contrastive learning idea proposed in SimCLR (Chen et al. 2020) to detect out-of-class samples and treat them in a different way than in-class unlabeled samples during semi-supervised learning. It also explores the idea of auxiliary batch normalization (from Xie et al. 2020) in the open-set SSL setting but the results of the ablation study suggest the level of improvement achieved by this normalization is negligible and the most of the improvement comes from more accurate detection of out-of-class samples through using the projection header function introduced in SimCLR paper. Although results across multiple benchmark datasets report significant improvement over other SSL techniques these improvements could be artificial as other techniques have no way of handling out-of-class samples. Integrating these other out-of-class detection techniques in ReMixMatch and comparing results with the proposed technique would offer a more compelling argument. Overall, the paper proposes a nice practical idea but for publication in ICLR one would like to see more theoretical insight along with empirical evidence. 

Class conditional likelihoods has been shown to be not very useful in detecting out-of-distribution samples in cross-entropy loss learning. What aspect of contrastive learning makes it more useful for open-set classification?

Cross-entropy term in equation 8 does not make much sense. It is simply computing the loss with respect to an incorrect label. If a sample belongs to an unknown class it is not clear why this would help SSL. If the purpose here is to capture shared characteristics of the samples then SimCLR trained with both labeled and unlabeled data already takes care of it. The authors try to justify this by considering that some seen classes would be similar to the unseen one but for a fine-grained classification task such samples may still hurt predictive performance. If there are no similar classes among labeled classes the authors argue that this loss will have a uniform affect for all classes. Again, this is not a compelling argument. No matter how dissimilar the unseen sample to labeled classes are it would be more similar to some of the classes than some others. Due to the normalization affect the weight distributions will significantly deviate from a uniform distribution. 

Baseline techniques are all SSL techniques. These are guaranteed to perform worse than the proposed technique because they have no way of handling out-of-class samples. What about other baselines? From Table 3 one can see that the main contribution comes from the detection of out-of-class samples. The authors compare the detection performance of their technique against other standard outlier detection techniques and show that the proposed detection outperforms all of them to achieve the highest AUC in the "isolated" detection task. However, it is still not clear why the projection header g achieves something that softmax probabilities cannot do. In other words why do class-conditional probabilities obtained from g are useful for out-of-class detection but the probabilities that one would obtain from the softmax layer of an architecture (such resnet) trained by cross-entropy loss is not much useful for the same task. Interesting  that not much insight has been provided in the SimCLR paper either. Along the same lines, as a contrastive loss function, triplet loss also seems to do well in open-world settings. 

Minor:

Please correct the following:

page1 Compared to prior approaches approaches have that bypassed ...
review_title: The paper uses contrastive learning idea proposed in SimCLR (Chen et al. 2020) to detect out-of-class samples and treat them in a different way than in-class unlabeled samples during semi-supervised learning.

Meta-review: This paper proposes OpenCos for semi-supervised learning that can leverage unsupervised information in open-set scenarios where samples can be out-of-class.  They first pre-train by learning an unsupervised representation using SimCLR on both the labeled and unlabeled data.  Then, they detect out-of-class samples in the unlabeled set based on similarity measures on the representation learned in the previous step.  The unlabeled data can now be split into in-class and out-of-class.  OpenCos optimizes (8) which combines a semi-supervised loss for in-class unlabeled data and an auxiliary cross-entropy loss with soft-labels for the out-of-class samples.  Finally, they perform an auxiliary batch normalization.

The paper is easy to read and clearly structured.  It also places the work well with respect to related work.

The proposed approach makes sense; however, as pointed out by the reviewers, the novelty is marginal.  The technical innovation seems to be an extension of SimCLR and the auxiliary batch norm of Xie et al.

Decision: Reject