Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: From the notion and empirical analyses from barely supervised learning, it introduces super class based regularization which improves the discriminability of unlabeled data instances. It largely improves the performances with various benchmark datasets with high confidences. The introduction of clustered super class is somewhat novel in the semi-supervised learning.
strengths_and_weaknesses: Strengths :

Introduction of super class is the main novelty of this paper. It mitigates the imbalanced ability of barely supervised learning from the side of immutability and separability, which is sound. 

Weaknesses :

Application on the semi-supervised learning with more labels provides ineffective performances, which is not explainable from the view of immutability and separability. If it really matters, the performances can be calibrated from the hyper-parameter tuning between consistent learning and the introduced discriminative learning.
questions: Stated in weaknesses session.
limitations: Stated in weakness session.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles the problem of barely-supervised learning – an SSL setting in which only a few-labels per class are annotated. The authors propose a novel approach that extends the FixMatch with a discriminative information learning module. The key idea of this module is to form super-classes by clustering the data using K-means clustering and then rely on the similarity between super-classes and samples to guide the training. The number of clusters is gradually increased during the training. In that way, super-classes are dynamically formed and gradually become more fine-grained. The final objective function consists of standard cross-entropy loss on the labeled examples, standard consistency loss based on the weakly and strongly augmented versions of the data and the newly introduced discriminative distribution loss based on the formed super-classes. The performance of the approach is compared to state-of-the-art SSL and barely-supervised methods on CIFAR-10, CIFAR-100 and STL-10 datasets.
strengths_and_weaknesses: **Strengths:**
- Paper is well written, well structured and it was a pleasure to read it. The explanation with immutability and separability motivates the paper and the proposed approach really well. 
- Barely-supervised learning is a challenging setting, and the proposed method solves the challenge in a simple, elegant and effective way
- The proposed  method achieves state-of-the art performance on the barely-supervised learning setting on three datasets, and comparable performance on the standard SSL setting.

**Weaknesses:**
- (Transductive) few-shot learning should be discussed in the related work and in the introduction and the differences between the settings need to be explained. Barely supervised learning should be better defined in the introduction and put in the relation with the more studied problem of the few-shot learning.
- The performance does depend on K and it is not clear how to set it. For example, it is not clear why for the CIFAR-100 K was set in the range {5,10,20} and differently then for the CIFAR-10 and STL-10 datasets. How does setting K differently affect performance on the more challenging CIFAR-100 dataset? The analysis is only done on the CIFAR-10 dataset.
- Reliance on the K-means clustering. Clusters can only be sperical in shape and the method may fail for the unbalanced dataset.


questions: - Why K ∈ {5,10,20} for CIFAR-100? What is the performance on CIFAR-100 when the K goes to the maximum of 100 classes?
- Why is K increased only in the 30% of iterations? How is that determined and how does a different setup affect the performance?
- How does the algorithm compare to others on the SVHN benchmark dataset on the barely-supervised learning as well as standard SSL setting?
- How does the algorithm perform on the unbalanced dataset?
- Authors motivate super-class idea by the fact that the pseudo-labelling approach can be very noisy. But how accurate are the resulting clusters (super-classes) during model training? It would be beneficial to compare the accuracy of pseudo-labels of FixMatch with the the quality of generated clusters in the proposed approach (e.g., whether examples that belong to the same ground-truth class are assigned to the same super-class during training)

**Minor comments:**
- Line 194: it seems it should be \lambda_con=\lambda_dis=1 not L
- Line 294: sentence is not finished

limitations: No. I suggest authors comment on the limitations of their reliance on the K-means clustering.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses Barely Supervised Learning (BSL), an SSL problem when only a few labeled samples are available.
The key is the introduction of a new loss function to the general consistency-based framework of SSL; the new loss measures the consistency between unlabeled samples and the centroids of "superclasses," which are obtained by applying K-means to the unlabeled data with $K <=$ # classes. K is gradually increased as the learning progresses, like a “curriculum learning” idea.
Experiments with CIFAR-10, CIFAR-100, and STL-10 have shown that the proposed method outperforms the existing SoTA SSL/BSL methods under BSL settings.

strengths_and_weaknesses: --------------------------------------Pros--------------------------------------

P1. The paper addresses BSL, an SSL problem with only a few labels. While it is important and practical, few studies have been reported in the literature.

P2. This paper provides a good insight that BSL can be improved by a very simple method of adding consistency loss with K-means centroids.

P3. The method outperforms SoTA SSL/BSL methods.

P4. Ablation study on $K$ is reported.

--------------------------------------Cons--------------------------------------

C1. Batch size and $K$: Algorithm 1 says that K-means is performed batch by batch. Just to make sure, does this mean that K-means is applied to the samples within a batch, i.e., 64 samples? If yes, I wonder the resulting clusters will be quite different between iterations and may not be meaningful when the number of clusters is large.

C2. Computation Time: Since every iteration K-means is performed, the training time may increase. There is no discussion on this point.

C3. Results: I have some questions about the results. For example, in the CIFAR-100 100-label setting in Table 1, the score of LESS [4] is 28.2 $\pm$ 3.0, which is consistent with the number reported in [4]. However, in the 200-label setting, there is a gap between the numbers reported in this paper (39.5 $\pm$ 3.2) and those given in the original [4] (42.5 $\pm$ 3.2). It is unclear what is causing this difference. Such inconsistencies are also observed in Table 3.

C4. Analysis: The ablation study on the proposed loss term $L_{dis}$ would be essential, as it is the main idea of this paper. Some sensitivity analysis and discussions on the weight for $L_{dis}$, $\lambda_{dis}$, would be necessary.

C5. Novelty: The method looks a little incremental. The proposed method only adds the consistency loss with K-means centroids to the typical SSL framework (i.e., consistency between weak and strong augmented samples).

C6. The discussion of immutability and separability given in the introduction is interesting, but its significance is not clear. The relationship to the idea of the proposed method seems somewhat tenuous, and these measures are not used in the evaluations.

C7. Typos：
- L37 and others: “model collapse” -> "mode collapse" would be better.
- L49: “two different samples” -> "two samples from different classes" would be what the authors want to mean.
- L140 and others: “super-class super-class” -> “super-class”
- L190: “the consistent information learned model” -> "consistent information learning module" (the same is applied to L192: “the discriminative information learned model”)
- L221: “52.52%” -> “76.76%”

questions: There are several unclear points. In particular, I would like the authors to clarify C1-C3.
limitations: I did not find any discussion on limitations. Related to C2 above, if the training time increases when running K-means on a batch basis, it would be a clear weakness.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a semi-supervised learning method for barely supervised setting. The core contribution is the design of discriminative information learning module with clustered super classes. The experiment on barely supervised setting demonstrates the consistently improved results on small datasets. 
strengths_and_weaknesses: Strengths:
The paper is well-written and organized, easy to follow.
The improvement on the studied settings is significant and consistent.

Weakness:
The design choice on auxiliary distribution loss (eq.3) is not shown
There is a mismatch of EMA of model parameter in main paper 0.999 (L201) versus 0.99 in appendix 0.99 (in table1)
questions: Questions:
For eq.3, why a constative-like loss is exploited. Is simple CE loss on the super classes not working? Cosine similarity is adopted in eq.3, but for softmax probabilities.

It important to show the training speed with the proposed discriminative information learning module. Would running K-means for each training iteration slow down the training? Comparison between the training speed would be expected.

For barely supervised setting, random seed often plays important roles for the distribution of the labeled data. Can you provide more details on how the random seed is set? Is it randomly selected or with more sophisticated selection? For example, in FixMatch, they expain how they choise the 10 labels for CIFAR-10

All experiments are run with small datasets. Also, a challenging dataset SVHN is not shown. SVHN is more challenging because its noisy background. It would be interesting to see the performance on SVHN and larger datasets such as (Mini)-ImageNet.
To my understanding, for barely supervised setting, the quantity of the samples accepted for calculating the loss is especially important for early training. If a high threshold (tau_1, tau_2) is employed, the SSL algorithms would collapse to one class as shown in Fig.1 because not diverse enough samples are enrolled during training. This is also demonstrated in Fig.4(c), when tau_2 is higher, the performance drops more significantly than when tau_2 is lower. In that case, compare the enrolled rate of samples and the pseudo-label accuracy to FixMatch and FlexMatch is important to demonstrate the effectiveness of the proposed method.

The Super class concept is very similar to SimMatch (https://arxiv.org/abs/2203.06915) yet no discuss and comparision is provided. 
limitations: see above
review_title: null

Meta-review: metareview: This paper proposes a simple but effective method for barely-supervised learning, where the label is scarce, for instance, 1 example per class. The method is based on k-means clustering of the unlabeled data to form superclass and rely on a contrastive-type of loss to enforce discriminativeness. The proposed method improves class separation in the presence of scarce labels and significantly improves the overall performance.

Though the proposed method is rather heuristic, the idea to address barely supervised learning is novel and interesting.  The superior performance is demonstrated via strong empirical studies. I would recommend acceptance of this paper given the novelty and simplicity of the idea and the strong empirical evidence. 

confidence: Certain
Decision: Accept