Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors notice that popular SSL evaluation protocols are often constrained to computer vision tasks and are generally time-consuming and environmentally unfriendly. Based on this, they construct USB by selecting several tasks on which they evaluate dominant semi-supervised learning(SSL) methods. To reduce the cost of SSL, they also provide a pre-training and fine-tuning paradigm. The used tasks are well divided according to their domains and algorithms are categorized. 
strengths: This paper is well motivated. As the authors point out, previous work typically trains deep neural networks from scratch, which is time-consuming and environmentally unfriendly. I applaud the author's efforts in experiments. To clearly and comprehensively clarify the strengths and weaknesses of different algorithms in different domains, they conduct quite enormous experiments. The authors also use nearly every dataset commonly used in different tasks. This paper has many contributions from both technical and conceptual perspectives. I applaud the authors' efforts to clearly categorized every algorithm according to its own features.
weaknesses: As the authors imply, they still lack some other SSL tasks such as imbalanced semi-supervised learning, open-set semi-supervised learning, etc. Although the authors use 14 different algorithms to conduct experiments, there are still some algorithms remaining. The table used to store experiment data seems too small, like tables 5, 6, and 7.


correctness: Yes, I think the evaluation methods and experiment design appropriate and performed correctly.
clarity: This paper clearly describes the experienment.
relation to prior work: This work clearly discusses the differences between previous work.
additional feedback: None.
review_title: Review of USB: A Unified Semi-supervised Learning Benchmark

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work proposes a new benchmark for semi-supervised learning, USB, which covers evaluation on 15 different tasks in 3 different domains: vision, NLP, and audio. The benchmark is constructed to not be too computationally expensive to run. Additionally, a number of different SSL algorithms are compared on the benchmark and there is some analysis performed about which methods succeed.
strengths: - The benchmark includes a large and diverse evaluation suite: 15 different tasks in 3 different domains, including the audio domain, for which USB is the first to evaluate SSL on.
- 14 different SSL algorithms are implemented and compared on the benchmarks (save for those algorithms which do not make sense to implement in certain modalities).
- The benchmark is constructed with computational expense in mind, such that it requires much fewer GPU-hours to evaluate a method across the suite of tasks compared to previous SSL benchmarks.
- Using pretrained backbones is an important axis of variation missing in previous benchmarks, and USB investigates the impact of the use of a pretrained backbones, finding that most SSL algorithms become more efficient.
- USB compares the rankings of different algorithms across the different domains. I believe this analysis/result is the first for SSL algorithms to be compared between domains.
weaknesses: My main criticism of this work is that there is little justification given for a couple core experimental choices in the USB benchmark. This is a considerable missed opportunity, since USB is positioned as an compute-friendly SSL benchmark that academics or other researchers can hill-climb on. As such, this work can do a better job of convincing the reader that results from this benchmark will transfer to other settings, wherever possible, by performing additional experimental validation. Let me be concrete:

- The choice of pretraining backbone. I do agree that it is reasonable to leverage pretrained backbones as a way to improve the performance of SSL. However, this is a significant departure from the standard SSL setting and deserves some justification. Why are the specific pretrained ViT, BERT, or Wav2Vec defined in USB chosen (is there a reason they are better than alternatives)? Do the rankings or conclusions of which SSL algorithm is best change if the pretrained backbone is changed (for a different architecture, or one trained on a different dataset)? If the choice of pretraining backbone has a large influence on the SSL algorithm ranking, this should be discussed in more detail. If not, then the experiments need to be presented to inspire confidence in the current choice of backbone.

- Why is ImageNet excluded from USB? This seems like an odd choice, especially since *most* SSL work has been developed for ImageNet, and we know ImageNet is a reasonable dataset to hill-climb on since many previous works have shown that ImageNet gains routinely transfer to other datasets. There is some discussion about computational cost for the reason, but its exclusion is odd given that USB positions itself as "Unified." One way around this would be to conduct an evaluation on ImageNet and compare the rankings to the CV portion of USB, and if the rankings are consistent, this would be evidence that ImageNet results closely track the rest of USB-CV and thus perhaps redundant/not needed. But this needs to be run & shown.

Another weakness of this work is the choice of overall metric. Having a rank as the overall metric seems to be a very poor choice since the score for any method depends on the rest of the methods implemented in the benchmark at that point in time. If two new methods A and B were proposed at the same time separately, there would be no way to compare which of A and B is better by looking at their reported ranks until they were both implemented into the USB codebase and the ranks were recalculated. This seems like a major downside. What about just using average accuracy or something similar? It's a much more standard reporting metric and allows time-independent comparisons.
correctness: For hyperparameter tuning, the appendix seems to imply that the parameters of FixMatch were tuned for each dataset, and then those parameters were used for evaluation of the rest of the SSL algorithms. Is this correct? If so, this would seem to greatly bias the FixMatch results - why aren't hyperparameters tuned for each method individually?
clarity: The clarity of the writing and presentation of results could be significantly improved in parts. To be concrete:

- In the abstract, the last line is very unclear. Is it trying to highlight the evaluation speedup? Is it trying to highlight the dataset diversity? Two different pieces are being mentioned and it's hard to understand what contributes to the speedup if the two benchmarks are different and gpu-days are incomparable. 

- There is a lot of space devoted to talking about the existing protocol. As a reader, having lots of information that isn't central to the story of USB was distracting. I think Tables 3 and 5 are useful but can be safely moved to the appendix. 

- The last paragraph on page 4. This was hard to get through. I'm not sure what purpose this paragraph serves - it's just Table 4 written in prose. I would suggest using this space to talk about the different columns of Table 4 and what they mean instead.
relation to prior work: I would suggest mentioning TorchSSL whenever the text says "existing popular protocol"
additional feedback: I would be happy to increase my score, if the changes I outlined are made and depending on the result of the discussion.

***************** UPDATE AFTER REBUTTAL *****************************

I had 3 main concerns, each of which was satisfactorily answered by the authors:

1. Choice of pretraining backbone. The authors ran ablations for different backbones for each of the CV/NLP/Audio domains. It seems like relative algorithm ordering can indeed vary a bit in the top performing algorithms (ranks 1-6). While this is unideal for a benchmark like USB, the authors are willing to include a discussion of these ablations in the text, as well as have promised to update the benchmark regularly to new backbones. Together, this response seems satisfactory to me. While there may be a larger discussion around whether algorithm design or backbone improvements drive the lion's share of improvements in SSL, I feel this is out of scope for the benchmark as it is.

2. ImageNet results. The authors have run a few algorithms on ImageNet to compare with the rest of the CV USB datasets, and have included a compare/contrast discussion in the main text. This seems reasonable to me.

3. Better metrics. The authors have included mean error rate as a metric in addition to rank. I think this addition significantly improves results comparison for the paper.

Overall, all my concerns have been satisfactorily answered, and I believe this benchmark will be a solid contribution to the community, especially to those on low resource budgets.
review_title: More justification is needed for the experimental choices in the USB benchmark.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors propose USB: a Unified SSL Benchmark to facilitate general SSL research. USB offers benchmarks across five CV datasets, five NLP datasets, and five Audio datasets. By using pre-trained Vision Transformers, the authors found that it can largely decrease the training time when evaluating the performance of an SSL algorithm. The authors implement 14 SSL algorithms and will open-source the codebase.

strengths: The problem is interesting and important. The SSL problem has been widely studied in the machine learning community. It is a fundamental problem in CV and NLP.

The authors implement 14 well-known SSL algorithms and put them into an open-sourced codebase for easy comparison. Comprehensive experiments clearly demonstrate the differences. 

The authors provide analysis from the experiments such as the difference between training from scratch and using pre-training in SSL, and the effectiveness of SSL when using the state-of-the-art neural models as the backbones.

weaknesses: The authors should be more clear about the motivation, and why this work/open-sourced codebase is useful for the community. For example, in the introduction, the authors mentioned that existing benchmarks are mostly constrained to CV tasks, but do not clearly illustrate why this is a problem, and what advantages can be gained by building a unified SSL benchmark. 

Lack of insights or explanations for the experimental results. For example, in section 4.3, when answering why should we evaluate an SSL algorithm on diverse tasks across domains. The authors said, “the differences between ranks of SSL algorithms in different domains cannot be ignored, hence it is crucial to introduce diverse tasks from multiple domains when evaluating an SSL algorithm”. Why can it not be ignored? After introducing diverse tasks, what insights can we get?

correctness: The evaluation methods and experiment design are appropriate.
clarity: The tables and figures are clear and supportive of the experiments. However, the authors need to provide more illustrations and clarifications about the motivation.
relation to prior work: The authors discussed the difference compared to previous contributions.
additional feedback: Could you summarize the insights of this paper? What can we learn by comparing so many SSL algorithms across different tasks?
review_title: an interesting paper, but need more insights/illustrations

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces USB, a benchmark for semi-supervised learning which covers classification problems in several domains including vision (CIFAR-100, STL-10, EuroSAT, TissueMNIST, Semi-Aves), language (IMDB, AG-News, Amazon Review, Yahoo! Answer, and Yelp) and audio (ESC-50, GTZAN, UrtaSound8k, FSDnoisy18K). This benchmark aims to cover more domains than the standard approaches for benchmarking, as well as to reduce the amount of computation required (37 GPU days vs. 335 GPU days for FixMatch). The paper also introduces code for 14 existing SSL algorithms, which are evaluated according to the benchmark. Notably, the paper shows that under the USB benchmark, FlexMatch gains 5 places in ranking over the previous CRMatch approach in computer vision. 

The paper's major contributions are:
- Introducing a collection of tasks for benchmarking SSL
- Demonstrating, with some analysis, the performance of some SSL algorithms on this task
strengths: In general, the paper is well motivated: No existing benchmark exists for semi/self-supervised learning algorithms which covers a wide range of domains, and approaches. The paper does a fairly good job at selecting efficient, yet relevant, classification tasks within the three target domains of language, vision, and audio. Given the performance on the USB benchmark, I would be relatively confident in the performance of the semi-supervised learning algorithm.

In addition, the paper has relatively strong analysis of the tested algorithms. I am much more confident in the performance and experiments given here than in many papers, and from the code provided, it is clear that the results are reproducible, and extensible. The framework for evaluation is also reasonable, and none of the methods for summarizing the performance over the tasks seem too opinionated. 

I appreciate specifically that the paper includes multiple settings and seeds as part of the evaluation time and benchmarking process, which is often lacking in existing implementations. 

I further really appreciate the fact that the amount of training time is taken into account, when exploring the 
weaknesses: Generally, I'm worried about the scope of the paper, and the exact choices made for the benchmark. The USB benchmark as stated claims to be a "unified" benchmark for semi-supervised learning, however the chosen tasks largely ignore many of the tasks that individual communities consider challenging or difficult. More specifically, I'm concerned that the benchmark is limited to classification tasks, which are traditionally global in nature, and require little fine-grained detail to solve. Such a decision may lead to research prioritizing models which are designed for classification tasks, and ignore other equally important and interesting tasks.

- In vision, many applications of semi/self-supervised learning focus on fine-grained and zero-shot classification, which are mostly missing from the benchmark (Semi-Aves is the only included example which is even close). Additionally, semi/self supervised learning has been shown to be applicable in tasks in object detection, grounding, 3D vision, and image generation - which are all widely applicable, and interesting to explore.
- In NLP, the chosen datasets are more widely explored, but ignore important tasks such as language generation/general language modeling, summarization, and open-domain QA. Additionally, datasets like IMDB are largely considered solved in the supervised domain, which can make it hard to use as a differentiating task (even in SSL). 
- In audio, the benchmarks focus on audio classification, but many other tasks are relevant including prosody classification, speaker recognition, automated speech recognition, and others (for a good summary of tasks, see the SUPERB benchmark, [46] in the paper). 

**In particular, I would like to see a wider range of tasks in a "unified" benchmark, which closer reflect the current SOTA applications for semi and self-supervised learning.**

Another concern I have is with the chosen algorithms. As discussed in the limitations section of the paper, there are a significant number of SSL-based algorithms which are missing. Further, the chosen algorithms to be implemented are largely from a single direction (*-match) in the SSL space, which could bias future research towards algorithms structured in significant ways. **I would really like to see a wider range of field-representative algorithms, rather than a detailed exploration of single algorithm variants, included in the benchmark**

**Finally, there is no related work section in the paper, which makes it difficult to place in context.**

Some other minor comments:
- While Table 4 has a few interesting properties of each of the algorithms, there's no analysis of the impact of these properties on the USB benchmark scores. It would be interesting, and relevant, to explore how such properties impact the process.
- There is little discussion as to why each of the datasets was chosen beyond "they are in use in the field". 
- I would really like to see some figures which demonstrate the correlation between the standard benchmark and the proposed USB benchmark. It's a bit hard to parse from the table if the results are correlated, and if they are, how closely they are. Related: I would appreciate some more discussion of the numerical correlations between the standard approach and the proposed approach, and I additionally wonder if a subset of the chosen experiments could be used to reduce training time, and would correlate as well or better with the full chosen experiment set. Is it possible to demonstrate that every benchmark chosen is necessary?
- The analysis of the TSNE in Figure-3 is somewhat superficial, and doesn't really add much to the paper. What makes for a "better" TSNE embedding is under-specified at best, and does not always indicate separability (see [here](https://distill.pub/2016/misread-tsne/)). 
- The backbones that are chosen for the approach are somewhat arbitrary. It would be good to include a discussion of possible backbones, and to understand why each backbone was chosen.
- Performance on some of the benchmarks was a bit confusing, as it was presented as error-rate instead of accuracy (which is standard for many of the datasets shown).



 
correctness: I am confident that the included code is correct, and the benchmark is sound. The experimental design seems correct, and I particularly appreciate the inclusion of multiple seeds in the benchmarking procedure. I believe that the experiments could be reproduced, and the code is clean enough to be used in a potential benchmark. 
clarity: The paper is sufficiently clear. That being said, I believe (in accordance with the weaknesses of the paper) that some of the claims of being a "unified" benchmark should be toned down, since the benchmark is hardly unified across a wide range of applications. Appendix C. is very detailed, and shows a lot of useful information. The related work section is not present, which, although not *strictly necessary* would be helpful for readers in understanding and contextualizing information quickly.
relation to prior work: There is no related work section, which makes the paper difficult to contextualize with existing benchmarking procedures. It is not entirely clear how this work differs from other attempts at benchmarking (if such attempts exist). If no such attempt to benchmark semi-supervised learning has ever been made, a small related work section discussion existing methods and their specific drawbacks (beyond those superficially mentioned in the introduction) would be appreciated. The definition of the "typical protocol" is not really well motivated, and we have to take it at face value that such a protocol is typical, and widely in use.
additional feedback: In summary, while I believe that this paper represents a solid technical effort and is globally well motivated, I am concerned about its utility as a benchmark, and novelty compared to existing work (which is hard to evaluate due to the distributed nature of the related work section). I would be more willing to accept a paper if it were re-scoped from "unified" to more correctly describe the target tasks, or if better motivation was included for the selection of each task, which was more aware of the trends in self/semi-supervised learning. I'm also somewhat concerned about the impact of this paper as a benchmark, as it might bias future research in ways that are not necessarily in the best interests of the field - such as an overwhelming focus on classification over other tasks, or a focus on exploration of *Match style algorithms over other directions in the field. That being said, I really appreciate the attempt to unify SSL benchmarks under one approach (which I believe is necessary, and important work), as well as the attention to the training time and technical correctness that this paper presents.
review_title: Review: USB: A Unified Semi-supervised Learning Benchmark

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a comprehensive benchmarking for semi-supervised learning on vision, NLP and audio. And this paper provides an environmentally friendly and low-cost evaluation protocol with pre-training & fine-tuning paradigm, reducing the cost of SSL experiments. Additionally, this paper implements14 SSL algorithms and open-source a modular codebase and config files for easy reproduction of the reported results in this work.

strengths: 1. This paper builds a comprehensive benchmarking on Semi-supervised Learning, which contains vision, NLP and audio.

2. The work opens the source code for further extension, which is beneficial for the ML community in the SSL field.

3. This paper is well-written and provides details on the datasets and experiment setup.

4. It is great that "We plan to evolve the benchmark in the future iterations over time by extending with more tasks." (Line 220-221) This benchmarking could be impactful for the SSL community.

weaknesses: 1. Although the experiment results are comprehensive, the insights and analysis are relatively limited. It would be better if the authors could provide more insights and discoveries from the experiment results. These insights are critical for inspiring SSL model designs in the future. 

2. It is suggested that what the benchmark can help for the SSL community could be described more clearly and explicitly in the paper.

3. Since there are already a lot of existing SSL benchmarks, the relation and comparison with existing benchmarking may need more detailed descriptions, which could be put in the appendix.
correctness: Overall well-written.
clarity: All is clear.
relation to prior work: The authors claim that existing benchmarks are  (1) mostly constrained to plain CV, NLP, audio (2) time-consuming and environmentally unfriendly.
additional feedback: All points are covered in the main response.
review_title: A comprehensive SSL benchmarking on vision, NLP and audio

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Existing SSL methods evaluation protocols are limited because of (i) only implementing them using CV tasks, and (ii) the requirement to train deep neural models from scratch. This paper evaluates different SSL algorithms across different domains (CV, NLP, and Audio Tasks) using vision Transformers (ViT) instead of training conventional structures such as ResNet from scratch. The main contribution is providing the pre-trained models that serve as a benchmark for SSL algorithms. These evaluation models are based on ViT.
strengths: 1- The paper, in general, is well-motivated specially as it proposes to facilitate the development of SSl algorithms to research labs with limited computational resources. 

2- The variety of the datasets across multiple domains, and the considered 14 SSL algorithms. 

weaknesses: 1- In general, further clarifications are needed for some parts of the paper (points 2 and 3 below). 

2- For the sentence "Compared with training neural models from scratch, pre-training has much reduced cost in SSL, yet relatively few benchmarks offer a fair test bed for SSL with the pre-trained versions of neural models.", can authors provide references? If there are other test beds for SSL methods, it is recommended to mention them and exploit their limitations by which USB is addressing in this paper. I understand that the authors show that difference in GPU days needed to evaluate SSL algorithms in Table 1a with referencing [21]. However, further explanations and details into the weaknesses of [21] would improve the paper. 

3- No structural details and layers stacks are provided for the utilized ViT. Given that the main contribution of this work is the use of pre-trained ViT as an evaluation benchmark, I believe describing how the ViT is utilized along with its structure is of high importance. Also, it is not clear, for instance, if the same architecture is used for all tasks. It is mentioned that for CV, authors use a model similar to [27]. For NLP and Audio, on the other hand, it is mentioned that a transformer model similar to [64-66] is adopted. Does this mean that for each task, a different pre-trained model is used? 

4- While the Limitation section is appreciated, it is not clear why none of the mentioned methods in this section is implemented. A justification is needed in this case which leads to the question: Can USB be considered unified if popular GAN-based and/or GCN-based SSL methods are not implemented? 

correctness: The claims made are mostly correct. The proposed evaluation method can be strengthened if GAN-based and/or GCN-based methods are included. This would be more valuable when compared to using similar methods such as MiXMatch, ReMixMatch, and FixMatch.
clarity: The paper is, in general, well-written. 
relation to prior work: The authors mention plenty of previous SSL works. However, further details are needed to compare against existing evolution protocols in reference [21]. See the weaknesses section. 
additional feedback: 1- Not all SSl algorithms achieve optimal results on Pre-Trained ViT-S-P2-32 (Figure 1b). Can this observation be justified?
 
2- Run-time results for Tables 5-9.
review_title: Paper is well-motivated. The proposed benchmark is of high importance in developing SSL algorithms. However, more clarifications and algorithms are required. 

Meta-review: metareview: This article observed an active rebuttal period and definitely went to a final form that should be published at this venue, as highlighted by the latest updates of all reviewers. It will be a good add to the broad (multi domain) semi-supervised community!


confidence: 5: The area chair is absolutely certain
Decision: Accept