Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: In this work, the authors extend the DABS benchmark [1] to include five more datasets. The new datasets are carefully chosen to cover a variety of underserved domains -- i) bacterial genomics, ii) semiconductor wafer manufacturing, iii) particle physics, iv) protein biology, v) satellite imagery. The authors provide evaluation of two new variations of masked autoencoders and contrastive learning under varying levels of corruption (masking) in the data for the five new datasets and the seven datasets from the original DABS benchmark.  Generalized MAE does well on the satellite image dataset but it is not consistently the case across domains. Furthermore, the optimal amount of corruption is not just domain dependent and can depend on other factors making it hard to choose.

[1] Tamkin, Alex, et al. "Dabs: A domain-agnostic benchmark for self-supervised learning." Neurips 2021 datasets and benchmarks track. 
strengths: The authors consider an interesting set of datasets from a wide range of disciplines. Given the wide range of datasets covered the updated DABS benchmark should be of interest to the broad community. To the best of my understanding, the authors have properly answered questions regarding the datasets containing offensive content. 
weaknesses: I have several concerns regarding the paper. 

Significance of the contribution:

1. Out of the five datasets that the authors use, three of them (higgs, genomics, eurostat) are directly available in tensorflow already. This implies the datasets are already in a usable format and I do not see what is the authors contribution in terms of these datasets.  The introduction and description in the main text feels misleading.  

2. The two algorithms that the authors use are a minor variation of MAEs and contrastive learning. Therefore, I do not think there is any insightful algorithmic variation that the authors introduce either. I understand that this is the benchmarks track and we do not expect completely new methods. Despite that I think what is introduced should not be stated as a new universal SSL method but should be honestly acknowledged as natural extensions/variations of well known approaches. 

3. Thirdly, the authors show a table of results. I did not gather any new insights from the results and that was not very pleasing. 

4. Finally, there is a bunch of experiments that the authors state ran into an issue and have been marked as pending (for some other results are marked OM). I understand that issues can happen and do not want to penalize authors for this. At the same time, it seems unfair to others. By accepting this we are allowing authors extra time to gather results that others did not have. 
correctness: In the main table, I am guessing authors are reporting accuracy? Also, what is going on with OM and DV? 
clarity: The paper is reasonably well written. 
relation to prior work: Yes, the authors have discussed prior work. However, i feel the contributions on top of DABS benchmark are limited and do not seem to merit a full paper yet. 
additional feedback: I do not have additional feedback. I look forward to authors responses as I might have missed something.
review_title: Review for DABS 2.0: Improved Datasets and Algorithms for Universal Self-Supervision

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Five new real-world datasets in science and engineering are added to the original 7 datasets in DABS 1.0. Also, two additional unsupervised learning methods are introduced - Capri, and a generalized masked encoding. Both these algorithms and the ShEd algorithm from DABS 1.0 are then applied to all the 12 datasets and the variation of influence of corruption fraction on performance, across the 12 domains (and the three algorithms) is observed. The process and results convey the potential of using DABS 2.0 for assessing robustness of domain-agnostic SSL methods and also for studying the effect of design decisions across domains.

strengths: - Supports generalization of SSL algorithms to less-studied and diverse domains and modalities
- Enables assessment of how certain design choices for one domain affects the performance on another domain. In this work, the cross-domain difference in the influence of masking / permuting of embeddings becomes evident.
- Tasks associated with the datasets introduced are real-world tasks.
- One of the datasets - Bacterial Genomics includes in-distribution and out-of-distribution data enabling an assessment of robustness of models to distributional shifts.
- Each newly introduced dataset is of a different size which can help get an idea of how a domain-agnostic algorithm is able to scale.
- The baseline results indicate that the same SSL algorithm and settings do not perform well in all domains. This is the challenge for SSL algorithms - to generalize to multiple domains without needing manual domain-specific tuning. The fact that the results indicate this challenge, supports the purpose of this dataset.

weaknesses: - Easy execution of pre-training and transfer learning with the help of newly added code is mentioned but the code / dataset are not made available for review.
- Claims surrounding 'Universal SSL' may be a bit too broad, since graphs, point clouds, etc. are not included.
correctness: Yes, the paper appears correct.
clarity: Yes, the paper is clear.
relation to prior work: Yes, the authors discuss prior work.
additional feedback: _Minor Errors:_
- The DABS 1.0 image in Fig. 1 can be a bit bigger.
 
_Notes to the author:_
-  Performance could be more easily interpreted if the performance data in the table were presented in a different format (plots with performance as a function of corruption level)
review_title: Review for DABS 2.0.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper extends the DABS benchmark with datasets from five new domains, proposes a new universal SSL algorithm extending masked auto-encoding. The paper also investigates an interesting metric, the corruption rate across the different domains.
strengths: * Extension of a well-established benchmark
* Significant contribution with 5 new datasets domains 
* New domains are novel in terms of being less studied than extensively studied, such as images, text, speech, etc...
* Interesting investigation of corruption rate and how it differs based on the domain at hand
* Discussion of external and internal validity


weaknesses: The scope of the paper is relatively wide with the multiple domains and the two new algorithms. This also makes it difficult to supply a datasheet for all newly introduced datasets, elaborate on the datasets in terms of split, distribution, etc... Additionally, it makes it challenging to cover the details of the benchmarking with a good level of detail.

correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: The paper is well-written, I only noticed these minor issues:

* Line 101, an extra “a” is there before “the”
* Line 153, probably a semicolon is better before "namely" for readability
* Line 182, capitalize words of section titles for consistency
* Line 206, probably a simple diagram with inputs (script, domain, algorithm, corruption_frac) is better than having the python command, but I leave this up to the authors.
* Line 266, I think a comma is missing after “However”
review_title: Extension of a universal self-supervision framework including new datasets and algorithms

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors contribute DABS 2.0, an extension to DABS dataset for universal self-supervision now with additional five new domains. The authors also propose two new algorithms and evaluate them over all the 12 domains with different corruption fractions.
strengths: The paper takes universal SSL one step further in both dimensions: datasets and algorithms. This is a good contribution for the community.
weaknesses: The 5 new domain datasets are legitimate contributions; however, in my view, MAE does not account as a novel contribution to this research. A lot of implementation details (in section 3) could have been provided to make it a thorough read.
correctness: The claims, dataset, algorithms, and experiments look correct.
clarity: The paper is interesting to read and is written well.
relation to prior work: Yes, but it is an extension to a prior published work.
additional feedback: -> A typo on line 573. The section is not referred correctly.
-> Line 151 seems to be grammatically incorrect.
review_title: Extension of DABS 1.0 in terms of dataset and universal SSL techniques.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposed a benchmark DABS 2.0, which contains extended real-world science and engineering dataset domains and a universal SSL algorithm. 
strengths: 1. A universal SSL algorithm is proposed along with the benchmark and works as a strong baseline.
2. The insight of evaluating and employing the universal self-supervised on real-world science and engineering datasets would be helpful for both AI for Good and AI for Science.
3. Comprehensive discussion including both internal validity and external validity
weaknesses: 1. The results part mainly explored the masked ratio experiments on the proposed benchmark, compared with two baseline methods. More details on the experiments, models, hyper-parameters are expected to be provided.
2. The documentation of the codebase is constrained into README file without the detailed APIs .
correctness: The claims are correct.
clarity: The paper is well-organized by introducing the dataset and then algorithms. The results and discussions are also provided.
relation to prior work: Yeah, the paper can be viewed as an extension of DABS 1.0 with proposed SSL algorithm.
additional feedback: 1. It would be helpful to see more analysis on the benchmark. For example, for each domain, what is the minimum data requirement to train a strong SSL model. Collecting data of different domains usually has different difficulty. 
2. The training details are not very clear. The model used, the parameters, the training time, etc.
review_title: Good Benchmark

Meta-review: metareview: DABS 2.0 extension of DABS to include five more datasets, and serves as benchmark for self-supervised representation learning. The new datasets cover new domains such as genomics, industrial images, biology, satellite imagery, etc. In addition, two new self-supervised learning methods are evaluated and their robustness to domains and hyper parameters are evaluated. In addition a new technique called Capri is introduced, which combines benefits of masked auto-encoders and contrastive learning to learn representations. 

The reviewers are generally positive except R1 who recommends reject. The main concern of R1 is that the contributions over the DABS is not significant (e.g., several datasets are available in tensor flow). Contrast this with R2, R3, R4, R5 who find the contributions valuable. Given the importance of unsupervised representation learning and the significant effort being put in the research community this work is valuable. Thus they recommend accept. 
confidence: 5: The area chair is absolutely certain
Decision: Accept