Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduced a community library, BIGBIO, that provides curated set of meta-datasets and data governance capabilities for biomedical NLP tasks evaluation. This work differs from existing BioNLP benchmark datasets as it provides tooling to automate ML workflows through programmatically accessing and ingesting data.
strengths: * Direct contribution to dataset quality, data governance, task schema harmonization, ML workflow integration, etc.
* Highly-credibility crowd-sourced effort in the dataset curation through a hackathon
* Identifying the performance limitation of existing LMs towards solving BioNLP tasks. The results from the general LM were somewhat expected and aligned with the previous research on domain-specific LM performance (e.g., PubMedBERT performance comparison).
* Results from T0 class of models were promising for shaping future research direction on specialized domains like biology or chemistry; The performance may suggest that the models that were capable to learn the task or functionality directly may generalize to new scientific domains.

weaknesses: * Lack of discussion towards few-shot NER implementation. NER finetuning performance is reported in the Section 5.2 with MTL model. However, there seems no performance advantage over the SOTA baseline. Bit skeptical on the poor performance of MTL over the LinkBERT though MTL encoder was initialized with the BioLinkBERT-base.
* How does the prompt designing on BioNLP tasks differ from general NLP tasks? The difficulty on MedNLI, GAD, and BIOSSES remained due to the lack of selecting best prompts?
* Since the authors proposing a framework design, there might be additional elements that they need to consider to adopt this library in the scientific ML domain. For example, complex wide data policy/strategy, complex security consideration, E.g., classified data, propriety data, scientific value, scale data infrastructure and frameworks for HPC facilities. 

correctness: The claims made in the submission is correct for the most parts. However, the reasoning on poor performance of MTL over the SOTA baseline can be improved.
clarity: Yes.
relation to prior work: Yes
additional feedback: N/A
review_title: Important community library for BioNLP tasks, but more research needed to show the utility of the datasets.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper has introduced a community library- BIGBIO which has more than 126 biomedical NLP datasets, covering multiple tasks and supporting various languages. It allows programmatic access to datasets and even supports platforms for prompt engineering along with evaluation of few/zero shot language model.
strengths: The BIGBIO differs from all previous works by providing programmatic access to datasets that are unit-tested, supporting various languages. They provide guides for new datasets. BIGBIO can be used in zero-shot biomedical language evaluation as well as multi-task learning. They have provided a Python package that supports 126 biomedical tasks that can be used for multiple tasks.  BIGBIO increases robustness for data-centric machine learning, and it also reduces the amount of time and work needed for training and evaluating models. Most importantly, it is a useful interface for loading large number of biomedical datasets. Furthermore, the paper is easy to follow except in few instances (which are discussed below).
weaknesses: Figure 1 and Figure 2 lack any explanation in the paper. They have not been cited in the text. The paper does not elaborate much on how it supports the datasets that need credentialed access from the users as such datasets have restricted usage. Also, there is no information about the values represented in Table 2. Is it F1 scores? Furthermore, line 222 mentions "trained or finetuned". Is it supposed to be "pre-trained" instead of "trained"? That part of the text is unclear and should be properly explained. Some concepts like "majority class" have not been explained. 

Minor weaknesses:
Missing citation in line 219: "We replicate the finding in Sanh et al. that models". 
The text in Figure 2 is too small.
correctness: Yes
clarity: The paper is clear and well-written.
relation to prior work: Yes, the authors have described the lack of such a community library for biomedical datasets which is the motivation behind the work. They have clearly explained that the BIGBIO differs from all previous works by focusing on the infrastructure and curation needed for generation of meta-datasets. Unlike the previous works, BIGBIO provides programmatic access to datasets.
additional feedback: The library introduced in the paper has the potential of being a good resource for the biomedical community.
review_title: Official review for Paper249

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper introduces a biological natural language processing benchmark with a large number of different tasks and provides complete community tools and third-party library support.

strengths: 1. The paper provides a professional and complete system solution for large-scale multi-task learning in biology
2. The article provides a useful exemplar for designing unified dataloaders and schema harmonisation.
3. This work systematically collects and organises a large number of existing datasets, which is a tedious but important task that should contribute significantly to the standardisation and advancement of the field.
weaknesses: 1. The paper writing can be improved. Some descriptions are hard to follow (e.g., line 24).
2. In experiments such as Figure 3 and Table 2 only generative pre-trained language models are included, but there are also many important masked pre-trained language models, such as RoBERTa and PubmedBert, which should not be completely ignored.
3. There may be data-overlap between the included task and the training data for pre-trained language models, which may introduce unfair comparisons: a model may perform well simply because it has seen more repeated data. How to avoid such bias?

correctness: The dataset and evaluation methods are presented in a sound way
clarity: The paper is generally clear, although the writing could be improved.
relation to prior work: Related works are clearly discussed. 
additional feedback: NA
review_title: This work collects and organises a large number of different biological natural language processing tasks, and provides a complete set of benchmark tools to support them.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes BIGBIO, a community library of 126+ biomedical NLP datasets, currently covering 12 task categories and 10+ languages. BIGBIO facilitates reproducible meta-dataset curation via programmatic access to datasets and their metadata. Moreover, it is compatible with current platforms for prompt engineering and few/zero-shot language model evaluations. This paper discusses the process for task schema harmonization, data auditing, and contribution guidelines and outlines two illustrative use cases. Overall, this paper is well-written and well-motivated and releases a very useful and important resource for the community.
strengths: 1. This paper proposes  BIGBIO, a community resource for programmatically accessing biomedical NLP 33 datasets at scale and encouraging reproducibility when generating meta-datasets, which are well-written and well-motivated.

2. This paper contains the comprehensive process of task schema harmonization, data auditing, and contribution guidelines.

3. This paper conducts the zero-shot evaluation of biomedical prompts and large-scale, multi-task learning.
weaknesses: 1. I think the writing can be further improved to highlight the motivation in the introduction section.
correctness: The claims in this paper are correct, and the datasets are constructed in a sound way. The evaluation methods and experiment design are correct.
clarity: This paper is well-written.
relation to prior work: Yes.
additional feedback: I recommend revising the introduction section to improve the writing and highlight the motivation.
review_title: A very good paper with a focused contribution.

Meta-review: metareview: This submission introduces a "meta dataset", consolidating and harmonizing multiple bio-medical NLP datasets across several languages.

The submission generated good discussions between the reviewers and the authors as it shows potential utility of the corresponding dataset. There is hope that this dataset will spur forward research in biomedical NLP.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept