Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces CBLUE, a chinese biomedical NLU benchmark. As part of assembling the benchmark, the authors annotated multiple datasets for different tasks that cover a wide spectrum of biomedical tasks. The authors provide many baseline numbers and an (amateur) human performance estimate, alongside in-depth documentation on how to use the data. 

Overall, if the documentation issues could be addressed and the evaluation process expanded, I'd be okay with seeing this paper be accepted. 
strengths: This paper describes a vast amount of work to both introduce datasets and a benchmark that uses them and make the benchmark accessible by providing introductory notebooks that work on servers with free GPU hours. In addition to the expansive starter code, there is also a lot of exhaustive technical documentation of the dataset formats, which makes working with the data much easier. 
weaknesses: My main concern with this paper is about its lack of documentation of anything that is not the data format. There is insufficient information about the crowdsourcing process and no consideration of negative impacts, both of which are crucial in the biomedical domain where high-quality annotations are incredibly important.

Specifically, my points of critique are 

1. The benchmark, while including much long-tailed data, does not explicitly differentiate between those examples and head examples in the reporting of accuracy, which can lead to an overestimate of real-world utility. Summarizing performance with a single accuracy number has been rightfully critiqued and new benchmarks should not fall into the same trap. 
2. It reads from the description in 4.2 that all the crowd workers were untrained. While I sincerely hope that this was not the case for the collection of the data, this requires a disclaimer in the text itself that the “human performance” is not the performance of a trained professional, but that of amateurs with no medical experience. For example, in the table it should be labeled as “majority vote of amateur humans”. The section also doesn’t mention how many humans were chosen to annotate each datapoint and how many dev data points they had to annotate during training, or how long the task took or how much was paid. Moreover, for both the data collection and human baseline, there was no provided information about annotator agreement or other data quality indicators. 
3. Nowhere in 4.3 does it state why the case studies are particular to the Chinese language, instead of general findings that you would also see in other languages (note that I do not want to discredit the value of non-English datasets, just that the particular instances don’t say much about Chinese)

correctness: AI is not changing medical practice, as suggested by the abstract, and instead any system should be critically examined (see, e.g., https://www.nature.com/articles/s42256-021-00307-0)

clarity: There are a couple of unclear points in the text: 

- How was the “meaningless text” found in the CHIP-CTC processing?
- While it is stated that no PII is included in the data, it would be nice to empirically verify this or describe the process through which this is ensured, similar to how MIMIC does it
- How were diagnosis items sampled for CHIP-CDN

relation to prior work: n/a
additional feedback: While the KUAKE datasets sound super interesting, I wonder if it is actually too different from the other tasks and may be worth more exploring in a separate benchmark.
review_title: Amazing resource of epic scope with a few suggested documentation and writing improvements.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a new benchmark called CBLUE, Chinese Biomedical Language Understanding Evaluation, for medical language understanding. 
strengths: - The authors unified eight biomedical language understanding tasks in Chinese to form a benchmark, which bio-data is relatively sensitive and hard to obtain.
- The dataset has been applied and investigated by researchers in the field.
weaknesses: - No dataset comparison with existing Chinese understanding tasks, for example, CLUE (https://github.com/CLUEbenchmark/CLUE), in terms of size, task diversity, etc.
- I would expect to have some results and baselines that are fine-tuned from existing Chinese NLU work. For example, what is the gap between standard Chinese NLU and Chinese bio-NLU?
correctness: - My biggest concern about the correctness is that whether the collected annotation is representative and correct. 
- For example, for a well-trained annotator (based on the training description in section 3.3), the F1 scores in several tasks such as CMeEE, CMeIE, and CDN are only 60-70% as human performance.  
- Therefore, I have a concern that whether the annotation could be different if we ask experts and doctors to do the job?
- What is the confident from 1 to 10 that the authors believe their gold labels are correct?
clarity: - Please bold the highest performance in Table 2.
- Figure 1 (a) is not informative, please provide details of each ratio.
relation to prior work: As I mentioned in the weakness part, I believe it is better to compare with the existing Chinese NLU datasets that are not necessary to be in the bio-domain.
additional feedback: N/A
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents a benchmark for Chinese Biomedical NLP. 
It contains eight datasets (a combination of existing and new datasets), including NER, classification, measuring sentence similarity. 
These tasks are framed as either sentence-level or token-level tasks.

In addition to scholarly articles, clinical notes, this benchmark contains queries of search engines, which to my knowledge is under explored data source.
strengths: * A platform for evaluating Chinese Biomedical NLP models, convering various tasks.
* A bunch of existing BERT-based models are evaluated on the proposed benchmark.
weaknesses: * The novely and target audience of proposed benchmark seems limited, because it follows previous efforts (GLUE, CLUE) and builds a domain-specific language-specific benchmark.
* The authors mention that the data are proprietary (no license?), and this may pose restrictions on researchers to use it.

correctness: Yes
clarity: * The paper is well written. 
relation to prior work: * Yes
additional feedback: * Section 1, 'Anglo-centric' may not be the proper word, since most of these datasets you mention are US-based.
* Section 3.3, Fleiss' Kappa seems unbelievable high. 0.9 is not 'moderate', it may be described as 'almost perfect agreement'
review_title: A GLUE-like benchmark for Chinese Biomedical NLP

Meta-review: The reviewers feel that the new dataset is interesting but the contributions are not sufficient for acceptance. (1) The authors need to clarify which part was annotated by trained professionals and which part was annotated by amateurs. (2) The authors need to compare CBLUE and CLUE. 
Decision: Reject