Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Authors build a new language understanding benchmark for Korean language. They build dataset for 8 NLU tasks namely topic classification, semantic textual similarity, natural language inference, named entity recognition, relation extraction, dependency parsing, machine reading comprehension and dialogue state tracking. Along with the dataset authors also release pre-trained model for Korean language based on BERT, RoBERTa etc. Authors have documented details about benchmark construction and testing process. 
strengths: - Large scale dataset on wide variety of language understanding task 
- A unified benchmark for evaluating Korean (13th most used language)  language models in the world will encourage research in Korean language.
- Benchmark has been built from ground up instead of translating existing English benchmark into Korean.
weaknesses: An evaluation with existing English benchmarks like GLUE translated to Korean language would have served as one weak baseline.
correctness: seems good
clarity: Overall the paper is well written. But paper could have been structured in a bit better manner. Also, papers need one careful reading for grammatical mistakes.

relation to prior work: Probably limitation of existing English dataset translated to Korean language could have been discussed in a bit detailed manner.
additional feedback: Read abstract and check it for Grammatical mistakes and correct it.
review_title: A new language understanding benchmark in Korean

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper ***1) introduces Korean Language Understanding Evaluation (KLUE) benchmark***, including eight different natural language understanding (NLU) tasks. Meanwhile, the authors ***2) release pre-trained language models*** and ***3) give insights for future works building similar resources***, both of which are equally valuable for the community.
strengths: The paper is well written with a well-designed structure. The three contributions listed above are significant, especially the third one sets up a typical example for future similar studies. The accessibility and accountability are clear with the data source and nice documentation. 
weaknesses: I have not found any important shortcomings in this paper. To make the paper stronger I would suggest extending the part of design principles to clarify the relationships to each task. This will give more insights for future studies.
correctness: Yes, the dataset is constructed in a sound way and the released pre-trained language models are reliable.
clarity: Yes, this paper is well written with a clear logic flow.
relation to prior work: Yes, the discussion of related work is detailed.
additional feedback: I look forward to seeing this paper accepted.
review_title: Nice work! 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes Korean Language Understanding Evaluation (KLUE), consisting of 8 NLU tasks, including classification, regression, machine reading comprehension, NER, etc. Different from other NLU benchmarks, all eight tasks in KLUE are newly annotated. To examine the benchmark, several new Korean pre-trained language models are built, which are based on BERT/RoBERTa. The results show that the KLUE-RoBERTa_large could achieve the best performance on most tasks than mBERT or XLM-R.
strengths: 1. The first general NLU benchmark for Korean.
2. All tasks introduced in KLUE are newly annotated with careful considerations.
3. Several new Korean pre-trained language models are introduced and evaluated on KLUE.
weaknesses: 1. Though I acknowledge the huge works by the authors, including constructing every task and pre-trained models, etc., the novelty seems to be limited, as most of the annotation procedures are not new. I'm not sure whether a non-Korean researcher could learn something new from this paper, other than its solid contribution to the Korean NLP community.
2. Another concern is that the proposed benchmark might only be interested in a limited research community (for studying Korean NLP). And I am not sure if it is suitable for NeurIPS. 
3. It is not clear whether the data for pre-training PLMs will also be released.
4. There is no human performance (or estimated) on KLUE.
correctness: Yes. But a limitation is that no human performance is provided.
clarity: Yes.
relation to prior work: Well-discussed. 
additional feedback: 1. Line 136: superGLUE -> SuperGLUE
2. Possible missing reference for the whole word masking strategy (line 258). And also, what tool was used for Korean word segmentation for WWM?
3. Unlike most of other NLU benchmarks, there is no overall score in KLUE. It is quite hard to compare different PLMs as a whole. If this is not designed, I'd suggest adding an overall score to your benchmark leaderboard.
4. line 292: The illustration 'a few interesting observation' is somewhat confusing, as these observations are quite normal and expected.
5. Will the pre-training model and data also be released to the public?
review_title: A Korean NLU benchmark

Meta-review: This work presents KLUE, which provides a set of benchmarks for natural language understanding. All reviewers agree this is high-quality work, and that the process of creating data was done in a thoughtful manner, with attention to detail. High-quality datasets in a multitude of datasets are indispensable and this work will contribute to work on low resource languages in general.
Decision: Accept