Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents LEPISZCZE, a multi-task benchmark for evaluating neural models' Polish language understanding abilities. The benchmark includes five datasets from the KLEJ benchmark (released two years ago) and eight datasets mostly re-used from existing publications. LEPISZCZE covers various tasks across multiple domains, including text classification, sequence tagging, and sequence-to-sequence tasks. The authors conduct a series of extrinsic experiments using the proposed benchmark and monolingual and cross-lingual language models for Polish. The results indicate that the models perform strongly on many of the LEPISZCZE's tasks. At the same time, the work provides a flexible infrastructure that supports adding and documenting new tasks and models in a unified manner and practical recommendations on benchmarking.
strengths: 1. The paper fits the track well and addresses benchmarking for non-English languages, making a great attempt to create more resources for language model evaluation in Polish. The proposed benchmark also prompts the study of cross-lingual models' knowledge transfer abilities. 
2. The benchmark's infrastructure, including data and experiment versioning, is an important consideration for evaluating models and updating the benchmark with new tasks or potential dataset edits. 

weaknesses: 1. The motivation behind choosing the benchmark datasets is unclear. In particular:
* Why the five KLEJ tasks are included and the others are not? 
* I also wonder why the authors include the tasks (e.g., CDSC-E, NKJP POS) that do not seem to be that challenging for the considered models. 

2. The authors state that they allow sorting submissions by various metrics (lines 158--159). However, it is unclear if the metrics are appropriate for all considered tasks and how the variability in the evaluation would reflect the progress on the benchmark. Besides, as far as I understand, some metrics are task-specific (e.g., https://huggingface.co/datasets/clarin-pl/2021-punctuation-restoration). Does the evaluation design account for them in the model ranking?

3. The authors provide recommendations on creating benchmarks for resource-lean languages in Section 3. Recommendations based on 3.1.2 and 3.1.3 do not have an empirical foundation and can be formulated as future work. At the same time, it is unclear how the benchmark design, apart from the infrastructure choices, can be replicated in other languages (e.g., lines 49--51). 

4. The limitations of this work are paid little attention to. Could you please add the Limitations section?
correctness: As mentioned in the **Weaknesses** section, unifying task-specific metrics for all datasets can be confusing and lead to spurious conclusions about the model evaluation. Given that homogeneous metrics are used under such an approach, it is quite natural to rank the models by averaging their performance across all tasks. The model ranking on the potential LEPISZCZE leaderboard is not discussed, and how the ranking would change if the user wants to evaluate the models by another supported metric. The authors could also draw a few conclusions on the model performance across the domains. 


The experimental setup lacks simple baselines, such as random guessing and simple linear classifiers, where applicable. I would also suggest adding more details on how the encoders are fine-tuned for the punctuation restoration task. However, the unified model finetuning and evaluation infrastructure is definitely a strong point.
clarity: To me, the motivation of the work can be more focused on in the Introduction section. Contributions (1 and 3) and (2 and 4) can be combined. The authors could also provide more details on the dataset collection. To do this, the authors could remove content that is duplicated in the dataset cards (e.g., https://huggingface.co/datasets/laugustyniak/abusive-clauses-pl). 

Based on the dataset description, the task formulation is not always precise. I would suggest adding examples for each dataset for better understanding.
relation to prior work: The work discusses related benchmarks for non-English languages and highlights the limitations of the closely related KLEJ benchmark. I would suggest straightforwardly stating how the proposed benchmark overcomes the KLEJ limitations. 
additional feedback: **Appendix**

The authors do not provide appendices and supplementary materials that should include the discussion on the related work (lines 88--89, 97--98).

**Suggestions**

Overall, in the revision, I would suggest:
1. providing the dataset examples, more details on the datasets, and the supplementary materials
2. clarifying the motivation behind the dataset choices and how the KLEJ limitations are addressed in the proposed benchmark
3. discussing the limitations and ethical considerations
4. continuing a discussion on the other mentioned comments, particularly the metric choice and model ranking.


**Questions**
Are there any plans on establishing the human baseline?
review_title: This is the way - lessons learned from designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents an expansive NLP benchmark task for Polish NLP, modeled after GLUE (LEPISZCZE is the Polish word for glew). The benchmark addresses an urgent and important need to improve NLP for popular languages spoken around the world.
strengths: * The contribution addresses an urgent and important need in NLP
* The benchmark is clearly described and well-documented
* The paper includes initial results on five different language models
weaknesses: While the core contribution of the paper is strong, I noticed a few ways the writing and organization could be improved before publication. My main suggestion is that the authors include more justification and reasoning for their choices. Currently, the paper clearly describes the choices made but not always *why* they were made.

I also have the following technical suggestions:
* In-text citations should be changed so they are parenthetical (this is probably just a matter of using a different latex command). For example, on page 2, `protocol Pineau et al. (2021)` should be `protocol (Pineau et al., 2021)`. This is a recurring error throughout the paper.
* The caption for Figure one says `NLU benchmarks`
* On page 3, I suggest a table for the benchmarks listed (rather than the long paragraph of authors)
* Is there a missing citation on line 91? (The text says `edition of Ethnologue?:`.)
* The header for 3.1.1 and 3.2 seem very similar and imply redundant sections
* On line 190, the sentence `We conclude agreements over the Internet daily` is unclear. Would "make agreements" or "accept agreements" fit?
* I suggest the authors include a discussion of limitations (or potential limitations) in section 5, as well as a discussion of ethical considerations.
correctness: The main claims of the paper are correct and supported with evidence.
clarity: The paper is clear and concise.
relation to prior work: Section 2 describes related work in sufficient detail.
additional feedback: No additional feedback.
review_title: Promising NLP Benchmark for Polish

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This papers offers a novel benchmark for Polish. The benchmark consists of 13 existing dataset. Experimental evaluation covers 5 mono- and cross-lingual models. The papers makes a solid contirbution to the NLP benchmarking area.
strengths: - The paper extends the scope of mono-lingual benchmarking. The LEPISZCZE benchmark can be used to track progress in developing mono-lingual models for Polish and cross-lingual models overall.
- All datasets but proprietary ones are distributed via HuggingFace hub for better accessability.
- The baseline movel evaluation is well documented via MLOps tools.
weaknesses: - The paper aggregates 13 existing datasets into a single benchmark. No novel datasets are introduced. The focus of chosen datasets is not well explained — it is clear, that the authors tried to account for a large number of datasets, but it is not clear how these particular datasets were chosen. These datasets do not cover a standart NLU / NLP scope or any domain-specific problem.
- Mono-lingual datasets sich as GLUE or SuperGLUE include a human evaluation as a reference. The LEPISZCZE benchmark lacks of human evaluation. The papper does not provide inisghts, whether a human baseline will be introduced as a starting points.
- The choice of models seems unclear. The only cross-lingual model used is a sentence model. All mono-lingual models are encoders. Why not use a cross-lingual encoder such as RemBERT, mBERT, XLM-R?
- There is no overall metric in the benchmark. To this end, it is not clear, how to choose best model.
correctness: Some of the choices in the benchmark design are questionable.(i) the choice of datasets and (ii) the choice of models. Human evaluation is not a must, but could provide better understanding of model’s skills.  There is not aggregated metric in the benchmark, so it is not clear how exactly the models should be compared. 
clarity: - How the Fig.1 was designed, how is the choice of languages motivated? How the number of native speakers was estimated?
- What are the strategies to cope with standart splits problem and how are they implemented in the LEPISZCZE benchmark?
- How constiniuos benchmarking is implenented in the LEPISZCZE benchmark?
relation to prior work: The paper is grounded on previous reserach in NLP benchmarking. 
additional feedback: Please see the comments above.
review_title: A benchmark for Polish, 13 tasks and five models 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper extends an existing benchmark for Polish NLP (i.e., Klej) and presents LEPISZCZE, a new benchmark that integrates five tasks from the Polish benchmark and eight new datasets. Also, the paper conducts experiments on the benchmark based on the F1 macro performance of some recent LMs for Polish. 
strengths: - The benchmark construction and improvement for Polish NLP is well motivated. And the discussion of limitations of previous benchmarking seems reasonable to this reviewer.
- Codes, models, datasets, and parameters are clearly provided, which promotes reproducibility.

weaknesses: - The benchmark construction process is unclear: 
  - The selection criteria of the five datasets from Klej and new datasets are not clarified. Moreover, what is the overall design idea behind all the datasets/tasks in LEPISZCZE?
   - The evaluation metrics for different datasets are not described.
   - The organization of different datasets. In the paper (line 186), the authors claim that they unify different datasets into the same form. How the authors do that is not described.
    - The splits of train, dev, and test sets are not clear. Do the authors use the original splits of the corresponding datasets?

- The limitations of previous benchmarking are not addressed. Given the existing benchmark Klej, the paper does not show much improvement besides adding new datasets:
    - Single metric evaluation is identified as a limitation of prior work. But the paper seems to also use a single metric for model evaluation. 
    - The paper does not show improvements over standard split problems.
    - The paper does not show sufficient proof of ensuring continuous benchmarking.
    - The benchmark datasets seem to all relate to classification tasks, other important tasks such as generation tasks (e.g., summarization, translation) are not included.
    - According to the evaluation, HerBERT seems a clear winner. It outperforms other models on the majority of newly proposed tasks, which weakens benchmark contributions to some extent compared to the situations where different models perform better at different tasks (in Klej).
correctness: As mentioned in the weaknesses, the paper does not clarify the benchmark construction process. Moreover, the paper claims that the proposed benchmark uses different metrics while it does not report and compare them in the evaluation (it just uses the F1 score), which is confusing to this reviewer.
clarity: The paper is a bit hard to follow, especially since the selection criteria of datasets in the benchmark are not clarified. And the paper can reduce the length of background/motivation about benchmarking Polish NLP.
relation to prior work: No. The paper points out the limitations of Klej (an existing Polish NLP benchmark), including task formulation, and single evaluation metric. However, the paper does not show and clarify the corresponding improvements over the aforementioned limitations. 
Moreover, this reviewer suggests that there should be more discussion about related work on benchmark construction (e.g., tasks design and formulation, evaluation metrics) of other languages, especially low-resourced languages.

additional feedback: I did not see the supplementary material in this submission as mentioned in the paper (Line 89,98). More importantly, I did not see the leaderboard (Line 111, 117) as promised in the paper.
review_title: A good benchmark for Polish NLP

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Summary:
This paper introduces the benchmark LEPISZCZE to benchmark for Polish NLP. Specifically, results on 13 tasks from give LMs for Polish are reported in this paper.

Contributions:
1) more dataset in Polish are included to evaluate Polish LMs
2) design the benchmark from the perspectives of data splits, continuous maintainess, etc
3) lessons learned during benchmark design are summarized and are expected to be helpful for benchmarks concerning other low-resource languages 
strengths: 1) multiple splits are considered to evaluate performance more comprehensively
2) a variety of NLU tasks for Polish are considered and evaluated in this paper
weaknesses: 1) The contribution to datasets is limited since all of the used datasets are not new and have been investigated in other literature before.
2) From the perspective of benchmarking, most of the studied models are bert-based. However, there are quite a few recent multilingual LMs and the comparison would be more complete if these models are considered, e.g., Roberta (sdadas/polish-roberta-base-v2), mT5 from Google (google/mt5-base), InfoXLM from Microsoft (microsoft/infoxlm-base), XGLM from meta (facebook/xglm-564M), mGPT from SberDevices (sberbank-ai/mGPT), etc.
3) Since the paper is aimed at benchmarking NLP tasks for Polish (as paper title suggests), more important NLP tasks besides text/token classification could be considered, e.g., generation, question answering, summarization, translation, etc.
correctness: The claims made in the paper are correct.
clarity: 1) PAC dataset description is not clear, mixing examples and dataset descriptions
2) In line 172 of page 5:  what does “other splits” mean here? 
3) From line 210 to 219 on page 6: details especially information links could be provided as footnotes. Otherwise, it becomes difficult to distinguish websites and normal texts. Meanwhile, there is a undefined citation (question mark) for the corpus of polish call center dialogs.
4) From line 244 to 247 on page 6: is this a summarization task or classification task? Since the original dataset is built for summarization and previous work formalized it as text-similarity, how does this paper define the task on this dataset? According to Table 1, PSC is defined as a paraphrase classification task. It could be much more clear if the dataset description part could declare more straightforwardly.
5) Considering the variety of datasets, the description could be more clear if the datasets are listed according to their task type, following what has been done in glue paper (single-sentence tasks, similarity and paraphrase tasks, inference tasks)
relation to prior work: The paper has mentioned this work as an extension of the existing benchmark for Polish NLP KLEJ. However, there are a few points unclear related to KLEJ and LEPISZCZE:
1) Why regression tasks studied in KLEJ are not studied here: CDSC-R and AR
2) Why CBD (Cyberbullying detection) and NKJP-NER (NER classification) are not included in this work? 
additional feedback: I have a few related questions and hope they could be answered by the authors:

1) It has been mentioned that “Using static data splits leads to over-fitting and results in quick benchmark saturation. ", as the motivation for multiple split evaluation. But for some of the benchmarks, the ground-truth of test splits is unprovided and we can only get performance by submitting predictions. Why this kind of benchmark still face the problem of "over-fitting" and "quick saturation"?
2) The hyperparameter names and ranges are not provided in Section 4.2. It would be great if a table reporting those details is included in the main paper or appendix
3) Do you mind adding the resource and year of numbers for Figure 1? Since the number of speakers for a specific language changes year by year, some values I searched online does not align well with those demonstrated in the figure, e.g., 
- English speaker is around: 1350 million in 2021 (https://www.babbel.com/en/magazine/how-many-people-speak-english-and-where-is-it-spoken#:~:text=Out%20of%20the%20world's%20approximately,%2C%20in%20third%2C%20comes%20English.)
- Chinese speaker is around: 1300 million in 2021 (https://www.babbel.com/en/magazine/the-10-most-spoken-languages-in-the-world#:~:text=1.,spoken%20language%20in%20the%20world.)
- Indian speaker is around: 615 million in 2019 (https://newsonair.gov.in/News?title=Hindi-is-3rd-most-spoken-language-in-the-world-with-615-million-speakers-after-English%2C-Mandarin&id=381514#:~:text=Hindi%20is%20the%203rd%20most,list%20with%201%2C132%20million%20speakers.)


Below are some typos and could be corrected for the next version of this paper:
1) should be "reveal" rather than "revile" (line 166 of page 5)
2) "started working" rather than "started work" (line 183 of page 5)
3) left double quote should be “  rather than  ”​ (multiple occurrences between line 188 and line 200 for PAC dataset description)
review_title: This paper benchmarked existing language models (either pretrained on Polish or multilingually) on a variety of NLP tasks in Polish

Meta-review: metareview: This work is about a multi-task benchmark for evaluating Polish NLU, called LEPISZCZE. This curates 5 datasets from an existing benchmark and adds eight more with a bit of reusage. There are a variety of domains and tasks covered. The authors conduct various experiments on the benchmark to evaluate performance of monolingual and crosslingual language models. Models perform pretty well on this benchmark already. Overall, the authors should include more motivation on exactly how these benchmarks were chosen and no new datasets are contributed. However, the benchmark clearly has value and creating benchmarks for more languages is important work for the community. Further, authors & reviewers had excellent and thorough discussion on reviewer concerns. 
confidence: 5: The area chair is absolutely certain
Decision: Accept