Title: This is the way: designing and compiling
LEPISZCZE, a comprehensive NLP benchmark for
Polish
Benchmark: LEPISZCZE
Summary: Authors introduce LEPISZCZE, a new, comprehensive benchmark for
Polish NLP with a large variety of tasks and high-quality operationalization of the
benchmark.  LEPISZCZE was designed with flexibility in mind. Including new models,
datasets, and tasks is as simple as possible while still offering data versioning and
model tracking. In the first run of the benchmark, 13 experiments (task
and dataset pairs) were tested based on the five most recent LMs for Polish. Five
datasets from the Polish benchmark are reused and eight novel datasets are added. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: model performance on Polish language across various tasks (13)
Phenomenon definition: The ability of language models to understand and process Polish language across a diverse range of NLP tasks, evaluated using 13 task-dataset pairs that include classification, natural language inference, and sequence labeling tasks.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Each task in the LEPISZCZE benchmark is defined as a standard NLP problem—such as classification, sequence labeling, or natural language inference—applied to Polish-language datasets. These tasks test specific linguistic capabilities of models, like sentiment analysis, named entity recognition, part-of-speech tagging, and others.
Task item definition: there are datasets for 13 tasks.
Task definition detail:  Entailment Classification, Q&A Classification,  Sentiment Analysis, Paraphrase Classification, Abusive Clauses Detection, Aspect-based Sentiment Analysis , NER, POS  Tagging,  Political Advertising Detection,  Punctuation Restoration, Punctuation Restoration.
Dialogue Acts Classification
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 30,003
Dataset size notes: 204,504 and 9,970
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: mean and standard deviation
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Multilinguality / 
New taxonomy: General Capability / Multilinguality