Title: README++: Benchmarking Multilingual Language Models for
Multi-Domain Readability Assessment
Benchmark: ReadMe++
Summary: ReadMe++ is a multilingual and multi-domain dataset for readability assessment according to the Common European Framework of Reference for Languages (CEFR) scale in Arabic, English, French, Hindi, and Russian. The dataset is human-annotated and publicly available. The dataset can benchmark supervised, unsupervised, and few-shot approaches, and is measured by the Pearson Correlation between predictions and ground-truth labels (supervised, few-shot) or the Ranged Sentence Readability Score (unsupervised). 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Readability assessment
Phenomenon defined by authors: Yes
Phenomenon definition: Readability assessment is the task of determining how difficult it is for a specific audience to read and comprehend a piece of text. 
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: The model must classify the readability of a sentence according to the 6-point Common European Framework of Reference for Languages (CEFR). The scale proceeds as 1 (A1), 2 (A2), 3 (B1), 4 (B2), 5 (C1), 6 (C2), where A is for basic, B is for independent, and C is for proficient; the paper provides the full annotation criteria in the appendix. 
Task item definition: A single item is a sentence with its associated language, domain, sub-domain, paragraph, context, and readability assessment label. The paragraph and context are optional and provided for human annotators to aid in manual labeling. 
Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task source detail: Data is sourced from 21 types of text (e.g. textbooks, legal documents, etc.) from various open-source datasets or open-access resources. 
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Task ecology detail: The task would probably be integrated into user applications, but not directly asked for by the user. Provided real-world applications of readability assessment were controllable text-simplification, ranking search engine results by their level of difficulty, and selecting appropriate reading material for language learners. 
Dataset size: 9757
Dataset size notes: 60/10/30 train/validation/test
Dataset metadata available: Yes
Dataset metadata detail: Language, Domain, Sub-Domain, Context, Paragraph
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice
Metric definition: Distribution (perplexity, calibration, correlation)
Metric definition detail: The model has two metrics. Pearson correlation requires just model output, but Ranked Sentence Readability Score requires model access to access the LLM's distribution. 
Metric aggregation: Simple Mean
Metric statistics: Min, max, average
Metric subscores: Yes
Metric subscores detail: Unseen Domains per Data Source, Cross-Lingual Transfer
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors assess their construct validity when justifying the originality or contribution of their benchmark. They expand an existing scale grounded in literary research to be multilingual and balance several domains, which current assessments fail to do, to ensure the most reliable assessment of readability. 
Taxonomy: NLP / 
New taxonomy: General Capability / Multilinguality