Title: Revisiting non-English Text Simplification: A Unified Multilingual Benchmark
Benchmark: MULTI-SIM
Summary: MULTI-SIM, benchmark for multilingual text simplification, containing complex-simple sentence pairs across 12 languages. They show improvements from multilingual training for non-English languages and strong performance of Russian for cross-lingual transfer.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: automatic text simplification, multilingual
Phenomenon defined by authors: Yes
Phenomenon definition: "Automatic text simplification (ATS) is the task of reducing the complexity of a text without chang- ing its original content and meaning"
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Task definition: Automatic Text Simplification transforms complex sentences into simpler versions that maintain the original meaning but reduce linguistic complexity.
Task item definition: Pair of sentences: a complex sentence and its corresponding simpler version that preserves the original meaning.
Task source: Real task examples (e.g. GitHub issues), Expert-crafted task examples (e.g. hand-written examples), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: Sentence pairs come from real-world simplification efforts intended for actual audiences with lower literacy levels.
Dataset size: train: 653,468
Dataset size notes: test: 6,306; dev: 6,728
Dataset metadata available: Yes
Dataset metadata detail: language, script, domain, collection approach, simplification type
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric aggregation: Simple Mean
Metric statistics: Automatic Evaluation Metrics (SARI; BLEU); Measure inter-annotator agreement using Krippendorff's alpha
Metric subscores: Yes
Metric subscores detail: Scores for each language dataset separately, single-dataset fine-tuning, joint language fine-tuning, zero-shot cross-lingual transfer
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: Yes
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: 

(1) human evaluation to assess the quality of model outputs on their benchmark; 
(2) inter-annotator agreement using Krippendorff's alpha; (3) analyze corpus statistics to understand dataset quality; (4) acknowledge limitations "we cannot guarantee the quality of each resource or validate the methods that the original authors used to create them"
Taxonomy: NLP / Summarization
New taxonomy: General Application / Summarization