Title: MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark
Benchmark: MULTITuDE
Summary: MULTITuDE, a benchmark dataset for multilingual machine-generated text detection. It contains human-written and machine-generated text across languages from multilingual LLMs. The authors evaluate how zero-shot and fine-tuned detectors generalize across languages and LLMs.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Multilingual machine-generated text detection
Phenomenon defined by authors: Yes
Phenomenon definition: Machine-generated text detection is a binary classification of a text to be human-written or machine-generated.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Binary classification of texts as either human-written or machine-generated (multilingual).
Task item definition: News article text sample in one of 11 languages, metadata on language, human-written / machine-generated,  LLM that generated it
Task source: Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Realistic detection scenario in which one would need to identify machine-generated news content in multiple languages, e.g., relevant for combating misinformation.
Dataset size: 44,786 (train set)
Dataset size notes: 29,295 (test set)
Dataset metadata available: Yes
Dataset metadata detail: language, text generation model, script type, language family
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation), Correlation (Matthew's correlation, Pearson's r)
Metric aggregation: Simple Mean
Metric statistics: Mean, ANOVA, post-hoc pairwise tests
Metric subscores: Yes
Metric subscores detail: Language; Generator LLM; Detection type; Monolingual or multilingual fine-tuning
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Detection
New taxonomy: General Application / LLM Detection