Title: M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection
Benchmark: M4GT-Bench
Summary: M4GT-Bench, a multilingual, multi-domain benchmark for detecting machine-generated text (MGT). The benchmark extends the previous dataset and contains three distinct tasks: Binary MGT Detection (classifying text as human-written or machine-generated), multi-way Generator Detection (which LLM generated text), change Point Detection (text transitions from human-written to machine-generated).
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Machine-generated text (MGT) detection
Phenomenon defined by authors: Yes
Phenomenon definition: Machine-generated text detection as the task of identifying and differentiating LLM-generated text from genuine human-generated text.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Benchmark tasks: (1) Binary classification to determine if text is human-written or machine-generated, (2) Multi-way classification to identify which specific LLM generated a text, and (3) Boundary detection to identify where text transitions from human-written to machine-generated content.
Task item definition: Text sample, label human-written/machine-generated, model (if applicable)
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Represent realistic scenarios where one would want to detect fully machine-generated text, identify which model generated text, or find the point where human writing ends and machine generation begins.
Dataset size: 73,288 (total machine)
Dataset size notes: Mixed Dataset, PeerRead, e.g. ChatGPT as Generator: 3,649 train; 1,522 test; 505 dev
Dataset metadata available: Yes
Dataset metadata detail: Domain; language; generating LM; boundary position
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), Mean Absolute Error
Metric aggregation: Simple Mean
Metric statistics: Simple means for the main metrics
Metric subscores: Yes
Metric subscores detail: LM generator, language, domain
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Human evaluation for multi-way generator detection, where they test how well humans can distinguish between different LLM generators.
Taxonomy: NLP / 
New taxonomy: General Application / LLM Detection