Title: DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios
Benchmark: DetectRL
Summary: Evaluate LLM-generated text in realistic scenarios. The benchmark collects human-written texts from high-risk domains, generates comparable texts using popular LLMs, and applies various attack methods to simulate real-world conditions. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: LLM-generated text detection
Phenomenon defined by authors: Yes
Phenomenon definition: Defines LLM-generated text detection as the task of discriminating between human-written and LLM-generated texts with a focus on real-world scenarios.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Detect whether text is human-written or LLM-generated, with a focus on real-world scenarios that include various prompt usages, human revisions, and writing noises.
Task item definition: Text sample that is either human-written or generated by an LLM, along with its corresponding label.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: DetectRL simulates realistic conditions under which LLM-generated text detection would need to operate - while not being a complete real task.
Dataset size: 100,800 human-written samples
Dataset size notes: Task-dependent train and test split (Table 2, p. 4)
Dataset metadata available: Yes
Dataset metadata detail: domain, LLM type, attack type, text length
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Multiple choice
Response format detail: Requires a binary classification (human-written vs. LLM-generated) for each text sample.
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric aggregation: Simple Mean
Metric statistics: Simple mean to aggregate across different settings. For each detector, they report AUROC and F1 Score values for each specific condition and the average across those conditions. 
Metric subscores: Yes
Metric subscores detail: domain, LLM type, attack type, text length intervals 
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Specifically discuss the validity of their benchmark by comparing it to existing benchmarks and explaining why DetectRL better represents real-world scenarios. 
Taxonomy: NLP / Detection
New taxonomy: General Application / LLM Detection