Title: Counter Turing Test (CT2): AI-Generated Text Detection is Not as Easy as You May Think – Introducing AI Detectability Index
Benchmark: CT2
Summary: Counter Turing Test (CT2) is a benchmark to evaluate the robustness of AI-generated text detection techniques. The AI Detectability Index (ADI) is a metric to rank LLMs according to how detectable their outputs are as machine-generated versus human-written.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: AI-generated text detection
Phenomenon defined by authors: Yes
Phenomenon definition: "If a work’s traditional elements of authorship were produced by a machine, the work lacks human authorship", Detectability of AI-generated text
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: The Counter Turing Test (CT2) evaluates the robustness of AI-generated text detection techniques: (i) watermarking, (ii) perplexity estimation, (iii) burstiness estimation, (iv) negative log-likelihood curvature, (v) stylometric variation, and (vi) classifier-based approaches. 
Task item definition: Pair of texts on the same topic (one human-written and one AI-generated).
Task source: Real task examples (e.g. GitHub issues), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 100000
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric aggregation: Formula combining perplexity and burstiness
Metric statistics: mean, standard deviation, entropy calculations, z-scores, p-values, bootstrapping, Le Cam's lemma, multiplicative damping factors
Metric subscores: Yes
Metric subscores detail: Detection method, LLM used
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: No, no link is provided
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: NLP / Detection
New taxonomy: General Application / LLM Detection