People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text
Authors: Jenna Russell, Marzena Karpinska, Mohit Iyyer
Accepted at: acl25

People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text

Jenna Russell1  Marzena Karpinska2  Mohit Iyyer1,3
1University of Maryland, College Park   2Microsoft   3UMass Amherst
{jennaruss,miyyer}@umd.edu, mkarpinska@microsoft.com

Abstract
In this paper, we study how well humans can detect text generated by commercial LLMs (GPT-4O, CLAude-3.5-SONNET, O1-PRO). We hire annotators to read 300 non-fiction English articles, label them as either human-written or AI-generated, and provide paragraph-length explanations for their decisions. Our experiments show that annotators who frequently use LLMs for writing tasks excel at detecting AI-generated text, even without any specialized training or feedback. In fact, the majority vote among five such “expert” annotators misclassifies only 1 of 300 articles, significantly outperforming most commercial and open-source detectors we evaluated even in the presence of evasion tactics like paraphrasing and humanization. Qualitative analysis of the experts’ free-form explanations shows that while they rely heavily on specific lexical clues, they also pick up on more complex phenomena within the text that are challenging to assess for automatic detectors. We release our annotated dataset and code to spur future research into both human and automated detection of AI-generated text.

https://github.com/jenna-russell/human_detectors

1 Introduction
AI-generated text is rampant in today’s world, spurring research into automatic detectors to help combat academic plagiarism (Zhu et al., 2024) and fake content creation (Gameiro et al., 2024). Unfortunately, automatic detectors suffer from low detection rates, poor robustness to evasion attempts, and limited explainability to end users (Sadasivan et al., 2024; Ji et al., 2024).1

In this paper, we instead study how well humans can detect AI-generated text. Unlike prior work on this topic, which was mainly conducted in the pre-ChatGPT era (Ippolito et al., 2020; Brown et al., 2020; Clark et al., 2021), we focus on text generated by modern LLMs (GPT-4O, CLAude-3.5-SONNET, O1-Pro) and in the presence of evasion attempts (paraphrasing, humanization). We hire human annotators to read non-fiction English articles, label them as written by either a human or by AI, and provide a paragraph-length explanation of their decision-making process (Figure 1). Overall, we collect 1790 annotations on 300 unique articles at a total cost of $4.9K USD, which allows us to compare humans to automatic detectors and analyze what kind of clues they rely on.

Experience matters: Unsurprisingly, annotators who rarely or never use LLMs are poor detectors of AI-generated text; in fact, they overestimate their own ability to perform the task by providing high confidence scores for their decisions. We identify a subset of five high-performing annotators who frequently use LLMs for writing tasks (e.g., editing, copywriting, creative writing). The majority vote of this subset of “expert” annotators fails to predict

![A human expert’s annotations of an article generated by OpenAI’s O1-Pro with humanization. The expert provides a judgment on whether the text is written by a human or AI, a confidence score, and an explanation (including both free-form text and highlighted spans) of their decision.](page_820_613_484_246.png)

Figure 1: A human expert’s annotations of an article generated by OpenAI’s O1-Pro with humanization. The expert provides a judgment on whether the text is written by a human or AI, a confidence score, and an explanation (including both free-form text and highlighted spans) of their decision.

1This paper focuses only on post-hoc detection of AI-generated text, which, unlike watermarking (Kirchenbauer et al., 2023) requires no cooperation from LLM providers.
the correct label on only one out of 300 articles. We emphasize that we do not provide any training to our annotators: they are given no feedback on which articles they labeled incorrectly.

Human experts outperform automatic detectors: The majority vote of our five expert annotators outperforms almost every commercial and open-source detector we tested on these 300 articles, with only the commercial Pangram model (Emi and Spero, 2024) matching their near-perfect detection accuracy. In our most complex configuration, which requires detecting articles generated and humanized by o1-Pro, the expert majority vote is perfect (true positive rate of 100%), while open-source methods such as Binoculars (6.7%) and Fast-DetectGPT (23.3%) struggle. We conclude that hiring expert human annotators to perform detection is a viable strategy, particularly in high-stakes settings where explainability is critical.

What do expert annotators focus on? An analysis of our expert’s explanations reveals that usage of “AI vocabulary” (e.g., vibrant, crucial, significantly) form the most common giveaways. Close behind are formulaic sentence and document structures (e.g., optimistically vague conclusions) and originality (how creative or engaging an article is). We observe that neither paraphrasing nor humanization effectively removes all of these signatures; that said, these evasion tactics and defenses for them are still underexplored in the research community.

Can LLMs mimic human detectors? We attempt to automatically replicate the decision-making process of our human experts by providing a candidate text to an LLM, along with a guidebook compiled from human expert explanations, and instructing the model to use the guidebook to judge whether the text is written by an AI or by a human. This simple prompting strategy is competitive with existing detectors on easier configurations but struggles to detect humanized articles. Moving forward, we believe LLMs can be used in conjunction with high-performing detectors such as Pangram to offer an explanation of the judgment to end users.

2 How good are humans at detecting AI-generated text?

How well do human detectors stack up against their automatic counterparts? What features do humans find most predictive of AI writing? To answer these questions, we conducted five experiments that ask humans to judge increasingly more deceptive AI-generated texts.

Task setup: Each experiment consists of 60 articles, 30 human-written and 30 LLM-generated. Each human-written article has a corresponding AI-generated counterpart with the same title and subtitle to control for high-level content and topics. Articles are shown to annotators in a randomized order. For each, they provide (1) a binary label (human-written or AI-generated), (2) a confidence rating on a scale from 1 (least confident) to 5 (most confident), (3) highlighted spans used as clues, and (4) a paragraph-length explanation. We evaluate with True Positive Rate (TPR), which measures the percentage of AI-generated articles that are successfully detected, as well as False Positive Rate (FPR), which measures the percentage of human-written articles marked as AI-generated.2

Article selection: We restrict our study to English nonfiction articles of fewer than 1K words.3 While we cannot make claims about human detection performance in other domains (e.g., scientific papers or social media posts),4 we focus on such articles because (1) malicious LLM users generate large volumes of fake articles, making it practically impactful; (2) such articles do not require specialized knowledge to understand or judge; and (3) many previous studies also ask humans to judge AI-generated text in this domain (Ippolito et al., 2020; Clark et al., 2021; Puccetti et al., 2024).

Generating paired AI articles: For each human-written article, we generate a corresponding AI article by prompting an LLM with its title, subtitle, desired length, and publication source. This ensures that AI-generated articles are directly comparable in topic and length.5 By constructing these human-AI article pairs with only authorship as the varying factor, we create minimal pairs that allow direct comparisons (Gardner et al., 2020; Warstadt et al., 2020; Karpinska et al., 2022, 2024). For

2 Prior work in automatic detection reports either TPR at a fixed low FPR such as 1% (Krishna et al., 2023; Hans et al., 2024; Dugan et al., 2024) or AUROC (Mitchell et al., 2023; Hu et al., 2023; Bao et al., 2023), which we cannot easily do with human annotators.

3 Human-written articles in our experiments come from eight different American publications: Associated Press, Discover Magazine, National Geographic, New York Times, Reader’s Digest, Scientific American, Smithsonian Magazine, and Wall Street Journal.

4 We report results on §2.1 for fictional stories in §C.

5 We include a word count target to ensure comparable lengths between article pairs, reported in Table 6.
experiments 3 and 5, we modify the prompt to include perturbations designed to evade detectors (paraphrasing and humanization). We employ a within-subjects design for our experiments, where each annotator judges both the human-written and AI-generated articles. This reduces variability from individual differences and requires fewer annotators (Allen, 2017). We further minimize bias by randomizing article order and ensuring annotators are unaware of the pairing.6

Annotator details: We recruited annotators via the Upwork freelancing platform, all of whom self-identified as native English speakers.7 All annotators were required to read the project guidelines and sign a consent form prior to the task, provided in §A. Additionally, we surveyed each annotator to determine their level of education, profession, English dialect, and familiarity with LLMs, including how they use these models (see Table 4 for details). They were compensated $2 per candidate text, resulting in an hourly rate of about $15 - $30 USD. Ultimately, we collect 1740 annotations from 9 annotators at a total cost of $4.9K USD.

Justification of sample size: Most experiments in this paper are conducted by five expert annotators (Experiment 1 includes four nonexpert annotators).8 While a larger sample size would strengthen population-level claims; prohibitive costs9 limit us to the current sample size. We employ a within-subjects design for our experiments, where each annotator judges both the human-written and AI-generated articles, reducing individual variability and requiring fewer annotators (Allen, 2017). Our set of five expert annotators provide an insightful view of the human upper bound at AI-generated text detection without additional training. Furthermore, collecting detailed explanations of these annotators’ decision-making process, while expensive, enables fine-grained qualitative analysis and ensures that annotators actually read the texts.10

<table>
  <tr>
    <th>METRIC</th>
    <th>NONEXPERTS</th>
    <th>EXPERTS</th>
  </tr>
  <tr>
    <td>Avg. TPR</td>
    <td>56.7</td>
    <td>92.7</td>
  </tr>
  <tr>
    <td>Avg. FPR</td>
    <td>51.7</td>
    <td>4.0</td>
  </tr>
  <tr>
    <td>Avg. Confidence</td>
    <td>4.03</td>
    <td>4.39</td>
  </tr>
</table>

Table 1: On average, nonexperts perform similar to random chance at detecting AI-generated text, while experts are highly accurate.

2.1 Experiment 1: What kinds of annotators can reliably detect AI-generated articles?

LLM: GPT-4O (2024-08-06)
Evasion tactic: none

Is there a population of humans that is highly accurate at our detection task, and if so, are there any commonalities between them?

We first ask five annotators with varying backgrounds and levels of expertise with LLMs to attempt a batch of 60 articles. We observe both ends of the spectrum in terms of performance: annotators who rarely or never use LLMs for writing tasks performed near random, while one annotator who uses LLMs daily to edit LLM-generated text performed almost perfectly. More details on our initial study can be found in §C. Inspired by this result, we recruit four more annotators with similar backgrounds as our top performer, who all of whom perform similarly well on the same batch of articles.

Annotators with limited LLM experience are unreliable detectors: The four annotators who self-report either not using LLMs at all, or using LLMs only for non-writing tasks, detect AI-generated text at a similar rate to random chance, achieving an average TPR of 56.7% and FPR of 52.5% (Table 1).11 Despite their poor performance, these annotators reported high confidence (avg. 4.03) in their decisions, suggesting they may be overestimating their detection abilities. We term this population nonexperts for the rest of this paper.

Annotators who frequently use LLMs for writing tasks are highly accurate: In contrast, the five annotators who have significant experience with using LLMs for writing-related tasks are able to detect AI-generated text very reliably, achieving

11We report average TPR and FPR instead of majority vote since there were only four nonexpert annotators. For the remainder of the paper, we report the majority vote of the five expert annotators (i.e., at least three out of five agreed).

6We set temperature=0 for Experiments 1-3 and use ChatGPT (Pro) interface for Experiments 4-5 as the API for o1 was not yet available.
7See Table 4 for survey details.
8See §A.1 for details on expert recruitment.
9It cost us $865 USD per annotator for five experiments.
10Collecting explanations mitigates issues raised by Karpinska et al. (2021), whose Mechanical Turk annotators “read” long-form texts in a matter of mere seconds before evaluation.
a TPR of 92.7%.12 The average FPR for this population of annotators was 3.3%, meaning that they rarely mistake human-written text as AI-generated. The majority vote out of these five annotators correctly determined authorship of all 60 articles; the “GPT-4o” column of Table 2 contains more details. We term this population experts for the rest of this paper; all subsequent experiments rely on only these five expert annotators.

What do expert annotators see that nonexperts don’t? To understand why experts far outperformed nonexperts at detecting AI-generated text, we analyze the comments each annotator provided in their explanations. Overall, nonexperts often mistakenly fixate on certain linguistic properties compared to experts. One example is vocabulary choice, where nonexperts take the inclusion of any “fancy” or otherwise low-frequency word types as signs of AI-generated text; in contrast, experts are much more familiar with exact words and phrases overused by AI (e.g., testament, crucial).13 Nonexperts also believe that human authors are more likely than AI to form grammatically-correct sentences, while experts realize the opposite is true: humans actually make more grammatical errors. Finally, nonexperts attribute any text written in a neutral tone to AI, resulting in many false positives.

2.2 Experiment 2: Can experts detect articles generated by a different LLM?

LLM: CLAUDE-3.5-SONNET
Evasion tactic: none

While ChatGPT is the most widely-used LLM (both in general and by our expert annotators), competitors such as Anthropic’s Claude (Anthropic, 2023) also have a wide user base. Are our experts overfitting to artifacts specific to GPT-4o, or do they detect patterns that generalize to other LLMs? We address this question in Experiment 2 by evaluating our experts on a second batch of 60 articles, this time with 30 new human-written articles and 30 corresponding articles generated via CLAUDE-3.5-SONNET (Anthropic, 2024) using the same instructions as in Experiment 1.

Experts reliably detect articles generated by Claude: Despite the change in model, experts are reliable at detecting AI-generated content. TPR is almost unchanged from Experiment 1 and the expert majority vote is again 100% accurate (“CLAUDE-3.5-SONNET” column of Table 2), with two annotators achieving a perfect score individually; however, we find one annotator performs worse on Claude-generated texts than those of GPT-4o.14 We note that one expert (Annotator 2) achieved higher performance in the experiments where an OpenAI model was used, with their TPR falling to 80% in this experiment. Annotator 2 did not have any prior experience using Claude, although three other annotators who were also unfamiliar with Claude achieved TPRs of at least 96.7%. However, Annotator 2 focuses more heavily on the existence of “AI vocab” in the article than the other annotators, and they were not familiar with the distribution of words that are overused by Claude.

2.3 Experiment 3: How robust are experts to paraphrasing attacks?

LLM: GPT-4O (2024-08-06)
Evasion tactic: paraphrasing

Users of LLMs who hope to evade detection often resort to simple post-hoc modifications of LLM outputs such as paraphrasing. These tactics significantly drop detection rates of automatic methods (Krishna et al., 2023; Sadasivan et al., 2024), but how do they affect our human experts? We evaluate the five expert annotators on a third set of 60 articles, where the 30 AI-generated articles go through an additional round of paraphrasing,15 showing they maintain high TPR and low FPR.

Experts are robust to paraphrased AI-generated articles: Overall, TPR and FPR remain almost unchanged in this experiment compared to the first two, suggesting that paraphrasing is not an effective attack on expert human detectors (see “GPT-4O PARAPHRASED” column in Table 2). The majority vote among the five experts again correctly detects the labels of all 60 articles.

2.4 Experiment 4: Can experts keep up with advances in LLM reasoning capabilities?

LLM: o1-PRO
Evasion tactic: none

While we were in the middle of conducting experiments, OpenAI released their o1 model (Ope-

12We compare expert vs. nonexpert clues in §D.2.
13A complete list of “AI vocab” found in the detection guide (Table 11) is detailed in Table 12.
14This annotator’s performance is detailed in §D.3.
15Details on paraphrasing setup are in §B.2.
<table>
  <tr>
    <th rowspan="2">DETECTION METHOD</th>
    <th colspan="3">GENERATION METHOD</th>
    <th colspan="4">TPR% (FPR%)</th>
  </tr>
  <tr>
    <th>GPT-4O<br>TPR% (FPR%)</th>
    <th>CLAUDE<br>TPR% (FPR%)</th>
    <th>GPT-4O PARAPHRASED<br>TPR% (FPR%)</th>
    <th>O1-PRO<br>TPR% (FPR%)</th>
    <th>O1-PRO HUMANIZED<br>TPR% (FPR%)</th>
    <th>OVERALL<br>TPR% (FPR%)</th>
  </tr>
  <tr>
    <th colspan="7">(A) Expert human detectors</th>
  </tr>
  <tr>
    <td>EXPERT MAJORITY VOTE</td>
    <td>100 (0)</td>
    <td>100 (0)</td>
    <td>100 (0)</td>
    <td>96.7 (0)</td>
    <td>100 (0)</td>
    <td>99.3 (0)</td>
  </tr>
  <tr>
    <td>ANNOTATOR 1</td>
    <td>96.7 (3.3)</td>
    <td>100 (0)</td>
    <td>100 (0)</td>
    <td>96.7 (6.7)</td>
    <td>90.0 (23.3)</td>
    <td>96.7 (6.7)</td>
  </tr>
  <tr>
    <td>ANNOTATOR 2</td>
    <td>96.7 (0)</td>
    <td>80.0 (30)</td>
    <td>86.7 (10)</td>
    <td>90.0 (10)</td>
    <td>86.7 (10)</td>
    <td>88.0 (12)</td>
  </tr>
  <tr>
    <td>ANNOTATOR 3</td>
    <td>86.7 (6.7)</td>
    <td>100 (0)</td>
    <td>93.3 (0)</td>
    <td>16.7 (0)</td>
    <td>0 (3.3)</td>
    <td>59.3 (2)</td>
  </tr>
  <tr>
    <td>ANNOTATOR 4</td>
    <td>90.0 (6.7)</td>
    <td>96.7 (13.3)</td>
    <td>100 (10)</td>
    <td>100 (0)</td>
    <td>100 (0)</td>
    <td>97.3 (0)</td>
  </tr>
  <tr>
    <td>ANNOTATOR 5</td>
    <td>93.3 (0)</td>
    <td>93.3 (6.7)</td>
    <td>93.3 (0)</td>
    <td>93.3 (0)</td>
    <td>93.3 (0)</td>
    <td>93.3 (1.3)</td>
  </tr>
  <tr>
    <th colspan="7">(B) Automatic detectors</th>
  </tr>
  <tr>
    <td>PANGRAM HUMANIZERS</td>
    <td>100 (0)</td>
    <td>100 (3.3)</td>
    <td>100 (0)</td>
    <td>100 (0)</td>
    <td>96.7 (10)</td>
    <td>99.3 (2.7)</td>
  </tr>
  <tr>
    <td>PANGRAM</td>
    <td>100 (0)</td>
    <td>100 (3.3)</td>
    <td>100 (0)</td>
    <td>100 (0)</td>
    <td>90.0 (6.7)</td>
    <td>98.0 (2)</td>
  </tr>
  <tr>
    <td>CLAUDE</td>
    <td>100 (0)</td>
    <td>96.7 (0)</td>
    <td>100 (0)</td>
    <td>76.7 (0)</td>
    <td>46.7 (3.3)</td>
    <td>85.3 (0.7)</td>
  </tr>
  <tr>
    <td>FACT-DetectGPT (FPR=0.05)</td>
    <td>100 (0)</td>
    <td>96.7 (3.3)</td>
    <td>56.7 (3.3)</td>
    <td>86.7 (0)</td>
    <td>23.3 (3.3)</td>
    <td>80.0 (7.2)</td>
  </tr>
  <tr>
    <td>BINOCULARS (ACCURACY)</td>
    <td>100 (0)</td>
    <td>93.3 (0)</td>
    <td>60.0 (6.7)</td>
    <td>73.3 (0)</td>
    <td>66.7 (0)</td>
    <td>66.7 (1.3)</td>
  </tr>
  <tr>
    <td>BINOCULARS (LOW FPR)</td>
    <td>96.7 (0)</td>
    <td>80.0 (0)</td>
    <td>13.3 (0)</td>
    <td>10.0 (0)</td>
    <td>0 (0)</td>
    <td>40.0 (0)</td>
  </tr>
  <tr>
    <td>RADAR (FPR=0.05)</td>
    <td>66.7 (0)</td>
    <td>0 (0)</td>
    <td>10 (3.3)</td>
    <td>0 (3.3)</td>
    <td>0 (3.3)</td>
    <td>15.3 (2)</td>
  </tr>
  <tr>
    <th colspan="7">(C) Prompt-based detectors</th>
  </tr>
  <tr>
    <td colspan="7">Detector LLM: GPT-4O-2024-11-20</td>
  </tr>
  <tr>
    <td>ZERO-SHOT</td>
    <td>100 (10)</td>
    <td>93.3 (10)</td>
    <td>100 (6.7)</td>
    <td>56.7 (3.3)</td>
    <td>6.7 (3.3)</td>
    <td>71.3 (6.7)</td>
  </tr>
  <tr>
    <td>ZERO-SHOT + COT</td>
    <td>63.3 (3.3)</td>
    <td>33.3 (0)</td>
    <td>96.7 (6.7)</td>
    <td>16.7 (0)</td>
    <td>0 (0)</td>
    <td>42.0 (2.0)</td>
  </tr>
  <tr>
    <td>ZERO-SHOT + GUIDE</td>
    <td>100 (10)</td>
    <td>96.7 (10)</td>
    <td>100 (13.3)</td>
    <td>80 (6.7)</td>
    <td>3.3 (3.3)</td>
    <td>76.0 (8.7)</td>
  </tr>
  <tr>
    <td>ZERO-SHOT + COT + GUIDE</td>
    <td>100 (10)</td>
    <td>100 (13.3)</td>
    <td>100 (16.7)</td>
    <td>86.7 (6.7)</td>
    <td>3.3 (3.3)</td>
    <td>78.0 (10.7)</td>
  </tr>
  <tr>
    <td colspan="7">Detector LLM: O1-2024-12-17</td>
  </tr>
  <tr>
    <td>ZERO-SHOT</td>
    <td>93.3 (3.3)</td>
    <td>66.6 (6.7)</td>
    <td>96.7 (6.7)</td>
    <td>40.0 (3.3)</td>
    <td>20.0 (6.7)</td>
    <td>42.2 (5.6)</td>
  </tr>
  <tr>
    <td>ZERO-SHOT + Cot</td>
    <td>83.3 (6.7)</td>
    <td>53.3 (3.3)</td>
    <td>96.7 (3.3)</td>
    <td>20 (3.3)</td>
    <td>16.7 (3.3)</td>
    <td>54 (4)</td>
  </tr>
  <tr>
    <td>ZERO-SHOT + GUIDE</td>
    <td>93.3 (0)</td>
    <td>30.0 (0)</td>
    <td>96.7 (0)</td>
    <td>13.3 (0)</td>
    <td>0 (0)</td>
    <td>36.7 (0)</td>
  </tr>
  <tr>
    <td>ZERO-SHOT + COT + GUIDE</td>
    <td>86.7 (0)</td>
    <td>43.3 (0)</td>
    <td>90.0 (0)</td>
    <td>6.7 (0)</td>
    <td>0 (0)</td>
    <td>53.3 (0.6)</td>
  </tr>
</table>

Table 2: Performance of expert humans (top), existing automatic detectors (middle), and our prompt-based detectors (bottom), where each cell displays TPR (FPR). Colors indicate performance bins where darkest teal is best, orange is middling, and purple is worst. We further mark closed-source (■) and open-weighted (●) detectors. The majority vote of expert humans ties Pangram Humanizers for highest overall TPR (99.3) without any false positives, while substantially outperforming all other detectors. While the majority vote is extremely reliable, individual annotator performance varies, especially on o1-Pro articles with and without humanization. Prompt-based detectors are unable to match the performance of either expert humans or closed-source detectors.

nAI, 2024), ushering in a new paradigm of test-time scaling. This release offered us a unique opportunity: since none of our experts had been previously exposed to a model with such advanced reasoning capabilities (Zhong et al., 2024), how well would they do at detecting its output? Interestingly, our experts remain reliable detectors of articles generated by o1-Pro (using the same prompt as in Experiments 1 and 2),16 although their comments show that they often perform a more detailed analysis of articles to make their judgments.

Four out of five experts are robust to o1-Pro: For four of five experts, detection ability of o1-generated content remains largely consistent with prior experiments, with a majority vote TPR of 96.7% and FPR of 0% (see “o1-Pro” column of Table 2).17 Average confidence dropped to 4.21 out of 5, compared to average confidence of 4.39, 4.38, and 4.48 from Experiments 1,2, & 3 respectively (see Figure 2 for details). Verbal feedback from experts, along with the drop in confidence, demonstrates the increased challenge posed by o1-generated articles. However, their steady aggregate performance highlights that even new model paradigms are still detectable by human experts.

Experts provide more nuanced explanations when detecting o1-Pro: In prior experiments, expert explanations focused primarily on whether or not a candidate text possesses characteristics of AI. However, experts shift focus with o1-Pro by more frequently commenting on identifying characteristics that make text sound “human”. For instance, experts frequently point to how humans repeatedly write the word says, while AI tries to use more descriptive synonyms like notes and explains. Annotator 3, the only one consistently fooled by o1-Pro outputs, relied too much on signs of informality not find enough clues in those categories to judge the article as AI-generated. The full text of the article is in Table 14.

16 o1-Pro is available through the ChatGPT Pro subscription. OpenAI states o1-Pro produces more reliably accurate and comprehensive responses than o1.
17 One AI-generated article was misclassified by the expert majority vote. Analyzing their explanations, the three annotators most focused on vocabulary and sentence structure did
(e.g., contractions, slang usage, usage of just and actually) as a sign of human writing, with 66.7% of their explanations relating to formality (Figure 12).

2.5 Experiment 5: Are experts robust to humanization?

LLM: o1-PRO
Evasion tactic: humanization

Many users attempt to evade detectors via humanization methods, which explicitly attempt to make AI-generated text more human-like (Wang et al., 2024a,b). Are such methods effective at evading our human experts? Since no well-established humanization methods exist, we first create our own humanizer by prompting o1-Pro with a detailed set of instructions derived from expert explanations from Experiments 1-4. Despite considerably degrading performance of many automatic detectors, humanization does not fool our experts in aggregate (“o1-Pro HUMANIZED” column of Table 2).

Implementing a prompt-based humanizer: We modify the article generation prompt used in prior experiments to include instructions on specific AI signatures to avoid. To obtain these instructions, we pay our experts $45 each to provide us with a list of clues that they look for during detection. We then then manually organize these disparate clues into a “guidebook” with different sections (e.g., vocabulary, tone) that each contain explanations and examples of how AI writing differs from human writing (see Table 11 for a truncated version of the guidebook). Then, we prompt o1-Pro with the guidebook and an instruction to generate an article that would evade a detector that was following the guidebook (see Table 13 for prompt).18 More details on the development of our humanization method can be found in §B.2.

Experts remain robust to humanized articles: Despite our best efforts to generate articles that our experts would find undetectable, most of their detection rates remain largely unchanged from prior experiments, and the expert majority vote is again perfect on all 60 articles. Annotator 3, who struggled to detect non-humanized o1-Pro articles in Experiment 4, performed remarkably poorly on this batch, achieving a TPR of zero by marking almost every article as human-written. While detection rates remained steady, overall confidence dropped, with 15.1% of annotations having a confidence of 1, showing the increased difficulty presented by the humanized articles (Figure 2).

![Expert confidence in their decisions drops when judging humanized articles generated by o1-Pro.](page_1012_1092_388_246.png)

Figure 2: Expert confidence in their decisions drops when judging humanized articles generated by o1-Pro.

AI clues still remain after humanization: The clues used by experts remained consistent from prior experiments, showing that humanization does not completely remove the “AI signature” from texts. For example, while humanization increases variety in AI-generated names, often even including people relevant to the article who exist in real life, our experts found that humanized articles use titles for these people (e.g., Dr., Prof.) much more frequently than in human-written articles. Our results highlight the need for increased public research on humanization, as more advanced methods can challenge both human and automated detectors while offering better training data for more robust detection systems.

3 Analysis of expert performance

We first provide in-depth comparisons between experts and automatic detectors.19 Next, we perform a fine-grained coding of expert explanations into different categories (e.g., vocabulary, originality, formality), which allows us to examine what they focus on during the different experiments. Finally, we analyze differences between annotators, highlighting implications for the future training of human annotators for AI-generated text detection.

Human experts vs. automatic detectors: We compare human experts with five AI detectors, including two closed-source classifiers, Pangram (Emi and Spero, 2024; Masrour et al., 2025) and

18We only use o1-Pro for this experiment as the previous experiments show it was more difficult to detect than GPT-4o and CLAUS-3.5-SONNET.
19We release our dataset of annotations under under the MIT License.
<table>
  <tr>
    <th>CATEGORY</th>
    <th>FREQ</th>
    <th>DEFINITION</th>
    <th>EXAMPLE EXPLANATIONS</th>
  </tr>
  <tr>
    <td>VOCABULARY</td>
    <td>53.1%</td>
    <td>LLMs use specific words and phrases more often than human writers, which often results in repetitive, unnatural, or overly complex wording.</td>
    <td>
      <b>Human:</b> “Furthermore, I very much doubt AI would have used adventurous adjectives like ‘chunky’, ‘mucky’ or ‘thin’ to describe food. Nor would it have used verbs like ‘blitzing’ or ‘bolstering’.”
      <br>
      <b>AI (01-HUMANIZED):</b> “Odd word choices: wheat that ‘stores’ a lineage; genes that ‘encode’.”
    </td>
  </tr>
  <tr>
    <td>Sentence Structure</td>
    <td>35.9%</td>
    <td>AI-generated sentences follow predictable patterns (e.g., high frequency of “not only ... but also ...”, or consistently listing three items), while human-written sentences vary more in terms of length.</td>
    <td>
      <b>Human:</b> “Short choppy sentences and paragraphs.”
      <br>
      <b>AI (01-PRO):</b> “One pattern I’ve been noticing with AI, and I think I’ve stated this before, is the comparison of ‘it’s not just this, it’s this’ and I’m seeing it here, along with listings of specifically three ideas.”
    </td>
  </tr>
  <tr>
    <td>Grammar & Punctuation</td>
    <td>24.8%</td>
    <td>AI-generated text is usually grammatically perfect (also avoiding dashes and ellipses), while human-written text often contains minor errors.</td>
    <td>
      <b>Human:</b> “There’s a lot of variety in the article’s grammar use, with dashes, brackets, quotes intermixed with sentences, and short spurts of comma sections throughout.”
      <br>
      <b>AI (GPT-40-PARA):</b> “there’s nothing off about the grammar or syntax in this piece…”
    </td>
  </tr>
  <tr>
    <td>Originality</td>
    <td>23.7%</td>
    <td>AI-generated writing is generally straightforward, “safe,” and lacking in surprises or humor, leaving annotators bored or disengaged.</td>
    <td>
      <b>Human:</b> “It’s offset by some great analogies and creative phrasing that works well to convey the topic, such as with “amateur sleuths”, “catnip for a certain type of Reddit user”.”
      <br>
      <b>AI (01-PRO):</b> “What happens when AI tries to be creative? Penguins “stand on their own flippers.””
    </td>
  </tr>
  <tr>
    <td>Quotes</td>
    <td>22.3%</td>
    <td>AI-generated quotes sound overly formal, lack the varied nuances of real conversation, and often mirror the article’s main text too closely in style.</td>
    <td>
      <b>Human:</b> “The quotes being short snippets also makes me think they’re real, as the writer had to find a way to fit them into the text, rather than them just perfectly stating either side’s views.”
      <br>
      <b>AI (GPT-40):</b> “The quotes also feel fake, every expert speaks the same way and it’s too homogenous with the text.”
    </td>
  </tr>
  <tr>
    <td>Clarity</td>
    <td>19.5%</td>
    <td>AI-generated text often lacks concise flow by over-explaining or including irrelevant details, effectively “telling” rather than “showing”.</td>
    <td>
      <b>Human:</b> “Words like “meander” are used, but are used sparingly to create better flow of ideas, and the writing style is simplified in the best way possible.”
      <br>
      <b>AI (GPT-40-3.5):</b> “The sentences are constructed to provide the best possible precision with its word choice, but the article lacks flow and clarity.”
    </td>
  </tr>
</table>

Table 3: Truncated taxonomy of clues used by experts to explain their detection decisions (see Table 17 for full version). For each category, we report the frequency of explanations that mention that category (regardless of correctness) and provide sample explanations for human-written and AI-generated articles. While vocabulary is the most common clue, complex phenomena like originality, quotes and clarity are also distinguishing features.

GPTZero (Tian and Cui, 2023), and three open-source methods: Binoculars (Hans et al., 2024), Fast-DetectGPT (Bao et al., 2023), and RADAR (Hu et al., 2023).20 Table 2 shows that only Pangram Humanizers (average TPR of 99.3% with FPR of 2.7% for base model) matches the human expert majority vote, and it also outperforms each expert individually, faltering just slightly on humanized 01-Pro articles. GPTZero struggles significantly on 01-Pro with and without humanization. Open-source detectors degrade in the presence of paraphrasing and underperform both closed detectors. We again note that our experts are untrained at this detection task, and they could likely improve their individual performance if provided with feedback.

Coding expert explanations: In contrast to opaque automatic detectors, we can easily solicit explanations from human annotators. We use GPT-40 to code these explanations into a schema (Table 3) developed by the authors after careful manual analysis. Details of the coding process used to label explanations can be found in §D. Vocabulary-related clues are mentioned in the majority (53.1%) of all explanations, while sentence structure (35.9%), grammar (24.8%), and originality (23.7%) are also common. We note that many of these categories (e.g., originality, factuality, tone) are much more difficult to assess automatically than others (e.g., vocabulary), and these may currently be areas where humans have an advantage over automatic detectors.

When experts correctly detect AI-generated articles, what clues do they use? Figure 3 shows the frequency that clue categories are mentioned in explanations for which the expert makes the correct decision. We observe several shifts as the complexity of the article generation setup increases: for example, 57.1% of correct explanations about 01-Pro articles mention vocabulary, compared to only 42.3% for the humanized 01-Pro articles prompted to avoid “AI vocab”. Somewhat counterintuitively, explanations about paraphrased articles note AI vocab in 88% of explanations, compared to only 69.8% of non-paraphrased GPT-40 articles. Quotations are mentioned in a surprisingly high 33.8% of explanations about paraphrased articles: a close reading reveals that experts flagged quotes that were always in the same format and style (e.g., only placed at the end of each paragraph). Finally, the frequency of originality clues highlights the gulf in creativity between humans and LLMs (Chakrabarty et al., 2024).

20We report TPR and FPR of automatic detectors using recommended thresholds whenever possible. For Fast-DetectGPT, we use a threshold calibrated to an FPR of 5% on a held-out set of 40 human-written articles. Thresholds used are found in §E.1. We evaluate Pangram in both its base and humanized modes, ignoring the “Possibly AI” label.
Figure 3: A heatmap displaying the frequency with which annotators mentioned specific categories in their explanations when they were correct. Interestingly, vocabulary becomes a less frequent clue for o1-Pro-generated articles, especially with humanization. A heat map of corresponding incorrect explanation is displayed in Figure 9. Details of each category can be found in Table 3.

Annotators don’t always focus on the same clues:
Our experts focus on different properties of the text to arrive at their decisions, as described further in §D.3. Annotator 1 is the only one to pick up on “AI names”, while in fact, 63.3% of GPT-4o and 70% of CLAUDE-3.5-SONNET articles include either the name Emily or Sarah.21 Annotators 2 and 3 emphasize grammaticality or the tendency of LLMs to always list examples in groupings of three. Annotator 4 focuses on article flow, analyzing the specificity of detail and motivation behind an article, and Annotator 5 examines how natural any quotations sound. This diversity explains why an ensemble of expert annotators performs so well and it also suggests room for training so that all experts are at least aware of most distinguishing features. implies additional training can improve individual experts.

When experts are incorrect, what clues lead them astray? Figure 9 shows clue frequencies in incorrect expert explanations (see §D.3 for more). For example, o1-Pro’s frequent use of contractions and colloquialisms fooled Annotator 3: they mention formality in none of their explanations for Experiments 2 and 3, compared to 66.7% of o1-Pro and 83.3% of humanized o1-Pro explanations (Figure 12). Studying expert false positives is also insightful (Figure 3): 31% of explanations here mention vocabulary, typically when human-written content contains “AI vocab” like delve and crucial.

21O1-PRO favors names of real people instead of fictional names when generating articles.

When is it worth it to use human detection?
Human experts deliver detailed, freeform explanations and confidence ratings - features that automatic detectors generally lack, as they usually provide only a scalar score or binary label, limiting their trustworthiness in critical settings (Lipton, 2018). For example, before accusing a student of plagiarism or flagging a high-stakes legal document, someone may want to point to specific properties of the text that raised suspicion, or ensure that a false positive won’t have severe consequences. In our study, the majority vote of five expert annotators—without any specialized training—misclassified only 1 out of 300 articles, on par with the most accurate commercial detector (Pangram), even under adversarial paraphrasing and humanization. However, manual review cannot scale to high-volume, low-stakes tasks such as online forum moderation, where automated systems remain the only practical option.

4 Can LLMs be prompted to mimic human expert detectors?
We have established that human experts are more accurate, robust, and interpretable than automatic detectors. Can we prompt LLMs to mimic the decision-making process of human experts? In this section, we develop a prompt-based detector that also generates explanations, using the guidebook from §2.5. While the approach promisingly outperforms Binoculars and RADAR, it lags human experts and closed-source detectors like Pangram.
Implementing a prompt-based detector: We prompt both GPT-4o and o1 to decide whether a candidate text is human-written or AI-generated and explain its decision based on the criteria in the guidebook (see Table 22 for prompt template).22 While we can control this detector’s behavior via its prompt (e.g., with and without chain-of-thought), it produces the label text instead of a scalar score, making it impossible to set an FPR threshold.

Prompt-based detection shows promise but struggles with humanization and high FPRs: For GPT-4o, the best detector configuration is achieved with both CoT and the guidebook (average TPR 78%), while o1 is best with CoT but without the guidebook (54%). We observe that o1 is more conservative than GPT-4o, demonstrating lower FPRs on average at the cost of lower TPR. Our best configuration with GPT-4o-2024-11-20 performs comparably to Binoculars and Fast-DetectGPT, although at a much higher cost. We hypothesize that the gap between prompt-based detection and humans can be reduced by fine-tuning.

5 Related work

Human detection of AI-generated text: Research conducted prior to ChatGPT’s release concludes that while naïve annotators do not reliably detect AI-generated texts (Ippolito et al., 2020; Brown et al., 2020; Clark et al., 2021; Karpinska et al., 2021), some annotators perform very well (Ippolito et al., 2020; Dugan et al., 2022). Other research explores detection in (Porter and Machery, 2024) and other languages (Puccetti et al., 2024; Wang et al., 2025). Dugan et al. (2020) and Dugan et al. (2022) address a more complex task of identifying the boundary between human and AI-generated text.

Automatic detection: Successful automatic detection methods rely on either perplexity (Mitchell et al., 2023; Bao et al., 2023; Hans et al., 2024) or trained classifiers (Solaiman et al., 2019; Emi and Spero, 2024; Verma et al., 2023). Prior work compares detectors’ performance in different domains and regimes (Dugan et al., 2024; Li et al., 2024; Zhang et al., 2024). Detectors can be vulnerable to paraphrasing attacks (Krishna et al., 2023; Sadasivan et al., 2024) and style changes (Doughman et al., 2025). Studies have also examined humanization attacks to bypass detection (Wang et al., 2024b; Lu et al., 2023; Shi et al., 2023; Wang et al., 2024a), while Zhou et al. (2024) and Masrour et al. (2025) boost detector robustness.

Analyzing differences between human-written and AI-generated text: Many frameworks group errors in AI-generated text similarly to our paper (Gehrmann et al., 2019; Dou et al., 2021). Ma et al. (2023) discover gaps in depth and content quality between AI-generated and human-written scientific text, while Shaib et al. (2024) notes the repetitiveness of AI-generated text by introducing syntactic templates. Ji et al. (2024) categorize human nonexpert detection explanations.

6 Conclusion

Our paper demonstrates that a population of “expert” annotators—those who frequently use LLMs for writing-related tasks—are highly accurate and robust detectors of AI-generated text without any additional training. The majority vote of five such experts performs near perfectly on a dataset of 300 articles, outperforming all automatic detectors except the commercial Pangram model (which the experts match). Analysis of explanations provided by our expert annotators reveals that they pick up on not just vocabulary and sentence structure-related clues but also more complex properties like originality and tone. We observe that experts focus on different aspects of the text, and we conjecture that with explicit training, human annotators can be made even more robust to advances in LLMs as well as evasion tactics (e.g., humanization). Future work can also explore human annotators working alongside automatic detectors like Pangram to improve detection accuracy and explainability. We find it apt to end with a comment from one of our experts about a particularly formulaic conclusion generated by GPT-4o:

This time I went to the end of the piece and said:
“Hello, AI.” There it was in all its glory: the “testament” serving “as a beacon of hope and inspiration” and “demonstrating” to us humans “that anything is possible.” Sometimes I feel sorry for AI—it must have a dreary time trying to satisfy its human interlocutor’s desire to “showcase” advocacy, social change, inclusivity, gender equality, equity, and representation in an essay of fewer than a thousand words.

— ANNOTATOR 5

22We set temperature=0 for GPT-4o and reasoning efforts to ‘medium’ for o1 (note that OpenAI has fixed temperature=1 for o1). Further experiments can be found in Table 23.
Limitations

Our study is limited to articles in American English, chosen for their consistent formatting and high quality (i.e., professionally written and proofread). We also did not investigate factual accuracy, as it did not appear to be a significant cue for our annotators, who covered a broad range of topics. While we selected articles from reputable sources, there remains a possibility that some included AI-generated edits beyond our scope of detection.

Ethical Considerations

This study was reviewed by the UMass Institutional Review Board (IRB #5927) and deemed exempt. All annotators were briefed on the purpose of the project and provided informed consent prior to participating. Those who wished to be acknowledged by name explicitly agreed to do so in their consent forms. We ensured fair compensation for annotators in recognition of their time and expertise. We acknowledge the potential risks of misinformation and hallucinated content, especially when AI outputs are presented as human-written. Our goal is to examine these issues and inform best practices, rather than endorse or facilitate deceptive uses of AI.

Acknowledgments

We would like to extend our gratitude to our Upwork expert annotators for their dedicated efforts, and link their Upwork accounts for future work requiring human expertise on AI text detection: Benjamin Stewart,23 Elizabeth Bergman,24 Lynette Stewart,25 Roberta King,26 and Shannon West.27 We thank Pangram and GPTZero for providing credits to access their APIs. We are also grateful to the members of the UMass NLP lab for their insightful input. This project was partially supported by awards IIS-2202506 and IIS-2312949 from the National Science Foundation (NSF) as well as an award from Open Philanthropy.

23Benjamin’s Upwork: https://www.upwork.com/freelancers/~01cfc42298090772b8?mp_source=share
24Elizabeth’s Upwork: https://www.upwork.com/freelancers/~01e0d3569ef95d391d7?mp_source=share
25Lynette’s Upwork: https://www.upwork.com/freelancers/~01cc5594cf2405eae5?mp_source=share
26Roberta’s Upwork: https://www.upwork.com/f1/robertaking?mp_source=share
27Shannon’s Upwork: https://www.upwork.com/freelancers/~01daaa89e049285294?mp_source=share

References

Mike Allen. 2017. The SAGE encyclopedia of communication research methods. SAGE Publications, Inc, 2455 Teller Road, Thousand Oaks California 91320.

Anthropic. 2023. Claude: A language model by anthropic. Accessed: 2025-01-20.

Anthropic. 2024. Claude 3 model card addendum. https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf. Accessed: 2024-12-30.

Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2023. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. In The Twelfth International Conference on Learning Representations.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv preprint. ArXiv:2005.14165 [cs].

Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. 2024. Art or artifice? large language models and the false promise of creativity. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for Computing Machinery.

Yapei Chang, Kalpesh Krishna, Amir Houmansadr, John Frederick Wieting, and Mohit Iyyer. 2024. PostMark: A robust blackbox watermark for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8969–8987, Miami, Florida, USA. Association for Computational Linguistics.

Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021. All that’s ‘human’ is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7282–7296, Online. Association for Computational Linguistics.

Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, and Yejin Choi. 2021. Is gpt-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. In Annual Meeting of the Association for Computational Linguistics.
Jad Doughman, Osama Mohammed Afzal, Hawau Olamide Toyin, Shady Shehata, Preslav Nakov, and Zeerak Talat. 2025. Exploring the Limitations of Detecting Machine-Generated Text. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4274–4281, Abu Dhabi, UAE. Association for Computational Linguistics.

Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. 2024. RAID: A shared benchmark for robust evaluation of machine-generated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12463–12492, Bangkok, Thailand. Association for Computational Linguistics.

Liam Dugan, Daphne Ippolito, Arun Kirubarajan, and Chris Callison-Burch. 2020. RoFT: A Tool for Evaluating Human Detection of Machine-Generated Text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 189–196, Online. Association for Computational Linguistics.

Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Callison-Burch. 2022. Real or fake text?: Investigating human ability to detect boundaries between human-written and machine-generated text. In AAAI Conference on Artificial Intelligence.

Bradley Emi and Max Spero. 2024. Technical Report on the Pangram AI-Generated Text Classifier. arXiv preprint. ArXiv:2402.14873 [cs].

Henrique Da Silva Gameiro, Andrei Kucharavy, and Ljiljana Dolamic. 2024. LLM Detectors Still Fall Short of Real World: Case of LLM-Generated Short News-Like Posts. arXiv preprint. ArXiv:2409.03291 [cs].

Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihaio Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Iharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307–1323, Online. Association for Computational Linguistics.

Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical Detection and Visualization of Generated Text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116, Florence, Italy. Association for Computational Linguistics.

Abhimanyu Hans, Avi Schwarzschild, Valeria Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Spotting LLMs with binoculars: Zero-shot detection of machine-generated text. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 17519–17537. PMLR.

Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2023. RADAR: robust ai-text detection via adversarial learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808–1822, Online. Association for Computational Linguistics.

Jiazhou Ji, Ruizhe Li, Shujun Li, Jie Guo, Weidong Qiu, Zheng Huang, Chiyu Chen, Xiaoyu Jiang, and Xinru Lu. 2024. Detecting Machine-Generated Texts: Not Just "AI vs Humans" and Explainability is Complicated. arXiv preprint. ArXiv:2406.18259 [cs].

Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021. The perils of using Mechanical Turk to evaluate open-ended text generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265–1285, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Marzena Karpinska, Nishant Raj, Katherine Thai, Yixiao Song, Ankita Gupta, and Mohit Iyyer. 2022. DEMETR: Diagnosing evaluation metrics for translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9540–9561, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. One thousand and one pairs: A “novel” challenge for long-context language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17048–17085, Miami, Florida, USA. Association for Computational Linguistics.

John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 17061–17084. PMLR.

Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2023. Paraphrasing evades detectors of ai-generated text, but retrieval
is an effective defense. In Advances in Neural Information Processing Systems, volume 36, pages 27469–27500. Curran Associates, Inc.

Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. 2024. MAGÉ: Machine-generated Text Detection in the Wild. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 36–53, Bangkok, Thailand. Association for Computational Linguistics.

Zachary C. Lipton. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31–57.

Ning Lu, Shengcai Liu, Ruidan He, and Ke Tang. 2023. Large language models can be guided to evade ai-generated text detection. Trans. Mach. Learn. Res., 2024.

Yongqiang Ma, Jiawei Liu, Fan Yi, Qikai Cheng, Yong Huang, Wei Lu, and Xiaozhong Liu. 2023. Ai vs. human – differentiation analysis of scientific content generation. Preprint, arXiv:2301.10416.

Elyas Masrour, Bradley Emi, and Max Spero. 2025. DAMAGE: Detecting Adversarially Modified AI Generated Text. arXiv preprint. ArXiv:2501.03437 [cs].

Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In International Conference on Machine Learning.

OpenAI. 2024. OpenAI o1 System Card. arXiv preprint. ArXiv:2412.16720 [cs].

B Porter and Edouard Machery. 2024. Ai-generated poetry is indistinguishable from human-written poetry and is rated more favorably. Scientific Reports, 14.

Giovanni Puccetti, Anna Rogers, Chiara Alzetta, Felice Dell’Orletta, and Andrea Esuli. 2024. AI "News" Content Farms Are Easy to Make and Hard to Detect: A Case Study in Italian. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15312–15338. ArXiv:2406.12128 [cs].

Vinu Sankar Sadasivan, Aounou Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2024. Can AI-Generated Text be Reliably Detected? arXiv preprint. ArXiv:2303.11156 [cs].

Chantal Shaib, Yanai Elazar, Junyi Jessy Li, and Byron C. Wallace. 2024. Detection and measurement of syntactic templates in generated text. In Conference on Empirical Methods in Natural Language Processing.

Zhouxing Shi, Yihan Wang, Fan Yin, Xiangning Chen, Kai-Wei Chang, and Cho-Jui Hsieh. 2023. Red teaming language model detectors with language models. Transactions of the Association for Computational Linguistics, 12:174–189.

Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, and Jasmine Wang. 2019. Release Strategies and the Social Impacts of Language Models. ArXiv.

Edward Tian and Alexander Cui. 2023. Gptzero: Towards detection of ai-generated text using zero-shot and supervised methods".

Vivek Kumar Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. 2023. Ghostbuster: Detecting text ghost-written by large language models. In North American Chapter of the Association for Computational Linguistics.

James Wang, Ran Li, Junfeng Yang, and Chengzhi Mao. 2024a. RAFT: Realistic Attacks to Fool Text Detectors. arXiv preprint. ArXiv:2410.03658 [cs].

Tianchun Wang, Yuanzhou Chen, Zichuan Liu, Zhanwen Chen, Haifeng Chen, Xiang Zhang, and Wei Cheng. 2024b. Humanizing the Machine: Proxy Attacks to Mislead LLM Detectors. ArXiv. Publisher: arXiv Version Number: 1.

Yuxia Wang, Rui Xing, Jonibek Mansurov, Giovanni Puccetti, Zhuohan Xie, Minh Ngoc Ta, Jiahui Geng, Jinyu Su, Mervat Abassy, Saad El Dine Ahmed, Kareem Elozeiri, Nurkhan Laiyk, Maiya Goloburda, Tarek Mahmoud, Raj Vardhan Tomar, Alexander Aziz, Ryuto Koike, Masahiro Kaneko, Artem Shelmanov, Ekaterina Artemova, Vladislav Mikhailov, Akim Tsvigun, Alham Fikri Ajji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2025. Is human-like text liked by humans? multilingual human detection and preference against ai. Preprint, arXiv:2502.11614.

Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392.

Qihui Zhang, Chujie Gao, Dongping Chen, Yue Huang, Yixin Huang, Zhenyang Sun, Shilin Zhang, Weiyi Li, Zhengyan Fu, Yao Wan, and Lichao Sun. 2024. Llm-as-a-coauthor: Can mixed human-written and machine-generated text be detected? In NAACL-HLT.

Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, Chao Cao, Hanqi Jiang, Hanxu Chen, Yiwei Li, Junhao Chen, Huawen Hu, Yihen Liu, Huaqin Zhao, Shaochen Xu, Haixing Dai, Lin Zhao, Ruidong Zhang, Wei Zhao, Zhenyuan Yang, Jingyuan Chen, Peilong Wang, Wei Ruan, Hui Wang, Huan Zhao, Jing Zhang, Yiming Ren, Shihuan
Qin, Tong Chen, Jiaxi Li, Arif Hassan Zidan, Afrar Jahin, Minheng Chen, Sichen Xia, Jason Holmes, Yan Zhuang, Jiaqi Wang, Bochen Xu, Weiran Xia, Jichao Yu, Kaibo Tang, Yaxuan Yang, Bolun Sun, Tao Yang, Guoyu Lu, Xianqiao Wang, Lilong Chai, He Li, Jin Lu, Lichao Sun, Xin Zhang, Bao Ge, Xintao Hu, Lian Zhang, Hua Zhou, Lu Zhang, Shu Zhang, Ninghao Liu, Bei Jiang, Linglong Kong, Zhen Xiang, Yudan Ren, Jun Liu, Xi Jiang, Yu Bao, Wei Zhang, Xiang Li, Gang Li, Wei Liu, Dinggang Shen, Andrea Sikora, Xiaoming Zhai, Dajiang Zhu, and Tianming Liu. 2024. Evaluation of OpenAI o1: Opportunities and Challenges of AGI. arXiv preprint ArXiv:2409.18486 [cs].

Ying Zhou, Ben He, and Le Sun. 2024. Humanizing machine-generated content: Evading ai-text detection through adversarial attack. In International Conference on Language Resources and Evaluation.

Tiffany Zhu, Kexun Zhang, and William Yang Wang. 2024. Embracing AI in Education: Understanding the Surge in Large Language Model Use by Secondary Students. arXiv preprint. ArXiv:2411.18708 [cs] version: 1.

A Human Evaluation

In this section of the appendix we provide additional details about our experts the data collection pipeline.

Annotators: The annotations for the experiment 1 were done by 5 annotators recruited on Upwork. All annotators are native English speakers from the US or South Africa. All annotators hold university degrees, worked in varying professions, and had varying levels of familiarity with AI assistants like ChatGPT. One had never used AI, 2 had little experience, and 2 used AI every day. Our five expert annotators are native English speakers hailing from the US, UK, and South Africa. Most work as editors, writers, and proofreaders and have extensively used AI assistants. See Table 4 for more information about annotators.

Collecting Human Annotations All annotators were required to read the guidelines (Figure 4) and sign a consent form (Figure 5) prior to the labeling task. Collecting all labels usually required additional communication with the annotators, resulting in about 20 hours of work from the author involved in this process. We estimate that the annotators were able to read and label between 8 and 12 articles per hour based on self-reported time and records in the spreadsheets. Figure 6 shows the interface annotators use to complete the annotation process. They read and highlight an article, then complete the annotation by providing their decision, confidence score, and explanation. Since completing annotations for 60 article batches takes a long time (estimated 6-8 hours of work), we have implemented an interface that made it possible for the annotators to save their work and come back at any time, allowing annotators to allocate their time as they saw best. Annotators were given 1 week to complete batches of 60 articles, with more time allowed when needed. Note that unlike Figure 1, annotators did not see the titles of the article when completing annotations. This decision was made to prevent obvious linkage of AI-generated and human-written text pairs.

Study Design : We employ a within-subjects design for our experiments, where each annotator judges both the human-written and AI-generated articles. This reduces variability from individual differences and requires fewer annotators (Allen, 2017). To further minimize bias, annotators are unaware of the pairing, and the article order is randomized.

A.1 Finding Expert Annotators

In our pilot study (§C), only the annotator who uses LLMs daily to edit AI-generated content achieved high accuracy. Prior work has noted on average, annotators cannot reliably detect AI-generated content (Brown et al., 2020; Clark et al., 2021), but vary widely in ability (Dugan et al., 2022); in this paper, we aim to investigate the upper bound of human performance. To find this limit, and understand the lexical clues humans use to recognize AI-generated content, we recruit four additional annotators with similar backgrounds to the high-performing annotator.

The annotations for understanding the limitations of humans to detect AI-generated texts were done by 5 annotators who passed performance requirements based on the article experiment of subsection 2.1. Only one annotator from the original experiment met these requirements. We recruited 10 more native English speakers on Upwork to take a 5 question sample of the original article task, questions which at most 2 out of 5 of the original annotators got correct. These 10 recruited annotators all had experience editing LLM generated content, frequently used LLMs, and had a professional background in editing or writing. Those who got at least 4 out of 5 correct (80%) given the rest of the 60 articles, and annotators who got at least
<table>
  <tr>
    <th>Annotator</th>
    <th>Education Level</th>
    <th>English Dialect</th>
    <th>LLM Usage</th>
    <th>AI Models Used</th>
    <th>Occupation</th>
  </tr>
  <tr>
    <th colspan="6">Non-Expert</th>
  </tr>
  <tr>
    <td>-</td>
    <td>Professional Degree or Doctorate</td>
    <td>American English</td>
    <td>Few times</td>
    <td>ChatGPT</td>
    <td>Writer</td>
  </tr>
  <tr>
    <td>-</td>
    <td>Bachelor’s Degree</td>
    <td>American English</td>
    <td>Daily</td>
    <td>ChatGPT, Gemini, Llama</td>
    <td>Finance/Operations</td>
  </tr>
  <tr>
    <td>-</td>
    <td>Some college, no degree</td>
    <td>American English</td>
    <td>Few times</td>
    <td>None</td>
    <td>Transcription, editing, data entry</td>
  </tr>
  <tr>
    <td>-</td>
    <td>Bachelor’s Degree</td>
    <td>American English</td>
    <td>Never</td>
    <td>None</td>
    <td>English tutor, copywriter, author</td>
  </tr>
  <tr>
    <th colspan="6">Expert</th>
  </tr>
  <tr>
    <td>Annotator 1</td>
    <td>Bachelor’s Degree</td>
    <td>South African English</td>
    <td>Daily</td>
    <td>ChatGPT, Microsoft Copilot</td>
    <td>Freelance editing, writing, proofreading</td>
  </tr>
  <tr>
    <td>Annotator 2</td>
    <td>Bachelor’s Degree</td>
    <td>South African English</td>
    <td>Daily</td>
    <td>ChatGPT</td>
    <td>Editing and Proofreading</td>
  </tr>
  <tr>
    <td>Annotator 3</td>
    <td>Master’s Degree</td>
    <td>British English</td>
    <td>Daily</td>
    <td>ChatGPT</td>
    <td>Copywriting, Proofreader</td>
  </tr>
  <tr>
    <td>Annotator 4</td>
    <td>Bachelor’s Degree</td>
    <td>American English</td>
    <td>Weekly</td>
    <td>ChatGPT, Llama, Huggingface models</td>
    <td>Freelancer Content Writer</td>
  </tr>
  <tr>
    <td>Annotator 5</td>
    <td>Master’s Degree</td>
    <td>South African English</td>
    <td>Weekly</td>
    <td>ChatGPT, Claude</td>
    <td>Language teacher</td>
  </tr>
</table>

Table 4: Survey of Annotators, specifically their backgrounds relating to LLM usage and field of work. Note that expert Annotator #1 was one of the original 5 annotators (along with the 4 non-experts) and remained an annotator for all expert trials.

90% correct (54/60) were considered to be experts and recruited for the rest of the labeling rounds. 5 out of 10 annotators passed the 5 question trial; out of those 5, 4 passed the 60 question set.

While some individuals unfamiliar with LLMs may prove to be reliable detectors, we focus on the reliability of people who frequently use LLMs for writing tasks (e.g. English teachers, editors, publishers), as our pilot study suggests their expertise enables them to consistently identify systematic patterns characteristic of AI-generated text.

B Dataset

In this section of the appendix we provide more details on our article corpus (B.1), how our AI articles were generated (B.2).

B.1 Article Corpus

Here we include more details about the articles collected for this study. Table 5 lists all publications of articles included in the corpus,28 with section distribution presented in Figure 7. Table 7 provides the statistics for articles by publication.

B.2 AI Text Generation

We generate the articles by prompting the models with the articles title, subtitle, approximate length, publication, and section. For stories, we prompt models with the title of the reddit thread. All closed-source models were prompted using the provider’s API.29 All models were prompted and articles with the prompt presented in Table 8. Instructions for rounding to an approximate length were included to ensure that articles on the same topic would be of similar length. Statistics on the distribution of lengths by trial are presented in Figure 8. An example article generation prompt is below:

You are given the following title and subtitle of a general article from the Science section of New York Times and asked to write a corresponding article of around 750 words. Include quotations from relevant experts and make sure the article is concise and easily understandable to a lay audience.

Title: The Science That Makes Baseball Mud ‘Magical’

Subtitle: Scientists dug up the real dirt on the substance applied to all the baseballs used in the major leagues.

Article:

Paraphrasing Attack We use the sentence-level paraphrasing approach outlined in PostMark (Chang et al., 2024), changing the exact language in the prompts slightly for our use case. This approach paraphrases the text on a sentence by sentence level. For the initial sentence, only that sentence is given to be paraphrased. For all following sentences, the portion of the text already paraphrased is passed to the LLM, intending to improve the overall flow of the paraphrased article. Additionally, since our experts easily identified AI-generated named entities (e.g., Sarah Thompson) in the previous experiments, we extract named entities from the human-written article and instruct GPT-4o to include these names in its generated article. We use GPT-4o as the paraphraser model with a temperature set to 0. The prompt for the initial sentence can be found in Table 9 and the prompt for all following sentences can be found in Table 10.

28All publications in the corpus were purchased by the researchers.
29The estimated cost for generating all data with each model is as follows: GPT-4o $1.85USD, O1-PRO $3.81 USD, CLAUDE-3.5-SONNET $0.51 USD
<table>
  <tr>
    <th>Publication</th>
    <th>Example Sections</th>
    <th>Pub. Date Range</th>
  </tr>
  <tr>
    <td>Associated Press</td>
    <td>Science, Oddities, Animals</td>
    <td>May 15, 2024 - Dec 5, 2024</td>
  </tr>
  <tr>
    <td>Discover Magazine</td>
    <td>Mind, The Sciences, Environment, Planet Earth</td>
    <td>July 10, 2024 - Nov 16, 2024</td>
  </tr>
  <tr>
    <td>National Geographic</td>
    <td>Animals, Environment, Science, History & Culture, Travel</td>
    <td>Feb 8, 2023 - Nov 19th, 2024</td>
  </tr>
  <tr>
    <td>New York Times</td>
    <td>US News, Science, Travel, Arts</td>
    <td>July 19, 2024 - Dec 6, 2024</td>
  </tr>
  <tr>
    <td>Readers Digest</td>
    <td>Knowledge, Holidays</td>
    <td>March 16, 2023 - Nov 15, 2024</td>
  </tr>
  <tr>
    <td>Scientific American</td>
    <td>Mind & Brain, Social Sciences, Technology</td>
    <td>May 3, 2024 - Nov 22, 2024; 2024</td>
  </tr>
  <tr>
    <td>Smithsonian Magazine</td>
    <td>Smart News, Mind & Body, History, Innovation, Travel, Science</td>
    <td>Oct 7, 2022 - Dec 8, 2024</td>
  </tr>
  <tr>
    <td>Wall Street Journal</td>
    <td>Science, Personal Technology, Workplace</td>
    <td>July 7, 2023 - Oct 29, 2024</td>
  </tr>
</table>

Table 5: List of publications included in HUMAN DETECTORS. The section is provided as listed as the section of the publication website where the article was published. All articles were taken from publications that wrote using American English.

<table>
  <tr>
    <th>Exp # Model</th>
    <th>Human-written<br>(n=30)</th>
    <th>AI-generated<br>(n=30)</th>
  </tr>
  <tr>
    <td>1 GPT-4o</td>
    <td>652.5<sub>170.9</sub></td>
    <td>625.1<sub>126.9</sub></td>
  </tr>
  <tr>
    <td>2 CLAUDE-3.5-SONNET</td>
    <td>793.1<sub>170.3</sub></td>
    <td>710.8<sub>133.5</sub></td>
  </tr>
  <tr>
    <td>3 GPT-4o paraphrased</td>
    <td>777.8<sub>175.9</sub></td>
    <td>654.9<sub>110.6</sub></td>
  </tr>
  <tr>
    <td>4 o1-PRO</td>
    <td>728.4<sub>151.1</sub></td>
    <td>813.8<sub>233.6</sub></td>
  </tr>
  <tr>
    <td>5 o1-PRO humanized</td>
    <td>739.0<sub>128.3</sub></td>
    <td>815.6<sub>171.2</sub></td>
  </tr>
</table>

Table 6: Mean and standard deviation (subscripted) of article length in words across experiments, computed by splitting on whitespaces.

Humanization Efforts: To humanize the articles in §2.5, we tried many prompt-based approaches before settling on a prompting framework that was capable of evading the first author’s own AI-detection skills. Initially, we tried the following approaches that humanized already generated texts:

1. Generate, then Humanize: Firstly, we tried to replicate what we believe most students or novice LLM users would do to humanize their AI-generated text. We asked o1-Pro to first generate an article using the prompt template depicted in Table 8, then instructed the model to make it sound more human. Qualitative analysis from the first author found this method to result in an effect similar to paraphrasing.

2. Step-by-Step: To add a fine-grained approach to the humanization efforts, we next tried a step-by-step approach to humanization. We generated a base article, then iteratively prompted the model to humanize the article, focusing on a different element at each step. For example, we first asked it to make the article more creative, then alter the tone. Final steps included editing for grammar and replacing typical AI vocab. While this was better, the first author observed that the numerous calls to LLMs was adding more characteristics of AI, so we abonded this method.

3. Two-step Humanization: Next, we tried taking a 2-step approach to humanization. We gave the humanizing LLM the AI detection guide found in Table 11, and asked the LLM to make a list of everything identifiable as AI-generated in the article, and suggested edits. Then, giving the LLM the list of suggested edits, we prompted the LLM to edit the article to make it sound more human-written. Out of the humanization efforts, this yielded the best results.

Evader Details: While approaches to humanization were improving, we found many of the largest identifiable traits of AI were due to the base generation. We decided to switch our approach from humanizing already existing AI-generated text to having an LLM generate the article in a humanized fashion all in one step. Our final prompt includes the set of instructions, examples of reference articles, detection guide, and the initial article prompt. The evader prompt template can be found in Table 13. The examples used were human-written and AI-generated articles from Experiments 1,2,3, and 4 from the same source. For example, when evading an article from New York Times, we provided all human-written New York Times articles and AI-generated articles based off those article titles.

Additional Existing Humanizers Recently, the general population is much more interested in evading automatic detectors, 30 many commercial humanization services exist to fill this demand. Some

30 Relevant discussions on Reddit: "AI Humanizer Recommendations?" (2024), "How to humanize ai-generated texts?" (2024), "How to humanize the AI generated content?" (2024)
<table>
  <tr>
    <th rowspan="2">Articles</th>
    <th colspan="2">Associated Press<br>(n=25)</th>
    <th colspan="2">Discover<br>(n=20)</th>
    <th colspan="2">National Geographic<br>(n=25)</th>
    <th colspan="2">New York Times<br>(n=20)</th>
    <th colspan="2">Readers Digest<br>(n=15)</th>
    <th colspan="2">Scientific American<br>(n=15)</th>
    <th colspan="2">Smithsonian Magazine<br>(n=25)</th>
    <th colspan="2">Wall Street Journal<br>(n=15)</th>
  </tr>
  <tr>
    <th>Mean</th><th>St. Dev.</th><th>Mean</th><th>St. Dev.</th><th>Mean</th><th>St. Dev.</th><th>Mean</th><th>St. Dev.</th><th>Mean</th><th>St. Dev.</th><th>Mean</th><th>St. Dev.</th><th>Mean</th><th>St. Dev.</th><th>Mean</th><th>St. Dev.</th>
  </tr>
  <tr>
    <td>TOKENS (nltkoken)</td>
    <td>860.0</td><td>240.7</td><td>949.5</td><td>229.9</td><td>1070.7</td><td>203.2</td><td>1009.1</td><td>138.4</td><td>865.6</td><td>171.9</td><td>874.2</td><td>347.1</td><td>922.0</td><td>210.7</td><td>927.5</td><td>202.5</td>
  </tr>
  <tr>
    <td></td>
    <td>1236.0</td><td>342.0</td><td>1463.0</td><td>561.0</td><td>1497.0</td><td>710.0</td><td>1254.0</td><td>645.0</td><td>1318.0</td><td>634.0</td><td>1760.0</td><td>455.0</td><td>1454.0</td><td>532.0</td><td>1336.0</td><td>408.0</td>
  </tr>
  <tr>
    <td>WORDS (nltk.space)</td>
    <td>674.0</td><td>180.8</td><td>734.6</td><td>174.5</td><td>809.7</td><td>149.9</td><td>784.1</td><td>106.3</td><td>675.3</td><td>136.6</td><td>684.2</td><td>265.9</td><td>708.1</td><td>160.0</td><td>720.5</td><td>153.9</td>
  </tr>
  <tr>
    <td></td>
    <td>1026.0</td><td>342.0</td><td>1117.0</td><td>415.0</td><td>1102.0</td><td>536.0</td><td>951.0</td><td>513.0</td><td>1008.0</td><td>501.0</td><td>1352.0</td><td>362.0</td><td>1095.0</td><td>388.0</td><td>1019.0</td><td>491.0</td>
  </tr>
</table>

Table 7: Number of tokens and words across articles by source.

Article Generation Prompt

You are given the following title and subtitle of a general article from the YOUR SECTION section and asked to write a corresponding article of around YOUR WORD COUNT words. Include quotations from relevant experts and make sure the article is concise and easily understandable to a lay audience.

Title: YOUR TITLE
Subtitle: YOUR SUBTITLE

Table 8: Prompt Template for Experiment 2 Paraphrasing

Paraphrase Prompt (Initial Sentence Only)

Paraphrase the given sentence. Only return the paraphrased sentence in your response. Make it seem like a human wrote the article and that it is from the YOUR SECTION section of YOUR PUBLICATION.

Sentence to paraphrase: YOUR SENTENCE
Your paraphrase of the sentence:

Table 9: Prompt Template for Experiment 2 Paraphrasing

humanization services include Undetectable AI, MyEssayWriter.ai and Stealth Writer.

C Pilot Study

In this section, we provide details on the pilot study, where five annotators with varying frequencies of familiarity with LLMs detect if texts are human-written or AI-generated. The pilot study follows the same annotation process described in §2. In the study, annotators completed two rounds of testing, one using the same articles as identified in §2.1, another using stories from the subreddit r/WritingPrompts.

Story Corpus: Here we include more details about the stories collected for this study. We collect 30 stories from r/WritingPrompts. We generate corresponding AI-generated stories with the prompt in Table 15.

Paraphrase Prompt

Paraphrase the given sentence. Only return the paraphrased sentence in your response. Make it seem like a human wrote the article and that it is from the YOUR SECTION section of YOUR PUBLICATION.

Previous context: YOUR PARAPHRASED ARTICLE SO FAR
Sentence to paraphrase: YOUR SENTENCE
Your paraphrase of the sentence:

Table 10: Prompt Template for Experiment 2 Paraphrasing

Story Results : Overall, nonexperts faired better on stories than articles, with an average TPR of 69.3% (including performance of the expert annotator). The nonexperts had a TPR of 62.5% not including our expert annotator, as shown in Table 16. We found that many of the differences spotted were superficial due to the very informal nature of reddit, compared to our prompt which did not instruct the LLM to generate stories of that nature. In reddit, people use acronyms, for example shortening ‘you’ to ‘u’, type in ALL CAPS, or generally write in a very R-rated manner. While these are real differences between AI-generated and human-written texts, we wanted to pursue clues humans could use in settings where a person was genuinely attempting to pass off AI-generated writing as human-written, such as students writing an essay or someone publishing a news article. Future work may explore human detection abilities on fictional work, where the human-written references are more edited than the reddit thread stories.
D Comment Analysis

In this section, we outline the framework for analyzing expert explanations and provide additional analysis of explanations.

D.1 Categorization of Comments

To categorize explanation comments, we first define the explanation categories found in Table 3. The first two authors individually annotated a sample of 25 expert explanations. The sample is a stratified sample of one comment per expert per each of the five experiments. Authors then came to an agreement on final human labels, refining categories as needed. We then prompt GPT-4o-2024-11-20 with the prompt found in Table 18 to categorize the sample explanations. Once the prompt was able to classify the majority of the sample explanations in alignment with the authors labels, we prompt GPT-4o-2024-11-20 to categories the explanations of all 1500 explanations. Each explanation can contain labels for multiple categories, since most explanations touch on multiple reasons of why a text is AI-generated. The total cost of classifying comments using GPT-4o was $6.02 USD.

D.2 What do expert annotators see that nonexperts don’t?

D.3 Individual Experts Commentary

Each expert had clues they favored using throughout all experiments. Annotator 1, whose category mention frequencies can be found in Figure 10, Figure 11 depicts Annotator 2 comments, Figure 12 shows Annotator 3 comments, Figure 13 shows Annotator 4 comments and Figure 14 has commentary frequencies from Annotator 5. The individual heatmaps highlight the range of clues used by annotators, who individually had no reference of clues other experts used. We again note that our experts are untrained at this detection task, and they could likely improve their individual performance if provided with feedback on their errors.

Annotators don’t always focus on the same clues:
Our experts focus on different properties of the text to arrive at their decisions. Annotator 1 is the only one to pick up on “AI names”: in fact, 63.3% of GPT-4o and 70% of CLAude-3.5-Sonnet articles include either the name Emily or Sarah.31 Annotator 2 and 3 emphasize linguistic features, such as humans not always following “proper” writing conventions or the tendency of LLMs to always list examples in groupings of three. Annotator 4 focuses on the flow of the articles, analyzing the specificity of detail and motivation behind an article, while Annotator 5 frequently mentions whether a quotation sounds natural.

Commentary on Paraphrasing Attack in Experiment 3: Annotators continue to pick up on many of the same clues within the paraphrased articles that were also apparent in Experiments 1 & 2, such as high frequency of “AI vocab” (even after paraphrasing), formulaic sentence structures, and cheerful summary conclusions. Somewhat counterintuitively, explanations about paraphrased articles note AI vocab in 88% of explanations, compared to only 69.8% of non-paraphrased GPT-4o articles. Similarly, quotations are mentioned in 33.8% of explanations about paraphrased articles, a much higher rate than other configurations: a close reading of explanations reveals that experts flagged quotes that were always in the same format and style (e.g., only placed at the end of each paragraph).

One Annotator Finds o1 Content Hard to Detect
Interestingly, Annotator 3’s TPR drops considerably, as they prioritize signs of human writing over signs of AI writing when making their judgments. Annotator 3, the only one consistently fooled by o1-PRO outputs, relied too much on signs of informality (e.g., contractions, slang usage, usage of just and actually) as a sign of human writing, with 66.7% of their explanations relating to formality (Figure 12).

E Automatic Detection

E.1 Automatic Detectors Benchmarked

In this section, we detail the detectors used as automatic benchmarks in Table 2.

• Pangram (Emi and Spero, 2024) is a closed-source commercial detector implemented via a Transformer classifier trained via an iterative process that uses hard negative mining and synthetic data to improve data efficiency. We run both their base model and a newer model (PANGRAM HUMANIZERS) trained to distinguish humanized data (Masrou et al., 2025). We directly use the labels (i.e., thresholds) produced by the Pangram API and do not award credit for the neutral label “Possibly AI”.

31 o1-PRO favors names of real people instead of fictional names when generating articles.
• GPTZero (Tian and Cui, 2023) is also a closed-source commercial detector that runs a classifier sentence-by-sentence across the document. We directly use the binary labels produced by the GPTZero API.

• Binoculars (Hans et al., 2024) is an open-source detector that relies on the cross perplexity computed by two different language models to perform detection. We run Binoculars with the two FPR thresholds recommended by its authors (“Accuracy” and “Low FPR” modes).

• Fast-DetectGPT (Bao et al., 2023) is an open-source method that samples and scores many perturbations of the text to estimate conditional probability curvature. We threshold this method at a FPR of 0.05, set on our held-out dev set of 40 articles.

• RADAR (Hu et al., 2023) train an open-source classifier adversarially against a paraphraser. We threshold this method at a FPR of 0.05, set on our held-out dev set of 40 articles.

Thresholding for Automatic Detectors: Some automatic detectors benchmarked in Table 2 and Table 23 did not provide suggested thresholds for model usage. In these scenarios, we test models on a held-out test set 40 human-written articles, finding a threshold for a 5% FPR. The respective thresholds used were 0.6051510572 for RADAR, 0.96 for Fast-DetectGPT, and 0.8963184953 for e5-lora.

E.2 Explainable Detection
In this section we detail the prompt based detector used for experiments in §4. While humans experts are found to be robust detectors, an obvious drawback is that hiring humans is expensive and slow: on average, we paid $2.82 per article including bonuses, and gave annotators a week to complete each experiment. We wondered if we could prompt an LLM to do the same.

Implementation Details The prompt-based detector is set up to mimic how humans think about AI-generated text detection rather rather than how current automatic detectors do.

• Zero-Shot: To find a baseline performance of detector models, we prompt the model using the template in Table 19 to return if the candidate text is Human-written or AI-generated.

• Zero-Shot + COT: We prompt the model using the template in Table 20 to return if the candidate text is Human-written or AI-generated and an explanation of why they text is human-written. This ablation was conducted to understand the effect explanations may have on over LLM detection performance.

• Zero-Shot + Guide: In this experiment, we prompt the model using the template in Table 21 to return if the candidate text is Human-written or AI-generated. This experiment serves to understand the effect including the guide in the prompt has on LLM performance.

• Zero-Shot + COT + Guide: In this experiment, we prompt the model using the template in Table 22 to return if the candidate text is Human-written or AI-generated and an explanation of why they text is human-written. THis set up is set up to fully mimic how a human detects, thinking through clues and providing explanations of what makes them think a text is either human-written or AI-generated.

We observe that expert annotators perform better when using a majority vote strategy. Future work in prompt-based detectors could employ majority voting as a strategy to increase detection accuracy and avoid false positives.

E.3 Results
Detector Model We initially test GPT-4o-2024-08-06, GPT-4o-2024-11-20, o1, and CLAUDE-3.5-SONNET as potential detector models. Based on early results, we continue with ablations of o1 due to its low FPR and GPT-4o-2024-11-20 due to having the highest TPR. The cost of all o1 detection experiments on the 300 texts is $100.60 USD and $25.24 USD for GPT-4o-2024-11-20 detection experiments.

Additional Results: While testing both GPT-4o-2024-08-06 and GPT-4o-2024-11-20 as detectors, we find extremely varied performance, with GPT-4o-2024-11-20 performing much better, particularly at detecting paraphrased and o1 articles. CLAUDE-3.5-SONNET failed to detect content generated by itself and o1We also tested e5-lora, one of the top performing models on the Raid (Dugan et al., 2024) benchmark, which we found was unable to correctly classify any AI-generated articles at a FPR of 5%. See all additional results in Table 23.
Guidelines

1. Why are we collecting this data?

We are a university research lab based in [location]. You were most likely contacted by [researcher name] Currently, we are investigating the ability of humans to distinguish between texts written by humans and those written by large language models (e.g., ChatGPT, Clouds).

We are interested in creating a dataset of short stories annotated with spans of text that people can fact check or signals to determine whether a story was written by human or AI. We also want to check whether the text was randomly labeled or authored by a human or AI.

Prompt:
You are not a good person. Your party was made of good people, and you need to be good because you liked having allies. But now they're all missing, so they won't see the lengths you're willing to go to to save them.

Human-Generated Text | Machine-Generated Text

I understand why you might think that, it's true that I've always been surrounded by people who inspire goodness, and perhaps my actions have been influenced by their presence. But now, with them all gone, I find myself at a crossroads. The truth is, I care deeply about them, and I am willing to do whatever it takes to bring them back safely.

Maybe my methods won't align with the ideals of a "good person," but sometimes the lines between right and wrong blur when the stakes are this high. I am driven by a fierce loyalty and a sense of responsibility to those who have stood by me. If that means stepping into the shadows and making difficult choices, then so be it.

I hope that when they return, they will understand why I did what I did. I hope they see that my actions, though perhaps not traditionally good, were born out of love and desperation. In the end, I believe that a good person is not just in their intentions, but in their willingness to fight for what truly matters. And for me, that's them.

Label: Machine-Generated Text

Overall, our goal is to:

(i) Create a dataset that clearly labels all the words/sentences in a story that motivate a human to think that either a human or AI wrote the text.

(a) Guidelines: Page 1

(2) Read the short story, which is highlighted in the red box. While you are reading, please use both the Human-Generated Text and Machine-Generated Text annotation buttons to highlight clues. You do this in 2 ways:

(a) clicking on the annotation label, and then highlighting the words or phrase in the story you would like to annotate
(b) clicking the corresponding number key on your keyboard and then highlighting the words or phrases in the story you would like to annotate (s for Human-Generated Text, 2 for Machine-Generated Text)

Here is an example of a text annotated with labels (these labels were randomly chosen and are NOT an example of a well labeled paragraph). Ultimately, there is no "correct" answer to this task. We are interested in your perception. What looks like a machine-generated text? Why? What looks like a human-written text? Why?

Human Generated Text | Machine-Generated Text

I understand why you might think that. It's true that I've always been surrounded by people who inspire goodness, and perhaps my actions have been influenced by their presence. But now, with them all gone, I find myself at a crossroads. The truth is, I care deeply about them, and I am willing to do whatever it takes to bring them back safely.

Maybe my methods won't align with the ideals of a "good person," but sometimes the lines between right and wrong blur when the stakes are this high. I am driven by a fierce loyalty and a sense of responsibility to those who have stood by me. If that means stepping into the shadows and making difficult choices, then so be it.

I hope that when they return, they will understand why I did what I did. I hope they see that my actions, though perhaps not traditionally good, were born out of love and desperation. In the end, I believe that a good person is not just in their intentions, but in their willingness to fight for what truly matters. And for me, that's them.

(3) Please respond to the following questions after reading the story

(a) Human or Machine: If the text was written by a human or by a machine

(b) Your confidence: Your confidence in your conclusion of who wrote the text

(c) Guidelines: Page 3

(2) Understand how confident people are in their ability to determine the source of story authorship.

2. What is your task?

We want to ask you to read a writing prompt, the short story response, and annotate the story. You should do the following:

(i) Read the prompt at the top of the page, which is highlighted in the purple box. This is what the story will be about.

Please annotate the following text with all words or phrases that made you think the text was written by a human or machine.

Human Generated Text | Machine-Generated Text

You are not a good person. Your party was made of good people, and you need to be good because you liked having allies. But now they're all missing, so they won't see the lengths you're willing to go to to save them.

Please use the two colored boxes to annotate Human-Generated and Machine-Generated text in the same story.

Human Generated Text | Machine-Generated Text

Understand why you might think that. It's true that I've always been surrounded by people who inspire goodness, and perhaps my actions have been influenced by their presence. But now, with them all gone, I find myself at a crossroads. The truth is, I care deeply about them, and I am willing to do whatever it takes to bring them back safely.

Maybe my methods won't align with the ideals of a "good person," but sometimes the lines between right and wrong blur when the stakes are this high. I am driven by a fierce loyalty and a sense of responsibility to those who have stood by me. If that means stepping into the shadows and making difficult choices, then so be it.

I hope that when they return, they will understand why I did what I did. I hope they see that my actions, though perhaps not traditionally good, were born out of love and desperation. In the end, I believe that a good person is not just in their intentions, but in their willingness to fight for what truly matters. And for me, that's them.

(ii) Short form explanations: Please explain your choice. Feel free to reference the clues you highlighted in the earlier part of the text, any intuition you may have, or any knowledge you have that made you decide one way over another. Please mention why you rated yourself at a certain confidence level with your choice.

Please choose how you think the text was generated

Human Generated | Machine Generated

Please note your confidence you are in a state of S.C. with 1 being the least confident and 5 being the most confident.

Please explain your choice in a few sentences. What made you think the text was written by a human or by AI? How confident are you in your response?

(c) Guidelines: Page 4

4. Summary

Overall, we asked you to:

(1) Read a writing prompt and the accompanying short story.
(2) Highlight any clues that make the story seem written by a human or by AI.
(3) Choose who wrote the story (human or machine)
(4) Rank how confident you are in your choice
(5) Write a short explanation about why you felt the story was written by the selected author and how confident you were in your choice.

Figure 4: Guidelines provided to the annotators for the annotation task. The annotators were also provided additional examples and guidance during the data collection process.
Guide to Distinguishing AI-Generated Text from Human Writing Template

## Vocabulary / Word Choice Patterns
- Certain words crop up unusually frequently throughout AI-generated text compared to human writing
- Words like 'delve' and 'tapestry' are overused in AI-generated text but infrequently used in human writing ...

## Grammar
- Human writing generally less strictly adheres to English grammar rules and punctuation than AI-generated text
- AI-generated text uses a very formal writing style unless explicitly told not to ...

## Sentence Structure
- AI-generated sentences often follow the complex sentence structure, with multiple dependent and independent clauses, while human writing contains more of a mixture of simple, complex, and compound sentences
- example AI-generated sentence: “When it comes to celebrating Halloween, this holiday is a testament to the importance of empathy and community.” ...

## Formatting
- When AI makes lists, it typically uses the format of creating a bold header per bullet point, followed by a colon and then description of that list item.
- If a book title is referenced in a text, AI-generated text will always italicize the title, while human writing does not always follow this convention.
- Some pieces of text, especially articles and essays, contain headers and sub-headers. The headers written by AI are quite repetitive ...

## Tone
- The tone of AI-written text is flowery and formal, and its sentences are frequently structured as a reflective, onlooking statement, regardless of topic.
- AI tends to be inherently positive, attempting to emotionally uplift the reader, especially towards the conclusion.
- AI prioritizes efficiency, sometimes sacrificing clarity or depth in its messaging ...

## Introductions
- AI-written introductions often contain a strong scene-opener with a description of a specific time or place, such as "On a drab November morning..." or "On December 8, 1660, a London audience gathered ..." ...

## Conclusions
- AI-generated text always ends with a neat conclusion, instead of just ending the article naturally.
- AI-generated conclusions are often overly long and summarize everything that has already been written in an article ...

## Content
- Unless specifically prompted, AI will avoid controversial topics at all costs. - AI will avoid any type of swear word, including mild ones like ‘darn’, or any other offensive vocabulary ...

## Contextual Accuracy and Factuality
- Human writing in the domain of non-fiction is factually accurate and contains many specific factual claims.
- In human writing, people, places, brands, and other named objects can be verified or are highly plausible ...

## Creativity & Originality
- AI-generated text is much less creative than that of humans, lacking originality and sticking to an 'obvious' way to answer a prompt.
- Humans incorporate twists, unexpected insights, and twists that AI hasn’t seemed to master quite yet ...
AI Vocabulary Included in Detection Guide

Nouns: aspect, challenges, climate, community, component, development, dreams, environment, exploration, grand scheme, health, hidden, importance, landscape, life, manifold, multifaceted, nuance, possibilities, professional, quest, realm, revolution, roadmap, role, significance, tapestry, testament, toolkit, whimsy

Verbs: capturing, change, consider, delve/dive into, elevate, embrace, empower, enact, enhance, engage, ensure, evoking, evolving, explore, fostering, guiding, harness, highlights, improve, integrate, intricate, jeopardizing, journey, navigating, navigate, notes, offering, partaking, resonate, revolutionize, shape, seamlessly, support, tailor, transcend, underscores, understanding

Adjectives: authentic, complex, comprehensive, crafted, creative, critical, crucial, curated, deeper, diverse, elegant, essential, groundbreaking, key, meaningful, paramount, pivotal, powerful, profound, quirky, robust, seamless, significant, straightforward, structured, sustainable, transformative, valuable, vast, vibrant, vivid, whimsical

Adverbs: additionally, aptly, creatively, moreover, successfully

Phrases: as we [verb] the topic, cautionary tale, connect with, has shaped the, in a world of/where, in conclusion, in summary, it’s crucial to, it’s important to note, it’s not about ___ it’s about ____, manage topic issues/problems, not only ___ but also, packs a punch/brings a punch, paving the way, personal growth, quality of life, remember that, simple yet ____, step-by-step, such as, the effects of, the rise of, their understanding of, they identified patterns, to form the, to mitigate the risk, weaving, when it comes to topic

Table 12: All ‘AI’ Vocabulary our expert annotators noted, as listed in the Detector Guide. See the full detection guide prompt in Table 11.
Evader Template

You are a journalist writing an article for . Your goal is to write an article given a title and subtitle, but make it look like it was written by a human. To assist you, we provide the following detection guide to determine if something is written by a human or by AI.

AI Detection Guide

You are given some examples of human and AI written articles to help you.
<start examples>
Example Articles
<end examples>

You are also given the following guidelines to make your article seem like a human wrote it.

<start of guidelines>
- Follow the detection guideline to ensure writing does not sound AI-generated.
- Do not use any of the "Overused AI Words/Phrases", "Overused AI Metaphors", "AI Grammar Patterns", AI sentence structures, AI tones, common AI names, or AI content described in the guide.
- Include words/phrases/grammar patterns/tone/content that are consistent with human writing as described in the guide.
- Use human-written examples and explanations as references for formatting and content.
- Use the example explanations to see what humans identify as 'AI' or 'Human'
- Avoid formulaic introductions that elaborate on scene-setting or time, immediately quote experts, or bring in too much historical context.
- Avoid formatting the first setting by mentioning the setting, then a comma, then some detail or context. Avoid mentioning atmospheric details or time (of day, month, year, etc ...) in the first sentence as well.
- Avoid generic, vague, and forward-looking conclusions, and do not summarize the article in the conclusion to make articles sound more human-like. Concluding sentences don’t have to wrap everything up nicely.
- Avoid using generic statements to start paragraphs, such as 'For now' or 'In the end'
- Add references to darker topics when appropriate.
- Add specific references to places, people, brands, items, facts, and metrics when appropriate.
- Always opt for specifics over broad details. For example, mention correct statistics instead of broad claims and mention company names instead of saying the type of company.
- Make sure that none of the claims made in your article are factually incorrect or implausible.
- Use language aligned with the publication you are representing as a journalist
- If your article would benefit from being organized into sections (e.g., scientific or historical content), please use section headers to do so. There are 1/3 odds an article has headers.
<end of guidelines>

Now, by following the guidelines above, examples, and the information in the detection guide, please write an article without any AI signatures that also includes signatures of human writing. No reader should doubt that your article could have been published by the given source. Readers with access to the information in the detection guide should not be able to detect the article was written by AI.

Article Generation Prompt
Article:

Table 13: Prompt used for evader. The first insert into the prompt is filled by Table 11. The second insert is filled by examples of human and machine-generated articles.
New Telescope Could Potentially Identify Planet X
Are there hidden planets in our solar system? New technologies, like the powerful Rubin Observatory, brings us closer to answers.

For decades, astronomers have speculated that there might be another large planet lurking in the outer reaches of our solar system—an elusive “Planet X.” Often referred to as “Planet Nine” by modern researchers, this hypothetical world is believed to be so distant that it has escaped detection by existing telescopes, leaving only subtle gravitational clues in its wake. Now, a new generation of telescopes, particularly the upcoming Vera C. Rubin Observatory, promises a more powerful set of eyes on the night sky. With these tools, scientists hope to either pinpoint this mysterious planet’s location or finally put the idea to rest.

Why the Search Matters
The concept of a hidden planet is not new. In the 19th century, the discovery of Neptune followed suspicions that Uranus’s odd orbital path hinted at an unseen gravitational pull. Today, attention has turned to the distant suburbs of our solar system, where far-off objects in the Kuiper Belt—frozen remnants of planetary formation—appear to cluster in strange ways. Some astronomers argue that only a large, unseen planet’s gravity could explain these odd orbits.

“The evidence is subtle, but it’s there,” said Dr. Konstantin Batygin, a planetary astrophysicist at the California Institute of Technology (Caltech) who, along with colleague Dr. Mike Brown, first proposed the existence of a Planet Nine in 2016. “We’re seeing several distant Kuiper Belt objects all tilted and clustered in a peculiar manner, and a ninth planet several times Earth’s mass, orbiting far beyond Neptune, could be the simplest explanation.”

Not everyone is convinced. Skeptics point to the small number of known distant objects and argue that the clustering could be a statistical fluke. Others think unseen observational biases might make it look like these objects are behaving strangely. Either way, the mystery remains unsolved—at least for now.

Enter the Vera C. Rubin Observatory
Scheduled to begin its full operations in the near future, the Vera C. Rubin Observatory (formerly known as the Large Synoptic Survey Telescope, or LSST) in Chile represents a leap forward in astronomical capability. With a massive 8.4-meter mirror and a state-of-the-art digital camera, it will repeatedly scan the entire southern sky over the course of a decade, providing a dynamic, time-lapse-like portrait of celestial motions.

“The Rubin Observatory is really a game-changer,” said Dr. Meg Schwamb, an astronomer at Queen’s University Belfast who studies the outer solar system. “Instead of looking at a small patch of sky, we’ll be looking at pretty much everything visible from Chile, over and over again. This repeated coverage means we can detect faint, distant objects that move slowly across the sky—exactly the kind of signature we’d expect from a far-flung planet.”

By systematically imaging the sky every few nights, the Rubin Observatory’s Legacy Survey of Space and Time (LSST) will reveal thousands—or even tens of thousands—of new objects in the outer solar system. Among them could be Planet Nine, if it exists. Even if the telescope doesn’t directly spot the planet, it might detect more distant objects whose orbits can be mapped with unprecedented precision, allowing astronomers to figure out once and for all if a hidden giant is lurking out there.

Technological Edge
One reason Planet Nine (or any Planet X) has been so hard to pin down is that if it exists, it’s incredibly faint and slow-moving, possibly hundreds of times farther from the Sun than Earth is. Traditional telescopes rely on painstaking surveys that cover only small portions of the sky at a time. In contrast, Rubin’s wide field of view—about 40 times the size of the full Moon—means it will cover the visible sky every few days.

“This isn’t just another telescope—it’s a new way of doing astronomy,” said Dr. Lynne Jones, an LSST researcher at the University of Washington. “We’re moving from static snapshots to continuous movies. If there’s something out there, no matter how faint, as long as it’s moving, we have a good chance of picking it up over time.”

Ruling In or Ruling Out
If the Rubin Observatory does find Planet Nine, the discovery would reshape our understanding of the solar system’s architecture. A distant giant could be a leftover core of a gas giant that got tossed out in the early days of planetary formation, or it might have formed in situ, far from the Sun’s warmth. Such a find could help scientists piece together the chaotic early era when planets were jostling for position and smaller bodies were flung into distant orbits.

On the other hand, if Rubin’s comprehensive survey concludes after several years without any sign of Planet Nine—or if newly discovered distant objects don’t line up as predicted by the planet hypothesis—then the case for a hidden giant will weaken. Instead, astronomers might refine models of how solar system objects distribute themselves naturally. Understanding these patterns could still teach us valuable lessons about how gravity and cosmic debris shape the outskirts of our celestial neighborhood.

Beyond Planet X
The quest for Planet Nine is just one highlight of what the Rubin Observatory offers. It will also track near-Earth asteroids that could pose future hazards, study dark matter and dark energy by observing the distribution of galaxies, and contribute to countless other fields of astronomy.

Still, for many sky-watchers, the idea of a hidden planet holds a particular allure. The notion that our solar system might still hold major surprises underscores how much we have yet to learn about our own cosmic backyard.

“As much as astronomy has advanced, we’re still explorers,” said Batygin. “The Rubin Observatory gives us the tools to either find this planet or lay the mystery to rest. In the next decade, I expect we’ll know a lot more about what’s really out there—and what isn’t.”

Until then, the search continues, driven by powerful new telescopes and the human desire to uncover the unseen.
Deciding Authorship of Short Stories

participate in this study because you fit the criteria of being an adult over the age of 18 and are a native English speaker.

Why am I doing this research study?
The goal of this project is to determine how well people can detect if text is written by a human or by an Artificial Intelligence (AI).

Who can participate in this research study?
Any adult over the age of 18 who is a native English speaker can participate in this study.

What will I be asked to do and how much time will it take?
I will read short stories and answer questions about them.

You will be asked to: (1) access the label studio web application shared with you, (2) read through all instructions, (3) label words or phrases in the text that seem human or AI generated, and (4) determine if you have enough information to make a decision. You may also provide a short explanation of why you made your choices in your answers and provide a short explanation behind your decisions of who wrote the story.

The content of the stories you will read may be sensitive, stories may contain themes of sadness, depression, violence, magic, etc. Content may include sensitive language.

We estimate that you will be able to complete ~12 stories per hour. Depending on how many stories you have agreed to participate in, this could take anywhere from 5 minutes to 3 hours. For example, if you are annotating 60 stories, it will take about 5 hours to complete.

Will help in this research study be given in any way?
You may not directly benefit from this research; however, we hope that your participation in the study will help inform if people can detect whether text they are reading is written by another human or by AI models. This will contribute further our understanding of how often humans truly recognize the source of content they are consuming.

What are my risks of being in this research study?
We believe there are minimal risks associated with this research study; however, a risk of breach of confidentiality is present. If you have any thoughts or feelings to express this risk is outlined in a section below.

How will my personal information be protected?
To the best of our ability your answers in this study will remain confidential. We will minimize any risk by not collecting personally identifiable information.

Will I be paid money or other compensation for being in this research study?
You will be paid $10 to complete the pre-study survey linked here.

You will be paid at a rate of $2 per story via the Upwork platform. You will be paid according to the total number of stories agreed via Upwork upon satisfactory completion of all agreed upon work.

What happens if I say yes, but change my mind later?
You may withdraw from the study at any time. If you agree to be in the study but later change your mind, you may drop out at any time. There are no penalties or consequences of any kind if you decide that you do not want to participate.

Who can I talk to if I have questions?

By signing this consent, I acknowledge that:

• I voluntarily agree to participate in this research study.
• I understand that I will be asked to read 20 short stories and answer the corresponding questions, unless agreed otherwise via Upwork.
• I have been informed of the purpose and nature of the study and I have had the opportunity to ask questions about the study and what data I have the right to ask questions during the task.
• I understand participation involves:
    1. Labeling words or phrases in a story that I feel need to be human or AI-generated
    2. Determining if I think the story was written by a human or AI
    3. Providing a short explanation of why I think the story was written by a human or AI
• I understand that all information I provide for this study will be handled confidentially.
• I understand that in any report on the results of this research my identity will remain anonymous.
Please sign below if you have read the above terms and fully agree with them.

I have read and understood the guidelines. I understand that I can ask any additional questions now or during the task via Upwork.
[ ] Yes
[ ] No

I agree that all the data (announcements, short story, comments, validation, and other annotations) can be used in academic experiments.
[ ] Yes
[ ] No

I will complete the pre-task survey after signing this consent form, linked here, which will be stored in anonymized form, and agree that all data created can be used for data analysis.
[ ] Yes
[ ] No

By clicking “I agree” below you are indicating that you are at least 18 years old, have read this consent form and agree to participate in this research study. You are free to skip any question that you choose.

Please print a copy of this page for your records.
Signature

Short answer test

Story Generation Prompt

Please write a story around WORD COUNT words responding to the following prompt: STORY PROMPT.

Table 15: Prompt Template for Story Generation, where STORY PROMPT is the writing prompt from r/WritingPrompts that the human-written story was written about and WORD COUNT is the length of the story to generate.

<table>
  <tr>
    <th>Annotator</th>
    <th>TPR (%)</th>
    <th>FPR (%)</th>
  </tr>
  <tr>
    <td>Nonexpert Annotator 1</td>
    <td>73.3</td>
    <td>76.7</td>
  </tr>
  <tr>
    <td>Nonexpert Annotator 2</td>
    <td>43.3</td>
    <td>80.0</td>
  </tr>
  <tr>
    <td>Nonexpert Annotator 3</td>
    <td>66.7</td>
    <td>80.0</td>
  </tr>
  <tr>
    <td>Expert Annotator 1</td>
    <td>96.7</td>
    <td>93.3</td>
  </tr>
  <tr>
    <td>Nonexpert Annotator 4</td>
    <td>66.7</td>
    <td>76.7</td>
  </tr>
  <tr>
    <th>Average</th>
    <th>69.3</th>
    <th>81.3</th>
  </tr>
</table>

Table 16: Story performance of initial 5 annotators, which included 4 nonexpert annotators and expert annotator #1, who was used in all article experiments.

Figure 5: Consent form which the annotators were asked to sign via GoogleForms before collecting the data.
A full-house villagers — Spinalonga, on the Kritsouni Bay’s west bank, gathered for a special turkey Thanksgiving in a small poolside circle overnight. They weren’t expecting rain. No one would have bet on the app-edition givers, out where single-engine aircraft from the nearby air strip was on call. This one was waiting for a second pair of all-weather conditions, dropping Thanksgiving turkeys almost every other day to nearby fishing waters.

“Turkeys are a big part of our Christmas and Thanksgiving traditions,” said Anna Mavrogianni, 48, who owns a small restaurant. “We don’t eat much turkey here, but half our community is Greek,” said Pericles, standing a head above the others in his dark sweater and hat. “It’s cold. What I just put on is my sweater.”

He shrugged as if that was the most ordinary idea, then laughed. “Folks around here get used to it.”

Because the villagers are from Spinalonga, it is a bare rush connecting their home to Athens by ferry, and residents rely on small boats or occasionally for daylight. With no roads, the area’s coffee - an island blend - looks positively fresh, a bit like the air you breathe after a long walk through the mountains. The island has been closed off since 1957 because of leprosy, and the last person with the disease died in 2006. Today, the island is a popular tourist destination and a symbol of hope and healing.

Pericles, 31, grew up in a house with his parents and three sisters. He was born with a hearing impairment and now uses a hearing aid. His mom runs a small restaurant and shop that sells hand-crafted items from a weaving family. “I’m not sure how I’ll ever leave the island,” he said.

In the village, residents spend the morning fixing up a church and a cemetery as a way of giving back to the community. “The people here are very close-knit,” said Pericles. “We care about each other.”

The island is known for its traditional music and dances. People sing and play instruments at festivals and weddings. “We never had something like this when I lived in Athens,” Pericles said. “This is a real Greek tradition.”

The turkey was prepared by the local priest, who cooks the bird at the church’s annual thanksgiving event. The turkey is roasted in an open fire and served with potatoes, carrots, and onions. The family also makes a traditional Greek salad and drinks wine.

Figure 6: Interface for annotators, with an example annotation from Annotator #4 with a humanized article from §2.5. This is the same article displayed in Figure 1. An annotator can highlight texts, make their decision, put confidence, and write an explanation. This AI-generated article was based off of In Alaska, a pilot drops turkeys to rural homes for Thanksgiving, written by Mark Thiessen & Becky Bohrer, and was originally published by Associated Press on Nov. 28, 2024.
<table>
  <tr>
    <th>CATEGORY</th>
    <th>FREQ</th>
    <th>DEFINITION</th>
    <th>EXAMPLE EXPLANATIONS</th>
  </tr>
  <tr>
    <td>VOCABULARY</td>
    <td>53.1%</td>
    <td>LLMs use specific words and phrases more often than human writers, which often results in repetitive, unnatural, or overly complex wording.</td>
    <td>Human: "Furthermore, I very much doubt AI would have used adventurous adjectives like 'chunky', 'musky' or 'thin' to describe food. Nor would it have used verbs like 'blezing' or 'bolstering'.<br>AI (O1-HUMANIZED): 'Odd word choices: wheat that 'stores' a lineage; genes that are 'honed'.'"</td>
  </tr>
  <tr>
    <td>Sentence Structure</td>
    <td>35.9%</td>
    <td>AI-generated sentences follow predictable patterns (e.g., high frequency of "not only ... but also ...", or consistently listing three items), while human-written sentences vary more in terms of length.</td>
    <td>Human: "Short choppy sentences and paragraphs."<br>AI (O1-PRO): "One pattern I've been noticing with AI, and I think I've stated this before, is the comparison of 'it's not just this, it's this' and I'm seeing it here, along with listings of specifically three ideas."<br>AI (GPT-40-PARA): "there's nothing off about the grammar or syntax in this piece..."</td>
  </tr>
  <tr>
    <td>Grammar & Punctuation</td>
    <td>24.8%</td>
    <td>AI-generated text is usually grammatically perfect (also avoiding dashes and ellipses), while human-written text often contains minor errors.</td>
    <td>Human: "There's a lot of variety in the article's grammar use, with dashes, brackets, quotes intermixed with sentences, and short spurts of comma sections throughout."</td>
  </tr>
  <tr>
    <td>Originality</td>
    <td>23.7%</td>
    <td>AI-generated writing is generally straightforward, "safe," and lacking in surprises or humor, leaving annotators bored or unengaged.</td>
    <td>Human: "It's offset by some great analogies and creative phrasing that works well to convey the topic, such as with 'amateur sleuths' - 'catnip' for a certain type of Reddit user."<br>AI (O1-PRO): "What happens when AI tries to be creative? Penguins 'stand on their own flippers'."</td>
  </tr>
  <tr>
    <td>Quotes</td>
    <td>22.3%</td>
    <td>AI-generated quotes sound overly formal, lack the varied nuances of real conversation, and often mirror the article's main text too closely in style.</td>
    <td>Human: "The quotes being short snippets also makes me think they're real, as the writer had to find a way to fit them into the text, rather than them just perfectly stating either side's views."<br>AI (GPT-40): "The quotes also feel fake, every expert speaks the same way and it's lost homogeneity in the text."</td>
  </tr>
  <tr>
    <td>Clarity</td>
    <td>19.5%</td>
    <td>AI-generated text often lacks concise flow by over-explaining or including irrelevant details, effectively "telling" rather than "showing".</td>
    <td>Human: "Words like 'meander' are used, but are used sparingly to create better flow of ideas, and its writing style is simplified in the best way possible."<br>AI (CLAUDE-3.5-SONNET): "The sentences are condensed to provide the best possible precision with its word choice, but the article lacks flow and clarity."</td>
  </tr>
  <tr>
    <td>Formatting</td>
    <td>15.0%</td>
    <td>AI-generated formatting is overly consistent (e.g., fully capitalized headings, bolded lists, paragraphs of similar length).</td>
    <td>Human: "the formatting itself is varied, with topic headers focusing on lowercase lettering, longer explanations, and the bullet points not going into a title: one to two sentence explanation format ..."<br>AI (O1-PRO): "... it has a structured article format of commonly used headers..."</td>
  </tr>
  <tr>
    <td>Conclusions</td>
    <td>13.1%</td>
    <td>AI-generated conclusions tend to be repetitive and overly optimistic summaries, while human-written conclusions end more abruptly and less tidily.</td>
    <td>Human: "Both the introduction and the conclusion were humorous and unique."<br>AI (CLAUDE-3.5-SONNET): "Most of the conclusion leads to that summarizing, flowery tone."</td>
  </tr>
  <tr>
    <td>Formality</td>
    <td>12.3%</td>
    <td>AI-generated text (especially without humanization) rarely contains filler words, contractions, slang, or abbreviations, favoring fully spelled-out terms and a polished tone.</td>
    <td>Human: "It includes filler words like 'just', 'very' and 'really'. Colloquial language like 'naked'."<br>AI (CLAUDE-3.5-SONNET): "Phrases like 'maintains that extending habitats corpus rights to animals' and 'fundamentally alter the ability of accredited zoos to conduct their vital conservation work' are so strictly formal, so tight and dense with wordage."</td>
  </tr>
  <tr>
    <td>Names & Titles</td>
    <td>11.7%</td>
    <td>LLMs frequently generate the same names regardless of the prompt (e.g., "Emily Carter," "Sarah Thompson"). Furthermore, if an article contains multiple people, they tend to refer to all of them with the same title (e.g., "Dr.") without offering unique details. Unique or context-specific names (including real brands or products) are associated with human writing.</td>
    <td>Human: "A couple of the experts also have quite unique names, and none of them are referred to as Dr. X (bonus points for none of them being named Emily)."<br>AI (GPT-40): "However, the introduction of Dr. Sarah Thompson and Dr. Emily Carter (who has more than a lifetime's worth of qualifications), means that it has to be AI text."</td>
  </tr>
  <tr>
    <td>Tone</td>
    <td>9.3%</td>
    <td>The tone of AI-generated text is consistently neutral or positive, lacking depth or emotional variety compared to human writing.</td>
    <td>Human: "The article seems to display some bias against TikTok while trying to remain impartial or impartial-seeming. The same goes for its political slant which although subtle, seems to be present."<br>AI (O1-HUMANIZED): "...the majority of the writing is filled with the same language it uses to describe everything - inspirational, stunning, essential, and resonating - using formal words, an inherent positivity bias, and a reflective, romantic tone that doesn't give advice on why this topic matters, why we still need to know who she is, and whether or not she should be allowed to continue her work."<br>AI (CLAUDE-3.5-SONNET): "The article has a very generic introduction and conclusion, especially the introduction, which essentially tells you what the entire article is about."</td>
  </tr>
  <tr>
    <td>Introductions</td>
    <td>7.3%</td>
    <td>AI-generated introductions are usually generic or focus on scenic details without providing key background, while human introductions have more compelling hooks and relevant context.</td>
    <td>Human: "The introduction is unique as it starts with the subject of the article watching a movie rather than instantly explaining the entire point of the article."<br>AI (CLAUDE-3.5-SONNET): "The article has a very generic introduction and conclusion, especially the introduction, which essentially tells you what the entire article is about."</td>
  </tr>
  <tr>
    <td>Factuality</td>
    <td>7.2%</td>
    <td>AI-generated text contains factual inconsistencies or hallucinations more often than human writing.</td>
    <td>Human: "The article is very factual, with details about the dates, materials used, and sales portions."<br>AI (GPT-40-PARA): "Incorrect information delivered with customary AI confidence: the 'audience' is another name for the backdrop."</td>
  </tr>
  <tr>
    <td>Topics</td>
    <td>3.1%</td>
    <td>Unlike human authors, LLMs generally avoid darker or more mature topics (e.g., violence, graphic descriptions).</td>
    <td>Human: "Phrases throughout the article, including 'burned to ashes and scattered at sea to prevent the crowd from venerating them as relics' and 'to templete on a brass likeness of Jesus or the Virgin Mary—a blasphemous act,' actively show what happened, the horrors and tragedies that occurred during that time."<br>AI (GPT-40): "It spends very little time talking about the horrors of the disease, and instead focuses on future research, hopeful quotes, and potential cures, even referring to it as 'embarrassing or a disappointment'."</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>2.6%</td>
    <td>Other clues that do not fall into any of the above categories, often based on the annotator's intuition or overall impression.</td>
    <td>Human: "There are no clues here apart from the highlighted sentence which seems to have a human 'ring'."<br>AI (O1-HUMANIZED): "The article feels slightly artificial but I cant really find any clear clues for it."</td>
  </tr>
</table>

Table 17: Full taxonomy of clues used by experts to explain their detection decisions. For each category, we report the frequency of explanations that mention that category (regardless of if the annotator was correct) and provide examples of explanations for both human-written and AI-generated articles. While vocabulary and sentence structure form the most frequent clues, more complex phenomena like originality, clarity, formality, and factuality are also distinguishing features.
Comment Categorization Prompt

We hired an annotator to determine whether an article is AI-generated or human-written. Alongside their “machine-generated” or “human-generated” label, they provided an **explanation** detailing the specific clues that led them to their decision.

Your task is to **review** the annotator’s explanation and **categorize** each clue they mention. For each clue, identify:
1. The **category** it falls under (e.g., Vocabulary, Grammar, etc.).
2. The **label** (AI-generated or Human-written) the annotator associates with that clue.
3. The exact **quote** from the annotator’s explanation that shows this clue.

Below is a list of categories and **example** indicators the annotator might reference. **Note:** These examples do **not** cover all possibilities. If a comment fits two categories, choose the best one.

{DEFINITIONS_OF_ALL_CATEGORIES}

Instructions:
- Read the annotator’s explanation.
- Identify any category that applies.
- Indicate whether the annotator says it points to AI-generated or human-written.
- Provide the exact quote from the annotator's explanation that led you to this conclusion.

**If the annotator cites both AI and human clues within the same category**, please create **two separate entries** (one for AI-generated, one for human-written).
**If no category is applicable**, categorize it under “other.”

**Answer Format** (use this structure for each separate category/label pair):

<category>YOUR CATEGORY HERE</category>
<label>YOUR LABEL HERE (AI-generated or Human-written)</label>
<quote>RELEVANT QUOTE FROM EXPLANATION</quote>

<category>YOUR CATEGORY HERE</category>
<label>YOUR LABEL HERE (AI-generated or Human-written)</label>
<quote>RELEVANT QUOTE FROM EXPLANATION</quote>

(Repeat as needed, using a new block for each category/label pair.)
Example:
<category>Vocabulary</category>
<label>AI-generated</label>
<quote>"The article mentioned the word 'crucial' and used a lot of unusual synonyms."</quote>

The annotators explanation is as follows:

<explanation>EXPLANATION</explanation>

Table 18: Truncated prompt used for comment analysis. The first insert is filled by a list of definitions of the categories found in Table 3 and the second insert is filled by an annotator explanation.
Figure 7: Section distribution of articles across all trials.

Figure 8: Word Count Distribution Per Experiment

<table>
  <tr>
    <th colspan="2">Zero Shot Detector Template</th>
  </tr>
  <tr>
    <td>You are given a candidate text and an AI detection guide. Your task is to carefully read the candidate text and determine whether it was either written by a human or generated by AI.</td>
  </tr>
  <tr>
    <td>Answer HUMAN-WRITTEN if the candidate text was likely written by a human.</td>
  </tr>
  <tr>
    <td>Answer AI-GENERATED if the candidate text was likely generated by an AI.</td>
  </tr>
  <tr>
    <td><start of candidate text> Candidate Text </end of candidate text></td>
  </tr>
  <tr>
    <td><question>Is the above candidate text HUMAN-WRITTEN or AI-GENERATED? </question></td>
  </tr>
  <tr>
    <td>Provide your final answer in this format.</td>
  </tr>
  <tr>
    <td><answer> YOUR ANSWER </answer></td>
  </tr>
</table>

Table 19: Prompt Template for the zero-shot detector set up

<table>
  <tr>
    <th colspan="2">Zero-shot + CoT template</th>
  </tr>
  <tr>
    <td>You are given a candidate text. Your task is to carefully read the candidate text and determine whether it was either written by a human or generated by AI.</td>
  </tr>
  <tr>
    <td>Answer HUMAN-WRITTEN if the candidate text was likely written by a human.</td>
  </tr>
  <tr>
    <td>Answer AI-GENERATED if the candidate text was likely generated by an AI.</td>
  </tr>
  <tr>
    <td><start of candidate text> Candidate Text </end of candidate text></td>
  </tr>
  <tr>
    <td><question>Is the above candidate text HUMAN-WRITTEN or AI-GENERATED? </question></td>
  </tr>
  <tr>
    <td>First, concisely describe the features of the candidate text that exemplify either AI or human writing. Then, provide your final answer.</td>
  </tr>
  <tr>
    <td><description> YOUR DESCRIPTION </description></td>
  </tr>
  <tr>
    <td><answer> YOUR ANSWER </answer></td>
  </tr>
</table>

Table 20: Prompt Template for the Zero-shot + CoT detector setup

<table>
  <tr>
    <th colspan="2">Zero-shot + Guide Template</th>
  </tr>
  <tr>
    <td>You are given a candidate text and an AI detection guide. Your task is to carefully read the candidate text and determine whether it was either written by a human or generated by AI.</td>
  </tr>
  <tr>
    <td>Answer HUMAN-WRITTEN if the candidate text was likely written by a human based on the information in the provided guide.</td>
  </tr>
  <tr>
    <td>Answer AI-GENERATED if the candidate text was likely generated by an AI based on the information in the provided guide.</td>
  </tr>
  <tr>
    <td>DETECTION GUIDE</td>
  </tr>
  <tr>
    <td><start of candidate text> Candidate Text </end of candidate text></td>
  </tr>
  <tr>
    <td><question>Based on the detection guide, is the above candidate text HUMAN-WRITTEN or AI-GENERATED? </question></td>
  </tr>
  <tr>
    <td><answer> YOUR ANSWER </answer></td>
  </tr>
</table>

Table 21: Prompt Template for the Zero-shot + Guide detector setup
Figure 9: Same as Figure 3, except only computed over explanations when experts were incorrect. Formality is a big source of misdirection for o1-Pro articles, while fixating on sentence structure can lead experts to false positives. Details of each category can be found in Table 3.

Figure 10: Annotator 1 Frequency of Categories Mentioned in Explanations

Figure 11: Annotator 2 Frequency of Categories Mentioned in Explanations
Figure 12: Annotator 3 Frequency of Categories Mentioned in Explanations

![Annotator 3 Frequency of Categories Mentioned in Explanations](page_186_120_1077_495.png)

Figure 13: Annotator 4 Frequency of Categories Mentioned in Explanations

![Annotator 4 Frequency of Categories Mentioned in Explanations](page_186_620_1077_495.png)

Figure 14: Annotator 5 Frequency of Categories Mentioned in Explanations

![Annotator 5 Frequency of Categories Mentioned in Explanations](page_186_1120_1077_495.png)
Zero-shot + CoT + Guide Template

You are given a candidate text and an AI detection guide. Your task is to carefully read the candidate text and determine whether it was either written by a human or generated by AI.

Answer HUMAN-WRITTEN if the candidate text was likely written by a human based on the information in the provided guide.
Answer AI-GENERATED if the candidate text was likely generated by an AI based on the information in the provided guide.

DETECTION GUIDE

<start of candidate text> Candidate Text </end of candidate text>

<question>Based on the detection guide, is the above candidate text HUMAN-WRITTEN or AI-GENERATED? </question>

First, use the provided guide to concisely describe the features of the candidate text that exemplify either AI or human writing. Then, provide your final answer.

<description> YOUR DESCRIPTION </description>
<answer> YOUR ANSWER </answer>

Table 22: Prompt Template for the Zero-shot + CoT + Guide detector setup
<table>
  <tr>
    <th rowspan="2">Detection Method</th>
    <th colspan="6">Generation Method</th>
    <th rowspan="2">Overall TPR% (FPR%)</th>
  </tr>
  <tr>
    <th>GPT-4o<br>TPR% (FPR%)</th>
    <th>CLAUDE<br>TPR% (FPR%)</th>
    <th>GPT-4o PARA.<br>TPR% (FPR%)</th>
    <th>O1-PRO<br>TPR% (FPR%)</th>
    <th>O1-Pro Human.<br>TPR% (FPR%)</th>
  </tr>
  <tr>
    <th colspan="8">(B) Automatic detectors</th>
  </tr>
  <tr>
    <td>LS-LORA (FPR=0.05)</td>
    <td>0 (0)</td>
    <td>0 (0)</td>
    <td>0 (0)</td>
    <td>0 (0)</td>
    <td>0 (0)</td>
    <td>0 (0)</td>
    <td>0 (0)</td>
  </tr>
  <tr>
    <th colspan="8">(C) Prompt-based detectors</th>
  </tr>
  <tr>
    <td>Detector LLM: GPT-4o-2024-08-06</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ZERO-SHOT GPT-4o-2024-08-06</td>
    <td>100 (13.3)</td>
    <td>100 (3.3)</td>
    <td>26.7 (0)</td>
    <td>3.3 (0)</td>
    <td>0 (0)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ZERO-SHOT + CoT + GUIDE (GPT-4o-2024-08-06)</td>
    <td>96.7 (3.3)</td>
    <td>100 (10)</td>
    <td>70 (3.3)</td>
    <td>46.7 (6.7)</td>
    <td>0 (3.3)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Detector LLM: CLAUDE-3.5-SONNET-2024-12-17</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ZERO-SHOT + CoT + GUIDE</td>
    <td>86.7 (0)</td>
    <td>43.3 (0)</td>
    <td>90.0 (0)</td>
    <td>6.7 (0)</td>
    <td>0 (0)</td>
    <td>53.3 (8.6)</td>
    <td></td>
  </tr>
</table>

Table 23: Each cell displays TPR (FPR), with TPR in normal text and FPR in smaller parentheses. Colors indicate performance bins: TPR is darkest teal (100) at best, medium teal (90–99), and burnt orange (89–70). Scores 69 and below are in purple (<70). FPR is darkest teal (0) at best, medium teal (1–5), burnt orange (6–10), and purple (>10) at worst. No percentage signs appear in the cells, but the numeric values represent percentages (e.g., “90” means 90%). We further mark closed-source (♣) and open-weights (♦) detectors.

Reviewer paper summaries: The paper explores the ability of human annotators—particularly those experienced with large language models (LLMs)—to detect AI-generated text. The study finds that such annotators can outperform automated detection systems, demonstrating remarkable accuracy in identifying machine-generated content. The research highlights the strengths of human intuition in detecting AI text
Summary of strengths: - Paper shows that expert LLM users can detect AI-generated text with very high accuracy. A majority vote among five expert annotators misclassified only 1 out of 300 articles, significantly outperforming many commercial and open-source AI detectors.

- Unlike automated detectors, expert human annotators were not easily fooled by paraphrased or humanized AI text. This shows that linguistic and stylistic cues are crucial in distinguishing AI-generated content, even after paraphrasing or humanization.

- Authors benchmark human performance against several AI detection models, including both closed-source (e.g., Pangram) and open-source approaches (e.g., Fast-DetectGPT, Binoculars). The results show that while some automated systems perform well, they struggle against adversarial modifications, unlike expert annotators.
Summary of weaknesses: Even though it was mentioned already by the authors in the paper, I see the problem only in **Limited Generalizability**—the experiments were conducted solely **on non-fiction English articles from reputable sources**. Additionally, **only five expert annotators** were used in key experiments, and **the sample size of detected passages was small**, making it difficult to generalize the findings to a broader population or different text domains.
Comments, suggestions and typos: A related study, **"Is Human-Like Text Liked by Humans? Multilingual Human Detection and Preference Against AI"** by Wang et al., further supports the idea that human expertise is crucial in AI text detection. Their research found an upper-bound human detection accuracy of **87.6%** across **multiple languages and domains**. This reinforces the claim that trained annotators can outperform automated AI detectors, challenging previous conclusions that AI text detection by people is no better than random guessing. I recommend reading and citing this work as it provides a broader perspective on AI text detection across different languages and cultural contexts.
Overall assessment scores: 4