People who frequently use ChatGPT for writing tasks are accurate and
  robust detectors of AI-generated text
Authors: name: Jenna Russell; name: Marzena Karpinska; name: Mohit Iyyer
Published: 2025-01-26 19:31:34

Abstract: In this paper, we study how well humans can detect text generated by
commercial LLMs (GPT-4o, Claude, o1). We hire annotators to read 300
non-fiction English articles, label them as either human-written or
AI-generated, and provide paragraph-length explanations for their decisions.
Our experiments show that annotators who frequently use LLMs for writing tasks
excel at detecting AI-generated text, even without any specialized training or
feedback. In fact, the majority vote among five such "expert" annotators
misclassifies only 1 of 300 articles, significantly outperforming most
commercial and open-source detectors we evaluated even in the presence of
evasion tactics like paraphrasing and humanization. Qualitative analysis of the
experts' free-form explanations shows that while they rely heavily on specific
lexical clues ('AI vocabulary'), they also pick up on more complex phenomena
within the text (e.g., formality, originality, clarity) that are challenging to
assess for automatic detectors. We release our annotated dataset and code to
spur future research into both human and automated detection of AI-generated
text.