DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of
  GPT-Generated Text
Authors: name: Xianjun Yang; name: Wei Cheng; name: Linda Petzold; name: William Yang Wang; name: Haifeng Chen
Published: 2023-05-27 03:58:29

Abstract: Large language models (LLMs) have notably enhanced the fluency and diversity
of machine-generated text. However, this progress also presents a significant
challenge in detecting the origin of a given text, and current research on
detection methods lags behind the rapid evolution of LLMs. Conventional
training-based methods have limitations in flexibility, particularly when
adapting to new domains, and they often lack explanatory power. To address this
gap, we propose a novel training-free detection strategy called Divergent
N-Gram Analysis (DNA-GPT). Given a text, we first truncate it in the middle and
then use only the preceding portion as input to the LLMs to regenerate the new
remaining parts. By analyzing the differences between the original and new
remaining parts through N-gram analysis in black-box or probability divergence
in white-box, we can clearly illustrate significant discrepancies between
machine-generated and human-written text. We conducted extensive experiments on
the most advanced LLMs from OpenAI, including text-davinci-003, GPT-3.5-turbo,
and GPT-4, as well as open-source models such as GPT-NeoX-20B and LLaMa-13B.
Results show that our zero-shot approach exhibits state-of-the-art performance
in distinguishing between human and GPT-generated text on four English and one
German dataset, outperforming OpenAI's own classifier, which is trained on
millions of text. Additionally, our methods provide reasonable explanations and
evidence to support our claim, which is a unique feature of explainable
detection. Our method is also robust under the revised text attack and can
additionally solve model sourcing. Codes are available at
https://github.com/Xianjun-Yang/DNA-GPT.