EvoBench: Towards Real-world LLM-Generated Text Detection Benchmarking for Evolving Large Language Models
Authors: Xiao Yu, Yi Yu, Dongrui Liu, Kejiang Chen, Weiming Zhang, Nenghai Yu, Jing Shao
Accepted at: acl25

EvoBench: Towards Real-world LLM-Generated Text Detection Benchmarking for Evolving Large Language Models

Xiao Yu1*, Yi Yu2*, Dongrui Liu2, Kejiang Chen1†,
Weiming Zhang1, Nenghai Yu1, Jing Shao2‡
1University of Science and Technology of China, China
2Shanghai Artificial Intelligence Laboratory
yuxiao1217@mail.ustc.edu.cn, yuyi@pjlab.org.cn
chenkj@ustc.edu.cn, shaojing@pjlab.org.cn

Abstract

With the widespread of Large Language Models (LLMs), there has been an increasing need to detect LLM-generated texts, prompting extensive research in this area. However, existing detection methods mainly evaluate on static benchmarks, which neglect the evolving nature of LLMs. Relying on existing static benchmarks could create a misleading sense of security, overestimating the real-world effectiveness of detection methods. To bridge this gap, we introduce EvoBench, a dynamic benchmark considering a new dimension of generalization across continuously evolving LLMs. EvoBench categorizes the evolving LLMs into (1) updates over time and (2) developments like finetuning and pruning, covering 7 LLM families and their 29 evolving versions. To measure the generalization across evolving LLMs, we introduce a new EMG (Evolving Model Generalization) metric. Our evaluation of 14 detection methods on EvoBench reveals that they all struggle to maintain generalization when confronted with evolving LLMs. To mitigate the generalization problems, we further propose improvement strategies. For zero-shot detectors, we propose pruning the scoring model to extract shared features. For supervised detectors, we also propose a practical training strategy. Our research sheds light on critical challenges in real-world LLM-generated text detection and represents a significant step toward practical applications. The EvoBench is now available at: https://github.com/happy-Moer/EvoBench.

1 Introduction

Large Language Models (LLMs), such as Chat-GPT (OpenAI, 2022b), Claude (Anthropic, 2024), and LLaMA (Touvron et al., 2023a), have demonstrated remarkable capabilities in natural language understanding and task processing, leading to their widespread application (M Alshater, 2022; Yuan et al., 2022; Christian, 2023). However, concerns have been raised about the misuse of these models in areas like social media (Ahmed et al., 2021), education (Lee et al., 2023), and academic writing (Mitchell, 2022; Patrick Wood, 2023). For instance, LLMs can be used to manipulate the public by generating comments or to fabricate experimental data and statistical results in support of unverified hypotheses (Solaiman et al., 2019; Goldstein et al., 2023). The potential misuse of LLMs highlights the urgent need to detect LLM-generated text (Kaur et al., 2022; Chen and Shu, 2023).

The academic community has carried out extensive research to detect LLM-generated text effectively (Liu et al., 2019; Gehrmann et al., 2019; Su et al., 2023; Solaiman et al., 2019). Current methods can be categorized into supervised methods (Hu et al., 2023; Yu et al., 2024a; Chen et al.,

![Current benchmarks (Guo et al., 2023; Bao et al.; Wang et al., 2024b,a; Kwan et al., 2024) primarily focus on specific versions of LLMs and neglect vast ecosystem of evolving LLMs beneath the surface, including updates over time or developments through fine-tuning or pruning. The figure primarily includes LLMs before Jan 2025.](page_1016_384_384_384.png)

Figure 1: Current benchmarks (Guo et al., 2023; Bao et al.; Wang et al., 2024b,a; Kwan et al., 2024) primarily focus on specific versions of LLMs and neglect vast ecosystem of evolving LLMs beneath the surface, including updates over time or developments through fine-tuning or pruning. The figure primarily includes LLMs before Jan 2025.
Figure 2: Detection accuracy (measured in AUROC) of Fast-DetectGPT when faced with the evolving LLMs. Figure (a) shows a clear decline in average detection performance as the LLM updates. Figure (b) illustrates the developments of LLMs, including fine-tuning and pruning.

2024b; Yu et al., 2024b; Guo et al., 2024) and zero-shot methods (Ippolito et al., 2020; Yang et al.; Mitchell et al., 2023; Su et al., 2023; Bao et al.). Supervised methods are typically trained with a binary classifier to distinguish between texts generated by LLMs and those created by humans, while zero-shot methods primarily rely on statistical features gathered from pre-trained large language models. Although these methods show strong performance on existing benchmarks (Bao et al.; Hu et al., 2023; Chen et al., 2024b; Yu et al., 2024a), they often fall short in real-world applications as current benchmarks neglect the evolving nature of LLMs (Wang et al., 2024b; Guo et al., 2023; Wang et al., 2024a; He et al., 2024; Macko et al., 2023). As illustrated in Figure 1, existing benchmarks include a limited versions of LLMs, much like observers seeing only the tip of an iceberg while overlooking the vast evolving LLMs hidden beneath the surface.

In real-world applications, LLMs are continuously evolving via regular updates, fine tuning (Touvron et al., 2023b; Zhang et al., 2023), or pruning (Sun et al.; Liang et al., 2021), all of which affect LLMs’ output (Tao et al., 2024; Touvron et al., 2023a; Gunasekar et al., 2023), thereby impacting the detection performance. Figure 2 illustrates that the detection performance of current detection methods suffers up to 25% drop as LLMs evolve.1 Therefore, relying on current static benchmarks for evaluation could create a misleading sense of security, leading the research community to overestimate the real-world effectiveness of detection methods. Therefore, it is crucial to enhance existing benchmarks to better capture the ongoing evolving LLMs.

In this paper, we propose EvoBench, a dynamic benchmark that extends traditional benchmarking to account for the evolving LLMs. EvoBench aims to provide a more accurate evaluation of detection methods by incorporating two dimensions of evolving LLMs: updates and developments, as shown in Figure 1. Updates refer to changes made by the LLM publishers, including LLM updates, i.e., GPT-4o undergoes updates approximately every 3 months 2. On the other hand, developments refer to optimizations made by LLM developers for specific application scenarios, such as fine-tuning and pruning. EvoBench introduces a new dimension of generalization, enabling the evaluation of detection methods across evolving LLMs, covering 7 widely used LLMs and their 29 evolving versions.

Using EvoBench, we evaluate 14 widely used detection methods and find that all struggle to adapt to the evolving nature of LLMs. To intuitive quantify the generalization across evolving LLMs, we introduce the EMG (Evolving Model Generalization) metric. Results reveal that the performance of 14 current detection methods, including widely used Fast-DetectGPT (Bao et al.) and RADAR (Hu et al., 2023) detectors, significantly drop when faced with evolving versions, as reflected by low EMG values, while some achieve high AUROC (0.91) scores on individual models3. This phenomenon highlights the limitations of current detection methods in generalization capabilities.

To mitigate this challenge, we explore two complementary strategies targeting zero-shot and supervised detection methods, respectively. For zero-shot detectors, we propose pruning the scoring

1The widely used detection method, Fast-DetectGPT, declines up to 25% drop when detecting the Claude-3-5-haiku-20241022 compared to Claude-3-haiku-20240307.

2These versions are gpt-4o-2024-05-13, gpt-4o-2024-08-06, gpt-4o-2024-11-20, and chatgpt-4o-latest.

3Fast-DetectGPT achieve 0.91 AUROC when detecting Claude-Haiku-2024-03-07, but drop to 0.65 AUROC when detecting the version of 2024-10-22.
Figure 3: The construction pipeline and the dataset statistics of EvoBench. EvoBench includes three dimensions of generalization: dataset domains, generation strategies, and evolving LLMs. The evolving LLMs are further divided into two categories: updates and developments.

model to extract shared features that enhance detection generalization. This strategy is not limited to Fast-DetectGPT (Bao et al.), which we adopt as a representative case, but is applicable to a broader class of zero-shot methods. Our experiments demonstrate that this approach consistently improves performance across a range of detectors—including Likelihood, Rank, LogRank, Fast-DetectGPT, and BiScope—when using Sheared-LLaMA as the scoring model in detecting text generated by developments of LLaMA-2. For supervised detectors, we propose a practical training strategy aimed at enhancing generalization: including training data from the initial version of an LLM family can significantly improve performance on its subsequent updates.

Overall, EvoBench serves as a complementary benchmark that focuses on the evolving nature of LLMs, offering a more accurate framework for evaluating the real-world applicability of detection methods. While it does not claim to be a one-size-fits-all solution, we envision EvoBench evolving alongside detection methods and other benchmarks, helping guide future developments in this area. In particular, we suggest that extracting shared features from evolving LLMs could enhance the generalization of detection methods. With EvoBench, the community can more effectively evaluate and refine detection methods, contributing to more robust and practical detection systems that mitigate LLM misuse in real-world applications.

2 Related Work

2.1 Evolving LLMs

Updates of Pre-trained LLMs. The advent of LLMs, such as ChatGPT (OpenAI, 2022a) and GPT-4 (OpenAI, 2023), has marked a paradigm shift in text generation. However, these pre-trained models are not static (Tao et al., 2024; Touvron et al., 2023a; Gunasekar et al., 2023; Biderman et al., 2023); rather, they undergo continuous evolution, with models like ChatGPT frequently updating their parameters to enhance performance and adapt to new application scenarios (Zheng et al., 2024; Touvron et al., 2023a).

Developments on Pre-trained LLMs. The rise of open-source LLMs has facilitated their widespread usage in diverse domains. Developers can flexibly tailor LLMs to specific tasks through fine-tuning (Touvron et al., 2023a,b; Zhang et al., 2023), pruning (Sun et al.; Liang et al., 2021; Xia et al.), and quantization (Du et al., 2024; Chen et al., 2024a; Dettmers et al., 2023) techniques that modify the model’s parameters or structure, ultimately influencing output characteristics. In this process, developers could use domain-specific datasets to enhance the model’s understanding of specialized fields and their contextual knowledge (Kerner, 2024). Furthermore, depending on practical application scenarios, users may select models with varying parameter scales to strike an optimal balance between performance, speed, and resource consumption (Kim et al., 2024; Zhao et al., 2023).

2.2 LLM-generated Text Detection

Supervised Methods. Supervised methods (Liu et al., 2019; Chen et al., 2024b) are usually trained to differentiate between texts generated by LLMs and texts created by humans. For example, RADAR (Hu et al., 2023) introduces the idea of adversarial learning to train a detector that can resist paraphrase attacks. Text Fluoroscopy (Yu et al., 2024a) extracts discriminative features from the intermediate layers of the language model and utilizes them to train a binary classifier, which enhances the generalization of supervised detectors across texts from different semantic domains.
However, as LLMs continue to evolve, frequent adaptation mechanisms—including updates, fine-tuning, pruning, and other optimization strategies—introduce changes in their generated text, making it difficult to guarantee the effectiveness of existing detection methods on newer model versions. This creates a significant gap between academic detection models and real-world applications, ultimately limiting the practical utility of these methods.

Zero-shot Methods. Existing zero-shot methods primarily rely on statistical features extracted using pre-trained large language models (Bao et al.; Mitchell et al., 2023). These features include likelihood (Gehrmann et al., 2019), probability curvature (Mitchell et al., 2023), divergence between multiple completions of a truncated passage (Yang et al.), and conditional probability curvature (Bao et al.). Zero-shot detection methods are immune to domain-specific degradation, demonstrating superior generalization in detection tasks (Gehrmann et al., 2019; Mitchell et al., 2023). However, the effectiveness of zero-shot detection heavily depends on the alignment between the pre-trained LLM used for detection and the generation model that produced the text to be detected (Bao et al.; Mitchell et al., 2023). While current methods achieve state-of-the-art results on existing benchmarks (Guo et al., 2023; Bao et al.; Wang et al., 2024b,a; Kwan et al., 2024), the evolving nature of LLMs introduces significant shifts in this alignment. As models continue to evolve, the pre-trained LLMs used for detection may become increasingly misaligned with the updated generation models, leading to a decline in detection performance. This explains why many methods, despite ranking highly on leaderboards, often fail when deployed in real-world scenarios.

3 EvoBench

3.1 Definition of Evolving LLMs

EvoBench primarily considers two evolving pathways for LLMs: (1) updates, which are released by publishers, and (2) developments, which involve optimization made by developers. These two evolving pathways present significant challenges to current detection methods.

Updates. We focus on two types of updates that occur over time: model (Brown et al., 2020; Tao et al., 2024) and version updates (Brown et al., 2020). Model updates typically involve significant changes to the model’s architecture. For example, the transition from GPT-4 to GPT-4o represents a major architectural shift (Achiam et al., 2023). In contrast, version updates occur more frequently, with shorter cycles and less perception, such as the regular releases of new versions of GPT-4 every 2-3 months, such as updates in May, August, and November 2024.

Developments. In EvoBench, model developments include fine-tuning (Hu et al., 2022; Mangrulkar et al., 2022), pruning (Liang et al., 2021), often used for domain adaptation and optimizing efficiency for real-world applications. Unlike updates, these actions are nearly imperceptible to the detector (Bhattacharjee et al., 2023). For example, developers may fine-tune LLMs using private datasets, making the detection more challenging.

3.2 Dataset Collection

3.2.1 Data Collection Principle

EvoBench, incorporating the two evolving pathways of LLMs, extends existing benchmarks (Mitchell et al., 2023) widely used in current detection methods, including DetectGPT (Mitchell et al., 2023), Fast-DetectGPT (Bao et al.), and ImBD (Chen et al., 2024b) to assess three key dimensions of generalization: (1) evolving LLMs, (2) domain, and (3) generation paradigms. Figure 3 illustrates our data collection pipeline to cover these three key dimensions.

Generalization to LLM Evolution. To comprehensively assess evolving, we consider 7 widely used LLM families: GPT-4o, GPT-4, Claude-Haiku, Claude-Sonnet, Qwen, LLaMA3, and LLaMA2, encompassing a total of 29 evolving versions. For updates, we focus on families of GPT-4o, GPT-4, Claude, Gemini, Qwen, and LLaMA3, tracking how detection methods perform across updates of these models. For developments, we focus on the LLaMA2 family, incorporating fine-tuning and pruning techniques. Detailed specifications of all LLMs are provided in the Appendix C.

Generalization to Domains. EvoBench includes five datasets spanning diverse tasks: XSum (Narayan et al., 2018) for (1) news articles, WritingPrompts (Fan et al., 2018) for (2) creative story writing, and PubMed (Jin et al., 2019) for (3) biomedical research question answering. Additionally, we also include two datasets: SocialMedia (Kula and Gregor, 2024) for (4) datasets on social media and PeerRead (Kang et al., 2018) for
(5) peer-reviewed academic writing.

Generalization to Generation Paradigms.
EvoBench incorporates three distinct generation paradigms: (1) continuation-based generation, including XSum, Writing, and PeerRead; (2) question-answering generation, including PubMed; (3) paraphrased generation, including SocialMedia. In the SocialMedia dataset, we specifically focus on paraphrastic rewriting of text, which presents a particularly challenging scenario for detection. By integrating these diverse paradigms, EvoBench offers a more comprehensive framework to assess the robustness of detection methods in various text generation paradigms.

In addition, EvoBench accounts for variation in text length. PubMed and SocialMedia primarily consist of shorter texts, while PeerRead includes medium-length samples, and XSum and Writing represent long-form text generation. This diversity in length further enhances the benchmark’s ability to test the generalization capabilities of detection methods under varied textual conditions.

3.2.2 Dataset Collection Process

We detail our systematic dataset construction process following the three-dimensional generalization framework. For each dataset, we first carefully selected 150 human-authored texts as reference samples. To collect AI-generated texts, we employed consistent prompting strategies across all evaluated models. Specifically, for continuation-based generation tasks (Xsum, Writing, PeerRead), we provided the prefix of human-written texts as context. For PubMed, we maintained the original question-answer format, while for SocialMedia, we implemented a paraphrasing approach where models were tasked with preserving semantic meaning while using different words. Details of prompts are shown in Appendix E.

3.3 Dataset Statistics

EvoBench consists of 7 LLM families, encompassing a total of 29 evolving versions. For each evolving LLM, we collect samples across 5 distinct datasets, with each dataset containing a balanced distribution of 150 human-authored texts and 150 machine-generated texts. In total, EvoBench comprises \( 29 \times 5 \times 150 = 21750 \) machine-generated samples, resulting in a comprehensive evaluation benchmark of 21,750 pairs of text samples.

3.4 Evaluation Metrics

To quantify the generalization ability of detection methods to evolving LLMs, we introduce the EMG \( \Phi_E \) (Evolving Model Generalization) metric, inspired by the coefficient of variation. It evaluates the consistency and trend of detection performance across evolving LLMs.

For an LLM family with \( m \) evolving versions, the widely evaluated version is selected as the base model, and the remaining \( m - 1 \) versions as evolving models. The performance (measured by AUROC \( \Phi_A \)) change \( \Delta \Phi_A^i \) between the \( i \)-th models and the base model is:

\[
\Delta \Phi_A^i = \Phi_A^i - \Phi_A^{base},
\]

where \( \Phi_A^i \) and \( \Phi_A^{base} \) is the \( \Phi_A \) of the \( i \)-th evolving model and base model, respectively.

The average performance change \( \mu_E \) of \( \Delta \Phi_A \), representing the overall performance change, is:

\[
\mu_E = \frac{1}{m-1} \sum_{i=1}^{m-1} (\Delta \Phi_A^i).
\]

The volatility \( \sigma_E \) of \( \Delta \Phi_A \) using the standard deviation is:

\[
\sigma_E = \sqrt{\frac{1}{m-1} \sum_{i=1}^{m-1} (\Delta \Phi_A^i - \mu_E)^2}.
\]

The proposed \( \Phi_E \) is defined as follows:

\[
\Phi_E = \frac{\mu_E}{\sigma_E + \lambda} \times \gamma,
\]

where \( \lambda \) is a regularization term to prevent fluctuations, and \( \gamma \) is a scaling factor. In this study, we set \( \lambda \) and \( \gamma \) as 1 and 100, respectively.

3.5 Detection Methods

We evaluate a range of supervised and zero-shot detection methods to assess their generalization ability against evolving LLMs. Additionally, we conducted repeated generation processes to ensure that output diversity does not significantly impact detection performance in Appendix B.

3.5.1 Existing Methods

For supervised detectors, we tested the GPT-2 detectors developed by OpenAI (Liu et al., 2019), the RADAR detector (Hu et al., 2023), Text Fluoroscopy (Yu et al., 2024a), and ImBD (Imitate Before Detect) (Chen et al., 2024b). For zero-shot detectors, we included Likelihood (average
log probabilities) (Gehrmann et al., 2019), LRR (a hybrid method combining log probability and log-rank) (Su et al., 2023), LogRank (mean of the log ranks sorted in descending order of probabilities) (Solaiman et al., 2019), Entropy (average token-level entropy from the predictive distribution)(Ippolito et al., 2020), DNA-GPT (Yang et al.), DetectGPT (Mitchell et al., 2023), and its enhanced variants NPR (Su et al., 2023) and Fast-DetectGPT (Bao et al.).

3.5.2 Optimizing Strategies
To address the challenges mentioned above, we introduce a baseline method for EvoBench and explore it from two perspectives.

Enhancing Zero-shot Detector via Extracting Shared Features. To enhance the detection generalization of the zero-shot method, we propose to prune the scoring model \( p_{\theta}^{\text{pruned}} \) to extract the shared features from the developments of LLMs. Take Fast-DetectGPT as an example, we define our detection metric \( d(x, p_{\theta}^{\text{pruned}}) \) based on conditional probability curvature:

\[
d(x, p_{\theta}^{\text{pruned}}) = \frac{\log p_{\theta}^{\text{pruned}}(x|x) - \tilde{\mu}}{\tilde{\sigma}},
\]

where \( \tilde{\mu} \) represents the expected log probability under the pruned model:

\[
\tilde{\mu} = \mathbb{E}_{\tilde{x} \sim p_{\theta}^{\text{pruned}}(\tilde{x}|x)}[\log p_{\theta}^{\text{pruned}}(\tilde{x}|x)],
\]

and \( \tilde{\sigma}^2 \) captures the variance of these log probabilities:

\[
\tilde{\sigma}^2 = \mathbb{E}_{\tilde{x} \sim p_{\theta}^{\text{pruned}}(\tilde{x}|x)} \left[ (\log p_{\theta}^{\text{pruned}}(\tilde{x}|x) - \tilde{\mu})^2 \right].
\]

In addition, this optimization strategy is not limited to Fast-DetectGPT; rather, it is designed for the broader class of zero-shot detectors. These detectors (Gehrmann et al., 2019) use a scoring model to extract statistical features from human- and AI-generated text, and then perform detection based on these features.

Enhancing Supervised Detectors via Optimizing Training Data. To improve the generalization of supervised detectors across evolving LLMs, we propose an incremental training strategy that focuses on optimizing the training data. Specifically, for each LLM family, we include only the initial version in the training set and evaluate the detector’s performance on subsequent, unseen versions. This approach reduces the need for frequent retraining while enhancing robustness to model updates.

4 Experimental Results

4.1 Setup
Detection Methods. To mimic real-world scenarios, we set up a black-box environment in which the detector is assumed to be unaware of the source model of the text to be detected under evaluation. We tested 14 detection methods, including 5 supervised detectors and 9 zero-shot detectors. Among them, RADAR (Hu et al., 2023) and Fast-DetectGPT (Bao et al.) are well-known methods in their respective categories. Details of experimental settings can be found in the Appendix D and Appendix F.

4.2 Main Results
To begin with, we select two leading detection methods, Fast-DetectGPT and RADAR detector, and evaluate them on our EvoBench, the results are shown in Table 1.

First, a trend is that texts generated by more advanced LLMs are harder to detect in the same LLM family. For instance, comparing Claude-3-Haiku\(^4\) and Claude-3-Opus\(^5\), the latter demonstrates more advanced capabilities. When detecting these two LLMs, Fast-DetectGPT and RADAR showed decreased detection AUROC of 3.58% and 2.77%, respectively, as shown in Table 1. Similarly, comparing detecting SocialMedia datasets generated by GPT-4o-mini with GPT-4o, two detectors showed a decline in detection AUROC of 3.22% and 6.69%, respectively.

Our results also reveal an intriguing pattern: existing detection methods often over-optimize for specific LLM versions, rather than maintaining robust generalization capabilities across different evolving LLMs. Specifically, detectors with high AUROC on widely used LLM families often show performance degradation with newer versions of the same family. For example, Fast-DetectGPT performed excellently on the GPT-4o-05-13 version, achieving an average detection accuracy of 0.8003 on EvoBench, but dropped to 0.7422 when facing the GPT-4o-latest version. Similar patterns were observed in other widely used LLM families, such as GPT-4 and Claude-Sonnet. In contrast, when evaluating less widely used LLMs like LLaMA3, detectors tend to demonstrate better generalization capabilities across evolving LLMs. For example, the detection performance of the RADAR remained

\(^4\)claude-3-haiku-20240307
\(^5\)claude-3-opus-20240229
<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th colspan="6">Fast-DetectGPT</th>
    <th colspan="6">RADAR</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Version Name</th>
    <th>This</th>
    <th>Writing</th>
    <th>PubMed</th>
    <th>SocialMedia</th>
    <th>PeerRead</th>
    <th>Avg.</th>
    <th>Xsum</th>
    <th>Writing</th>
    <th>PubMed</th>
    <th>SocialMedia</th>
    <th>PeerRead</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="5">GPT-4o</td>
    <td>2024-05-13</td>
    <td>0.98</td><td>0.71</td><td>0.75</td><td>0.48</td><td>0.85</td><td>0.80</td><td>0.99</td><td>0.83</td><td>0.63</td><td>0.60</td><td>0.66</td><td>0.73</td><td></td>
  </tr>
  <tr>
    <td>2024-06-04</td>
    <td>0.97</td><td>0.84</td><td>0.79</td><td>0.90</td><td>0.87</td><td>0.97</td><td>0.99</td><td>0.86</td><td>0.65</td><td>0.67</td><td>0.69</td><td>0.70</td><td></td>
  </tr>
  <tr>
    <td>(-2.75%) (-2.41%)(-1.28%) (-1.56 %) (-3.09 %)
      (-2.57%) (-1.61%)(+4.48%) (+3.49%) (+7.83%)
      (+5.15%)(+1.14%)(+2.12%)(+8.75%)(+1.10%)</td>
    <td colspan="7"></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>(-2.34%)(-2.12%)(-3.48%)(-6.34%)(+3.48%)(+2.43%)(+1.36%)(+5.98%)(+0.50%)</td>
  </tr>
  <tr>
    <td>2024-11-20</td>
    <td>0.72</td><td>0.90</td><td>0.70</td><td>0.57</td><td>0.78</td><td>0.73</td><td>0.99</td><td>0.72</td><td>0.79</td><td>0.62</td><td>0.66</td><td>0.75</td><td></td>
  </tr>
  <tr>
    <td>Latest</td>
    <td>0.70</td><td>0.91</td><td>0.70</td><td>0.58</td><td>0.82</td><td>0.74</td><td>0.99</td><td>0.75</td><td>0.80</td><td>0.64</td><td>0.67</td><td>0.77</td><td></td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>(-28.0%)(-12.8%)(-34.1%)(-12.0%)(-46.9%)
      (-19.4%)(-)11.7%(+)10.6%)(+42.6%)(+44.2%)(-14%;)(+132.4%)(-21%)</td>
    <td colspan="7"></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>2023-04-13</td>
    <td>0.93</td><td>0.98</td><td>0.76</td><td>0.49</td><td>0.75</td><td>0.78</td><td>0.89</td><td>0.87</td><td>0.90</td><td>0.90</td><td>0.86</td><td>0.85</td><td>0.86</td>
  </tr>
  <tr>
    <td>2023-11-06</td>
    <td>0.92</td><td>0.93</td><td>0.73</td><td>0.55</td><td>0.83</td><td>0.79</td><td>0.99</td><td>0.79</td><td>0.80</td><td>0.66</td><td>0.60</td><td>0.77</td><td></td>
  </tr>
  <tr>
    <td>GPT-4-large</td>
    <td>(-0.85%)(-3.47%)(-2.24%)(-1.43%)
      (1.43%) (7.21%)(-0.49%)(-1.49%)(-1.81%)
      (4.89%)(-7.14%)(-12.3%)(-5.88%)</td>
    <td colspan="7"></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>2024-01-25</td>
    <td>0.88</td><td>0.91</td><td>0.72</td><td>0.58</td><td>0.79</td><td>0.76</td><td>0.99</td><td>0.78</td><td>0.88</td><td>0.60</td><td>0.70</td><td>0.80</td><td>0.76</td>
  </tr>
  <tr>
    <td>2024-04-09</td>
    <td>0.83</td><td>0.91</td><td>0.72</td><td>0.58</td><td>0.79</td><td>0.77</td><td>0.99</td><td>0.79</td><td>0.82</td><td>0.60</td><td>0.60</td><td>0.76</td><td></td>
  </tr>
  <tr>
    <td>Claude>Sonnset</td>
    <td>2024-02-29</td>
    <td>0.95</td><td>0.98</td><td>0.81</td><td>0.48</td><td>0.93</td><td>0.98</td><td>0.97</td><td>0.88</td><td>0.88</td><td>0.41</td><td>0.61</td><td>0.80</td><td></td>
  </tr>
  <tr>
    <td>2024-06-20</td>
    <td>0.87</td><td>0.94</td><td>0.87</td><td>0.66</td><td>0.92</td><td>0.90</td><td>0.89</td><td>0.90</td><td>0.88</td><td>0.69</td><td>0.70</td><td>0.82</td><td>0.70</td>
  </tr>
  <tr>
    <td>Claude-Hauku</td>
    <td>2024-02-29</td>
    <td>0.81</td><td>0.92</td><td>0.64</td><td>0.59</td><td>0.94</td><td>0.95</td><td>0.93</td><td>0.84</td><td>0.70</td><td>0.68</td><td>0.82</td><td>0.84</td><td></td>
  </tr>
  <tr>
    <td>Claude>Opus</td>
    <td>2024-02-29</td>
    <td>(-13.5%)(-10.9%)(-11.6%)(-20.7%)
      (+8.5%)(-14.8%)(+12.0%)(-11.9%)(+14.1%)
      (+18.8%)(-13.4%)(-14.1%)(-24.0%)</td>
    <td colspan="7"></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen1.1-TB, Qwen2.7B, Qwen5.7B</td>
    <td></td>
    <td>0.92</td><td>0.91</td><td>0.66</td><td>0.49</td><td>0.91</td><td>0.90</td><td>0.93</td><td>0.91</td><td>0.66</td><td>0.85</td><td>0.85</td><td>0.76</td><td></td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>0.99</td><td>1.00</td><td>0.74</td><td>0.46</td><td>0.89</td><td>0.82</td><td>0.89</td><td>0.87</td><td>0.82</td><td>0.52</td><td>0.54</td><td>0.73</td><td></td>
  </tr>
  <tr>
    <td>LLaMA1</td>
    <td>LLaMA3.1TG, LLaMA3.2B</td>
    <td>(-1.0%)(+2.9%)(-2.3%)(+5.3%)
      (+13.6%)(+18.0%)(-12.0%)(+3.1%)(+2.2%)
      (+10.7%)(+13.0%)(+6.7%)</td>
    <td colspan="7"></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LLaMA2.7B-chu4</td>
    <td>LLaMA-2.7B-pruned</td>
    <td>(-6.2%)(-4.18%)(-8.53%)(-10.9")
      (-11.8%)(-13.75%)(-13%)(-6.3%)(-11.9%)
      (-11.05%)(-8.76%)(-7.6%)(-3.09%)</td>
    <td colspan="7"></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Vicuna-1.5-TB, WizardMath-7B</td>
    <td></td>
    <td>(-3.6%)(-0.6%)(-4.7%)(-1%) (+1%)
      (+1.9%)(-10.5%)(-7.2%)(+1.7%)
      (+1.7%)(+8.95%)(-1.18%)(-10.8%)</td>
    <td colspan="7"></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Sheared-LLaMA1.3-TB, Sheared-LLaMA2.7B, Sheared-LLaMA2.7B-pruned</td>
    <td></td>
    <td>(+2%)(+8.4%)(-4.5%)(-13.6%)(-8.95%)(-1.7%)
      (+15.0%)(-15.8%)(+4.91%)(-15.1%)(-8.05%)(-9.4%)(+10.6%)</td>
    <td colspan="7"></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 1: The detection performance (measured in AUROC) of two leading detection methods, Fast-DetectGPT and RADAR, on EvoBench. For each LLM family, we set the LLMs that are widely evaluated in text generation detection benchmarks as anchor points. ‘Latest’ refers to the LLM currently used in the web version of GPT-4o.

stable across different versions of the LLaMA3 family, with AUROC scores across five datasets ranging from 0.7989 to 0.8348.

This suggests that the essence of generalization lies in the overfitting problem. Current detectors are often optimized to achieve superior performance on existing benchmarks, which leads to overfitting to a specific LLM version rather than being truly effective. This also reveals the vulnerability of current detection methods across evolving LLMs. From a practical perspective, a key priority for future research lies in developing more adaptable detection frameworks that can simultaneously preserve detection performance while demonstrating robust generalization across dataset domains and evolving LLMs.

4.2.1 Detection Generalization across Evolving LLMs

To visually demonstrate the generalization ability of detection methods across evolving LLMs, we evaluate the 14 detection methods using the EMG metric, with results shown in Table 2.

Table 2 demonstrates that no method can maintain stable performance across all evolving LLMs. Specifically, detectors like ImBD and Text-Fluoroscopy perform well on complex models like GPT-4o and GPT-4. Meanwhile, detectors like entropy and RADAR exhibit better generalizations of newer model families, including Qwen, LLaMA3, and the developments of LLaMA2. This variation in performance across different detectors suggests that improving generalization might require adopting strategies that combine the strengths of different approaches, which may help improve the
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="5">Updates</th>
    <th colspan="2">Developments</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>GPT-4o</th>
    <th>GPT-4</th>
    <th>Claude-Sonnet</th>
    <th>Claude-haiku</th>
    <th>Qwen</th>
    <th>LLaMA3</th>
    <th>Fine-tuning</th>
    <th>Pruning</th>
  </tr>
  <tr>
    <td colspan="9">Supervised Detectors</td>
  </tr>
  <tr>
    <td>RoBERTa-base</td>
    <td>-2.252</td>
    <td>-4.593</td>
    <td>-8.325</td>
    <td>-15.95</td>
    <td>+5.862</td>
    <td>+1.457</td>
    <td>-2.378</td>
    <td>+6.871</td>
    <td>-2.413</td>
  </tr>
  <tr>
    <td>RoBERTa-large</td>
    <td>-1.878</td>
    <td>-5.577</td>
    <td>-10.06</td>
    <td>-14.63</td>
    <td>+4.646</td>
    <td>+3.448</td>
    <td>-0.725</td>
    <td>+12.54</td>
    <td>-1.530</td>
  </tr>
  <tr>
    <td>RADAR</td>
    <td>-0.537</td>
    <td>-5.775</td>
    <td>-4.934</td>
    <td>-2.394</td>
    <td>+2.053</td>
    <td>+1.372</td>
    <td>+11.96</td>
    <td>+7.302</td>
    <td>+1.131</td>
  </tr>
  <tr>
    <td>Text-Fluoroscopy</td>
    <td>-0.493</td>
    <td>-0.655</td>
    <td>+0.902</td>
    <td>-0.123</td>
    <td>-5.241</td>
    <td>+1.895</td>
    <td>-11.03</td>
    <td>-15.77</td>
    <td>-3.815</td>
  </tr>
  <tr>
    <td>Imitate Before Detect</td>
    <td>+0.440</td>
    <td>+2.397</td>
    <td>-1.572</td>
    <td>-4.445</td>
    <td>+0.023</td>
    <td>-0.054</td>
    <td>+8.942</td>
    <td>-9.842</td>
    <td>-0.513</td>
  </tr>
  <tr>
    <td colspan="9">Zero-Shot Detectors</td>
  </tr>
  <tr>
    <td>Likelihood</td>
    <td>-0.873</td>
    <td>-0.047</td>
    <td>-3.810</td>
    <td>-8.280</td>
    <td>-1.174</td>
    <td>+0.439</td>
    <td>-6.991</td>
    <td>-32.99</td>
    <td>-6.717</td>
  </tr>
  <tr>
    <td>Rank</td>
    <td>-2.127</td>
    <td>+1.071</td>
    <td>-0.178</td>
    <td>-12.09</td>
    <td>+4.249</td>
    <td>+1.023</td>
    <td>-9.661</td>
    <td>-7.109</td>
    <td>-3.103</td>
  </tr>
  <tr>
    <td>LogRank</td>
    <td>-1.316</td>
    <td>-0.106</td>
    <td>-4.272</td>
    <td>-9.443</td>
    <td>-0.608</td>
    <td>+0.837</td>
    <td>-6.851</td>
    <td>-30.14</td>
    <td>-6.488</td>
  </tr>
  <tr>
    <td>Entropy</td>
    <td>-3.202</td>
    <td>+0.416</td>
    <td>+0.602</td>
    <td>-3.411</td>
    <td>+4.138</td>
    <td>+2.145</td>
    <td>+5.150</td>
    <td>+25.12</td>
    <td>+3.869</td>
  </tr>
  <tr>
    <td>LRR</td>
    <td>-2.511</td>
    <td>-0.075</td>
    <td>-5.864</td>
    <td>-14.26</td>
    <td>+2.629</td>
    <td>+1.911</td>
    <td>-6.964</td>
    <td>-17.45</td>
    <td>-5.325</td>
  </tr>
  <tr>
    <td>NPR</td>
    <td>-2.846</td>
    <td>+7.436</td>
    <td>-2.237</td>
    <td>-15.26</td>
    <td>+7.132</td>
    <td>+0.866</td>
    <td>-8.063</td>
    <td>-31.18</td>
    <td>-5.520</td>
  </tr>
  <tr>
    <td>DNA-GPT</td>
    <td>-3.805</td>
    <td>+1.249</td>
    <td>-4.621</td>
    <td>-12.26</td>
    <td>-1.332</td>
    <td>+1.337</td>
    <td>-6.355</td>
    <td>-36.11</td>
    <td>-7.738</td>
  </tr>
  <tr>
    <td>DetectGPT</td>
    <td>-4.095</td>
    <td>+10.04</td>
    <td>+2.258</td>
    <td>-11.09</td>
    <td>+2.357</td>
    <td>-0.336</td>
    <td>-5.442</td>
    <td>-20.18</td>
    <td>-3.312</td>
  </tr>
  <tr>
    <td>Fast-DetectGPT</td>
    <td>-3.558</td>
    <td>-0.271</td>
    <td>-4.115</td>
    <td>-13.16</td>
    <td>+2.652</td>
    <td>+1.094</td>
    <td>-5.095</td>
    <td>-27.94</td>
    <td>-6.302</td>
  </tr>
</table>

Table 2: EMG performance of 14 detection methods on EvoBench. Red indicates a negative EMG, signifying a decrease in AUROC when facing evolving LLMs, while a larger EMG value reflects a better generalization of the detection method.

<table>
  <tr>
    <th rowspan="2">Developments of LLaMA-2</th>
    <th rowspan="2">Detection Methods</th>
    <th colspan="4">LLaMA-2-7b (baseline)</th>
    <th colspan="4">Sheared-LLaMA-1.3B</th>
    <th colspan="4">Sheared-LLaMA-2.7B</th>
    <th colspan="4">Wanda-LLaMA</th>
  </tr>
  <tr>
    <th>Likelihood</th>
    <th>Rank</th>
    <th>LogRank</th>
    <th>Entropy</th>
    <th>Likelihood</th>
    <th>Rank</th>
    <th>LogRank</th>
    <th>Entropy</th>
    <th>Likelihood</th>
    <th>Rank</th>
    <th>LogRank</th>
    <th>Entropy</th>
  </tr>
  <tr>
    <td rowspan="2">Fine-tuning</td>
    <td>Likelihood</td>
    <td>0.8616</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.8515</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.8529</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.8593</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Rank</td>
    <td>0.6288</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6781</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6469</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6163</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Pruning</td>
    <td>Likelihood</td>
    <td>0.8762</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.8551</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.8632</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.8714</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LogRank</td>
    <td>0.3908</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.3362</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.3524</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.3881</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Fine-tuning and Pruning</td>
    <td>Likelihood</td>
    <td>0.9350</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.9152</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.9376</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.9339</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Average</td>
    <td>0.7384</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.7272</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.7306</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.7338</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Fine-tuning</td>
    <td>Likelihood</td>
    <td>0.3921</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.7948</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.5378</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.4633</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Rank</td>
    <td>0.6583</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.7950</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.7179</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6113</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Pruning</td>
    <td>Likelihood</td>
    <td>0.4273</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.8209</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.5806</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.5046</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LogRank</td>
    <td>0.7343</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.4614</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.5407</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6871</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Fine-tuning and Pruning</td>
    <td>Likelihood</td>
    <td>0.6847</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.8913</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.8066</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.7518</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Average</td>
    <td>0.5793</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.7526</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6367</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6036</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Fine-tuning</td>
    <td>Likelihood</td>
    <td>0.6269</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.8232</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6954</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6613</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Rank</td>
    <td>0.6436</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.7366</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6824</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6138</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Pruning</td>
    <td>Likelihood</td>
    <td>0.6518</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.8380</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.7219</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6880</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LogRank</td>
    <td>0.5625</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.3988</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.4465</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.5376</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Fine-tuning and Pruning</td>
    <td>Likelihood</td>
    <td>0.8099</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.9033</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.8721</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.8429</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Average</td>
    <td>0.6589</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.7400</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6837</td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.6687</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 3: Average Detection performance across LLaMA-2 developments, including fine-tuning and pruning versions, using five detection methods with different scoring models.

adaptability of the detector and reduce sensitivity to specific LLM evolution trends.

4.2.2 Performance of Optimizing Strategies
Extracting Shared Features To improve the generalization of current detection methods across evolving LLMs, we have preliminarily explored two possible approaches, with results presented in Table 3.

For zero-shot detection methods, we propose a general optimization strategy called extracting shared features, which is not limited to Fast-DetectGPT but is designed for the broader class of zero-shot detectors. Specifically, we prune the scoring model to preserve the discriminative features that are shared across different generative models, aiming to improve generalization against model developments such as fine-tuning and pruning. To validate the effectiveness of this strategy more broadly, we conduct experiments across multiple detectors, including Likelihood, Rank, LogRank, and Entropy. Additionally, we examine how different pruning strategies for the scoring model impact detection performance. We use LLaMA-2-7b as the baseline and evaluate three pruned versions: Sheared-LLaMA-1.3B, Sheared-LLaMA-2.7B, and Wanda-LLaMA.
<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th colspan="3">Text Fluoroscopy</th>
  </tr>
  <tr>
    <th>Version Time/Version Name</th>
    <th>Original Dataset</th>
    <th>Adding Initial Version</th>
  </tr>
  <tr>
    <td rowspan="4">GPT-4o</td>
    <td>GPT-4o-mini</td>
    <td>0.8373</td>
    <td><b>0.8464</b></td>
  </tr>
  <tr>
    <td>2024-08-06</td>
    <td>0.8240</td>
    <td><b>0.8327</b></td>
  </tr>
  <tr>
    <td>2024-11-20</td>
    <td>0.7910</td>
    <td><b>0.8035</b></td>
  </tr>
  <tr>
    <td>Latest</td>
    <td>0.7986</td>
    <td><b>0.8057</b></td>
  </tr>
  <tr>
    <td rowspan="3">GPT-4</td>
    <td>2023-11-06</td>
    <td>0.8117</td>
    <td><b>0.8633</b></td>
  </tr>
  <tr>
    <td>2024-01-25</td>
    <td>0.8061</td>
    <td><b>0.8549</b></td>
  </tr>
  <tr>
    <td>2024-04-09</td>
    <td>0.8131</td>
    <td><b>0.8584</b></td>
  </tr>
  <tr>
    <td rowspan="2">Claude-3.5-Sonnet</td>
    <td>2024-06-20</td>
    <td>0.8699</td>
    <td><b>0.8720</b></td>
  </tr>
  <tr>
    <td>2024-10-22</td>
    <td>0.8218</td>
    <td><b>0.8401</b></td>
  </tr>
</table>

Table 4: Average Detection Performance across updates of three LLM families with and without the initial version in the training dataset under Text Fluoroscopy evaluation.

We assess the average detection AUROC of each configuration under two types of model development—fine-tuning and pruning. The results, shown in the table below, demonstrate that using Sheared-LLaMA-1.3B as the scoring model consistently leads to higher detection performance across all detectors. This suggests that the pruned model retains key shared features, making it a strong general-purpose scoring model for detecting AI-generated text across a variety of model variants.

Including the Initial Version For supervised detectors, we introduce a training set-based generalization strategy that can be applied beyond a single method. Rather than tailoring our approach to a specific architecture, we propose a practical and broadly applicable training strategy, including the initial version of an LLM family in the training data. This enables the detector to generalize more effectively to subsequent, unseen updates of the same LLM family, without requiring frequent retraining.

To validate this strategy, we extended our experiments to include both Text Fluoroscopy (Yu et al., 2024a) and the Roberta-based detector (Liu et al., 2019). Details of experiment settings are shown in Appendix D. In each case, we incorporated the initial version of an LLM family into the training dataset and evaluated performance on later versions of that family. For example, for the GPT-4o family, we selected GPT-4o (2024-05-13) as the initial version and added its data to the training set. We then tested the detector’s generalization on multiple subsequent versions: GPT-4o-mini (2024-07-18), GPT-4o (2024-08-06), GPT-4o (2024-11-20), and GPT-4o (latest). This setup allowed us to evaluate the model’s ability to generalize across updates directly.

As shown in the Table 4, both supervised detectors demonstrated improved generalization after including the initial LLM version in training. Notably, for the GPT-4 family, adding data from GPT-4 (2023-06-13) led to substantial gains. For the GPT-4 family, after only adding data from the initial version (GPT-4-2023-06-13), the performance on GPT-4-2023-11-06 increased from 0.8117 to 0.8633, and on GPT-4-2024-01-25, it improved from 0.8061 to 0.8549. Therefore, when developing detectors, it is essential to consider model updates. To address this, it is crucial to at least include the initial version of each LLM family in the training data. This can improve performance on future unseen updates of the same LLM family without the need for frequent retraining.

5 Conclusion

In this paper, we introduce EvoBench, a novel benchmark for evaluating the generalization of LLM-generated text detection methods across evolving LLMs. EvoBench defines two key dimensions of LLM evolution: (1) updates made by LLM publishers over time and (2) developments carried out by developers, ensuring a comprehensive understanding of how evolving LLMs impact detection performance. To quantify this evolving dimension generalization, we propose the EMG (Evolving Model Generalization) metric.

We evaluate 14 widely used detection methods using EvoBench, revealing their vulnerabilities when facing evolving LLMs. In response, to improve the generalization of zero-shot methods across developing LLMs, we propose two strategies: for zero-shot methods, we suggest pruning the scoring model to extract shared features across LLM developments; for supervised methods, we recommend augmenting training data with data from LLM updates. Our benchmark represents a key step towards bringing detection methods into real-world scenarios for evolving LLMs. We also envision continuously updating this benchmark to cover a broader range of evolving LLMs, enabling more comprehensive evaluations across various domains and evolving LLMs.

Acknowledgments

This work was partially supported by the National Natural Science Foundation of China under Grant 62472398 and U2336206, and by the Shanghai Artificial Intelligence Laboratory.
Limitations

Although we have thoroughly considered the three key dimensions of generalization toward real-world scenarios, certain limitations remain that warrant further exploration in future research. First, it is important to emphasize that EvoBench is not intended to replace existing benchmarks for evaluating the generalization of LLM-generated text detection methods but rather to complement them. Therefore, EvoBench primarily focuses on three key dimensions in real-world scenarios: domains, generation strategies, and evolving models. We have not covered other aspects of generalization, such as language. A potential direction for future research is to extend EvoBench to incorporate additional dimensions of generalization.

Second, robustness has not been addressed in this study, as our focus was mainly on detection generalization. However, EvoBench could serve as a foundation for inspiring current robustness research, as robustness also involves tasks such as rewriting text using different LLMs.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shayamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Alim Al Ayub Ahmed, Ayman Aljabouh, Praveen Kumar Donepudi, and Myung Suh Choi. 2021. Detecting fake news using machine learning: A systematic literature review. arXiv preprint arXiv:2102.04458.

AI Anthropic. 2024. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card, 3(6).

Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. In The Twelfth International Conference on Learning Representations.

Amrita Bhattacharjee, Tharindu Kumarage, Raha Moraffah, and Huan Liu. 2023. ConDA: Contrastive domain adaptation for AI-generated text detection. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 598–610, Nusa Dua, Bali. Association for Computational Linguistics.

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Halahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR.

Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Canyu Chen and Kai Shu. 2023. Combating misinformation in the age of llms: Opportunities and challenges. arXiv preprint arXiv: 2311.05656.

Hong Chen, Chengtao Lv, Liang Ding, Haotong Qin, Xiabin Zhou, Yifu Ding, Xuebo Liu, Min Zhang, Jinyang Guo, Xianglong Liu, and Dacheng Tao. 2024a. DB-LLM: Accurate dual-binarization for efficient LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, pages 8719–8730, Bangkok, Thailand. Association for Computational Linguistics.

Jiaqi Chen, Xiaoye Zhu, Tianyang Liu, Ying Chen, Xin-hui Chen, Yiwen Yuan, Chak Tou Leong, Zuchao Li, Tang Long, Lei Zhang, et al. 2024b. Imitate before detect: Aligning machine stylistic preference for machine-revised text detection. arXiv preprint arXiv:2412.10432.

Jon Christian. 2023. Cnet secretly used ai on articles that didn’t disclose that fact, staff say. Futurism, January.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115.

DaYou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. 2024. BitDistiller: Unleashing the potential of sub-4-bit LLMs via self-distillation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 102–116, Bangkok, Thailand. Association for Computational Linguistics.

Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898.

Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019. Gltr: Statistical detection and visualization of generated text. In Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics (ACL).
Josh A Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. 2023. Generative language models and automated influence operations: Emerging threats and potential mitigations. arXiv preprint arXiv:2301.04246.

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644.

Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597.

Hanxi Guo, Siyuan Cheng, Xiaolong Jin, Zhuo Zhang, Kaiyuan Zhang, Guanhong Tao, Guangyu Shen, and Xiayu Zhang. 2024. Biscope: Ai-generated text detection by checking memorization of preceding tokens. Advances in Neural Information Processing Systems, 37:104065–104090.

Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2024. Mgtbench: Benchmarking machine-generated text detection. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 2251–2265.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.

Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2023. Radar: Robust ai-text detection via adversarial learning. Advances in Neural Information Processing Systems, 36:15077–15095.

Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808–1822.

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577.

Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylem, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1647–1661, New Orleans, Louisiana. Association for Computational Linguistics.

Davinder Kaur, Suleyman Uslu, Kaley J Rittichier, and Arjan Durresi. 2022. Trustworthy artificial intelligence: a review. ACM Computing Surveys (CSUR), 55(2):1–38.

Tobias Kerner. 2024. Domain-specific pretraining of language models: A comparative study in the medical field. arXiv preprint arXiv:2407.14076.

Sanghoon Kim, Dahyun Kim, Chanjun Park, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim. 2024. SOLAR 10.7B: Scaling large language models with simple yet effective depth upscaling. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pages 23–35, Mexico City, Mexico. Association for Computational Linguistics.

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

Sebastian Kula and Michal Gregor. 2024. Multilingual models for check-worthy social media posts detection. arXiv preprint arXiv:2408.06737.

Wai-Chung Kwan, Xingshan Zeng, Yufei Wang, Yusen Sun, Liangyou Li, Yuxin Jiang, Lifeng Shang, Qun Liu, and Kam-Fai Wong. 2024. M4LE: A multi-ability multi-range multi-task multi-domain long-context evaluation benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15568–15592, Bangkok, Thailand. Association for Computational Linguistics.

Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. 2023. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, pages 3637–3647.

Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. 2021. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370–403.

Yinhai Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Muneer M Alshater. 2022. Exploring the role of artificial intelligence in enhancing academic performance: A case study of chatgpt. Available at SSRN.

Dominik Macko, Robert Moro, Adaku Uchendu, Jason Lucas, Michiharu Yamashita, Matúš Pikuliak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, et al. 2023. Multitude: Large-scale multilingual machine-generated text detection benchmark. In Proceedings
of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9960–9987.

Sourab Mangulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft.

Alex Mitchell. 2022. Professor catches student cheating with chatgptl: ‘i feel abject terror’. Retrieved from https://nypost.com/2022/12/26/students-using-chatgpt-to-cheat-professor-warns/. Accessed on February 17, 2023.

Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In International Conference on Machine Learning, pages 24950–24962. PMLR.

Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807.

OpenAI. 2022a. ChatGPT. https://chat.openai.com/.

OpenAI. 2022b. Introducing chatgpt.

OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774.

Mary Louise Kelly Patrick Wood. 2023. ‘everybody is cheating’: Why this teacher has adopted an open chatgpt policy. Retrieved from https://www.npr.org/2023/01/26/1151499213/chatgpt-ai-education-cheating-classroom-wharton-school. Accessed on January 26, 2023.

Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203.

Jinyan Su, Terry Zhuo, Di Wang, and Preslav Nakov. 2023. Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12395–12412.

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations.

Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou. 2024. A survey on self-evolution of large language models. arXiv preprint arXiv:2404.14387.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.

Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Pucetti, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024a. M4GT-bench: Evaluation benchmark for black-box machine-generated text detection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3964–3992, Bangkok, Thailand. Association for Computational Linguistics.

Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, et al. 2024b. M4: Multi-generator, multi-domain, and multi-lingual black-box machine-generated text detection. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1369–1407.

Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations.

Xianjun Yang, Wei Cheng, Yue Wu, Linda Ruth Petzold, William Yang Wang, and Haifeng Chen. Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text. In The Twelfth International Conference on Learning Representations.

Xiao Yu, Kejiang Chen, Qi Yang, Weiming Zhang, and Nenghai Yu. 2024a. Text fluoroscopy: Detecting llm-generated text through intrinsic features. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15838–15846.

Xiao Yu, Yuan Qi, Kejiang Chen, Guoqiang Chen, Xi Yang, Pengyuan Zhu, Xiuxwei Shang, Weiming Zhang, and Nenghai Yu. 2024b. Dpic: Decoupling prompt and intrinsic characteristics for llm generated text detection. Advances in Neural Information Processing Systems, 37:16194–16212.
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: story writing with large language models. In 27th International Conference on Intelligent User Interfaces, pages 841–852.

Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyuan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372.

A Examples of Texts Generated by Different LLMs

In this section, we provided examples of responses from different updates of the same LLMs to the same question, offering a visual representation of how changes in versions lead to variations in the model’s output in Table 5 and Table 6.

B Additional Experiment

Additionally, to eliminate the influence of output diversity on detection performance, we first generated 150 texts from each model for every dataset, resulting in a total of 300 texts. Additionally, we conducted repeated experiments with several models to ensure that output diversity does not significantly impact changes in detection difficulty. By doing so, we ensure that the variations in detection difficulty primarily stem from model evolution itself, rather than biases introduced by the diversity of the generated texts. The results of this experimental process are presented in Table 7, where we highlight the potential range of biases caused by the diversity of generated texts. This helps further to understand the relationship between model evolution and detection performance.

C Details of Evolving LLMs

The details of evolving LLMs are shown in Table 8.

D Details of Optimizing Strategies

In this section, we provide the details of two optimizing strategies. For the zero-shot detection method, we chose to optimize the Fast-DetectGPT detector. We use the pruned model 'princeton-nlp/Sheared-LLaMA-2.7B-Pruned' to replace the scoring model 'meta-llama/Llama-2-7b-hf' model as the scoring model. For the supervised detector, we chose to optimize the Text Fluoroscopy. Following Text Fluoroscopy, we used the first 200 entries of the open-source Human-ChatGPT Comparison Corpus (HC3) (Guo et al., 2023) dataset collected by previous researchers as a training set. The ratio for splitting the training and validation is 8 : 1. We use a specific LLM of the LLM families to regenerate machine text in the training dataset and then use the new dataset and the original dataset to retrain the detector. Specifically, we regenerated this part of the training set using GPT-4-0613 and tested the results on the GPT-4 family. For the GPT-4o family, we used GPT-4o-2024-05-13, and for the
Table 5: An example of content generated by different versions of the GPT-4o and GPT-4o-mini

<table>
  <tr>
    <th>Does airway surgery lower serum lipid levels in obstructive sleep apnea patients?</th>
  </tr>
  <tr>
    <th>Human Answer</th>
    <td>Surgery may improve blood lipid levels in patients with OSA, especially in patients with preoperative dyslipidemia, potentially yielding a major benefit in metabolism and cardiovascular sequelae. Prospective studies should examine this ...</td>
  </tr>
  <tr>
    <th>GPT-4o<br>2024-05-13</th>
    <td>Airway surgery for obstructive sleep apnea (OSA) has been shown to reduce serum lipid levels in some patients. The improvement is likely due to better oxygenation and reduced inflammation following successful surgery, ...</td>
  </tr>
  <tr>
    <th>GPT-4o<br>2024-08-06</th>
    <td>Airway surgery can potentially lower serum lipid levels in obstructive sleep apnea (OSA) patients. Surgical interventions that alleviate OSA symptoms may improve metabolic profiles, including lipid levels, due to reduced hypoxia and ...</td>
  </tr>
  <tr>
    <th>GPT-4o<br>2024-11-20</th>
    <td>Yes, airway surgery in obstructive sleep apnea (OSA) patients may lower serum lipid levels. Studies suggest that by improving oxygenation, reducing inflammation, and addressing the metabolic impacts of OSA, surgeries like ...</td>
  </tr>
  <tr>
    <th>GPT-4o<br>Latest</th>
    <td>Yes, airway surgery for obstructive sleep apnea (OSA) can lower serum lipid levels. By improving airway patency and reducing hypoxia, surgery may enhance lipid metabolism and decrease systemic inflammation, which are linked ...</td>
  </tr>
  <tr>
    <th>GPT-4o-mini<br>2024-07-18</th>
    <td>Airway surgery in obstructive sleep apnea patients may lead to improvements in overall health and potentially lower serum lipid levels. However, the extent of lipid level changes can vary among individuals and ...</td>
  </tr>
</table>

Table 6: An example of content generated by different versions of the Claude-Haiku and Claude-Opus

<table>
  <tr>
    <th>Neck pain treatment with acupuncture: does the number of needles matter?</th>
  </tr>
  <tr>
    <th>Human Answer</th>
    <td>For this pathology, the number of needles, 5 or 11, seems not to be an important variable in determining the therapeutic effect when the time of stimulation is the same in the two groups.</td>
  </tr>
  <tr>
    <th>Claude-Haiku<br>2024-03-07</th>
    <td>Acupuncture for neck pain can be effective, but the number of needles used may not be the most critical factor. The placement and selection of acupuncture points tailored to the individual’s condition are more important ...</td>
  </tr>
  <tr>
    <th>Claude-Haiku<br>2024-10-22</th>
    <td>Research suggests that the number of acupuncture needles may not significantly impact neck pain treatment. Effectiveness depends more on precise needle placement, practitioner skill, and individual patient response. Further studies are needed to definitively determine ...</td>
  </tr>
  <tr>
    <th>Claude-Opus<br>2024-02-29</th>
    <td>The number of needles used in acupuncture for neck pain treatment may impact its effectiveness. However, factors such as needle placement, depth of insertion, and stimulation technique are likely more important than the specific number ...</td>
  </tr>
</table>

Claude-Haiku family, we used Claude-3-sonnet-20240229. we use gte-Qwen1.5-7B-instruct\footnote{https://huggingface.co/Alibaba-NLP/gte-Qwen1.5-7B-instruct} as the encoder and the classifier consists of three fully connected layers with Tanh function. The dimensions of the intermediate layers in the classifier are 1024 and 512, respectively. The batch size is set to 16, and Adam (Kingma and Ba, 2014) optimizer is employed with an initial learning rate of \(3e - 3\).

E Data Collection.

In this section, we describe the process of data collection. Most of these datasets were generated following the dataset construction methodology
<table>
  <tr>
    <th></th>
    <th>Fast-DetectGPT</th>
    <th>RADAR</th>
    <th>Text-Fluoroscopy</th>
    <th>Imitate-Review-base</th>
    <th>RodBERT-base</th>
    <th>RodBERT-large</th>
    <th>Likelihood</th>
    <th>Rank</th>
    <th>LogRank</th>
    <th>Entropy</th>
    <th>DetectGPT</th>
    <th>LRB</th>
    <th>NPR</th>
    <th>UNA-LPI</th>
  </tr>
  <tr>
    <td>GPT-3.5-turbo-2024-01-23</td>
    <td>0.8420</td>
    <td>0.8319</td>
    <td>0.8917</td>
    <td>0.9011</td>
    <td>0.6629</td>
    <td>0.6696</td>
    <td>0.7822</td>
    <td>0.6511</td>
    <td>0.7787</td>
    <td>0.4118</td>
    <td>0.5008</td>
    <td>0.7120</td>
    <td>0.6276</td>
    <td>0.8266</td>
  </tr>
  <tr>
    <td>(+1.3%)</td>
    <td>(-0.3%)</td>
    <td>(+6.1%)</td>
    <td>(+1.8%)</td>
    <td>(+0.8%)</td>
    <td>(-3.6%)</td>
    <td>(-2.3%)</td>
    <td>(-1.2%)</td>
    <td>(-1.2%)</td>
    <td>(-1.2%)</td>
    <td>(-3.6%)</td>
    <td>(-3.6%)</td>
    <td>(-2.3%)</td>
    <td>(-2.3%)</td>
    <td>(-1.2%)</td>
  </tr>
  <tr>
    <td>GPT-4-mono-2024-01-18</td>
    <td>0.8079</td>
    <td>0.7976</td>
    <td>0.8777</td>
    <td>0.8806</td>
    <td>0.5596</td>
    <td>0.5827</td>
    <td>0.7264</td>
    <td>0.6494</td>
    <td>0.7154</td>
    <td>0.4063</td>
    <td>0.5417</td>
    <td>0.6556</td>
    <td>0.6101</td>
    <td>0.7291</td>
  </tr>
  <tr>
    <td>(+0.2%)</td>
    <td>(+0.2%)</td>
    <td>(+6.2%)</td>
    <td>(+2.5%)</td>
    <td>(+2.7%)</td>
    <td>(-2.8%)</td>
    <td>(-2.8%)</td>
    <td>(-1.2%)</td>
    <td>(-1.2%)</td>
    <td>(-1.2%)</td>
    <td>(-4.1%)</td>
    <td>(-4.1%)</td>
    <td>(-2.8%)</td>
    <td>(-2.8%)</td>
    <td>(-1.2%)</td>
  </tr>
  <tr>
    <td>GPT-4-turbo-2024-04-09</td>
    <td>0.7644</td>
    <td>0.7459</td>
    <td>0.8355</td>
    <td>0.8534</td>
    <td>0.5053</td>
    <td>0.5146</td>
    <td>0.6959</td>
    <td>0.6318</td>
    <td>0.6486</td>
    <td>0.4526</td>
    <td>0.5229</td>
    <td>0.6183</td>
    <td>0.5982</td>
    <td>0.7114</td>
  </tr>
  <tr>
    <td>(+0.1%)</td>
    <td>(+0.1%)</td>
    <td>(+6.2%)</td>
    <td>(+2.5%)</td>
    <td>(+2.5%)</td>
    <td>(-1.8%)</td>
    <td>(-1.8%)</td>
    <td>(-0.6%)</td>
    <td>(-0.6%)</td>
    <td>(-0.6%)</td>
    <td>(-3.9%)</td>
    <td>(-3.9%)</td>
    <td>(-0.6%)</td>
    <td>(-0.6%)</td>
    <td>(-0.6%)</td>
  </tr>
</table>

Table 7: We selected several LLMs and produced LLM-generated text many times, and then conducted inspections to evaluate the impact of diversity on the detector.

used in DetectGPT and Fast-detectGPT, which are widely recognized in the literature on AI-generated text detection. This setting minimizes the influence of factors, other than model evolution, on the detection performance.

To assess the domain generalization capability of detection methods, we included the following datasets: Writing, PubMed, Community, and Peer-Read. We also incorporated three different generation paradigms: continuation, question-answering, and paraphrasing. Specifically, the prompts used for generating each dataset are as follows:

• Xsum: "Please write an article with about 150 words starting exactly with: <prefix>"

• Writing: "Please write an article with about 150 words starting exactly with: <prefix>"

• PubMed: "Please answer the question in about 50 words: <question>"

• SocialMedia: "Generate text similar to the input social media text but using different words and sentence composition: <Full text>"

• PeerRead: "Please write a peer review with about 150 words starting exactly with: <prefix>"

F Experimental Setup Details

In this section, we provide a detailed description of the experimental setup. Specifically, for zero-shot, we follow Fast-DetectGPT (Bao et al.) and use GPT-J-6B (Wang and Komatsuzaki, 2021) and GPT-Neo-2.7B (Black et al., 2021) as the scoring models for zero-shot methods. For supervised detectors, we used the pre-trained detectors provided by the authors. All experiments are conducted on a workstation equipped with 4 NVIDIA A100 GPUs.

For the valuation metric, we measure the detection performance using two metrics: AUROC(the area under the receiver operating characteristic) and EMG (evolving model generalization). A higher AUROC value indicates better detection quality, while a higher EMG metric indicates better generalization ability across evolving LLMs.

G The explanation of EMG

In this section, we provide an explanation of EMG. A positive EMG value indicates that the detection performance improves as the LLM evolves, while negative values reflect a decline in generalization. The magnitude of the EMG value also reflects the degree of improvement or decline in generalization; smaller values indicate worse stability. EMG reflects the direction and degree of fluctuations. However, it does not directly indicate the actual detection performance regarding AUROC.

For example, as shown in Table 2, in the GPT-4o family, the EMG values for RADAR and Text-Fluoroscopy are −0.537% and −0.493%, respectively. However, as shown in Table 1, the AUROC values for RADAR range from 0.8416 to 0.8694, while those for Text-Fluoroscopy fluctuate between 0.75426 to 0.79618. Although the EMG values are similar, the AUROC differs significantly, suggesting that both methods have comparable generalization abilities in the face of evolving LLMs, but their actual detection performance varies. Therefore, we recommend evaluating both AUROC and EMG together.
Table 8: Details of evolving LLMs.

<table>
  <tr>
    <th>Evolving LLMs</th>
    <th>Version Time</th>
    <th>Source</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>2024-05-13</td>
    <td>gpt-4o-2024-05-13</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>2024-08-06</td>
    <td>gpt-4o-2024-08-06</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>2024-11-20</td>
    <td>gpt-4o-2024-11-20</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>Latest</td>
    <td>chatgpt-4o-latest</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>2024-07-18</td>
    <td>gpt-4o-mini-2024-07-18</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>2023-06-13</td>
    <td>gpt-4-0613</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>2023-11-06</td>
    <td>gpt-4-1106-preview</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>2024-01-25</td>
    <td>gpt-4-0125-preview</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>2024-04-09</td>
    <td>gpt-4-turbo-2024-04-09</td>
  </tr>
  <tr>
    <td>Claude-Sonnet</td>
    <td>2024-02-29</td>
    <td>claude-3-sonnet-20240229</td>
  </tr>
  <tr>
    <td>Claude-Sonnet</td>
    <td>2024-06-20</td>
    <td>claude-3-5-sonnet-20240620</td>
  </tr>
  <tr>
    <td>Claude-Sonnet</td>
    <td>2024-10-22</td>
    <td>claude-3-5-sonnet-20241022</td>
  </tr>
  <tr>
    <td>Claude-Haiku</td>
    <td>2024-03-07</td>
    <td>claude-3-haiku-20240307</td>
  </tr>
  <tr>
    <td>Claude-Haiku</td>
    <td>2024-10-22</td>
    <td>claude-3-5-haiku-20241022</td>
  </tr>
  <tr>
    <td>Claude-Opus</td>
    <td>2024-02-29</td>
    <td>claude-3-opus-20240229</td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>Qwen1.5-7B</td>
    <td>Qwen/Qwen1.5-7B-Chat</td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>Qwen2-7B</td>
    <td>Qwen2-7B-Instruct</td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>Qwen2.5-7B</td>
    <td>Qwen/Qwen2.5-7B-Instruct</td>
  </tr>
  <tr>
    <td>Llama3</td>
    <td>Llama-3.1-8B</td>
    <td>meta-llama/Meta-Llama-3.1-8B-Instruct</td>
  </tr>
  <tr>
    <td>Llama3</td>
    <td>Llama-3.1-70B</td>
    <td>meta-llama/Meta-Llama-3.1-70B-Instruct</td>
  </tr>
  <tr>
    <td>Llama3</td>
    <td>Llama-3.2-1B</td>
    <td>meta-llama/Meta-Llama-3.2-1B-Instruct</td>
  </tr>
  <tr>
    <td>Llama3</td>
    <td>Llama-3.2-3B</td>
    <td>meta-llama/Meta-Llama-3.2-3B-Instruct</td>
  </tr>
  <tr>
    <td>Llama3</td>
    <td>Llama-3.3-70B</td>
    <td>meta-llama/Meta-Llama-3.3-70B-Instruct</td>
  </tr>
  <tr>
    <td>Fine-tuning</td>
    <td>LLaMA-2.7B-chat-hf</td>
    <td>meta-llama/Llama-2-7b-chat-hf</td>
  </tr>
  <tr>
    <td>Fine-tuning</td>
    <td>Vicuna-1.5-7B</td>
    <td>Imsys/vicuna-7b-v1.5</td>
  </tr>
  <tr>
    <td>Fine-tuning</td>
    <td>Wizardmath-7B</td>
    <td>WizardLMTeam/WizardMath-7B-V1.0</td>
  </tr>
  <tr>
    <td>Pruning</td>
    <td>Sheared-LLaMA1.3B</td>
    <td>princeton-nlp/Sheared-LLaMA-1.3B</td>
  </tr>
  <tr>
    <td>Pruning</td>
    <td>Sheared-LLaMA1.3B-pruned</td>
    <td>princeton-nlp/Sheared-LLaMA-1.3B-Pruned</td>
  </tr>
  <tr>
    <td>Pruning</td>
    <td>Sheared-LLaMA2.7B-pruned</td>
    <td>princeton-nlp/Sheared-LLaMA-2.7B</td>
  </tr>
  <tr>
    <td>Pruning</td>
    <td>Sheared-LLaMA2.7B</td>
    <td>princeton-nlp/Sheared-LLaMA-2.7B-Pruned</td>
  </tr>
</table>

Reviewer paper summaries: The paper introduces EvoBench, a benchmark for evaluating LLM-generated text detection methods across evolving LLMs. It addresses the gap that current detection methods are evaluated on static benchmarks, while real-world LLMs continuously evolve via updates and developments. EvoBench covers multiple LLM families and versions, and is evaluated to be a challenging dataset. The authors also suggest optimization strategies to improve detection performance.
Summary of strengths: 1. EvoBench addresses an important real-world gap in LLM-generated text detection
2. The benchmark is comprehensive, covering multiple LLM versions, domains, and data generation methods
3. The EMG metric quantifies generalization ability across evolving LLMs
Summary of weaknesses: 1. The optimizing strategies the authors propose are fairly basic. The "optimizing training data" approach requires re-training on evolving data which may not reflect the detection model's generalization ability to unseen data. The "extracting shared features" approach simply replaces the scoring model in Fast-DetectGPT, and lacks analysis on why the pruned Llama2 is effective for development and prune data, but not for finetune data.
2. Despite the detectors generally performing worse on evolving data, some exhibit better performance in specific cases. For example, RADAR performs better on average for model developments, and Fast-DetectGPT performs better on newer versions of Qwen. The paper would benefit from deeper analysis of these findings.
Comments, suggestions and typos: It would be informative to include results from closed-source detectors, such as GPTZero.
Overall assessment scores: 3