Venue: NAACL 2025

# DART@: An AIGT Detector using AMR of Rephrased Text

Hyeonchu Park, Byungjun Kim, and Bugeun Kim

Department of Artificial Intelligence, Chung-Ang University, Republic of Korea

{phchnu0429, k36769, bgnkim}@cau.ac.kr

###### Abstract

As large language models (LLMs) generate more human-like texts, concerns about the side effects of AI-generated texts (AIGT) have grown. So, researchers have developed methods for detecting AIGT. However, two challenges remain. First, the performance of detecting black-box LLMs is low because existing models focus on probabilistic features. Second, most AIGT detectors have been tested on a single-candidate setting, which assumes that we know the origin of an AIGT and which may deviate from the real-world scenario. To resolve these challenges, we propose DART@, which consists of four steps: rephrasing, semantic parsing, scoring, and multiclass classification. We conducted three experiments to test the performance of DART. The experimental result shows that DART can discriminate multiple black-box LLMs without probabilistic features and the origin of AIGT.

+
Footnote †: Equal contribution.

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 710–721

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

As large language models (LLMs) continue to advance, it becomes increasingly difficult for humans to discern AI-generated text (AIGT). This poses issues in society and research, such as spreading fake news and tainting AI training data. Researchers have developed AIGT detectors to address these issues. Despite their success, two challenges related to real-world applicability persist.

One challenge with applying AIGT detectors is low performance in detecting recent black-box LLMs. Traditionally, AIGT detectors rely on probabilistic features such as logits. However, in commercial black-box models, including GPT [4, 8] or Gemini [10], we cannot access their computation procedure which provides logits. That is, traditional approaches cannot detect such black-box models. So, researchers have also designed detectors using syntactic features that do not require accessing computational procedures. Yet, these detectors struggle to detect black-box models because their syntactic naturalness is comparable to that of humans.

The other challenge is the vagueness of the origin of AIGTs. In the inference time of a detector, it receives a text without any information about its origin. So, similar to the inference scenario, we should verify whether a detector can successfully discriminate AIGT regardless of source models. However, existing studies mainly tested their detectors under the assumption that a candidate LLM is known in advance; they tested whether a binary detector can distinguish a 'human-written text' from an 'AIGT by the predefined candidate.' So, whether existing detectors can detect the origin without the assumption is questionable.

To address these challenges, we propose a Detector using AMR of Rephrased Text [1]. DART utilizes the semantic gap between given input and rephrased text, using Abstract Meaning Representation (AMR). This rephrasing idea was first introduced by RAIDAR [12]; we adopted a similar idea to reveal such a semantic gap. To examine the real-world detection performance, we assess DART in three settings: single-candidate, multi-candidate, and leave-one-out. Experimental results show that DART can successfully discriminate humans from four cutting-edge LLMs, including GPT-3.5-turbo, GPT-4o, Llama 3-70b [19], and Gemini-1.5-Flash.

Thus, this paper has the following contributions:

* We present a semantics-based detection framework for AIGT, leveraging semantic gaps between given input text and rephrased texts.
* DART can discriminate different LLMs and outperform other models. On average, DART beat others by more than 19% in F1 score.



* Also, DART can generalize its knowledge on detecting unseen source models. Specifically, DART achieved a 85.6% F1 score on leave-one-out experiment.

## 2 Background

In this section, we categorize existing studies regarding numbers (_single_ or _multi_) and transparency (_white box_ or _black box_) of candidate LLMs.

Single white-box candidatesAIGT detectors first attempted to extract candidate-specific features. As the candidate is a known white-box model, some researchers designed algorithms adopting probabilistic features from the model Gehrmann et al. (2019); Mitchell et al. (2023). For example, DetectGPT Mitchell et al. (2023) used log probabilities of tokens as features. Other researchers used neural models that can learn features from the given texts Solaiman et al. (2019); Hu et al. (2023). However, as many black-box LLMs recently emerged, the performance of existing detectors should be revalidated on those LLMs.

Single black-box candidatesSome AIGT detectors then attempted to extract features regardless of the candidate Bao et al. (2024); Yu et al. (2024); Yang et al. (2023); Kim et al. (2024), as black-box candidates may not provide probabilistic features. Fast-DetectGPT Bao et al. (2024) extended DetectGPT by extracting probabilistic features from a proxy white-box model (e.g., GPT-J). Since such a proxy can provide less accurate results, other studies used syntactic or surface-level features without using a proxy Yang et al. (2023); Kim et al. (2024). For example, DNA-GPT Yang et al. (2023) used \(n\)-grams from multiple paraphrased texts generated by the candidate. However, such syntactic features are insufficient to detect recent LLMs because recent models generate text with human-level syntax.

Multiple candidatesAs a single-candidate performance is far from real-world scenarios, recent AIGT detectors were designed to detect multiple candidates Li et al. (2023); Abburi et al. (2023); Wang et al. (2023); Shi et al. (2024); Antoun et al. (2024). For example, POGER Shi et al. (2024) extends resampling methods to estimate probability using about 100 paraphrases. Because of such an excessive regeneration, POGER incurs high computational costs. Besides, SeqXGPT Wang et al. (2023) used a Transformer-based detector with a proxy model estimating probabilistic features. However, these studies mainly focused on surface-level features and the limited range of LLMs (e.g., GPT family), raising questions about detecting other cutting-edge LLMs.

## 3 The DART Framework

As shown in Figure 1, DART utilizes semantic gaps between a given text and rephrased texts. To train a detector capturing such gaps, DART uses a four-step procedure: _Rephrasing_, _Semantic parsing_, _Semantic gap scoring_, and _Classification_.

Step 1, Rephrasing:We hypothesized that rephrasing texts could reveal the difference between humans and AI in the way they express semantics. To obtain the rephrased texts, DART uses a _rephraseer_ module that generates semantically closer text \(T_{1}\) from a given text \(T_{0}\). Further, we let the rephraser generate another rephrased text \(T_{2}\) by giving \(T_{1}\) to attain additional features. To avoid generating rephrased texts irrelevant to the given input, we need a reliable rephraser that can preserve semantics. So, we adopted GPT-4o-20240513 as our rephraseer because the model showed the highest performance in semantics-related tasks OpenAI (2024a). Appendix A.2 details the prompts used in the rephrasing step.

Step 2, Semantic parsing:DART adopts a semantic parser to transform texts into semantic representations. We especially adopted AMR as a

Figure 1: The DART framework



semantic representation because AMR has widely been adopted to abstract the given text into semantics (Banarescu et al., 2013). For the parser, we adopted Naseem et al. (2022). As a result, the parser constructs an AMR graph \(A_{i}\) from each \(T_{i}\).

Step 3, Semantic gap scoring:DART uses metrics for semantic parsers to measure semantic gaps between texts. As we adopted AMR as a semantic representation in the previous step, we utilize a fast and efficient algorithm for scoring AMR similarity called SEMA Anchieta et al. (2019); Ki et al. (2024). To obtain semantic gaps between \(A_{0}\) and \(A_{i}\) (\(i>0\)), DART computes precision \(p_{i}\) and recall \(r_{i}\) scores generated by SEMA, resulting a feature vector \(v=[p_{1},p_{2},r_{1},r_{2}]^{\top}\) for the next step.

Step 4, Classification:DART has a classifier that predicts one possible origin of \(T_{0}\). DART uses interpretable classifiers, including support vector machine (SVM) or decision tree (DT), though any classifier that maps \(v\) to origins can be used.

## 4 Experiments

To evaluate the performance of DART, we conducted three experiments: (1) single-candidate, (2) multi-candidate, and (3) leave-one-out settings. First, in the single-candidate setting, we formulate AIGT detection as a binary classification task. Assuming that AIGTs are exclusively produced by a specific LLM, a detector should predict whether the given text is produced by the LLM. Second, in the multi-candidate setting, we formulate the task as a multi-label classification. After training on AIGTs from multiple candidate sources, a detector should decide the source of the given input text among the candidates. Third, in the leave-one-out setting, we test the generalizability of detectors. We examined whether a detector can successfully classify AIGTs from models that were unseen during the training.

We ran each experiment 10 times for each experiment to achieve reproducibility. Further, we analyzed DART's training efficiency by examining the decreasing rate of detecting performance as the size of the training dataset.

### Datasets

To train DART, we need human-written texts and AIGTs. First, we used four English datasets as human-written text datasets: XSum Narayan et al. (2018), SQuAD 1.1 Rajpurkar et al. (2018), Reddit Fan et al. (2018), and PubMedQA Jin et al. (2019). Following the practice of previous research Mitchell et al. (2023); Wang et al. (2023), we randomly sampled texts from these datasets. We split training and validation sets with an 8:2 ratio.

Second, we generated AIGT datasets based on the human dataset. Following Mitchell et al. (2023), we collected English AIGT from each human-written text. Four cutting-edge LLMs are used to generate AIGTs: GPT-4o, GPT-3.5-turbo, Llama 3-70B, and Gemini-1.5 Flash. We obtained AIGTs by providing the first 30 tokens of each human-written text to an LLM. Because PubMedQA contains many texts shorter than 30 tokens, we provided corresponding questions instead of the first 30 tokens. Appendix A.1 illustrates the detailed prompts used for generating AIGTs. As a result, we obtained about 3,989 human-written texts and 15,956 AIGTs (= 3,989 texts \(\times\) 4 LLMs). See Appendix B.2 for the statistics of the collected dataset.

### Baselines

As baselines, we used five open-source state-of-the-art detectors: DetectGPT Mitchell et al. (2023), Fast-DetectGPT Bao et al. (2024), DNA-GPT Yang et al. (2023), Roberta-base Solaiman et al. (2019), and SeqXGPT Wang et al. (2023). Among these models, DetectGPT, Fast-DetectGPT, and SeqXGPT used probabilistic features generated by third-party LLMs in order to detect cutting-edge LLMs. Meanwhile, DNA-GPT and Roberta-base used shallow semantic features, such as \(n\)-grams or contextual embeddings. DART stands out from these models because it uses AMR-based semantics rather than probabilistic features.

We used a different set of detectors for the three experiments, considering experiments reported with five baselines. For the single-candidate experiment, we compared DART with all five detectors. For the multi-candidate and the leave-one-out experiments, we compared DART only with SeqXGPT, as it is the only existing detector that can trained on multiple candidates simultaneously. To ensure a fair comparison, all detectors used in the experiment are trained on our dataset from scratch1. To measure the performance, we used the F1 score.

 



## 5 Result and Discussion

Single-candidate experiment:DART outperformed existing models. As shown in Table 1, our DART\({}_{\text{DT}}\) and DART\({}_{\text{SVM}}\) achieved 96.5% and 82.8% F1 scores on average, which are 19.3%p and 5.6%p higher than the Roberta-base model (77.2%). Also, DART\({}_{\text{DT}}\) can detect all four cutting-edge models with over 85% of F1 score. Meanwhile, other existing models showed F1 scores lower than 70%, on average. Moreover, DNA-GPT and SeqXGPT sometimes showed F1 scores lower than the random binary baseline (50%).

We suspect that DART\({}_{\text{DT}}\) can achieve such outstanding performance because our semantic scoring step can successfully form several clusters according to their origins. To support this argument, we further analyzed the feature vectors of DART using principal component analysis. We found that texts sharing the same source usually form several independent clusters rather than spread over the space. Detailed results are presented in Appendix C.3.

Multi-candidate experiment:DART also outperformed SeqXGPT. As shown in Table 2, our DART\({}_{\text{DT}}\) and DART\({}_{\text{SVM}}\) achieved 81.2% and 65.0% macro F1 scores, which are 22.0%p and 5.8%p higher than SeqXGPT (59.2%). Interestingly, SeqXGPT achieved the lowest F1 score on detecting Llama 3 (44.8%), but DART\({}_{\text{DT}}\) achieved the lowest score on detecting GPT-4o (76.6%).

We suspect how the detectors extract features using an LLM affects the performance. We present a contingency table of SeqXGPT and DART\({}_{\text{DT}}\) to support this claim, as shown in Figure 2. The figure shows that (i) SeqXGPT struggled in distinguishing models other than Llama 3, and (ii) DART\({}_{\text{DT}}\) struggled in distinguishing the GPT family and humans. Since SeqXGPT in our experiment used GPT-2 as a proxy model, and DART\({}_{\text{DT}}\) used GPT-4o as a _rephraser_ module, the characteristics of the used LLMs affected the detection performances. For example, as DART\({}_{\text{DT}}\) utilizes semantic gaps between the original and rephrased texts, origins should reveal distinguishable gaps to identify them successfully. So, when the gaps are too similar between origins to discriminate them, DART\({}_{\text{DT}}\) faces difficulty in the classification step.

Since GPT-4o has a similar language understanding ability to humans [1], GPT-4o and humans may be less distinguishable through

Figure 2: Contingency matrix from multi-candidate experiment. Top (a) and Bottom (b) correspond to SeqXGPT and DART\({}_{\text{DT}}\). Actual and predicted classes are depicted as horizontal and vertical axes.



gaps. Similarly, as GPT-3.5-turbo may share some core knowledge with GPT-40, GPT-40 can be confused with GPT-3.5-turbo in \(\text{DART}_{\text{DT}}\).

Leave-One-Out experiment:DARTDT showed the best performance. As shown in Table 3, \(\text{DART}_{\text{DT}}\) achieved 85.6% average F1 score, followed by SeqXGPT (77.9%) and \(\text{DART}_{\text{SVM}}\) (56.5%). Besides, \(\text{DART}_{\text{DT}}\) scored 62.5% F1 on detecting the unseen Gemini-1.5, though \(\text{DART}_{\text{DT}}\) recorded more than 75% on detecting others.

This result indicates that \(\text{DART}_{\text{DT}}\) can generalize trained knowledge to detect unseen source models. That is, \(\text{DART}_{\text{DT}}\) can discriminate new candidate models from humans. Specifically, compared to the single-candidate result (Table 1), our model showed almost similar performance on detecting GPT-3.5-turbo and Llama 3 without training on those models. As in the single-candidate experiment, we believe that our semantic scoring step helped to detect unseen models because they form clusters independent from humans. Also, when the cluster becomes indiscernible with humans, \(\text{DART}_{\text{DT}}\) struggles to detect new models. For example, \(\text{DART}_{\text{DT}}\) showed a big performance drop when excluding Gemini-1.5 from the training set because \(\text{DART}_{\text{DT}}\) often confused Gemini-1.5 with humans (top-right corner on Figure 2b).

Training efficiency of Dart:Here, we discuss the general tendency of the result. Figure 3 shows the performance changes when we decrease the size of the training set. Detailed result of training efficiency is presented in Appendix C.4. The result shows that \(\text{DART}_{\text{DT}}\) is robust even though we use a small amount of training data. Specifically, \(\text{DART}_{\text{DT}}\) maintained a similar F1 score until we used 1/32 of the training set (about 500 examples). Meanwhile, the performance of SeqXGPT and \(\text{DART}_{\text{SVM}}\) monotonically decreases as we reduce the size of the training set.

## 6 Conclusion

We introduced an AIGT framework, \(\text{DART}^{\circledr}\) to tackle challenges in applying AIGT detectors to real-world scenarios. DART employed _rephraser_ and semantic gap scoring module to address the challenges of black-box models. To evaluate whether DART can address vagueness of origin, we assessed DART in three experimental settings: single-candidate, multi-candidate, and leave-one-out settings. As a result, DART achieved outstanding performance compared to existing AIGT detectors, demonstrating successful capture of differences across origins with semantic gaps.

Figure 3: F1 score of detectors when we decrease the amount of training data in multi-candidate experiment.

\begin{table}
\begin{tabular}{l c|c c c c c} \hline \hline  & Macro F1 & GPT-3.5-turbo & GPT-4o & Llama3-70B & Gemini-1.5 & Human \\ \hline SeqXGPT* & 59.2\(\pm\)0.66 & 54.3\(\pm\)1.44 & 66.4\(\pm\)1.08 & 44.8\(\pm\)0.95 & 61.4\(\pm\)1.68 & 69.3\(\pm\)0.93 \\ \hline \multirow{2}{*}{
\begin{tabular}{c} DART\end{tabular} } & 65.0\(\pm\)0.77 & 67.4\(\pm\)0.81 & 71.0\(\pm\)1.20 & 54.0\(\pm\)1.26 & 67.0\(\pm\)0.94 & 65.4\(\pm\)1.16 \\  & **81.2\(\pm\)1.71** & **80.6\(\pm\)4.61** & **76.6\(\pm\)1.16** & **85.8\(\pm\)5.42** & **85.5\(\pm\)2.36** & **77.3\(\pm\)0.88** \\ \hline \hline \end{tabular}

* Models used GPT-2 as a proxy model, except Llama 3.

\end{table}
Table 2: F1 scores of detectors in the **multi-candidate** experiment, with standard deviations reported.

 



### Limitations

Despite the outstanding performance of DART, this paper has three limitations. First, we tested DART only with a single rephraser LLM, GPT-4o. Though GPT-4o provided enough semantic information to distinguish AIGTs successfully, it is questionable whether DART can be used with other rephrase LLMs, such as Llama 3, Gemini Pro, or others. Also, we recognize the cost implications of utilizing GPT-4o as a rephraser, which could restrict its applicability in resource-limited environments. Since different language models may provide different rephrased texts with lower costs, we need further study to determine how much rephraser LLM affects the performance.

Second, the performance of the adopted AMR parser may affect the detection performance of DART. Though the AMR parser rarely introduces errors in the DART framework, such errors may lead to huge changes in detection performance when they occur. Using a publicly available AMR parser Naseem et al. (2022), DART showed the lowest bound of its performance. Thus, we need further study to improve the performance using other semantic parsers.

Third, DART tested on a narrow range of blackbox models. While narrow LLMs have become publicly available through paid APIs or pretrained parameters, we tried our best to include recent LLMs, such as Gemini Pro or Claude 3. However, we finally excluded those models because their safeguards prevented from generating AIGTs based on a given human-written text when preparing the AIGT dataset. To generalize our findings to other origins, we need to conduct further studies in a broader range of models and design a new method of generating AIGTs.

## Acknowledgments

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2021-0-01341, Artificial Intelligence Graduate School Program, Chung-Ang University)

## References

* Abburi et al. (2023) Harika Abburi, Michael Suesserman, Nirmala Pudota, Balaji Veeramani, Edward Bowen, and Sanmitra Bhattacharya. 2023. Generative ai text classification using ensemble llm approaches. In _IberLEF@SEPLN_, volume 3496 of _CEUR Workshop Proceedings_. CEUR-WS.org.
* Anchieta et al. (2019) Rafael Torres Anchieta, Marco Antonio Sobrevilla Cabezudo, and Thiago Alexandre Salgueiro Pardo. 2019. Sema: an extended semantic evaluation for amr. In _(To appear) Proceedings of the 20th Computational Linguistics and Intelligent Text Processing_. Springer International Publishing.
* Antoun et al. (2024) Wissam Antoun, Benoit Sagot, and Djame Seddah. 2024. From text to source: Results in detecting large language model-generated content. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_, pages 7531-7543, Torino, Italia. ELRA and ICCL.
* Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
* Bao et al. (2024) Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2024. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhi, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akili Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _Preprint_, arXiv:2407.21783.
* Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
* Gehrmann et al. (2019) Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical detection and visualization of generated text. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 111-116, Florence, Italy. Association for Computational Linguistics.
* 16, 2023_.
* Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A

dataset for biomedical research question answering. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2567-2577, Hong Kong, China. Association for Computational Linguistics.
* Ki et al. (2024) Kyung Seo Ki, Bugeun Kim, and Gahgene Gweon. 2024. Inspecting soundness of AMR similarity metrics in terms of equivalence and inequivalence. In _Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024)_, pages 402-409, Mexico City, Mexico. Association for Computational Linguistics.
* Kim et al. (2024) Zae Myung Kim, Kwang Lee, Preston Zhu, Vipul Raheja, and Dongyeop Kang. 2024. Threads of subtlety: Detecting machine-generated texts through discourse motifs. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5449-5474, Bangkok, Thailand. Association for Computational Linguistics.
* Li et al. (2023) Linyang Li, Pengyu Wang, Ke Ren, Tianxiang Sun, and Xipeng Qiu. 2023. Origin tracing and detecting of llms. _Preprint_, arXiv:2304.14072.
* Mao et al. (2024) Chengzhi Mao, Carl Vondrick, Hao Wang, and Junfeng Yang. 2024. Raidar: generative AI detection via rewriting.
* Mitchell et al. (2023) Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. DetectGPT: Zero-shot machine-generated text detection using probability curvature. In _Proceedings of the 40th International Conference on Machine Learning_, volume 202 of _Proceedings of Machine Learning Research_, pages 24950-24962. PMLR.
* Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
* Naseem et al. (2022) Tahira Naseem, Austin Blodgett, Sadhana Kumaravel, Tim O'Gorman, Young-Suk Lee, Jeffrey Flanigan, Ramon Astudillo, Radu Florian, Salim Roukos, and Nathan Schneider. 2022. DocAMR: Multi-sentence AMR representation and evaluation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3496-3505, Seattle, United States. Association for Computational Linguistics.
* OpenAI (2024a) OpenAI. 2024a. Hello gpt-4o. [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/). Accessed: 2024-10-13.
* OpenAI (2024b) OpenAI. 2024b. Introducing chatgtrt. [https://openai.com/index/chatgpt/](https://openai.com/index/chatgpt/). Accessed: 2024-10-13.
* Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
* Shi et al. (2024) Yuhui Shi, Qiang Sheng, Juan Cao, Hao Mi, Beizhe Hu, and Danding Wang. 2024. Ten words only still help: Improving black-box ai-generated text detection via proxy-guided efficient re-sampling. In _Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24_, pages 494-502. International Joint Conferences on Artificial Intelligence Organization. Main Track.
* Solaimal et al. (2019) Irene Solaimal, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. 2019. Release strategies and the social impacts of language models. _Preprint_, arXiv:1908.09203.
* Team et al. (2024) Gemini Team, Petko Georgiev, Ying Ian Lei, Ryan Burnell, Libin Bai, Annol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _Preprint_, arXiv:2403.05530.
* Wang et al. (2023) Pengyu Wang, Lingyang Li, Ke Ren, Botian Jiang, Dong Zhang, and Xipeng Qiu. 2023. SeqXGPT: Sentence-level AI-generated text detection. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 1144-1156, Singapore. Association for Computational Linguistics.
* Yang et al. (2023) Xianjun Yang, Wei Cheng, Yue Wu, Linda Petzold, William Yang Wang, and Haifeng Chen. 2023. Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text.
* Yu et al. (2024) Xiao Yu, Yuang Qi, Kejiang Chen, Guoqiang Chen, Xi Yang, Pengyuan Zhu, Xiwwei Shang, Weiming Zhang, and Nenghai Yu. 2024. Dpic: Decoupling prompt and intrinsic characteristics for llvm generated text detection. _Preprint_, arXiv:2305.12519.

 

Prompts

### AIGT datasets

In general, we followed the prompts used in SeqXGPT (Wang et al., 2023) when generating the AIGT dataset. We collected AIGTs by providing LLMs with the first 30 tokens of human-written texts and letting them generate the rest of the texts, except for the PubMedQA dataset. Besides, we asked LLMs to answer the questions in the PubMedQA dataset instead of providing the 30 tokens of text, borrowing the collecting method of Mitchell et al. (2023). We used different methods for PubMedQA because most of the texts in PubMedQA were shorter than 30 tokens. In addition, to avoid collecting AIGTs with irrelevant phrases (e.g., "Here is the generation of ..."), we added a constraint clause in the prompts for Llama 3-70B and Gemini-1.5 Flash.

We understand that different datasets and different prompting methods may affect the performance of the detectors. Therefore, we conducted additional per-subset experiments to investigate whether those differences influenced the detecting performance. The findings are detailed in Appendix C.2.

For GPT familyWhen collecting AIGTs with GPT-3.5-turbo and GPT-4o, we used the following prompts except for the PubMedQA dataset. Please provide a continuation for the following content to make it coherent: {first 30 tokens} For PubMedQA, we used the following prompts: Please answer the question: {question} For Llama 3-70B and Gemini-1.5-Flash When collecting AIGTs with Llama 3-70B and Gemini-1.5-Flash, we used the following prompts except for the PubMedQA dataset. Please provide a continuation for the following content to make it coherent: {first 30 tokens} Provide the continuation without any prefix. ------------------------------------------------answer: For PubMedQA, we used the following prompts: Please answer the question: {question} Providethe continuation without any prefix. ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------answer: ------------------------------------------------: ------------------------------------------------answer: ------------------------------------------------: ------------------------------------------------answer: ------------------------------------------------: ------------------------------------------------answer: ------------------------------------------------: ------------------------------------------------answer: ------------------------------------------------: ------------------------------------------------answer: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: ------------------------------------------------: : 

Experimental setting

### Environment

Hardware configuration:The experiments were conducted on a system with an AMD Ryzen Threadripper 3960X 24-Core Processor and four NVIDIA RTX A6000 GPUs. The four NVIDIA RTX A6000 GPUs are used to train existing detectors and execute AMR parsers. The semantic gap scoring module was run on a single core of the CPU.

LLM APIs:We used commercial APIs of LLMs to collect AGTs and rephrased texts. GPT models are called with OpenAI's official API. Llama 3-70B is called with a free API provided by groq.com. Lastly, Gemini-1.5-Flash is called with OpenRouter's API.

ImplementationWe used Python 3.11.7 for implementing DART@. Using scikit-learn library, we implemented DART\({}_{\text{SVM}}\) and DART\({}_{\text{DT}}\) with SVC and DecisionTreeClassifier. We mostly used the basic settings of those classes without conducting a hyperparameter search. The only exception is the depth of the pruned tree in DART\({}_{\text{DT}}\), and we set it as 5 based on our heuristic.

### Dataset statistics

Table 7 in page 10 shows the statistics of the collected dataset. We used four datasets, which belong to different domains: Xsum Narayan et al. (2018), SQuAD Rajpurkar et al. (2018), Reddit Fan et al. (2018), and PubMedQA Jin et al. (2019). Xsum is a dataset of news articles and summaries. SQuAD is a question-answering dataset whose questions are based on Wikipedia articles. Reddit is a dataset of human-written stories with writing prompts. PubMedQA is a question-answering dataset on a specialized medical domain.

The statistics show that the average lengths of texts in each dataset are different. For example, Gemini-1.5 usually generates long texts on the PubMedQA dataset, while the model generates short texts on the Xsum and Reddit datasets. On average, it seems that the length of a given text is not a significant factor for discriminating origin.

## Appendix C Additional analysis

### Precision, Recall

As we discussed in Section 3, DART computes precision \(p\) and recall \(r\) scores with SEMA. Note that \(p_{i}\) and \(r_{i}\) refer to the semantic similarity between the original text \(T_{0}\) and the \(i\)-th rephrased text \(T_{i}\). DART assumes that the differences between those rephrased texts in terms of \(p\) and \(r\) values can be used to identify AIGTs. In this section, we provide evidence that supports the assumption by comparing the trend of \(p\) and \(r\) values.

Table 6 on page 9 illustrates the average of precision and recall values we collected. On average, the table shows that \(p_{2}\) and \(r_{2}\) are smaller than \(p_{1}\) and \(r_{1}\), respectively. This indicates that \(T_{2}\) was semantically far from \(T_{0}\) than \(T_{1}\). So, as we apply rephraseer more times on \(T_{0}\), the semantics of rephrased text becomes farther from \(T_{0}\).

Also, the result shows that \(p\) and \(r\) values are lower in human-written texts than AIGTs. For example, human-written text showed \(p_{1}\) of 0.619, which is lower than AIGTs (ranging from 0.636 to 0.651). So, it is reasonable to use these values to distinguish between human-written texts and AIGTs.

### Effect of prompt and domain changes

Since we used different prompting methods and datasets in generating AIGTs, we conducted the per-subset experiment to investigate whether those differences affected the performance of detectors. Specifically, we conducted multi-candidate experiments for each subset. For example, instead of using all data, we trained and tested models only with texts from PubMedQA.

Table 5 on page 8 shows the results of the per-subset experiment. Though the domains and prompting methods are different across those subsets, DART\({}_{\text{DT}}\) achieved consistently high-performance scores by showing 98.6% macro F1. Also, DART\({}_{\text{SVM}}\) (ranging from 80.0% to 93.2%) showed better consistency than SeqXGPT (ranging from 57.0% to 75.1%). This result indicates that DART@ models are robust on changes of domains or prompting methods compared to SeqXGPT.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \(p_{1}\) & \(p_{2}\) & \(r_{1}\) & \(r_{2}\) \\ \hline Human & 0.619 & 0.582 & 0.600 & 0.561 \\ GPT-3.5-T & 0.645 & 0.605 & 0.631 & 0.595 \\ GPT-4o & 0.636 & 0.596 & 0.623 & 0.587 \\ Llama3 & 0.648 & 0.610 & 0.631 & 0.594 \\ Gemini-1.5 & 0.651 & 0.615 & 0.633 & 0.596 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Precision and Recall values for text comparisons between \(T_{0}\), \(T_{1}\) and \(T_{0}\), \(T_{2}\) 



### Principal components of features

Figure 4 and 5 in page 11 display PCA plots of features used in DART. The figures show that each source makes several clusters. Here, we attempt to interpret DART's experimental results by analyzing the PCA results. The distribution of feature vectors may affect the performance of SVM and DT classifiers. As SVM seeks a global decision boundary that maximizes margin, SVM may not find a clear decision boundary between multiple mini clusters. Meanwhile, DT can split such mini clusters based on multiple criteria. So, DT could achieve high performance in discriminating AIGTs from human-written texts. For example, we can easily discriminate humans from others and iteratively build different decision boundaries between smaller clusters. As a result, \({\text{DART}_{\text{DT}}}\) can clearly discriminate sources and showed higher performance than \({\text{DART}_{\text{SVM}}}\).

### Training efficiency on single-candidate setting

Figure 6 in page 12 shows the training efficiency on the single-candidate experiment. In general, the performance drops as the size of the dataset decreases. Among those models, \({\text{DART}_{\text{DT}}}\) demonstrates the best performance across all models, even with small datasets. \({\text{DART}_{\text{SVM}}}\) experiences a more rapid decrease in its performance.

We suspect that the distribution of the data may affect the classification performance. In other words, SVM or a neural network may not have sufficient data to distinguish small clusters whose features are close to each other when we use a small dataset.

## Appendix B

\begin{table}
\begin{tabular}{l r r r} \hline \hline  & \# char & \# tokens & \# sample \\ \hline PubMedQA dataset & & & \\ \hline Human & 265.9 & 41.8 & 995 \\ LLMs & 1132.4 & 188.1 & 3980 \\ GPT-3.5T & 496.2 & 78.2 & 995 \\ GPT-4o & 1181.4 & 192.6 & \\ Llama 3-70B & 1327.5 & 212.7 & \\ Gemini-1.5F & 1524.7 & 268.9 & \\ \hline Xsum dataset & & & \\ \hline Human & 2194.5 & 428.9 & 999 \\ LLMs & 909.5 & 160.5 & 3996 \\ GPT-3.5T & 773.7 & 136.5 & 999 \\ GPT-4o & 1627.8 & 282.4 & \\ Llama 3-70B & 671.9 & 121.6 & \\ Gemini-1.5F & 564.7 & 101.5 & \\ \hline Reddit dataset & & & \\ \hline Human & 2962.7 & 641.0 & 997 \\ LLMs & 1135.5 & 237.3 & 3988 \\ GPT-3.5T & 852.3 & 176.7 & 997 \\ GPT-4o & 1986.7 & 413.5 & \\ Llama 3-70B & 1009.5 & 213.0 & \\ Gemini-1.5F & 691.4 & 146.0 & \\ \hline SQuAD dataset & & & \\ \hline Human & 740.2 & 135.1 & 998 \\ LLMs & 947.5 & 157.0 & 3992 \\ GPT-3.5T & 503.4 & 79.1 & 998 \\ GPT-4o & 1803.1 & 303.6 & \\ Llama 3-70B & 809.7 & 142.4 & \\ Gemini-1.5F & 673.9 & 102.8 & \\ \hline \hline \end{tabular}
\end{table}
Table 7: Statistics of collected datasets

Figure 4: PCA Plot between the first principal component and the second

Figure 5: PCA Plot between the first principal component and the third




* [14] M. C. Bader, J.-M. Chen, and S.-H. Lee, "A new approach to the design of a single-candidate experiment using a multi-objective optimization framework for the problem of interest in the search problem," _Journal of Machine Learning Research_**2018**(3):1579-1603, 2017.
* [15] A. D. Krizhevsky, I. Sutskever, and R. Fergus, "Image recognition with deep convolutional networks," _IEEE Transactions on Pattern Analysis and Machine Intelligence_**2017**(1):1-10, 2016.
* [16] P. L. Bergstrom, "The alternating nature of neural networks," _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pp. 1-10, 2006.
* [17] G. E. Hinton, "Deep learning with applications to image processing," _arXiv preprint arXiv:1406.6098_**(1):1002.1189.
* [18] T. H. Friedman, "An introduction to machine learning," _Foundations of Statistics_**19**(1):1-10, 1994.
* [19] Y. Bengio, "Online learning with applications to data mining," _Neural Computation_**19**(2):1-10, 2003.
* [20] N. Srivastava, "Dropout classification by reducing the number of parameters," _Nature_**2004**(7):867-874, 2004.
* [21] V. Nair, "Very Large Scale Structure Models," _Advances in Neural Information Processing Systems_**2004**(1):1-10, 2004.
* [22] W. Teh, "Learning from multiple images," _IEEE transactions on pattern analysis and machine intelligence_**2004**(1):1-1, 2004.

[MISSING_PAGE_POST]