Venue: NAACL 2025

# Bemo: Benchmark of Expert-edited Machine-generated Outputs

Ekaterina Artemova\({}^{1}\) Jason Lucas\({}^{2}\) Saranya Venkatraman\({}^{2}\)

**Jooyoung Lee\({}^{2}\) Sergei Tilga\({}^{1}\) Adaku Uchendu\({}^{3}\) Vladislav Mikhailov\({}^{4}\)**

###### Abstract

The rapid proliferation of large language models (LLMs) has increased the volume of machine-generated texts (MGTs) and blurred text authorship in various domains. However, most existing MGT benchmarks include _single-author_ texts (human-written & machine-generated). This conventional design fails to capture more practical _multi-author_ scenarios, where the user refines the LLM response for natural flow, coherence, and factual correctness. Our paper introduces the **B**enchmark of **E**xpert-**e**dited **M**achine-generated **O**utputts (Beemo), which includes 6.5k texts written by humans, generated by ten instruction-finetuned LLMs and edited by experts for various use cases, ranging from creative writing to summarization. Beemo additionally comprises 13.1k machine-generated & LLM-edited texts, which allows for diverse MGT detection evaluation across various edit types. We document the Beemo's creation protocol and present the results of benchmarking 33 configurations of MGT detectors in different experimental setups. We find that expert-based editing evades MGT detection, while LLM-edited texts are unlikely to be recognized as human-written. Beemo and all materials are publicly available.

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 6992–7018

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

The rapid advancement of large language models (LLMs) has significantly improved their ability to assist users with a wide range of writing tasks Gero et al. (2022); Yang et al. (2022); Chakrabarty et al. (2024b). While the benefits of LLMs are commendable, their widespread adoption has raised concerns regarding authenticity of textual content and potential malicious uses within the domains of news, social media, science, and education Lucas et al. (2023); Crothers et al. (2023); Gupta et al. (2024); Chamezopoulos et al. (2024); Tang et al. (2024).

Over the last few years, a broad range of machine-generated text (MGT) detection benchmarks have been created to facilitate the development of reliable detectors, aimed at mitigating the risks associated with the misuse of LLMs across different domains and languages (e.g., Macko et al., 2023, 2024; Pu et al., 2023; Dugan et al., 2024; Tripto et al., 2024; Wang et al., 2024c, b). However, most of them focus on a _single-author_ scenario, comprising only machine-generated and human-written texts. This well-established design does not account for common applications of LLMs, where the user refines the LLM's response for natural flow, coherence, and factual correctness.

This paper introduces the **B**enchmark of **E**xpert-**e**dited **M**achine-generated **O**utputts (Beemo1), which consists of 6.5k texts written by humans, generated by ten open-source instruction-finetuned LLMs and edited by expert annotators, who are well-experienced in refining LLM-generated content. Furthermore, each MGT is edited by two state-of-the-art LLMs using several diverse editing prompts, which results in 13.1k machine-generated & LLM-edited texts. Beamo covers five use cases: open-ended generation, rewriting, summarization, and open & closed question answering (QA). Our design enables various diagnostic evaluation scenarios, ranging from out-of-domain MGT detection to analyzing how an MTG detector's behavior changes after the LLM response is refined by experts and "humanized" by other LLMs.

Our main contributions are: (i) we create Beemo, one of the first _multi-author_ benchmarks of LLM-generated & expert-edited responses for fine-grained MGT detection, which counts 19.6k texts in total; (ii) we evaluate 33 configurations of zero-shot and pretrained MGT detectors; (iii) we release Beemo2 and all annotation materials3.





## 2 Related Work

Table 1 summarizes commonly used monolingual and cross-lingual MGT detection evaluation resources. To the best of our knowledge, only three of them comprise machine-generated & LLM-edited texts, and only two of them include machine-generated & human-edited texts. Below, we provide an overview of the resources by their type and task formulation with the main focus on English.

### Standard MGT Detection Benchmarks

MGT detection features various task formulations and labelling schemes: binary classification, neural authorship attribution, boundary-detection, and multi-author classification.

Binary ClassificationBinary classification is a well-established design of MGT benchmarks [8, 9, 40, 37]. The task is to determine if a given text is machine-generated or not.

Authorship AttributionAuthorship Attribution aims to identify the author of a given text [6, 5, 10, 11], a multi-class classification task with humans and LLMs as the labels.

Boundary DetectionBoundary detection is a less explored task formulation, which aligns with application of LLMs for text continuation tasks. Here, the goal is to detect a change point in the text, where a natural text transitions into a neural one [12, 23].

Multi-author ClassificationRecent research has proposed several fine-grained MGT benchmarks, which help to explore how varying degrees of LLM intervention in writing tasks affect the behavior of MGT detectors. MixSet [37] comprises human-written, machine-generated, and human/LLM-refined MGTs and focuses on multi-author binary classification. LLM-DetectAIve [1] offers a four-way classification task with two other classes (human-written/machine-generated & machine-polished), which reflects a common usage of LLMs for enhancing a human-written text. LAMP [1] consists of 1k non- and fiction MGTs edited by professionals and LLMs and explores automatic detection and rewriting of "problematic" spans in MGTs. Our work differs from these studies in the following aspects: (i) in line with Zhang et al.; Chakrabarty et al., we present one of the first attempts to create a benchmark of expert-edited MGTs; (ii) we use an instruction-tuning dataset as the source of prompts and human-written responses, which ensures coverage of various domains and use cases; (iii) our annotation protocol is not based on a predefined taxonomy of operations (MixSet) and edits (LAMP); instead, our annotators make the edits based on their expertise in refining

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Resource** & **Size** & **\# Models** & \multicolumn{2}{c}{**Machine-generated \& LLM-edited**} & **Machine-generated \\  & & & \& LLM-edited & **\& Human-edited** \\ \hline TuringBench [6] & 200k & 19 & ✗ & ✗ \\ RoFT [12] & 21k & 2 & ✗ & ✗ \\ MULTITuDE [1] & 75k & 8 & ✗ & ✗ \\ OpenLMLText [13] & 340k & 5 & ✗ & ✗ \\ HC3 Plus [23] & 210k & 1 & ✗ & ✗ \\ MAGE [14] & 447k & 27 & ✗ & ✗ \\ RAID [12] & 6.2M & 11 & ✗ & ✗ \\ MultiSocial [1] & 472k & 7 & ✗ & ✗ \\ BUST [15] & 25k & 7 & ✗ & ✗ \\ M4 [24] & 122k & 6 & ✗ & ✗ \\ M4GT-Bench [23] & 217k & 8 & ✗ & ✗ \\ LLM-DetectAIve [1] & 303k & 14 & ✓ & ✗ \\ MixSet [37] & 3.6k & 8 & ✓ & ✓ \\ LAMP [1] & 1k & 7 & ✓ & ✓ \\
**Beemo (ours)** & 19.6k & 10 & ✓ & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of publicly available monolingual and cross-lingual MGT detection evaluation resources. Beemo is one of the first benchmarks that contains machine-generated & LLM/human-edited texts.



LLM-generated content; (iv) Beemo includes the largest number of expert-edited MGTs compared to MixSet (1.2k) and LAMP (1k); (v) we analyze how the behavior of binary MGT detectors changes after edits are made w.r.t. edit ratio and use case.

### Editing MGTs

There are several approaches to editing generated content within MGT detection: operation-based, prompt-based, and expert-based editing.

Operation-based EditingZhang et al. (2024a) propose five operations to refine an MGT at the token-, sentence-, and paragraph-level: polish, complete, rewrite, humanize, and adapt. This hybrid editing approach emulates real-world cases where humans aim to modify MGT to suit their preferences, improve quality, or align with the intended purpose.

Prompt-based EditingEmerging research leverages prompt-based LLM editing to mitigate the risks of "humanizing" LLM-generated content at scale. Hu et al. (2023a) employ few-shot in-context learning with human demonstrations to enhance MGT edits. Mitchell et al. (2023); Yang et al. (2023) use prompts to simulate humans limited editing behavior using T5 Raffel et al. (2020). Abassy et al. (2024) utilize various prompts to refine MGTs and polish human-written texts, such as improving grammar and fluency. While prompt-based editing show benefits to assist humans, the output quality depends heavily on prompts and LLM editors, emphasizing the importance of careful prompt design and LLM considerations Kamalloo et al. (2023); Zhang et al. (2023, 2024b).

Expert-based EditingPrior work employed human experts to edit and evaluate text across different domains Nahar et al. (2024); Reid and Neubig (2022); Du et al. (2022); Roberts and Moran (1983); Lucas et al. (2023). However, limited research utilizes such experts for editing MGTs. The above mentioned works by Zhang et al. (2024a); Chakrabarty et al. (2024a) highlight human experts' unique value in refining machine-generated content, particularly in adapting to academic genres. Our work incorporates the prompt- and expert-based editing approaches, summarizes the experts' strategies to refine LLM content, and presents the results of analyzing the detectors' performance w.r.t. edit percentages.

## 3 Beemo

Figure 1 outlines our high-level methodology for creating Beemo, which includes the following stages: generating instruction-finetuned LLMs' responses (SS3.1), editing the responses by expert annotators (SS3.2) and state-of-the-art LLMs (SS3.3).

### Machine-generated Data Collection

No RobotsRajani et al. (2023) is a human-created instruction-finetuning dataset, which is used as the source of prompts and corresponding human-written responses across the following categories: open-ended generation, rewriting, summarization, and open and closed QA4. We randomly sample each prompt to generate a response with one of

Figure 1: Overview of the Beemo’s creation pipeline. (a) Use No Robots Rajani et al. (2023) as the source of prompts and human-written responses across five categories. Generate responses from ten open-source instructions-finetuned LLMs. (b) Refine the LLMs’ responses with a team of expert editors. (c) Refine the LLMs’ responses using two state-of-the-art LLMs and editing prompts (P1-P3). Each of 2,187 instances includes nine text versions.

 

ten open-source instruction-finetuned LMs (see Table 2), which range in size from 2B to 70B. We use the default HuggingFace Wolf et al. (2020) chat templates and inference hyperparameters.

### Expert-based Editing

We run an in-house annotation to create expert-edited versions of the MGTs. Our team consists of two lead editors and 25 expert annotators, who are native English speakers and well-experienced in refining content produced by LLMs5. Refer to a voluntary survey results from 17 respondents in Table 5 (see Appendix A) for sociodemographic details. The lead editors collaborate closely with the annotators throughout the annotation project, suggesting areas for improvement, exchanging feedback in a group chat, and manually validating each annotator's submission. We provide detailed annotation guidelines to the annotators before they start the editing phase (see Appendix B). Each annotator receives one example at a time (a category, a prompt, and an LLM's response) and is asked to (i) carefully read the prompt and the response; (ii) judge the response's relevance to the prompt and the category; (iii) fact check the response if required; (iv) edit the response by correcting factual inconsistency, removing hallucinations, and improving style, coherence, and fluency; and (v) proofread the edited version before submission. The recommended ratio of edits ranges between 20% and 40% of the response. The average pay rate is SS20/hr. The annotator can skip an example (i) if it does not require any edits and aligns with the prompt intent or (ii) if it requires significant improvements or does not follow the prompt closely. We discard such examples and use only the refined and manually validated responses to create Beemo.

Footnote 5: Our annotators are not experts in NLP but are well-experienced in annotating content generated by LLMs. Our annotators have diverse backgrounds, including professional writing, editing, and translation across various domains, from media and communication to education and technologies.

### LLM-based Editing

We utilize two open-source and proprietary LLMs for LLM-based editing: Llama3.1-70B-Instruct and GPT-4o (see Table 2). We automatically refine the MGTs using three types of prompts based on distinct motivations (see Table 9 in Appendix D): (P1) Focuses on grammatical correctness and human-like qualities; (P2) Aims to remove artifacts in an LLM-generated text (e.g., unwanted text such as _"Sure, here is the summarized text"_); (P3) Emphasizes producing a more natural and native-sounding text. Our aim here is to address the diversity of LLM-based edits in real-world scenarios, which target various aspects of making a machine-generated text more human-like Zhang et al. (2024); Gero et al. (2022). We specify the recommended range of edits in (P1) and (P2) as 20%-40% and do not provide it in (P3) to ensure the LLM editors are not restricted in refining the generated response.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **Base** & **License** & **Source** & **SST Corpus** \\ \hline  & \multicolumn{4}{c}{**Data generation**} \\ \hline zephyr-7b-beta & Mistral-7B-v0.1 & MIT & Tunstall et al. (2023) & UltraChat, Urtrackback \\ tulu2-lb & Llama 2 7B & AI2 ImpACT & Ivison et al. (2023) & human-created, synthetic \\ gemm-2b-it & Gemma 2B & & & \\ gema-7b-it & Gemma 7B & & & \\ llama2-7b-chat-hf & Llama 2 7B & & & \\ llama2-13b-chat-nf & Llama 2 13B & Liama & Touvron et al. (2023) & Misc. \\ llama2-7bb-chat-hf & Llama 2 70B & & & \\ \#istral-7B-Instruct-v0.1 & Mistral-7B-v0.1 & Apache-2.0 & Jiang et al. (2023) & Misc. \\ \#istral-8x7B-Instruct-v0.1 & Mistral 8x7B & Apache-2.0 & Jiang et al. (2024) & Misc. \\ \hline  & \multicolumn{4}{c}{**LLM-based editing**} \\ \hline Llama3.1–70B-Instruct & Llama 3.1 70B & Llama & Dubey et al. (2024) & Misc. \\ GPT-4o & GPT-4 & OpenAI & OpenAI (2024) & Misc. \\ \hline \hline \end{tabular}
\end{table}
Table 2: The LLMs used to generate (§3.1) and edit (§3.3) responses. The supervised finentuning (SST) / instructions-tuning corpus description is based on public information. We evaluate potential overlap between the SFT corpora and LLMs’ outputs in §3.4. Corpora references: UltraChat Ding et al. (2023); UltraFeedback Cui et al. (2023).

 



### Benchmark Analysis

General StatisticsWe summarize the Beemo's general statistics by category (see Table 3) and LLM (see Table 10, Table 11 in Appendix E) based on count, stylometric, edit distance, and embedding-based similarity metrics computed via TextDescriptives Hansen et al. (2023), edtidistance6 and Evaluate7: the average text length in tokens, the average number of stopwords in a text, the Flesch-Kincaid grade of a text (FKG), Levenshtein distance (LD), and BERTScore similarity Zhang et al. (2019) between different text versions.

Footnote 6: github.com/roy-ht/editdistance

Footnote 7: github.com/huggingface/evaluate

We observe that the number of stopwords as a distinctive measure of repetitiveness Frohling Zubiaga (2021) depends on the LLM rather than its size. However, larger LLMs generally produce text with higher readability scores (FKG), with the exception of the Llama2 LLMs. Analyzing the statistics by category, we find that the LLMs' responses and their expert-edited versions are generally longer than the human-written and LLM-edited texts. This implies that the LLM editors tend to shorten the generated responses, which is indicated by higher LD values. The average LD between the machine- and human-written responses ranges from 153 to 256, which suggests a minimal overlap between No Robots and the LLMs' instruction-finetuning corpora. The similarity between machine-generated texts and their edited versions remains consistent across LLMs based on BERTScore values. GPT-4o edits are the closest to the original machine-generated texts, while Llama3.1-70B-Instruct edits show lower similarity, reflecting diverse editing styles among LLM editors.

Editing AnalysisWe compare the edit percentages between the MGTs and their Llama3.1-70B-Instruct-, GPT-4o- and expert-edited versions using difflib8. Figure 2 and Figure 3 present a comparison of editing behaviors among the expert and LLM editors. The experts demonstrate an average edit percentage of 70%, falling between GPT-4o (60%) and Llama3.1-70B-Instruct (80%). The error bars indicate considerable variability in edit percentages across different instances. This variability is particularly pronounced for the LLM editors, indicating less consistency in their editing behavior than experts. Both LLMs make more extensive edits than specified in the prompts (SS3.3), which suggests their potential limitation in controlled editing.

Footnote 8: docs.python.org/library/difflib

Effect of PromptFigure 5 (see Appendix E) illustrates the LLMs' editing behavior across the three prompts. Overall, Llama3.1-70B-Instruct makes the most extensive edits irrespective of the prompt. Both LLMs exhibit an increase in edit percentage from (P1) to (P2) of up to 15%, al

\begin{table}
\begin{tabular}{l c r r r r r r} \hline \hline
**Category** & **\# Examples** & **\# Tokens (P)** & **\# Tokens (H)** & **\# Tokens (M)** & **\# Tokens (E)** & **\# Tokens (L)** & **\# Tokens (G)** \\ \hline Closed QA & 1,845 & 268.0 \(\pm\) 230.3 & 26.7 \(\pm\) 17.4 & 83.3 \(\pm\) 73.2 & 60.9 \(\pm\) 66.4 & 74.2 \(\pm\) 35.5 & 62.4 \(\pm\) 37.4 \\ Generation & 5,265 & 40.3 \(\pm\) 34.3 & 225.5 \(\pm\) 142.9 & 305.6 \(\pm\) 175.4 & 278.1 \(\pm\) 171.4 & 115.8 \(\pm\) 23.1 & 115.9 \(\pm\) 28.5 \\ Open QA & 4,347 & 15.4 \(\pm\) 44.9 & 89.1 \(\pm\) 55.0 & 186.3 \(\pm\) 145.4 & 128.4 \(\pm\) 121.5 & 100.7 \(\pm\) 30.2 & 91.7 \(\pm\) 37.7 \\ Rewrite & 4,725 & 296.7 \(\pm\) 249.7 & 240.6 \(\pm\) 201.0 & 250.3 \(\pm\) 160.2 & 242.7 \(\pm\) 170.3 & 115.7 \(\pm\) 22.1 & 112.2 \(\pm\) 28.2 \\ Summarize & 3,501 & 274.9 \(\pm\) 162.9 & 73.7 \(\pm\) 452.4 & 143.4 \(\pm\) 1106.9 & 98.2 \(\pm\) 599.0 & 102.5 \(\pm\) 25.0 & 85.9 \(\pm\) 30.8 \\ Overall & 19,683 & 159.4 \(\pm\) 204.0 & 155.3 \(\pm\) 151.6 & 216.3 \(\pm\) 164.0 & 184.2 \(\pm\) 160.4 & 101.8 \(\pm\) 27.2 & 93.6 \(\pm\) 32.5 \\ \hline \end{tabular}
\end{table}
Table 3: General statistics by category. **P=No Robots** prompts (2,187 prompts); **H=human-written (2,187 texts); **M**=machine-generated (2,187 texts); **E**=expert-edited (2,187 texts); **L=Llama3.1–70B-Instruct-edited (6,561 texts); **G**=GTP-4o-edited (6,561 texts). **L** and **G** are aggregated over three editing prompts (§3.3).

Figure 2: Distribution of edit percentages across five edit ranges for expert annotators, GPT-4o (P1, P2, **P3**), and Llama3.1-70B-Instruct (P1, P2, **P3**). The bars represent the number of instances falling within each edit percentage range for each editor type.



though (P2) aims to control the edit range. Using (P3) results in a slight decrease in edit percentages for both LLMs compared to (P2), but they remain higher than (P1). The results suggest that prompt engineering significantly influences the LLM's ability to follow specified editing ranges, which aligns with Wang et al. (2023).

Survey on Editing StrategiesWe send a voluntary survey to our editors to better understand their main editing strategies and identify common issues in the generated responses (see Appendix F). Our analysis of 14 respondents' answers reveals several common issues with the MGTs: (i) format inconsistencies (e.g., a partial or inconsistent ordering of items in a list); (ii) hallucinations and factual errors; (iii) struggling with creative writing (e.g., writing poems and haikus); (iv) complicated and repetitive vocabulary, and their responses can feel over-explained; and (v) responses often lack natural flow and exhibit a repetitive structure. Our expert editors follow a consistent approach based on their individual experience to refine the generated outputs: (i) carefully reading the prompt and LLM's response; (ii) fact-checking if required; (iii) focusing on complicated use of passive voice, repeated elements, and odd openings as common properties in our data; (iv) editing for natural flow, improving structure, and changing vocabulary for clarity and richness; (v) ensuring the response adheres to the required format; (vi) double-checking for grammar, phrasing and spelling errors; and (vii) reading through or aloud for a final quality check. Refer to Table 8 in Appendix C for examples of identified issues in machine-generated texts and corresponding edits. Our experts report that it can be challenging to follow the recommended edit range of 20%-40%, and rewriting entire sections can be more effective than editing specific parts, particularly for summarization, rewriting, and open-ended generation. However, the core content of the LLM response remains consistent, even at higher edit ratios.

## 4 Experimental Setup

We use Beemo as an out-of-domain benchmark to evaluate generalization abilities of 33 zero-shot and pretrained MGT detectors' configurations in seven binary classification task formulations, which rely on single- and multi-author text versions. We list the detectors below and detail them in Appendix G.

Zero-shot MGT DetectorsThe zero-shot MGT detectors utilize log probabilities, entropy, and curvature-based properties to score a given text: (i) Binoculars Hans et al. (2024); (ii) Log probability Solaiman et al. (2019); (iii) Rank Gehrmann et al. (2019); (iv) Log-Rank Ippolito et al. (2020); (v) Entropy Gehrmann et al. (2019); Mitchell et al. (2023); (vi) DetectLLM Su et al. (2023a) Likelihood Log-Rank ratio (LRR) and Normalized Perturbed Log-Rank (NPR); and (vii) DetectGPT Mitchell et al. (2023)9. We use the codebase by Hans et al. and Su et al. to run the detectors, which supports GPT2-XL10, OPT-1.3B11, Falcon-7B12, and Owen2-7B13 as the default backbone models.

Footnote 10: http.co/openai-community/gpt2-xl

Footnote 11: hf.co/facebook/opt-1.3b

Footnote 12: hf.co/lituae/falcon-7b

Footnote 13: hf.co/Owen2-7B

Footnote 15: hf.co/TrustSaFal/RADAR-Vicuna-7B

Footnote 16: hf.co/ytabultanian/AIG_detector_env

Footnote 17: hf.co/yoful/MAGE

Footnote 18: hf.co/openai/roberta-large-openai-detector

Footnote 19: footnotemark:

#### Pretrained MGT Detectors

We consider the following pretrained detectors: (i) RADAR14Hu et al. (2023b); (ii) AIGC MPU15; (iii) MAGEG16Li et al. (2024); and (iv) OpenAI RoBERTabase17/large18.

Footnote 15: hf.co/ytabultanian/AIG_detector_env

Footnote 17: hf.co/oyraul/MAGE

Footnote 18: hf.co/openai/roberta-base-openai-detector

Footnote 19: footnotemark:

#### Task Formulations

We explore two research questions in our work: (1) _Do the detectors identify MGTs as generated after they are refined by an expert or by an LLM?_ and (2) _Do the detectors identify MGTs as human-written after they are refined_ 

by an expert or by an LLM?_ We consider the corresponding binary classification task formulations by using different text versions as the input:

1. human-written (**H**; label=0) vs. machine-generated, expert-, Llama3.1-7@B-Instruct-, and GPT-4o-edited (**M/E/L/G**; label=1). We treat a text as machine-generated even if the experts edit it.
2. expert-, Llama3.1-70B-Instruct-, and GPT-4o-edited (**E/L/G**; label=0) vs. machine-generated (**M**; label=1). We treat a text as human-written even if the LLMs "humanize" it.

Performance MetricFollowing Verma et al. (2024); Mitchell et al. (2023), we evaluate the detector performance using Area Under the Receiver Operating Characteristic curve (AUROC). AUROC values represent the probability that a detector assigns higher scores to a randomly selected machine-generated text than a randomly selected human-written text. AUROC is commonly used in zero-shot scenarios - where the choice of the scoring threshold is crucial - as it considers the range of all possible thresholds Krishna et al. (2024).

## 5 Results & Analysis

This section describes our empirical evaluation results on Beemo. We report the overall results in Table 4 and results by category in Appendix H. Overall, we observe that Binoculars is the best

\begin{table}
\begin{tabular}{l l|c c|c|c|c|c|c} \hline \multicolumn{1}{l}{**Detector**} & \multicolumn{1}{c}{**H vs. M**} & \multicolumn{1}{c}{**E vs. M**} & \multicolumn{1}{c}{**H vs. E**} & \multicolumn{1}{c}{**L vs. M**} & \multicolumn{1}{c}{**H vs. L**} & \multicolumn{1}{c}{**G vs. M**} & \multicolumn{1}{c}{**H vs. G**} \\ \hline \multicolumn{8}{c}{**Zero-shot MCT Detectors**} \\ \hline
**Binoculars** & & 83.90 & 76.79 & 61.24 & [59.90] & 79.90 & [57.93] & 78.15 \\ \hline \multirow{4}{*}{**Log Probability**} & GPT-2NL & 69.72 & 64.73 & 56.56 & 63.70 & 58.88 & 60.32 & 61.23 \\  & GPT-1.18 & 74.82 & 66.52 & 60.86 & 66.58 & 62.78 & 61.12 & 66.95 \\  & Falcon-7R & 86.07 & 65.48 & 78.41 & 65.36 & 80.78 & 59.65 & 82.44 \\  & Quez-7R & 87.72 & 68.68 & 78.02 & 56.23 & 84.52 & 62.45 & 81.27 \\ \hline \multirow{4}{*}{**Rank**} & GPT-2NL & 60.36 & 56.70 & 54.50 & 59.85 & 50.65 & 56.79 & 53.64 \\  & GPT-1.18 & 65.95 & 57.44 & 59.94 & 63.56 & 53.48 & 57.10 & 59.86 \\  & Falcon-7R & 74.52 & 56.56 & 70.66 & 62.95 & 63.64 & 55.87 & 70.61 \\  & Quez-7R & 74.22 & 57.65 & 69.11 & 52.55 & 72.52 & 55.10 & 70.81 \\ \hline \multirow{4}{*}{**Log-Rank**} & GPT-2NL & 70.26 & 64.95 & 56.78 & 65.25 & 57.42 & 61.20 & 60.68 \\  & GPT-1.18 & 73.03 & 66.11 & 59.03 & 67.21 & 59.30 & 61.83 & 63.90 \\  & Falcon-7R & 84.81 & 65.07 & 77.02 & 66.12 & 78.26 & 60.26 & 80.32 \\  & Quez-7R & 86.91 & 68.13 & 77.04 & 56.04 & 83.62 & 62.09 & 80.33 \\ \hline \multirow{4}{*}{**Entropy**} & GPT-2NL & 88.93 & 42.42 & 46.06 & 36.74 & 51.72 & 41.95 & 46.64 \\  & GPT-1.18 & 37.63 & 42.74 & 44.18 & 35.41 & 51.33 & 41.07 & 45.72 \\  & Falcon-7R & 20.92 & 41.65 & 24.42 & 36.02 & 29.24 & 40.96 & 25.66 \\  & Quez-7R & 10.36 & 32.02 & 17.24 & 44.01 & 12.82 & 38.01 & 15.28 \\ \hline \multirow{4}{*}{**DetectLLM**} & GPT-2NL & 68.73 & 63.72 & 56.18 & 67.45 & 51.86 & 62.27 & 57.07 \\  & GPT-1.18 & 64.37 & 63.09 & 51.90 & 67.21 & 47.17 & 62.49 & 52.10 \\  & Likelihood Log-Rank Ratio & Falcon-7R & 76.73 & 62.16 & 68.41 & 66.77 & 64.68 & 61.28 & 68.31 \\  & Quez-7R & 79.68 & 64.12 & 69.40 & 54.71 & 76.25 & 59.41 & 72.83 \\ \hline \multirow{4}{*}{**DetectLLM**} & GPT-2NL & 64.06 & 63.07 & 62.00 & 54.08 & 63.34 & 51.07 & 63.93 \\  & GPT-1.18 & 65.29 & 63.86 & 62.86 & 56.98 & 64.04 & 51.36 & 65.12 \\ \cline{1-1}  & Random-7R & 70.14 & 63.18 & 66.71 & 56.65 & 68.40 & 51.37 & 69.84 \\ \cline{1-1}  & Quez-7R & 74.12 & 68.30 & 68.11 & 54.52 & 72.49 & 50.68 & 73.84 \\ \hline \multirow{4}{*}{**DetectGPT**} & GPT-2NL & 64.91 & 63.71 & 62.75 & 59.15 & 63.53 & 58.97 & 63.63 \\ \cline{1-1}  & GPT-1.18 & 67.27 & 66.87 & 64.22 & 64.39 & 64.68 & 61.65 & 65.32 \\ \cline{1-1}  & Falcon-7R & 70.69 & 65.35 & 67.47 & 64.15 & 67.73 & 62.67 & 68.18 \\ \cline{1-1}  & Quez-7R & 72.76 & 69.37 & 68.21 & 61.36 & 70.14 & 63.05 & 69.90 \\ \hline \end{tabular} 
\begin{tabular}{l l|c|c|c|c|c|c} \hline \multicolumn{1}{c}{**Retrained MCT Detectors**} \\ \hline
**RADAR** & & 51.41 & 50.93 & 50.80 & 54.51 & 62.65 & 52.96 & 60.16 \\
**MAGE** & & 73.72 & 60.88 & 64.09 & 57.95 & 67.79 & 60.59 & 65.37 \\
**AIGC MPU** & & 70.52 & 70.35 & 50.66 & 65.71 & 57.17 & 59.66 & 62.69 \\
**OpenAI RoBERTn-base** & & 66.95 & 66.83 & 49.13 & 66.95 & 50.00 & 66.95 & 50.00 \\
**OpenAI RoBERTn-large** & & 62.93 & 63.96 & 48.76 & 64.71 & 48.57 & 59.67 & 53.74 \\ \hline \end{tabular}
\end{table}
Table 4: The AUROC scores (%) of the 33 MOT detectors’ configurations on Beemo. A random classifier has an AUROC of 50%. **H**=human-written; **M**=machine-generated; **E**=expert-edited; **L**=Llama3.1-70B-Instruct-edited; **G**=gpt-4o-edited. **L** and **G** are aggregated over three prompts (§3.3).

 

performing zero-shot detector, while MAGE is the strongest pretrained detector in most scenarios. Entropy performs worse or on par with a random guessing classifier. We summarize our findings below w.r.t. task formulation, editing type, MGT detector type, category, and edit percentages.

Expert-based Editing Evades DetectionAnalyzing the results in the **H vs. M/E/L/G** scenarios, we find that all detectors often fail to recognize an MGT after it is refined by experts, with the AUROC scores decreasing by up to 22% for zero-shot (e.g., Binoculars, Log Probability, and Logank) and pretrained (e.g., AIGC MPU, OpenAI RoBERTa-base/large) detectors. In contrast, many zero-shot detectors identify LLM-edited texts as MGTs, with Binoculars, Rank, Log-Rank, and Log Probability demonstrating stronger generalization abilities. However, RADAR, AIGC MPU, and OpenAI RoBERTa-base/large exhibit a random guessing performance across various edit types, which suggests a strong distribution shift.

LLM-edited Texts are Less Likely to Be Recognized as Human-authoredA comparison of the results in the **E/L/G vs. M** scenarios confirms our previous finding that the expert-edited texts are more likely to be classified as human-written compared to the LLM-edited ones. Notably, GPT-40-edited texts are generally easier to identify as MGTs compared to expert- and Llama3.1-70B-Instruct-edited.

Analysis by Detector TypeHere, we compare the results between the zero-shot and pretrained detectors in more detail. The zero-shot detectors (i) better distinguish between human-written and machine-generated texts (**H vs. M**); (ii) are more generalizable to expert- and LLM-edited texts, which is indicated by lower AUROC \(\delta\)-scores (e.g., up to 10% for Log-Rank, DetectLLM LLR/NPR, DetectGPT and up to 20% for AIGC MPU, MAGE, and OpenAI RoBERTa-base/large); and (iii) perform better with a larger backbone model in the **H vs. M/E/L/G** scenarios, however, this effect is less pronounced in the **E/L/G vs. M** scenarios.

Analysis by CategoryThe key finding here is that the detectors' behavior w.r.t. different edit types remains the same, but the AUROC scores across most task formulations depend on category (see Appendix H). The detectors perform consistently better on open-ended generation and open QA (e.g., Binoculars=up to 86%; Log Probability=up to 93%; DetectLLM LLR=up to 86%; MAGE=up to 76% in **H vs. M/E/L/G**). The rewriting, summarization, and closed QA are more challenging, where the detectors exhibit significant performance drops after expert edits (e.g., Log Probability=up to 20%; Rank=up to 17%; Log-Rank=up to 23%; DetectLLM=up to 18%) or struggle with distinguishing between different MGT versions (e.g., RADAR and MAGE often perform worse than a random classifier). We attribute these trends to category-specific text generation issues identified by our experts in SS3 and shorter text length (see Table 3), which remains an unresolved challenge in MGT detection.

Effect of Edit PercentageWe analyze the effect of the edit percentage in the **E vs. M** scenario to assess the impact of expert edits (see Figure 4). The overall trend is that the detectors' performance remains moderate and stable across the experts' edit ratios, showing general improvement as the ratio increases. Most detectors achieve similar AUC ROC scores for both moderately (20%-40%) and significantly (60%-80%) edited texts. Lower AUCROC scores at the lower edit ranges indicate that moderate editing can confuse the detectors.

## 6 Conclusion & Future Work

This work introduces Becmo, one of the first multi-author benchmarks of expert-edited and LLM-refined MGTs for English. Becmo covers five common use cases of instruction-finetuned LLMs, ranging from creative writing to summarization. We describe the Becmo creation approach and common

Figure 4: Results in the “Expert-edited” (label=0) vs. “machine-generated” (label=1) scenario divided into seven groups by the edit range.

 

issues in LLM responses and strategies for mitigating them based on the feedback from our expert editors. We conduct an extensive out-of-domain evaluation of 33 MGT detectors' configurations and analyze their performance in seven signle- and multi-author binary classification tasks. Our key empirical results demonstrate that detectors can be confused by moderate expert edits, while editing with state-of-the-art LLMs does not significantly influences the detection behavior. Furthermore, we find that zero-shot detectors are more generalizable to both expert- and LLM-edited MGTs than pretrained detectors. Our _future_ work includes (i) exploration of other MGT detection task formulations, including authorship attribution, k-way classification (human-written, machine-generated, machine-generated & expert-/LLM-edited), and extraction of spans; (ii) establishment of the human baseline across various editing types; (iii) ablation studies on exploring the supervised detectors' robustness towards unseen instruction-finetuned LLM and writing task; and (iv) ablation studies on the effect of the LLM-edited MGTs in the training data on the detectors' generalization.

### Limitations

Evaluation DesignWhile we present the results of extensive empirical evaluation of a broad range of MGT detectors' configurations, we acknowledge certain limitations in our evaluation design. First, the LLM-based editing inherently depends on the prompt, and the best prompt configuration is LLM-specific [25]. Although we address this sensitivity by employing three distinct types of editing prompts, our results may still be affected by the effect of prompt engineering. Second, our MGT detectors represent commonly used open-source approaches, which does not address the analysis of commercial and API-based MGT detectors often used to check if the LLM's response refinement passes them. Furthermore, ensemble approaches can be considered, which are more generalizable to the out-of-domain data [19]. Last but not least, there might be a distribution shift with respect to how a non-expert user can refine their LLM responses in terms of editing quality. However, a direct comparison of expert and non-expert editors falls outside the scope of this work.

Need for Continuous Data CollectionThe design of MGT benchmarks is fragile due to the rapid development of novel LLMs and their instruction-finetuned versions. This raises the need to continuously update Beemo to keep it with the current state of the field; however, it is expensive to collect expert-edited texts at scale. We encourage the NLP researchers and practitioners to contribute machine-generated and machine-generated & LLM-refined texts to account for the diversity of LLMs and editing prompts.

Lack of Human BaselineSimilar to closely related studies on multi-author MGT detection [40, 6, 7], our work does not present the human baseline results due to limited resources. We aim to establish the human baseline across various editing types in our future work.

## Ethics Statement

Expert-based EditingOur team of expert editors is based in the United States and Canada, and their pay rate exceeds the corresponding hourly minimum wage. The annotation and voluntary survey results are collected and saved anonymously. The experts are warned about potentially sensitive and harmful content in the prompts and LLMs' responses related to various topics, including but not limited to politics, culture, sexual orientation, and religion.

Use of AI-assistantsWe use Grammarly19 to correct grammar, spelling, phrasing, and style errors in our paper. Therefore, specific text segments can be detected as machine-generated, machine-edited, or human-generated & machine-edited.

Footnote 19: gramarlarly.com

Computational CostsEvaluating an MGT detector on Beemo does not require any finetuning. To reduce the evaluation costs in SS4, we pre-compute and save the detectors' predictions for each text version. This allows us to estimate the performance efficiently by manipulating these pre-computed results based on the task formulation. Further inference costs can be reduced with the help of distributed inference libraries (e.g., accelerate20 and vllm21).

Footnote 20: github.com/huggingface/accelerateer

Footnote 21: github.com/vlln-project/vllm

Potential MisuseWe acknowledge that Beemo can be misused for malicious purposes, including 

but not limited to training multi-author MGT detectors to evade further detection at scale. We release Beemo for research and development purposes and encourage responsible use of our benchmark.

TransparencyWe release Beemo and all annotation materials following the standard open-source research practices. Our GitHub repository and HuggingFace dataset card provide comprehensive documentation on our benchmark creation process and data annotation guidelines.

Licensing InformationThe prompts and human-written responses from No Robots are under the original dataset's license (CC-BY-NC-4.0). The MGTs and their LLM-edited versions are subject to the underlying instruction-finetuned LLMs' licensing terms (see Table 2). The expert-edited MGTs are available under the MIT license, unless otherwise specified in the underlying instruction-finetuned LLMs' licensing terms.

## Acknowledgements

We thank Nontobeko Magala for her contribution to the data collection, Alexey Artemov for his contribution to the figure design, Natalia Fedorova for her support, and our anonymous reviewers for their feedback.

## References

* A. Abassy, K. Elozeiri, A. Aziz, M. Ngoc Ta, R. V. Tomar, B. Adhikari, S. E. Dine Ahmed, Y. Wang, O. Mohamed Afzal, Z. Xie, et al. (2024)LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection. arXiv preprint arXiv:2408.04284. Cited by: SS1.
* I. Beltagy, M. E. Peters, and A. Cohan (2020)Longformer: the Long-Document Transformer. arXiv:2004.05150. Cited by: SS3.
* T. Chakrabarty, P. Laban, and C. Wu (2024a)Can AI writing be salvaged? mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits. arXiv preprint arXiv:2409.14509. Cited by: SS1.
* T. Chakrabarty, V. Padmakumar, F. Brahman, and S. Muresan (2024b)Creativity Support in the Age of Large Language Models: An Empirical Study Involving Professional Writers. In Proceedings of the 16th Conference on Creativity & Cognition, pp. 132-155. Cited by: SS1.
* S. Chameopoulos, D. Herrmannova, A. De Waard, D. Herrmannova, D. Rosati, and Y. Kashnitsky (2024)Overview of the dagpap24 shared task on detecting automatically generated scientific paper. In Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024), Bangkok, Thailand, pp. 7-11. External Links: Link Cited by: SS1.
* Y. Chen, H. Kang, V. Zhai, L. Li, R. Singh, and B. Raj (2023)Token prediction as implicit classification to identify llm-generated text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13112-13120. Cited by: SS1.
* J. Cornelius, O. Lithgow-Serrano, S. Mitrovic, L. Dolamic, and F. Rinaldi (2024)Bust: Benchmark for the evaluation of detectors of llvm-generated text. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 8022-8050. Cited by: SS1.
* E. N. Crothers, N. Japkowicz, and H. L. Viktor (2023)Machine-generated Text: A Comprehensive Survey of Threat Models and Detection Methods. IEEE Access11, pp. 70977-71002. Cited by: SS1.
* G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun (2023)UltraFeedback: boosting language models with high-quality Feedback. Preprint arXiv:2310.01377. Cited by: SS1.
* W. Cui, L. Zhang, Q. Wang, and S. Cai (2023)Who said that? benchmarking social media ai detection. arXiv preprint arXiv:2310.08240. Cited by: SS1.
* N. Ding, Y. Chen, B. Xu, Y. Qin, Z. Zheng, S. Hu, Z. Liu, M. Sun, and B. Zhou (2023)Enhancing Chat Language Models by Scaling High-quality Instrumental Conversations. arXiv preprint arXiv:2305.14233. Cited by: SS1.
* W. Du, V. Raheja, D. Kumar, Z. Myung Kim, M. Lopez, and D. Kang (2022)Understanding iterative revision from human-written text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3573-3590. Cited by: SS1.
* A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahale, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. (2024)The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: SS1.
* L. Dugan, A. Hwang, F. Trllik, J. Magnus Ludan, A. Zhu, H. Xu, D. Ippolito, and C. Callison-Burch (2024)RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors. arXiv preprint arXiv:2405.07940. Cited by: SS1.
* L. Dugan, D. Ippolito, A. Kirubarajan, S. Shi, and C. Callison-Burch (2023)Real 

or fake text? Investigating human ability to detect boundaries between human-written and machine-generated text. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 375. Cited by: SS1.
* T. Fagni, F. Falchi, M. Gambini, A. Martella, and M. Tesconi (2021)TweepFake: about detecting deepfake tweets. PloS one6 (5), e0251415. External Links: Link, Document Cited by: SS1.
* L. Frohling and A. Zubiaga (2021)Feature-based detection of automated language models: tackling GPT-2, GPT-3 and Grover. PeerJ Computer Science7,e443. Cited by: SS1.
* S. Gehrmann, H. Strobelt, and A. Rush (2019)GLTR: statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Florence, Italy, pp. 111-116. External Links: ISBN 978-3-540-46556-5, Link, Document Cited by: SS1.
* G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sitfe, M. Riviere, M. Sanjay Kale, J. Love, et al. (2024)GEMA: open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. Cited by: SS1.
* K. Ionka Gero, V. Liu, and L. Chilton (2022)Sparks: inspiration for science writing using language models. In Proceedings of the 2022 ACM Designing Interactive Systems Conference, Vol. 1002-1019. Cited by: SS1.
* V. Gupta, P. Narayanan Venkit, S. Wilson, and R. J. Passonneau (2024)Sc sociodemographic bias in language models: a survey and forward path. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pp. 295-322. Cited by: SS1.
* A. Hans, A. Schwarzschild, V. Cherepanova, H. Kazemi, A. Saha, M. Goldblum, J. Geiping, and T. Goldstein (2024)Spotting LLMs with binoculars: zero-shot detection of machine-generated text. In Forty-first International Conference on Machine Learning, Cited by: SS1.
* L. Hansen, L. Ronbo Olsen, and Kenneth Enevoldsen (2023)TextDescriptives: a Python package for calculating a large variety of metrics from text. arXiv preprint arXiv:2301.02057. Cited by: SS1.
* 16, 2023, Cited by: SS1.
* D. Ippolito, D. Duckworth, C. Callison-Burch, and D. Eck (2020)Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 1808-1822. External Links: Link, Document Cited by: SS1.
* H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith, I. Beltagy, et al. (2023)Camels in a changing climate: enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702. Cited by: SS1.
* A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. Singh Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. (2023)Mistral 7b. arXiv preprint arXiv:2310.06825. Cited by: SS1.
* A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, Blanche Savary, C. Bamford, D. Singh Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, et al. (2024)MixRAT of experts. arXiv preprint arXiv:2401.04088. Cited by: SS1.
* E. Kamalloo, N. Dairi, C. Clarke, and D. Rafiei (2023)Evaluating open-domain question answering in the era of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 5591-5606. External Links: Link, Document Cited by: SS1.
* K. Krishna, Y. Song, M. Karpinska, J. Wieting, and M. Tyyer (2024)Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems36. Cited by: SS1.
* Y. Li, Q. Li, L. Cui, W. Bi, Z. Wang, L. Wang, S. Yang, S. Shi, and Y. Zhang (2024)MAGE: machine-generated text detection in the wild. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangskok, Thailand, pp. 36-53. Cited by: SS1.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)RoBERTa: a robustly Optimized BERT Pretraining Approach. Preprint arXiv:1907.11692. Cited by: SS1.
* V. Liyanage, D. Buscaldi, and A. Nazarenko (2022)A benchmark corpus for the detection of automatically generated text in academic publications. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 4692-4700. Cited by: SS1.
* J. Lucas, A. Uchendu, M. Yamashita, J. Lee, S. Rohatgi, and D. Lee (2023) 

2023. Fighting fire with fire: The dual role of lfms in crafting and detecting elusive disinformation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 14279-14305.
* Macko et al. (2024) Dominik Macko, Jakub Kopal, Robert Moro, and Ivan Srba. 2024. MultiSocial: Multilingual Benchmark of Machine-Generated Text Detection of Social-Media Texts. _arXiv preprint arXiv:2406.12549_.
* Macko et al. (2023) Dominik Macko, Robert Moro, Adaku Uchendu, Jason Lucas, Michiharu Yamashita, Matus Pikuliak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, and Maria Bielikova. 2023. MULTITuDE: Large-scale multilingual machine-generated text detection benchmark. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 9960-9987, Singapore. Association for Computational Linguistics.
* Mitchell et al. (2023) Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In _International Conference on Machine Learning_, pages 24950-24962. PMLR.
* Nahar et al. (2024) Mahjabin Nahar, Haeseung Seo, Eun-Ju Lee, Aiping Xiong, and Dongwon Lee. 2024. Fakes of varying shades: How warning affects human perception and engagement regarding llm hallucinations. _arXiv preprint arXiv:2404.03745_.
* OpenAI (2024) OpenAI. 2024. GPT-40 Mini: Advancing Cost-Efficient Intelligence. Accessed: 2024-09-30.
* Pu et al. (2023) Jiameng Pu, Zain Sarwar, Sifat Muhammad Abdullah, Abdullah Rehman, Yoonjin Kim, Parantapa Bhattacharya, Mobin Javed, and Bimal Viswanath. 2023. Deepfake text detection: Limitations and opportunities. In _2023 IEEE symposium on security and privacy (SP)_, pages 1613-1630. IEEE.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-training.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of machine learning research_, 21(140):1-67.
* Rajani et al. (2023) Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. 2023. No Robots. [https://huggingface.co/datasets/HuggingFaceH4/no_robots](https://huggingface.co/datasets/HuggingFaceH4/no_robots).
* Reid and Neubig (2022) Michel Reid and Graham Neubig. 2022. Learning to model editing processes. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 3822-3832.
* Roberts and Moran (1983) Teresa L Roberts and Thomas P Moran. 1983. The evaluation of text editors: methodology and empirical results. _Communications of the ACM_, 26(4):265-283.
* Solaiman et al. (2019) Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, and Jasmine Wang. 2019. Release Strategies and the Social Impacts of Language Models.
* Su et al. (2023a) Jinyan Su, Terry Zhuo, Di Wang, and Preslav Nakov. 2023a. DetectLLM: Leveraging log rank information for zero-shot detection of machine-generated text. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 12395-12412, Singapore. Association for Computational Linguistics.
* Su et al. (2023b) Zhenpeng Su, Xing Wu, Wei Zhou, Guangyuan Ma, and Songlin Hu. 2023b. Hc3 plus: A semantic-invariant human chatgpt comparison corpus. _arXiv preprint arXiv:2309.02731_.
* Tang et al. (2024) Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. 2024. The Science of Detecting LLM-generated Text. _Communications of the ACM_, 67(4):50-59.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Tripta et al. (2024) Nafis Triiza Tripte, Saranya Venkatraman, Dominik Macko, Robert Moro, Ivan Srba, Adaku Uchendu, Thai Le, and Dongwon Lee. 2024. A ship of theseus: Curious cases of paraphrasing in llvm-generated texts. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024_, pages 6608-6625. Association for Computational Linguistics.
* Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clementine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of LM alignment. _arXiv preprint arXiv:2310.16944_.
* Uchendu et al. (2023) Adaku Uchendu, Thai Le, and Dongwon Lee. 2023. Attribution and obfuscation of neural text authorship: A data mining perspective. _ACM SIGKDD Explorations Newsletter_, 25(1):1-18.
* Uchendu et al. (2020) Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. 2020. Authorship attribution for neural text generation. In _Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP)_, pages 8384-8395.
* Uchendu et al. (2021) Adaku Uchendu, Zeyu Ma, Thai Le, Hui Zhang, and Dongwon Lee. 2021. Turingbench: A benchmark environment for turing test in the age of neural text generation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 2001-2016.

 

Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. 2024. Ghostbuster: Detecting text ghostwritten by large language models. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 1702-1717, Mexico City, Mexico. Association for Computational Linguistics.
* Voronov et al. (2024) Anton Voronov, Lena Wolf, and Max Ryabinin. 2024. Mind Your Format: Towards Consistent Evaluation of In-context Learning Improvements. _arXiv preprint arXiv:2401.06766_.
* Wang et al. (2023) Yiming Wang, Zhuosheng Zhang, and Rui Wang. 2023. Element-aware summarization with large language models: Expert-aligned evaluation and chain-of-thought method. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8640-8665.
* Wang et al. (2024a) Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvignun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, and Thomas Arnold. 2024a. SemEval-2024 task 8: Multidomain, multimodel and multilingual machine-generated text detection. In _Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)_, pages 2057-2079, Mexico City, Mexico. Association for Computational Linguistics.
* Wang et al. (2024b) Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvignun, Osama Mohammad Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024b. M4GTbench: Evaluation benchmark for black-box machine-generated text detection. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3964-3992, Bangkok, Thailand. Association for Computational Linguistics.
* Wang et al. (2024c) Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jiayan Su, Artem Shelmanov, Akim Tsvignun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, Thomas Arnold, Alham Fikri Aji, Niizar Habash, Iryna Gurevych, and Preslav Nakov. 2024c. M4: Multi-generator, multi-domain, and multi-lingual black-box machine-generated text detection. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1369-1407, St. Julian's, Malta. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remin Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Yang et al. (2022) Daijin Yang, Yanpeng Zhou, Zhiyuan Zhang, Toby Jia-Jun Li, and Ray LC. 2022. AI as an Active Writer: Interaction strategies with generated text in human-AI collaborative fiction writing. In _Joint Proceedings of the ACM IUI Workshops_, volume 10, pages 1-11. CEUR-WS Team.
* Yang et al. (2023) Xianjun Yang, Wei Cheng, Yue Wu, Linda Ruth Petzold, William Yang Wang, and Haifeng Chen. 2023. Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text. In _The Twelfth International Conference on Learning Representations_.
* Zhang et al. (2023) Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine translation: A case study. In _International Conference on Machine Learning_, pages 41092-41110. PMLR.
* Zhang et al. (2024a) Qihui Zhang, Chujie Gao, Dongping Chen, Yue Huang, Yixin Huang, Zhenyang Sun, Shilin Zhang, Weije Li, Zhengyan Fu, Yao Wan, and Lichao Sun. 2024a. LLM-as-a-coauthor: Can mixed human-written and machine-generated text be detected? In _Findings of the Association for Computational Linguistics: NAACL 2024_, pages 409-436, Mexico City, Mexico. Association for Computational Linguistics.
* Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In _International Conference on Learning Representations_.
* Zhang et al. (2024b) Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024b. Benchmarking large language models for news summarization. _Transactions of the Association for Computational Linguistics_, 12:39-57.

 



## Appendix A Annotator Profiles

\begin{table}
\begin{tabular}{l r} \hline \hline  & **What gender do you identify as?** \\ Male & 30.0\% \\ Female & 70.0\% \\ Nonbinary / other & 0\% \\ \hline  & **What ethnicity do you identify as?** \\ White / Caucasian & 58.8\% \\ South Asian & 5.9\% \\ Indigenous / Native American / Alaskan Native & 0.0\% \\ East Asian & 0.0\% \\ Middle Eastern & 0.0\% \\ Latinx & 0.0\% \\ Black / African & 17.6\% \\ Mixed / Mixed race & 11.8\% \\ \hline  & **What is your nationality?** \\ American & 17.5\% \\ British & 17.5\% \\ Indian & 5.9\% \\ South African & 17.5\% \\ Canadian & 17.5\% \\ Kenyan & 11.8\% \\ Malaysian & 5.9\% \\ Irish & 5.9\% \\ \hline  & **What is your native language?** \\ English & 75\% \\ Multiple languages including English & 25\% \\ \hline  & **What is your age?** \\
20-29 & 53.2\% \\
30-39 & 17.5\% \\
40-49 & 17.5\% \\
50-59 & 11.8\% \\
60+ & 0\% \\ \hline  & **What is your highest attained level of education?** \\ High school degree & 0.0\% \\ Undergraduate degree & 52.9\% \\ Postgraduate degree & 23.5\% \\ Master's degree & 23.5\% \\ Doctorate degree & 0\% \\ \hline  & **How many years of work experience do you have?** \\
1-3 years & 17.6\% \\
4-6 years & 11.8\% \\
7-9 years & 17.6\% \\
10-12 years & 17.6\% \\
13-15 years & 11.8\% \\
16+ years & 17.6\% \\ \hline \hline \end{tabular}
\end{table}
Table 5: Annotator profiles. Voluntary survey results from 17 respondents.





## Appendix B Annotation guidelines

**Objective** The goal of this task is to refine machine-generated texts to make them more **human-like** in terms of fluency, factual correctness, coherence, and style.

**Task** You are provided with an instruction pair, comprising two components: (1) an instruction given to an AI assistant, such as ChatGPT, and (2) the assistant's response. Your objective is to refine the response, making it sound more human-like and ensuring it is free of errors.

**How many edits should be done?** Attempt to edit 20% to 40% of the provided text. Reject the text if it requires more significant improvements or does not follow the instructions completely. Aim for real-life settings: if a text misses the mark or requires a lot of refinement, most likely, the user would ask the AI assistant for a better response. To **reject** the text, hit the "**Too bad**" button. If there is **nothing to edit** in the text, hit the "**Already perfect**" button and proceed to the next example. No need to fill in the "**Edit the AI Response**" field in either of these two cases.

1. **Read:** Carefully read the provided text, paying attention to the instruction and to how well the AI assistant followed it.
2. **Check:** Check that the prompt matches the category. See the fields **Category** and **Prompt** at the top part of the interface. If the prompt doesn't fall into the specified category, reject the text by hitting the "**Too bad**" button.
3. **Assess:** Evaluate the AI assistant's response for factual correctness, coherence, grammar, style, and overall human-like quality. Identify any errors or areas that need improvement.
4. **Edit:** Make necessary revisions to the AI assistant's response to enhance its human-like quality. This may include rephrasing sentences, correcting grammar mistakes, ensuring coherence, and adding a personal touch to it. See the list of the issues you may want to correct below. We provide the recommended range of edits 20%-40% but it is not strict; make the edits based on your experience in editing and working with data generated by language models.
5. **Proofread:** Review the edited response to ensure it is free of errors and flows smoothly. Check for any remaining issues or inconsistencies.
6. **Submit:** Once you are satisfied with the edits and the AI response meets the criteria of being human-like and error-free, submit your final version.

\begin{table} \end{table}
Table 6: A shortened version of the annotation guidelines for expert-based editing in §3.2 (Part 1). The full version with the editing examples are provided in our GitHub repository at github.com/Toloka/beemo/guidelines.





**Issues of AI-generated text that you may want to correct**

_This list is not exhaustive. Follow your intuition and edit the text where you feel it's reasonable to do so._

1. **Repetitions.** The same word, phrase, or sentence is repeated multiple times. **Do:** Remove repeated words, phrases, and sentences. This is an editing example.
2. **Awkward phrasing.** The text contains awkward or unnatural phrasing that makes it difficult to read or understand. It contains generic and simplistic language or, on the contrary, uses unnecessarily sophisticated and outdated words and phrases. **Do:** Improve the response for natural flow. This is an editing example.
3. **Tone and style.** The tone and style of the text do not align with the intended audience and purpose. **Do:** Adjust the tone and style. This is an editing example.
4. **Texts written from the AI assistant's point of view.** AI assistants tend to add some unnecessary sentences to the beginning or the end of the text. These AI's introductory sentences are unnecessary and should be omitted. **Do:** Remove markers of a text generated by an AI assistant. This is an editing example.
5. **Grammar errors.** The text is ungrammatical and contains unacceptable sentences. **Do:** Edit so the text is grammatically correct, fix the sentence structure and correct the typos. This is an editing example.
6. **Relevance.** The response does not fully follow the provided instruction. **Do:** Remove any irrelevant or unnecessary information from the AI response. Add important details or context. If the AI response is completely irrelevant to the instruction, reject the text. This is an editing example.
7. **Personal touch.** The text is too generic. **Do:** You may want to add a personal touch to make the text more engaging. This is an editing example.
8. **Fact-check the AI response.** Check the facts provided by the AI assistant. If you can fix the facts based on common knowledge or your personal experience, you don't need to provide anything. If you searched online for the information to fix the facts, provide links to the found sources in the "**Comment**" that you used to fact-check the AI response. **Do:** Decide to what degree it is possible to edit the AI response to improve factual correctness. If it requires substantial changes, discard the text. This is an editing example.
9. **Incomplete response.** If the response is interrupted accidentally, complete it. This is an editing example.
10. **Other.** You may find other issues that you want to edit. Let us know about them in "**Comment**".

Example of web interface

**Category** This is a toy category.

**Prompt** This is a toy prompt.

**AI Response** This is a toy LLM's response.

Too bad

Already perfect

**Edit the AI Response**

**Comment**

Proofread before submission. Thank you!



## Appendix C Examples of Common Issues in Machine-generated Texts

\begin{table}
\begin{tabular}{p{142.3pt} p{85.4pt} p{76.9pt} p{108.1pt}} \hline \hline
**Prompt** & **Model Output** & **Human Edit/Correct Answers** & **Explanation** \\ \hline
**Format Inconsistencies** & _Category: Generation (Llama2-7B-chat-fly)_ & & \\ Write a shopping list for everything you need to make burritons. & Sure! Here are five essential items you’ll need for burrions. & 1. Four tortillas & The model-generated list is **too vague and incomplete, making it multiplepli-ful for someone trying to shop for burrits ingredients.** The human edit & **for some tracing trying to shop for burriting ingredients.** The human edit  expands and specifies the required items, ensuring completeness and clarity. \\  & & & \\ \multirow{2}{*}{
\begin{tabular}{c} **H hallucinations and Factual Errors** \\ What are the colors that light up in the children's game Simo? \\ \end{tabular} } & _Category: Open QA (Table-2, 138)_ & 1. Four tortillas 2. Cooked seasoned ground bed or shredded chicken & The model-generated list is **too vague and incomplete, making it multiplepli-ful for someone trying to shop for burrits ingredients.** The human edit  expands and specifies the required items, ensuring completeness and clarity. \\  & & & **full for someone trying to shop for burrits ingredients.** The human edit \\  & & & **burrits ingredients.** The human edit  expands and specifies the required items, ensuring completeness and clarity. \\  & & & **clarity.** \\ \hline \multicolumn{3}{p{142.3pt}}{Simon is a children’s game that involves four colors; red, blue, blue, yellow; and green.} & Simon is a popular electronic game that features four colored buttons: red, blue, yellow; and green. Players must follow an increasing sequence of lights and sounds by pressing the correct colors in order. \\ \hline
**Struggling with Creative Writing** & _Category: Generation (Llama2-7B-chart-fly)_ & & \\ Write a short porn about a sheep adventuring on a hill. & Sure! Here is a short poem about a sheep adventuring on a hill. & A lone sheep climbs the emerald crest, Chains dreams fearless next. Sheaf people osw, the sheep goes down, Wandering freely all around, Soft who used and tiny feet, Through the fields it loves to greet. & This example highlights how the mobile struggles would create creativity, offer denaining to simplistic, predictable rhymes. The human edit improves imagery, rhythm, and poetic depth. \\ \hline
**Complicated and Repetitive Vocabulary** & _Category: Saumarize (Mistral-7B-tastner)_ & & \\ Give me a TLDR of what Professor Kaptchuk is researching. & Professor Ted Kaptechuk of Harvard-affiliated Beth Israel researchers hod doctor-patient interactions and environmental factors contribute the placebo effect. Itis work explores how the brain and body react to placeds, and be is particularly interested in understanding the biological, neurological, and psychological mechanisms underlying placebo responses. He studies how fiedors such as docute-patient interventions, belief systems, and environmental cues influence placeups. His research provides insights into how cluttered places can be integrated into medical treatments to enhance therapeutic outcomes and improve patient care. \\ \hline
**Lack of Natural Flow Repetitive Structure** & _Clategory: Rewrite (Gamma-2B-TIF)_ & The model output is overly long and if FOCS, That’s what you are male life special. You are the most ambitious focus. Love you, & **repetitive, making it sound unnatural for a simple post-it note.** The human edit keeps it **consele, peoti, and engaging** aligning with the task’s request for a short love note. \\  & & & \\ \hline \hline \end{tabular}
\end{table}
Table 8: Examples of common issues in MGTs along with corresponding human edits. The examples are on the issues described in §3.2: (i) format inconsistencies; (ii) hallucinations and factual errors; (iii) challenges in creative writing; (iv) overly complicated and repetitive vocabulary; and (v) lack of natural flow with repetitive structure.



Details on LLM-based Editing

\begin{table}
\begin{tabular}{p{14.2pt} p{390.7pt} p{56.9pt} p{85.4pt}} \hline \hline
**Prompt** & **geema–2b-it** & **Expert-edited** & **Llama3.1–70B-Instruct, edited** \\ P1 & You are given a prompt and a text generated by AI using this prompt. Word task is to edit the Al-generated text to author’s online it sound human-like through these values, they can provide a positive work overall edits do not exceed 40% of the generated text and the edited text follows the other request. Output only the desired context and do not explain your edits. [propnet] & According to the statement, Chick-fil-A’s core values are purpose, culture, and our teamwork. They believe that making it sound human-like through these values, they can provide a positive work overall edits do not exceed 40% of the generated text and the edited text follows the other request. Output only the edited text and do not explain your edits. [propnet] & According to the statement, Chick-fil-A’s core values are purpose, culture, and our teamwork. They believe that drawing them “excluded” from their own counterparts. They believe that making it sound human-like through these values, they can provide a negative work overall edits do not exceed 40% of the generated text and the edited text follows the other request. Output only the edited text and do not explain your edits. [propnet] & According to the statement, Chick-fil-A’s core values are purpose, culture, and our teamwork. They believe that drawing them “excluded” from their own counterparts. They believe that making it sound human-like through these values, they can provide a positive work overall edits do not exceed 40% of the generated text and the edited text follows the other request. Output only the edited text and do not explain your edits. [propnet] & According to the statement, Chick-fil-A’s core values are purpose, Culture, and our teamwork. They believe that drawing them “excluded” from their own counterparts. They believe that drawing them “excluded” from their own counterparts. They believe that drawing them “excluded” from their own counterparts. They believe that drawing them “excluded” from their own counterparts. They believe that drawing them “excluded” from their own counterparts. They believe that drawing them “excluded” from their own counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They argue that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts. They think drawing them “excluded” from their counterparts. They believe that drawing them “excluded” from their counterparts 

Details on Benchmark Analysis

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline  & **BERTScore(M,H)** & **BERTScore(M,E)** & **BERTScore(M,L)** & **BERTScore(M,G)** \\ \hline zephyr-7b-beta & 0.72 \(\pm\) 0.05 & 0.83 \(\pm\) 0.07 & 0.79 \(\pm\) 0.06 & 0.84 \(\pm\) 0.07 \\ tu1u2-2b & 0.73 \(\pm\) 0.06 & 0.85 \(\pm\) 0.07 & 0.79 \(\pm\) 0.06 & 0.86 \(\pm\) 0.07 \\ tulu2-1b & 0.73 \(\pm\) 0.06 & 0.83 \(\pm\) 0.08 & 0.79 \(\pm\) 0.06 & 0.85 \(\pm\) 0.07 \\ gemma-2b-it & 0.71 \(\pm\) 0.06 & 0.81 \(\pm\) 0.09 & 0.77 \(\pm\) 0.06 & 0.83 \(\pm\) 0.07 \\ germa-7b-it & 0.72 \(\pm\) 0.06 & 0.83 \(\pm\) 0.09 & 0.78 \(\pm\) 0.06 & 0.84 \(\pm\) 0.06 \\ Llama2-7b-chat-hf & 0.73 \(\pm\) 0.06 & 0.80 \(\pm\) 0.07 & 0.79 \(\pm\) 0.05 & 0.84 \(\pm\) 0.07 \\ Llama2-13b-chat-hf & 0.71 \(\pm\) 0.06 & 0.80 \(\pm\) 0.09 & 0.76 \(\pm\) 0.07 & 0.81 \(\pm\) 0.08 \\ Llama2-7b0-chat-hf & 0.72 \(\pm\) 0.06 & 0.83 \(\pm\) 0.07 & 0.79 \(\pm\) 0.06 & 0.83 \(\pm\) 0.07 \\ Mistral-7B-Instruct-v0.1 & 0.74 \(\pm\) 0.06 & 0.83 \(\pm\) 0.08 & 0.79 \(\pm\) 0.06 & 0.85 \(\pm\) 0.07 \\ Mistral–8x7B-Instruct-v0.1 & 0.74 \(\pm\) 0.05 & 0.85 \(\pm\) 0.08 & 0.80 \(\pm\) 0.06 & 0.86 \(\pm\) 0.07 \\ Overall & 0.72 \(\pm\) 0.06 & 0.83 \(\pm\) 0.08 & 0.79 \(\pm\) 0.06 & 0.84 \(\pm\) 0.07 \\ \hline \end{tabular}
\end{table}
Table 11: Similarity scores by LLM. **H** = human-written, **M** = machine-generated, **E** = expert-edited; **L**=Llama3.1-70B-Instruct-edited; **G**= GPT-4o-edited; **BERTScore(M, H/E/L/G)**=BERTScore similarity. **L** and **G** are aggregated over three prompts (§3.3).

Figure 5: Comparison of average edit percentages between GPT-4o and Llama3.1-70B-Instruct across three different prompts. The bars represent the mean edit percentage for each prompt, with error bars indicating the standard deviation.

 

Details on Annotation Survey

**Survey: Editing AI-generates Responses**

Answer the following questions related to editing AI-generated responses. The questions are divided into two groups. First, you are asked to judge general issues in the AI-generated responses based on how often you have observed them. Second, you are asked to share your strategies in the editing AI-generated responses in the free form.

_How often do you observe the following issues in the AI-generated responses?_

Rate each issue on a scale from 1 to 5, where 1 stands for "never," 3 stands for "sometimes," and 5 stands for "very often."

**Q1: Repetitions.** The same word, phrase, or sentence is repeated in the AI-generated text multiple times.

\begin{tabular}{c c c c c c c}  & 1 & 2 & 3 & 4 & 5 \\
**Never** & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & **Very often** \\ \end{tabular}

**Q2: Awkward phrasing.** The AI-generated text contains awkward or unnatural phrasing that makes it difficult to read or understand. It contains generic and simplistic language or, on the contrary, uses unnecessarily sophisticated and outdated words and phrases.

\begin{tabular}{c c c c c c}  & 1 & 2 & 3 & 4 & 5 \\
**Never** & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & **Very often** \\ \end{tabular}

**Q3: Tone and style.** The tone and style of the AI-generated text do not align with the user intent.

\begin{tabular}{c c c c c c}  & 1 & 2 & 3 & 4 & 5 \\
**Never** & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & **Very often** \\ \end{tabular}

**Q4: Grammar errors.** The AI-generated text is ungrammatical and does not sound natural.

\begin{tabular}{c c c c c c}  & 1 & 2 & 3 & 4 & 5 \\
**Never** & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & **Very often** \\ \end{tabular}

**Q5: Relevance.** The AI-generated text does not fully follow the user's prompt or complete the target task.

\begin{tabular}{c c c c c c}  & 1 & 2 & 3 & 4 & 5 \\
**Never** & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & **Very often** \\ \end{tabular}

**Q6: Code fragments.** The AI-generated text includes fragments that resemble programming language and are not contextually relevant.

\begin{tabular}{c c c c c c}  & 1 & 2 & 3 & 4 & 5 \\
**Never** & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & **Very often** \\ \end{tabular}

**Q7: Incomplete response.** The AI-generated text is incomplete.

\begin{tabular}{c c c c c c}  & 1 & 2 & 3 & 4 & 5 \\
**Never** & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & **Very often** \\ \end{tabular}

**Q8: Factual inconsistency.** The facts described in the AI-generated texts are wrong.

\begin{tabular}{c c c c c c}  & 1 & 2 & 3 & 4 & 5 \\
**Never** & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & **Very often** \\ \end{tabular}

**Q9: Incoherence.** Segments of the AI-generated text are not connected or contradict each other.

\begin{tabular}{c c c c c c}  & 1 & 2 & 3 & 4 & 5 \\
**Never** & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & \(\circ\) & **Very often** \\ \end{tabular}

Table 12: A survey on AI-generated response issues and editing strategies (Part 1).

 

_Answer the following questions in 2-5 sentences._

1. Do you have specific strategies for editing AI-generated texts?

[frame=none]

2. In your experience, how does editing AI-generated texts compare with editing texts written by humans?

3. What difficulties or challenges have you encountered when editing the AI-generated texts?

Thank you for your time and efforts.

Leave a comment if you want to add something.

[frame=none]

Figure 6: Results of the survey on issues in the AI-generated responses. **Q1**=Repetitions; **Q2**=Awkward phrasing; **Q3**=Tone and style. **Q4**=Grammar errors; **Q5**=Relevance; **Q6**=Code fragments; **Q7**=Incomplete response; **Q8**=Factual inconsistency; **Q9**=Incoherence.

 

Details on MGT Detectors

#### 6.1.3 Zero-shot MgT Detectors

* Binoculars utilizes the ratio of perplexities of a text obtained using two pre-trained LLMs to compute cross-perplexity. This relative measure of perplexity from the lens of two LLMs helps detect LLM-generated texts.
* Log Probability utilizes the average of the log probabilities of each token in a text and classifies text with a higher average log probability as machine-generated. This follows the notion that the probabilities of the tokens, as measured by the LLM that generated them, should be higher.
* Rank. In this approach, text with a higher average rank of each token in the text is determined to be machine-generated text. Rank is calculated by sorting the vocabulary tokens in decreasing order of likelihood.
* Log-Rank. This method classifies text with higher average values as machine-generated by calculating the average of the observed log-ranks of all tokens.
* Entropy attributes higher entropy text as machine-generated. Entropy is calculated as the negative sum of the probability times log probabilities of all tokens at each position of the text.
* DetectLLM has two variations: DetectLLM-LRR, which replaces log-likelihood computation with log-rank, and DetectLLM-NPR, which computes normalized perturbed log-rank. The former is fast and efficient, while the latter is slow but with higher detection accuracy.
* DetectGPT leverages the structure of LLMs' probability curve to identify machine-generated texts. Specifically, it assumes that LLMs' generations tend to occupy negative curvature regions.

**Pretrained MGT Detectors**

* RADAR uses an adversarial learning setup in which a paraphraser LM learns to alter the machine-generated text to evade detection. A detector LM then learns from this adversarially paraphrased text to better its performance.
* AIGC MPU augments LLM training with an additional length-sensitive multiscale positive-unlabeled (MPU) loss term and a text multiscaling module to enhance the training data. The method works with the assumption that MGT can be formulated as a partial Positive-Unlabeled (PU) problem by assuming short-length machine texts as partially unlabeled.
* MAGE is the LongFormer model Beltagy et al. (2020) finetuned on 447K human-written and machine-generated texts collected from a wide range of sources.
* OpenAI's RoBERTa-based detectors: OpenAI released pretrained detectors using the RoBERTa models (base and large; Liu et al., 2019) trained on text generated by GPT-2 



## Appendix H Details on Empirical Evaluation Results

\begin{table}
\begin{tabular}{l l|c c|c|c|c|c|c} \hline \multicolumn{1}{l}{**Detector**} & \multicolumn{2}{c}{**H vs. M**} & \multicolumn{2}{c}{**E vs. M**} & \multicolumn{2}{c}{**H vs. E**} & \multicolumn{2}{c}{**L vs. M**} & \multicolumn{2}{c}{**H vs. L**} & \multicolumn{2}{c}{**G vs. M**} & \multicolumn{2}{c}{**H vs. G**} \\ \hline  & \multicolumn{6}{c}{**Zero-shot MGT Detectors**} \\ \hline
**Binoculars** & 93.64 & 82.84 & 76.19 & 75.26 & 81.80 & 74.44 & 77.03 \\ \hline \multirow{4}{*}{**Log Probability**} & qP2-XL & 74.41 & 67.21 & 63.02 & 72.21 & 56.50 & 68.20 & 61.55 \\  & qP1-1.38 & 82.13 & 70.03 & 70.28 & 76.52 & 63.23 & 70.35 & 71.00 \\  & Falcon-7R & 90.44 & 71.53 & 83.59 & 77.07 & 80.20 & 71.01 & 84.83 \\  & Qem2-7R & 91.14 & 75.88 & 81.57 & 58.62 & 87.95 & 67.23 & 84.76 \\ \hline \multirow{4}{*}{**Rank**} & qP2-XL & 67.87 & 61.35 & 59.02 & 67.60 & 49.58 & 62.81 & 54.87 \\  & qP1-1.38 & 75.29 & 60.95 & 68.88 & 72.03 & 54.29 & 64.33 & 63.56 \\  & Falcon-7R & 83.47 & 62.21 & 78.61 & 72.42 & 65.54 & 64.37 & 74.84 \\  & Qem2-7R & 81.74 & 64.23 & 73.14 & 54.74 & 78.87 & 59.49 & 76.01 \\ \hline \multirow{4}{*}{**Log-Rank**} & qP2-XL & 76.28 & 68.08 & 63.65 & 74.53 & 54.43 & 69.61 & 61.05 \\  & qP1-1.38 & 80.64 & 69.97 & 68.07 & 77.28 & 58.28 & 71.04 & 67.01 \\  & Falcon-7R & 89.56 & 71.44 & 82.15 & 77.95 & 76.94 & 71.55 & 82.50 \\  & Qem2-7R & 90.44 & 74.75 & 80.79 & 58.25 & 87.22 & 66.50 & 84.01 \\ \hline \multirow{4}{*}{**Entropy**} & qP2-XL & 36.66 & 42.43 & 43.56 & 82.97 & 60.51 & 33.93 & 53.34 \\  & qP1-1.38 & 33.92 & 42.95 & 39.53 & 26.89 & 58.58 & 33.40 & 50.31 \\  & Falcon-7R & 19.50 & 37.62 & 23.40 & 25.86 & 38.92 & 30.69 & 32.09 \\  & Qem2-7R & 8.26 & 26.53 & 14.41 & 42.18 & 10.31 & 34.35 & 12.36 \\ \hline \multirow{4}{*}{**DetectLLM**} & qP2-XL & 77.86 & 69.39 & 63.26 & 78.35 & 47.81 & 71.95 & 57.65 \\  & qP1-1.38 & 71.68 & 67.46 & 57.28 & 77.28 & 40.81 & 71.08 & 50.44 \\  & Falcon-7R & 83.28 & 68.72 & 73.11 & 78.37 & 60.35 & 71.06 & 68.87 \\  & Qem2-7R & 84.26 & 68.21 & 73.92 & 56.07 & 80.81 & 62.14 & 77.37 \\ \hline \multirow{4}{*}{**DetectLLM**} & qP2-XL & 71.76 & 66.26 & 69.49 & 58.19 & 70.65 & 53.20 & 71.37 \\  & qP1-1.38 & 73.27 & 66.76 & 70.56 & 63.17 & 71.64 & 54.51 & 72.79 \\  & Falcon-7R & 77.85 & 68.01 & 73.84 & 63.70 & 75.43 & 56.75 & 77.00 \\  & Qem2-7R & 81.68 & 71.68 & 75.23 & 63.20 & 79.00 & 56.41 & 80.81 \\ \hline \multirow{4}{*}{**DetectGFPT**} & qP2-XL & 72.51 & 68.19 & 71.51 & 70.82 & 71.81 & 68.17 & 71.98 \\  & qP1-1.38 & 74.50 & 70.40 & 73.04 & 77.60 & 72.83 & 72.93 & 73.30 \\  & Falcon-7R & 76.74 & 69.53 & 74.95 & 77.85 & 74.29 & 74.60 & 74.68 \\  & Qem2-7R & 78.34 & 73.19 & 75.48 & 78.25 & 75.29 & 75.53 & 75.62 \\ \hline \multirow{4}{*}{**RADAR**} & 64.13 & 55.84 & 60.73 & 22.19 & 81.38 & 27.00 & 78.27 \\  & MAGE & 78.86 & 59.90 & 71.01 & 67.36 & 64.61 & 72.90 & 59.47 \\  & MGC MPU & 75.30 & 71.70 & 55.86 & 56.28 & 72.17 & 47.66 & 77.83 \\ \cline{1-1}  & OpenAI RoBERTa-base & 74.60 & 71.41 & 52.85 & 71.32 & 56.89 & 70.81 & 58.30 \\ \cline{1-1}  & OpenAI RoBERTa-large & 70.45 & 69.87 & 51.16 & 63.07 & 60.52 & 56.74 & 66.00 \\ \hline \end{tabular}
\end{table}
Table 14: The AUROC scores (%) of the 33 MGT detectors’ configurations on Beemo in the **Generation** category. A random classifier has an AUROC of 50%. **H**=human-written; **M**=machine-generated; **E**=expert-edited; L=Llama3.1-7OB-Instruct-edited; **G**=6PT–40-edited. **L** and **G** are aggregated over three prompts (§3.3).





\begin{table}
\begin{tabular}{l l|c c|c|c|c|c|c} \hline
**Detector** & **H vs. M** & **E vs. M** & **H vs. E** & **L vs. M** & **H vs. L** & **G vs. M** & **H vs. G** \\ \hline \multicolumn{8}{c}{**Zero-shot MCT Detectors**} \\ \hline
**Binoculars** & & 85.97 & 81.31 & 56.46 & 74.09 & 69.67 & 70.06 & 69.48 \\ \hline  & GPT2-XL & 76.15 & 75.18 & 50.95 & 70.37 & 60.05 & 60.10 & 70.79 \\
**Log Probability** & OPT-1.3B & 79.22 & 76.83 & 53.02 & 73.49 & 61.33 & 62.16 & 73.14 \\  & Falcon-7B & 92.30 & 75.66 & 80.35 & 72.94 & 86.18 & 58.85 & 92.81 \\  & Quen2-7B & 93.68 & 77.76 & 83.45 & 59.25 & 90.22 & 68.51 & 86.86 \\ \hline  & GPT2-XL & 63.60 & 62.21 & 51.78 & 62.24 & 51.17 & 56.24 & 57.96 \\
**Rank** & OPT-1.3B & 66.54 & 64.21 & 53.64 & 65.54 & 51.78 & 56.92 & 61.35 \\  & Falcon-7B & 77.41 & 62.01 & 68.83 & 65.28 & 64.02 & 54.43 & 75.68 \\  & Quen2-7B & 77.53 & 60.33 & 72.00 & 53.44 & 75.69 & 56.89 & 73.84 \\ \hline  & GPT2-XL & 77.45 & 76.13 & 51.31 & 72.50 & 59.18 & 61.92 & 70.68 \\
**Log-Rank** & OPT-1.3B & 78.12 & 76.84 & 51.38 & 74.30 & 58.45 & 63.33 & 70.66 \\  & Falcon-7B & 91.79 & 75.65 & 79.41 & 73.96 & 84.35 & 60.07 & 91.81 \\  & Quen2-7B & 93.30 & 77.78 & 82.40 & 59.26 & 89.67 & 68.52 & 86.03 \\ \hline  & GPT2-XL & 35.06 & 34.36 & 50.88 & 35.35 & 48.77 & 44.33 & 39.67 \\
**Entropy** & OPT-1.3B & 37.08 & 35.25 & 51.85 & 34.43 & 51.53 & 42.63 & 43.20 \\  & Falcon-7B & 12.26 & 32.29 & 20.55 & 31.01 & 19.34 & 40.37 & 13.64 \\  & Quen2-7B & 6.10 & 24.96 & 12.57 & 41.65 & 8.26 & 33.31 & 10.41 \\ \hline  & GPT2-XL & 74.80 & 73.75 & 50.88 & 72.67 & 54.49 & 62.71 & 65.20 \\
**DetectLLM** & OPT-1.3B & 69.69 & 72.35 & 46.67 & 72.13 & 48.71 & 62.79 & 59.14 \\
**Likelihood Log-Rank Ratio** & Falcon-7B & 84.28 & 71.15 & 69.35 & 72.70 & 70.31 & 61.47 & 79.23 \\  & Quen2-7B & 86.58 & 72.14 & 71.58 & 57.38 & 81.58 & 64.76 & 76.58 \\ \hline  & GPT2-XL & 55.89 & 67.17 & 53.56 & 59.94 & 54.54 & 50.94 & 55.97 \\
**DetectLLM** & OPT-1.3B & 56.81 & 68.31 & 54.02 & 64.00 & 54.75 & 52.58 & 56.60 \\
**Normalized Perturbed Log-Rank** & Falcon-7B & 63.13 & 68.05 & 58.78 & 64.72 & 59.87 & 52.75 & 62.89 \\  & Quen2-7B & 68.05 & 72.53 & 60.61 & 64.39 & 64.04 & 51.52 & 67.94 \\ \hline  & GPT2-XL & 58.80 & 69.55 & 55.31 & 62.10 & 56.40 & 58.44 & 57.30 \\
**DetectGPT** & OPT-1.3B & 61.25 & 75.15 & 56.14 & 69.21 & 57.05 & 63.20 & 58.59 \\  & Falcon-7B & 65.51 & 71.80 & 60.04 & 66.81 & 60.93 & 62.49 & 62.44 \\  & Quen2-7B & 68.14 & 74.24 & 61.11 & 62.56 & 64.31 & 62.64 & 64.70 \\ \hline RADAR & 47.78 & 46.69 & 51.43 & 37.84 & 57.84 & 40.62 & 55.58 \\
**MAGe** & 76.05 & 64.39 & 63.31 & 58.99 & 70.21 & 62.23 & 67.34 \\
**AIGC MPU** & 75.33 & 79.87 & 40.69 & 85.27 & **31.55** & 80.63 & 39.59 \\
**OpenAI RoBERTa-base** & 76.68 & 76.55 & 48.95 & 80.72 & 41.85 & 78.99 & 43.93 \\
**OpenAI RoBERTa-large** & 72.60 & 71.87 & 51.46 & 78.94 & 40.87 & 73.95 & 47.36 \\ \hline \end{tabular}
\end{table}
Table 15: The AUROC scores (%) of the 33 MQT detectors’ configurations on Beemo in the **Open QA** category. A random classifier has an AUROC of 50%. **H**=human-written; **M**=machine-generated; **E**=expert-edited; **L**=_lama3.1–7bB-Instruct-edited; **G**=GTP-4o-edited. **L** and **G** are aggregated over three prompts (§3.3).





\begin{table}
\begin{tabular}{l l|c c|c|c|c|c|c} \hline
**Detector** & **H vs. M** & **E vs. M** & **H vs. E** & **L vs. M** & **H vs. L** & **G vs. M** & **H vs. G** \\ \hline \multicolumn{8}{c}{**Zero-shot MCT Detectors**} \\ \hline
**Binoculars** & & 8025 & 76.05 & 55.97 & 88.28 & 88.65 & 39.19 & 85.97 \\ \hline  & GPT-XL & 70.21 & 64.29 & 57.11 & 54.74 & 67.61 & 59.37 & 61.70 \\
**Log Probability** & OPT-1.3B & 74.95 & 66.61 & 60.53 & 57.90 & 70.88 & 59.25 & 67.11 \\  & Falcon-7B & 87.88 & 63.56 & 81.00 & 54.90 & 88.29 & 55.22 & 84.99 \\  & Quen2-7B & 90.41 & 67.70 & 81.50 & 55.90 & 87.44 & 61.80 & 84.47 \\ \hline  & GPT-XL & 57.24 & 51.93 & 55.43 & 53.79 & 54.16 & 53.28 & 54.16 \\
**Rank** & OPT-1.3B & 62.02 & 53.70 & 58.76 & 57.24 & 55.81 & 52.42 & 59.73 \\  & Falcon-7B & 70.81 & 49.83 & 70.69 & 55.17 & 66.15 & 49.63 & 70.25 \\  & Qwen2-7B & 70.71 & 51.04 & 70.21 & 50.35 & 70.54 & 50.69 & 70.38 \\ \hline  & GPT-XL & 69.88 & 63.95 & 57.05 & 56.47 & 65.67 & 60.01 & 60.51 \\
**Log-Rank** & OPT-1.3B & 73.26 & 65.56 & 59.45 & 58.76 & 67.94 & 59.99 & 64.40 \\  & Falcon-7B & 87.02 & 62.35 & 80.41 & 56.08 & 86.54 & 56.16 & 83.39 \\  & Qwen2-7B & 90.04 & 66.86 & 81.06 & 55.62 & 87.05 & 61.24 & 84.05 \\ \hline  & GPT-XL & 35.56 & 42.34 & 42.83 & 42.84 & 41.56 & 44.49 & 40.63 \\
**Entropy** & OPT-1.3B & 34.05 & 42.23 & 41.07 & 41.60 & 40.85 & 44.02 & 39.29 \\  & Falcon-7B & 16.02 & 45.02 & 18.24 & 45.45 & 15.79 & 47.90 & 16.29 \\  & Qwen2-7B & 5.78 & 29.58 & 13.81 & 43.19 & 8.46 & 36.30 & 11.13 \\ \hline  & GPT-XL & 64.75 & 59.97 & 55.57 & 59.14 & 56.95 & 59.89 & 54.74 \\
**DetectLLM** & OPT-1.3B & 64.29 & 60.76 & 54.67 & 60.52 & 55.65 & 60.95 & 53.84 \\
**Likelihood Log-Rank Ratio** & Falcon-7B & 78.55 & 56.93 & 73.60 & 58.83 & 73.77 & 58.20 & 71.78 \\  & Open2-7B & 83.14 & 62.02 & 74.66 & 54.01 & 80.31 & 58.01 & 77.49 \\ \hline  & GPT-XL & 61.93 & 56.39 & 60.78 & 45.90 & 62.60 & 49.25 & 62.08 \\
**DetectLLM** & OPT-1.3B & 63.48 & 59.90 & 61.74 & 51.23 & 63.23 & 49.81 & 63.60 \\
**Normalized Perturbed Log-Rank** & Falcon-7B & 68.11 & 58.33 & 66.39 & 49.17 & 68.32 & 47.17 & 68.82 \\  & Qwen2-7B & 72.17 & 66.17 & 68.07 & 45.76 & 72.91 & 47.08 & 72.89 \\ \hline  & GPT-XL & 60.80 & 59.71 & 59.88 & 46.92 & 62.09 & 53.66 & 60.98 \\
**DetectGPT** & OPT-1.3B & 62.98 & 66.23 & 60.66 & 52.93 & 62.99 & 55.29 & 62.72 \\  & Falcon-7B & 67.54 & 67.08 & 64.56 & 53.33 & 67.41 & 56.32 & 66.91 \\  & Qwen2-7B & 70.40 & 72.84 & 65.65 & 48.42 & 71.07 & 56.65 & 69.44 \\ \hline RADAR & 41.12 & 46.81 & 44.16 & 41.96 & 46.88 & 45.31 & 44.75 \\
**MAGe** & 69.88 & 59.94 & 61.09 & 46.81 & 72.77 & 48.72 & 70.96 \\
**AIGC MPU** & 71.85 & 68.09 & 54.00 & 70.64 & 50.94 & 64.91 & 56.09 \\
**OpenAI RoBERTa-base** & 49.03 & 55.56 & 42.30 & 55.24 & 42.88 & 55.23 & 43.43 \\
**OpenAI RoBERTa-large** & 47.66 & 52.87 & 44.33 & 55.00 & 42.13 & 52.18 & 45.54 \\ \hline \end{tabular}
\end{table}
Table 16: The AUROC scores (%) of the 33 MOT detectors’ configurations on Beemo in the **Summarize** category. A random classifier has an AUROC of 50%. **H**=human-written; **M**=machine-generated; **E**=expert-edited; **L**=_lama3.1–7OB-Instruct-edited; **G**=GTP-4o-edited. **L** and **G** are aggregated over three prompts (§3.3).





\begin{table}
\begin{tabular}{l l|c c|c|c|c|c|c} \hline
**Detector** & **H vs. M** & **E vs. M** & **H vs. E** & **L vs. M** & **H vs. L** & **G vs. M** & **H vs. G** \\ \hline \multicolumn{8}{c}{**Zero-shot MCT Detectors**} \\ \hline
**Binoculars** & & 83.83 & 77.66 & 61.90 & 49.35 & 82.51 & 46.54 & 81.91 \\ \hline  & GPT-XL & 66.56 & 61.63 & 56.03 & 63.39 & 54.23 & 62.44 & 54.77 \\
**Log Probability** & OPT-1.3B & 72.73 & 62.53 & 62.20 & 65.91 & 59.42 & 62.31 & 62.24 \\  & Falcon-7B & 84.17 & 62.26 & 77.70 & 64.38 & 77.68 & 61.58 & 78.08 \\  & Qwen2-7B & 84.78 & 65.91 & 75.18 & 55.30 & 81.58 & 60.61 & 78.38 \\ \hline  & GPT-XL & 56.44 & 52.87 & 54.43 & 57.68 & 48.82 & 56.91 & 49.23 \\
**Rank** & OPT-1.3B & 64.11 & 53.05 & 62.36 & 61.82 & 53.20 & 56.43 & 58.19 \\  & Falcon-7B & 72.94 & 53.61 & 71.12 & 61.98 & 62.50 & 56.47 & 67.63 \\  & Qwen2-7B & 73.23 & 56.84 & 68.20 & 52.28 & 71.55 & 54.56 & 69.88 \\ \hline  & GPT-XL & 66.19 & 61.05 & 56.18 & 64.29 & 52.51 & 62.85 & 53.66 \\
**Log-Rank** & OPT-1.3B & 70.04 & 61.91 & 59.80 & 66.24 & 55.44 & 62.98 & 58.26 \\  & Falcon-7B & 82.13 & 61.80 & 75.54 & 65.03 & 73.98 & 62.24 & 74.56 \\  & Qwen2-7B & 83.28 & 65.37 & 73.57 & 55.12 & 80.04 & 60.25 & 76.81 \\ \hline  & GPT-XL & 45.41 & 47.10 & 48.15 & 36.49 & 58.94 & 41.54 & 53.82 \\
**Entropy** & OPT-1.3B & 42.17 & 47.53 & 44.18 & 34.54 & 57.56 & 40.72 & 51.19 \\  & Falcon-7B & 26.84 & 46.09 & 28.43 & 36.53 & 37.46 & 41.45 & 33.24 \\  & Qwen2-7B & 14.62 & 34.95 & 22.56 & 44.98 & 17.27 & 39.97 & 19.91 \\ \hline  & GPT-XL & 63.75 & 58.94 & 55.98 & 65.69 & 47.73 & 63.35 & 49.88 \\
**DetectLLM** & OPT-1.3B & 58.88 & 58.20 & 51.28 & 65.42 & 43.10 & 63.18 & 45.27 \\
**Likelihood Log-Rank Ratio** & Falcon-7B & 71.86 & 58.70 & 65.68 & 65.17: & 58.81 & 63.02 & 60.02 \\  & Qwen2-7B & 74.21 & 61.41 & 65.15 & 53.80 & 71.19 & 57.61 & 68.17 \\ \hline  & GPT-XL & 64.30 & 62.54 & 62.24 & 50.66 & 64.13 & 50.25 & 64.31 \\
**DetectLLM** & OPT-1.3B & 65.81 & 63.51 & 63.35 & 52.73: & 65.25 & 50.47 & 65.87 \\
**Normalized Perturbed Log-Rank** & Falcon-7B & 70.27 & 63.41 & 66.98 & 53.06 & 69.74 & 51.08 & 70.34 \\  & Qwen2-7B & 73.57 & 69.32 & 67.83 & 48.16 & 74.28 & 49.63 & 74.00 \\ \hline  & GPT-XL & 64.00 & 63.18 & 61.76 & 60.89 & 62.23 & 61.87 & 61.97 \\
**DetectGPT** & OPT-1.3B & 66.43 & 66.27 & 63.51 & 66.98 & 63.61 & 65.36 & 63.79 \\  & Falcon-7B & 69.91 & 64.56 & 67.07 & 69.00 & 66.68 & 68.00 & 66.68 \\  & Qwen2-7B & 71.48 & 69.51 & 67.43 & 64.94 & 68.96 & 67.85 & 68.12 \\ \hline RADAR & & 57.24 & 54.46 & 53.01 & 24.69 & 77.32 & 27.91 & 74.72 \\
**MAGe** & 70.67 & 59.56 & 62.05 & 59.23 & 62.77 & 60.29 & 61.34 \\
**AIGC MPU** & & 71.29 & 69.80 & 52.30 & 52.49 & 70.62 & 47.35 & 73.71 \\
**OpenAI RoBERTa-base** & & 65.61 & 64.11 & 51.08 & 58.16 & 59.77 & 58.70 & 58.83 \\
**OpenAI RoBERTa-large** & & 63.01 & 62.77 & 49.84 & 56.49 & 57.51 & 52.17 & 61.49 \\ \hline \end{tabular}
\end{table}
Table 17: The AUROC scores (%) of the 33 MGT detectors’ configurations on Beemo in the **Rewrite** category. A random classifier has an AUROC of 50%. **H**=human-written; **M**=machine-generated; **E**=expert-edited; **L**=_lama3.1–7OB-Instruct-edited; **G**=GTP-4o-edited. **L** and **G** are aggregated over three prompts (§3.3).





\begin{table}
\begin{tabular}{l l|c c|c|c|c|c|c} \hline
**Detector** & **H vs. M** & **E vs. M** & **H vs. E** & **L vs. M** & **H vs. L** & **G vs. M** & **H vs. G** \\ \hline \multicolumn{8}{c}{**Zero-shot MGT Detectors**} \\ \hline
**Binoculars** & 74.83 & 76.09 & 49.49 & 52.07 & 75.44 & 47.29 & 76.63 \\ \hline  & GPT2-XL & 70.06 & 63.69 & 57.60 & 54.76 & 68.95 & 54.42 & 67.08 \\
**Log Probability** & OPT-1.3B & 71.61 & 65.11 & 58.35 & 56.31 & 69.54 & 54.34 & 68.67 \\  & Falcon-7B & 83.29 & 60.60 & 76.37 & 53.72 & 83.81 & 51.56 & 82.89 \\  & Qwen2-7B & 87.08 & 64.19 & 76.77 & 54.73 & 83.64 & 59.46 & 80.21 \\ \hline  & GPT2-XL & 54.86 & 54.46 & 50.74 & 52.09 & 53.42 & 52.24 & 52.94 \\
**Rank** & OPT-1.3B & 58.22 & 57.86 & 51.15 & 55.57 & 54.44 & 53.75 & 55.08 \\  & Falcon-7B & 64.51 & 55.45 & 59.75 & 54.25 & 61.84 & 50.23 & 64.21 \\  & Qwen2-7B & 65.20 & 55.52 & 59.51 & 51.84 & 63.30 & 53.68 & 61.41 \\ \hline  & GPT2-XL & 70.84 & 63.71 & 58.69 & 56.00 & 68.61 & 54.63 & 67.50 \\
**Log-Rank** & OPT-1.3B & 70.90 & 64.72 & 57.83 & 56.64 & 68.26 & 54.80 & 67.49 \\  & Falcon-7B & 82.52 & 60.46 & 75.92 & 54.05 & 82.97 & 52.14 & 81.66 \\  & Qwen2-7B & 87.11 & 64.87 & 76.75 & 54.96 & 83.66 & 59.91 & 80.20 \\ \hline  & GPT2-XL & 31.34 & 42.60 & 37.21 & 46.67 & 31.46 & 48.52 & 31.83 \\
**Entropy** & OPT-1.3B & 31.09 & 41.61 & 38.04 & 46.15 & 31.67 & 48.47 & 31.60 \\  & Falcon-7B & 18.33 & 46.14 & 19.95 & 49.14 & 16.26 & 50.68 & 16.97 \\  & Qwen2-7B & 10.18 & 38.04 & 15.74 & 46.01 & 12.03 & 42.03 & 13.89 \\ \hline  & GPT2-XL & 69.18 & 60.24 & 59.98 & 57.93 & 64.22 & 54.47 & 65.69 \\
**DetectLLM** & OPT-1.3B & 65.08 & 61.43 & 55.58 & 56.15 & 61.35 & 54.87 & 61.09 \\
**Likelihood Log-Rank Ratio** & Falcon-7B & 74.48 & 58.24 & 68.55 & 53.86 & 73.54 & 53.23 & 72.05 \\  & Qwen2-7B & 80.44 & 62.98 & 70.09 & 54.33 & 76.99 & 58.65 & 73.54 \\ \hline  & GPT2-XL & 62.14 & 62.06 & 60.72 & 56.56 & 61.28 & 55.24 & 61.61 \\
**SpectellLM** & OPT-1.3B & 62.62 & 61.25 & 61.30 & 56.50 & 61.69 & 52.51 & 62.34 \\
**Normalized Perturbed Log-Rank** & Falcon-7B & 64.85 & 58.15 & 63.51 & 52.37 & 64.32 & 52.22 & 64.59 \\  & Qwen2-7B & 68.00 & 66.66 & 64.54 & 52.47 & 67.06 & 55.54 & 67.17 \\ \hline  & GPT2-XL & 63.87 & 60.33 & 61.14 & 52.12 & 62.50 & 53.01 & 62.68 \\
**DetectGPT** & OPT-1.3B & 65.05 & 63.49 & 62.05 & 55.13 & 63.12 & 52.66 & 63.94 \\  & Falcon-7B & 67.20 & 60.53 & 64.05 & 50.31 & 65.62 & 52.63 & 65.71 \\  & Qwen2-7B & 68.80 & 66.55 & 64.67 & 46.95 & 67.81 & 55.01 & 66.92 \\ \hline RADAR & 39.41 & 52.57 & 34.99 & 71.24 & 13.34 & 74.10 & 11.02 \\
**MAGe** & 71.88 & 64.21 & 58.50 & 46.43 & 74.24 & 43.57 & 77.66 \\
**AIGC MPU** & 76.62 & 73.45 & 52.60 & 68.83 & 56.73 & 65.70 & 58.58 \\
**OpenAI RoBERTa-base** & 57.17 & 62.46 & 42.64 & 65.45 & 38.65 & 70.30 & 33.11 \\
**OpenAI RoBERTa-large** & 48.72 & 57.38 & 40.69 & 74.68 & 22.41 & 69.15 & 28.56 \\ \hline \end{tabular}
\end{table}
Table 18: The AUROC scores (%) of the 33 MGT detectors’ configurations on Beemo in the **Closed QA** category. A random classifier has an AUROC of 50%. **H**=human-written; **M**=machine-generated; **E**=expert-edited; **L**=_lama3.1–7OB-Instruct-edited; **G**=GTP-4o-edited. **L** and **G** are aggregated over three prompts (§3.3).