TSM-Bench: Detecting LLM-Generated Text in Real-World Wikipedia Editing Practices
Authors: Gerrit Quaremba
Elizabeth Black
Denny Vrandecic
Elena Simperl
Venue: ICLR 2026 Poster (2026)
Keywords: llm-generated text detection
editing tasks
wikipedia
benchmark
multilingual
Primary area: datasets and benchmarks
TL;DR: We propose an LLM-generated text detection benchmark for realistic editing tasks and show that state-of-the-art detectors considerably underperform.

Abstract: Automatically detecting machine-generated text (MGT) is critical to maintaining the knowledge integrity of user-generated content (UGC) platforms such as Wikipedia. 
Existing detection benchmarks primarily focus on \textit{generic} text generation tasks (e.g., ``Write an article about machine learning.'').  
However, editors frequently employ LLMs for specific writing tasks (e.g., summarisation).  
These \textit{task-specific} MGT instances tend to resemble human-written text more closely due to their constrained task formulation and contextual conditioning. 
In this work, we show that a range of MGT detectors struggle to identify task-specific MGT reflecting real-world editing on Wikipedia.  
We introduce \textsc{TSM-Bench}, a multilingual, multi-generator, and multi-task benchmark for evaluating MGT detectors on common, real-world Wikipedia editing tasks.  
Our findings demonstrate that (\textit{i}) average detection accuracy drops by 10--40\% compared to prior benchmarks, and (\textit{ii}) a generalisation asymmetry exists: fine-tuning on task-specific data enables generalisation to generic data---even across domains---but not vice versa. 
We demonstrate that models fine-tuned exclusively on generic MGT overfit to superficial artefacts of machine generation.  
Our results suggest that, in contrast to prior benchmarks, most detectors remain unreliable for automated detection in real-world contexts such as UGC platforms.  
\textsc{TSM-Bench} therefore provides a crucial foundation for developing and evaluating future models.

Published as a conference paper at ICLR 2026
TSM-BENCH: DETECTING LLM- LLM-GENERATED TEXT IN REAL- WORLD WIKIPEDIA EDITING PRACTICES
Gerrit Quaremba', Elizabeth Black', Denny Vrandečic Elleaa Simperl'
'King's College London, Wikimedia Foundation
gerrit quaremba@kcl. ac. uk
ABSTRACT
Automatically detecting machine-generated text (MGT) is critical to maintain- ing the knowledge integrity of user-generated content (UGC) platforms such as Wikipedia. Existing detection benchmarks primarily focus on generic text gen- eration tasks (e.g., "Write an article about machine learning."), However, editors frequently employ LLMs for specific writing tasks (c.g., summarisation). These task-specific MGT instances tend to resemble human-written text more closely due their constrained task formulation and contextual conditioning. this work, we show that range of SOTA MGT detectors struggle to identify task-specific MGT reflecting real world editing on Wikipedia. We introduce lingual, multi- generator, and multi-task benchmark for evaluating MGT detectors on common, real-world Wikipedia editing tasks. Our findings demonstrate that (i) average detection accuracy drops by 10-40% compared to prior benchmarks, and (ii) generalisation asymmetry exists: fine-tuning on task-specific data en- ables generalisation
a multi-
generic data We demonstrate that models fine-tuned exclusively on generic MGT overfit to su- perficial artefacts of machine generation. Our results suggest that, in contrast prior benchmarks, most detectors remain unreliable for automated detection in real-world contexts such as UGC platforms. TSM-BENCY therefore provides a critical foundation for developing and evaluating future models.
across domains not vice versa.
INTRODUCTION
Wikipedia serves vital source of reliable human-written text (HWT) for the artificial intelli- gence (AI) community. As one of the largest high-quality multilingual corpora on the internet, features in the training data of most large language models (LLMs) (Deckelmann, 2023; Longpre et al., 2024). However, the Wikimedia Foundation warns that the proliferation of maachne--gneraatee text (MGT) across Wikipedia could undermine its knowledge integrity." The unchecked spread of MGT risks degrading the very data underpinning much of recent progress in AI. Generative mod- els trained on uncurated data may deteriorate over time, potentially resulting in fatal model col- lapse (Shumailov et al., 2024). Therefore, differentiating MGT from HWT is essential task with wide-ranging downstream implications, resulting in automatic MGT detection becoming an active area of research (Wu et al., 2025a).
Prior work on benchmarking MGT detectors (c.g. Guo et al., 2023; Macko et al., 2023; Li et al., 2024; He al., 2024; Wang et al., 2024b;a; Wu et al., 2024) has largely relied on simple text gen- eration prompts such as: "Write an article about machine learning." In practice, however, editors typically employ LLMs support a range of specific writing tasks (Ford et al., 2023; Zhou al., 2025). Compared with earlier work's free-form generic text generation, prompts in real-world edit- ing scenarios are narrower in scope and often contextually constrained (e.g., summarisation). We refer to this as task-specific text generation. Crucially, generic and task-specific text differ in that the former is often linguistically and semantically less similar human text, whereas the latter- because of its constraints -tenstteddss to align more closely in style and meaning. Figure 1 illustrates this distinction across four metrics, comparing HWT, generic and the task-specific MGT generated
Wikipedia Community Call Notes 2023-24

---

Published as a conference paper at ICLR 2026
0.175
0.155
0.125
Demsby
0.000
0.052
Norm. Levenshtein Distance
Covime Similarit
1.0
Unigram Duerlap
0.002
30
Perplesity
200
Human
Paragraph Wriking
Summarication
Style
Figure 1: Comparison of textual characteristics between human text, generic MGT, and three task- specific English MGTs. Task-specific MGT's more closely resemble human text. The same pattern observed in other languages (see Appendix D.2).
for the three Wikipedia editing tasks considered in this study. As detectors learn from such textual patterns, well established that detection performance decreases as the total variation distance between human and machine distributions narrows (Sadasivan et al., 2023). We therefore expect de- tectors to face greater challenges on task-specific data than on prior benchmarks limited to generic data. Assessing their reliability in more realistic MGT scenarios is critical, as detectors safeguard the content integrity of UGC and thus ensure high-quality, uncontaminated data for downstream use across diverse Al applications.
Benchmark Construction
Tadls
Antonnaio Poueeg Brodeation
MGT Geevration
182,100 L.M
Wil pe boloe Maaciie
CETERVER:
Perg Costiration
Expertments
Off dulff comettos
Sucauri Bluntick slot
opervised
Text Trunddr
Fazur xulpic
Mard Qua 25-78
cutrodaa laaning
Outof-kerain geeenisaaice
Coosttll grametiction
Figure 2: Overview of TSM-BENCH: ! We define four editing tasks informed by research how editors employ LLMs. 2 For each task, we adopt two prompts from the natural language generation literature and automatically evaluate them against simple baseline. 3 Using the highest-scoring prompt, we generate MGT from six LMs. 4 Finally, we run five experiments on these data and draw key conclusions about the effectiveness of detectors in identifying real-world MGT instances.
In this work, we introduce Task-Specific MGT Benchmark generator, and multi-task MGT detection benchmark (see Figure 2), designed to move beyond generic MGT detection and evaluate detectors on text that more closely reflects how users em- ploy LLMs in real-world workflows. Drawing real-world accounts of how Wikipedia editors LLMs (Ford et al., 2023; Zhou et al., 2025), we define four common editing tasks. For each task, adopt two prompts from the natural language generation literature and compare them with minimal baseline using automatic metrics. We then employ the highest-scoring prompts to generate MGT with six LMs of varying sizes across three languages (English, Portuguese, and Vietnamese). We select languages with different resource levels to study communities beyond the frequently ex- amined English Wikipedia. We define resource level using two indicators: (1) the number of active Wikipedia users and (2) the language's is share in the Common Crawl corpus,² standard training source for LLMs. 3 On these data, we conduct extensive experiments benchmarking 12 SOTA de-
a multilingual, multi-
'ttpps: : / www.///cwww.....oom oorg/
"Appendix Table reports the metrics used guide language selection.
2

---

Published as a conference paper at ICLR 2026
tectors from different model families, testing their out-of-domain and cross-task generalisability. Finally, to strengthen our case for moving beyond generic MGT, we analyse feature importance in models trained on generic versus task-specific data.
Our contributions are as follows:
Benchmark We introduce TSM-BENCH, a multilingual, multi-generraorr, and multi-task bench- mark for task-specific MGT detection Wikipedia, comprising 152,910 MGTss We among the first to study task-specific MGT detection. We plan to maintain this benchmark by adding more detectors, tasks, and languages. Code and data are available GitHub.
Experiments We evaluate detectors from different model families, assess their generalisability across domains and tasks, and conduct a feature analysis to identify the linguistic cues detectors exploit.
Results We demonstrate that (i) accuracy decreases by up to 32%, 20%, and 10% for zero-shot, off-the-shelf, and supervised detectors, respectively, compared evaluations on generic data; and (ii) a generalisation asymmetry exists: models fine-tuned on task-specific data generalise to generic MGT both within and across domains, but not vice versa. Feature importance analysis shows that models trained on generic data overfit to LLM artefacts, exposing limitation of prior benchmarks.
Implications Compared with prior MGT benchmarks, our results suggest that earlier evaluations likely overestimated detector performance due to simplified generation settings, In practice, our findings indicate that most detectors cannot reliably support automated MGT detection in real- world contexts such UGC platforms. TSM-BENCH therefore provides a valuable foundation for developing and evaluating more robust detectors, and we recommend training future models on a diverse combination of task seskifpec data.
2 RELATED WORK
Wikipedia Editing Tasks Wikipedia articles consist of lead section, tabular infobox, and body organised into sections. Their content is written and maintained by volunteer editors who per- form wide range of tasks (Johnson et al., 2024). Paragraph Writing involves generating new en- cyclopaedic content, which is central to expanding knowledge on Wikipedia. Research has focused on expanding Wikipedia with agentic systems (Shao et al., 2024) or through RAG (Zhang et al., 2024b). Summarisation refers to producing the lead section of the article body, which introduces its most important points.* 4 The literature treats lead section generation either as a multi noctmet (Liu et al., 2018; 8; Gholipour Ghalandari et al., 2020; Hayashi et al., 2021) al., 2021; Perez-Beltrachini & Lapata, 2021) summarisation task. Text Style Transfer (TST) the task of modifying the style of a sentence while preserving its meaning (Toshevska & Gievska, 2022). On Wikipedia, maintaining a Neutral Point of View (NPOV) is a core content policy requir- ing all content to be written from neutral perspective. Pryzant et al. (Pryzant al., 2020) introduce the Wikipedia Neutrality Corpus (WNC), a large-scale collection of biased and neutralised sentence pairs retrieved from NPOV-related revisions.
single-document (Gao
MGT Detection Benchmarks Extensive work has benchmarked MGT detectors across domains, languages, and generators (Wu et al., 2025a). TuringBench (Uchendu et al., 2021) is one of the first benchmarks to study the Turing test and authorship attribution, using multiple generators in the news domain. MULTTTuDE (Macko et al., 2023) expands MGT data beyond English, testing detectors in multilingual settings. MAGE (Li et al., 2024) covers multiple domains, generators, and detectors, benchmarking across eight increasingly challenging detection scenarios. M4 (Wang et al., 2024b) comprchensively includes various generators, languages, and domains, while M4GT (Wang et al., 2024a) expands M4 by incorporating additional languages and introducing human-machin mixed detection. Alongside the release of detection datasets (Guo et al., 2023; Su et al., 2023b; Yu et al., 2025), recent work has increasingly focused on adversarial attacks to evade detectors (He et al., 2024; Wu et al., 2024; Zheng et al., 2025).
wtwkkiiiwwwiediieeeeee Manual_on_aaaaaa hhttps::/en...ikiipddia... /iikiee Neutral laraallppotttttt view
3

---

Published as a conference paper at ICLR 2026
Compared to prior MGT detection benchmarks, work is among the first to consider MGT arising from task-specific scenarios. Most existing work relies on generic prompts such as "Write an article about neural networks." We illustrate the of such generic prompts in related work (Wang et al., 2024b;a; Li et al., 2024; Macko et al., 2023) in Appendix 3. In contrast, our benchmark uses task- specific prompts defined in Section 3.1.
Al-Assisted Editing Wikipedia Wikipedia has long history of Al-assisted tools. For ex- ample, ORES (Halfaker Geiger, 2020) provides edit and page-quality predictions using multiple independent classifiers. Descartes (Sakota et al., 2023) generates short descriptions for Wikipedia articles using mBART. The emergence of LLMs has prompted researchers to investigate their impact on Wikipedia. Both Reeves et al. (2025) and Huang al. (2025) find only marginal evidence of an effect on user engagement. Brooks et al. (2024) attempt to identify MGT on Wikipedia and estimate that roughly 5% of new articles may be Al-generated.
This paper extends our previous work (Quaremba et al., 2025), in which we introduced two datasets (WikiPS and mWNC) in English, Portuguese, and Vietnamese. TSM-BENCH extends the bench- mark by adding more tasks, detectors, and LLMs, and provides more extensive experiments includ- ing evaluations of SOTA detectors, domain and cross-task generalisation, and reauure-importance analyses. This allows us to draw critical conclusions about the reliability of detectors on UGC platforms such as Wikipedia.
TSM-BENCH
TSM- TSM-BENCH BENCH a multilingual, multi-generator, and multi-task MGT detection benchmark designed reflect real-world, task-specific LLM--gnerated text Wikipedia. Our tasks empirically grounded in research on Wikipedia editors' perceived use cases for L.LM-assisted editing (Ford al., 2023; Zhou et al., 2025), ensuring their relevance to practical applications. Our benchmark corpus comprises 152,910 parallel human- and machine-written texts across tasks, languages, and generators. Appendix Table 4 presents dataset statistics.
3.1 TASK DEFINITIONS
MGT Detection We define MGT detection as a binary classification task. Given dataset D {( yi ) each instance consists of text 2; and a label E 1]. where 0 denotes a human- written text and a machine- generated machine-generated text. detector learns a function f X + R that assigns real-valued score to each text r E X. Using threshold the predicted label is defined as ÿ = 1 if f=) 2 r and ğ = 0 otherwise.
Task-specific Generation Let fo denote language model with parameters that produces tex- tual output o. Let I = fi1,...,4c it} be set of detailed user instructions for t natural language generation tasks, and let C. = (c........... J be set of contexts associated with task For example, Ct may consist of retrieved evidence passages used to generate new paragraph. We define generic generation Og. fo(g)), where 9: an unconstrained, free-form prompt with minimal task instruction for task l. We define task-specific generation O₂ = fo(is, Ct), where the model com- pletes constrained task using additional context C.. This setting corresponds to the four Wikipedia editing tasks we investigate.
Wikipedia Editing Tasks We consider three editing tasks grounded in Ford et al. (2023) and Zhou al. (2025), who survey Wikipedia editors about LLM-assisted editing practices. The three tasks are: 1 Paragraph Writing, which involves generating new multi-sentence content or extending existing text. We define two subtasks: Introductory Paragraph, the task of writing the opening para- graph of a new section; and Paragraph Continuation, which extends an incomplete human-written paragraph. These subtasks allow us to test detection performance both on purely machine-written text and on text blending HWT and MGT. 2 Summarisation. where the model generates lead section of comparable length to human-written written reference, conditioned on the article's 's content. We frame this as a single-documen abstractive summarisation task, following Wikipedia's Manual of Style+ and prior work on Wikipedia summarisation (Gao et al., 2021; Beltrreelia Lapata, 2021). 3 Text Style Transfer, defined as neutralising revision-level NPOV violations (Pryzant
4

---

Published as a conference paper at ICLR 2026
et al., 2020). We provide biased sentence paragraph context and instruct the model vise in line with Wikipedia's 's neutrality guidelines. Focusing on NPOV violations ensures direct alignment with one of Wikipedia's 's core content policies."
3.2 BENCIMARK CONSTRUCTION
Data We use WikiPS, a collection of paragraphs and summary-arten article pairs, and mWNC (Quaremba et al., 2025), an extension of WNC (Pryzant et al., 2020), as the human- written corpus, available in English, Portuguese, and Vietnamese. We randomly sample 2,700 HWT per task and language from the corresponding subsets. For the Paragraph Writing and Summarisation tasks, we balance each subset by length tertiles. For TST, we evaluate at the sentence level for all languages and at the paragraph level for English only, due to limited data in the other languages.
PROMPT EVALUATION
Language
RungeL
BERTSoore
QAFactEval
Transfer
Introdessrry Pungrrash- RAG
English
0.47 0.25(40:23)
0.68
0.38 (-0.33)
Porsgazzee
033
Vietammese
0.30(+0.20)
0.55 (+0255)
(+900))))7)
0.35 (+10300)
Parograph Contitaction -> RAG
Euglish
(40.55
(40.44)
0.42
Portiguree
0.25 (40.25
0,49
(-900+00000
(+42
Vietammee
0.32(10.30)
(10.6))
(+92()))07)
0.35 (+1.344
Suumorisation
English
0
(10.00)
0.45 (+0.01)
Porlagacnc
(40,0)
0.29 (40.06)
0.88 (40.01)
(.40010.01)
Varmames
0.120400055
(40.0))))
(+0.01)
0.46 (-0.00)
English
(40112)))
0,95 (+0.01)
0.91 (+0.01)
Portsquese
(155(1):14)
(10,00)
(.90
0.91 (+0.05)
Vicasmee
(+2.00)
Paglish P
0.58
(40.12)))
0.96 (-0.01)
Table 1: Prompt evaluation results. We report the highest-scoring prompt per language and task. For each metric, we present the highest score achieved, with the improvement over the baseline shown in percentage points (in parentheses).
For each task, we adapt prompts from the natural language generation literature shown to be most effective (Zhang al., 2024a; Mukherjec & Dušck, 2024; Gao et al., 2024), and automatically eval- uate them against a simple baseline. We conduct the evaluation on a length-stratified 10% sample of the target data using GPT-4o mini (Hurst et al., 2024), and select the highest-scoring prompt gen- erate MGT for our benchmark. Appendix B provides implementation details and prompt templates. The prompts consider are as follows:
Paragraph Writing Minimal provides the generic baseline, instructing the model to write or con- tinue paragraph given article and section titles. Content Prompts extend Minimal by including up to ten content-related questions about the target HWT paragraph (e.g.. "What are neural networks inspired by?"), generated using GPT-4o. Naive retrieval-angmented generation (RAG) (Gao et al., 2024) further augments Content Prompts with relevant retrieved content.
Summarisation & TST For Summarisation and TST, we use conceptually identical prompts. Minimal is a simple zero-shot prompt that instructs the model to summarise the article content/neu- tralise biased text. Instraction adds a detailed definition of the lead section/NPOV policy, alongside the bascline instructions to compile the lead section/neutralise the input text. Few-shot includes representative examples for each task in addition to the Instruction prompt.
hhtpss//enn.wikieediss..or /wwkg////kkkk wiki / kipiiei Neutral a st_oof_vies
5

---

Published as a conference paper at ICLR 2026
Evaluation Metrics We use BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) for n-gram overlap, and BERTScore (Zhang et al., 2020)? for semantic similarity. For Paragraph Writing and Summarisation, assess factuality using QAFactEval (Fabbri et al., 2022). 3 For TST, fine-tune pre-trained language models per language and report binary style classification accuracy.
Evaluation Table presents our prompt evaluation results. For each task, we report the best- performing prompt, along with the percentage improvement over the baseline (in parentheses). Overall, prompts with richer context or more detailed instructions achieve greater gains across eval- uation metrics: RAG substantially outperforms Minimal, while Few-shot prompts provide smaller improvements. The results demonstrate that task-specific MGT is of higher quality than generic MGT. For our benchmark, we select: Paragraph Writing - RAG. Summarisation One-shot, and TST + Five-shot.
MGT Generation For each mask-lanuuae subset, we generate MGT with six generators using their respective best-performing prompts. For LLMs, we use GPT-4o and GPT-4o Mini (Hurst al., 2024), Gemini 2.0 Flash (Team al., 2023), and DeepSeek (Guo et al., 2025). We also in- clude two small language models (SLMs): Qwen2.5-7B (Yang et al., 2024a) and Mistral-7B (Jiang et al., 2023).
EXPERIMENTAL SETUP
We design five experiments to benchmark off-the-shelf, supervised, and zero-shot detectors, test their out-of-domain generalisability. analyse their behaviour through feature analysis, and evalu- ate cross-task transfer inform future detector development. Further details implementation, detectors, data, and training are provided in Appendix C.
Experiment 1: Off-the-shelf detectors We evaluate the performance of widely used off-the-shelf
detectors on our tasks. For comparison, we also assess these detectors on generically generated Wikipedia articles, reflecting the setups prior work (e.g., Guo et al., 2023; Macko et al., 2023; Li et al., 2024; He et al., 2024; Wang et al., 2024b;a; Wu et al., 2024). We generate these instances using the prompt "Write a Wikipedia article about <hitle>", ". We select RADAR (Hu et al., 2023), Binoculars (Hans et al., 2024), Desklib, and e5-small (Dugan et al., 2024), these models achieved the strongest performance on the RAID shared task (Dugan et al., 2024). We restrict this analysis to GPT-4o and English, these detectors primarily trained English data.
Experiment 2: Zero-shot and supervised detectors We evaluate nine zero-shot and supervised detectors across each tosk-language-gneent configuration. For each configuration, we fine-tune supervised models using hyperparameter search, and for zero-shot methods we calibrate the optimal classification threshold with Youden's J.
For supervised detectors, we use M-RBBRTa (Conneau et al., 2020) and mDeBERTa (He al., 2023). As zero-shot white-box detectors, we include Binoculars (Hans et al., 2024), LLR (Su et al., 2023a), and FastDetectGPP (White-Box) (Bao et al., 2024). As zero-shot black-box detec- tors, we use BiScope (Guo et al., 2024), Revise-Detect (Zhu et al., 2023), GECScore (Wu et al., 2025b), and FastDetectGPT (Black-Box (Bao et al., 2024).
Out-of-domain generalisation We evaluate the generalisability of detectors trained on task-specific versus generic MGT, both within Wikipedia and oo-main We con- sider two aditor-driven domains where reliable MGT detection is equally important: social reviews (Yelp (Zhang et al., 20155 (EN), B2W (Real al., 2019) (PT), ABSA (Nguyen et al., 2018) (VI) and news (CNN/DM (Nallapati et al., 2016) (EN), Folha (Emdemor, 2023) (PT), News25 (Quang, 2022) (VD).. For all domains, we generate MGT using generic prompts.
Experiment 3:
7 We use XLMM-R (Conneau et al., 2020) as the backhone.
As FaaactEval is English-only, we translated the Portuguese and Vietnamese prompt-evaluation sets using GPT-4o. Appendix D.I shows that translation bias in this sample minimal.

---

Published as a conference paper at ICLR 2026
Experiment 4: Feature analysis As our central argument concerns task-specific MGT, we inves- tigate what models learn when trained on generic versus task-specific data. To this end, we train our best-performing model from Experiment with the same configuration each English dataset and compute Shapley Additive Explanations (SHAP) (Lundberg & Lee, 2017). SHAP values highlight which patterns models exploit to distinguish HWT from MGT.
Experiment 5: Cross-task generalisation Different writing tasks may leave different traces of MGT. We examine how well detectors generalise across tasks by training a model on the full data of one task and evaluating it on all others. This experiment is crucial for understanding how to train future detectors for optimal performance.
We restrict Experiments 4 and 5 to GPT-4o and Qwen 2.5, using the best-performing model from Experiment Given the parallel structure of our benchmark data, we report accuracy as our primary metric and additionally provide Fl scores for Experiment 2.
5
RESULTS
5.1
EXPERIMENT 1: OFF-TIE--SIELF DETECTORS
4.-
Denilh
65 mmmil
Medel
Figure 3: Comparison of off-the-shelf detectors on generic and task-specific MGT.
Off-the-shelf detectors underperform on task-specific MGT. Figure shows the accuracy of four off-the-sheff detectors generic and task-specific data. All detectors achieve near-perfect accuracy of >93% on generic data, including the zero-shot method B nocul a rs. However, across tasks, accuracy drops between 47% and 73%. This indicates that detectors which appear effective on generic data are likely to underperform in real-world scenarios where users rely on LLMs for specific tasks.
5.2 EXPERIMENT ZERO-SHOT AND SUPERVISED DETECTORS
Most detectors struggle to detect task-specific MGT. Table 2 reports detection results by task and language, averaged across the six generators." Overall, all tasks pose challenges detectors from every model family. Supervised models consistently outperform zero-shot methods, achiev- ing average accuracies between 79.7% and 91.8% (excluding sentence-level TST), while zero-shot models fail to exceed 64.7% average.
For Introductory Paragraph, supervised detectors achieve an average accuracy of 85.9% across lan- guages, whereas white-box (57.7%) and black-box (62.3%) methods perform considerably worse. While both supervised detectors perform similarly, nicucll ars achieves the highest average accuracy among white-box methods (61.8%), and GEcScore leads among black-box methods (69.7%). For Paragraph Continuation, most zero-shot methods drop to near random-chance ac- curacy, with the exception of BiScope. This likely reflects blurred statistical disparities caused We report precision-recall curves for closer inspection of trained models in Appendix D.4.

---

Published as a conference paper at ICLR 2026
Introductory Paragraph
Paragraph Continuation
Detector
English
Portuguese
Vietnamese
English
Portuguese
Vietnamese
ACC
ACC
ACC
ACC
F1
ACC
F1
ACC
F1
Binoculars
57.8
60.4
61.3
66.4
67.3
52.1
41.7
55.8
55.0
60.6
60.0
LLR
50,9
63.2
52.6
56.0
55.4
30.9
50.6
23.5
51.6
45.2
52.2
25.1
FDGPT (WB)
53.6
43.8
58.4
60.6
63.0
56.5
50.7
36.7
54.5
49.1
57.8
40.3
Avg. White-box
54.1
55.8
57.4
59.4
61.6
51.6
51.1
34.0
54.0
49.8
56.8
41.8
BiScope
69.1
68.7
65.1
64.5
69.0
68.9
61.8
60.6
61.9
59.4
68.3
66.9
Revise
52.5
52.1
54,4
54.4
53.0
46.0
51.3
42.7
52.2
47.1
52.2
56.8
GiCCScore
75.6
74.0
70.8
72.3
62.7
62.8
56.1
54.7
55.0
40.8
52.8
36.9
FDGPT (BB)
53.6
41.7
58.7
55.3
62.2
56.8
51.3
17.2
55.7
41.6
59.1
43.1
Avg. Black-box
62.7
59.1
62.3
61.6
61.7
58.6
55.1
43.8
56.2
47.2
58.1
50,9
Ron-RoBERTa
84.6
82.2
85.5
84.3
84.2
85.2
84.8
88,1
mDeBERTa
89.0
88.8
83.8
83.1
86.3
85.6
86.2
86.1
86.3
86.2
87.4
87.4
Avg. Supervised
86.8
86.5
83.0
82.2
85.9
85.1
85.3
85.2
85.8
85.5
87.8
87.7
Summarisation Portuguese
Text Style Transfer
English
Vietnamese
English
Portuguese
Victnamese
English P.
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
Binoculars
60.4
61.4
67.6
69.2
66.4
69.6
51,9
30.7
55.8
52.2
55,9
45.7
$7.1
45.3
54.5
54.7
54.1
50.1
51.9
21.5
51.7
35.8
52.6
26.9
FDGPT (WB)
57.8
59.2
64.7
65.4
64.5
56.6
52.5
60.8
54.6
44.5
55.9
45.5
59.3
42.0
Avg. White-box
57.6
62.2
62.3
65.0
61.7
61.1
51.5
31.2
54.1
39.4
54.5
42.3
56.4
42.0
BiScope
70.7
69.9
68.0
66.0
70.5
70.2
57.3
56.9
60.3
59.6
59.6
58.3
57.3
56.9
Revise
54.0
56.3
53.3
57.0
53.0
57.9
55.1
60.0
53.3
54.1
56.1
60.1
57.4
$8.0
GECScore
75.8
76.5
68.5
70.1
62.3
64.3
64.2
61.1
61.4
59.8
58.6
44.1
73.8
73.6
FDGPT (BB)
58.2
60.0
63.8
64.8
62.7
53.9
52.0
37.7
53.6
36.0
55.3
40.4
57.9
51.0
Avg. Black-box
64.7
65.6
63.4
64.5
62.1
61.6
57.1
53.9
57.1
52.4
57.4
50.7
61.6
59.9
xIm-RoBERTa
91.5
91.4
91.2
91.1
90.2
89.9
64.6
63.0
64.5
63.5
63.4
61.8
78.8
77.9
mDeBERTa
90.4
90.2
92.5
92.4
89.3
63.4
61.8
68.2
66.4
66.2
65.2
80.6
79.4
Avg. Supervised
90.9
90.8
91.8
91.8
89.8
89.5
64.0
62.4
66.4
64.9
64.8
63.5
79.7
78.7
Table 2: Detector accuracies (ACC) and Fl-scores (F1) on task-specific MGT for each task and language, averaged across generators, English P. = English Paragraphs.
by the mixing of human and machine text, which undermines the signal on which these methods rely. We also observe a slight increase in average accuracy across detector families from English to Vietnamese, most pronounced for white-box detectors (e.g., Binoculars: arou 52.1 - 60). This linear trend occurs only in this task.
Summarisation yields the highest detection scores across model families and languages. Compared Introductory Paragraph, both zero-shot families perform marginally better, while supervised mod- els increase to or approach 90% accuracy across languages. Although most LLMs undergo extensive post-training on summarisation tasks, Wikipedia lead sections follow a distinctive style that provides strong cues for detectors. BiScope (69.7%) and Binoculars (64.8%) achieve the best perfor- mance within their respective families. Across languages. supervised detectors maintain consistent accuracy with slight drop for Vietnamese; white-box methods perform best on Portuguese (62.3%), while black-bor models reach the highest accuracy on English (64.7%).
For sentence-level TST, supervised detectors achieve an average accuracy of 65.1%, considerably lower than in the other tasks. Most zero-shot detectors perform only slightly above random chance across languages, with the notable exception of GECScore, which performs comparably to su- pervised detectors (e.g., 64.2% for English). We attribute the low detection scores in part to the sentence-level setting. When comparing English sentence- paragraph- data (English P.), observe substantially higher average accuracies for the latter, most notably a 15.7% increase for supervised detectors.
8

---

Published as a conference paper at ICLR 2026
EXPERIMENT 3: OUT-OF-DOMAIN GENERALISATION 5.3
Enn
Porfucuume
Vtetnamace
56.2
62.0
96.6
92.8
65.4
66.2
81.5
100
72.0
$1.0
58.8
100
54.0
50.3
23.5
46.0
50.3
73.3
72.3
88.0
15.3
53.2
50.7
52.2
59.6
52.0
54.7
52.1
99.6
Figure 4: Out-of-domain accuracies of mDeBERTa by language with GPT-4o. Our dataset balances Introductory Paragraphs and Summarisation. WikiG (Wikipedia generic prompting), news (CN- N/DM, FOLHA, News25), and social reviews (Yelp, B2W, ABSA) represent generic MGT.
Fine-tuning task-specific data generalises to generic data within and across domains, but not vice versa. Figure shows confusion matrix accuracies by language for the best persforefnn model from our second experiment, mDeBERTa. We observe generalisation asymmetry: when fine-tuned on task-specific data, mDeBERTa generalises well to generic data both within and across domains. However, when fine-tuned on generic data, the model fails generalise to our data-even within the same domain. For example, mDeBERTa fine-tuned on our English data achieves an average accuracy of 89.7% across test sets (first matrix, first row). In contrast, when fine-tuned on generic data, no domain yields more than 73.3% test set accuracy on task-specific data. This pattern consistent across most configurations, though most pronounced test set accuracies of 92.8-100% reinforce the findings of Experiments I and 2, underscoring that generic MGT is easy to detect across domains. Appendix Figure 10 reports results for Qwen 2.5, showing the same pattern.
English. Moreover, diagonal
5.4 EXPERIMENT 4: FEATURE ANALYSIS
Fine-tuning on generic data tends to overfit surface-level features. To analyse the results of Experiments 1-3, we compare features learned by mDeBERTa when trained generic versus task- specific English Wikipedia data. Figure 5 presents the five features with the highest SHAP values in each setting. mDeBERTa fine-tuned on our data assigns greater weight to semantically meaningful tokens, indicating stronger reliance on transferable MGT patterns. In contrast, when fine-tuned on generic data, SHAP values reveal heavy dependence on superficial cues such as section formatting (e.g.. "==" or ").
Toro
Canart
mox Vall
Figure 5: SHAP features for mDeBERTa.
5.5
EXPERIMENT CROSS-TASK GENERALISATION
91.3
62.6
61.9
36.9
81.
$9.9
$9.7
60.2
17.2
70.1
65.9
58.7
50.2
92.0
40.8
43.3
49.8
62.9
89.3
75.0
53.2
45.3
18.9
75.1
14.8
53.
50.3
50.4
47.9
03.0
2.4
Figure 6: Cross-task accuracies of mDeBERTa by language with GPT-4o. IP Introductory Para- graph, PC Paragraph Continuation, SUMS Summarisation, TST Text Style Transfer.

---

Published as a conference paper at ICLR 2026
Cross-task performance is generally low. Figure 6 presents cross-task accuracies by language for mDeBERTa. Overall, detection performance across tasks remains relatively low. For English, the average cross-task accuracy is 72.1% % for Summarisation, compared with 60.5% for Introductory Paragraph and close to random chance for the other two tasks. The same trend holds for Portuguese and Vietnamese, as well as when using Qwen 2.5 (Appendix Figure 11). These results suggest that tasks exhibit distinct patterns that do not easily generalise across tasks. We therefore conclude that future detectors should be trained on a combination of different tasks.
DISCUSSION AND CONCLUSION
Discussion We find that most detectors struggle considerably on task-specific data. Through cross- domain experiments and feature analysis, we demonstrate that models trained on generic data tend to overfit to superficial MGT artefacts. This explains their strong in-domain but weak out-of-domain performance. In contrast to prior benchmarks (e.g. Guo et al., 2023; Macko et al., 2023; Li et al., 2024: He et al., 2024; Wang et al., 2024b;a; Wu et al., 2024), our results suggest that evaluations on generic data likely overestimate detector performance. This conclusion aligns with the findings of Doughman
al. (2025), who also highlight that classifier performance is often overestimated. Because we ground our task setup in observed editing practices of L.M usage (Ford et al., 2023; Zhou al., 2025), we argue that most detectors are insufficient for supporting the automatic de- tection of MGT in real-world contexts. Recent work on adversarial attacks (He et al., 2024; Wu al., 2024; Zheng et al., 2025) also reports reduced detection performance. However, our data are more challenging as they more realistically capture real-world generation, rather than relying on adversarial perturbations applied post- genergennrtion. For future work, we recommend developing and evaluating detectors on diverse combination of common writing tasks. T'M--BENE provides rich multilingual data to facilitate research in this direction. Future extensions could include additional languages, tasks, detectors, new domains, and analyses of the effects of combining tasks.
Conclusion We present TSM-BENCII, a multilingual, multi-generatorr, and multi-task benchmark for MGT detection, featuring diverse real-world LLM text generation tasks Wikipedia. We show that most detectors underperform on task-specific MGT and highlight the limitations of evaluating detectors solely on generic MGT. Our findings suggest that existing benchmarks likely overestimate detector performance on UGC platforms and indicate that automatic MGT detection in real-world contexts remains unreliable.
Limitations First, we focus on three common editing tasks, although other equally important tasks exist (e.g., translation). We base these tasks on qualitative evidence of editors' LLM usage, although we cannot guarantee that all editors employ LLMs in these ways. Second, some of our style classi- fiers used for TST prompt evaluation perform poorly despite extensive fine-tuning. We address this limitation in Appendix C.2.1, but note that NPOV style classification remains challenging. Third, we stratify by length avoid confounding effects, but we not explore in detail how text length influences task-specific MGT detection. We leave this to future work.
ETHICS STATEMENT
Our work uses publicly available content from Wikipedia, licensed under CC BY-SA. No private sensitive information is included, and experiments pose risk Wikipedia editors or the Wikipedias under study. Sensitive data about individual contributors are not identifiable or exposed in any way.
We obtain machine-generated data using four L.MMs under their respective licences:
GPT-4-mini: No specific license. OpenAl welcomes research publications."
Gemini 2.0: Apache 2.0
Qwen 2.0: Apache 2.0
phttps://gil wttphh/...oooooggeeeeeeeeeee Phttps:/ wttp...wm/wwwee/////////oe
10

---

Published as a conference paper at ICLR 2026
Mistral: Apache 2.0
All other datasets used in our work are publicly available. The license details are as follows:
CNN/Daily Mail (News): Apache 2.0 License
FOLHA: CCO: Public Domain!
News25: CCO:
Public Domain
Yelp Reviews: Licensed under the Yelp Dataset License Agreement." 17 Permits usage for academic and non-commercial research purposes only.
:2W-Revvews001: CC BY-NC-SA 4.0 License
VLSP 2018 ABSA: No specific license is provided, but the dataset is intended for research use."
This study addresses limitations in previous evaluations of MGT detectors by assessing their per- formance within realistic editorial contexts. The objective is to provide more accurate and practical insights into the feasibility and utility of MGT detection in scenarios where humans employ LLMs in diverse ways to generate text for specific tasks. The experiments aim to inform the potential of MGT detectors automated metrics or as tools support users of UGC platforms in identifying machine-genene generated content.
LLM Usage We use LLMs to correct spelling. grammar, and punctuation mistakes in our text. We do not copy and paste the corrected text but instead prompt for detailed list of errors and incorporate them manually.
REPRODUCIBILITY STATEMENT
We have taken several measures to ensure that our benchmark reproducible and usable for future research. For data generation, describe prompt design and evaluation in Section 3.2 and pro- vide the full prompts alongside additional details on data generation in Appendix B. We outline our experimental setup in Section 4 and include further information in Appendix C, such hyperpa- rameter settings and hardware configurations. All experiments are run with fixed random seed to guarantee full reproducibility of our results. We make all code and data publicly available.
ACKNOWLEDGEMENTS
This work was supported by the Engineering and Physical Sciences Research Council [grant number [PP/009900//] through funding from Responsible AI UK (KP0011), I), and by UK Research and Innovation [grant number EP/S023356/1] through the UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (www.affenndrusttdaa..rg
htte------t-------tot-l-eeeee hauool
wisranssee-oonineeeessssaassss
"Yelp Dataset

---

Published as a conference paper at ICLR 2026
REFERENCES
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectGPT: Efficient zero-shot detection of machine-generater text via conditional probability curvature. In The Twelfth International Conference on Learning Representations, 2024. URL https: / //openzeview. hloeenavveew.ntt/fnnnnaee weeet//frrmmdddppogpgee
Creston Brooks, Samuel Eggent, and Denis Peskoff. The rise of Al-generated content in Wikipedia. In Lucie Lucie-Aimée, Angela Fan, Tajuddeen Gwadabe, Isaac Johnson, Fabio Petroni, and Daniel van Strien (eds.), Proceedings of the First Workshop on Advancing Natural Language Pro- cessing for Wikipedia, pp. 67-79, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 12.18665//1/2224.waag org/2024, wikinlp-11 1.2/
1.12. URL https: / wocccannhhooggg.
Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. Generating literal and implied subquestions to fact-check complex claims. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3495-3516, Abu Dhabi, United Arab Emirates, December 2022. Asso- ciation for Computational Linguistics. doi: 10. // //aclannthlloyy.... wttpoogy...g///// org/ 2202. emnlp- mano-main... 229/.
1//p:22.emmllmmmaieei main.229. URL https:
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov, Un- supervised cross-lingual representation learning at scale. In Dan Jurafsky. Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp.
Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/112202..acllm mainmm..77 URL https: aalaacaannnoooggoy.ooooooo org/ 2020. acl-main. 747/-
Selena Deckelmann. 2023. URL w wikin wikipedde wnw.aaale----ee-a--eo-------
Wikipedia's value in the age of generative ai. July 12 hngg/nwws/22255 /12/
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers). pp. 4171-4186, 2019.
Jad Doughman, Osama Mohammed Afzal, Hawau Olamide Toyin, Shady Shehata, Preslav Nakov, and Zeerak Talat. Exploring the limitations of detecting machine-generater text. In Owen Ram- bow, Leo Wanner, Marianna Apidianaki, Hend AI-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 3 International Conference on Computational Linguis- tics, pp. 4274-4281, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL https: //t:://aala /aaaaaaaaanhhooao/aa Logy- wwt.o//////// 202.. ling main. 288/
Liam Dugan, Alyssa Hwang, Filip Trhlfk, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Cuurison-Burch. RAID: A shared benchmark for robust evaluation of machine- generated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 12463-12492, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https::t/aaaacc http://aaaaaanaa/aaaaaaammm .org/ 2024.ac1- long. 674.
Emdemor. News of the brazilian newspaper, 2023. URL https://hugingfaae co/ httssetss/mdmmrr/////
of the chhe-braziiiie lian newspaper. Accessed: 2025-05-10.
Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. QAFactEval: Improved QA-based factual consistency evaluation for summarization. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 2587-2601, Seattle. United States, July 2022. Association for Computational Linguistics. doi: 10. 12222222222aaaa nar.llnnacllmaian 202 naagl - /
URL https: : / //a/claanttt aaclanthloogy /rr/
12

---

Published as a conference paper at ICLR 2026
Heather Ford, Michael Davis, and Timothy Koskie. Implications of chatgpt for knowledge integrity
on wikipedia, July 2023. URL https://meta wikimedia wwtpoo/wgwgw///aa..eee.ee Implicat settioos_ooos of__hhatPCT____ for knowledge integrity on Sttggrittt__o_iiiiiiiiiaaa Nikipedia. Ac- cessed: 2025-04-06.
Shen Gao, Xiuying Chen, Chang Liu, Dongyan Zhao, and Rui Yan. Biogen: Generating biography summary under table guidance on wikipedia. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4752-4757, 2021.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval--uggneetedd generation for large language models: A survey, 2024. URL https: arxiv. org/ www.aaa/////oaaa//ea/..//o/eaaaa
Demian Gholipour Ghalandari, Chris Hokamp. Nghia The Pham, John Glover, and Georgiana Ifrim. A large-scale multi-document summarization dataset from the Wikipedia current events portal. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 1302-1 1308, Online, July 2020. Association for Computational Linguistics. doi: 10.18653//1/2020...o acl- maiemmmamnnnaa 120. URL https: / a antho ogy .rg.orgggoo..... 2020 acl ma
i in. 120/-
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yu- peng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection, 2023. URL https:://// / arxiv. org/ /abs//////// / 07597..
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. 2025.
Hanxi Guo, Siyuan Cheng, Xiaolong Jin, Zhuo Zhang, Kaiyuan Zhang, Guanhong Tao, Guangyu Shen, and Xiangyu Zhang. Biscope: Ai-generated text detection by checking memorization of preceding tokens. Advances in Neural Information Processing Systems, 37:104065-104090 2024.
Aaron Halfaker and R Stuart Geiger. Ores: Lowering barriers with participatory machine learning in wikipedia. Proceedings of the ACM on Human-Compute Interuction, 4(CSCW2):1-37, 2020.
Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Spotting ilms with binoculars: zero-shot detection of machine-generated text. In Proceedings of the 41st International Conference Machine Learning, ICML'24. JMLR.org, 2024.
Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. Wikiasp: A dataset for multi-domain aspect-based summarization. Transactions of the Association for Computational Linguistics, 9:211 2021.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangles embedding sharing. The Eleventh International Conference on Learning Representations, 2023. URL
net/ htt///oormmiddeeee
XhLxHA.
Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. Mgtbench: Benchmark- ing machine-genere generated text detection. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 2251-2265, 2024.
Xiaomeng Hu, Pin-Yu Yu Chen, and Tsung-Yi Yi Ho. Radar: Robust ai-text detection via adversarial learning, Advances neural information processing systems, 56:15077-1 2023.
Siming Huang, Yuliang Xu, Mingmeng Geng, Yao Wan, and Dongping Chen. Wikipedia the era of lmms: Evolution and risks. arXiv preprint XxXi::2503302879,, 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21226, 2024.
13

---

Published as a conference paper at ICLR 2026
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Ane Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //arxiv.org//aaeeeeeeee http://.wrrrbb///...////// 06825.
Isaac Johnson, Lucie-Aimés Kaffee, and Miriam Redi. Wikimedia data for AI: review of wiki- media datasets for NLP tasks and Al-assisted editing. In Lucie Lucie-Aimée, Angela Fan, Tajud- deen Gwadabe, Isaac Johnson, Fabio Petroni, and Daniel van Strien (eds.), Proceedings of the First Workshop on Advancing Natural Language Processing for Wikipedia, pp. 91-101, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024. 90244wiiknllp-111 URL https::tpp........t....tt / a www.annhhllloooooooooooooooooolo org/ 2024. w
144/
Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang. Linyi Yang. Shuming Shi, and Yue Zhang. MAGE: Machine-generated text detection in the wild. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 36-53, Bangkok, Thailand, Au- gust 2024, Association for Computational Linguistics. doi: 1/20 https: / /a/aacaann aaclanthhllloooooooooooooooooooa logy. ogy- org/ 2024 aclllllloonn 3/.
URL
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74 81, 2004.
Peter J. Liu, Mohammad Saleh*, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In International Confer- ence on Learning Representations, 2018. URL https: /wwp//opeeeeeeeeeeeeeeeeee net http:/urt//oommiddte HygOvbWC-.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL https: / arriv. org/ htrgg/abs/// abs 1907 111922.
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, toxicity. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 1: Long Papers), pp. 3245-3276, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v 1/266/////222.mmacl-ooog naacl-log..-ln.11 179. URL ht https: htttp:://cclatthooe
wrgg//20002 mww.2244aaaaoooooo... 179/.
Scott Lundberg and Su-In Lee. A unified approach to interpreting model predictions, 2017. URL http://wwxx....//aaa////////ee/a alos/ 077777
Dominik Macko, Robert Moro, Adaku Uchendu, Jason Lucas, Michiharu Yamashita, Matúš Piku- liak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, and Maria Bielikova. MULTITuDE: Large-scale multilingual machine-generated text detection benchmark. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 9960-9987, Singapore, December 2023. Associa- tion for Computational Linguistics. doi: 10.18653/v1/22 htt////w/222...2 www.mmmmin.ioooammmmmmmm URL https: / //aclannhooogyy org/2223. emn 11-mamin.... 616/
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of fac- tual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 12000, Singapore. December 2023. Association for Computational Linguis- tics. doi: 100188633/1/2022 101533//112223.mmlpppp wwn.p-mminnnnneeeee m411.741. URL https: http://aaaaantttaaaaaaa org 2023. emr.ppmaa in. 741/.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. De- tectgpt: zero-shot machine- machine-generated text detection using probability curvature. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023. 2023.
14

---

Published as a conference paper at ICLR 2026
Sourabrata Mukherjee and Ondrej Dušek. Text style transfer: An introductory overview, 2024. URL attpss://rxxin org/ als /200 24442.482...
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gulçehre, and Bing Xiang. Abstrac- tive text summarization using sequence-to-sequenee RNNs and beyond. In Stefan Riezler and Yoav Goldberg (eds.), Proceedings of the 20th SIGNLL Conference on Computational Natu- ral Language Learning, pp. tational Linguistics. doi: 10. 10.18653/11/K16-11 1028. URL
Berlin, Germany, August 2016. Association for Compu- .org/
K1 2216-1000 6-1028/.
Thi Minh Huyen Nguyen, Viet Hung Nguyen, The Quyen Ngo, Xuan Luong Vu, Mai Vu Tran, Xuan Bach Ngo, and Anh Cuong Le. Visp 2018 shared task: Aspect-basee sentiment analysis. In Proceedings of the Fifth International Workshop on Vietnamese Language and Speech Processing (VLSP 2018), Hanoi, Vietnam, 2018. URL https: /wlsp.llpppppp org. wwtpss/wwwwlp...//m/eeee /
sa. Accessed: 2025-05-10.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https: /eteps:/aazee wwaaiiivorrr abs/ 02155.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational I Linguistics, pp. 311-318, 2002.
Laura Perez-Beltrachnni and Mirella Lapata. Models and datasets for cross-lingual summari- sation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference
Empirical Methods in Natural Language Pro- cexsing, pp. 9408-9423, Online and Punta Cana, Dominican Republic, November 2021. Asso- ciation for Computational Linguistics. doi: 10. 10.18653/w1/021..com www.:p/.a..mmmmmmmm..ammmm URL https: / /aclantholooyy. org 2021..221.. wwnnp-mnamaa.eeee 742./
Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. Automatically neutralizing subjective bias in text. In Proceedings of the auai conference on artificial intelligence, volume 34, pp. 480-489, 2020.
Hai Tran Quang. Vietnamese online dataset, 2022. URL datasets a hhaa i ranquango aalaalavveetttttmeeeeee www.onnneeeeweeaaeeeeee
www.wwwww.w....coopp.ppwwww..... com/
Ac-
cessed: 2025-05-10.
Gerrit Quaremba, Elizabeth Black, Denny Vrandecic, and Elena Simperl. WETBench: A benchmark for detecting task-specific machine-gent -eeeerree text on Wikipedia. In Akhil Arora, Isaac Johnson, Lucie-Aimée Kaffee, Tzu-Sheng Kuo, Tiziano Piccardi, and Indira Sen (eds.), Proceedings of the 2nd Workshop on Advancing Natural Language Processing for Wikipedia (WikiNLP 2025), pp. 10-30, Vienna, Austria, August 2025, Association for Computational Linguisties. ISBN 979- 8-89176-284-8. doi: 10. 10.118533v1/2055. http://///..././/po/.o... wikinlp-1.66 URL https: hat/a/acaaanntoooo... httpp//aoooopp////aaaccccooooooo 2025 wikinlp-1.6 6/.
org/
Livy Real, Marcio Oshiro, and Alexandre Mafra. B2w-reviews01: An open product reviews corpus. 2019. URL https: /gi thub. com/amert canas- wwwteeewhheeeeeeeeeeeeeeeeeeeeee e1 Accessed: 2025-05-10.
Neal Reeves, Wenjie Yin, and Elena Simperl. Exploring the impact of chatgpt on wikipedia engagement. Collective Intelligence, 403)26633913225113333333 2025, doi: 10.1177/
26339137251372599, URL https: /tt::///ooiiie/ http:////....1///331111111177770
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.1/156 2023.
Marija Sakota, Maxime Peyrard, and Robert West. Descartes: Generating short descriptions of wikipedia articles. In Proceedings the ACM Web Conference 2023, WWW '23, pp. 1446-1456 ACM, April 2023, doi: 10.11455/543350735832200 URL http: /ttp:::/dd.eeoe doi www.:/ociooooooeeoo 3543507. 3583220.
www//o.//............ooo........ 1145/
15

---

Published as a conference paper at ICLR 2026
Yijia Shao, Yucheng Jiang, Theodore Kanell, Peter Xu, Omar Khattab, and Monica Lam. As- sisting in writing Wikipedia-like articles from scratch with large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), pp. 6252-626 6278,, Mexico City, Mexico, June 2024. As- sociation for Computational Linguistics. doi: 11118653/112024.... maaa02.aaactmooog....... URL https: //aclanthology wwg/2024 naacl
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 631(8022)7555-55, 2024.
Jinyan Su, Teny Zhuo, Di Wang, and Preslav Nakov. Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 12395-12412, 2023a.
Zhenpeng Su, Xing Wu, Wei Zhou, Guangyuan Ma, and Songlin Hu. Hc3 plus: A semantic-invariant human chatgpt comparison corpus. arXiv preprint 0rXiv22309.0233,, 2023b.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yean-Baptiste Alayrac. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.118000 2023.
Martina Toshevska and Sonja Gievska. A review of text style transfer using deep learning. IEEE Transactions on Artificial Intelligence, 3(5):669-644, October 2022. ISSN 2691-4581. doi: 10. (109/ta..0211311550 URL http: dx doi wwwp:rggg....... 1109/ TAI. 2021. 3115992.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 0rXiv:2307.09288, 2023.
Adaku Uchendu, Zeyu Ma. Thai Le, Rui Zhang, and Dongwon Lee. TURINGBENCH: A bench- mark environment for Turing test in the age of neural text generation. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 18653/v 1/2021 findings-emnlp..11 URL ptttps://aclaanthote
wttg////.1122.... 2021. findingsn-eenee emn 1.....enm1ppp.....mmm 172//
Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arool, Alham Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. M4GT-bench: Evaluation benchmark for black- box machine-generan generated text detection. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Pupers), Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18665 2024. acl- Long. 218/-
URL https: http://aaaaantoooooo.ooo.o /
Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, et M4: Multi-generator, multi-domain, and multi-lingual black-box machine- geeerraee text detection. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume Long Papers), pp. 1407, 2024b.
Junchao Wu, Runzhe Zhan, Derek Wong, Shu Yang, Xinyi Yang, Yulin Yuan, and Lidia Chao. Detectrl: Benchmarking llm-generated text detection in real-world scenarios. Advances in Neural Information Processing Systems, 37: 37:110039-100401 2024.
Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia Sam Chao, and Derek Fai Wong. A survey on IIm- iin-generated text detection: Necessity, methods, and future directions. Computational Linguistics, pp. 1-66, 2025a.
16

---

Published as a conference paper at ICLR 2026
Junchao Wu, Runzhe Zhan, Derek F Wong, Shu Yang, Xuebo Liu, Lidia S Chao, and Min Zhang. Who wrote this? the key to zero-shot llm- generated text detection is gecscore. In Proceedings of the 31st International Conference Computational Linguistics, pp. 10275-10292, 2025b.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024a.
Xianjun Yang, Liangming Pan, Xuandong Zhao, Haifeng Chen, Linda Ruth Petzold, William Yang Wang, and Wei Cheng. A survey on detection of LLMs-generated content. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2024, pp. 9786-9805, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653/v 1/2024 http:////22....nnnnnso.ooo findings-emnll.55 https: / aa antho logy..oggy... org/ 2024. fi ndi ngs-emm 1 p 572//
URL
Peipeng Yu, Jiahan Chen, Xuan Feng, and Zhihua Xia. Cheat: A large-scale dataset for detecting chatgpt-written abstracts. IEEE Transactions on Big Data, 2025.
Haopeng Zhang, Philip S Yu, and Jiawei Zhang. A systematic survey of text summarization: From statistical methods to large language models. arXiv preprint arXiv:2406. / 1289, 2024a.
Jiebin Zhang, Eugene J. Yu, Qinyu Chen, Chenhao Xiong, Dawei Zhu, Han Qian, Mingbo Song, Weimin Xiong, Xiaoguang Li, Qun Liu, and Sujian Li. Wikigenbench: Exploring full-length wikipedia generation under real-world scenario, 2024b. URL https://arxiv. org/ abs/ 2402. 18264.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evalu- ating text generation with bert. In International Conference on Learning Representations, 2020. URL htwpt: http:///peeeeeeeeeeeeeeeeee net forum? Ld-SSeeucrdd wtw.eeenne///eeeineeeeeeeeoe
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
Jingyi Zheng, Junfeng Wang. Zhen Sun, Wenhan Dong, Yule Liu, and Xinlei He. Th-bench: Evalu- ating evading attacks via humanizing ai text on machine-generatee text detectors. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD '25, pp. 5948-5959, New York, NY, USA, 2025, Association for Computing Machinery. ISBN 9798400714542. doi: 10.1145/8711189..3777418 URL 01111896.333744
Moyan Zhou, Soobin Cho, and Loren Terveen. Llms in wikipedia: Investigating how 11ms impact participation in knowledge communities. arXiv preprint 3Xii:2509.07879 2025.
Biru Zhu, Lifan Yuan, Ganqu Cui, Yangyi Chen, Chong Fu, Bingxiang He, Yangdong Deng, Zhiyuan Liu, Maosong Sun, and Ming Gu. Beat lims their own game: Zero-shot Imm-generated text detection via querying chatgpt. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7470-7483, 2023.
17

---

Published as a conference paper at ICLR 2026
A DATA
MGT Generation Examples of Prior Work Table presents the MGT generation prompts of three selected prior works, and explanation of how they differ from TSM-BENCII.
Paper
Prompt
Explanation
M4GT-Bench (Wang al., 2024a)
Generate Wikipedia summary for "stitle>" (English).
Extends M4 by adding new detection but not generation tasks. The dataset expansion follows generic gener- ation paradigm, since summaries are produced without conditioning on arti- cle content, matching the generic setup described Section 2.1.
MAGE (Li et al., 2024)
"Write news article with the follow- ing headline: theadline>"
Uses fully generic text-generation setup for the news domain. This matches generic news prompt in our Experiment used to construct generic news instances.
MULTITuDE (Macko
"Write (language_name) [..].
news
article
in
A multilingual MGT detection bench- mark focused exclusively on generic news generation. It does not include task-specific settings and follows the same generic paradigm.
al., 2023)
Table 3: Examples of benchmarks relying generic text-generation prompts, as defined in Sec- tion 3.1.
Benchmark Statistics Table presents benchmark statistics. We extend the data of our prior work (Quaremba al., 2025) by adding more generators and new generation task, Paragraph Continuation. This increases the number of MGT samples by approximately half the size of the original data.
We performed data collection steps to mitigate the risk of MGT contamination. Specifically, all data was collected prior to the introduction of ChatGPT on 30 November 2022, we removed all text edited by bots, we considered only full articles rather than article stubs, and we included only text blocks containing at least one reference, ensuring a minimum standard of textual quality. While cannot perfectly ensure that the data contains no MGT, we have taken these measures to minimise this risk.
Corpus
Subset
Level
Language
Corpus N
Eval N
Experiment N
MGT N
EN
286,626
270
2,700
16,200
Sentences
PT
7,877
270
2,700
16,200
mWNC
Text Style Transfer
1.185
1,185
7,110
Paragraphs
EN
4,671
270
2,700
16,200
EN
96,860
270
2,700
16,200
Paragruph Writing
PT
72,965
270
2,700
16,200
98,315
270
2,700
16,200
WikiPS
EN 53,203
270
2700
16,200
Summarisation
PT 36,075
270
2,700
16,200
45,500
2,700
16,200
Total
2,700
25,485
152,910
Table 4: TSM-Bendh dataset statistics. Corpus N denotes the size of the data; Experiment N denotes the number of numan-written texts; and MGT N denotes the total number machine-generated texts.
18

---

Published as a conference paper at ICLR 2026
Language Selection Criteria Table 5 presents details our language selection. Our selection is guided by the goal of including high-, medium-, and low-resource languages, thereby inform- ing communities beyond the English Wikipedia about the potential and limitations of Al detectors. Concretely, we selected languages that vary along two key dimensions. Firstly, the number of ac- tive editors. Secondly, we aimed to choose languages for which the resource ranking Wikipedia aligns with the resource ranking in common LLM training corpora. To verify the latter, we relied on language distribution statisties from Common Crawl.
Language
Resource Level
Active Users
CC Share (%)
English
High
224.120
44.2668
Portuguese
Medium
8,343
2.1796
Vietnamese
Low
3,686
1.0326
Table 5: Overview of languages, resource levels, active Wikipedia user counts, and share language
Common Crawl data (CC-MAIN-2025-44),
B TASK DESIGN DETAILS
Content Prompts We model editors' LLM-assisted content generation through Content Prompts. This prompt variant is motivated from the literature Wikipedia article generation (Shao et al., 2024), which models the process as information-seeking behaviour guided by asking questions. The underlying idea is that editors construct content by iteratively posing questions about the subject. Our Content Prompts are designed to simulate this cognitive process.
For instance, editor aiming to expand a Wikipedia article might prompt model to generate a paragraph in response to factual questions about a specific topic (e.g., "What is the difference between supervised and unsupervised learning?"" "What is reinforcement learning used for?"), within a given section. For each human-written paragraph in our dataset, we prompt GPT-4 to generate a minimum of five content prompts for low-tertile paragraphs, and eight for medium- and high-tertile paragraphs. Although this method does not exhaustively cover all factual content from the HWT, substantially improves the alignment of factual information between HWT and MGT.
When generating these prompts, a valid concern is that the resulting questions may contain halluci- nations, which would deteriorate the quality of the generated texts. We rely on supportive evidence from fact-checking literature (Chen et al., 2022; Min et al., 2023). Min et al. (2023), which uses LLMs to generate atomic questions, finds that such questions are "effective and close human," consistent with findings from prior work (Chen et al., 2022). Additionally, as reported in Table 11 in the Appendix, we show that text generated via Content Prompts leads significant gains in our evaluation metrics, particularly in the factuality metric QAFactEval. Finally, Content Prompts include detailed instructions on how to generate questions, which helps minimise the risk of irrele- vance and hallucination.
Naive RAG We implement a web-based Naive RAG setup to reflect an editing scenario in which an editor, addition to providing task instructions and content prompts, also supplies relevant con- text to minimise factual inaccuracies. Our RAG pipeline follows the indexing, retrieval, and gener- ation modules of the Naive variant (Gao et al., 2024), with two key modifications: we prepend the pipeline with Content Prompts and Web Search modules.
Content Prompts and Web Search For each paragraph, we generate diverse content prompts as de- scribed above. Each content prompt is used to query the Google Custom Search 2PI retrieving the top 10 most relevant URLs. This results minimum of 80 URLs for low-tertile paragraphs and at least 100 URLs for medium- and high-teriile paragraphs.
Indexing We download the raw HTML each scrappable web page and apply a series of prepro- cessing and cleaning steps. Each page is then split into chunks using LangChain's 's RecursiveChar-
htttps://deeel ooo.s. http://tttooaaecc/e//oooeoo///
19

---

Published as a conference paper at ICLR 2026
acterTexttpliiter. We compute BGE-M3" embeddings for each chunk and store them in vector database.
Retrieval and Generation Each content prompt is treated query, for which compute embedding and retrieve the two most similar chunks from the vector database based on cosine sim- ilarity. These retrieved chunks appended to the content prompt context, guiding the model's generation. For the Paragraph Continuation task, we apply RAG only to the second half each.
In-context Learning For Summarisation, we include 1-3 high-quality lead-content pairs re- trieved from the respective Wikipedia Featured Articles page. 23 For TST, we include 1-5 randomly sampled neutallee examples.
B.1 PARAGRAPH WRITING PROMPTS
For brevity, we present prompts in English only.
INTRODUCTORY PARAGRAPII PROMPTS
Minimal
Please write the first paragraph for the section " "(section_title) in the Wikipedia article ":page_title]" using no more than (n_words) words. Only return the paragraph.
Content Prompts
Please write the first paragraph for the section " (sscttio_tiitle" in the Mikipedia article [ipage_title"""
Address the following key points in your response: [content_prompts]
Use no more than (n_words) words. Only return the paragraph.
RAG
Use the following context ensure factual accuracy when writing: (context)
Please write the first paragraph the section "(section_title) the Wikipedia article "ipage_titlo)",
Address the following key points in your response: (content_promptsa)
Use the context above to inform your response, in addition to any relevant knowledge have. Use no more than (ords words. Only return the paragraph in (language).
PARAGRAPH CONTINUATION PROMPTS
Minimal
Please continue writing the follewing paragraph for the section "! section_title in the Mikipedia article "(page_titie)"
LLangChain RecmsiieChannterTTluSpptteee documentation httttp:://hugggnggffceee.. -3 httpa//ppaaapaa.oaaoooaaaaaaaaaa Haat
20

---

Published as a conference paper at ICLR 2026
Existing paragraph: "Ip first)"
Use no more than (n_words) words. Please only return the continuation of the paragraph.
Content Prompts
Please continue writing the follewing paragraph for the section "i section_title)" in the Wikipedia article "(page_title)".
Existing paragraph: "(p_first)"
Make sure that the continuation addresses these key points: (content__rromtts)
Use no more than In words words. Please only return the contimation of the paragraph.
RAG
Use the following context to ensure factual accuracy when writing: (context)
Please continue writing the belew paragraph for the section "[ secticn_titlej" in the Wikipedia article "(page_title]",
Make sure that the continuation addresses these key points: (content__rompts)
Existing paragraph:
"(trgt_first)"
Use the context above to inform your response, in addition to any relevant knowledge you have. Use no more than (trgt_n_toks) words. Only return the continuation of the paragraph in (language).
B.2 SUMMARISATION PROMPTS
Minimal
Your task is summarize the belcw article with than n_toks_trgt) words. Article:
(SrC)
Instructton/Few-Shot
Your task is to summarize an article to create a Wikipedia lead sect on. In wikipedia, the 1ead section is introduction to an article and summary of its most inportant contents.
Apart from basic facts, significant information should not appear the lead if it is not covered in the remainder of the article.
Generate the lead for the article titled "(page_title)" using the article " body above with no more than (n_toke_trgt) words. Article:
(src) """
21

---

Published as a conference paper at ICLR 2026
B.3 TST PROMPTS
Minimal
Please make this sentence/paragrnnn more neutral. **Make as few changes possible and use more than (trgt_r_wordss words for the neutralised sentence/paragrapp... Sentencee/Paragraph:
{src) ""*
Instruction/Few-Shot
Please edit this biased Wikipedia sentence/parrggaahi make it more neutral, aligning with Wikipedia's s neutral point of view policy:
Achieving what the Mikipedia community understands as neutrality means carefully and critically analyzing variety of reliable sources and convey to the reader the information contained in iepportionatelyy and as far as possible without
then attempting
them fairly, editorial bias. Mikipedia alms to describe disputes, but not engage in them. The aim is to inform, not infiuence. Editors, while naturally having their faith to provide complete information and not promote one particular point of view over another. The neutral point of view does not mean the exclusion of certain points of view; rather, it means including all verifiable points of view which have sufficient due weight. Observe the following principles to help achieve the level of neutrality that is appropriate for an encyclopedia:
points of view, should strive in good
Avoid stating opinions as facts.
Avoid stating seriously contested assertions facts.
Avoid stating facts opinions.
Prefer nonjudgmental language.
Do not editorialize.
Indicate the relative promi nence of opposing views.
Make as few changes as possible and use no more than (trgt n. words; words for the neutralised pattence/paragraph.. Output only the neutralized sentence/ppragraaple Sentence/Faragraph:
n*. (src) """
C ADDITIONAL EXPERIMENTAL DETAILS
C.I DETECTOR DETAILS AND IMPLEMENTATIONS
We follow the taxonomy for detecting MGT proposed by (Yang et al., 2024b), which categorises detectors into three types: zero-shot, 2) training-based, and 3) watermarking, although exclude the latter from our experiments. The taxonomy further divides zero-shot methods into white-box and black-box, depending whether the detector has access to the generator's logits other model internals. For all zero-shot models, when the originally used backbone LLM does not support one of our languages, replace it with a multilingual model of the comparable size.
ZERO-SHOT WHITE-BOX
LLR (Su et al., 2023a) The Log-Likelihood Log-Rank Ratio (LLR), intuitively leverages the ratio of absolute confidence through log-lik lillicildoood to relative confidence through log rank about sequence. We implement this detector with Bloom-3B." 24
"https://huuginggface gsci wnc.e//////eeeeeeeeccccccccc bommm3b
22

---

Published as a conference paper at ICLR 2026
Binoculars (Hans et al., 2024) Binoculars introduces metric based the ratio of perplex- ity to cross-perplent perpleeerl,,,,y, where the latter measures how surprising the next-token predictions one model are another. We implement this detectors using Qwen2.5-7B" for the observer model and and Qwen2.5-7B-Inss S for the performer model.
FastDetectGPT White-Box (Bao et al., 2024) DetectGPT (Mitchell et al., 2023) exploits that MGT tends to be located at negative curvature regions of the log probability function, from which curvature-based detection criterion is defined. FastDetectGPT (WB) is an optimised version of DetectGPT that builds on the conditional probability curvature. We implement the white-box version with Bloom-3B." 24
ZERO-SHOT BLACK-BOX
BiScope (Guo et al., 2024) BíScope measures cross-entropy losses beyween output logits and original token and between output logitsaand the preceding input token. From statistics of these losses, they train classifier predict whether the text is machine-generated We implement this detector as in the original paper with Llama 2-7B (Touvron et al., 2023).
Revise (Zhu et al., 2023) Revise builds on the hypothesis that C performs fewer re- visions when generating MGT, and thus bases its detection criterion the similarity between the original and revised articles. We implement this detector as the original paper with GPT-3.5- turbo.2 28
GECScore (Wu et al., 2025b) Grammar Error Correction Score assumes that HWT contain more grammatical errors and calculates Grammatical Error Correction score. We implement this detectors as the original paper with GPT-3.5-turbo."
FastDetectGPT Black-Box (Bao et al., 2024) In the black-box version, scoring model different from the reference model. We use BLOOM-3B for the reference model and BLOOM-1.7B for the scoring model.
SUPERVISED
XLM-RoBERTa (Conneau et al., 2020) XIM-RoBERTa is the multilingual version of RoBERTa (Liu et al., 2019) for 100 languages. RoBERTa is improved version of BERT (Devlin et al., 2019) through more and longer training and dynamic masking modelling.
mDeberTaV V3 mDeberTale V3 is the multilingual version of DeBERTa (He et al., 2023) which improves BERT and RoBERTa through disentangled attention and enhanced mask decoder.
C.2 EXPERIMENTAL SETUPS
C.2.1 TST STYLE CLASSIFIERS
We fine-tune four style classifiers: for each language the sentence level, and an additional classifier for English at the paragraph level. The hyperparameter settings are provided in Table 6.
For English, adopt the hyperparameters from the best- pess-peeforming neutrality classifier available on Hugging Face."! 31 As the English data contain nearly a quarter million English sentence pairs, we conduct fine-tuning smaller subset of the most recent 150k pairs, specifically filtered to include the keyword NPOV in the revision content, in order to further enhance precision. For Portuguese,
2 "nttps://hugginggtan <tteet/e//aggiiiieeeeooo//////02 Inn--nnattttt
" "hups///openai.cco
fication onsujjecttt veetiive-eeee neut
23

---

Published as a conference paper at ICLR 2026
1 Laggug//ool
Models
Learning Rate
Batch Sizes
Egochs
Weight Deeay
ENiSent
ruberia- buse
1e-6
0.01
PTTSen..
xim noherohesta base.
MRFKT
3o-5, 1e-5, 56-6
16, 32
2,5,8
0.001
VinSent
xim roberta base, mBERT
Se le-5.
16,32
2,4,6
0.0.01
EN/Para
ruberia-usee
Se.6
16,32
3,6,9
0.0.01
Table 6: Style Classifier Hyperparameter Settings.
we apply commonly used hyperparameter values, while for Vietnamese and English paragraphs, we extend the search space, initial experiments yielded low detection performance.
Level
Language
Pairs
Test Accuracy
English
300,000
73%
Sentences
Portuguese
S738
63%
Vietnamese
2370
58%
Paragraphs English
9342
58%
Table 7: Style Transfer Classifier Performance. Pairs denote biased and neutralised samples.
Table 7 reports the style classifier hyperparamter fine-tuning results. While fine-tuned models for English and Portuguese sentences yield satisfactory results, style accuracy for English paragraphs and Vietnamese sentences is low. In the following, we provide qualitative analysis of both subsets and explain how we address these low performances.
Low Style Classifier Performance Analysis Table 8 presents two representative examples of NPOV revisions from each subset. The first example in each case illustrates a clear NPOV vio- lation. For instance, the phrase "considered the best footballer" in Vietnamese and "not as strong" in English are both subjective. However, as illustrated with the second examples, NPOV filtering also captures revisions related to political or historical content, which often rely on (subjectively) factual corrections rather than systematic semantic cues.
As we observed this pattern consistently across both subsets, we conducted additional data pro- cessing and hyperparameter tuming for the classifiers. We explored several strategies, including: (I) extending the list NPOV-related keywords, (2) allowing multiple edit chunks per revision, (3) per- mitting multi-sentence edits within single chunk, and (4) expanding the range of hyperparameter settings and model types. However, none of these approaches significantly improved style classifier performance.
Therefore, we selected the configuration that yielded the highest precision, adopting a conservative approach extract NPOV-relevant revision pairs. Despite the relatively low classifier accuracy, are confident that our dataset includes high proportion of true positives.
C.2.2 EXPERIMENT 2: ZERO-SHOP SHOT AND SUPERVISED DETECTORS
We fine-tune both training-based models per task and language on an 80/10/10 split with the hyper- parameter choices displayed in Table 9.
Hyperparameter
Values
Batch Size
16, 32 1e-5, 5c-6, 1e-6
Learning Rate
Epochs
3, 5
Seed
42
Resource
1x NVIDIA A100 40GB
Table 9: Hyperparameter settings for supervised-deteccors.
24

---

Published as a conference paper at ICLR 2026
Subset
Biased Examples
coi là th xut nht th gii và là th vī nht mi thi (Greatest of All Time GOAT), Ronaldo là ch nhân ca 5 Qu bong väng cháu Au vão cáe näm 2008, 2013, 2014, 2016, 2017 vā cing là ch nhân 4 Chic giāy vāng châu Au, hai là k Ic ca mi cu th chau Au cing nhiu danh hiu cao quý khác. (EN: Considered the best football player in the world and the greatest of all time (GOAT). Ronaldo has won Ballon d'Or awards the years 2008, 2013, 2014, 2016, and 2017, as well as 4 European Golden both records for European player-along with many other prestigious titlcs.)
Vietnamese
Öng tng phc Lý Hodi Tiên, ing di quyn nghch te S T Minh Ngy Yên. (EN: He once served Lý Hoài Tiên, general under command the rebel S Minh of Ngy Yên.)
He is not strong, although still exceptional warrior. Agamemnon clearly has stubborn streak that one can argue makes him even more arrogant than Achilles. Although takes few risks in battle, Agamemnon still accomplishes great progress for the Greeks.
English Paragraphs
The population of Bangladesh ranks seventh in the world, but its area of ap- proximately ranked ninety-fourth, making it one of the most densely pop- ulated countries in the world, or the most densely populated country if small island nations and sity-states are not included. It is the third targett Muslim- majority nation, but has a smaller Muslim population than the Muslim minor- ity in India Geographically dominated the fertile Ganges-Brahmapun Brahmaputra Delta, the country has annual monsoon floods, and cyclones are frequent.
Table 8: NPOV Revision Examples. Parentheses contain English translations. Highlighted words indicate words that were edited.
For zero-shot detectors, we use either a single NVIDIA A100 80GB or two NVIDIA A100 40GB GPUs. All models are trained on an HPC cluster.
C.2.3 EXPERIMENT 3: GENERALISATION
For each language- generator domain combination, we randomly sample 2,700 HWT from each dataset to generate generic MGT instances. These data reflects MGT genera setups of prior work (e.g., Guo al., 2023; Macko al., 2023; et al., 2024; He et al., 2024; Wang et al., 2024b;a; Wu et al., 2024) (see prompt templates below). For task-specific MGT, we randomly sample an equal number of instances from our Introductory Paragraph and Summarisation tasks, excluding text of the lowest length tertile.
Due to the open-ended nature of the test data, we truncate all outputs including our texts to 160 tokens to ensure comparable text lengths. All detectors are trained the full training set (N 2, 700) and evaluated on 300 randomly drawn instances. We use the same hyperparameter settings as for Experiment 2 (see Table 9).
We run this experiment on a single NVIDIA A100 40GB.
TEST DATA
We consider two additional domains news and social reviews for which reliable MGT detection equally important as on Wikipedia.
WIKIPEDIA
For all three languages, we randomly sample from our base WikiPS dataset to create full articles consisting of the lead section and article body with minimal formatting.
25

---

Published as a conference paper at ICLR 2026
NEWS
CNN/DM CNN/Daily Mail (Nallapati al., 2016) an English dataset containing over 300,000 news articles from CNN and the Daily Mail, each paired with summary composed of bullet-pointed highlight sentences. We use only the full article text in our experiments.
Folha de São Paulo (FOLHA) (Emdemor, 2023) is large-scale collection of 167,053 news titles and articles from the Brazilian newspaper the same name. The dataset covers the period from January 2015 to September 2017.
FOLHA
News25 The Vietnamese Online News Dataset (News25) (Quang, 2022) is a large-scale collection of over 150,000 news articles from the 25 most popular Vietnamese news sites, collected in July 2022. Each entry includes title and the main article body, along with additional metadata.
SOCIAL REVIEWS
Yelp The Yelp dataset (Zhang et al., 2015) is a large-scale collection of approximately 700,000 business reviews written the Yelp platform. It covers businesses across eight metropolitan areas in the United States and Canada.
B2W B2W-Reviews01 (Real al., 2019) is Portuguese dataset containing over 130,000 e- commerce customer reviews. The reviews were collected from the Americanas.com website be- tween January and May 2018.
ABSA The VLSP 2018 Aspect-Based Sentiment Analysis (ABSA) dataset (Nguyen et al., 2018) includes 4,751 restaurant reviews and 5,600 hotel reviews in Vietnamese. We consider only the restaurant domain, which consists of reviews collected from www.foody.vn.
PROMPT TEMPLATES
For brevity, we present prompts in English only.
WIKIPEDIA TST
Write Nikipedia article with the title "[title)", the article should at least have 250 words.
CNN/DM
Write news article given the fel fellewing highlights: """hhighlights:""
Yelp
Given the first few words of the review, continuc the review with a minimum of 20 words. Review beginning: " "(beginning)"
D ADDITIONAL RESULTS
D.1 TRANSLATION QUALITY
Table reports a quality assessment of our back-translations using the Deepl. API, one of the leading commercial translation systems. Across tasks and languages, standard machine translation metrics (BLEU, ROUGE, and BERTScore) indicate consistently high translation quality. This sug- gests that, while we cannot fully rule out translation bias, its impact on the factuality evaluation of Portuguese and Vietnamese prompts is likely minimal. This interpretation is further supported by the near-identical factuality scores across languages in Table 3.2. Finally, emphasize that translation is used only for evaluating prompt factuality with QAFactEval, all detection experiments operate solely the original source texts.
26

---

Published as a conference paper at ICLR 2026
Task (Language)
BLEU
ROUGE-1
ROUGE-2
BeRT
Puragruphs (PT)
71.50
0.7838
0,6468
0,9671
Puragruphs (VT)
67.95
0.8475
0.7164
0,9486
Summarization
74.24
0.8767
0,8013
0,9736
Summarization (VI)
63.46
0.7520
0,9411
Table 10: Back-translation quality metrics (BLEU, ROUGE, and BERTScore) across tasks and languages.
D.2
LINGUISTIC DESCRIPTIVE ANALYSIS
Figures and 8 show the results of the linguistic descriptive analysis for Portuguese and Vietnamese, respectively.
12
8-
0.20
10
0.15
Dencity
4-
0.10
2
0.0
0.5
1.0
0.4
0.6
0.0
0.00
100
150
Norm. Levenshlein Distance
Casine Similarity
Unigram Overlap
Perplexity
Human
Generic MGT
Paragraph Writing
Sum Sumaaisation
Text Style Transfer
Figure 7: Comparison of textual characteristics between human, generic, and our task-specific MGT
Portuguese.
0.20
10
Density
2
0.05
0.6
0.0
0.00
150
Levenshteln Distance
Similarity
Unigram Overlap
Perplexity
Human
Generic
Paragraph Writing
Sum murrissaaion
Text Style Transfer
Figure 8: Comparison of textual characteristics between human, generic, and our task-specific MGT in Vietnamese.
27

---

Published as a conference paper at ICLR 2026
D.3 PROMPT EVALUATION
Language
Technique
BLEU
ROUGE-1
ROUGE-2
ROUGE-L
BERTSScoe
Introdiactoyy Poragraph
Minimal
0.02
0.29
0.06
0.17
0.76
0,06
English
Content Proempt
0.22
0.57
RAG
0.25
0.61
0.35
0.47
0.88
0.38
Minimal
Portngioo
Content Prompts
0.20
0.56
0.30
0.41
0.91
0.25
Minimal
0.04
0.67
0.26
0.32
0.55
0.06
Vietnames:
Concent Prompts
0.28
0.78
0.54
RAG
0.30
0.79
0.54
0.55
0.92
0.36
Paragruph Cunticnution
0.01
English
Content Prompts
0.21
0.58
0.32
0.45
0.88
0.30
Minimal
0.01
0.25
0.04
0.15
0.36
0.83
Portuguese
Concent Prompts
0.20
0.57
RAG
0.25
0.60
0.38
0.49
0.92
0.42
Minimal
0.31
0.62
0.21
0.31
0.85
0.04
Vietnamese
Content Prommptx
0.31
0.78
RAG
0.32
0.78
0.54
0.57
0.82
0.38
Table Paragraph Writing Prompts Evaluation Results.
Table 11 presents our prompting evaluation results. We find that our Naive RAG approach con- sistently outperforms both Minimal and Content Prompts across subtasks and languages. The low evaluation scores for Minimal prompts highlight that MGT produced in prior work is often synthet- ically divergent from its human- written references. While Content Prompts substantially improve performance, Naive RAG further enhances generation quality-particulatn
in terms of factual con- sistency, which is critical for encyclopedic content." 32 Based these findings, we adopt Naive RAG as the prompting strategy for the paragraph writing task in our MGT detection experiments.
Language
Technique
BLEU
ROUGE-1
ROUGE-2
ROUGE-
BERTSeore
QAFactEval
English
Instruction
0.13
0.44
0.21
0.33
0.52
0.46
One-shot
Two-shoi
0,18
0.47
0.24
0.36
0,83
0.46
Three sbot
0.16
0.46
0.23
0.35
0.83
0.46
Portugaeso
Minimal
0.06
0.48
Instruction
0.11
0.42
0.19
0.30
0
0.48
One-4hot
Two-shot
0.11
0.43
0.19
0.30
0
0.47
Three sbot
0.12
0.43
0.20
0.30
0.58
0.47
Vietnamese
Minimal
0.07
0.63
0.28
0.36
0.45
Instriction
0.11
0.64
0.31
0.38
0.87
0.43
Two shot
0.12
0.66
0.33
0.38
0.57
0.44
0.57
Table 12: Summarisation Prompts Evaluation Results.
Table 12 presents the summarisation prompt evaluation results, showing that across languages, In- struction and Few-shot achieve higher overlap and semantic similarity scores, although Few-shot only marginally improves over Instruction. Factuality scores remain relatively stable across prompts, presumably because summarisation is a core task in aligning LLMs through reinforcement learning from human feedback (Ouyang et al., 2022). Given that increasing the number of shots does not yield further improvements, and considering the context window of smaller LLMs, we select one- shot prompting for experiments.
http:///g//////iiiiie/eaaaeeee fiabilits
28

---

Published as a conference paper at ICLR 2026
Language
Technique
BLEU
ROUGE-1
ROUGE-2
ROUGE-L
BERTScore
ST
English
Minimal
0.35
0.68
0.52
0.66
0.92
0.90
Instruction
"
One-shot
0.52
0.65
0.95
Two shot
0.47
0.75
0.61
0.73
0.94
0.90
Three-shot
0.54
0.79
0.67
0.78
0.95
0.89
Four-shot
0,95
Portuguess
Minimal
0.41
0.71
0.58
0.69
0.94
0.86
Instruction
0.40
0.70
0.57
0.67
0.94
0.88
One-sihot
0.50
0.75
0.61
0.74
0.9%
0.65
Three shot
0.53
0.78
0.66
0.76
0.96
Four-shot
0.58
0.81
0.70
0.79
0.96
0.92
Five-shat
0.55
0.79
0.68
0.77
0.96
0.1
Vietnamese
0.65
Instruction
0.45
0.67
0.94
One-shot
0.44
0.78
0.66
0.71
0.98
0.88
Two-shot
0.51
0.82
0.70
0.76
0.95
0.87
Three-shot
0.50
0.81
0.70
0.75
0.95
Five-shot
0,55
0.83
0.73
0.78
0,96
0.84
English Para.
Minimal
0.35
0.68
0.52
0.66
0.92
0.97
Instriction
0.36
0,68
0.52
0.66
0.92
0.99
One-shot
0.52
0.65
0.95
Three-shot
0.54
0.79
0.67
0.78
0.95
0.96
Four-shot
0.56
0.80
0.69
0.79
0.95
0.5
Five-shot
0.55
0.80
0.68
0.78
0.95
0.96
Table 13: TST Prompts Evaluation Results.
Table 13 presents the prompt evaluation metrics for the TST task, evaluated at the sentence level for all languages, and additionally at the paragraph level for English. Across languages and levels, we find that four- and five-shot prompting consistently outperforms Minimal and Instruction prompts. While differences in semantic similarity and style transfer are marginal across prompts, we observe substantial improvements in overlap-based metrics as the number of few-shot examples increases. These improvements can be attributed to the fact that neutralisation edits in mWNC tend to be relatively minimal. For instance, in the English sentence subset, average only 14% of words are deleted and 7% added similar trends hold for the other subsets. As a result, the model appears to leam from the examples apply similarly sparse edits, thereby producing outputs that match the reference text more closely in terms of n- n-gram overlap. Based on these findings, we adopt five-shot prompting to generate MGT in our subsequent experiments.
29

---

Published as a conference paper at ICLR 2026
D.4 EXPERIMENT 2: PRECISION-RECALL CURVES ACROSS TASKS AND LANGUAGES
0.7
Recal
(a) Introductory Paragraph (mDeBERTa)
(b) Introductory Paragraph (XLM-RoBEEL) RoBERTa)
3.
0.5
(c) Paragraph Extension (mDeBERTa)
(d) Paragraph Extension (XLM-RoBll) RoMETa))))
0.1
Facel
(c) Summarisation (mDeBERTaa)
(f) Summarisation SXLM-RoBEIL
(g) Text Style Transfer (mDeBERTa)
(h) Text Style Transfer
Figure 9: Precision-recall curves across tasks and languages (with GPT-4o the generator.
30

---

Published as a conference paper at ICLR 2026
D.5 EXPERIMENT 2: RESULTS BY LANGUAGE, MODEL AND TASKS
Tables 14 and 15 present the full results of experiment 2.
152
Table 14: Detector accuracies (ACC) and Fl-scores (F1) on task-specific MGT for each task, lan- guage, and generator.
English Text Style Trandee
GPT-4o
GPT--o nini
Geniné 2.0
Despheck
Qwen 2.5
Mistal
Asg
ACC
F1
ACC
F1
ACC
F]
ACC
Fi
ACC
F1
ACC
F1
ACC
FI
0.55
LLR
0.60
0.71
0.52
0.25
0.51
0.22
0.50)
0.00
0.50
0.03
0.53
0.40
0.53
0.37
FDGPT (WB)
0.80
0.78
0.60
0.63
0.56
0.60
0.50
0.00
0,52
0.60
0.58
0.61
0.59
0.54
Avg (White-box)
0.00
0.71
0.54
0.43
0.52
0.54
0.56
0.51
0.56
0.42
BoScope
0.55
0.56
0.55
0.58
0.57
0.65
0.56
0.55
0.54
0.53
0.57
0.57
Revise
0.80
0,80
0.53
0.62
0.52
01.66
0.54
0.54
0.53
0.42
0.52
0.55
0.57
0.58
FDGPT (BB)
0.75
0.71
0.59
0.62
0.54
0155
0.50
0.00
0.51
0.63
0.58
0.54
0.58
0.51
Ang (Blcc-bss))
0024
6.13"
0.63
10.65
00.57
0.50
0.60
0144
0.55
0.57
0.56
0.58
0.62
0.60
RoBERTa
0.76
0.78
0.78
0.77
0.78
mDeBliiia
0.82
0.81
0.63
0.83
0.77
0.76
0.92
0.92
0.81
0.61
0.87
0.64
0.81
0.79
(Supervised)
0.79
0.77
0.87
0.30
0.78
0.57
0.92
092
0.80
0.79
0.09
0.50
0.79
Table 15: Detector accuracies (ACC) and F1-scores (F1) task-specific MGT for TST of English paragraphs.
Mistral Error Analysis We observe anomalous evaluation metrics for Vietnamese texts written by Mistral. While both zero-shot detectors achieve random chance accuracy and often zero Fl-scores, training-based detectors achieve almost perfect metrics. After checking the data, we observe that Mistral-in contrast to the other models-does not follow the instructions in prompts. Typical errors include outputting text mid-sentence or returning English text, despite the last sentences of prompt emphasizing to return text in Vietnamese. These flaws explain the strong performance of training- baased detectors, as they pick up these syntactic imperfections, whereas zero-shot detectors seem unable find clearly distinctive patterns based model internals token patterns.
31

---

Published as a conference paper at ICLR 2026
D.6
EXPERIMENT 3: RESULTS FOR QWEN 2.5
Eng
Portuqusss
Vilhharress
59.5
91.2
15.9
99.4
59.4
9122
72.2
96.1
74.2
52.0
57.7
62.0
100
14.0
6:11
82.4
180.0
91.1
50.4
93.1
96.0
70.
50.7
49.9
$6.7
54.2
13.
82.6
05.0
Figure 10: Out-of-domain accuracies of mDeBERTa by language with Qwen 2.5. Our dataset bal- ances Introductory Paragraphs and Summarisations. WikiG (Wikipedia generic prompting). news (CNN/DM, FOLHA, News25), and social reviews (Yelp, B2W, ABSA), generic MGT.
D.7 EXPERIMENT 5: RESULTS FOR QWEN 2.5
B6.1
66.66
66.
10.2
11:7
64.0
64
62.4
86.9
65.6
7%.1
50.1
73,5
82.0
68.8
61.3
65.3
86.3
$3.0
57.4
17.8
90.0
53.4
54.3
56
65.9
50.7
1AM
10
3370
S4MM
Figure 11: Cross-task domain accuracies of mDeBERTa by language with Qwen 2.5. Introduuuyyyory Paragraph, PC=Paragraph Continuation, SUMSSSummarisatiio,, TST=Text Style Transfer.
32


Reviews:
[{"review_id": "eXMCbCmRfT", "reviewer": "Reviewer_TeVN", "summary": "The authors introduce TSM-BENCH, a new benchmark for evaluating machine-generated text (MGT) detectors in realistic Wikipedia editing scenarios. TSM-BENCH comprises 152,910 machine-generated texts across three languages, four editing tasks grounded in actual Wikipedia editing practices, and six different language models. Results show taht Off-the-shelf detectors achieving over 93% accuracy on generic MGT drop to between 47-73% on task-specific data. The authors discover a \"generalization asymmetry\": detectors trained on task-specific data can generalize to generic data and even across domains, but detectors trained on generic data fail catastrophically when applied to task-specific scenarios. Through feature analysis, the authors find that Models trained on generic data overfit to superficial artifacts like section formatting markers rather than learning genuine linguistic patterns that distinguish machine from human text.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. TSM-BENCH is large-scale and comprehensive, featuring 152,910 machine-generated texts across 3 languages, 4 tasks, and 6 different generators.\n\n2. The paper identifies a significant limitation in existing machine-generated text (MGT) detection benchmarks: previous work has primarily focused on generic text generation, while real-world users employ LLMs for specific, constrained tasks.\n\n3. The authors evaluate 12 different detectors across multiple model families and conducts five comprehensive experiments including zero-shot performance, supervised learning, out-of-domain generalization, feature analysis, and cross-task transfer. The findings are significant and help explain why existing detectors may be unreliable in real-world settings.", "weaknesses": "1. While authors adapt prompts from the natural language generation literature shown to be most effective for each task, this might not reflect how people using LLMs for the task.\n\n2. The study relies heavily on accuracy as the primary metric, with F1 scores provided but not deeply analyzed. In AI content detection task,  metrics like false positive rates also provide valuable insight.\n\n3. The feature analysis in Experiment 4 shows that mDeBERTa trained on generic data overfits to surface-level features, while the model trained on task-specific data focuses more on semantically meaningful tokens. However, this analysis only examined mDeBERTa, Is this pattern of feature learning generalizable across other detection model families?\n\n4. The prompt evaluation uses GPT-4o Mini as the sole judge for selecting the best prompts. This introduces potential biases and lacks validation against human judgments or alternative automatic metrics, which could affect the quality assessment of generated text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xsGJEgNMCw", "reviewer": "Reviewer_qQWg", "summary": "This paper introduces TSM-BENCH, a multilingual, multi-generator, and multi-task benchmark for detecting machine-generated text (MGT) in realistic Wikipedia editing scenarios. The key insight is that existing benchmarks primarily evaluate detectors on generic text generation (e.g., \"Write an article about X\"), while real-world Wikipedia editors use LLMs for specific constrained tasks such as summarization, paragraph writing, and style transfer. The authors demonstrate that task-specific MGT more closely resembles human-written text (HWT) than generic MGT, making detection significantly more challenging.", "questions": "See weakness above.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper makes a valuable contribution by shifting focus from generic to task-specific MGT detection, which better reflects real-world LLM usage patterns. The identification of generalization asymmetry between task-specific and generic training data is an interesting finding. The multi-dimensional benchmark design (multilingual, multi-task, multi-generator) is comprehensive.\n2. The paper is well-written with clear motivation and effective use of figures (especially Figure 1 and Figure 2). The experimental setup is described in detail, and results are presented in an organized manner. The distinction between generic and task-specific MGT is clearly articulated.", "weaknesses": "**1. Data Provenance and Contamination Concerns**\n\nA fundamental assumption of this work is that the sampled Wikipedia texts are purely human-written. However, recent studies have documented that Wikipedia already contains LLM-generated content mixed with human contributions. For instance, [Detecting LLMs in the Wild: A Field Study of Latent Misuse](https://openreview.net/pdf?id=R2bp8xLLao) and [Wikipedia in LLM era](https://arxiv.org/abs/2503.02879) provide evidence of LLM usage in Wikipedia editing. The paper does not address this contamination issue or provide any verification methodology to ensure the \"human-written\" samples are actually purely human-written. This is a critical concern that potentially undermines the benchmark's validity. The authors should at minimum: (i) discuss this limitation explicitly, (ii) provide some analysis or filtering approach to mitigate contamination risk, or (iii) acknowledge the uncertainty in their ground truth labels.\n\n**2. Missing Comparison with Highly Related Work**\n\nThe paper fails to discuss or compare with WETBench (cited as Quaremba et al., 2025 in the references), which appears to be a highly related concurrent work also focusing on detecting task-specific MGT on Wikipedia. This is a significant omission. The authors should provide a detailed comparison explaining: (i) how TSM-BENCH differs from WETBench, (ii) whether there is overlap in the data sources, and (iii) how the findings complement or contradict each other. Without this comparison, it's difficult to assess the novelty and positioning of this work. Moreover, like \"Wikipedia in the Era of LLMs\" and \"Studying the Role of LLMs on Wikipedia\" also discuss the LLM usage in Wikipedia while do not get any discussion and citation.\n\n**3. Oversimplified Task Formulation for \"Real-World\" Claims**\n\nWhile the paper claims to reflect \"real-world Wikipedia editing practices,\" the task formulations are actually quite coarse-grained and unrealistic:\n\n- **Paragraph-level detection is insufficient**: Modern Wikipedia editing with LLMs rarely involves writing entire paragraphs or complete continuations purely with AI. Real-world usage is much more fine-grained, involving sentence-level or even token-level human-AI collaboration. Recent work on collaborative writing has extensively studied this: [Coauthor (Clark et al., 2018)](https://arxiv.org/pdf/2201.06796) and [LLM-as-a-Coauthor](https://arxiv.org/abs/2401.05952) all demonstrate that human-AI co-writing happens at much finer granularities.\n\n- **Binary classification may not be appropriate**: For the Paragraph Continuation task, the authors mention \"blending HWT and MGT\" but still frame it as binary classification at the paragraph level. If the continuation mixes human and machine text, shouldn't the detection task be formulated at a finer granularity? Labeling an entire human-machine hybrid paragraph as simply \"MGT\" seems oversimplified and may not align with practical needs. You can refer to LLM-as-a-Coauthor and Llm-detectaive for fine-grained HWT and MGT detection.\n\n- **Impact on results interpretation**: This coarse-grained formulation might partially explain why white-box and black-box methods show such poor performance (<60% accuracy in Table 2). These methods rely on statistical patterns that may be diluted when human and machine text are mixed at the paragraph level. The strong performance of supervised methods (which can learn task-specific artifacts) compared to zero-shot methods supports this concern. Is the low performance due to the inherent difficulty of the task, or due to the mismatch between the evaluation setup and the actual detection problem?\n\n**4. Language Selection Lacks Justification**\n\nThe choice of Portuguese and Vietnamese as the multilingual evaluation languages is not well motivated. More widely-studied and resource-rich languages like Chinese or French would seem like natural choices, given that: (i) they have substantial Wikipedia presence, (ii) there are more available detection tools and LLMs for these languages, and (iii) they would increase the benchmark's applicability to a broader research community. The paper should provide clear rationale for the specific language selection beyond data availability.\n\n**5. Limited Discussion of Related Work on Human-AI Collaborative Writing**\n\nThe related work section misses an important body of literature on human-AI collaborative writing and mixed-authorship text. Works on co-writing, human-in-the-loop generation, and collaborative authorship have direct relevance to the \"Paragraph Continuation\" task and the real-world editing scenarios the paper aims to capture. This omission weakens the positioning of the work within the broader context of human-AI collaboration research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hz2188yi9K", "reviewer": "Reviewer_gz51", "summary": "This paper introduces TSM-Bench, a multilingual, multi-generator, and multi-task benchmark designed to evaluate machine-generated text (MGT) detectors in real-world Wikipedia editing scenarios. Unlike prior benchmarks that rely on generic prompts (e.g., \"Write an article about X\"), TSM-Bench focuses on task-specific LLM usage—such as paragraph writing, continuation, summarization, and text style transformation across English, Portuguese, and Vietnamese. The benchmark comprises approximately 152,000 texts and evaluated using 12 detectors. The authors find that existing detectors suffer a 10–40 percentage point drop in accuracy on task-specific MGT, exhibit asymmetric generalization (models trained on task-specific data generalize to generic data, but not vice versa), and tend to overfit to superficial artifacts when trained on generic outputs. Overall, the study compellingly argues that realistic, editor-aligned benchmarks are essential for assessing detector reliability on user-generated content (UGC) platforms like Wikipedia.", "questions": "- Could the benchmark be extended to mixture-aware or segment-level labeling to better reflect real Wikipedia editing?\n- How significant is translation bias in PT/VI QAFactEval via GPT-4?\n- In what specific aspects does TSM-BENCH advance beyond prior work (such as M4GT, MAGE, and MULTITuDE) in design or analysis?\n- Did supervised models trained on English task-specific data show any zero-shot gains on Portuguese or Vietnamese? \n- Could the authors expand on whether overfitting features are artifacts of Wikipedia formatting specifically or general stylistic cues common across corpora?\n\n\nIf the authors can substantively address the questions and weaknesses outlined above, I would be inclined to raise my score.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses a pressing concern: maintaining content integrity in Wikipedia and similar UGC ecosystems in the era of LLM-generated text. The motivation is clear and important for both the NLP and broader AI ethics communities.\n- Comprehensive evaluation across 12 detectors and 5 experimental dimensions; inclusion of SHAP analysis gives interpretable evidence of overfitting to surface artifacts.\n- The observed generalization asymmetry, where task-specific detectors generalize to generic data but not vice versa, is a notable and reproducible pattern that could inform future model training paradigms.", "weaknesses": "- The paper convincingly grounds its task design in empirical studies of Wikipedia editors'  LLM use and policy structures such as Manual of Style and Neutral Point of View. However, the benchmark ultimately treats detection as binary classification (HWT vs. MGT), despite acknowledging blended paragraphs where human- and machine-generated text co-exist. This abstraction may be practical for dataset construction, but undermines the ecological validity of the \"real-world\" claim. Real Wikipedia editing may involve mixed authorship, such as post-editing of LLM drafts, LLM-assisted revisions of human text, and stylistic corrections. A benchmark faithful to these workflows would ideally include segment-level or mixture-aware labeling to capture such dynamics. Thus, while the task types are empirically grounded, the evaluation protocol remains synthetically simplified.\n- I kindly recommend that the authors improve the manuscript's presentation quality. For example, several key abbreviations and table entries are used without any definition at first mention, forcing readers to infer their meaning:\n  - \"TST\" first appears in Section 2.2, where it evidently denotes \"Text Style Transformation\", but the acronym is not explicitly defined in the caption of Figure 6 and again in Section 5. This redundancy and delayed definition confuse the reader and suggest a lack of editorial consistency. Readers must deduce its meaning from context or later descriptions.\n  - \"English P.\" appears in Table 1 but is never defined. From the nearby mention of \"paragraph-level English samples,\" one can infer that it stands for English Paragraph, yet this is not stated anywhere in the text or legend.\nSuch omissions may appear minor, but they significantly reduce readability for new readers and hinder reproducibility, as one cannot easily map metrics to task definitions.\n- The factuality evaluation for Portuguese and Vietnamese data relies on GPT-4 translation followed by QAFactEval (line 212–213). However, the paper does not justify or quantify how this translation step affects factual consistency. Without back-translation checks or metric comparisons, it is unclear whether observed performance differences reflect model ability or translation artifacts.\n- Prior work (such as M4GT, MAGE, and MULTITuDE) already explores multilingual or task-specific MGT; contribution is incremental without a stronger methodological leap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PNOLi5ScIY", "reviewer": "Reviewer_mKYw", "summary": "This paper proposes a multilingual, multi-generator, and multi-task benchmark for machine-generated text (MGT) detection on Wikipedia, encompassing 4 tasks, 152,910 examples and six text generators in total. An extensive set of experiments is conducted, including evaluation of 12 detectors from different methods (zero-shot, off-the-shelf, supervised) and model families, feature importance analysis, out-of-domain and cross-task generalisation. Results show degradation of metrics on task-specific generation data compared to evaluations on generic data, and generalisation asymmetry in favor of task-specific generation data. Importantly, the limitations of existing benchmarks and MGT detectors in real-world conditions are emphasized.", "questions": "1. Some details are unclear, such as the specific model used for BERTScore, the references for per-language fine-tuned models, and the definition of \"Wiki IO.\" in Exp. 3.\n2. The prompt evaluation relies on a single model, limiting the reliability of its conclusions.\n3. The reliability of the RAG pipeline, particularly the relevance of retrieved chunks, is not verified.\n4. The justification for the specific language selection is currently poor and needs strengthening.\n5. Below are some suggestions to fix typos and text errors:\n— L384 “accuracy of 89.7%”, in Appendix D.1 captions for figures are the same.\n— Appendix A: “DATA SET STATISTICS” –> “BENCHMARK STATISTICS”; “Corpus denotes the size of the data” –> “Corpus denotes the source of the data”.\n— Rephrase L070  “content integrity of UGC”.\n— Formulation “generic generation” is not clear.\n— The subsection \"PROMPT EVALUATION\" is missing a section number", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper identifies a critical gap in existing benchmarks, where inflated performance metrics often fail to translate to real-world applications. The authors propose a novel benchmark that is multilingual, incorporates multiple generators, and covers main tasks relevant to user-generated content platforms like Wikipedia. An underlying contribution is the original formulation dividing data generation into \"task-specific\" (constrained by context) and \"generic\" (free-form) categories. The experiments demonstrate a substantial performance decrease for all detector types on task-specific data and reveal a clear generalization asymmetry: task-specific training aids in detecting generic text but not the reverse. The feature importance analysis indicates that models trained on generic data tend to overfit to superficial cues.\n\nThe paper is well-structured, competently written, and backed by thorough analysis, including a prompt impact evaluation and an examination of performance gaps between different detector methods.\n\nThe public release of the code and data represents a meaningful contribution to the community, enabling reproducibility and future research, in particular the development and evaluation of MGT detectors.", "weaknesses": "1. The out-of-domain generalization experiment lacks transparency, as it conflates the effects of the domain and the generation conditioning (task-specific vs. generic). A more complete picture would require a fine-tuning experiment on the authors' own \"Our\" data using generic MGT to directly compare the performance impact of the training data's contextual conditioning generation.\n2. The benchmark, though covering crucial real-world tasks, omits others like translation and creative writing, and the scope of domains remains limited to Wikipedia.\n3. The paper misses an opportunity to provide examples of generic MGT cases and does not explore balancing task-specific and generic data for optimal detector training.\n4. The quality of the LLM-based translation is not checked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)