Wikipedia in the Era of LLMs: Evolution and Risks
Authors: Siming Huang
Yuliang Xu
Mingmeng Geng
Yao Wan
Dongping Chen
Venue: Accepted by TMLR ()

Abstract: In this paper, we present a comprehensive analysis and monitoring framework for the impact of Large Language Models (LLMs) on Wikipedia, examining the evolution of Wikipedia through existing data and using simulations to explore potential risks. We begin by analyzing article content and page views to study the recent changes in Wikipedia and assess the impact of LLMs. Subsequently, we evaluate how LLMs affect various Natural Language Processing (NLP) tasks related to Wikipedia, including machine translation and retrieval-augmented generation (RAG). Our findings and simulation results reveal that Wikipedia articles have been affected by LLMs, with an impact of approximately 1% in certain categories. If the machine translation benchmark based on Wikipedia is influenced by LLMs, the scores of the models may become inflated, and the comparative results among models could shift. Moreover, the effectiveness of RAG might decrease if the knowledge has been contaminated by LLMs. While LLMs have not yet fully changed Wikipedia's language and knowledge structures, we believe that our empirical findings signal the need for careful consideration of potential future risks in NLP research.

Published in Transactions on Machine Learning Research (02/2026)
Wikipedia in the Era of LLMs: Evolution and Risks
Siming Huang!
Huazhong University of Science and Technology
Yuliang Xu
Huazhong University of Science and Technology
Mingmeng Geng"
École Normale Supérieure (ENS) Université Paris Sciences et Lettres (PSL) & Laboratoire LATTICE International School for Advanced Studies (SISSA)
Yao Wan"
wonyao@hust.edu.co
Huezhong Universily of Science and Technology
Dongping Chen
Huazhong University of Science and Technology
Reviewed on OpenReview: https: // openreview. net/ forum? id= ah VKVLt
Abstract
In this paper, we present comprehensive analysis and monitoring framework for the im- pact of Large Language Models (LLMs) on Wikipedia, examining the evolution of Wikipedia through existing data and using simulations to explore potential risks. We begin by ana- lyzing article content and page views to study the recent changes in Wikipedia and assess the impact of LLMs. Subsequently, we evaluate how LLMs affect various Natural Language Processing (NLP) tasks related to Wikipedia, including machine translation and retrieval- augmented generation (RAG). Our findings and simulation results reveal that Wikipedia. articles have been affected by LLMs, with an impact of approximately 1% in certain cate- gories. If the machine translation benchmark based on Wikipedia is influenced by LLMs, the scores of the models may become inflated, and the comparative results among mod- els could shift. Morcover, the offectiveness of RAG might decrease if the knowledge has been contaminated by LLMs. While LLMs have not yet. fully changed Wikipedia's 's language and knowledge structures, we believe that our empirical findings signal the need for careful consideration of potential future risks in NLP research!!
1 Introduction
The creation of Wikipedia challenged traditional encyclopedias (Giles 2005 and the rapid development and wide adoption of Large Language Models (LLMs) have sparked concerns about the future of Wikipedia (Wag- ner & Jiang 2025 Vetter et. al., 2025). Researchers have begun examining the influence of LLMs on Wikipedia, and it is unlikely that Wikipedia has remained unaffected. For example, Reeves et al. (0224 an-
alyze Wikipedia user metrics such as page views and edit historics. Mcanwhile, Brooks et al. (0024) estimate the proportion of Al-generated content in newly created English Wikipedia articles using Machine- -enerrtteeeeeeeed Text (MGT) detectors. Given the richness and significance of Wikipedia, the impact of L.LMss on Wikipedia requires more comprehensive and detailed investigation.
Wikipedia is widely recognized valuable resource Singer et al. 2017 and its content is extensively utilized in AT research, particularly in Natural Language Processing (NTP)) tasks (Johnson et al. 2024b
Equal Contribution. * Corresponding Authors. f Project Lead.
1 We release all the experimental dataset and source code http:///tthaabcco///S//////// Wikipedia

---

Published in Transactions on Machine Learning Research (02/2020)
Direct Impact
HLP
Indirect Impact
HOLA
B
Word Frequency
Linguistic Style
Page Views
Machine Translation
RAG
Figure Our work analyzes the direct impact LLMs on Wikipodia, and explores the indirect impact of LLM-generated content on Wikipedia: Have LLMs already impacted Wikipedia, and if so, how might they influence the NLP community and human society?
For instance, Wikipedia pages are among the five datasets used to train GPT-3 (Brown et al. 2020 The sentences in the Flores-101 evaluation benchmark are extracted from English Wikipedia (Goyal et al. 2022) Lewis al. (2020)* 's work on Retrieval-Augmented Generation (RAG) treated Wikipodia as a source factual knowledge. Therefore, we aim investigate the influence of LLMs on machine translation and knowledge systems using Wikipedia as key resourcc.
Figure | illustrates the various tasks and research lopics discussed in this paper. Our first objective is to evaluate the direct impact of LLMs on Wikipedia, focusing word frequency, linguistic style, and page views. Then we explore the indirect effects on the broader NLP community, particularly in relation to machine translation benchmarks and RAG, both of which rely heavily on Wikipedia content for their corpora. This positions us to better observe and assess the evolutions and risks of Wikipedia in the era of LLMs. Our analysis yields number of significant insights:
Some Wikipedia pages have been influenced by LLMs, although the overall impact remains limited so far.
There has been slight decline in page views for certain scientific categories on Wikipedia, and similar trends are observed across multiple language editions, although the connection to LLMs remains uncertain.
If the sentences in machine translation benchmarks are drawn from I.M.-influnnced Wikipedia content, the evaluated models' scores may be inflated, potentially reversing their relative rankings.
Wikipedia content processed by LLMs could be less effective for RAG compared real Wikipedia content.
Based on these findings, we underscore the importance of carefully assessing potential risks and encourage further exploration of these issues. The key contributions of this paper are three-fold, as we are the first to: (1) quantify the impact of LLMs on Wikipedia pages across various categories; (2) analyze the impact of LLMSs on Wikipedia based on word usage and provide the corresponding estimates; and (3) examine how LLM-generated content affects machine translation evaluation and the efficiency of RAG systems.
This is also very likely the first paper to comprehensively analyze the impact of LLMs Wikipedia based data and simulations. It is important to note that, while some changes not obvious at the moment, the methods and perspectives we have proposed can be employed for long-term detection of the impact of LLMs on Wikipedia in the future.
2
Related Work
Wikipedia for NLP. Wikipedia has long been utilized in various NLP applications (Strube & Ponzel
2006
Mihalcea Csomai, 2007 Zesch et al.) 2008) Gabrilovich Markovitch, 2009 Navigli & Ponzetto In the era of LLMs, Wikipedia also plays an important role, such as in fact-checking (Hou et al.
2010
2024
and reducing hallucinations (Semnani
2023). Writing Wikipodia- articles is also one of the
LLM
applications (Shao et al. 2024
2

---

Published in Transactions on Machine Learning Research (02/2026)
LLMs for Wikipedia. Researchers are trying use LLMs to enhance Wikipedia, including articles (Adak Mihindukulasooriya et al. 2024) and edit process (Johnson et al. 2024a). Some researchers have compared ULM-genertted or rewritten Wikipedia articles with human-writter
et al. 2025 Wikidata (Peng et al. 2024
ones, yiclding differing conchusions (Skarlinski et al. 2024 Zhang et al. 2025a
There are studies on the impact of LLMs on its page views (Reeves et al, The detection of Al-gencrated content has been hot rescarch topic in recent Wang et al.) 2025 Zhang el al., 2024), including its application to Wikipedia ar- 2024). But MGT detectors have notable limitations (Doughman et al. 2024). and researchers are also exploring other methods for estimating the LLM impact, such as word frequency analy- sis Liang ct al. 2024 Geng & Trotta 2024]. Moreover, the emergence of LLM-assisted edits on Wikipedia has raised concerns about preserving the encyclopedia's style and editorial standards (Ashkinaze et al. 2024). Contamination of its articles with LLM-ggnnrated text can create harmful feedback loops in model training (Shumailov et al. 2023
Estimation of LLM Impact.
2024
Lyu ct al.
2025
years Wu et al.
2025
ticles Brooks et al.
3 Data Collection
In this paper, we focus on data from Wikipedia and Wikinews, both under the Wikimedia Foundation.
Wikipedia uses a hierarchical classification system for articles. It begins with top-level categories that cover broad fields, which are then divided into more specific subcategories. Only pages four or five levels away from our larget category are included in our study. Then we scrape the Wikipedia page versions from 2018 to 2025, using the January 1 snapshot of each year. To minimize the impact of topic-specific words, only those ranked within the top 10,000 in the Google Ngram dataset" are included in the calculations.
We are interested in Wikipedia pages that belong to the following categories: Arl, Biology, Compuler Science (CS), Chemistry, Mathematics, Philosophy, Physics, Sports. Among them, Philosophy has the smallest number of articles (31,132), and CS leads with the largest number (55,121). More details on data collection and processing are shown Appendix A For a better comparison, we also collect 6,690 Featured Articles (FA), along with their corresponding 2,029 simple English versions (where available) as Simple Articles (SA).
While Wikipedia is the main focus of this paper, we also collect Wikinews articles from 2020 to 2024 to generate questions in Section 5.2 There are over a hundred news per year, covering wide variety of topics.
4 Direct Impact from LLMs
4.1 Direct Impact 1: Word Frequency
Since LLMs are extensively applied writing-related tasks, we aim to investigate whether the text in Wikipedia articles has changed well. For example, we found that the frequency of certain words favored by LLMs has indeed increased, such "crucial" and "additionally" (Geng & Trotta 2024 Kobak et al. 2024), as shown in Figure 2
To further investigate whether the changes in word frequency are coincidental or part of a collective shift, we calculate the frequency changes of more words and estimate the impact of LLMs n in one set of Wikipedia articles S based on the following formula (Geng el al. 2025):
i(S) Eier (J:(S) (S))
(1)
Eier (ST(S))))²
A, f((S2) f((St)
(2)
S((S))
where (S) represents the frequency of word i in the set of texts S real dataset, (S) represents the one if LLMs do not affect the texts, the set of words used for estimation, fi(S,) and fe(S2) represent the frequency of word i for another set of articles before and after LLM processing, respectively.
"Gogle Ngram dataset: http::///....eeeeoccceeee////tts
3

---

Published in Transactions on Machine Learning Research (02/2026)
Frequency:
Wikipddia
Original
0.5958
Frequency Trend of 'crucial
Frequency Trend "ddittonally
teer mnrr
2.03
0.27
2.11
Frouucncy (or
D24
0.25
0.2055
0.02
0.5
0.01.6
0.0217
2013
2235
2023
2033
2022
2004
2218
2019
2030
2022
2002
2003
2004
2025
additionally
Year
9ear
Figure 2: Word frequency before and after LLM processing, and its evolution in Wikipedia articles.
We take the average of the word frequencies from the 2018, 2019, and 2020 versions of the pages as fi (S). To construct word frequency data fS22 reflecting the impact of LLMs, we use GPT-4o-mint to revise the January 2022, versions of Fealured Arlicles, with the prompt "Revise the following senlences:".
By setting thresholds for f and f, we select commonly-used and LLM-sensitive word combinations / to estimate the impact of LLMs. We perform grid search over the parameter space used for vocabulary selection. Specifically, f ranges from 500 to 20,000 with a step size of 500, and f ranges from 0.05 to 1.0 with a step size of 0.05. For cach (f"; () combination, we begin by estimating the LLM impact over the pre-LLM period (2018 2022). Using only these pre-LLM estimates, we fit a linear regression to model the natural evolution trend of Wikipedia articles.
We then evaluate cach paramcter combination based two criteria: (i) how well the linear model fits the data, measured by the coefficient of determination R², and (ii) the stability of the pre-LLM baseline, measured by the absolute value of the fitted slope. Parameter combinations with R² close to 1 and slopes close to indicate good baseline before LLM adoption. To avoid reliance on single criterion, we select the intersection of the TOPk parameter combinations ranked by R² and by slope.
LLM Impact the Pull Texts of Wikipedia Arti cles
LLM Impact on the Full Texts of Wikipedia Articles
Art
Math
Art
Math
Blo
Phio
50
Po
Chem
Phy
Cham
Phy
C5
Sports
C5
Sports
0.8
1-pact/5) Takkttin velue
0.6
0.6
Impoct(9) Relttve
-02
-02
2018
2019
2020
2023
2023
2024
2025
2018
2019
2022
2025
(a) Different word combinations.
(b) Same word combinations.
Figure 3: Impact of LLMs on Wikipedia pages, estimated based on simulations of Featured Articles.
For the f value, we propose two strategies: First, the target words should frequently appear in the first section of Featured Articles, as we use this part of the actis for LLM simulation when estimating f; second, the target words should frequently appear in the target category. For the first strategy, when calculating the impact of the LLM on different pages, the selected vocabulary combination remains the same. For the second strategy, the influence on pages of different categories will be estimated using the vocabulary combination corresponding to each category.
4

---

Published in Transactions on Machine Learning Research (02/2026)
Finally, the selected words to estimate impacts for the post-LLM period (2023 (023-. We subtract the extrapolated pre-LIM linear trend from the estimates to separate causality. When setting TOPk 250, the LLM impact is approximately 1% for the articles in certain categories, as illustrated in Figures 3 But different texts still lead to different estimations, and using different words for estimation will also produce different results. Detailed results parameter selection and impact estimation are shown Appendix B
Finding 1: While the estimation results vary, the influence of LLMs on Wikipedia is likely to become more significant over time. In some categories, the impact has exceeded 1%.
4.2 Direct Impact 2: Linguistic Style
We also investigate the current and futurr impact of LLMs Wikipedia from more linguistic perspectives, examining the evolution of Wikipedia content at Word, Sentence, and Paragraph levels by comparing the texts before and after LLM processing under the same standards.
Criteria
Effect of LLM processing
Trends in real data
Figures
Auxiliary Verb Rate %
Ma
1d
To be Verb Rate %
Corrected Type-Token Ratio (CTTR)
Long Words Rate
Conjunction Rate %
18
Noun Rate %
19
Preposition Rate %
20
Promouns Rate %
One-syllable Word Rate %
22
Average Syllables per Word
23
Passive Voice Rate %
4b
4e
Long Sentence Rate %
24
Average Sentence Length
Average Parse Tree Depth
26
Clause Rate %
27
Promoun-initial Sentence Rate %
28
Article-iniiial Sentence Rate %
29
Dale-Chall Readability
4c
30a
Automated Readability Index
30b
Fresch-Kincaid Grade Level
4c
4f
Flesch Reading Ease
40
30c
Coleman- Liau Index
4c
30d
Gunning Fox Index
30c
Summary linguistic style trends. The second column indicates the effects of LLM processing. The third column shows Wikipedia trends over time.
Table 1:
Word Level. In addition to word frequency used before, we can also consider it from a broader perspective at the level of words. For instance, the frequency of auziliary verbs indicates the ability of model to convey complex reasoning and logical relationships (Yang et al.) 2024). Lexical diversity, often measured by the corrected type-token ratio (CTTR), reflects the varioty of words (Wróblewska (2023) revealed that the lexical diversity of humans is higher than that of ChatGPT-3 but lower than that of ChatGPT-4, suggesting newer models have surpassed human writing in that metric. Furthermore, the
al.) 2025). Herbold et al.
5

---

Published in Transactions on Machine Learning Research (02/2026)
0.200
Audi 32 Netoo Wegerro
0.177
0.100
OineCraa Goom
T-X Goade
0,50
0.125
0.100
va
broa
CA-2211
0.875
56
SA-Gem
TAIGTT NeGemiitc-on
SN-GPT
PX-GPT
Cokman-
(a) Auxiliary verbs proportion.
(b) Passive voice proportion.
(c) Readability metries comparison.
0.00
M0b00 2004 (80
1/2
Boldi 90ki (8)
1.100
1.0
0.075
Relative Velic
-0.04
0.8
0.050
-0.00
0.6
0.000
0.025
0.2
-0,050
0.12
000775
2025
2024
2024
2024
202
An
Bio
Chem
CS
Math
piso
Phy
Sports
2018
2019
2021
2022
Art
Bio
Chem
Math
mile
Phy
Sports
2018
2020
2011
Bi
Chem
CS
Math
Pholo
my
Sports
2015
202-
2022
(d) Change in auxiliary verbs rate.
(e) Change in passive voice rate.
(I) Change Flesch Kincaid.
Figure 4: The results of linguistic style comparison, including the real Wikipedia pages and L.M-simulated pages. The three subplots below represent the differences compared to the data from 2020.
proportion of specific parts of speech (POS) commonly used as stylistic feature assessing the quality of Wikipedia articles (Moás & Lopes 2023 Georgiou (2025) showed that LLM-generated text employed more nouns, while human--ritlen lext employed more auxiliaries and pronouns.
Sentence Level. In terms of sentence structure, we focus on sentence length and the use passive voice AlAfnan & MohdZuki, 2023). Regarding sentence complexity, we analyze both the depth of the entire synlactic tree and the clause ratio (lavarone et al., 2021). Reinhart al. (2025) revealed that LLMs use present participial clauses at to 5 times the rate of human text while using the passive voice at roughly half the rate as human texts.
Paragraph Level. For the paragraph dimension, which essential for Wikipedia's oducational mis- 2024b), we seek guidance from readabilily evaluation (Moés & Lopes 2023 Trokhy-
sion (Johnson et al.
movych et al.) 2024). where six traditional formulas included in our study: Automated Readability 2018), Coleman-Lian Inder (Antunos & Lopes 2019), Dale-Chall Score (Patel et al.
Inder (Mehta et al.
2011H, Flesch Reading Ease [Eleyan et al. 2020 Flesch Kincoid Grade Level (Solnyshkina et al. 2017)
and Gunming Fog inder Swieczkowski & Kulacz 2021 The detailed definitions of all metrics are provided
in Appendix C.3
LLM Simulation. Wikipedia articles are not static, and their linguistic styles are difficult to remain the same under different measurement metrics. To understand the link between these trends and LLMs, we simulate the real Wikipedia with GPT-4o-mint and
then compare the changes before and after the process. Specifically, we instruct both models to revise Featured and Simple articles using prompt mentioned in Section 4.1 and additionally use GPT-4o-miné to generate Wikipedia-style articles using prompt "Generate Wikipedsa-stylee article tilled [tille} and return only the article body in plain text."

---

Published in Transactions on Machine Learning Research (02/2026)
Results. Table presents the summary of the trends in linguistic style in real Wikipedia articles and LLM simulations. The detailed outcomes are illustrated in Figure E and Appendix C Although we have plotted the results from 2020 in the these figures, the trends summarized in the table are based on the data in the LLM cra, that is, after 2023. For example, our simulation results reveal that LLMs substantially reduce the use of auziliary verbs, with Gemini employing even fewer than GPT, as shown in Figure 4a Consistent with this tendency, the usage of auzillary verbs on real Wikipedia pages shows marginal decline from 2020 2025, as depicted in Figure Ad However, this consistency not observed for all metrics. For instance, the trends of passive voice proportion in Figures 4b and A are not the same. For paragraph level, Figure Ac presents the results of six readability metrics, all of which indicate that L...-generated texts tend to be less readable. The Flesch Kincaid score in Figure initially decreases and then rises, and the score after LLM simulation also increases.
The observed changes in Wikipedia articles largely consistent with the preferences of LLMs under most metrics.
Finding 2:
4.3 Direct Impact 3: Page View
The analysis of Wikipedia's page view data (i.e., the number of times a Wikipedia page is accessed by users) can yield many interesting conclusions (Piccardi et al. 2021 2024). Similar to the work of Reeves et al. 2024), we transform the page view counts of Wikipodia articles using the inverse hyperbolic sine function.
2.9
Phy
7.25
Art
Philosophy
Chem
Sports
Math
2.00
575
2018
2019
2020
2023
2022
Jan.
2024
2025
Nov. 2025
(a) Page views Across Different Categories in English Wikipedia.
EN
Papostess
Jan.
Jan. Jan.
Jan.
(b) Page views Across Wikipedia of Different Languages
Figure 5: Daily page views of Wikipedia pages. The y-axis represents page view values after smoothing with seven-day time window and being transformed via the Inverse Hyperbolic Sine (IHS) function.
Figure ba shows page views across different categories in English Wikipedia. Notably, there was a slight decline in page views across scientific categories since 2024. Reeves et al. (2024) examined changes in

---

Published in Transactions on Machine Learning Research (02/2026)
Wikipedia page views across various languages from up to January 1, 2024, but analysis covers pages from different scientific categories and extends to 2025. The latest data actually leads to different findings. and one recent study has reached a similar conclusion (Lyu et al. )025).
To generalize our findings beyond English (EN) Wikipedia, we further analyze the page views of Featured Articles four major language editions, German (DE), Spanish (ES) and French (FR), as shown Figure 5b The page views of Wikipedia articles in these languages also exhibit decline from 2024, with the drop being especially sharp in the Spanish edition. Detailed data for different language editions is shown in Table and the means of the page view values plotted in Figures 31 in the appendix.
In the second half of 2024, there slight decline in page views across scientific calegories, and its connection to the use of LLMs requires further investigation.
Finding 3:
5 Indirect Impact from LLMs
5.1
Indirect Impact 1: Machine Translation
Overall. Sentences of some machine translation benchmarks are derived from Wikipedia. If these bench- marks also inftuenced by LLMs, what impact would it have the evaluation results?
Facebook-NLLB BLEU Performance on Original vs LLM-Influenncdd Benchmarks
100
+9.71
+20.99
Original
+18.12
+25.19
LLM..rocessed
+23.60
Score
+11.93
+8.80
+2.16
40
PT
FR
DE
ZH
IT
ES
RU
AR
HI
JA
KO
Figure 6: Facebook-NLLB BLEU scores on the original benchmark and the LLM-Influenced benchmark.
Helsinki-NLP BLEU Performance on Original vs LLM-Influenced Benchmarks
+22.61
Original ILM-Processed
+16.25
+4.98
+3.47
BLEU Score
+12.25
+2.52
-4.20
20
PT
FR
ZH
IT
ES
RU
AR
HI
KO
Figure 7: Helsinki-NLP BLEU scores the original benchmark and the LLM-Inftueneed benchmark.

---

Published in Transactions on Machine Learning Research (02/2026)
Benchmark Construction. We utilize the Flores datase which consists of multiple sentence sets, each containing parallel translations of the same Wikipedia sentence across many languages. For our experiments, we keep the original English (EN) sentence in each set, and use GPT-4o-mini to translate this English sentence into the remaining languages with the prompt "Translate the following text to (target language] We then replace the original non-English sentences with these L.M--generated translations, forming an LLM- influenced version of the benchmark. The following 11 widely used languages are included our simulations: Modern Standard Arabie (AR), Mandarin (ZH),, German (DE), French (PR), Hindi (HI), Italian (IT), Japanese (JA), Korean (KO), Brazilian Portuguese (PR), Russian (RU), Latin American Spanish (ES).
Evaluation Pipeline. We evaluate multiple machine translation models by translating the benchmark sentences and assessing the output with four metrics: BLEU (Post 2018 COMET (Rei et. al. 2020
ChrF PPopović 2015), and BERTScore (Zhang et al. 2019
Models. We compare the translation results from three models: Facebook-NT.L.F multilingual model supporting 200+ languages [Costa-Jussà et al. 2022]; Google (mT55 pre-trained on data covering 101 languages (Xue et al., 2021 and NL.P's bilingual Transformer model trained on OPUS
corpus Tiedemann & Thottingal 2020: Tiedemann et al. 2023
Results. In most cases, machine translation models achieve higher scores on the GPT-processed bench- mark, as shown in Figures G and 7 Moreover, the relative ranking of machine translation models could be reversed. For example, in the French translation Lask, Pacebook-NLLB gets a lower BLEU score (87.04) than Helsinki-NLP (88.39) in the original benchmark, but a better score in the GPTTprocessed benchmark (96.75 75
us 89.40). More results are shown Figure 32 33 34 and 35 in the Appendix.
The impact of LLMs on the benchmark could not only inflate the translation scores across different languages but also distort the comparison of translation abilitics between models, thereby undermining the benchmarks' ability to reflect true translation effectiveness.
Finding 4:
5.2 Indirect Impact 2: RAG
Overall. RAG can provide more reliable and up-to-date external knowledge to mitigate hallucination in LLM generation (Gao et al. 2023). Wikipedia is one of the most commonly applied general retrieval sets in previous RAG work, which stores factual structured information in scale (Fan al. [024]. In the process of translation using LLMs, some information may also be lost or distorted (Mohamed el al. 2025). Therefore, are curious how the effectiveness of RAG might change if Wikipedia pages are influenced by L.Mss. Our experiment procedure is illustrated in Figure S and the detailed steps are listed below.
Question Generation. GPT-4o-mini and Gemini- 1. .5-flash are used to generate multiple-choice questions (MCQs) based on Wikinews articles. In order to generate some questions that are not too easy for LLMs, we refer to the prompt in the work of Zhang et al. (025b),, shown in Figure 9
Knowledge Base. We construct the knowledge base using Wikinews articles from 2020 to 2024. Each
article is preprocessed and split into smaller text segments, then vectorized via BERT (Devlin et al. 2019 We then indexed these vectors using FAISS, library for efficient similarity search and clustering of dense vectors, to enable rapid retrieval (Douze el al. 2024).
Retrieval and Generation. The questions are vectorized using BERT and a similarity search is conducted with FAISS. The three most relevant segments are retrieved and used as contextual information. These
htttp:://gthub..cm//leatea uncased http:://ugginggfcccco///

---

Published in Transactions on Machine Learning Research (02/2026)
LLM Simulation
Questioning Method
Original Content
Based Original Content
Aske
LLM-Revised Content
Ask with Full Content
RAG Based on LLM Revised Content
PROMPT|
Question
O
Generation
LLM-Revised Content
Which country's President questioned the protection of Kharkiv without targeting missile launch sites? A) Germany
B) United Kingdom C) Ukraine
D) France
President Emmanuel Macron France questioned how the Ukrainians were expected protect
Kharkiv without targeting "the point from which the missiles are fired? ".... The United Kingdom also expressed desire for Ukraine to have the ability attack inside Russia.
A U.S. official confirmed Ukraine's s use of U.S. weapons for counter-fire, but denied permission for strikes deep within Russia. This follows Ukrainian request and aligns with statements from French President Macron and the U.K. supporting Ukraine's ability to target launch sites.
B
Figure 8: on the extracted Wikinows data. Various questioning methods are employed evaluate the impact of LLMs the RAG process. The LLMs being queried include GPT-4o-ming GPT-3.5, and DeepSeek- V3.
and Geminé- f-5-flas are used to generate multiple-chn choice questions (MCQs) based
Prompt
You are to generate three self- celf-contained multiple-choice questions based on the facts mentioned in the following content. Avoid questions that reference the content directly. Each question should include all relevant context and directly name any referenced items, avoiding pronouns like "i,, "the game, " or "the person." Do not inchude phrases that reference the source or context, such as "mentioned in the article" or "according to the text."
Figure 9: Prompt for Wikinews-baase questions.
segments are then combined with the question in a prompt template to query GPT-3.5 and GPT-4o-mini. The final answer is generated based on both the LLMs" prior knowledge and the retrieved content.
Questioning Methods. We conduct experiments using three types of queries. First, we can query the LLMs directly to obtain answers, using the prompt shown in Figure 36 (Direct Ask). Second, the Wikinews articles used generate the question included the prompt, as shown in Figure 37 To explore the impact of LLM-generated text, we apply the prompt "Revise the following news" " to Wikinews articles. We consider three versions of each article: the original text. (Full (Original)), and revisions processed by GPT-40-mini (Full (GPT)) or Gemini-1.5-FP Flash (Full (Gemini)). Finally, in the RAG-based setting, relevant information is first retrieved from constructed knowledge base, after which the models are queried using the prompt shown in Figure 10 The knowledge base can be built from original Wikinews articles (RAG (Original)) or from articles revised by GPT-4o-mini (RAG (GPT)) or Gemini-1.5-Flas (RAG (Gemini)).
Results. Figure illustrates the summary the accuracy rates of the LLM responses under different scenarios, with more detailed results provided in Appendix The analysis based on these results leads the following conclusions:
10

---

Published in Transactions on Machine Learning Research (02/2026)
Prompt: Asking with knowledge base
prompt (
"Uee context to answer user questions."
f"Question: {question\n"
f"Reference context: (topk_ans)) an
f"Only need to give the correct option without explanation."
Figure 10:
Prompt used in the asking with a knowledge base setting,
Accuracy Heatmap of Different Questioning Pairs
Q: GPT-40 mini
74.35
86.31
86.69
81.76
95.90
95.55
88.05
A: GPT-4o-mini
90
85
Gemini-L.5-llash A: GPT-4o-mini
67.27
82.53
81.82
76.72
93.56
92.85
83.20
Percentage (%)
880
Q: GPT-4o-mini A: GPT-3.5
66.
78.49
78.01
74.99
88.20
88.93
85.77
75
70
Semini--L-5-flash A: GPT-3.5
59.87
73.08
72.36
67.76
79.34
80.19
76.70
65
Direct Ask
RAG (Original)
RAG (GPT)
RAG (Gemini)
Full (Original)
Full (GPT)
Full (Geminl))
The accuracy rate of LLM responses under different settings. For each case, more than 1,800 questions based on Wikinews articles from 2020 to 2024 are used for simulations. More detailed results are presented in Appendix
Figure 11:
Higher Accuracy with Knowledge Base. Providing external knowledge greatly improves perfor- mance. With knowledge base, the accuracy of responses often exceeds 80%. This confirms the effec- liveness of RAG in enhancing factual accuracy.
Maximal Performance with Full Content. Providing the full news as context yields the highest accuracy, demonstrating the limitations of retrieval-based approaches in selecting the most relevant in- formation. In most cases with GPT-4o-mini, the full content approach achieves more than 93% accuracy, setting benchmark for ideal retrieval performance.
Impact of LLM-Revised Content. Compared to the cases using real Wikinews articles, the accuracy of responses based ChatGPTTprocesse pages shows little change and responses based Gemini- processed pages show clear drop in accuracy. This suggests that Gemini's rewriting may lead to the loss of some key information.
Declining Accuracy for Recent Events. In the absence of RAG, all models exhibit lower accuracy when answering questions derived from recent Wikinews articles (e.g., GPT-4o-mint shown in Table 10 of the appendix: 66.67% in 2024, GPT-3.5: 61.25% in 2024), while their accuracy is much better for older events (e.g., (020-2022). The reason also straightforward: these news events are not included in their training data. Morcover, DeepSock-V3. achieves the highest accuracy, which may be attributed its later knowledge cutoff date. In addition, Table I0 also report results rewritten using the newly released Gemini-3. Compared to outputs rewritten with Gemini 1.5, both RAG (Gem3) and Full
11

---

Published in Transactions on Machine Learning Research (02/2026)
(Gem3) exhibit performance improvements. This suggests that as LLMs continue to advance, the risk of information loss introduced by T.LM-based rewriting may be partially mitigated.
Case Study. To explore the impact of LLM-generated texts, we focus on cases in which the model answers correctly with the original content but fails with the LLM-revised version. Fig- ure g has provided one interesting example. The original passage contained an unambiguous clue, "President Emmanuel Macron of France questioned how the Ukrainians were expected to protect Kharkiv which directly supports the correct answer "France." However, in the LLM-revised ver- sion, the model reformulated the information into a more abstract and compressed form: ". ....aligns with statements from French President Macron and the U.K. ... This revision removes the explicit verb "questioned," merges multiple entities, and relocates key details. As result, RAG systems relying on the revised text. may incorrectly associate the query with "the U.K." due to lexical proximity. This illustrates how LLM-style rewriting can distort relational information and impair factual grounding in RAG systems. More examples are included in Appendix F.3 and LLM-gencrated texts may decrease accuracy in RAG lasks for several reasons:
Information Fusion Misleading: When LLMs merge multiple distinct and clear pieces of information into a single sentence, it can lead to misinterpretation as shown in Figure
Keyword Replacement and Omission:
LLM might replace or omit key terms, altering the original
meaning and causing misinterpretation in Figures
38
39
and 40
Abbreviation Ambiguity Misleading: LLMs use abbreviations inappropriately, leading to misinter-
pretation as shown in Figure [1
Introduction of Modifiers: Adding adjectives or modifiers can change the context and impact. the
text's accuracy, as illustrated in Figure 42
Retrieval Mismatch: Revised texts may either reduce the similarity between the question and the correct news or increase the similarity with irrelevant ones. In some cases, even small edits to the article lead to a failure in matching.
The results suggest that LLM-processed content may perform less effectively in RAG systems compared to human-created texts in some cases. If such content has impacted high-quality communities like Wikipedia, it raises concerns about the potential decline information quality in knowledge bases.
Finding 5:
6 Discussion and Conclusion
The relationship between Wikipedia and LLMs is bidirectional. On the one hand, Wikipedia content has been a key factor in the growth of LLMs. On the other hand, researchers have used NLP methods, including LLMs, to improve Wikipedia (Lucie-Aiméé Aimée et al. 2024). Humans and LLMs are coevolving (Geng & Trotta, 2025), and Wikipedia may be one of the bridges in this process. Our study also provides new insights into the risks associated with rescarch that uses Wikipedia data.
In this paper, we collect a large amount of weal-world data and conduct comprehensive experimental simula- tions. Our findings suggest that LLMs are impacting Wikipedia and the impact could extend indirectly to some NLP tasks through their dependence chine translation may gradually shift towards the language style of LLMs, albeit in small steps. In addition, the accuracy of RAG tasks may decline when LLM-rrvisod Wikipedia pages are used, indicating potential risks of using LLMs to support Wikipedia or similar knowledge systems.
Wikipedia content. For instance, the target language for ma-
Although some of the changes may not be immediately apparent, our work offers a framework for long long-term term monitoring. These results will also serve as excellent illustrations of the impact of AT on society, given the significant amount of human engagement with Wikipedia. This kind of social impact is already taking place, but it may not have been adequately addressed by the AI community.
https://en. wikiness. www.neww..wg/.ikt.i..
tteesttttteteeeeeeeeeeeeeeeeeeee strike striieeRussian watpttttrrrtttrtreeeeeee Kharkiv
12

---

Published in Transactions on Machine Learning Research (02/2026)
Limitations
Although conduct several experiments to evaluate the impact of LLMs on Wikipodia, study has cerlain limitations. First, some analyses are primarily correlational, identifying palterns but not definitively attributing observed changes to LLMs. The causal relationships of some impacts, such as the pages views, roquire more detailed discussion.
Second, the lack of field experiments limits our insights inlo the actual human-in-the-loop editing processes behind Wikipedia article creation. Real-world editing involves complex interactions between humans and sophisticated LLM-based tools. These dynamics may not be fully captured by our simulated studies.
Additionally, when assessing the readability of Wikipedia pages, we rely only on traditional metrics based on formulas, such the Flesch-Kincaid score. However, recent advances in NLP have shifted towards computational models (François) 2015). Moreover, in the RAG task, our Wikinews dataset is not large enough compared to the Wikipedia page dataset, which may limit the generalization our findings.
Acknowledgments
This work benefited from funding from the French State, managed by the Agence Nationale de la Recherche, under the France 2030 program (grant reference ANR-23-JACL-00083) This work is also supported in part by the ENS-PST, BeYs Chair in Data. Science and Cybersecurity.
References
Sayantan Adak, Pauras Mangesh Meher, Paramita Das, and Animesh Mukherjee. Reversum: A multi- staged renrentel-angmeenedd generation method enhance wikipedia tail biographies through personal narratives. In Proceedings of the 31st International Conference Compulational Linguisties: Industry Track, pp. 732-750, 2025.
Mohammad Awad AlAfman and Siti Fatimah MohdZuki. Do artificial intelligence chatbots have a writing style? investigation into the stylistic features of chalgpt-4. Journal of Arlificial intelligence and technology, 3:85 94, 2023.
Hélder Antunes and Carla Teixeira Lopes. Analyzing the adequacy of readability indicators to a non-english language. In Experimental IR Meels Multilingualily,, Multimodalily, and Interaction: 10th International Conference of the CLEF Association, CLEF 2019, Lugano, Switzerland, September 9-12, 2019, Proceed- ings 10, pp. 149-155. Springer, 2019.
Joshua Ashkinaze, Ruijia Guan, Laura Kurek, Eytan Adar, Ceren Budak, and Eric Gilbort. Secing like an ai: How llms apply (and misapply) wikipedia neutrality norms. arXiv preprinl arXiv:2007004188 2024.
Creston Brooks, Samuel Eggert, and Denis Peskoff. The rise of ai-gearr content in wikipedia, anXiv preprint ar Xix:2224 10. 08.08444 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-100 1901, 2020.
Marta R Costa-Jussá, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint ar Xr:iv:2007.0 0477,4 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short popers), pp. 1171-4186, 2019.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Jad Doughman, Osama Mohammed Afzal, Hawau Olamide Toyin, Shady Shehata, Preslav Nakov, and Zeerak Talat. Exploring the limitations of detecting machine-generated text. arXiv preprint or orXiu:2406. 11073, 2024.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosscini, and Hervé Jégou. The faiss library. ar preprint arXiv:2401.08281 2024.
Derar Eleyan, Abed Othman, and Amna Eleyan. Enhancing software comments readability using flesch reading case score. Information, 11:430, 2020.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijic Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting llss: Towards retrieval-augmente large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6491-6501, 2024.
Thomas François. When readability meets computational linguistios: a new paradigm in readability. Recue française de linguistique appliquée, 20:79-97, 2015.
Evgeniy Gabrilovich and Shaul Markovitch. Wikipedia- brased based semantic interpretation for natural language processing. Journal of Artificial Intelligence Rescarch, 34:443-44 2009.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arxiv preprint arXiv:2312. 10997, 2023.
Mingmeng Geng and Roberto Trotta. Is chatgpt transforming academics' writing style? orXiv preprint arXiv:240408662 2024.
Mingmeng Geng and Roberto Trotta. Human IIm coevolution: Evidence from academic writing. In Findings of the Association for Computational Linguistics: ACI 2025, pp. 12689-12696, 2025.
Mingmeng Geng, Caixi Chen, Yanru Wu, Yao Wan, Pan Zhou, and Dongping Chen. The impact of large language models in academia: from writing to speaking. In Findings of the Association for Computational 2025.
Linguistics: ACL 2025, pp.
Georgios P Georgion. Differentiating between human-written written and ai-generated texts using automatically extracted linguistic features. Information, 16(11):979, 2025.
Jim Giles. Special report internet encyclopaedias go head to head. nature, 438:900-901, 2005.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Marc'Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522-538, 2022.
Steffen Herbold, Annette Hautli-Janisz, Ute Heuer, Zlata Kiktova, and Alexander Trantsch. A largo-scale comparison of human-written versus chatgpt-generale essays. Scienlific reports, 13(1):18617, 2023.
Yufang Hou, Alessandra Pascale, Javier Gaaneror--Cano, Tigran Tchrakian, Radu Marinescu, Elizabeth Daly, Inkit Padhi, and Prasanna Sattigeri. Wikicontradict: benchmark for evaluating Ilms on real-world knowledge conflicts from wikipedia. arXiu preprint arXiv:2406. 19805, 2024.
Benedetta lavarone, Dominique Brunato, Felice Dell'Orletta, et al. Sentence complexity in context. In CMCL 2021- Workshor on Cognétive Modeling and Computational Linguistics, Proceedings, pp. 186-199. Association for Computational Linguistics (ACL), 2021.
Isaac Johnson, Guosheng Feng, Robert West, et al. Edisum: Summarizing and explaining wikipedia edits at scale. arXiv preprint arXiv:2404.03. 2024a.
Isaac Johnson, Lucie-Aimée Kaffee, and Miriam Redi. Wikimedia data for ai: review of wikimedia datasets for nlp tasks and ai-assisted editing. Xiv preprint or orXin:24 10. 0.08918, 2024b.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Dmitry Kobak, Rita González-Márquez, Emöke-Agnes Horvát, and Jan Lause. Delving into chatgpt usage in academic writing through excess vocabulary. arXiv preprint arXin:2406.0770 07016, 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmentee generation for knowledge- intensive ulp tasks. Advances in Newral Information Processing Systems, 33:9459 9474, 2020.
Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al. Monitoring ai-modified content scale: A case study on the impact of chatgpt on ai conference peer reviews. arXiv preprint arXtt:2403.07118 2024.
Lucie Aucie-Aimés Angela Fan, Tajuddeen Gwadabe, Isaac Johnson, Fabio Petroni, and Daniel Van Strien. Proceedings of the first workshop on advancing natural language processing for wikipedia. In Proceedings of the First Workshop on Advancing Natural Language Processing for Wikipedia, 2024.
Liang Lyu, James Siderius, Hannah Li, Daron Acemoglu, Daniel Huttenlocher, and Asuman Ozdaglar. Wikipedia contributions the wake of chatgpt. arXiv preprint arXiv:2503. 00757, 2025.
Manish P Mehta, Hasani W Swindell, Robert W Westermann, James T Rosneck, and T Sean Lynch. Assess- ing the readability of online information about hip arthroscopy. Arthroscopy: The Journal of Arthroscopic Related Surgery, 34:2142 2149, 2018.
Rada Mihalcca and Andras Csomai. Wikify! linking documents to encyclopedic knowledge. In Proceedings of the sicleenth ACM conference on Conference on information and knowledge management, pp. 2007.
Nandana Mihindukuulasooriya, Sanju Tiwari, Daniil Dobriy, Finn Arup Nielsen, Tek Raj Chhetri, and Axel Polleres. Scholarly wikidata: Population and exploration of conference data in wikidata using llms. In International Conference on Knowleigge Engineering and Knowledgge Management, pp. 243 259. Springer, 2024.
Pedro Miguel Moás and Carla Teixeira Lopes. Automatic quality assessment of wikipedia articles sys- tematic literature review. ACM Computing Surveys, 56:1-37, 2023.
Amr Mohamed, Mingmeng Geng, Michalis Vazirgiannis, and Guokan Shang. Lim broken telephone: Iterative generation distorts information. arXiv preprint 1xXii:2502.20255 2025.
Roberto Navigli and Simone Paolo Ponzetto. Babelnet: Building a very large multilingual semantic network. In Proceedings of the 48th annual meeting of the association for computational linguistics, pp. 216 225, 2010.
Priti P Patel, Ian C Hoppe, Naveen K Ahuja, and Frank S Ciminello. Analysis of comprehensibility of patient information regarding complex craniofacial conditions. Journal of Craniofacial Surgery, 22:1179- 1182, 2011.
Yiwen Peng, Thomas Bonald, and Mehwish Alam. Refining wikidata taxonomy using large language models. In Proceedings of the 39rd ACM International Conference on Information and Knowledge Management, pp. 5395 5399, 2024.
Tiziano Piccardi, Miriam Redi, Giovanni Colavizza, and Robert West. On the value of wikipodia as a gateway to the web. In Proceedings of the Web Conference 2021, pp. 249 260, 2021.
Tiziano Piccardi, Martin Gerlach, and Robert West. Curious rhythms: Temporal regularities of wikipedia consumption. In Proceedings of the International AAAI Conference on Web and Social Media, volume 18, pp. 1249 1261, 2024.
Maja Popović. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the lenth workshop on statistical machine translation, pp. 392-30 222-395,22 2015.
Matt Post. A call for clarity in reporting bleu scores. Xiu preprint ar xrxi::8804.08771 2018.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Neal Reeves, Wenjie Yin, Elena Simperl, and Miriam Redi. the death of wilipeeda""-explorigg the impact of chatgpt on wikipedia engagement. Xiv preprinl or arXiv:2405. 10205, 2024.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. arXiv preprint (tt::20090090220 2020.
Comet: neural framework for mt evaluation.
Alex Reinhart, Ben Markey, Michael Laudenbach, Kachatad Pantusen, Ronald Yurko, Gordon Weinberg, and David West Brown. Do 1mss write like humans? variation in grammatical and rhetorical styles. Proceedings of the National Academy of Sciences, 122(3))22222451122, 2025.
Sina J Semnani, Violet Z Yao, Heidi C Zhang, and Monica S Lam. Wikichat: Stopping the hallucination of large language model chatbots by few-shot grounding on wikipedia. arXiv preprint arXiw:2305. 14292 2023.
Yijia Shao, Yucheng Jiang, Theodore A Kanell, Peter Xu, Omar Khattab, and Monica S Lam. Assisting in writing wikipedia- like articles from scratch with large language models. arXiv preprint K1v:22402.12007 2024.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.174 17493, 2023.
Philipp Singer, Florian Lemmerich, Robert West, Leila Zia, Ellery Wulezyn, Markus Strohmaier, and Jure Leskovec. Why we read wikipedia. In Proceedings of the 26th international conference on world wide web, pp. 1591-1600, 2017.
Michael D Skarlinski, Sam Cox, Jon M Laurent, James D Braza, Michaela Hinks, Michael J Hammerling, Manvitha Ponnapati, Samuel G Rodriques, and Andrew D) White. Language agents achieve superhuman synthesis of scientific knowledge. arXiv preprint arXiv:2409. 13740, 2024.
Marina Solnyshkina, Radif Zamaletdinov, Ludmila Gorodetskaya, and Azat Gabitov. Evaluating text com-
plexity and flesch-kincaid grade level. Journal of social studies education research, 8:238 248, 2017.
Michael Strube and Simone Paolo Ponzetto. Wikirelate! computing semantic relatedness using wikipodia. In AAAI, volume 6, pp. 1419 1424, 2006.
Damian Swicezkowski and Slawomir Kulacz. The use of the gunning fog index to evaluate the readability of polish and english drug leaflets in the context of health literacy challenges in medical linguistios: An exploratory study. Cardiology Journal, 28:627-623 2021.
Jörg Tiedemann and Santhosh Thottingal. OPUS-MT Building open translation services for the World. In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT), Lisbon, Portugal, 2020.
Jörg Tiedemann, Mikko Aulamo, Daria Bakshandaeva, Michele Boggia, Stig- Arne Grönroos, Tommi Niemi- nen, Alessandro Raganato Yves Scherrer, Raul Vazquez, and Sami Virpioja. Democratizing neural machine translation with OPUS-MT. Language Resources and Evaluation, pp. 713-755, 2023. ISSN 1574-0218. doi: 10.1007/s10579-0222 (20-000003-000000
Mykola Trochymovych, Indira Sen, and Martin Gerlach. An open multilingual system for scoring readability of wikipedia. arXiv preprint arXiv:2406.01883 01835, 2024.
Matthew A Vetter, Jialei Jiang, and Zachary J McDowell. An endangered species: how 1ims threaten wikipedia's sustainability. AT & SOCIETY, pp. 1-14, 2025.
Christian Wagner and Ling Jiang. Death by ai: Will large language models diminish wikipedia? Journal of the Association for Information Science and Technology, 2025.
Yuxia Wang, Artem Shelmanov, Jonibek Mansurov, Akim Tsvigun, Vladislav Mikhailov, Rui Xing, Zhuohan Xie, Jiahui Geng, Giovanni Puccetti, Ekaterina Artemova, et al. Genai content detection task 1: English and multilingual machine-generated text detection: Ai vs. human. Xiv preprint or xxx::25011.1000 11012, 2025.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Anna Wróblewska, Marceli Korbin, Yoed N Kenett, Daniel Dan, Maria Ganzha, and Marcin Paprzy- cki. Applying text. mining to analyze human question asking in creativity research. or arviv preprinl arXiv:2501.088 020000 2025.
Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia Sam Chao, and Derek Fai Wong. A survey Ammgeneratted text detection: Necessity, methods, and future directions. Computational Linguistics, pp. 1 66, 2025.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Adilya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Linguistics. doi: 10.1 1653/v1120222
wwwcl-main.nn.. URL httpe://aclanthologg 2tt.aaanttooooooooo/22222 www.///...////////..
41
Qiyuan Yang, Pengda Wang, Luke Plonsky, Frederick Oswald, and Hanjie Chen. From babbling to fluency: Evaluating the evolution of language models in terms of human language acquisition. arXiv preprint ar arXiv:2410.. 13259, 2024.
Torsten Zesch, Christof Müller, and Iryna Gurevych. Extracting lexical semantic knowledge from wikipedia and wiktionary. In LREC, volume 8, pp. 1646 1652, 2008.
Jiebin Zhang, Yu Eugene, Qinyu Chen, Chenhao Xiong, Dawei Zhu, Han Qian, Mingbo Song, Weimin Xiong, Xiaoguang Li, Qun Liu, et al. Wikigenbench: Exploring full-length wikipedia generation under real-world scenario. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 5191-5210, 2025a.
Qihui Zhang, Chujie Gao, Dongping Chen, Yue Huang, Yixin Huang, Zhenyang Sun, Shilin Zhang, Weiye Li, Zhengyan Fu, Yao Wan, and Lichao Sun. I.M--as--coouthhr: Can mixed human-written and machine- generated text be detected? In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 409-436, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v /2024 findings-naacl.n. URL https: //aclanthology.org///
findings-nasn naacl. .99/
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. Xi preprint ar XxXi:1904.006755 2019.
Yueheng Zhang, Xiaoyuan Liu, Yiyou Sun, Atheer Alharbi, Hend Alzahrani, Basel Alomair, and Dawn Song. Can Ilms design good questions based on context? arXiv preprint rxXiv:2501003491 2025b.
A Data Collection and Processing
The detailed classification in Wikipedia poses a problem in our data crawling process: When iteratively querying deeper subcategorics without limit, the retrieved pages may become less relevant to the original topic (i.e., the root category). To address this issue, we select an appropriate crawl depth for each calegory to balance the number of pages with their topical relevance, as shown in Table 2
We also exclude redirect pages, they do not contain independent content but link to other target pages. After crawling the pages, we clean the data by extracting the plain text and removing irrelevant sections such as "References, "See also," " "Further reading," " "External links," " "Notes, and "Footnotes." For Wikinews, we use the Extracss extension which provides an API to retrieve plain-text extracts of page content.
9rexx Extracts extension: http::////ww...iii....../r/////a ort.eeae.e.cc.......ee..eeeeeeer
17

---

Published in Transactions on Machine Learning Research (02/2026)
Category
Art.
Bio
Chem
CS
Math
Philo
Phy
Sports
Crawl Depth
4
4
5
5
5
Number of Pages
50,810
41,237
49,516
55,121
43,888
31,132
38, 14,144
48706
Table 2: Number of Wikipedia articles crawled per category.
Languages
German
Spanish
English
French
Number of Pages
2943
1363
6690
2254
Table 3: Number of Wikipedia Featured Articles different language.
B LLM Impact
Let Ty denote the threshold for the bascline frequency fi, and let T, denote the threshold for the scnsitivity measure fz. We define a discrete search space ② as follows:
= ((T)) Ty E Tr, E Tx}
(3)
where Ty (500, 1000,.... 20,000} represents the sequence of frequency thresholds, and Tr (0.05, 0.1 1.0} represents the sequence of sensitivity thresholds.
For each parameter pair 0 (r),TT) TT) θ, we construct a candidate vocabulary set le- This set consists of words that are both frequently used in the target texts and sensitive to LLMs, i.e., words that are either particularly preferred or clearly avoided by LLMs:
1 J
P +1
1e
STS
To evaluate the reliability of the candidate set To, we examine the natural evolution trend of the indicator f during the pre-LLM period, defined as Tpre = .2018...., 2022).. We model this trend using linear regression, where the slope of the fitted line is denoted by ag.
To avoid bias from reliance on single metric, let Sga and S. denote the TOP K parameter sets ranked by descending R2 and ascending Jool, respectively. The optimal parameter set O* is defined as their intersection:
θ SSp n..
(5)
When setting TOPx = 250 to estimate the full texts of Wikipedia articles, 18 (J*,f) combinations that satisfy the conditions are: (5500, 0.45), (6000, 0.45), (5000, 0.4), (4500, 0.35), (5000, 0.35), (7500, 0.45). (7000, 0.45), (6500, 0.45), (8000, 0.45), (8500, 0.5), (9000, 0.5), (9500, 0.5), (15000, 0.5), (10000, 0.5), (15500, 0.5), (10500, 0.5), (16500, 0.5), (17500, 0.5).
Specifically, when we take J. < 5500 and + 2 0.45+1 "run," " "called," " "players," " "taken," " "largely," " "seen," "struck," " "remains," " "mainly," "press," "make," " "ap- peared," " "long." " "launched," "sometimes," " "carlier," " "like," " "form," " "wide," " "player," " "sent," " "subsequently." "brought,"" " "had," " "upon," " "despite," " "significant," "killed," " "making," " "us," " "can," " "given," "parts," " "lead- ing," " "sce," " "came," " "primarily," "important," " "throughout," "worked," "failed," " "this," "p, "very," " "saw," "large," "due," " "features," " "usually," " "just," " "however," "attempt," " "built," "different," "because," " "vic- tory," " "popular," "men," "across," " "commonly," " "out," "there," "placed," " "mostly," " "went," " "particularly," "serving," " "often," " "having," "following," " "operations," " "died," " "established," " "wrote," " "forced," " "so," " "al- most," " "where," "but," " "whose," "lived," "next," " "helped," " "served," "various," "generally," "soon," " "while," " "number," " "written," " "win," " "people," " "initially," " "considered, " "used," "these," " "rest," "along," " "located," "won," 7 "role," " "limited," " "numerous," " "use," " "fought," "about," " "result," " "opened," " "up," " "subsequent," "then," " "ended," " "caused," " "within."
0.45² 115 words that satisfy the conditions are: "moved," "
Figures 12 13 and 14 illustrate LLM impact estimated using different TOPk-
18

---

Published in Transactions on Machine Learning Research (02/2026)
LLM Impact the First. Sections of Wikipedia Articles
LLM Impact on the First Sections of Wikipedia Articles
Phioo
Chem
0.5
CS
Sports
Impeet(%)) value
0.4
Im9art%)) Relattee
0.2
0.5
0.
-0.5
Mhy
2016
2020
2021
2024
2025
2010
2014
2022
2023
2025
(a) Different word combinations.
(b) Same word combinations.
Figure 12: Impact of LLMs on the first section of Wikipedia pages, estimated when setting TOPk = 250.
LLM Impact on the Full Texts Wikipedia Articles
LLM Impact the Full Texts Wikiped Articles
1.50
1.25
Sports
Impact/Si Reladoe value
1.00
Impartrii Readve
-025
2018
2013
2020
2021
2022
2023
2024
2025
2018
2013
2020
2023
2022
2023
2024
2025
(a) Different word combinations.
(b) Same word combinations.
Figure 13: Impact of LLMs on the full texts of Wikipedia pages, estimated when setting TOPk 300.
C Linguistic Style
C.1 Word Level
"To Be Verbs Figure 15 illustrates that LLMs significantly reduce the usage of "To Be" verbs (e.g., replacing "is important" " with "demonstrates significance "), with Gemini using fewer such verbs than GPT. Moreover, marginal decline in the usage of these verbs is observed in actual Wikipedia pages.
Lexical Diversity As shown in Figure 16 revised articles display slightly higher CTTR, with texts revised by GPT exhibiting greater lexical diversity than those revised by Gemini. When tasked with gener- ating wiki-style articles, GPT achieves the highest lexical diversity. Over time, the vocabulary used across different Wikipedia categories has become increasingly varied.
Long Words Figure shows that LLM-revised texts generally contain a higher proportion of long words than iaman-written pages, with Gemini producing the most pronounced increase. From 2020 to 2025, a substantial increase in the usage of long words observed in the first section of Wikipedia pages.
19

---

Published in Transactions on Machine Learning Research (02/2026)
LLM Impacc on the First Sections of Wilk pedia Articles
LLM Impact on the First Sections of Wikipedia Articles
Phlo
Ro
Ph
1.25
Chem
Phy
Phy
C5
Sports
1.0
CS
Sports
Impeet100) valow
Impare/3) Relattve
0.25
0.00
-0.75
2030
2019
2023
2024
2025
2018
2014
2022
2023
2025
(a) Different word combinations.
(b) Same word combinations.
Figure 14: Impact of LLMs on the first section of Wikipedia pages, estimated when setting TOPk = 300.
"o Be' Verbs Preportion
To Verbs Proportion Trend [Fisg
6.07
To Verbs Proportion Thems (Full)
2.175
0.06
0.00
0,150
5,325
1000
0.02
-0.02-
0.300
0.00
0.04
6.075
-0.05
0.02
-0.08
3.025
3.000
-0.10
54
SA-OPF 54 dem
Carpus
Figure 15: "To Be" Verbs.
19 20 and 21 show that LLMs lead to a slight increase in the use of nouns, accompanied by corresponding decrease in pronouns. Prepositions and conjunct ions remain stable after LLM simulation. On Wikipedia pages, the proportion prepositions has steadily increased, while the proportions of other parts of speech have remained stable.
Parts of Speech Figures 18
Syllables Figures 22 and 23 illustrates that the proportion of one-syllable words declines in articles revised by LLMs, with Gemini employing even fewer such words. Mcanwhile, the average syllables per word increase, suggesting a preference for polysyllabic words by LLMs. However, these two metrics remain relatively stable across different Wikipedia categories.
20

---

Published in Transactions on Machine Learning Research (02/2026)
Corrected Lexical Diversity
Math
Chan
Conecced Lesteal Diveraty TTT
0.05
Corrected Lesical Diversiy (CTTR)) Trend (Rrst)
0.35
Conrected Lexical Diversity (CTTR) Thend (Full.
0.06
0.25
0.03
0.20
0.02
6.01
0.30
0.00
-0.01
54 GFT SAGen
DA-GPT Gan wiiablee
-0.02
Corpus
Figure 16: Corrected Type- Type-Token Token Ratio (CTTR).
Long Words Proportion
Char
0.3
Long Words Propertion Trand (First
0.05-
Long Words Proportion Trend (Fal)
Lang Korth Propottion
0.2
Reletion Value
-0.05
0.1
-0.10
0.0
0.2-
-6.1
0.20
2A
5A-2P7 Sa-Gam
FAIHT Wi-SemWWlleyy
Corous
Figure 17: Long Words Rate.
Conjunctions Proportion
Conjunctions Proportion Trand (Frso
Conjunctions Proportion Trand (Rull
2.20
Conlunttion Tresetton
0.08
2,55
6.06
0.03
0.04
0.31
6.00
8.00
SA
S4-dum 1 Corpus
MI PA Wgyy
Figure 18: Conjunctions Proportion.
Nauns Proportion
Char
2.8
Nouns Proportion Trend (Rrst)
0.4
Nouns Proportion Trend (Full)
3.5
0.20
0.3
Weune trovettonn
0.15
Value
0.10
0.05
-0.1
0.05
SA. DF7
SA Sem
PA Wemaea
Cortus
Figure 19: Nouns Proportion.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Prepositions Preportion
Chan
0.30
0.15
Prepositions Proportion Trend (Rrst)
0.25
Prepasttinns Propartion Trend (Pull)
0.20
Propnke Frezee
0.10
Relatiun Valuee
0.35
0.05
0.30
0.00
0.05
0.00
2.00
-0.05
SA- SA- 54-Gam
16 Corpus
IA-FP:
-0.05
Figure 20: Prepositions Proportion.
Pronours Proportion
Math
Pronouns Proportion Tenn (FrsQ)
Pronouns Proportion Trand (Fund
0.12
2,30
Value 196)
0.04
Protaens Fro2e
6.04
3.04
0.02
0.00
6.62
0.00
-0.01
8.00
S4-CPT SA-Gem
M Corous
TN-FP
D.GamMeieyy
-0.02
Figure 21: Pronouns Proportion.
One-Syllable Words Preportion
One-Sy One-Syllable Words Propoction: Trand Peot!
One-Syllable Words Proportion Tunn (Ful)
Otte Sytteile Wards Perembon
0.1
0.15
0.0
-0.1
0.00
-0.05
5A
SA ger
54 dem
FAOFT PA dem Walagn
Corous
Figure 22: One-Syllable Words Proportion.
Syllables Word
Average Syliables per Word Trand (Firs))
0.002
Average Syllapics per Word Trand (Ful)
Ruarrga Sydabws Werr
3.75
0.008
0.001
Value
2,55
2.00
0.004
.0.001
3.35
0.022
0.002
2.50
-0.003
-0.002
SA SA-Gam M
IN-FP: Pa-
Corous
Figure 23: Average Syllables per Word.
22

---

Published in Transactions on Machine Learning Research (02/2026)
C.2 Sentence Level
Sentence Length Figure 25 shows that both the average sentence length and the proportion of long sentences show significant increase after being processed by the LLM. Additionally, the period from 2020 to 2025 has seen a notable rise in these two metrics across Wikipedia pages, indicating & trend towards longer sentence structures.
Long Sentence Rate
Math
Soats
Fly
Soutts
Philuedhyy
Long
Long
1.50
1.2
Lang Cettennceeaa
Balatow Value (5G
1.25
1.3-
1.00
0.8
0.6
0.50
0.25
8.6-
0.00
0.3-
2038
2019
2020
2021
2022
2023
2024
2025
2018
2019
2020
2021
2022
2023
2024
2025
Figure 24: Long Sentence Rate.
Average Sentence Length
Chee
Cen
Average Sentence Langth Trand (Are))
Average Sentence Langth Trand (Ful)
Aurrae Senvence Lanig..
0.25
0.20
6.20
0.35
0.30
0.10
0.05
Corous
2018
2019
2020
2022
2024
2025
2018
2013
2020
2023
2022
2023
2004
2025
Figure 25: Average Sentence Length.
Sentence Complexity According to Figures 26 and 27 after revisions by GPT, Simple Articles show increase in complexity, while Featured Articles exhibit only minor changes. This may suggest that LLMs do not generate sentences at the highest possible complexity, but instead maintain complexity at certain level. For real Wikipedia pages, a steady year-on-year increase in these two metrics has been observed, indicating shift towards more complex sentence structures.
Average Parse Tree Depth
Muth soophy
Spont
my
Spams
16
Depth Thand (Aiss)
Parse Trend
mneraeeere Pante The Deph
0.08
0.04
0.03
0.02
6.02
0.00
Wii-itle
0.01
2018
2019
2020
2001
2022
2023
2024
2025
2018
2019
2020
2023
2022
2023 2024 2025
Figure 26: Average Parse Tree Depth.
Pronoun and Article Sentences LLMs tend to avoid starting sentences with pronouns (c.g.,
"It") articles (e.g. "The"), shown in Figures 29 and
28
For example, it might replace "The team
23

---

Published in Transactions on Machine Learning Research (02/2026)
Proportion
Cher
chen
2.0
Clause Proportion Trand (Fins)
0.6
Cluse Propartion Trend (Ful)
Elus movertionn
0.4
Daliativ Value
-6.2-
SA-cem Sa-dem
FaiGPT Radden warlyy
Corpus
Figure 27: Clause Proportion
worked hard to finish the project time." with "Hard work from the team ensured the project was completed time." However, in real Wikipedia pages, Article-initaal sentences have increased, while pronoun-initial sentences remain stable from 2020 to 2025.
Iropottton Dentencae Stardno art hensus
Proportion Sentences Starting
Char
Dtem
1.0
Sentences Starting with Froncun Trand (Rrst)
0.4-
Senterces Starting with Bronoun Trend
0.3
0.8
Reletioon Value
0.6-
0.1
Relative
0.0
02
.03
5A
5A-2P7 Sk-lam
TAGHT Milaa Waleey
-02
Corous
Figure 28: Proportion of Sentences Starting with Pronoun.
Properce of Sentaton Statting wth Mth.A000 AMIC
Proportion of Senterces Starting with Article
Be
Soata
The exccht
feg
Pay
Sports
C5
Starting
Starting Article
0.2
6.2
Relatoow Value
6.1
Relative
022
0.1
FA
2018
2019
2020
2021
2022
2023
2024
2025
2018
2019
2020
2021
2022
2023
2024
2025
Figure 29: Proportion of Sentences Starting with Article.
C.3 Paragraph Level
We Textsta 1o to calculate six paragraph metrics. Textstat is casy-to-usc library to calculate statistics from the text. It provides range of functions to analyze readability, sentence length, syllable count, and other important textual features. Through the LLM simulation process, we discover that LLMss tend to generate articles that are harder to read. Figure 30 suggests that the readability of Wikipedia pages has shown only slight variation over the years and does not appear to be influenced by LLMs at this stage.
Dale-Chall Readability: uses the concept of difficult words, combining it with the average sentence size estimate readability.
24

---

Published in Transactions on Machine Learning Research (02/2026)
di.f fifuuttosssds 100) + 0.0496 words words
DC = 0.1579
(6)
senlences
Automated Readability Index: Estimates readability by combining the average word length with the average sentence size.
characters
0.5 words
ARI 4.71
sentences 21.43
(7)
words
Coleman-Liau Index: Similarly to ARI, estimates readability by combining the average word length with the average sentence size.
CL 5.88 characters 29.6 sentences
words 15.8
(8)
words
Flesch Reading Ease: Using the average sentence size and amount of syllables per word, computes a value between 0 and 100, where 0 indicates the text is difficult to understand.
words 84.6 syllables
FRE- 206.835 1.015 senlences
(9)
words
Flesch-Kincaid Grade Level: Same as FRE, but provides US grade levels instead of values between 0 and 100.
FK = 0.39 words sentences 11.8 syllables words 15.59
(10)
Uses the concept of complezwords, which the number of words with three or more syllables. The higher its value, the more difficult. is the text to read.
Gunning Fog Index:
words
+ 100 complexwords:
GFI 0.4( sentences
(11)
words
631
06
-06
622
-.04
02
2022
2011
2019
2013
Ham
2000
myy
(a) Dale-Chal Readability.
(b) Automated Readability.
(c) Flesch Reading Ease.
118
1
-005
6.15
-315
005
4.38
acto
2003
2533
Spoel
2012
2012
Buro
200
(d) Coleman- Liau Index.
(e) Gunning Fog Index.
Figure 30: Changes in readability metrics of Wikipedia pages.
25

---

Published in Transactions on Machine Learning Research (02/2026)
D Page Views
CS
Phy
Philesophy
Bio
Chem
Sports
600
106
Jan. 1, 2018
Jan 1. 2019
Jan. 1, 2020
Jan. 1, 2021
Jan. 2022
Jan. 1, 2023
Jan. 1, 2024
Jan. 1. 2025
Nov. 1, 2025
(a) Page views Across Different Categories in English Wikipedia.
EN
FR
1004
Uml
Jan 2015
2029
2020
2022
2023
2024
2023
Rov. 2025
(b) Page views Across Wikipedia of Different Languages
Figure 31: Daily page views of Wikipodia pages. The y-axis represents page view values after smoothing with a seven-day time window and being transformed via mean aggregation.
E
Machine Translation
These are the languages in our benchmarks: English Bongg-aatn-san112933 Modern Standard Arabic (arb- Arab-stan1318), Mandarin Camn-Hans-beij122 German (deu-Latunstan11955) French (fra-Latn-stan129) Hindi Siin-Deva-hind1260 Italian (ita-Latn-itall2822) Japancse (jpn-Jpan-mucl1143), Korean kore1280). Brozilian Portuguese boo-bLaatbbraz1246) Russian (rus- (tus-Cyrl-russ1233) Latin American Spanish (spa-Tatn-ann 1244aa
For Google-T55 T5 shown in Table 6 German (DE) initially has BLEU score of 30.24, which rises to 44.18 in the GPT-processed benchmark, marking another substantial improvement.
We also evaluate our results using BERTScore, as shown Tables many tokens in the candidate sentence are similar to tokens in the reference sentence, capturing how much of the candidate sentence is relevant to the reference. Recall measures how many tokens in the reference sentence are similar to tokens in the candidate sentence, capturing how much of the reference sentence is represented in the candidate. As for Fl Score, BERTScore combines precision and recall into an Fl score, the harmonic mean of the two. This balanced measure provides a single metric that reflects the accuracy and completeness of the candidate sentence relative to the reference.
and 7 Precision measures how
Overall, our conclusion that LLM-inffunneed benchmarks inflate translation scores across different languages still holds when using Score the evaluation metric.
26

---

Published in Transactions on Machine Learning Research (02/2026)
Facebook-NLLB ChrF Performance on Original vs LLM-Influenced Benchmarks
100
+12.61
+4.69
+18.12
+17.00
+4.72
Original LLM-Processed
+10.32
+11.13
+10.91
+5.39
+16.73
ChrF Score
60
20
PT
FR
DE
ZH
IT
ES
RU
AR
H1
KO
Figure 32: Facebook-NLLB ChrF scores the original benchmark and the LLM-Inftnnnced benchmark.
Facebook-NLLB COMET Performance on Original vs LLM-Influenced Benchmarks
-2.66
Original
+1.67
+1.51
+3.50
+4.46
+1.62
+0.85
LLM- Procerreeseed
COMET Score
+0.63
+0.26
60
20
RU
Figure 33: Facebook-NLLz COMET scores the original benchmark and the LLM-Influeneed benchmark.
Precision
Recall
Fl
Precision
Recall
F1
G
AR
0.869
0.891
0.854
0.880
0.861
0.886
AR
0.861
0.872
0.854
0.869
0.857
0.870
DE
0.890
0.915
0.874
0.900
0.882
0.907
DE 0.896
0.923
0.888
0.916
0.892
0.919
ES
0.885
0.953
0.867
0.943
0.876
0.948
ES 0.884
0.952
0.871
0.949
0.877
0.951
FR
0.919
0.944
0.902
0.930
0.910
0.937
FR 0.919
0.944
0.911
0.941
0.915
0.942
HI 0.876
0.900
0.865
0.892
0.870
0.896
HI 0.812
0.822
0.785
0.798
0.798
0.809
0.883
0.986
0.865
0.925
0.874
0.930
IT 0.880
0.935
0.869
0.931
0.874
0.933
JA 0.829
0.850
0.808
0.824
0.818
0.836
JA 0.608
0.612
0.625
0.629
0.617
0.620
KO 0.849
0.878
0.842
0.869
0.845
0.873
KO 0.610
0.614
0.602
0.605
0.605
0.609
PT 0.923
0.946
0.913
0.937
0.918
0.941
PT 0.929
0.954
0.926
0.951
0.927
0.952
RU
0.875
0.908
0.858
0.892
0.866
0.899
RU
0.868
0.896
0.859
0.889
0.863
0.892
ZH
0.839
0.861
0.778
0.797
0.806
0.827
ZH
0.852
0.870
0.825
0.845
0.838
0.857
Table 4:
BERTScore ovaluation results on the Facebook-NILL NLLB translation outputs.
Table 5: BERTScore evaluation results on the Helsinki-NLP machine translation outputs.
27

---

Published in Transactions on Machine Learning Research (02/2026)
Helsinki-NLP ChrF Performance on Original vs LLM-Influenced Benchmarks
+10.48
+0.14
+16.81
Original LLM-Processed
+11.66
+10.58
+6.02
ChrF Score
-3.03
20
PT
FR
DE
ZH
IT
ES
RU
AR
HI
KO
Figure 34: Helsinki-NLP ChrF scores the original benchmark and the LLM-Inftuenced benchmark.
Helsinki-NLP COMET Performance on Original vs LLM-Influenced Benchmarks
100
+1.60
+1.52
+4.33
Original
+1.54
+0.90
LLM-Processed
80
COMET Score
+0.87
+0.22
60
40
Figure 35: Helsinki-NLP COMET scores on the original benchmark and the JLM-Influenced benchmark.
F
RAG
F.1
Experiment Setup
Table S
presents the LLM parameters employed in RAG simulations, such the knowledge cutoff date, lemperature, and lop-p. Table g shows the annual number of questions generated by different LLMs.
Prompt: Direct Asking
prompt (
f Answer the following questions. The format should be per 1. C)..." f NeNed to answer all questions and mark the question number."
f f'Only need to give each answer without explanation. Questions: {text)"
f"The format should be as per 1. In2. C)..."
f"All questions are required to answered. Don't skip any."
Figure 36: Prompt used in the direct asking setting.
28

---

Published in Transactions on Machine Learning Research (02/2026)
BLEU
ChrF
COMET
O
0
G
G
DE
71.52
80.09
84.27
93.62
83.91
85.63
FR
68.33
65.93
87.86
86.32
85.49
87.01
Table 6: Google-T5 results BLEU, ChrF, and COMET metrics.
Precision
Recall
F1
G
DE
0.873
0.898
0.845
0.869
0.858
0.883
FR.
0.887
0.907
0.849
0.869
0.867
0.887
Table 7: BERTScore evaluation results (Precision, Recall, F1) on the Google T5 translation outputs.
Models
Knowledge Cutoff
Temperature
Top-p
GPT-3.5
September 2021
1.0
1.0
GPT-4o-mini
October 2023
1.0
1.0
Gemini-1.5-fal
May 2024
1.0
0.95
DeepSeek-V3
July 2024
1.0
1.0
Gemini Pro
January 2025
1.0
1.0
Table LLM parameters Used in RAG simulations.
Year
2020
2021
2022
2023
2024
Number of GPT generated Questions
348
453
390
126
240
Number of Gemini generated Questions
348
453
393
426
240
Table 9: Annual Number of Questions Generated by Different LLMs.
Prompt: Full Texts Provided
prompt f fUUse context to answer user questions." f f" ququesioo: {question)" f f"Reference context: {content)"
f f'Only need to give the correct option without explanation. Don't t miss ') or option." "
I'If there is no answer content, just return None. Don't give string."
Figure 37: Prompt used in the full texts provided setting.
29

---

Published in Transactions on Machine Learning Research (02/2026)
F.2 Detailed Results
and 15 present detailed RAG results. Questions generaled from Wikinews in 2024 are likely the most up date; therefore, we focus on their results, which correspond to the last row in each table. The results indicate that LLM-revised content tends to be less effective as a knowledge source, as accuracy based on LLM-revised texts often lower than that based on the original texts.
Tables 10
Y
Direct
RAG
R (GPT)
R (Gem1.5)
RAG (Gem3)
Full (Original)
Full (GPT)
Full (Gem1.5)
Full (Gem3)
20
75.86%
85.34%
85.63%
79.60%
85.63%
95.98%
95.40%
87.36%
94.25%
21
71.74%
86.31%
88.96%
79.69%
82.56%
96.03%
96.03%
88.08%
93.38%
22
80.00%
89.49%
87. 87.18%
84.10%
85.90%
95.64%
95.64%
88.97%
93.85%
77.46%
87 .7.0
87.09%
83.33%
84.98%
96.01%
94.84%
87.09%
93.19%
24
66.67%
83.33%
84.58%
82.08%
83.33%
95.83%
95.83%
88.75%
90.42%
Table 10: GPT-4o-mini performance RAG task (problem generated by GPT).
Year
Direct Ask
RAG
RAG (GPT)
RAG (Gem)
Full (Original)
Full (GPT)
Full (Gem)
2020
66.95%
82.76%
82.47%
75.86%
93.68%
91.38%
84.20%
2021
64.68%
81.90%
82.34%
75.06%
94.04%
93.82%
82.12%
2022
73.54%
86.01%
85.75%
78.88%
94.66%
93.89%
83.21%
2023
69.95%
82.39%
83.10%
78.40%
92.49%
92.25%
83.57%
2024
61.25%
79.58%
75.42%
75.42%
92.92%
92.92%
82.92%
Table 11: GPT-4o-mini performance on RAG task (problem generated by Gemini).
Year
Direct Ask
RAG
RAG (GPT)
RAG (Gem)
Full (Original)
Full (GPT)
Full (Gem)
2020
68.65 53.68%
77.59%
78.16%
74.14%
86.21%
87.93%
87.36%
2021
67.11% 17.11%
79.25%
79.25%
74.17%
87 47.42%
88.30%
84.99%
2022
70.26%
82.82%
80.77%
78.97%
88.46%
90.51%
88.46%
2023
64.08%
74.88%
76.06%
71.83%
86.85%
88.73%
84.27%
2024
60.42%
77.92%
75.83%
75.83%
92.08%
89.17%
83.75%
Table 12: GPT-335 Performance on RAG task (problem generated by GPTT).
30

---

Published in Transactions on Machine Learning Research (02/2026)
Year
Direct Ask
RAG
RAG (GPT)
RAG (Gem)
Full (Original)
Full (GPT)
Full (Gem)
2020
66.95%
72.70%
72.41%
68.97%
77.87%
79.31%
77.59%
2021
58. 72%
73.73%
71.74%
68.21%
81.02%
79.47%
74.17%
2022
62.09%
74.05%
72.77%
69.47%
82.44%
82.19%
80.41%
2023
56.57%
73.24%
74.88%
67.14%
77.46%
79.58%
74.65%
2024
55.00%
71.67%
70.00%
65.00%
77.92%
80.42%
76.67%
Table 13: GPT-3.5 Performance on RAGG task (problem generated by Gemini).
Year
Direct Ask
RAG
RAG (GPT)
RAG (Gem)
Full (Original)
Full (GPT)
Full (Gem)
2020
89.66%
81.03%
81.90%
78.45%
98.28%
97.70%
90.52%
2021
84.55%
77.26%
79.25%
69.09%
97.57%
97.79%
87.20%
2022
90.00%
80.77%
81.54%
75.90%
97.69%
97.44%
90.00%
2023
83.57%
73.00%
76.29%
69.72%
96.71%
95.54%
88.03%
2024
82.08%
75.42%
72.50%
75.42%
97.08%
96.67%
86.25%
Table 14: DeepSeck-Vi V3 Performance on RAG task (problem generated by GPT).
Year
Direct Ask
RAG
RAG (GPT)
RAG (Gem)
Full (Original)
Full (GPT)
Full (Gem)
2020
83.62%
74.14%
75.57%
69.54%
94.54%
95.11%
83.05%
2021
54.97%
69.98%
72.41%
63.36%
95.81%
94.70%
81.68%
2022
84.48%
78.12%
78.37%
65.39%
96.18%
94.91%
84.99%
2023
65.02%
73.00%
74.88%
64.55%
95.54%
94.37%
83.33%
2024
75.83%
74.17%
70.42%
70.00%
95.42%
93.75%
84.58%
Table 15: DeepSeck- V3 Performance on RAG task (problem generated by Gemini).
31

---

Published in Transactions on Machine Learning Research (02/2026)
F.3 Case Study
Example 1 Keyword Replacement
Title: NASA says object that hit Florida home is from International Space Station
Question: On which date did NASA release a pallet containing old nickel hydride batteries from the International Space Station?
A) March 8, 2021 B) March 11, 2021 C) April 22, 2024 D) March 8, 2020
Original Context: ...A A pallet containing old nickel hydride batteries released from the ISS March 11, 2021, after new batteries were installed.
LLM Revised Context: ... The debris, part of 5,800-1b cargo pallet released from the ISS in March 2021, unexpectedly survived atmospheric re-entry.
"https://en www.neww..e......eaeeammm....... www../ww..rg/..k./../// ottp/siki/HASS.aaas..oo www.eeeeae_ooeeeeeeeeeeeeeeeeeee thattttttttt__ee that_hit__ Florids.....r hwmeidd__ommm..e is. from. International. Space_
Station
Figure 38: The news revised by LLMs omits key information about the specific date that NASA released the pallet, causing the RAG system unable to determine the correct date and ultimately selecting A.
Example 2
Keyword Replacement.
Title: Latin American expedition of Viktor Pinchuk: meeting with the traveler took place in Yalta
Qucstion: What hobby involves collecting recordings of ethnic performors and is practiced by Viktor Pinchuk?
A) Philophony B) Ethnomusicology C) Hobo tourism D) Cultural preservation
Original Context: From every trip an expedition, Viktor Pinchuk brings CDs with record- ings of ethnic performers; the traveler's collection has already accumulated a significant number of them (not counting several hundred digital editions world-famous musicians). The hobby is called "philophony", and the subject it is called a philophonist.
LLM Revised Context: Pinchuk, self-described "philophonist," " has amassed hundreds of CDs and digital recordings of ethnic and world music.
/https://en wikineus. Lzaveler_took_place Yalta
http::////ewwwwwee..ee...e.. www.ki/ww...///../////s...... wwtp//k////tttt....kkkk orrreeoee//aaaaaaeeeeeeennnnn peteriica_eepeditiiio Viktor ViktoorPPicchutle heaeiluk__eeettg__tteeee the
Figure 39: The RAG system mistakenly selects when using the LLM-revised text because the revision omits key details, such the explicit mention of the hobby's name, "philophony."
32

---

Published in Transactions on Machine Learning Research (02/2026)
Example 3 Keyword Replacement
Title: New Zealand Navy ship HMNZS Manawanui capsizes one nautical mile from shore
Question: What was the name of the Royal New Zealand Air Force aircraft that assisted in the evacuation of the crew from HMNZS Manawanui?
A) Boeing P-8 Poscidon B) Airbus A320 C) Lockheed Martin C-130J D) Boeing 737, Original Context: They were rescued with assistance from the Rescue Coordination Centre (RCCNZ) and Royal New Zealand Airforce Boeing P-8 Poseidon.
LLM Revised Context: ... All 75 crew were safely evacuated with assistance from the Rescue Coordination Centre and the Royal New Zealand Air Forcc.
Whtpps: :tops://oen //t. www.nnww..neuss...eeemmmmm.. Silkinews..rrvvikk////...... org/wiks/Mexx Zealand, Aaealand_aavyy Navy_ ship HMNZS Manavanui Yanavanui_capsizes, pauticallllc... amo__autical_nileeee.. _ew___le___rree_e from_
Shore
Figure 40: LLMs omit key information, such as the aircraft's name.
Example 4 Abbreviation Ambiguity Mislcading
Title: At least 20 die in Odesa in Russian missile strike, Ukraine reports
Question: How many employees of the State Emergency Service of Ukraine were reported victims of the missile strikes in Odesa?
A) One B) Five C) Seven D) Ten
Original Context: ... Among the dead are an employee of the State Service of Ukraine for Emer- gency Situations (SSES) and a paramedic. Among the victims are seven employees of the State Emergency Service.
LLM Revised Context: Among the deceased are staff member of the State Service of Ukraine for Emergency Situations (SSES) and paramedic. Seven SSES personnel were among the injured, and medical workers also sustained injuries.
hhttps:/een. htttp::/en..wkkttew..oorrr wwkineus. httpi/eows.org/ttttttttLttL leasts ceats___ass_______ee_ _0_die__i____ in, Ode_,deaa__p in Russian wwwssaanmissilleeeeeeeaeeeeeaeee missile_strike, reports
Figure 41:
The original text use the full name "seven employees of the State Emergency Service," " while the LLM's revised text abbreviated this to "seven SSES personnel," causing RAG incorrectly choose A.
Example Introduction of Modifiers
Title: Arizona bans abortion for genetic abnormalitie
Question: What does Senate Bill 1457 in Arizona classify as a Class felony?
A) Seeking or performing an abortion because of a severe fetal abnormality
B) Seoking or performing an abortion due to the presence of genetic abnormality the child
C) Distributing abortion-inducing drugs via courier D) Soliciting funds for abortion
Original Context: ... The bill makes it Class 6 felony, the least severe, to seek perform abortion "because of a genetic abnormality of the child", defined as "the presence presumed presence of an abnormal gene expression in an unborn child," " but not a "severe fetal abnormality" considered "incompatible with life."
LLM Revised Context: ... Arizona Governor Doug Ducey signed Senate Bill 1457 into law on Tuesday, effectively banning abortions sought solely due to fetal genetic abnormalitics. The bill, which passed the Repuboicca-cootrrolled legislature after twice stalling and undergoing amend- ments to secure necessary votes, classifies seeking performing such abortions Class 6 felony.
http::///onniikiieessoo/r//iiiit wikineus wwtp:s:.....o.//......att httpiiiionaaaaaaaabbeeeeeaaaeeae caans__borttoo_fo_ggnmmeee www.gentticcabborralittttt
Figure 42: Although both versions explicitly exclude "severe fetal abnormalitics," " the revision changes "genetic abnormality" "fetal genetic abnormalities," causing LLMs to misinterpret the information.
33


Reviews:
[{"review_id": "Vj0iVq3F9p", "reviewer": "Reviewer_Y14v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present an empirical analysis of how LLMs are already influencing Wikipedia and how such influence may affect downstream NLP tasks, especially MT benchmarks and RAG.\n\nThe authors collect Wikipedia snapshots (2020–2025), Featured Articles, Simple English articles, and Wikinews articles. They conduct several direct and indirect impact analyses. Their direct analyses include estimating LLM influence through word-frequency patterns based on simulated GPT-edited pages. They also track linguistic-style trends at the word, sentence, and paragraph levels, finding that many recent changes in Wikipedia increasingly align with typical LLM writing tendencies. Beyond these surface effects, the authors demonstrate indirect effects: GPT-translated variants of the FLORES raise model scores while LLM-rewritten Wikinews articles reduce factual retrieval accuracy in RAG systems due to information fusion, omission, etc.\n\nStrengths:\n- The paper is clearly written and easy to follow.\n- The authors perform a multi-dimensional analysis (style, pageviews, MT, RAG) which is helpful to understand the problem.\n- The authors provide a strong basis with real Wikipedia data and extensive simulations.\n- The authors utilised a number of techniques in direct analysis to measure LLM impact. The trends provided are quite useful.\n- Declining accuracy for recent trends: I really liked this subsection how LLMs are really good when it comes to old data and do quite bad with new data.\n\nWeakness:\n- Many of the paper’s findings are correlational. The analyses rely on naturally occurring Wikipedia edits and pageview trends, which can be influenced by many factors besides LLMs. For example, shifting editor behavior, policy changes, etc. Moreover, the LLM rewriting process which authors talk about may not capture actual human-in-the-loop Wikipedia editing behaviour.\n - Adding to point above, one way of separating causality with correlation is to check if data from years before 2020 also follow certain trend when compared to post 2020?\n- The downstream effects on MT and RAG are demonstrated through controlled simulations rather than evidence of actual contamination. Therefore, the paper suggests possible mechanisms but does not establish definitive causal relationships.\n- The source text for FLORES200 came from Wikipedia but to ensure accuracy and fidelity, the translations were manually produced by professional, native-speaking translators. Picking up samples directly from Wikipedia for benchmarks is problematic because of data contamination-- LLMs have swallowed Wikipedia multiple times by now, existing LLMs will do really well on those benchmarks and as a result those benchmarks will become outdated the minute it comes out.\n- The LLMs produce “translationese” when prompted to translate from one language to another. As a result, other models score really high on the GPT benchmark whereas Wikipedia follows human-in-the-loop editing behavior in which contributors can replace translationese with culture-specific terms.\n- The authors mention a bunch of linguistic style trends in Table 1 but offer no definition for bottom 6 and how they are relevant to their objective.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides clear evidence for each major claim:\n- Word-frequency impact is quantified using an estimator applied to multiple word subsets. \n- Linguistic-style alignment is demonstrated with clear metrics and LLM simulations, showing consistent trends across stylistic categories.\n- MT inflation effects are supported with controlled experiments on 11 languages and multiple metrics, showing score increases.\n- RAG degradation is demonstrated with MCQs and multiple LLMs, including detailed case studies of where and why LLM-edited content fails.", "audience_yn": "Yes", "audience_explanation": "The paper talks about major NLP concerns: dataset contamination, benchmark reliability, LLM-driven language drift, and long-term risks for knowledge bases. Moreover, since Wikipedia data is involved in training and evaluating modern models, the audience will find these results relevant. These issues matter directly to:\n- NLP researchers using Wikipedia-based corpora.\n- MT researchers relying on FLORES-like benchmarks.\n- People working on RAG, knowledge-retrieval and LLM influence on human-generated content.", "requested_changes": "- My main concern is that the paper occasionally implies LLMs “caused” stylistic changes or page view declines, though evidence is correlational. State limits clearly and don’t imply cause-and-effect without evidence. It will be also great if authors can show the same trend for a language other than English to corroborate their claims.\n- Real Wikipedia edits may involve partial edits, human-machine interactions with different LLM quality. I believe that revising the whole article with the GPT approach may overestimate the stylistic impact. The changes should be a bit subtle just like how changes happen in Wikipedia.", "broader_impact_concerns": "- Wikipedia might need new policies for detecting or managing LLM-assisted edits to maintain the original linguistic styles. The paper should more explicitly discuss such mechanisms as well.\n- The authors showed how LLM contaminated articles will impact benchmarking but didn't how will it affect the models that are trained on such data. Will they continue doing normal or their performance will be impacted too?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RgXSxiTpBA", "reviewer": "Reviewer_SkDf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This article quantitatively analyzes LLM's impact on Wikepedia, including first-order impact such as frequency, linguistics style, page view; and second-order impact, such as machine translation benchmarking and RAG that rely on Wikepedia. It's a comprehensive study.\n\nIt calls upon an important problem, the impact of LLM on human knowledge production. Specifically, it uses Wikipedia as a key venue of human knowledge production: it's originally written by humans, read by humans, and now used to train LLMs, and also presumably edited by LLMs. Had any impact on Wikipedia by LLMs, the impact can take years to be seen, given the 2nd-order impact may take time to be noticed. Such impact can be subtle, nontheless being impact, and consequence being serious. \n\nThe results are important contributions. Results on both 1st-order and 2nd-order impact are notable and largely convincing. The authors stated LLM impact on Wikipedia is \"approximately 1%-2% in certain categories\"(figure 3), it notes that the linguistic style changes in Wikepedia highly correlated to the linguistic style changes in LLM simulations (table 1). The 2nd degree impact might be some of the earlies efforts on this rather neglected problem, suggesting that LLM's impact on Wikepedia does not stop at Wikipedia. \n\nIt identifies a set of methodologies to study LLM impact on human knowledge: word frequency compared to baseline frequency before GPT, linguistic style change that correlates to LLM-simulated linguistic changes, constructing benchmark and RAG based on LLM-revised wikipedia to understand the impact. Overall they are rather comprehensive.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall claims are overall supported by presented methdology and evidence, and they are mostly accurate and convincing. \n\n- The 1st-degree impact is overall robust. \"approximately 1%-2% in certain categories\" are calculated across catagories and are adopted from previously established methodology from (Geng et al., 2025). \n- The linguistic style changes largely correlates to the LLM-based simulations that authors ran.\n- The page view presents mixed results, and there can be reasons beyond LLM that explain the drop. But this is not the main results of this paper. \n- The 2nd-degree impact such as that on MT benchmark and RAG seem exploratory, as the authors construct (rather toy) benchmars and RAGs based on LLM-rewritten content, giving readers a flavor what might happen in such set up. The resuts are mixed, and the methologyies seem more for demonstration rather than rigorious study. I regarded them as some early effort to investigate 2nd-order problems. And the authors present some relevant findings.", "audience_yn": "Yes", "audience_explanation": "Yes. I believe the work on LLM impact on human knowledge production and transmission require attention from ML community, as LLM, different from other more conventional ML technologies are widely adopted in many research, educational, and information-processing tasks. Such impact is poorly studied, and it requires long-term monitor, investigation, and even causal analysis.", "requested_changes": "There are some methodological problems to be noted, and changes might be needed:\n\n- (crucial) In the assessment of word frequency, the data was taken as comparison from 2020-2021 on the same articles. There are two probems here (i) 2020-2021 seems a very unsuaul year in human history because of covid-19. An example of anomaly:  and notably there are far more studies about covid-19 relevant topics from other years; (ii) the comparison here is between \"word frequency of Wikipedia *with* LLM-editing\" and \"word frequency of Wikipedia *without* LLM-editing\", but the authors didn't investigate whether word frequency would usually change, too. The question here should be \"whether the word frequency change under LLM is rarther unsusal compared to baseline change? This is not investigated by the authors. \n- As authors noted themselves \"Second, the lack of field experiments limits our insights into the actual machine-in-the-loop editing processes behind Wikipedia article creation.\" Indeed, ineractive dynamic should be far more complex than what they did with simulations. And understanding that interactive dynamic might be hard. But can authors estimate whether their results are at upper or lower bound of the impact? This is a comment on linguistic style subsection. \n- (crucial) The 2nd-order impact overall is unrealitic. If your own estimation is 1-2% impact (adimittedly, what does 1-2% mean is unclear. It's linguistic frequency difference, but I am not sure whether we can intuitively think that in average 1~2% of each article is re-written by LLM), why do you try benchmark/RAG based on 100% rewritten by LLMs? As a natural result, the impact is inflated. Wouldn't it be more convincing to try lighter LLM revision? As you argue, the LLM impact grows over time, and in some catagories the impact exceeds 1-2%. So how about trying LLM revision in the bar park of 1-2% but slightly higher? This becomes a straw-man argument.\n- Gemini 1.5 Flash was released early 2024 and it's a rather low-capability model. I won't be too suprised that RAG / MT benchmark based on it would experience accuracy drop. But what about newer models? If RAG based on Gemini 3-revised Wikepedia can achieve human performance, would we claim this is a lesser concern over time? Also, if GPT-4o-mini does not compromise RAG performance as well, would this be an artifact with Gemini 1.5 Flash?", "broader_impact_concerns": "No ethical concerns applied. This is an empirical analysis of LLM impact on Wikipedia.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b5t10BrdlM", "reviewer": "Reviewer_dwkh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the impact of LLMs on Wikipedia, and they measure direct and indirect impacts of LLMs in their work. Based on their experiments, the authors conclude that LLMs do have some impact on Wikipedia but it is overall not significant/uncertain. They also believe that some machine translation model scores could be inflated due to LLMs' influence on Wikipedia.\n\nSome key strengths of the paper include:\n- a principled experimental design and really nice problem to study\n- clear organization of the paper\n- inclusion of impacts for practical tasks like machine translation and RAG pipelines\n- i also really liked the case study at the end, and it was arguably the most clear and impactful section to me in the paper\n\nSome key weaknesses of the paper are:\n- although the existing experimental setup is technically sound for that they are doing, they do not truly capture what the authors originally set out to measure\n- the main text is also not self-complete. there are many key details missing from protocol descriptions that make it hard for a reader to follow along smoothly\n- the direct impact analysis section is arguably weak and for the strong claims made in that section, the experiments need to be studied more thoroughly as well", "claims_and_evidence_yn": "No", "claims_explanation": "- When the authors say \"crucial\" and \"additionally\" are favored by LLMs, it is missing a citation. Figure 2 shows an increase in their frequencies but is it enough to establish a direct causal link between that LLMs? Consequently, the Impact metric computed is also based on the assumption that crucial and additionally are favored by LLMs (which may be true but I don't see enough evidence of that being true). Instead of word frequencies, I think perplexity would be a much more reliable metric to measure here as we *know* LLMs are trained to achieve lower perplexity. Another thing to do could be to ask LLMs to rewrite pre-LLM articles and plot the before and after word frequency trends (you don't need to do it for all the pre-LLM articles but sufficiently big and diverse sample should be enough, and using open LLMs might further help reduce the compute cost).\n\n- For 4.1, I am not convinced that the simulation experiment of rewriting a sentence with an LLM truly captures the LLM impact as the instruction to revise does not include any incentive for the LLM to actually replace some words. It can high some biases for sure, but it is not representative of what the authors are trying to measure. A much more representative task would be ask it to write some text from scratch on the same topic based on the same reference content and then compare the two versions (maybe you already use this data for indirect impact analysis, but I am not sure if you restrict the LLM to use the same references as the original article there).\n\n- For the linguistic style analysis, it is good to see different linguistic criteria but it is not clear from the text which ones have been proven to be preferred by humans. For instance, I would guess LLMs prefer more active voice than passive voice but has this been shown or studied before? Same argument for the remaining criteria. If they have, please clearly state them in the main text.\n\n- I am also not sure we can establish a direct causal connection between higher performance on the GPT-processed benchmark vs the original benchmark and the fact that this suggests that LLMs can inflate translation scores. First, GPT-4o-mini $\\ne$ all LLMs, and it is a pretty weak model. Second, in addition to those quantitative numbers, we should try to understand the failure cases in original v/s the success cases in the GPT-processed benchmark (where do GPT translations differ and how correct are they). Because even if GPT constructs incorrect translations, I don't understand how that would strictly boost a translation model's score (I think it's just as likely to drop). I am really curious to understand this.", "audience_yn": "Yes", "audience_explanation": "The impact of LLMs of existing bodies of text like Wikipedia and to extend it to the whole internet is a meaningful research problem as it also directly guides many pretraining/posttraining decisions for frontier models down the line.", "requested_changes": "- Please consider presenting Tables 2 and 3 visually. It is really hard to tally through > 100 numbers in a table. A simple bar chart on the other hand would've instantly told me the story (just an example, there may a better way to represent the data than this). This is a minor change though and it won't impact my decision\n- I also find the models used out of date, and while I understand that you cannot possibly study the impact of all the LLMs, include some good models from the recent few months (and preferably open models) would really help generalize your findings. Looking at the analysis for a more recent model's data would help me make a better decision for the paper.\n- Please address the concerns I have raised regarding the representation of your experimental setup as it would strongly impact my decision for the submission\n- I have also compiled some other cosmetic comments that would help improve the delivery of the paper.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "- I don't quite understand what is $f_{i}^{*}(S)$ even after reading the text. I am also confused as to how can you determine if some text is affected by an LLM based on the formula. What is $\\eta^{\\hat}(S)$ in Eq 1 supposed to capture, intuitively speaking and how does the interaction between different terms capture that?\n- Why is there no $i$ anywhere in $f(S_2)$ or $f(S_1)$ in Eq 2?\n- I don't quite understand the LLM Simulation setting, we have GPT-4o-mini and Gemini-1.5-Flash, but it is not clear from the main text what are they exactly simulating or benchmarked on. I took a quick look at the appendix but there's a lot going on there. It would really help if you could clarify the simulation setting more clearly in the main text.\n- The results section is also a bit hard to follow. In the sentence \"Although we have plotted...from 2020 in **these** figures\" do they mean the figures in the main text or the appendix? Following that, in \"...the trends summarized in **the** table are based on...\" which table does the highlighted \"the\" refer to? Table 1 caption does not clarify this either.\n- The term \"page view\" is a little ambiguous in this context as it can mean the number of views for a given wikipedia page (the intended meaning) or it could mean the rendered view of a Wikipedia page (which is how it reads like at the beginning).\n- What does the \"Full Content\" setting mean in Figure 8? The input is the question+the wiki article that was used to construct those questions? Those setting names from the figure should be referenced in the main text.\n- Where do the chatGPT processed articles come into picture? It abruptly appears the questioning methods paragraph and the exact protocol on how it is was used is not clear.\n- Results of the case study are really good.\n- The authors have more than 1 page to make the main text more complete without changing the category of the submission, and I recommend utilizing it as I felt a lot of disconnections when reading the paper as some key details that would've helped me understand the paper better were absent.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)