Mapping the Increasing Use of LLMs in Scientific Papers
Authors: Weixin Liang
Yaohui Zhang
Zhengxuan Wu
Haley Lepp
Wenlong Ji
Xuandong Zhao
Hancheng Cao
Sheng Liu
Siyu He
Zhi Huang
Diyi Yang
Christopher Potts
Christopher D Manning
James Y. Zou
Venue: COLM (2024)
Keywords: Academic Writing, Computational Social Science, Science and Technology Studies, Societal Impact and LLM Adoption
TL;DR: We conducted the first systematic analysis to quantify the impact of Large Language Models (LLMs) on academic writing over time.

Abstract: Scientific publishing lays the foundation of science by disseminating research findings, fostering collaboration, encouraging reproducibility, and ensuring that scientific knowledge is accessible, verifiable, and built upon over time. Recently, there has been immense speculation about how many people are using large language models (LLMs) like ChatGPT in their academic writing, and to what extent this tool might have an effect on global scientific practices. However, we lack a precise measure of the proportion of academic writing substantially modified or produced by LLMs. To address this gap, we conduct the first systematic, large-scale analysis across 950,965 papers published between January 2020 and February 2024 on the $\textit{arXiv}$, $\textit{bioRxiv}$, and $\textit{Nature}$ portfolio journals, using a population-level statistical framework to measure the prevalence of LLM-modified content over time. The statistical framework operates on the population level without the need to perform inference on any individual instance. Our findings reveal a steady increase in LLM usage, with the largest and fastest growth observed in Computer Science papers (up to 17.5\%). In comparison, Mathematics papers and the Nature portfolio showed the least LLM modification (up to 6.3\%). Moreover, at an aggregate level, our analysis reveals that higher levels of LLM-modification are associated with papers whose first authors post preprints more frequently, papers in more crowded areas, and papers with shorter lengths. Our findings suggests that LLMs are being broadly used in scientific papers.

Published as a conference paper at COLM 2024
Mapping the Increasing Use of LLMs in Scientific Papers
Weixin Liang" Yaohui Zhang; Zhengxuan Wu; Haley Lepp, Stanford University
Wenlong Ji, Xuandong Zhao, Stanford University, UC Santa Barbara
Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, Diyi Yang, Stanford University
Christopher Potts, Christopher D Manning! James Y. Zou Stanford University
Abstract
Scientific publishing lays the foundation of science by disseminating re- search findings, fostering collaboration, encouraging reproducibility, and ensuring that scientific knowledge is accessible, verifiable, and built upon over time. Recently, there has been immense speculation about how many people using large language models (LLMs) like ChatGPT in their aca- demic writing, and to what extent this tool might have an effect on global scientific practices. However, we lack a precise measure of the proportion of academic writing substantially modified or produced by LLMs. To ad- dress this gap, conduct the first systematic, large-scale analysis across 950,965 papers published between January 2020 and February 2024 on the Xio, bioRxiv, and Nature portfolio journals, using a population-level statistical framework to measure the prevalence of L.M.mmodified content over time. Our statistical estimation operates on the corpus level and is more robust than inference on individual instances. Our findings reveal a steady increase in LLM usage, with the largest and fastest growth observed in Computer Science papers (up to 17.5%). In comparison, Mathematics papers and the Nature portfolio showed the least LLM modification (up to 6.3%). Moreover, at an aggregate level, our analysis reveals that higher levels of L.M--odification are associated with papers whose first authors post preprints more frequently, papers in more crowded research areas, and papers of shorter lengths. Our findings suggests that LLMs are being broadly used in scientific writings.
Introduction
Since the release of ChatGPT in late 2022, anecdotal examples of both published papers (Okunyté, 2023; Deguerin, 2024) and peer reviews (Oransky & Marcus, 2024) which appear to be ChatGPT-generatee have inspired humor and concern.' While certain tells, such as "regenerate response" " (Conroy, 2023b;a) and "as an AT language model" (Vincent, 2023), found in published papers indicate modified content, less obvious cases nearly impossi- ble to detect at the indívidual level (Else, 2023; Gao et al., 2022). Liang et al. (2024a) present
"Co-first authors, Correspondence to: Weixin Liang wwwliangdstanford.com
"Co-supervised project, Correspondence James 7ou <jameszsstanford.c edu>
IIcreesedd attention to ChatGPT-use by multilingual scholars has also brought to the fore important conversations about entrenched linguistic discrimination in academic publishing (Lepp & Sarin, 2024; Khanna al., 2022).
1

---

Published as a conference paper at COLM 2024
20.0%
Computer Science(arXiv)
17.5%
Electrical Engineering and Systems Science(arXiv) caarXiv))
15.0%
bioRady
Physics(arXiv)
Estimated Alpha
12.5%
Nature portfolio
Mathematics(a///)
ChatGPT Launch Nov 30, 2022
10.0%
7.5%
5.0%
2.5%
0.0%
2021.1
2021.4
2021.7 2021.10
2022.1 2022.4 2022.7
2022.10
2023.1
2023.4 2023.7 2023.10 2024.1
Figure 1: Estimated Fraction of LL Sentences across Academic Writing Venues over Time. This figure displays the fraction (a) of sentences estimated to have been sub- stantially modified by LLM in abstracts from various academic writing venues. The analysis includes five areas within ar Xiv (Computer Science, Electrical Engineering and Systems Science, Mathematics, Physics, Statistics), articles from bioRxiv, and a combined dataset from 15 journals within the Nature portfolio. Estimates are based on the distributional GPT quantification framework, which provides population-level estimates rather than individ- ual document analysis. Each point in time independently estimated, with no temporal smoothing or continuity assumptions applied. Error bars indicate 95% confidence intervals by bootstrap. Further analysis paper introductions is presented in Supp Figure 6.
method for detecting the percentage of LLM-modified text in a corpus beyond such obvious cases. Applied to scientific publishing, the importance of this at-scale approach is two-fold: first, rather than looking LLM-use as a type of rule-breaking on an individual level, we can begin to uncover structural circumstances which might motivate its use. Second, by examining L.LM-use academic publishing at-scale, we can capture epistemic and linguistic shifts, miniscule at the individual level, which become apparent with birds-eye view.
Measuring the extent of LLM-use on scientific publishing has urgent applications. Concerns about accuracy, plagiarism, anonymity, and ownership have prompted some prominent scientific institutions to take a stance on the use of LLM-modified content in academic publications. The International Conference
Machine Learning (ICML) 2023, major machine learning conference, has prohibited the inclusion of text generated by LLMs like ChatGPT in submitted papers, unless the generated text is used as part of the paper's experimental analysis (ICMI, 2023). Similarly, the journal Science has announced an update to their editorial policies, specifying that text, figures, images, graphics generated by ChatGPT or any other LLM tools cannot be used in published works (Thorp, 2023). Taking steps to measure the extent of LLM-use can offer a first-step in identifying risks to the scientific publishing ecosystem. Furthermore, exploring the circumstances in which LLM- use is high can offer publishers and academic institutions useful insight into author behavior. Sites of high LLM-use can act as indicators for structural challenges faced by scholars. These range from pressures to "publish or perish" which encourage rapid production papers to concerns about linguistic discrimination that might lead authors to use LLMs as prose editors.
We conduct the first systematic, large-scale analysis to quantify the prevalence of LLM- modified content across multiple academic platforms, extending a recently proposed, state- of-the-art distributional GPT quantification framework (Liang et al., 2024a) for estimating the fraction of Al-modified content in a corpus. Throughout this paper, we use the term to refer to text content substantially updated by ChatGPT beyond basic
2

---

Published as a conference paper at COLM 2024
Frequency per one million words
pivotal
Intricate
100.0
realm
showcasing
ChatGPT Launch
Nov 30, 2022
50.0
0.0
2011
2012
2013
2014
2015
2016
2017
2018
2019
2021
2022
2023
2020
160.0
pivotal
140.0
intricate
Froquency per one million words
showcasing realm
120.0
100.0
ChatGPT Launch
80.0
Nov 30, 2022
60.0
20.0
0.0
2021.1
2021.7 2021.10
2022 2022.7
2023.1 2023.4 2023.7 2023 2024.1
Figure 2: Word Frequency Shift in arXiv Computer Science abstracts over 14 years (2010- 2024). The plot shows the frequency over time for the top 4 words most disproportionately used by LLM compared to humans, measured by the log odds ratio. The words are: realm, intricate, showeasingg pivotal. These terms maintained a consistently low frequency in arXiv CS abstracts over more than decade (2010-2022) but experienced sudden surge in usage starting 2023.
spelling and grammatical edits. Modifications we capture in our analysis could include, for example, summaries of existing writing or the generation of prose based on outlines.
key characteristic of this framework that it operates on the population level, without the need to perform inference any individual instance. As validated in the prior paper, the framework is orders of magnitude more computationally efficient and thus scalable, pro- duces more accurate estimates, and generalizes better than its counterparts under significant temporal distribution shifts and other realistic distribution shifts.
We apply this framework to the abstracts and introductions (Figure and Supp Figure 6) of academic papers across multiple academic disciplines,inclun journals within the Nature portfolio, such as Nature, Nature Biomedical Engineering, Nature Human Behaviour, and Nature Communications. Our study analyzes a total of 950,965 papers published between January 2020 and February 2024, comprising 773,147 papers from arxiv, 161,280 from bioRxiv, and 16,538 from the Nature portfolio journals. The papers from ar Xiv cover multiple academic fields, including Computer Science, Electrical Engineering and Systems Science, Mathematics, Physics, and Statistics. These datasets allow to quantify the prevalence of LLM-modified academic writing over time and across broad range of academic fields.
Xiv, bioRxiv, and 15
Our results indicate that the largest and fastest growth was observed in Computer Science papers, with a reaching 17.5% for abstracts and 15.3% for introductions by February 2024.

---

Published as a conference paper at COLM 2024
In contrast, Mathematics papers and the Nature portfolio showed the least increase, with a reaching 4.9% and 6.3% for abstracts and 3,5% and 6.4% for introductions, respectively. Moreover, our analysis reveals at aggregate level that higher levels of LLM-modification are associated with papers whose first authors post preprints more frequently and papers with shorter lengths. Results also demonstrate a closer relationship between papers with LLM-modifications, which could indicate higher use in more crowded fields of study (as measured by the distance to the nearest neighboring paper in the embedding space), or that generated-text is flattening writing diversity.
Related Work
GPT Detectors Various methods have been proposed for detecting L.M- text, including zero-shot approaches that rely on statistical signatures characteristic of machine- generated content (Lavergne et al., 2008; Badaskar et al., 2008; Beresneva, 2016; Solaiman et al., 2019; Mitchell et al., 2023; Yang et al., 2023a; Bao et al., 2023; Tulchinskii et al., 2023) and training-based methods that finetune language models for binary classification of human vs. LLM-modified text (Bhagat & Hovy, 2013; Zellers et al., 2019; Bakhtin et al., 2019; Uchendu et al., 2020; Chen et al., 2023; Yu et al., 2023; Li et al., 2023; Liu et al., 2023; Bhattacharjee et al., 2023; Hu et al., 2023a). However, these approaches face challenges such as the need for access to LLM internals, overfitting to training data and language models, vulnerability
adversarial attacks (Wolff, 2020), and bias against non-dominant language varieties (Liang et al., 2023). The effectiveness and reliability of publicly available LLM-modified text detectors have also been questioned (OpenAI, 2019; Jawahar et al., 2020; Fagni et al., 2021; Ippolito et al., 2023; Mitchell et al., 2023; Gehrmann et al., 2019; Heikkilä, 2022; Crothers et al., 2022; Solaiman et al., 2019; Kirchner et al., 2023; Kelly, 2023), with the theoretical possibility of accurate instance-level detection being debated (Weber-Wur Wulff et al., 2023; Sadasivan et al., 2023; Chakraborty et al., 2023). this study, we apply the recently proposed distributional GPT quantification framework (Liang et al., 2024a), which estimates the fraction of LLM-modified content in text corpus at the population level, circumventing the need for classifying individual documents or sentences and improving upon the stability, accuracy, and computational efficiency of existing approaches. A more comprehensive discussion of related work can be found in Appendix H.
3 Background: the distributional LLM quantification framework
We adapt the distributional LLM quantification framework from Liang et al. (2024a) to quantify the use of Al-modified academic writing. The framework consists of the following steps:
1. Problem formulation: Let P and Q be the probability distributions of human-written and LLM-modified documents, respectively. The mixture distribution is given by (x) = (1 - 1a)a)PP ax + a ((x), where a is the fraction of Al-modified documents. The goal is to estimate a based on observed documents [ Da-
2. Parameterizatio: To make a identifiable, the framework models the distributions of token occurrences in human-written and LLM-modified documents, denoted as Py and M
QT, respectively, for chosen list of tokens = { The occurrence probabilities of each token in human-written and LLM-modified documents, pt and qt, are used to parameterize P and Q7:
Pr(X)
I (teX) (1)x))) (x(X) FET
P
II
(tex) (1-)) 9t FET
3. Estimation: The occurrence probabilities pt and 9; estimated using collections of "e
known human-written and LLM-modified documents, (xp) ) and {x, J==1/ respec- tively:
)
1 Hp Pt Up
1 n0 1lt E x().
Σ 1(t X,), 4-= no Σ
4

---

Published as a conference paper at COLM 2024
4. Inference: The fraction is estimated by maximizing the log-likelihood of the observed documents under the mixture distribution xx,,(X) (1 (1- (x)
MLE 47
argmax
Σ log - ()-a))r(X))) ( (x,)) ae(0.1]
Liang et al. (2024a) demonstrate that the data points {X Dx can be constructed either as a document or as a sentence, and both work well. Following their method, we use sentences the unit of data points for the estimates for the main results. In addition, we extend this framework for our application to academic papers with two key differences:
We use two-stage approach to generate LLM-produced text, as simply prompting an LLM with paper titles or keywords would result in unrealistic scientific writing samples containing fabricated results, evidence, and ungrounded claims.
Generating Realistic LLM-Produced Training Data
Specifically, given a paragraph from a paper known to not include LLM-modification, we first perform abstractive summarization using an LLM to extract key contents in the form of an outline. We then prompt the LLM to generate full paragraph based the outline (see Appendix for full prompts).
Our two-stage approach can be considered a counterfactual framework for generating LLM text: given a paragraph written entirely by human, how would the text read if it conveyed almost the same content but was generated by an L.M? This additional abstractive summarization step can be seen as the control for the content. This approach also simulates how scientists may be using LLMs in the writing process, where the scientists first write the outline themselves and then use LLMs to generate the full paragraph based on the outline (Lee et al., 2024).
Using the Full Vocabulary for Estimation We use the full vocabulary instead of only adjectives,
validation shows that adjectives, adverbs, and verbs all perform well in our application (Supp Figures and 8). Using the full vocabulary minimizes design biases stemming from vocabulary selection. We also find that using the full vocabulary is more sample-efficient in producing stable estimates, as indicated by their smaller confidence intervals by bootstrap.
4
Implementation and Validations
4.1 Data Collection and Sampling
We collect data from three sources: arXiv, bioRxiv, and 15 journals from the Nature portfolio. For each source, randomly sample up to 2,000 papers per month from January 2020 to February 2024. The procedure for generating the LLM-generated corpus data is described in Section 3. We focused on the introduction sections for the main texts, as the introduction was the most consistently and commonly occurring section across diverse categories of papers. See Appendix D for comprehensive implementation details.
4.2 Data Split, Model Fitting, and Evaluation
For model training, we count word frequencies for scientific papers written before the release of ChatGPT and the LLM-modified corpora described in Section 3. We fit the model with data from 2020, and use data from January 2021 onwards for validation and inference. We fit separate models for abstracts and introductions for each major category.
To evaluate model accuracy and calibration under temporal distribution shift, we use 3,000 papers from January 1, 2022, to November 29, 2022, time period prior to the release of ChatGPT, as the validation data. We construct validation sets with LLM-modified content proportions (a) ranging from 0% to 25%, in 5% increments, and compared the model's estimated a with the ground truth Additionally, full vocabulary, adjectives, adverbs, and verbs all performed well in our application, with a prediction error consistently less than 3.5% at the population level across various ground truth a values (Supp Figures 7 and 8).
5

---

Published as a conference paper at COLM 2024
5 Main Results and Findings
5.1 Temporal Trends in Al-Modified Academic Writing
Setup We apply the model to estimate the fraction of L.M-mooiiiied content (a) for each paper category each month, for both abstracts and introductions. Each point in time was independently estimated, with no temporal smoothing or continuity assumptions applied.
Results Our findings reveal steady increase in the fraction of AT-modified content (a) in both the abstracts (Figure 1) and the introductions (Supp Figure 6), with the largest and fastest growth observed in Computer Science papers. By February 2024, the estimated for Computer Science had increased to 17.5% for abstracts and 15.5% for introductions. The second-fastest growth was observed in Electrical Engineering and Systems Science, with the estimated reaching 14.4% for abstracts and 12.4% for introductions during the same period. In contrast, Mathematics papers and the Nature portfolio showed the least increase. By the end of the studied period, the estimated for Mathematics had increased to 4.9% for abstracts and 3.9% for introductions, while the estimated a for the Nature portfolio had reached 6.3% for abstracts and 4.3% for introductions. We analyzed Nature portfolio journal papers using both submission and publication dates. The results were consistent, with Nature portfolio papers having among the lowest estimated alphas, even when plotted by submission date.
The November 2022 estimates serve pre-ChatGPT reference point for comparison, as ChatGPT was launched on November 30, 2022. The estimated a for Computer Science in November 2022 was 2.3%, while for Electrical Engineering and Systems Science, Mathemat- ics, and the Nature portfolio, the estimates were 2.9%, 2.4%, and 3.1%, respectively. These values are consistent with the false positive rate reported in the earlier section (Section 4.2).
5.2 Relationship Between First-Author Preprint Posting Frequency and GPT Usage in Computer Science
We found a notable correlation between the number of preprints posted by the first author arv and the estimated number of LLM-modifiee sentences in their academic writing across multiple fields in Computer Science. Papers were stratified into two groups based on the number of first-authored ar Xiv Computer Science preprints by the first author in the year: those with two or fewer (< 2) preprints and those with three or more (2 3) preprints (Figure 3). We used the 2023 author grouping for the 2024.1-2 data, as we don't have the complete 2024 author data yet.
By February 2024, abstracts of papers whose first authors had 2 preprints in 2023 showed an estimated 19.3% of sentences modified by AI, compared to 15.6% for papers whose first authors had s preprints (Figure 3a). We observe similar trend in the introduction sections, with first authors posting more preprints having an estimated 16.9% LLM-modified sentences, compared 13.7% for first authors posting fewer preprints (Figure 3b). Since the first-author preprint posting frequency may be confounded by research field, we conduct additional robustness check for our findings. We find that the observed trend holds for each of the three Xiv Computer Science sub-categories: Cs.CV (Computer Vision and Pattern Recognition), cs.LG (Machine Learning), and CL (Computation and Language) (Supp Figure 13).
Our results suggest that researchers in Computer Science posting more preprints tend to utilize LLMs more extensively in their writing. One interpretation of this effect could be that the increasingly competitive and fast-paced nature of Computer Science research communities incentivizes taking steps to accelerate the writing process. We do not evaluate whether these preprints were accepted for publication.
5.3 Relationship Between Paper Similarity and LLM Usage
We investigate the relationship between a paper's 's similarity to its closest peer and the estimated LLM usage in the abstract. To measure similarity, first embed each abstract

---

Published as a conference paper at COLM 2024
20%
Papers by first authors with 2 preprints the year Papers by first authors with 3 preprints the year
Estimated Alpha
Abstract
15%
10%
5%
0%
2022.1-3
2022.4-6
2022.7-9 2022.10-11 2023.1- 1-3 2023,4-6 2023.7-9 2023.10-1
10-2.10-12 2024.1-2
20%
Papers by first authors preprints the year Papers by first preprints year
Introduction
Estimated Alpha
15%
10%
5%
0%
2022.1-3
1-3 2022.4-6 2022.7-9 2022.10-11 2023.1-3 2023.4-6 2023.7-9 2023.10-13 2024.1- 1-2
Figure 3: Papers authored by first authors who post preprints more frequently tend to have higher fraction of LLM-modified content in Computer Science. Papers in ar aiv Computer Science are stratified into two groups based on the preprint posting frequency of their first author, as measured by the number of first-authored preprints in the year. Error bars indicate 95% confidence intervals by bootstrap.
from the arXiv Computer Science papers using Open textemmbedding-dda-00 model, creating vector representation for each abstract. We then calculate the distance between each paper's 's vector and its nearest neighbor within the Xiv Computer Science abstracts. Based on this similarity measure we divide papers into two groups: those more similar to their closest peer (below median distance) and those less similar (above median distance).
The temporal trends of LLM usage for these two groups are shown in Figure After the release of ChatGPT, papers most similar to their closest peer consistently showed higher LLM usage compared to those least similar. By February 2024, the abstracts of papers more similar to their closest peer had an estimated 22.2% of sentences modified by LLMs, compared to 14.7% for papers less similar to their closest peer. To account for potential confounding effects of research fields, we conducted an additional robustness check by measuring the nearest neighbor distance within each of the three ar Xiv Computer Science sub-categories: cs.CV (Computer Vision and Pattern Recognition), cs.LG (Machine Learning), and cs.CL (Computation and Language), and found that the observed trend holds for each sub-category (Supp Figure 14).
There are several ways to interpret these findings. First, LLM-use in writing could cause the similarity in writing or content. Community pressures may even motivate scholars to try to sound more similar to assimilate to the "style" of text generated by an LLM. Alternatively, LLMs may be more commonly used in research areas where papers tend to be more similar to each other. This could be due to the competitive nature of these crowded subfields, which may pressure researchers to write faster and produce similar findings. Future interdisciplinary research should explore these hypotheses.
5.4 Relationship Between Paper Length and Al Usage
We also explored the association between paper length and LLM usage in arXiv Computer Science papers. Papers were stratified by their full text word count, including appendices, into two bins: below or above 5,000 words (the rounded median).
Figure 5 shows the temporal trends of LLM usage for these two groups. After the release of ChatGPT, shorter papers consistently showed higher Al usage compared to longer papers. By February 2024, the abstracts shorter papers had an estimated 17.7% of sentences modified by LLMs, compared to 13.6% for longer papers (Figure 5a). A similar trend was observed in the introduction sections (Figure 5b). To account for potential confounding effects of research fields, we conducted an additional robustness check. The finding holds
7

---

Published as a conference paper at COLM 2024
Papers less similar to their closest peer (above modian distance)
Estimated Alpha
20%
Papers more similar to their closest peer (below median distance)
Abstract
15%
10%
5%
0%
2022.1-3
2022.4-6
2022.7-9 2022.10-11 2023.1-3 1-3 2023,4-6 2023.7-9 2023.10-12 2024.1-2
20%
Papers
peer (above median distance) peer (below distance)
Introduction
Estimated Alpha
Papers
5%
0%
2022.1-3
2022.4-6 2022.7-9 2022.10-11 2023.1-3 2023.4-6 2023.7-9 2023.10-12 2024.1-2
Figure 4: Papers in more crowded research areas tend to have a higher fraction of LLM- modified content. Papers in ar Xiv Computer Science are divided into two groups based on their abstract's embedding distance to their closest peer: papers more similar to their closest peer (below median distance) and papers less similar to their closest peer (above median distance). Error bars indicate 95% confidence intervals by bootstrap.
for both cs.CV (Computer Vision and Pattern Recognition) and cs.LG (Machine Learning) (Supp Figure 15). However, for cs.CL (Computation and Language), we found no significant difference in LLM usage between shorter and longer papers, possibly due to the limited sample size, as we only parsed subset of the PDPs and calculated their full length.
As Computer Science conference papers typically have a fixed page limit, longer papers likely have more substantial content in the appendix. The lower LLM usage in these papers may suggest that researchers with more comprehensive work rely less on LLM-assistance in their writing. However, further investigation is needed to determine the relationship between paper length, content comprehensiveness, and the quality of the research.
20%
Shorter papers worde)
Estimated Alpha
Longer papers
Introduction Abstract
15%
10%
5%
0%
2022.1-3
2022.4-6
2022.7-9 2022.10-11
2023.1-3
2023 4-6
2023.7
7-9
2023.10-12
2024.1-2
20%
Shorter papers 5,000 words)
Estimated A pha
Longer pepers 5,000 words)
15%
10%
5%
0%
2022.1-3
2022.4-6 2022.7-9 2022.10-11 2023.1-3 2023.4-6 2023.7-9 9 2023.10-1 2024.1-2
Figure 5: Shorter papers tend to have higher fraction of LLM-modified content. ar Xiv Computer Science papers are stratified by their full text word count, including appendices, into two bins: below or above 5,000 words (the rounded median). Error bars indicate 95% confidence intervals by bootstrap.
8

---

Published as a conference paper at COLM 2024
6 Discussion
Our findings show a sharp increase in the estimated fraction of LLM-modified content in academic writing beginning about five months after the release of ChatGPT, with the fastest growth observed in Computer Science papers. This trend may be partially explained by Computer Science researchers' familiarity with and access to LLMs. Additionally, the fast-paced nature of LLM research and the associated pressure to publish quickly may incentivize the use of LLM writing assistance (Foster al., 2015).
We expose several other factors associated with higher LLM usage in academic writing. First, authors who post preprints more frequently show a higher fraction of LLM-modified content in their writing. Second, papers in more crowded research areas, where papers tend to be more similar, showed higher LLM-modification compared to those in less crowded areas. Third, shorter papers consistently showed higher LLM-modification compared to longer papers, which may indicate that researchers working under time constraints are more likely to rely on AT for writing assistance. While these results suggest a potential link between competitive research environments and the pressure to publish quickly, further studies are needed to empirically validate this hypothesis.
In a research environment which values English-language publishing, scholars who are not heritage speakers of English may find it productive to have an Al model polish their writing (Lee et al., 2024). Additionally, LLMs offer the possibility of immediate feed- back on initial drafts compared consuming (Liang et al., 2024b). However, if the majority of modification comes from an LLM owned by private company, there could be risks to the security and independence of scientific practice. We hope our results inspire further studies of widespread LLM-modified text and conversations about how to promote transparent, epistemically diverse, accurate, and independent scientific publishing.
traditional peer review processes, which can be time-
Limitations While our study focused on ChatGPT, which accounts for more than three- quarters of worldwide internet traffic in the category (Van Rossum, 2024), we acknowledge that there are other large language models used for assisting academic writing. One potential confounder of our study the increased prevalence of research on LLMs after the launch of ChatGPT. This shift in research focus could potentially affect the accuracy of our method in detecting --oodiiiiied content. However, our validation has shown that our framework is robust under temporal distribution shifts of research topics. Still, future studies could further validate and analyze the robustness of our method with more systematic control the study content. Furthermore, while Liang et al. (2023) demonstrate that GPT-detection methods can falsely identify the writing of language learners as LLM-generated, our results showed that consistently low false positives estimates of in 2022, which contains significant fraction of texts written by multilingual scholars. We recognize that significant author population changes (MacroPolo, 2024) or other language-use shifts could still impact the accuracy of our estimates. Finally, the associations that we observe between LLM usage and paper characteristics are correlations which could be affected by other factors such as research topics. Investigation of the causal factors shaping this usage is an important direction for future research.

---

Published as a conference paper at COLM 2024
References
Scott Aaronson. Simons
Institute
Watermarking of Large URL https:/ //simos///mos///ec www.oo.eeeeeeleee.......e.....ee edu/talks/
Talk
Language Models, 2023.
ccott-aronso-uu--asiii------
Mikhail Atallah, Victor Raskin, Michael Crogan, Christian F. Hempelmann, Florian Ker- schbaum, Dina Mohamed, and Sanket Naik. Natural language watermarking: Design, analysis, and a proof-of-concept implementation. https://doi org/10. 1007/3- 540 10000--54969-90
In Information Hiding, 2001. URL
Sameer Badaskar, Sachin Agarwal, and Shilpa Arora. Identifying real or fake articles: Towards better language modeling. In International Joint Conference on Natural Language Processing (IJCNLP), 2008. URL https://aclanthologg/ org/I08-2215/
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc' Aurelio Ranzato, and Arthur Szlam. Real fake? Learning to discriminate machine from human generated text. Xiv preprint ar Xre:1996.003551 2019. URL https: http://awivv..oraa...a////
/900// 03351.
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang, Fast-DetectGPT: Efficient zero-shot detection machine-generated text via conditional probability curva- ture. ar Xiv preprint ar (rXii:23300051333 2023. URL https://aarke org/ abs 20t../033.1..
Daria Beresneva. Computer-generated text detection using machine learning: A systematic review. In International Conference on Applications of Natural Language Data Bases, 2016.
URL httht:://ooo... org/ / 1087/9978-3-31994177-----
Rahul Bhagat and Eduard H. Hovy. Squibs: What is a paraphrase? Computational Linguistics, 2013. URL httpttt://acaaathhlloooo wrt.:/yyy.oo//11/011110
Amrita Bhattacharjee, Tharindu Kumarage, Raha Moraffah, and Huan Liu. ConDA: Con- trastive domain adaptation for Al-generated text detection. In International Joint Conference on Natural Language Processing (IJCNLP), 2023. URL https://aclantholooe org/ 0023.
ijcnlp-main.48
Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. On the possibilities of Al-generated text detection. arXiv preprint arXiv:2304.04736, 2023. URL https://arxiv wtt.org/g//b.a.a/oee////// 04736.
Yutian Chen, Hao Kang, Vivian Zhai, Liang Li, Rita Singh, and Bhiksha Ramakrishnan. GPT-Sentinel: Distinguishing human and ChatGPT generated content. arviv preprint arXiv:2305.07969 2023. URL https://arxii: wwg. 2b30/ 07969.
Yuei-Lin Chiang, Lu-P'ng Chang, Wen-Tai Hsieh, and Wen-Chih Chen. Natural language watermarking using semantic substitution for chinese text. In International Workshop on Digital Watermarking, 2003. URL https://doi. wrw/o0..//./.// 1007 9978-9783333 3- 1-778-3-54---- 544--44244--111
Gemma Conroy. How ChatGPT and other AI tools could disrupt scientific publishing. pubiisihi Nature, October 2023a. URL https: wwwpwwww....ee..mmm htt.:/uure..oomaaaceee//////////
44-ww..
Gemma Conroy. sleuts spot CaachhonnsstthhCCh use in papers. Nature, Septemer 2023b. URL http:////www.ntuttu..///////////
Evan Crothers, Nathalie Japkowicz, and Herna Viktor. Machine generated text: A compre- hensive survey of threat models and detection methods. arXiv preprint arXiv:2210.07321,
2022. URL https: //axxivaaivv////a// www..or/g///////// 22220.0311.. 07321.
Mack Deguerin. Popular Science,
Al-generated nonsense is leaking into scientific journals. March 2024.
URL
wtt com/technoooggy.
ai
10

---

Published as a conference paper at COLM 2024
Fabrizio Dell' Acqua, Edward McFowland, Ethan R Mollick, Hila Lifshitz-Assaf, Katherine Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, and Karim R Lakhani. Nav- igating the jagged technological frontier: Field experimental evidence of the effects of AT on knowledge worker productivity and quality. Harvard Business School Technology & Operations Management, 2023. URL https: /www hbs wwt.eeeerr//uuaaaaattiiieeeeeeee
lles/
24-013 1011..d994068--9900 0e1-4445668997----000 42d6- a1 (ww.c6c767777782222. pdf.
Yanai Elazar, Akshita Bhagia, Ian Helgi Magmusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. What's in my big data? In International Conference on Learning Representations (ICLR), 2023. URL https://arxxt org/ wtgg/bs////eeoooooo /ttb////1... 20707.
Holly Else. Abstracts written by ChatGPT fool scientists. Nature, Jan 2023. URL https: www.:nattrrrcoooo wtt.tttrr.ccm/arrtceeee///aaae-0
Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi. TweepFake: About detecting deepfake tweets. Plos one, 16(5)xe0251415, 2021. URL
https://arxiv org/ 2200/ 00036.
Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. Three bricks to consolidate watermarks for large language models. 2023 IEÉE International Workshopp Information Forensics and Security (WIFS), 2023. URL pettps://arxle org/ / 2308. 2308.00113. 00113.
Jacob G Foster, Andrey Rzhetsky, and James A Evans. Tradition and innovation in scientists' research strategies. American sociological review, 80(5):875-908, 2015. URL https://arxiv. org/abs/1302. 6996.
Catherine A Gao, Frederick M Howard, Nikolay S Markov, Emma C Dyer, Siddhi Ramesh, Yuan Luo, and Alexander T Pearson. Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagia- rism detector, and blinded human reviewers. bioRxiv 2022.12.23.521610, 2022. URL
https://doi. org/10. 110111112222 /000222222.1111 12.....261...
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint xrii:2201.000027 2020. URL https: //arxiv.org/abs//2e0 000
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. GLTR: Statistical detection and visualization of generated text. In Association for Computational Linguistics (ACL): System Demonstrations, 2019. URL https://arxivv org/abs/ / 1906. 04044.
Soumya Suvra Ghosal, Souradip Chakraborty, Jonas Geiping, Furong Huang, Dinesh Manocha, and A. S. Bedi. Towards possibilities & impossibilities of Al-generated text detection: A survey. arxiv preprint arXiv:2310.15264, 2023. URL httph: //axxiv / 2320.15264.
Sourojit Ghosh and Aylin Caliskan. 'Person'== Light-skinned, western man, and sexualiza- tion of women of color: Stereotypes in stable diffusion. arXiv preprint arXiv:2310.19981, 2023. URL https: //axxiv http://aaxxvv.g.a/aaa/eae arg/abs/2310) 19981.
Melissa Heikkilä. How to spot Al-generated text. MIT Technology Review, URL https: //ww:.//wwwww. http:://www..eeoooooggrreew///// http://wwww...e///////////116630
Dec 2022. tow-to-spo-aae wewa--eeeeeeeeeeeeeee.
Xiaobing Hu, Pin-Yu Chen, and Tsung-Yi Yi Ho. RADAR: Robust Al-text detection via ad-
versarial learning. arXio preprint arXiv:2307.03833 2023a. URL https://arxxw.org/aaat 2307.
Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, and Heng Huang. Unbiased watermark for large language models. arv preprint ar XrX::22100.1066, 2023b. URL https::/anxiv.org/aas/////////// 10669.
11

---

Published as a conference paper at COLM 2024
ICML.
Clarification on large language 2023/ /llm-ooicy,, 2023.
model
policy
LLM.
https:/
Daphne Ippolito, Daniel Duckworth, Chris Callison- Burch, and Douglas Eck. Automatic de- tection of generated text is easiest when humans are fooled. ar Xiv preprint arXiv:1911.00650, 2023. URL https: www.a.orgaaaaaaa..../////p/a..ae 1911 10655.
Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. Automatic detection of machine generated text: A critical survey. arXiv preprint ar arXio:2011.01314 2020. URL https:///axiii http://axwvv..rggbbbb/s// wtg.obr///////a.a///
Samantha Murphy Kelly. ChatGPT creator pulls AI detection tool due to 'low rate of accuracy'. CNN Business, Jul 2023. URL https://wwwy .nw. 2tmm/2223307/25/teee html.
Saurabh Khanna, Jon Ball, Juan Pablo Alperin, and John Willinsky. Recalibrating the scope of scholarly publishing: A modest step in vast decolonization process. Quantitative Science Studies, 3(4):912-930, 12 2022. ISSN 2641-3337. doi: 10.1162/qss a 00228. URL
https://doi. orgooog//////
/arrg/10..0 10. 10....622/00 1162/qss a 00228.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, lan Miers, and Tom Goldstein. A watermark for large language models. In International Conference on Machine Learning (ICML), 2023, URL
Jan Hendrik Kirchner, Lama Ahmad, Scott Aaronson, and Jan Leike. New Al classifier for indicating Al-written text, 2023, URL https://openal. wttpenaai.cmmoollooomee com/blog/
cow-ai-classiiier-for
ai-written-text.
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-free watermarks for language models. arXiv preprint Xix::2330 Xit5:2307.15593,, 2023.
URL htttps://arxxv.org/abbsss 1551/
Thomas Lavergne, Tanguy Urvoy, and François Yvon. Detecting fake content with relative entropy scoring. Pan, 2008 URL https: httpps:/ccerrrws.org//ol--
horg/Vol-377 3012-/33pppppr4 /apper....... pdf.
Mina Lee, Katy llonka Gero, John Joon Young Chung, Simon Buckingham Shum, Vipul Raheja, Hua Shen, Subhashini Venugopalan, Thiemo Wambsganss, David Zhou, Emad A Alghamdi, et al. A design space for intelligent and interactive writing assistants. In The CHI Conference on Human Factors in Computing Systems (CHI), 2024. URL https:
//arxiv. www.vorg//aaoo
Haley Lepp and Parth Sarin. A global AI community requires language-diverse pub- lishing. In International Conference on Learning Representations (ICLR), 2024. URL https://globalat cul tures. gi thub. wttp//////////////////////////t/ hatpaa11aagglabaaa waw.lai.coooo wwt.i.ccoiitttt...mmmmm btriiittyrreee requireuuisss.eeee pdf.
Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. Deepfake text detection in the wild. arXiv preprint Xio:22305.13222, 2023. URL
https://arxiv htrg//bs/23300 13242.
Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Y. Zou. GPT detectors are biased against non-native English writers. arXiv preprint urXiv:2304.02819, 2023. URL https://arxiv. wwwpaaawwwi.oog/b//////../. 02819.
Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, and James Zou. Monitoring Al-modified content at scale: A case study the impact of ChatGPT Al conference peer reviews. In International Conference on Machine Learning (ICML), 2024a.
URL https::/aaxtti...... org htt.:/.ogggabb////./ee 2403. 2403.07183.
Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A. McFarland, and James Zou. Can large language models provide useful feedback on research papers? a large- scale empirical analysis. NEJM AI, 2024b. URL https: /aai nejm. org/doi/abs/10. 1056/ AIoa2400196.
12

---

Published as a conference paper at COLM 2024
Ryan Liu and Nihar B Shah. ReviewerGPT? An exploratory study using large language models for paper reviewing. arXio preprint arXio:2306.00622, 2023. URL https://arxiv. arxiv. org/abs/ 20020222260000..... 00622.
Xiaoming Liu, Zhaohan Zhang, Yichen Wang, Yu Lan, and Chao Shen. CoCo: Coherence- enhanced machine-generated text detection under data limitation with contrastive learn- ing. arXiv preprint drXi::2212.10341, 2023. URL www.tttps///aacaee httpp/////aaiiiooo//////////////
2rt. 10341.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. ar Xiv preprint ar XrXie:1907.1192, 2019. URL https://arxiv.
org/abs/1907. 11692.
MacroPolo. The global AI talent tracker, 2024.
URL https://macropollo org/
digital-proojcts//hee tattppooooccstttte-llllllll--aia gllbal-ai--aalentt-eeee-e----ee-
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. DetectGPT: Zero-shot machine- text detection using probability curvature. In International Conference Machine Learning (ICML), 2023. URL httph:t/ttrxiiii .ww.looorgggoooo orgg/b//////////////////oo/ 2301.11305 11305.
Paulina Okunyté. Google search exposes academics using ChatGPT in research papers. Cy- bernews, 2023. URL https: http///ybbbeeee/eeeeeeeeeeee//se ottpa////oooeeecoooooooooooooooo
OpenAI. GPT-2: 1.5b release. http:://ooooaa.ccc////eeeeerrrra 2019
2019.
Accessed: 2019-11-05.
Ivan Oransky and Adam Marcus. Papers and peer reviews with evidence of ChatGPT writing. Retraction Watch, 2024. URL https://eeraatiionatc.ccm
papers-and-peers ofw.e//wwwwwwttteeeeeeeeeeeeo-oo
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. In The Journal of Machine Learning Research (JMLR), 2020. URL https: wwtp://////bb////////g http//bs//111/ 10683.
Vinu Sankar Sadasivan, Aounon Kumar, S. Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can Al-generated text be reliably detected? arXiv preprint ar arXo:22303.11156 2023. WtKLhttp:://arkkt org/ abs/ 2303. 156.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning (ICML), 2023. URL https: / http://////axvvoor/aa/////// wwtp://..wr//bb///////// 2303. 27755
Zhouxing Shi, Yihan Wang, Fan Yin, Xiangning Chen, Kai-Wei Chang, and Cho-Jui Hsieh. Red teamiing language model detectors with language models. arv preprint arXiv:2305.197133 2023. URL https://arxiv
19711.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. arXiv preprint Xxiv:2305.17493 2023. URL https: http://aaxvvvv.g.aaaaa 2305. 17493.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. ar Xrv preprint ar wxxi::908.002000 2019.
H. Holden Thorp. ChatGPT is fun, but not an author. Science, 379(6630) science. sileence.org/doi//b//le...
2023. doi: 10.1126/ wt.1126//scencee sccience.ddg7777 URL https: htttp::/www.scceencee science. adg7879.
Mercan Topkara, Giuseppe Riccardi, Dilek Z. Hakkani-Tür, and Mikhail Atallah. Natural language watermarking: challenges in building a practical system. In Electronic Imaging, 2006a. URL https://disi wttpssii.iiiiiiiiiiiiiiii itt.iiiaiiiiiciiii..........
pdf.
13

---

Published as a conference paper at COLM 2024
Umut Topkara, Mercan Topkara, and Mikhail Atallah. The hiding virtues of ambiguity: quantifiably resilient watermarking of natural language text through synonym substi- tutions. In Workshop on Multimedia & Security, 2006b. URL https::/ddoi.org/ee... 1161366. 1161397.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar arXio:2302.13971 13971.
2023. URL https: penttss://ariii httpx/iw.org//b///////o/
Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, S. Baran- nikov, Irina Piontkovskaya, Sergey I. Nikolenko, and Evgeny Burnaev. Intrinsic dimension estimation for robust detection of Al-generated texts. ar preprint ar Xix:23306.004733 2023. URL https: //axxiv wwwgg///////aaeeee 04723.
Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. Authorship attribution for neural text generation. In Empirical Methods in Natural Language Processing (EMNLP), 2020. URL https://aclanthologg org/2020.00.. emnlp-main palp-mmain..m 673/.
Dann. Van Rossum. Generative Al top 150: The world's most used Al tools. https: //www. www.fllexooooommo wwwk///earn/geeerattiiii ai 150, February 2024.
Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks.
Xiv preprint ar drXv:23306.07899 2023. URL https://arxiv org/ /a////bs//3666 07899.
James Vincent. 'As an Al language model': The phrase that shows how Al is pollulating the web. The Verge, Apr 2023. URL https://www. theverge. com//2623/442//2367722 ai di-generated-spam fake- setee-uuer--rviewss www.riessss----------aaaagg--eee
Debora Weber-Wulff, Alla Anohina-Naumeca, Sonja Bjelobaba, Tomás Foltýnek, Jean Guerrero-Dib, Olumide Popoola, Petr Šigut, and Lorna Waddington. Testing of de- tection tools for Al-generated text. International Journal for Educational Integrity, 2023. URL https://doi. 0070://0909999920000000000
Max Wolff. Attacking neural text detectors. Xrv preprint arXiv:2002.11768, abs/ 2002.11768 2020. URL htttps://arxt: wttp://.wr///b///2.11111111
Yihan Wu, Zhengmian Hu, Hongyang Zhang, and Heng Huang. DiPmark: A stealthy, effi- cient and resilient watermark for large language models. arXiv preprint ar arXio:2310.07710, 2023. URL https://arxita http://rawvvv.orrs///////// 07710.
Xianjun Yang, Wei Cheng, Linda Petzold, William Yang Wang, and Haifeng Chen. DNA- GPT: Divergent n-gram analysis for training-free detection of gpt-generated text. arxiv preprint ar ftt:::305.133555 2023a. URL
org. abw.ab/........ 17359.
Xianjun Yang, Liangming Pan, Xuandong Zhao, Haifeng Chen, Linda Ruth Petzold, William Yang Wang, and Wei Cheng. A survey on detection of LLMs-generated content. arXiv preprint ar Xit::2230. 15654, 2023b. URL
a23//300bb//// 15654.
Kiyoon Yoo, Wonhyuk Ahn, Jiho Jang, and No Jun Kwak. Robust multi-bit natural language watermarking through invariant features. In Association for Computational Linguistics (ACL), 2023. URL
wttp.////..././........ 01..85.01900 01904.
Xiao Yu, Yuang Qi, Kejiang Chen, Guoqiang Chen, Xi Yang, Pengyuan Zhu, Weiming Zhang, and Neng H. Yu. GPT Paternity Test: GPT generated text detection with gpt genetic inheritance. arXiv preprint ar (tto:230.12511 2023. URL htttp::/arxiv..om org/ abss/305. 12519.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In Adoances in Neural Information Processing Systems (NIPS), 2019. URL http:::///xxvvvoo///aae/ 1985. 12616.
14

---

Published as a conference paper at COLM 2024
Yi-Fan Zhang, Zhang Zhang, Liang Wang, Tien-Ping Tan, and Rong Jin. Assaying the robustness of zero-shot machine-generated text detectors. arXiv preprint arXi::3312.12916, 2023. URL https://arxix: arg/abs/2312 12918.
Xuandong Zhao, Yu-Xiang Wang, and Lei Li. Protecting language generation models via invisible watermarking. In International Conference on Machine Learning (ICML), 2023. URL
(ICLR), 2024a. URL http:///aaxii..oogbbb/////// 0316 62.
Xuandong Zhao, Prabhanjan Vijendra Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for Al-generated text. In International Conference on Learning Representations aww/aass/55 17439.
Xuandong Zhao, Lei Li, and Yu-Xiang Wang. Permute-anddFllpp An optimally robust and watermarkable decoder for LLMs. arXiv preprint arXir:2402.058664 2024b. URL 058644.
15

---

Published as a conference paper at COLM 2024
A Estimated Fraction of LLM-Modified Sentences in Introductions
20.0%
Computer Science(arXiv)
17.5%
Electrical Engineering Systems Sdence(arXiv) ca(rXit
15.0%
Physics(arXiv))
Nature portfolic
Estimated Alpha
12.5%
MathematiicsaaxXiv
ChatGPT Launch Nov 30, 2022
10.0%
7.5%
5.0%
2.5%
0.0%
2021.1
2021.4
2021.7
2021.10 2022.1 2022.4 2022.7
2022.10
2023.1
2023.4 2023.7 2023.10 2024.1
Figure 6: Estimated Fraction of LLM-Modified Sentences in Introductions Across Aca- demic Writing Venues Over Time. We focused on the introduction sections for the main texts, as the introduction was the most consistently and commonly occurring section across different categories of papers. This figure presents the estimated fraction (a) of sentences in introductions which are LLM-modified, across the same venues Figure 1. We found that the results are consistent with those observed in abstracts (Figure 1). We did not include bioRxiv introductions there is no bulk download of PDFs available. Error bars indicate 95% confidence intervals by bootstrap.
16

---

Published as a conference paper at COLM 2024
B Fine-grained Validation of Model Performance
Electrical Engineering & Systems Scionce(arXiv)
Computer Science(arXiv)
Mathematica(arX(i
Vboeaaary
Vocasiley
Full Voceouery
20%6
ADI
25%
ADI
Abstract
Estimtec Aphe
20ns
Paltmuted Aghee
20%
20%
16%
15%
Estmotod
10%
10%
10%
10%
Ground Truth Alpha (16)
Ground Truth Alphia (%)
Ground TrphhApph 196
Physica(erX(i))
StatisticcsarXiiv
Nature Portfolio
9
bioRxiv
bogwerr Agphe (N0
20%
ADV
VERe
ADJ
Pu Vonatuaay
Matra
20%
25%
ADV
VEHB
ADV
Eadimgted Mptte
20%
ADV
Eatinabed Alphu
20%
27%
VENB
10%-
18%
15%
12%
10%4
10%
40%
0%
0%
5%
10%
15%,
20%
22%
15% 20% Ground ArthhAlpha (74
5% 12%
Ground Trush A pha (75)
10% 15% 20% 25%
Ground Truth Alhha (54)
5% 15% 25% 25%
Electrical Engineering Systems Science(arxiv)
h Computer Science(arXiv)
Matheeatticsar200v
FulVhombaryy
nut Vocatutery
25%
AGJ
Introduction
Eddmated Abhha 1N6
2074
00%-
Edliatted Aggha
15%
Extimetcc
15%
18%
80%
10n
15%
Ground Truhh (N)
Ground Truth Alpha (%)
Ground Truth (%)
Physics(arXiv)
Stadistics(arX(v)
Nature Portfollo
25%
Ful Vibulaulary ADJ
25%
Ful Houtsaarr ADU.
2894
ADI
Estinatod Nyto (%)
ADV
ASV
ADV VERS
201
VERS
20%
VERB
Estivated Appra
20%
Extmated
19%
10%
12%
10%4
0%
8%
0% 5%
0%
10% 15% Alphh
20% 40%
0%
0% 5% 10% 15% 22% 23%
0%
0%
5%
12%
15% 20% 20%
Alphi (1)
Alphe
Figure 7: Fine-grained Validation of Model Performance Under Temporal Distribution Shift. We evaluate the accuracy of our models in estimating the fraction of LLM-modified content (a) under challenging temporal data split, where the validation data (sampled from 2022-01-01 to 2022-11-29) are temporally separated from the training data (collected up to 2020-12-31) by at least year. The X-axis indicates the ground truth a, while the Y-axis indicates the model's estimated a. In all cases, the estimation error for a is less than 3.5%. The first 7 panels (a-g) are the validation on abstracts for each academic writing venue, while the later 6 panels (h-m) are the validation introductions. We did not include bioRxiv introductions due to the unavailability of bulk PDF downloads. Error bars indicate 95% confidence intervals by bootstrap.
17

---

Published as a conference paper at COLM 2024
Electrical Engineering Systems Scienco(arXiv)
Computer Science(arXiv)
Mathematics(arXiv)
25%
25%
25%
Wochsally
Full Vandelayr
Full Wooees dry ADL
Alty
Abstract
13%
10%
10%
10%
10%
0%
5%
19%
10%% 10%
10% 15% 25% 25% Ground Truth Alpnaa (16)
56
0% 5%
12%
10% 201 20%6
Ground Truit Alpha 175)
Ground Truth Alpho 1%1
Physics(arXiv)
Statistics/rrX(v
Nature Portfollo
bioRxiv
29%
25%
25%
Vildwway
Vocobelory
10%
ADV
ADJ
Vitra
20%
VERB
20%
VERS
25%
VERB
(50
15%
10%
Ente
15c
5%
5%
2%
0%
25%
0%
6%
106
15%
2/5 20%
0%
6%
10%
20% 25% (75)
15% 15% 2014
Grannd Truth prw
Electrical Engineering Systems Science(arXiv)
Computer Sclenco(arXtiv
Mathomatics(arttiy
28%
Fill Wocaudarry
Fill Vocctre
ADJ
ADJ ADV
Introduction
ADV
Enoer (57
15%
Eroor (55
mms
10%
10%4
0%
1utt 1ag Daji fagg
5% 10% 19% 20% 15% Ground Alphe (%)
5%. 5%
10% 15% 25% 25%
0% 5% 13% 19% 20% 20% Ground Truth Alphe (N)
Ground Truth Alphin (%)
PhysiosfarXiv)
Statistica(arXiv)
Nature Portfolio
TL
Till Voentu
20%
ADV
AbN
200
ADV
ADJ
20%
AD0
ADV
Egjor (8g
10%
VENS
15%
veeres
Entr (50
10%
VeRs
10%
074
0%
0%4
15%
Ground Truff Alpha (N)
Ground Truth Alsha (%)
Ground TruthAbhh (%)
Figure 8: Fine-grained Validation of Model Performance Under Temporal Distribution Shift. We evaluate the accuracy of our models estimating the fraction of L.o--doiiiied content (a) under challenging temporal data split, where the validation data (sampled from 2022-01-01 to 2022-11-29) are temporally separated from the training data (collected up to 2020-12-31) by at least year. The X-axis indicates the ground truth a, while the Y-axis indicates represents the error (difference between model's estimated a and ground truth a). In all cases, the estimation error for is less than 3.5%. The first 7 panels (a-g) are the validation on abstracts for each academic writing venue, while the later 6 panels (h-m) are the validation on introductions. We did not include bioRxiv introductions due to the unavailability of bulk PDF downloads. Error bars indicate 95% confidence intervals by bootstrap.
18

---

Published as a conference paper at COLM 2024
C LLM prompts used in the study
The aim here is to reverse- the author writing process by taking a piece of text from paper and compressing it into a more concise form. This process simulates how an author might distill their thoughts and key points into structured, yet not overly condensed form.
Now as first step, first summarize the goal of the text, e.g., is it introduction, or method, results? and then given complete piece of text from paper, reverse eenginer it into list of bullet points.
Figure 9: Example prompt for summarizing paragraph from human-authored paper into a skeleton: This process simulates how an author might first only write the main ideas and core information into concise outline. The goal is to capture the essence of the paragraph in a structured and succinct manner, serving as a foundation for the previous prompt.
Following the initial step of reverse eeverse-engineering the author 's writing process compressing text segment from paper, you now enter the second phase. Here, your objective is expand upon the concise version previously crafted. This stage simulates how an author elaborates on the distilled thoughts and key points, enriching them into detailed, structured narrative.
Given the concise output from the previous step, your task to develop into fully fleshed text.
Figure 10: Example prompt for expanding the skeleton into a full text: The aim here to simulate the process of using the structured outline as a basis to generate comprehensive and coherent text. This step mirrors the way an author might flesh out the outline into detailed paragraphs, effectively transforming the condensed ideas into fully articulated section of a paper. The format and depth of the expansion can vary, reflecting the diverse styles and requirements of different academic publications.
Your task proofread the pravided sentence for grammatical accuracy. Ensure that the corrections introduce minimal distortion the original content.
Figure 11: Example prompt for proofreading.
19

---

Published as a conference paper at COLM 2024
D Additional Information on Implementation and Validations
Supplementary Information about Data We collected data for this study from three publicly accessible sources: official APIs provided by ar and bioRxiv, and web pages from the Nature portfolio. For each of the five major ar categories (Computer Science, Electrical Engineering and Systems Science, Mathematics, Physics, Statistics), we randomly sampled 2,000 papers per month from January 2020 to February 2024. Similarly, from bioRxiv, we randomly sampled 2,000 papers for each month within the same timeframe. For the Nature portfolio, encompassing 15 Nature journals including Nature, Nature Biomedical Engineering, Nature Human Behaviour, and Nature Communications, we followed the same sampling strategy, selecting 2,000 papers randomly from each month, from January 2020 to February 2024. The procedure for generating the Al corpus data for a given time period is described in aforementioned Section 3.
When there were not enough papers to reach our target of 2,000 per month, we included all available papers. The Nature portfolio encompasses the following 15 Nature journals: Nature, Nature Communications,, Nature Ecology & Evolution, Nature Structural & Molecular Biology, Nature Cell Biology, Nature Human Behaviour, Nature Immunology, Nature Microbiol- ogy, Nature Biomedical Engineering, Communications Earth & Environment, Communications Biology, Communications Physics, Communications Chemistry, Communications Materials, and Communications Medicine.
Additional Information on Large Language Models In this study, we utilized the gpt-3.5- turbo-0125 0125 model, which was trained on data up to September 2021, to generate the training data for our analysis. The LLM was employed solely for the purpose of creating the training dataset and was not used in any other aspect of the study.
We chose to focus on ChatGPT due to its dominant position in the generative Al market. Ac- cording comprehensive analysis conducted by FlexOS in early 2024, ChatGPT accounts for an overwhelming 76% of global internet traffic in the category, followed by Bing Al at 16%, Bard 7%, and Claude at 1% (Van Rossum, 2024). This market share underscores ChatGPT's widespread adoption and makes it a highly relevant subject for our investigation Furthermore, recent studies have also shown that ChatGPT demonstrates substantially better understanding of scientific papers than other LLMs (Liang et al., 2024b; Liu & Shah, 2023).
We chose to use GPT-3.5 for generating the training data due to its free availability, which lowers the barrier to entry for users and thereby captures a wider range potential LLM usage patterns. This accessibility makes study more representative of the broad phe- nomenon of L.M--ssisted writing. Furthermore, the previous work by Liang et al. (2024a) has demonstrated the framework's robustness and generalizability to other LLMs. Their findings suggest that the framework can effectively handle significant content shifts and temporal distribution shifts.
Regarding the parameter settings for the LLM, we set the decoding temperature to 1.0 and the maximum decoding length to 2048 tokens during our experiments. The Top P hyperparameter, which controls the cumulative probability threshold for token selection, was set to 1.0. Both the frequency penalty and presence penalty, which can be used to discourage the repetition of previously generated tokens, were set to 0.0. Additionally, we did not configure any specific stop sequences during the decoding process.
20

---

Published as a conference paper at COLM 2024
E Word Frequency Shift in arXiv Computer Science introductions
160.0
Intricate
140.0
pivotal
Froquency per one million words
realm
120.0
showcasing
100.0
ChatGPT Launch Nov 30, 2022
80.0
40.0
20.0
0.0
2021.1
2021.4
2021.7
2021.10
2022.1 2022.4 2022.7
2022.10
2023.1
2023.4
2023.7 2023.10 2024.1
Figure 12: Word Frequency Shift in sampled arXiv Computer Science introductions in the past two years. The plot shows the frequency over time for the same words as demonstrated in Figure 2. The words are: realm, intricate, showcasing, pivotal. The trend similar for two figures. Data from 2010-2020 is not included in this analysis due the computational complexity of parsing the full text from large number of arXiv papers.
21

---

Published as a conference paper at COLM 2024
F Fine-grained Main Findings
CL Computation and Language
Papers by first authors with 6 preprints the year
Estimated Alpha
Papers by first authors with 3 preprints the year
20%
15%
5%
0%
2022.1-3
2-22.4-6
2022.7-9 2022.10-11 2023.1-3 1-3 2023.4-6 2023.7-9 2023.10-11 2024.1-2 CV- Computer Vision and Pattern Recognition Papers by first authors with preprints the
20%
Estimated Alpha
Papers by first authors with 3 preprints the
10%
5%
0%
2022.1-3
2022.4-6
2022.7-9 2022.10-11 1023.1-3 2023.4-6 2023.7-9 2023.10-12 2024.1-2
cs.LG Machine Learning
20%
Papers first authors with praprints the year Papers first authors with 3 preprints the year
Eslimated Alpha
15%
10%
5%
0%
2022.1-3
2022.4-6 2022.7-9 2022.10-11 2023.1-3
2023.4 0023.4-6 2023.7-9 2023.10 10-2 2024.1-2
Figure 13: The relationship between first-author preprint posting frequency and LLM usage holds across arXiv Computer Science sub-categories. Papers in each arXiv Computer Science sub-category (cs.CV, cs.LG, and cs.CL) are stratified into two groups based on the preprint posting frequency of their first author, measured by the number of first-authored preprints in the year: those with < 2 preprints and those with > 3 preprints. Error bars indicate 95% confidence intervals by bootstrap.
22

---

Published as a conference paper at COLM 2024
Cs. Computation Language
Papers less similar to their closest peer (above median distance)
Estimated Alpha
Papers more similar to their closest peer (below median distance)
20%
15%
10%
5%
0%
2022.1-3
2022.4-6
2022.7-9 2022.10-11 2023.1-3 2023.4-6 2023.7-9 2023.10-12 2024.1-2
cs. CV- Computer Vision and Pattern Recognition Papers similar to their closest pear (above madian distance) Papers more similar to their closest (below median distance)
20%
Esimated Alpha
15%
10%
5%
0%
2022.1-3
2022.4-6
2022.7-9 2022.10-11
2023.1- 1-3 2023 .-66 2023.7-9 2023.10-11 2024.1-2
Cs.LG Machine Learning
Papers similar peer (above distance)
Estimated Alpha
Papers
peer (below distance)
5%
0%
2022.1-3
2022.4-6 2022.7-9 2022.10-11 2023.1-3 2023.4-6 2023.74 7-9 2023.10-1 2024.1-2 1-2
Figure 14: The relationship between paper similarity and LLM usage holds across arXiv Computer Science sub-categories. Papers in each arXiv Computer Science sub-category (cs.CV, cs.LG, and cs.CL) are divided into two groups based on their abstract's embedding distance to their closest peer within the respective sub-category: papers more similar to their closest peer (below median distance) and papers less similar to their closest peer (above median distance). Error bars indicate 95% confidence intervals by bootstrap.
23

---

Published as a conference paper at COLM 2024
cs. CL Computation and Language Shorter papers 5,000 words)
Estimated Alpha
20%
Langer papers 6,000
15%
10%
5%
0%
2022.1-3
2022.4-6
2022.7-9 2022.10-11 2023.1-3
2023.4-6
2023.7-9
2023.10-13
2024.1-2
cs. CV. Computer Vision and Pattern Recognition Shorter papers words)
20%
Estimated Alpha
Longer papers words)
5%
0%
2022.1-3
2022.4-6
2022.7-9 2022.10-11
2023.1-3
2023.4-6 2023.7-9
2023.10-12
2024.1-2
cs.LG Machine Learning
20%
Shorter papers 5,000 words)
Estimated Alpha
Longer papers (> 5,000 words)
15%
10%
5%
0%
2022.1-3
2022.4-6 2022.7-9 2022.10-11 2023.1-3 2023.4-6 2023.7-9 2023.10-12 2024.1-2
Figure 15: The relationship between paper length and LLM usage holds for cs.CV and Ls. but not for Cs. Papers each arxiv Computer Science sub-category (cs.CV, cs.LG, and cs.CL) are stratified by their full text word count, including appendices, into two bins: below or above 5,000 words (the rounded median). For cs.CL, no significant difference in LLM usage was found between shorter and longer papers, possibly due to the limited sample size, as only a subset of the PDFs were parsed to calculate the full length. Error bars indicate 95% confidence intervals by bootstrap.
24

---

Published as a conference paper at COLM 2024
G Proofreading Results on arXiv data
20%
Before Proofread
Eslimated Alpha
After Proofread
15%
10%
5%
0%
Computer Sclence(arXiv) Electrical Engineering
Mathemsiice/arXOV)
Physica(erXiv)
Statistics(arXMM
Systems Science(arXiv)
Figure 16: Robustness of estimations to proofreading. The plot demonstrates slight increase in the fraction of LLM-modified content after using Large Language Models (LLMs) for "proofreading" across different arXiv main categories. This observation validates our method's robustness to minor LLM-generated text edits, such as those introduced by simple proofreading. The analysis was conducted on 1,000 abstracts from each arXiv main category, randomly sampled from the period between January 1, 2022, and November 29, 2022. Error bars indicate 95% confidence intervals by bootstrap.
25

---

Published as a conference paper at COLM 2024
H Extended Related Work
Zero-shot LLM detection. A major category of LLM text detection uses statistical sig- natures that are characteristic of machine-generated text, and the scope is detect the text within individual documents. Initially, techniques to distinguish Al-modified text from human-written text employed various metrics, such as entropy (Lavergne et al., 2008), the frequency of rare n-grams (Badaskar et al., 2008), perplexity (Beresneva, 2016), and log-probability scores (Solaiman et al., 2019), which are derived from language models. More recently, DetectGPT (Mitchell et al., 2023) found that AT-modified text is likely to be found in areas with negative log probability curvature. DNA-GPT (Yang et al., 2023a) improves performance by examining the divergence in n-gram patterns. Fast-DetectGPT (Bao et al., 2023) enhances efficiency by utilizing conditional probability curvature over raw probability. Tulchinskii et al. (2023) studied the intrinsic dimensionality of generated text to perform the detection. We refer to recent surveys by Yang et al. (2023b); Ghosal et al. (2023) for additional details and more related works. However, zero-shot detection requires direct access to LLM internals to enable effective detection. Closed-sourg commercial LLMs, like GPT-4, necessitate using proxy LLMss, which compromises the robustness of zero-shot detection methods across various scenarios (Sadasivan et al., 2023; Shi et al., 2023; Yang et al., 2023b; Zhang et al., 2023).
Training-based LLM detection. Another category is training-based detection, which in- volves training classification models on datasets that consist of both human and AT-modified texts for the binary classification task of detection. Early efforts applied classification algo- rithms to identify Al text across various domains, such as peer review submissions (Bhagat & Hovy, 2013), media publications (Zellers et al., 2019), and other contexts (Bakhtin et al., 2019; Uchendu et al., 2020). Recently, researchers have finetuned pretrained language model backbones for this binary classification. GPT-Sentinel (Chen et al., 2023) uses the constructed dataset OpenGPTText to train RoBERTa (Liu et al., 2019) and T5 (Raffel et al., 2020) classifiers. GPT-Pat (Yu et al., 2023) trains a Siamese neural network to compute the semantic similarity of AI text and human text. Li et al. (2023) build a wild testbed by gathering texts from various human writings and texts generated by different LLMs. Using techniques such as contrastive and adversarial learning can enhance classifier robustness (Liu et al., 2023; Bhattacharjee et al., 2023; Hu et al., 2023a). We refer to recent surveys Yang et al. (2023b); Ghosal et al. (2023) for additional methods and details. However, these publicly available tools for detecting Al-modified content have sparked debate about their effectiveness and reliability (OpenAI, 2019; Jawahar et al., 2020; Fagni et al., 2021; Ippolito et al., 2023; Mitchell et al., 2023; Gehrmann et al., 2019; Heikkilä, 2022; Crothers et al., 2022; Solaiman et al., 2019). OpenAl's decision to discontinue its Al-modified text classifier in 2023 due to "low rate of accuracy" further highlighted this discussion (Kirchner et al., 2023; Kelly, 2023).
Training-based detection methods face challenges such as overfitting to training data and language models, making them vulnerable to adversarial attacks (Wolff, 2020) and biased against non-dominant language varieties (Liang et al., 2023). The theoretical possibility of achieving accurate instance-level detection has also been questioned "Weber-Wulff et al., 2023; Sadasivan et al., 2023; Chakraborty et al., 2023).
LLM watermarking. Text watermarking introduces a method to detect Al-modified text by embedding an imperceptible signal, known as watermark, directly into the text. This watermark can be retrieved by a detector that shares the model owner's 's secret key. Early watermarking techniques included synonym substitution (Chiang et al., 2003; Topkara et al., 2006b) and syntactic restructuring (Atallah et al., 2001; Topkara et al., 2006a). Modern water- marking strategies involve integrating watermarks into the decoding process of language models (Aaronson, 2023; Kirchenbauer et al., 2023; Zhao et al., 2023). Researchers have developed various techniques, such as the Gumbel watermark (Aaronson, 2023), which uses traceable pseudo-random softmax sampling, and the red-green list approach (Kirchenbauer et al., 2023; Zhao et al., 2024a), which splits the vocabulary based on hash values of previous n-grams. Some methods focus on preserving the original token probability distributions (Hu et al., 2023b; Kuditipudi et al., 2023; Wu et al., 2023), while others aim to improve detectability and perplexity (Zhao et al., 2024b) incorporate mil watermarks (Yoo
26

---

Published as a conference paper at COLM 2024
et al., 2023; Fernandez et al., 2023). However, one major concern with watermarking is that it requires the involvement of the model or service owner, such as OpenAl, to implant the watermark during the text generation process. In contrast, the framework by Liang et al. (2024a) operates independently of the model or service owner's intervention, allowing for the monitoring of AT-modified content without requiring their active participation or adoption.
Implications for LLM Pretraining Data Quality The increasing prevalence of Al-modified content in academic papers, particularly on platforms like arviv, has important implications for the quality of LLM pretraining data. ar Xiv has become significant source of training data for LLMs, contributing approximately 2.5% of the data for models like Llama (Touvron et al., 2023), 12% for RedPajama (Elazar et al., 2023), and 8.96% for the Pile (Gao et al., 2020). Our findings suggest that growing proportion of this pretraining data may contain LLM- modified content. Preliminary research indicates that the inclusion of LLM-modified content (Veselovsky et al., 2023) in LLM training can lead several pitfalls, such as the reinforcement of stereotypes and biases against anyone who is not a middle-aged "European/ Cturopean/North North American man" (Ghosh & Caliskan, 2023; Santurkar et al., 2023), the flattening of variation in language and content (Dell' Acqua et al., 2023), and the potential failure of models to accurately capture the true distribution of the original content, which may result in model collapse (Shumailov et al., 2023). Santurkar et al. (2023) demonstrate that this phenomenon amplifies the effect of LLMs providing content that is unrepresentative of most of the world. As such, our results underscore the importance of robust data curation and filtering strategies even in seemingly unpolluted datasets.
27


Reviews:
[{"review_id": "EsIKuPg8rA", "reviewer": "Reviewer_5uwf", "summary": "The paper investigates the increase in the use of LLM-modified content in academic publications. In particular, changes in the percentage of LLM-modified content before and after the release of ChatGPT are analyzed from the viewpoints of the fields of papers, submission rates to arXiv, similarity between papers, and length of papers. The main finding is that the proportion of LLM-modified content in computer science papers was 17.5%, followed by Electrical Engineering and Systems Science at 14.4%, which is higher than that of other fields. The authors also report that higher levels of LLM revision are associated with papers in which the first author submits preprints more frequently, papers in crowded fields, and papers that are shorter in length.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your helpful feedback and support for the paper! We have carefully updated the paper to incorporate your suggestions. \n\n> The authors should indicate whether there are studies that analyze the proportion of LLM use as addressed in this paper... and how this paper stands compared to those studies.\n\nThank you for the comment. We clarified that we are among the first papers to measure the prevalence of LLM-modified content. Previous work, such as Liang et al. [1] and a follow-up study [2], focused on the prevalence of LLM-modified content in scientific peer review for AI conferences. In contrast, our paper investigates the prevalence of LLM-modified content in the writing of academic papers themselves. \n\n[1] Liang et al. \"Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews.\"ICML  (2024).\n\n[2] Latona et al. \"The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates.\" arXiv preprint arXiv:2405.02150 (2024).\n\n\n> Study Limitations\n\nThank you for the comment. We have added discussions to acknowledge this limitation: \"One potential confounder of our study is the increased prevalence of research on LLMs after the launch of ChatGPT. This shift in research focus could potentially affect the accuracy of our method in detecting LLM-modified content. However, our validation has shown that our framework is robust under temporal distribution shifts of research topics. Still, future studies could further validate and analyze the robustness of our method with more systematic control of the study content.\"\n\n\n\n> In the case of journal papers... Is the analysis based on the date of submission or publication? \n\nWe analyzed Nature portfolio journal papers using both submission and publication dates. The results were consistent, with Nature portfolio papers having among the lowest estimated alphas, even when plotted by submission date. This clarification has been added to the revised manuscript.\n\n> Whether the collected paper set is crowded or not depends on its coverage and the breadth of the field covered...\n\nWe clarified that we used only arXiv CS papers for our fine-grained analysis. To validate the robustness of our findings, we stratified the papers by arXiv CS subfields (cs.CV, cs.CL, cs.LG) and found consistent results (see Supplementary Figures 12-14).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses the interesting topic of the use of LLM in academic writing.\n- The analysis is based on relatively large data sets, and the analytical methods are considered to have a certain degree of reliability.\n- While there are no major surprises about the findings, the authors have found a clear trend in the use of LLMs.", "reasons_to_reject": "- The related work section is a weakness of this paper, as it only mentions studies related to determining whether a given sentence is LLM-modified. The authors should indicate whether there are studies that analyze the proportion of LLM use as addressed in this paper, if they exist, what they are, and how this paper stands compared to those studies.\n- Although there are related references in the “limitation” section, it cannot be denied that one of the reasons for the significant change in trends before and after the appearance of ChatGPT in the field of computer science is the existence of many papers on LLM and studies using LLM in this field, and the possibility that the number of contents that could be mistakenly identified as LLM-modified has increased.", "questions_to_authors": "- In the case of journal papers, a certain amount of time is considered to have elapsed between submission and acceptance. Is the analysis based on the date of submission or publication? If it is the latter, isn't some kind of correction necessary?\n- Whether the collected paper set is crowded or not depends on its coverage and the breadth of the field covered. As an extreme example, the field covered by Nature is very broad, and on the other hand, it does not cover all papers in the related field, so the paper set is unlikely to be crowded. Is it reasonable to discuss whether an area is crowded or not in a situation where different sources of the set of papers are used?", "extra_scores": {}}, {"review_id": "sgqZJsXnn0", "reviewer": "Reviewer_Rcc2", "summary": "This paper highlights growing concerns and interest in the prevalence of AI-generated text in academic publishing, spurred by anecdotal examples and evolving editorial policies. It discusses the need for systematic analysis to understand the extent and implications of AI-modified content, introducing a framework for quantifying such modifications at scale. The study applies this framework to a large dataset of academic papers across various disciplines, revealing significant growth in AI-modified text, particularly in Computer Science. It also identifies associations between AI-modification, author behaviors like preprint posting frequency, and paper characteristics like length.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Rcc2 for their positive comments and helpful feedback on our work. \n\nWe have added discussion on the positive aspects of AI usage as follows:“Researchers who are not native speakers of English may find it helpful to have an AI model polish their writing. Additionally, LLMs offer the possibility of immediate feedback on initial drafts compared to traditional peer review processes, which can be time-consuming.” We have added these discussions into the revised manuscript. \n\nWe again thank Reviewer Rcc2  for their review of our manuscript, and we hope that the above responses adequately address their concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The study offers a novel contribution to the field by introducing a framework for quantifying AI-modified content in academic publishing and applying it at scale across multiple disciplines.\n2. With the increasing concerns and debates surrounding the use of AI-generated content in academic publishing, the study's findings are highly relevant and timely.\n3. The study opens avenues for further research into the impacts of AI-generated content on scholarly communication, knowledge dissemination, and academic discourse. Future studies could build upon the framework and findings presented, exploring additional factors influencing AI-modification trends and their broader implications for the academic community.", "reasons_to_reject": "1. The study focuses primarily on quantifying the prevalence of AI-modified content in academic publishing without delving deeply into the potential positive aspects of AI technology in this context.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RoDAO49TwK", "reviewer": "Reviewer_T5PN", "summary": "The paper analyses the use of large language models (LLMs), specifically of ChatGPT in writing scientific paper in specific fields, i.e. in sciences (Computer Science, Mathematics, Statistics, etc.). For this, they collect data from various publishers (also arXiv) and identify the usage of ChatGPT with the distributional LLM quantification. Their results show that there is an increase of ChatGPT usage in scientific writing starting from 2020 with some disciplines using it more frequently (Computer Science) than the others (Mathematics).\n\nEmpiricism, Data, and Evaluation: The paper offers a strong empirical foundation, as the study uses a large collection of scientific papers (and also abstracts) and applying an established method with this data. In this way, it also offers reproducible results. \n\nAmbition, Vision, Forward-outlook: The growth in using LLMs in the recent years is very fast. The challenges posed are important to be addressed in such studies. This work is timely and highly relevant.\n\nUnderstanding Depth, Principled Approach: The authors show a good understanding of the approach used. They also provide detailed analysis of various aspects, such as correlation with publishing pre-prints, paper length, etc.\n\nClarity, Honesty, and Trust: The paper under review is clearly written. However, I have some comments on the structure. The introduction contains graphs which is very unusual. It would be better to restructure the paper in the way that the introduction describes aims and motivation, as well as what comes next (paper organiser) only. The other parts should come later, e.g. graph illustrations should be part of the results. \n\nFurther issues are rather minor and concern formatting. For instance, quotation marks are not properly formatted and point to one direction.\nThe authors should also indicate the date of access to the given URLs (e.g. in references).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We thank Reviewer T5PN for their positive comments and for providing thoughtful feedback on our work. \n\nFollowing your guidance, we have also revised the introduction section to improve clarity, and discussed the scope and the motivation of our study. Furthermore, we respectfully clarify that putting figures in the introduction has been a common practice in the literature [1,2,3] which we followed. \n\n\n**References**\n\n[1] Schaeffer, Rylan, Brando Miranda, and Sanmi Koyejo. \"Are emergent abilities of large language models a mirage?.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2] Mitchell, Eric, et al. \"DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature.\" (2023).\n\n[3] Wei, Jason, et al. \"Chain-of-thought prompting elicits reasoning in large language models.\" Advances in neural information processing systems 35 (2022): 24824-24837.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is highly relevant for the topic of the conference. The results are not only interesting but can also be inspiring for further research.", "reasons_to_reject": "I do not see any reasons to reject. However, I would recommend the authors to restructure the paper for the final version for a better readability.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hBjdXGt2rq", "reviewer": "Reviewer_i8UX", "summary": "Thanks to the authors for the hard work on this paper. The work is well-written and well-argued. It is a relatively small contribution: just applying an existing algorithm to existing data after validating/training with synthetic + real data. The analyses are nice, but can be combined into a single, more comprehensive one (I discuss this in more detail below). In summary, I like this work, but it should be expanded.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and the time you invested in reviewing our manuscript. We are delighted that you found our paper to be well-written, clear, and compelling.\n\n> [Major] These cutoffs don't seem principally motivated, ... I recommend instead a fixed-effects model where the output (estimated alpha) is modeled as an output of multiple input factors simultaneously....\n\nThank you for the suggestions. We clarified that the cutoffs are not arbitrary. The cutoffs in Sections 5.3 and 5.4 are determined by the rounded median. We used the rounded-up average for Section 5.2. We have added these clarifications in each of the figure captions of Figures 4, 5, and 6.\n\nWe appreciate the reviewer's thoughtful suggestions on the fixed-effect study. We clarified that our algorithm only provides a population-level estimate. Conducting a fixed-effect study is challenging without the availability of individual-level outcomes/estimates. Nevertheless, we have included a robustness analysis to validate the stability of our findings. We agree with you that certain subfields can be more crowded, and authors in these subfields may publish more papers simultaneously. Therefore, to further validate the robustness of our findings, we stratified the papers into Computer Vision (cs.CV), Computation and Language (cs.CL), and Machine Learning (cs.LG). We found that our results are consistent when stratifying by subfield (Supp. Figures 12, 13, and 14). This consistency indicates the robustness of our findings.\n\n\n> [Minor] Page 5: \"This approach also simulates how scientists may be using LLMs...\" Need a citation here ... supports your two-stage approach.\n\nThank you for the suggestion. Following your guidance, we have added a citation to Lee et al. [1], which characterizes the stages of human writing with AI assistants as planning and drafting, which aligns with our two-stage process. We have added the citation and the discussion into the methods section of the revised manuscript. \n\n[1] Lee, Mina, et al. \"A Design Space for Intelligent and Interactive Writing Assistants.\" Proceedings of the CHI Conference on Human Factors in Computing Systems. 2024.\n\n> [Minor] Figure 3 ... changing y-axis to \"error\" instead. \n\nThank you for the suggestion. We have added a figure where the y-axis has been changed to \"error\" in the supplementary figures for clarity. \n\nWe again thank Reviewer i8UX for their review of our manuscript, and we hope that the above response adequately addresses their concern.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is clear. The conclusions make sense and are compelling. The existing analysis seems well-done. Nice charts.", "reasons_to_reject": "- [Minor] Page 5: \"This approach also simulates how scientists may be using LLMs...\" Need a citation here or another piece of evidence that supports your two-stage approach. \n- [Minor] Figure 3 is hard to read. I recommend changing y-axis to \"error\" instead. That's the information I want to extract in any case. \n- [Major] Sections 5.2, 5.3, and 5.4 are all done independently and in somewhat arbitrary ways. I.e. for 5.2, paper are divided as two or fewer preprints vs three or more, and paper length as below or above 5000. These cutoffs don't seem principally motivated, and the results may be different if they change. Additionally, all of the analyses are done independently. For these kinds of analyses, I recommend instead a fixed-effects model where the output (estimated alpha) is modeled as an output of multiple input factors simultaneously. E.g.: datetime (bucketed), # of preprints (bucketed or not), arxiv category (categorical), distance to nearest neighbor (bucketed or not), pre-print posting. Adding new ones is straightforward both conceptually and computationally, and you are then able to look at how any individual variable is correlated (via its coefficient size) to the output when taking all others into account. It may be the case for example, that once you take into account # of preprints, the effect size of distance to nearest neighbor goes to 0. What other factors can you include in your model that may be confounders?", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper the authors study the use of / increase in LLM-generated text in scientific research articles. They do an analysis of ~1M papers published on arXiv, bioarxiv and Nature portfolio using the (previously published) distributional LLM quantification framework, which estimates the corpus-level proportion of ChatGPT-generated text by generating LLM versions of text from known-human authored papers. They perform an interesting analysis of the results with respect to field, submission rates, and paper lengths, finding that LLM-generated content is indeed increasing substantially over time since the launch of ChatGPT, and correlations between rate of publication (at the author and field level) with higher rates of LLM-generated content.

The reviewers agree that the paper addresses an interesting and timely topic using reliable analytical methods, and that it is a good fit for the conference. While the paper was sufficiently clearly written, the reviewers recommended some minor edits for grammar and clarity and additions to related work.
Decision: Accept