LLMs as Research Tools: A Large Scale Survey of Researchers’ Usage and Perceptions
Authors: Zhehui Liao
Maria Antoniak
Inyoung Cheong
Evie Yu-Yen Cheng
Ai-Heng Lee
Kyle Lo
Joseph Chee Chang
Amy X Zhang
Venue: COLM 2025 (2025)
Keywords: survey
large language model
research
societal impact
TL;DR: A large-scale survey of 816 researchers to study usage of LLMs in scientific research and the perception of such usage.

Abstract: The rise of large language models (LLMs) has led many researchers to consider their usage for scientific work. Some have found benefits using LLMs to augment or automate aspects of their research pipeline, while others have urged caution due to risks and ethical concerns. Yet little work has sought to quantify and characterize how researchers actually use LLMs and why or why not. We present the first large-scale survey of 816 verified research article authors to understand how the research community leverages and perceives LLMs as research tools. We examine participants' self-reported LLM usage, finding that 81% of researchers have already incorporated LLMs into aspects of their research workflow. We also find that some traditionally disadvantaged groups in academia (non-white, junior, and non-native English speaking researchers) report higher LLM usage and perceived benefits, suggesting potential for improved research equity. However, women, non-binary, and senior researchers have greater ethical concerns. Our study provides much-needed evidence, rather than speculation, about how LLMs are currently being used as research tools.

Published as a conference paper at COLM 2025
LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions
Zhehui Liao", Maria Antoniak", Inyoung Cheong', Evie Yu-Yen Cheng", Ai-Heng Lee*, Kyle Lo", Joseph Chee Chang", Amy X Zhang'
University of Washington, Paul G. Allen School of Computer Science & Engineering University of Colorado Boulder, Department of Computer Science
Princetonn University, Center for Information Technology Policy
Allen Institute for Artificial Intelligence
Abstract
The rise of large language models (LLMs) has led many researchers to consider their usage for scientific work. Some have found benefits using LLMs to augment or automate aspects of their research pipeline, while others have urged caution due to risks and ethical concerns. Yet little work has sought to quantify and characterize how researchers actually use LLMs and why or why not. We present the first large-scale survey of 816 verified research article authors to understand how the research community leverages and perceives LLMs research tools. We examine participants' self-reported LLM usage, finding that 81% of researchers have already incorporated LLMs into aspects of their research workflow. We also find that some traditionally disadvantaged groups in academia (non-white, junior, and non-native English speaking researchers) report higher LLM usage and perceived benefits, suggesting potential for improved research equity. However, women, non-binary, and senior researchers have greater ethical concerns. Our study provides much-needed evidence, rather than speculation, about how LLMs are currently being used as research tools.
1 Introduction
The burst popularity widely available generative Al tools, and findings from recent small-scale studies with researchers (Morris, 2023; Fecher et al., 2023) suggest that many in the research community have already found benefits in incorporating large-language models (LLMs)) into their research workflows. Adopting these tools has opened up many possibilities, such as improved efficiency, greater research equity, and inspiring novel ideas. At the same time, new tools raise familiar research risks and ethical concerns- like transparency, reproducibilityy plagiarism, and data fabrication introducing new dangers to the research process. Differences in perceptions about risks, ethics, and social acceptability across demographic groups and researcher backgrounds could also drive differences in adoption, so that any benefits accrue unevenly and exacerbate existing structural barriers in academia (Goyes and Skilbrei, 2023).
Due to high interest in the research community in developing and adopting LLMSs as research support tools, there have been a plethora of recent work on investigating researchers' usage of LLMs. However, most has focused on one specific research domain; for example, HCI (Kapania et al., 2024), psychology (Ke et al., 2024), machine learning (Russo et al., 2024), and management research (Williams et al., 2023). While there have also been few investigations that included multiple disciplines, they relied small-scale interviews (Morris, 2023, N=20), surveys (Fecher et al., 2023, N=52), or automated textual analysis of published papers (e.g., Kobak et al., 2024; Liang et al., 2024b).
1

---

Published as a conference paper at COLM 2025
In contrast, our work represents the first large-scale survey (N=816) focusing on researchers' use of LLMs across disciplines, providing empirical evidence for trends previously only speculated about. By asking researchers about both their general perceptions and personal usage, our survey allows us to describe developing norms in more detail, across disci- plines and demographic groups. Participant recruitment can be challenging for large-scale surveys but crucial for the validity of the results. For this, our participants were sourced from Semantic Scholar, which maintains a verified repository of more than 100 thousands published researchers from wide range of research domains, demographics, and research experiences. Our survey focused on understanding how researchers are actually using LLM-based researcher tools in their work today, and how they perceive the risks and ben- efits of leveraging LLMs for different research tasks. In particular, we were interested in researchers' perceptions about the acceptability of these tools and possible demographic differences, leading us to recruit researchers across nationalities, languages, career stages, disciplines, genders, and ages. The differences we uncover between these groups reveal rapidly changing social norms around the usage of AI tools in research, highlighting im- portant considerations around research equity and broader adoption. Our contributions include the following.
We find around 81% of respondents have used LLMs for research, with the tasks of information seeking and editing reported most frequently and data analysis and generation reported least frequently.
Researchers who are non-white, non-native English speaking, and junior researchers both use LLMs more frequently and perceive higher benefits and lower risks, but women and non-binary researchers have greater ethical concerns.
While LLMs are used across all fields, computer science researchers showed greater comfort with disclosure of usage and lower ethical concerns than other disciplines.
Researchers generally prefer to use LLMs from open source/non-profit entities.
Our findings suggest that LLMs might help improve research equity for people from demo- graphic groups traditionally facing certain structural barriers, according to the researchers from those groups. However, ethical issues around LLMs need be confronted and social norms established within each field for broader adoption.
Related Work
LLMs as Research Support Tools The Al and HCI research communities have been exploring research tools powered by LLMs to support all stages of the research workflow, from research ideation (Gero et al., 2022; Liu et al., 2024; Wang et al., 2024), paper reading (Lo et al., 2023; Fok et al., 2023), literature review (Lee et al., 2024b; Kang et al., 2023; Hsu et al., 2024), writing (Kim et al., 2023; Long et al., 2023; Gruda, 2024), peer review support (Sun et al., 2024; D'Arcy et al., 2024), and more. Additionally, there has also been increased commercial interests in building L.L.-based tools for science, such as Galactica (Taylor et al., 2022), Perplexity, Elicit, OpenAl and Gemini's Deep Research modes, and many more. There are even attempts to build fully autonomous end-to-end research agents based on LLMs such as Sakana AI (Castelvecchi,, 2024a;b) and FutureHouse (Sam Rodriques, 2024). These recent works show promise for LL.M-powered research support tools, and our survey further confirms already widespread adoption in the research community.
Risks and Ethical Implications of LLMs in Research There are many risks involved in using LLMs for research, including insufficient precision and accuracy (Alvarez et al., 2024), with a phenomenon of "hallucination" (Alkaissi and McFarlane, (eee)a-nnratiing plausible-sounding but fictional content. For example, Galactica, an LLM trained on sci- entific papers (Taylor et al., 2022), was taken down after producing convincing but false scientific articles (Heaven, 2022). Without vetting, inaccurate content could contribute to
Link to survey questions, analysis code, and results: rlm-research-uurvey
https://gi a lenai/
2

---

Published as a conference paper at COLM 2025
misinformation and erode trust in research (Kobiella et al., 2024; Gruda, 2024; Antoniak et al., 2024; Toma et al., 2023). These issues are particularly concerning because many researchers, particularly those lacking expertise resources, predominntly rely commercial closed models (Wulff et al., 2024; Toma et al., 2023).
Demographic Influences on LLM Perception and Adoption Individual perceptions and usage patterns of LLMs are shaped by various factors, including personality traits, age, gender, and educational background (Jakesch et al., 2022). For instance, people with a high level of agreeability and younger people tend to have more positive views of AT, while those susceptible to conspiracy theories often have more negative perceptions (Stein et al., 2024). Also, notable gender gap has been observed in LLM adoption, with men users outnumbering women users (Draxler et al., 2023). LLMs might reduce certain inequities in research and publishing by lowering barriers for non-native English speakers (Morris, 2023) and providing high-quality reviews to novice researchers (Chamoun et al., 2024).
3 Methods
3.1 Survey Design, Participant Recruitment, and Data Collection
Through literature reviews and formative survey X/Twitter, we iteratively designed questionnaire with feedback from researchers (see Appendix A.1). The survey asks about perceptions and usage of LLMs for different research tasks, with responses provided via Likert rating² and free response rationale. We recruited verified authors with at least one published paper listed on Semantic Scholar that has a dedicated data quality team maintaining academic author profiles. The study was reviewed and exempted by the IRB of the University of Washington. A recruitment email was sent to 100,187 verified authors, and 34,922 (34.9%) emails were opened. From November 2023 to April 2024, we collected 1,226 unfiltered survey responses via Qualtrics, which we filtered to exclude participants who did not progress past the first page or spent fewer than seconds on each question. This resulted in 816 total responses.
We collected fine-grained self-reported demographic information and then manually coded and categorized the responses (see Appendix A.3). Of the 816 total responses, 644 included demographic information, with the following distributions. Gender: Man (79%); Woman, Non-Binary, Other ;); Race: White (61%); Non-White (39%); Years of Research Experi- ence: 11+ (57%); 4-10 (32%); 0-3 (11%); Native Language: Native English (62%); Non-Native English (38%); Field of Study: Computer Science (40%); Social Science & Humanities (24%); Natural Science & Engineering (21%); Biology & Medicine (15%). We provide additional details in Table in the Appendix.
3.2 Quantitative Analysis
Each participant labeled with (up to) five demographic categories and contributes (up to) 36 Likert ratings (an LLM Usage Frequency question and five LLM Perception questions, each repeated for six LLM Usage Types). To address repeated measures, we fit linear mixed- effects models (1me4 in R) of the form
LikertRating, = Bo + 7i + Demographii; + $2UsgeeType, + +
(1)
to test the association between participant i's LikertRatings (answers to LLM usage or perception survey questions) and participant Demographic fixed-effects, while controlling for participant-specifii random effects 7 and the type of LLM usage in question
In total, we fit 30 models (6 types of Ratings x Demographic groups), use likelihood ratio (p. adjust in
tests for significant Demographic effect (Ho ß1 0), and use
M usage is rated on 6-pt scale while perception is rated on a 5-pt scale excluding option "T have never used L.L for this type of activity" for later analysis.
3

---

Published as a conference paper at COLM 2025
R) to correct for multiple comparisons within model fits for the same Rating type. Each model was fit on approximately three four thousand participant ratings after filtering out missing data; exact sample sizes per regression are in Table 4 in Appendix B. Given significant result, we then also conduct post hoc tests for significant pairwise differences (emmeans() () in R) in mean Rating between Demographics; for example, testing Ho Mwhite Pnon-white 0.
All survey questions, data, and R code is available in Supplementary Materials.
3.3 Qualitative Analysis of Free-Text Responses
To collect deeper insights, we analyzed free-text responses through an iterative open the- matic analysis approach (Boyatzis, 1998; Connelly, 2013). Three of the paper authors read through and annotated the same subset of the responses into thematic categories inde- pendently. Through discussion with the entire research team and additional rounds of independent annotation, we settled on a final set of themes. The same three authors anno- tated the full of responses (each response was annotated by one author). The full set of themes (one set per question), with definitions and examples, can be found in Appendix C, and we release our full set of annotations along with our other data and resources.
4 Results
4.1 How do researchers use LLMs?
Farely Oecasioraly Frequesty
Types
Meston Phaning
Gredt
Dite Choreaa Aettes
40%
20%
0%
20%
40%
45%
80%
Usage Frequeeccy Percentage
Figure The left diverging bar chart displays the distribution of usage frequency across six types of LLM usage. The midpoint (0%) is centered at "Occasionally: " The grey bar chart indicates percentages of responses that report "Never" using LLMs for each type.
Seeking
Editing
Cleaning Analysis
Oksover
Regramna oato ise
Claan and neturte
pppors
armarie
Senerate
Look syrongms
Socleticl reportng
oepannzioos
243
Paper for natting
Qualitatve
arody si
Dlazover
topice
Writing (Total: 352)
Ideation Framing (Total: 378)
(Total: 223)
mather assle
100
Realestom
ROu
144
100
Corti uS ways Thame paper
135
rabernlaaels and peerrplees
100
from ideas
Figure Each bar chart shows the number participants who reported using LLMs for the indicated research task. Participants could select multiple tasks and subtasks.

---

Published as a conference paper at COLM 2025
We asked participants to mark how frequently they used LLMs for six types of usage: Information Seeking, Editing, Ideation & Framing, Direct Writing, Data Cleaning & Analysis, and Data Generation. Overall, 80.88% (660 out of 816) of respondents reporting some use. The most popular tasks were Information Seeking and Editing (49% and 45% respondents rated at least occasional usage), with by far most usage on rewriting text to fix grammar or awkward phrasings (Figure Figure 2). Most respondents (69% and 73%) stated they never used LLMs for Data Cleaning & Analysis or Data Generation.
find that LLMs are frequently used by researchers, with
We find that researchers' racial identity is significantly associated with LLM usage, with non-white researchers reporting more frequent usage of LLMs than white researchers
(p < .0001)3 When using LLMs for editing, we see significantly greater usage by non-native English-speaking researchers (p < .0001), though this difference was not found for other tasks (see first column in Figure 3).
4.2
How do researchers perceive LLM usage?
Usage Frequency
Benefits (Is usetul?)
Risk (Is risky?)
Ethics (Is acceptable?)
Comfortable disciosing to peers?
Comfortable disciosing to revlowers?
Race
Ven Whit
Gender
Man
Language
Non-kodive Eng
Experience
6-ayes
410yys
13 yma
Field
Compute
Racual
Med
Socill 5d
Participants
Usage Frequency
Perception
Types Usage
Significant Differance Groups
Very Freq uently
Information seeking
Frequentyy
Osuasionally
Walky Soreewhor KooernbbaaaCoooootttt
Modenapely Uschw/Rssty
13- idear Framing
12- Editiig
Direct Witting
p-0
ps0.005 ***
p-0.01
Analysis
Sarely
Standy Uortu
15 Data Generation
Viry Fannyy
Louhu altibdy Unacceptabii Linoooorceeee
Figure 3: Each heat map square represents the average rating of this demographic group on the usage frequency or perception for the particular type of LLM usage. (***) shows significance of differences (p-values from regressions). Brackets indicate significance across all types of usage, whereas lines between squares indicate significance only for certain tasks.
We asked participants about their perceptions of the benefits and risks of LLM usage for research, the acceptability of such usage, and their comfort level with disclosing LLM usage to peers and reviewers, for each task category through Likert ratings. We also asked them to elaborate via free-response questions. In columns 2-6 of Figure 3, we present the average
'eee and in following results, we report as the mean of Likert ratings (min 1, max 5), and p p-values from host-hoc tests for significant pairwise differences among demographic groups using emmeans in Section 3.2.

---

Published as a conference paper at COLM 2025
rating given by participants, broken down by tasks and participants' backgrounds. Full significance tests can be found in Table 6 in the Appendix.
4.2.1 Perceptions of LLM benefits
Participants perceived greater benefits for Information Seeking (n = 3.2) and Editing (== = 3.4) compared to the other tasks (μ S 2.6). Overall, we found groups traditionally disadvantaged in the research community perceived more benefits. Non-white researchers perceived more benefits than White researchers (p < ,0001), as did researchers with 0-3 years of experience perceived more benefits than those with 11+ years of experience (p -0004). Connecting findings on usage frequency and the correlation between perceptions and usage, together they suggest perceived utility drives higher LLM usage among non-white and junior researchers. Similarly, non-native English speakers perceived greater benefits in using LLMs for Editing (p 0.0004), as well as reported using LLMs for Editing more frequently.
We also identified both language equity and other equity as major themes in respondents' discussions of LLM benefits ("For honest researchers in ceonurce-oonstranntt developing countries, with little to no research funding, availability and use of LLMs is game-changer leveling the playing field with other researchers in more fortunate climes." "). Benefits were especially highlighted by researchers facing systemic barriers, such as non-native English speakers, junior scholars, and researchers without much programming experience ("I am not native English speaker, so LLMs help me with the language barrier.") "). We list all themes in Table 8 of the Appendix.
4.2.2 Perceptions of LLM risks and ethical concerns
To distinguish risks from ethical concerns, we asked participants to separately rate their perception of risks, given known issues with LLMs today, and their perception of the acceptability of using LLMs given a future where LLMs can prevent hallucinations and can always attribute any copyrighted text (if generated) to the original sources. Overall, researchers perceived using LLMs for Editing not risky (μ 2.5), Direct Writing as moderately risky (μI 3), and the remaining categories as very to extremely risky (u 2 3.2). Non-white researchers perceived fewer risks in using LLMs for Data Cleaning & Analysis
(p < .0001), and Data Generation (p 0.0352).
But in their acceptability ratings, researchers reported Ideation & Framing (μ 2.9) and Data Cleaning & Analysis (μ = 2.96) as more unacceptable, while the remaining categories were found more acceptable (μ 2 3.4). Across all tasks, a researcher's ss gender was significantly associated with their perception of the ethics of using LLMs. Researchers who identified as Women, non-binary, and other genders perceived LLM usage in research as less acceptable than those who identified as men (p 0.0017). Similarly, senior researchers (11+ years of experience) perceived less acceptability than junior researchers (4-10 years of experience) for Editing (p 0.0215) and Data Cleaning & Analysis (p 0.0013). In contrast, non-white researchers perceived fewer ethical concerns for the more unusual usage categories, such as Data Generation (p 0.0230), in keeping with their lower perceptions of risks, higher perceptions of benefits, and higher usage. Finally, computer scientists perceived using LLMs for Editing as more acceptable than social scientists (p 0.0008), and they perceived Data Generation as more acceptable than natural scientists (p = 0.0075).
Participants used strong language to share their opinions about the risks and ethics of LLMs for research in their free-text responses ("LLMs are tools for automaled plagiarism and data fabrication that pose an existential threat to the network of trust essential for the integrity of academic work and the proper attribution of credit"). While many pointed to specific risks like data fabrication and plagiarism, others drew attention to higher level concerns that could affect all of academic research, such as pollution of the research ecosystem with low-quality work ("We need better judgment, slower science, and more thoughtful and ambitious work right now, not the opposite. Otherwise, we risk ridding science of its most special attributes just to crank out more papers."). Respondents worried about future generations of researchers whose skills, diligence, and creativity may be impacted by over-reliance on LLMs ("The main general risk is to flatten on 'average', which is the worst thing that may happen for a researcher...since this would block innovation"). They also worried about exacerbating existing problems, like
6

---

Published as a conference paper at COLM 2025
overwhelming numbers of papers needing review fear for deluge of Al-'assisted' (in the best case) papers that read somewhat fluently but are shallow, unoriginal, uninteresting, wrong in the details. This will overwhelm the peer review system") ) For detailed overview of these themes, drawn from our qualitative analysis, see Table the Appendix.
4.2.3 Comfort with disclosure
We asked participants to rate their comfort with disclosing their use of LLMs for each of the six tasks. Overall, participants were comfortable with disclosing to both peers and reviewers across all usage types (disclose to peers: 2 3.4, disclose to reviewers: 2 3.2). However, we also found that computer scientists reported more comfort with disclosure than social scientists (to peers: p 0.002; to reviewers: p 0.0043) and biologists & medical researchers (to peers: 0.001; to reviewers: 0.007).
Qualitatively, participants' opinions the disclosure of LLM usage and proper attribution were varied. One respondent mentioned "academic shame" as a reason to not disclose LLM usage. But other respondents highlighted the costs the research community of not disclosing LLM usage: "IIf researchers don't disclose using LLM-generated text], I fear that researchers can get lazy, and we start having a lot of 'repeated text' in articles... and eventually researchers may just ask LLMs to generate the whole paper." " Some respondents listed this as their main concern with LLM-assisted research, though long as LLM usage was disclosed, many described that usage as acceptable: "The same sort of disclosure of use [as with human assistance) should be sufficient. The same responsibility for the integrity of work applies whether part of the effort was provided by human assistant or an LLM." Finally, one respondent called for better processes to support disclosure: "..uniiersities have fotally different policies. It would be good if there was generic system of how to indicate that editing or drafting tools were used."
4.3 Does a researcher's usage of LLMs relate to their perceptions?
Emouency usage
Os risky??
Banefits li useful7)
lis accoptabler))
Confortabble disclosing to peers?
Comfortable reviewers?
very Peepurtty
Mespooissss
Requertly
Oxcabonaly
Rondy
kayy
Drtemally
Lorendy
Unacrestaca
fazepable
Uncerfs 1a01
Co onforaalee Uncomforate
Coinforrblee
* -0-401
= 0.62
T 0.389
* 0.22
0.100
Figure 4: Each heatmap is for one type of perception, and each cell represents the number of responses (log scaled) that fall under this level of frequency of perception. The Kendall's tau coefficient on the bottom indicates how strong the correlation is between the usage frequency and the perception of that usage. All perceptions are significantly correlated with usage (p- < .0001). Tests performed using cor. test and corrected with p. adjust.
Participants' perceptions of risks, benefits, and ethics and their willingness to disclose were all significantly associated with their usage of LLMs (p .0001). As expected, greater perceived risks and ethical concerns were associated with lower usage, and greater perceived benefits were associated with higher usage (Figure 4). However, perceived benefits had the strongest correlation to usage (T - 0.62, p < .0001), and weaker correlation for risk (T -0.401, p < .0001) and ethics (T 0.389, p < .0001, see Table in Appendix). Some who perceived few risks or ethical concerns still reported infrequent usage. We also found weaker but positive relationship between comfort with disclosure and frequency of LLM usage, with discomfort highest for those who never used LLMs for given usage category. This suggests possible lack of social norms around usage and disclosure may be hindrance to adoption.
7

---

Published as a conference paper at COLM 2025
4.4 Do researchers care about the organizational governance behind LLMs?
Participants were split on whether the source of an LLM, (i.e., non-profit versus for-profit entities), impacted their perception of benefits and risks. 54.81% of participants (359) reported their perception would change depending on the model source while 45.19% (296) reported no difference. In their free-text elaboration, 59.07% (228) of respondents preferred LLMs from open nource/non-profft entities,* 4 while only 2.85% (11) stated they preferred LLMs from for-profit corporations, and 38.08% (147) did not express a preference.
Qualitatively, the top reasons participants gave for their non-profit preference included the incentives of the organization, the transparency of the model, and ethics and bias considerations. These participants were skeptical of the incentives of commercial corpora- tions, and worried that they would "exploit user imput, manipulate LLM outputs for financial gain." " They also expressed concerns about monopolization and injecting bias to maximize profits. They favored non-profit entities because of the transparency in open-source models, increasing accountability, and users' trust. For the few participants who favored LLMs from for-profit entities, they believed those models were of higher quality due to the resources available to companies and their responsibility towards supporting customer issues. For participants who were indifferent, some held an attitude of neutrality: "the technology is the same" no matter which organization provided it. Some expressed that they cared more about the quality of the model, and would use the model with the best quality regardless of its source. Finally, other participants questioned the boundary between for-profit and non-profit entities: "as we have seen with OpenAL, non-profits can easily become commercial.." " Some respondents prioritized whether the model was open source over whether it was developed by a non-profit or for-profit entity. Other respondents were skeptical about the open source label: "No LLM is really open source. Most of them owe their existence to big commercial corporations, and even if they share the weights, we don't really know all the details about the training data. They are essentially black boxes."
Discussion
Deep and Pervasive Integration of LLMs in Research Our work revealed that most researchers have already found benefits in incorporating LLM-based tools into their current workflow, from literature review to data analysis. This confirms and expands upon prior assumptions about the prevalence of LLM usage in academia (Morris, 2023; Gruda, 2024; Bail, 2024; Koller et al., 2023; Kobak et al., 2024, summarized in $1). Participants in the free- form responses describe LLM tools with varying levels of autonomy and agency, ranging from direct manipulation ("just another tool in the loolbox") to data sources ("a cuslom Wikipedia page") to human-AT teaming ("a useful research collaborator or assistant," ") to fully autonomous agents ("an end-to-end Al researcher"), ) which points to a wide design space of future LLM- based research support tools and user interfaces (Lee et al., 2024a). As researchers across AI and HCI domains continue to devote resources to developing new tooling, we may see increasing benefits in adopting LLM-based research support tools and potential paradigm shift in the future of scientific work.
"A Game-changer Leveling the Field": Equity Benefits of LLMs Our survey reveals com- pelling insights into how LLMs reshape research equity across demographic groups, recurring theme of equity ($3.3, Appendix C) as a primary benefit of LLM use in research. Non-native English speakers described how LLMs allowed them to "level the playing field" by cutting down "tedious and time-consumiing editing tasks" to "more freely and precisely ex- press ideas in another language [English]." " For example, $4.2 demonstrates that traditionally underrepresented or disadvantaged groups in research (Linxen et al., (ezl)-spciffiallty non-white researchers, junior scholars, and non-native English speakers-nod only per- ceive greater benefits from LLMs but also report higher usage frequencies for certain tasks. Additionally, equity was mentioned in contexts such as enabling researchers without pro- gramming training to generate code for data cleaning, improving understanding of papers
In the free response, some participants used open source and non-profit interchangeably. Thus, for the sake of labeling, we created higher-level label of open source/ source/non-por pnon-- to capture that opinion.
8

---

Published as a conference paper at COLM 2025
that are technical from less familiar fields, or reducing monetary costs of proofreading services. This suggests that LLMs could help them overcome systemic barriers-including neo-colonial dynamics in research (Goyes and Skilbrei, 2023). To further build upon the eq- uity benefits, future work should examine the unique needs of traditionally under-resourced groups for development of LLM-based research tools; for instance, future tools could ed- ucate junior researchers about research skills or practice critical thinking (Ye et al., 2024), However, the equity benefits are under scrutiny: while LLMs supported non-native English speakers in overcoming language barriers, the distinct "ChatGPT style" can be used to infer author demographics, potentially to their detriment (Lepp and Smith, 2025).
Gender Gaps in LLM Usage and Ethical Concerns Despite some equity benefits, $4.2 indicates that women and non-binary participants expressed heightened ethical concerns regarding LLM use and demonstrated lower usage rates, though this difference did not reach statistical significance. This also extends beyond the research field: on public perceptions of AI, people who are nonbinary, transgender, and /or women reported significantly more negative AI attitudes compared to the majority groups, signaling that current Al tools fail to address their needs and concerns (Haimson et al., 2025). This pattern merits particular attention, given that these groups have historically faced disadvantages in academia. A concerning implication emerges: if ethical reservations lead these researchers to limit their LLM use, they might forfeit potential benefits and collaborative opportunities, potentially exacerbating existing inequities rather than alleviating them. The gender gap in LLM adoption has already been observed, where women and lower-earning workers were found less likely to use ChatGPT (Draxler et al., 2023; Humlum and Vestergaard, 2025). Future work should examine more deeply the specific ethical and other concerns expressed by these groups and strive to address their concerns in future LLM-based research tools. For instance, given prior research showing women scientists often struggle to receive appropriate credit for their work (Ni et al., 2021; Ross et al., 2022), their hesitancy about LLM use and attribution requires careful consideration.
The Costs of Commercial LLMs in Research Our survey reveals that while commercial models are sometimes perceived to offer higher quality results and better user support, they also raise concerns about transparency, reproducibílity, and more fundamentally, the misalignment of incentives between commercial entities and the research community. Many researchers have argued that the use of open-source models enhances the validity and integrity of research by allowing for greater scrutiny of research data and output (Sallou et al., 2024). These concerns should be carefully examined in high-stakes research areas such as medicine, bioengineering, and law, which can have direct, real-world impacts (Toma et al., 2023). In these fields, reproducibility and transparency are paramount, as they ensure the reliability of findings and provide justification for decisions that affect people's 's lives.
Emerging Standards Our survey reveals concerns about the risks associated with LLM use in research, such as deskilling, decreasing creativity, and decreasing diligence ($3.3, Appendix C), which might result in the proliferation of low-quality research (Bail, 2024). Interestingly, our survey shows that while disadvantaged groups are more likely to discuss LLM benefits, perceptions of risks appear to be more uniformly distributed across demo- graphics with few significant differences. For example, the problem of hallucinations in LLM outputs appears to be an equally significant concern for all researchers. This shared understanding indicates a collective awareness of LLMs' limitations and suggests the pos- sibility of developing uniform standards for LLM use that can be broadly agreed upon, irrespective of people's demographic characteristics. The academic community has begun exploring various mitigation strategies such as a peer-reviewers checklist (Watkins, 2023) and transparent disclosure of LLM use (Hosseini et al., 2023). Many conferences, publishers, and funding agencies have started to require LLM disclosure statements (ACL, 2024; ACM, 2024; AAAI, 2024; IEEE, 2024; 202, 2023; of Health, 2024; Kwong et al., 2024).
Our survey reveals that participants were broadly comfortable with disclosing LLM usage to both peers and reviewers across all tasks, though these varied by discipline, with computer scientists reporting significantly higher comfort compared to researchers in social sciences & humanities biology & medicine. This disparity reflects that while some disciplines may

---

Published as a conference paper at COLM 2025
accept LLMs standard research tools, others are still grappling with integrating these technologies into their established practices. The mention of "academic shame" by one respon- dent highlights the ongoing stigmatization of LLM use in some research communities. The lack of standardized policies across institutions further complicates this landscape, creating additional burdens for individual researchers who must navigate varying expectations and norms. This suggests that different academic communities are still negotiating their norms of acceptable LLM use, and continued discussion is crucial to fully leverage LLMs towards improving science.
5.1 Limitations
While our recruitment method covers a wide range of fields, more of our participants were computer scientists (40%), though other fields such as social sciences, biology, medicine, and natural sciences were also represented. There were also more men who responded to the survey (79%), which may partly reflect existing imbalances. The detailed distributions are reported in $4 and Appendix A.3. Our data also includes: (1) a high proportion of more senior white researchers; (2) more researchers who identified as men in the natural science & engineering field, and (3) fewer senior researchers in the computer science field. These could be the result of sampling bias or existing imbalances in these respective fields.
The survey responses were collected in batches of recruitment emails over a six-month period from November 2023 to April 2024. The uses and perceptions of researchers may change as LLM tools continue to evolve, but we hope this survey can give readers a snapshot of the current state of the community and a baseline data point in time for comparisons done by future studies, together supporting informed decisions as we continue to build consensus and norms around the use of LLMs for research.
Our quantitative analysis assumes that the 5-point rating data follow an unimodal distri- bution, which supports the use of parametric statistical methods. While this approach is acceptable (Harpe, 2015), we acknowledge that multimodal distributions could challenge the validity of these methods.
Finally, we discuss the limitations of our regression method. We chose to fit separate regression models for each demographic variable-one model to analyze race on per- ception and a separate model to analyze gender on perception-as way to improve statistical power and interpretability of results. Fitting a more complex model like Ratingg ~ Race; * Gender; Year;
* UsageType,j would lack degrees of freedom imped- ing statistical testing, well as result in larger regression model with over 500 estimated coefficients. Such models would be difficult to interpret and require making additional assumptions about the modeled relationships between demographic variables (e.g., are race and gender linearly related in their effects on perception). Our choice to fit individual models for each demographic variable eschews these problems, though does not capture interactions between pairs of demographic variables. We provide tests of independence between demographic pairs in Appendix Table 2, where we observe correlations between three pairs (Gender & Field, Race & Year, Year & Field).
Conclusion
In this study, we ran a large-scale survey of diverse groups of researchers about their usage and perception of LLMs for research, and revealed the widespread adoption of LLMs and distinct perceptions based on academic disciplines and demographics. Our work suggests that the research community is critical juncture, balancing growing LLM integration with the need to uphold originality, rigor, and ethical conduct as well as the potential for a more productive and inclusive global research landscape. It also underscores the need to better understand the implications of LLM usage-net not just technical integration but also sociological, ethical, and epistemological impacts across disciplines and researcher demographics. We call for studies that examine long-term effects of LLM use research quality, creativity, and the development of research skills as well as investigations into the potential of LLMs to increase fairness and representation in academia.
10

---

Published as a conference paper at COLM 2025
7 Ethics Statement
We recruited from verified authors listed on Semantic Scholar that has dedicated data qual- ity team maintaining academic author profiles. We could have tied the survey responses to participants' author metadata from Semantic Scholar to obtain high-precision demographic information (such as a list of publications, years of experiences, institutions, pronouns, etc.). However, for privacy concerns, we only used their email addresses for targeted recruitment of verified published authors. We instead relied on relorrigg using optional survey questions for demographic information, and did not tie survey responses to their author metadata.
We surveyed participants about their racial ethnic identity. Some participants who chose to self-describe their identities highlighted limitations in the provided options, such as including only one generalized "Asian" category, while others chose not to respond because they found the options to be U.S.-centric (see all survey questions in the supplementary material). We acknowledge these limitations in the design of this question and note the diversity of our participants, with 24% self-reporting U.S., the most common country of origin among respondents (see Appendix A.3.2 about other top countries).
The authors would like to thank our many anonymous survey respondents who provided long and thoughtful opinions and insights in the optional free-text survey questions. These responses enabled our qualitative analysis and showcased that many researchers are actively contemplating and engaging conversations around the use of LLMs as research support tool today.
References
2023. Writing the rules Al-assisted writing. Nature Machine Intelligence 5 (2023), 469 469. https://api. semanticscholar org/CorpusID::55846653
AAAI. 2024. Policy on use of AI systems in producing AAAI or AAAI-related publi- cations. Retrieved September 6, 2024 from https://aaal. org/ aaai ppbiiippbiiiiiiiioss piaal-ppbliaatioopoliicce wttpppooiiiiiiiiiieeeeieeeeeeeee guidelines/
ACL. 2024. ACL 2023 Policy on AI Writing Assistance. Retrieved September 6, 2024 from https://2823.1 aclweb. att.ee.orggbllg//A......o /ttpbboogAAA--202--p000/
ACM. 2024. ACM Policy on Authorship. Retrieved September 6, 2024 from https:
//www. acm. org /ttpo//ccctoosoooolccce//eeeeeer
Hussam Alkaissi and Samy 1 McFarlane. 2023. Artificial hallucinations in ChatGPT: implica- tions in scientific writing. Cureus 15, 2 (2023).
Amanda Alvarez, Aylin Caliskan, MJ Crockett, Shirley S Ho, Lisa Messeri, and Jevin West. 2024. Science communication with generative AI. Nature Human Behaviour 8, 4 (2024), 625-627.
Maria Antoniak, Aakanksha Naik, Carla S. Alvarado, Lucy Lu Wang, and Irene Y. Chen. 2024. NLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age LLMs. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAceT 24). Association for Computing Machinery, New York, NY, USA, 1446-1463. https://doi.. orgoo//////// 10. 1145/3630106. 3115836331000666688222
Christopher A. Bail. 2024. Can Generative Al improve social science? Proceedings of the National Academy of Sciences 121, 21 (2024), e2314021121.
Richard E. Boyatzis. 1998. Transforming Qualitative Information: Thematic Analysis and Code Development.
Davide Castelvecchi. 2024a. Researchers built an 'AI Scientist' - what can do? Nature
News (2024). /tdos://doi. org/10. 1038/d41586 020-58666--2200000000 -22--22-2------33333333
11

---

Published as a conference paper at COLM 2025
Davide Castelvecchi. 2024b. Researchers built an' AI Scientisttwwha can it do? Nature (2024).
Eric Chamoun, Michael Schlichtkrull, and Andreas Vlachos. 2024. Automated Focused Feedback Generation for Scientific Writing Assistance. In Findings of the Association for Computational Linguisfics ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand and virtual meeting, 9742-9763. https:/ aclanthololl .wweaa.......ooooooeeeee orgoorg///// 2024. findings-acl, aangs-aac..88 ac1
Lynne M. Connelly. 2013. Grounded theory. Medsurg nursing: official journal of the Academy of Medical-Surgical Nurses 22 2 (2013), 124, 127.
Mike D' D'Arcy, Tom Hope, Larry Birnbaum, and Doug Downey, 2024. Marg: Multi-agent
review generation for scientific papers. arXiv preprint ar 4rXio:2401.04259 (2024).
Fiona Draxler, Daniel Buschek, Mikke Tavast, Perttu Hämäläinen, Albrecht Schmidt, Juhi Kulshrestha, and Robin Welsch. 2023. Gender, Age, and Technology Education Influence the Adoption and Appropriation of LLMs. arXiv:2310.06556 [cs.CY] abs/2310. 06556
Benedikt Fecher, Marcel Hebing, Melissa Laufer, Jörg Pohle, and Fabian Sofsky. 2023. Friend or Foe? Exploring the Implications of Large Language Models on the Science System. ArXiv 21//2220600999 (2023). https: hops://appt /api. semanti cescmnntcscchlarr..... org/CorpusID: 259187447
Raymond Fok, Joseph Chee Chang, Tal August, Amy X Zhang, and Daniel S Weld. 2023. Qlarify: Bridging scholarly abstracts and papers with recursively expandable summaries. aiv preprint arXiv:2310.07581 (2023).
Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for science writing using language models. In Proceedings of the 2022 ACM Designing Interactive Systems Conference. 1002-1019.
David Rodríguez Goyes and May-Len Skilbrei. 2023. Rich scholar, poor scholar: inequalities in research capacity, "knowledge" abysses, and the value of unconventional approaches to research. Crime, Law and Social Change (2023). https://aph: semanticscholar org// CorpusID:259444322
Dritjon Gruda. 2024. Three ways ChatGPT helps me in my academic writing. Nature (April 2024). https://doi. org/10. 1038/14158666 15550/44411--020-14--44 Bandiera abtest: Cg type: Ca- reer Column publisher: Nature Publishing Group Subject term: Careers, Machine learn- ing, Authorship, Peer review, Publishing.
Oliver L. Haimson, Samuel Reiji Mayworm, Alexis Shore Ingber, and Nazanin Andalibi. 2025. AI Attitudes Among Marginalized Populations the U.S.: Nonbinary, Transgender, and Disabled Individuals Report More Negative AI Attitudes. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT '25). Association for Computing Machinery, New York, NY, USA, 1224-1237. https://doi. 1145/ 3715275.37322
Spencer F.. Harpe. 2015. How to analyze Likert and other rating scale data. Currents in Pharmacy Teaching and Learning 7, 6 (2015), 836-850. https://don org/10. 1016/ cptt. 2015. 08.001
Will Douglas Heaven. 2022.
Why Meta's latest large language
model survived only three days online. (Nov. 2022). -
MIT Technology Review
httpp:://wet technologyreeieew.... wtt.c/oooo//c./// http:///www.eeee//////////////// www.rreewwuuuuueriiaaaaaaalaaaie
Mohammad Hosseini, David B Resnik, and Kristi Holmes. 2023. The ethics of disclosing the
use of artificial intelligence tools writing scholarly manuscripts. Research Ethics 19,
(Oct. 2023), 449-465 https://doi.org:: /ttp://...oog//...117/////111111 1177/17478166123111888 188444
12

---

Published as a conference paper at COLM 2025
Chao-Chun Hsu, Erin Bransom, Jenna Sparks, Bailey Kuehl, Chenhao Tan, David Wadden, Lucy Lu Wang, and Aakanksha Naik. 2024. CHIME: LLM-Assisted Hierarchical Orga- nization of Scientific Studies for Literature Review Support. arXiv:2407.6114 [cs.CL]
kttps://arxivv http://aaxxvv..o//bbbba wrgg.bs//24//) 2abb//447.11144
Anders Humlum and Emilie Vestergaard. 2025. The unequal adoption of ChatGPT exacer- bates existing inequalities among workers. Proceedings of the National Academy of Sciences 122, (2025), 4241497
JEEE. 2024, IEEE Guidelines for Generative Al Usage. Retrieved September 6, 2024 from https: /tops://www./ ieee- ras. wtt.eraa.....lccaaaaaaaeeaeaaaaa http:/bbl.aatiioo/ggieeeeel.eeee for-iior----eeeeee-ee Sero--eerraatve-aae aisugge
Maurice Jakesch, Zana Buçinca, Saleema Amershi, and Alexandra Olteanu. 2022. How Different Groups Prioritize Ethical Values for Responsible Al. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT 22). Association for Computing Machinery, New York, NY, USA, 310-323. //doi. org/ 10. 145/3531 46. 3533097
https:
Hyeonsu B Kang, Tongshuang Wu, Joseph Chee Chang, and Aniket Kittur. 2023. Synergi: A Mixed-Initiative System for Scholarly Synthesis and Sensemaking. In Proceedings of the 36th Anmual ACM Symposium on User Interface Software and Technology. 1-19.
Shivani Kapania, Ruiyi Wang, Toby Jia-Jun Li, Tianshi Li, and Hong Shen. 2024. I'm categorizing LLM as productivity tool": Examining ethics of LLM use in HCI research practices. arviv preprint ar XrXi:2443..1877 (2024).
Luoma Ke, Song Tong, Peng Cheng, and Kaiping Peng. 2024. Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Review. ArXiv abs/ /0////2010011111 .11519 (2024). https://api: semanticscholee ottp://aaaatt.ttoooooooooooooooa
Jeongyeon Kim, Sangho Suh, Lydia B Chilton, and Haijun Xia. 2023. Metaphorian: leverag- ing large language models to support extended metaphor creation for science writing. In Proceedings of the 2023 ACM Designing Interactive Systems Conference. 115-135.
Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the SIGCHI conference on human factors in computing systems. 453-456.
Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work. 1301-1318.
Dmitry Kobak, Rita González-Márquezz Emōke-Agnes Horvát, and Jan Lause. 2024. Delving into ChatGPT usage in academic writing through excess vocabulary. arXiv:2406.0001 07001 (July 2024). https://doi... org/10 48550/ 48550/arXiv. 2406 07016 arXiv:2406.07016 [cs].
Charlotte Kobiella, Yarhy Said Flores López, Franz Waltenberger, Fiona Draxler, and Al- brecht Schmidt. 2024. "If the Machine Is As Good As Me, Then What Use Am I?" How the Use of ChatGPT Changes Young Professionals' Perception of Productivity and Accomplishment. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI 24)) Association for Computing Machinery, New York, NY, USA, 1-16.
https://doi. org/10. 10. 1145/3615311 3904. 3641964
Daphne Koller, Andrew Beam, Arjun Manrai, Euan Ashley, Xiaoxuan Liu, Judy Gichoya, Chris Holmes, James Zou, Noa Dagan, Tien Y Wong, al. 2023. Why we support and encourage the use of large language models in NEJM AI submissions. Ale2300128 pages.
Jethro C. C. Kwong, Serena C. Y. Wang, Grace Nickel, Giovanni E. Cacciamani, and Joseph C. Kvedar. 2024. The long but necessary road to responsible use of large language models in healthcare research. NPJ Digital Medicine 7 (2024). https:/ /api. semanticscholas....... .org/ CorpusID:27097250
13

---

Published as a conference paper at COLM 2025
Edith Law, Krzysztof Z Gajos, Andrea Wiggins, Mary L Gray, and Alex Williams. 2017.
Crowdsourcing as a tool for research: Implications of uncertainty, In Proceedings of the 2017 ACM conference computer supported cooperative work and social computing. 1544-1561.
Mina Lee, Katy Ilonka Gero, John Joon Young Chung, Simon Buckingham Shum, Vipul Raheja, Hua Shen, Subhashini Venugopalan, Thiemo Wambsganss, David Zhou, Emad A. Alghamdi, Tal August, Avinash Bhat, Madiha Zahrah Choksi, Senjuti Dutta, Jin L.C. Guo, Md Naimul Hoque, Yewon Kim, Simon Knight, Seyed Parsa Neshaei, Antonette Shibani, Disha Shrivastava, Lila Shroff, Agnia Sergeyuk, Jessi Stark, Sarah Sterman, Sitong Wang, Antoine Bosselut, Daniel Buschek, Joseph Chee Chang, Sherol Chen, Max Kreminski, Joonsuk Park, Roy Pea, Bugenia Ha Rim Rho, Zejiang Shen, and Pao Siangliulue. 2024a. A Design Space for Intelligent and Interactive Writing Assistants. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '4). Association for Computing Machinery, New York, NY, USA, Article 1054, 35 pages. https://doi. org/ 10. 1145/3611111 3904. 3642697
Yoonjoo Lee, Hyeonsu B Kang, Matt Latzke, Juho Kim, Jonathan Bragg, Joseph Chee Chang, and Pao Siangliulue. 2024b. PaperWeaver: Enriching Topical Paper Alerts by Contextualizing Recommended Papers with User-collected Papers. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-19.
Haley Lepp and Daniel Scott Smith. 2025. "You Cannot Sound Like GPT": Signs of language discrimination and resistance in computer science publishing. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT 25). Association for Computing Machinery, New York, NY, USA, 3162-3181. http:://doi.org//// 1145/ 3715275. 30..373332 3732202
Jenny T. Liang, Chenyang Yang, and Brad A. Myers. 2024a. A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE '24). Association for Computing Machinery, New York, NY, USA, Article 52, 13 pages. https://doi.org// 10. /145/3597503. 3608128
Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, Diyi Yang, Christopher Potts, Christo- pher D. Manning, and James Y. Zou. 2024b. Mapping the Increasing Use of LLMs in Scien- tific Papers. ArXiv abs/ 2005/24000126 (2024). https://apt semanticscholar www.atttisccooooooooooooooo org/CCrpusID: 268857142
Sebastian Linxen, Christian Sturm, Florian Brühlmann, Vincent Cassau, Klaus Opwis, and Katharina Reinecke. 2021. How weird is CHI?. In Proczedings of the 2021 chi conference on human factors in computing systems. 1-14.
Yiren Liu, Si Chen, Haocong Cheng, Mengxia Yu, Xiao Ran, Andrew Mo, Yiliu Tang, and Yun Huang. 2024. How AI Processing Delays Foster Creativity: Exploring Research Question Co-Creation with an LLM-based Agent. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '4)). Association for Computing Machinery, New York, NY, USA, Article 17, 25 pages. https: org/10. 1145/361 3904. 3642698
Yihe Liu, Anushk Mittal, Diyi Yang, and Amy Bruckman. 2022. Will Al console when I lose my pet? Understanding perceptions of Al-mediated email writing. In Proceedings of the 2022 CHI conference on human factors in computing systems. 1-13.
Kyle Lo, Joseph Chee Chang, Andrew Head, Jonathan Bragg, Amy X Zhang, Cassidy Trier, Chloe Anastasiades, Tal August, Russell Authur, Danielle Bragg, et al. 2023. The semantic reader project: Augmenting scholarly documents through ai-powered interactive reading interfaces. ar Xiv preprint ar Xio:2333. 143344 (2023).
Tao Long, Dorothy Zhang, Grace Li, Batool Taraif, Samia Menon, Kynnedy Simone Smith, Sitong Wang, Katy llonka Gero, and Lydia B Chilton. 2023. Tweetorial hooks: generative AT tools to motivate science social media. Xiv preprint arXiv:2305.12 12265 (2023).
14

---

Published as a conference paper at COLM 2025
Lisa Messeri and MJ Crockett. 2024. Artificial intelligence and illusions of understanding in scientific research. Nature 627, 8002 (2024), 49-58.
Meredith Ringel Morris. 2023. Scientists' Perspectives on the Potential for Generative Al in their Fields. ar Xiv preprint ar Xxx::2334..01220 (2023).
Chaoqun Ni, Elise Smith, Haimiao Yuan, Vincent Larivière, and Cassidy R Sugimoto. 2021. The gendered nature authorship. Science advances 7, 36 (2021), eabe4639.
National Institutes of Health. 2024. The Use of Generative Artificial Intelligence Technologies is Prohibited for the NIH Peer Review Process. Retrieved September 11, 2024 from https://grants nih.
wwtgggattgnndeenntttteeoeeeo -ool///oooi---iee///000000000000 149. html
Matthew B Ross, Britta M Glennon, Raviv Murciano-Goroff, Enrico G Berkes, Bruce A Weinberg, and Julia Lane. 2022. Women are credited less in science than men. Nature 608, 7921 (2022), 135-145.
Giuseppe Russo, Manoel Horta Ribeiro, Tim R. Davidson, Veniamin Veselovsky, and Robert West. 2024. The Al Review Lottery: Widespread Al-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates. ArXiv abs/2405.02150 (2024). https:
seppiissemmticschhollloo sot.maatttccccoolaaroooorrrrooss Dtt//CorpuSDD:26588777
June Sallou, Thomas Durieux, and Annibale Panichella. 2024. Breaking the Silence: the
Threats of Using LLMs in Software Engineering. In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE- NIER'24), Association for Computing Machinery, New York, NY, USA, 102-106.
https:
//doi.org/1e 10e.o..11553339990 33397776663666644
FutureHouse. Sam Rodriques. 2024.
FutureHouse Announcing FutureHouse. Retrieved September 9, 2024 from https://www. http:://ww..utuurehhuuss.
Jan-Philipp Stein, Tanja Messingschlager, Timo Gnambs, Fabian Hutmacher, and Markus Appel. 2024. Attitudes towards AI: measurement and associations with personality. Scientific Reports 14 (2024). https://aph semanticscholar org. 2 267498881
Lu Sun, Stone Tao, Junjie Hu, and Steven P Dow. 2024. MetaWriter: and Perils of AI Writing Support in Scientific Peer Review. Proceedings of the ACM on Human-Compuler Interaction 8, CSCW1 (2024), 1-32.
Exploring the Potential
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A Large Language Model for Science. arXiv:2211.00 1..90000088 (Nov. 2022). https://dol http://oooorg/////// org//0.48550/ arXiv.2211 221.229850905 arXiv:2211.09085 [cs, stat].
Augustin Toma, Senthujan Senkaiahliyan, Patrick R. Lawler, Barry Rubin, and Bo Wang. 2023. Generative AT could revolutionize health care - but not if control is ceded to big tech. Nature 624, 7990 (Dec. 2023), 36-38. https://doi.... org/10. mondiera_abtestt a Cg_type: Comment publisher: Nature Publishing Group Subject term: Machine learning, Health care, Medical research, Technology.
0//////11166263333 003300300-y
Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024. SciMON: Scientific Inspiration Machines Optimized for Novelty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 279-299. https://aclanthhllog: or 202./ acl -ong. 18
Skyler Wang, Ned Cooper, Margaret Fby, and Eun Seo Jo. 2023. From human-centered to social-centered artificial intelligence: Assessing ChatGPT's impact through disruptive events. arXiv preprint ar Xiv:23006 0022700002222000 (2023).
Ryan Watkins. 2023. Guidance for researchers and peer-reviewers the ethical use of Large Language Models (LLMs) in scientific research workflows. (May 2023). https:
//doi.org//i 1007 000-36881-2230-9-----
15

---

Published as a conference paper at COLM 2025
Nigel Williams, Stanislav Ivanov, and Dimitrios Buhalis. 2023. Algorithmic Ghost in the Re- search Shell: Large Language Models and Academic Knowledge Creation in Management Research. ArXiv abs//303.07304 (2023). https://api. semanticscholar.org//oooo
257496621
Dirk U. Wulff, Zak Hussain, and Rui Mata. 2024. The Behavioral and Social Sciences Need Open LLMs. OSF (Sept. 2024). https://dol org/ii...00100 31219/ost io//bvzs
Andre Ye, Jared Moore, Rose Novick, and Amy X. Zhang. 2024. Language Models as Critical Thinking Tools: A Case Study of Philosophers. Conference on Language Models (COLM) (2024).
16

---

Published as a conference paper at COLM 2025
A Methods
A.1
Survey Design
When designing the questionnaire in the survey, we used the four following approaches: First, for inspiration, we looked to recent literature on using LLMs productivity tool for research (Morris, 2023; Messeri and Crockett, 2024; Bail, 2024; Russo et al., 2024; Wang et al., 2023) and other scenarios (Liang et al., 2024a; Liu et al., 2022), which included qualitative interviews and survey results. Second, we reviewed historic papers on how the research community had adopted new tools the past, specifically around the use of crowdsoureing for data collection, user studies, and other productivity tasks (Law et al., 2017; Kittur et al., 2008; 2013). Third, we publicized an anonymous formative survey on X/ X/Twitter targeted towards researchers with open-ended questions about whether and how they use LLMs for research order to help define initial categories of usage that we later refined. Finally, we shared early drafts of the questionnaire with other researchers in own institutions for feedback and iteration. Through this process, we classified LLM usage for research into six broad categories: information seeking, editing, ideation & framing, directing writing, data cleaning & analysis, and data generation.
Demographic Groups
Lang
Gender 00
Field Study
Experience
Likert Ratings
Info- Sewking
Benefits?
Ediding
Benefits?
Ethios?
Ethics?
Disclose
Dieclore
Gisckcee (Peer))
nieddos
[Peors??
[Berisoo
(Aaciuamer))
Ideeation
Usage?
Writing
Usage?
Riske?
Ethice?
Riske?
Echics?
Diectose
Disclose
Diecioen
Disckose
[Peers]?
[Reviewer]?
(Peen0?
(Reviewe??
Data Cleaning
Usage?
Benefits?
Uaagge?
Risks?
Riske?
Ethics?
Disclose
Datra
Disciose
Disckose
(Poors??
(Reviewer)?
(Dewn)?
(Reviewe/?
Figure 5: An example of data collected per participant from sections of the survey that relate to our statistical modeling. Each participant provides answers to (up to) five demographic questions and (up to) 36 Likert ratings in response to questions about LLM usage frequency, perceptions, and usage types. Participants are not required to report on every question. In this example, the gender information is missing from the participant.
A.2 Recruitment
recruitment email was sent to 100,187 verified authors, using a Semantic Scholar email list compiled over the past decade, although the email addresses were not guaranteed to be active. Out of all sent emails, 34,922 emails were opened (34.9% open rate) and 1,834 emails were clicked (1.8% click-through rate). After click-through, 1,226 participants signed the consent form to start the survey, and thus the survey response rate lies between 1.2% to 3.5%, depending on what proportion of our targeted email addresses were no longer active, Finally, after filtering out survey results that did not progress past the first page or spent fewer than seconds on each question, we ended up with 816 total responses.
A.3 Participants Demographics and Quantitative Method
We transformed the dataset to generate the final demographic groups in which some of the response options were grouped to form coarser buckets (e.g., years of research experience),
17

---

Published as a conference paper at COLM 2025
and free responses (e.g., fields of study) were manually coded and discretized for analysis. Answers that did not fit into any categories, such as "Prefer Not to Answer," " were filtered out from demographic-speciff analysis. We ended up with 611 responses with gender identity, 527 with racial identity, 644 with years of research experience and native language information, and 635 with field of study information.
Man
Woman, Non-
Binary, Other
0-3
4-10
11+
White
243
74 34
White
27
96
198
Non-White
168
Non-White
31
81
94
p 0.7578
p 0.0101
(a) Race and Gender
(b) Race and Years Experience
CS
Bio
Nat.Sci
Soc.Sci
CS
Bio
Nat.Sci
Soc.Sci
Man
194
65
113
110
0-3
37
11
12
Woman, Non-
4-10
102
30
39
35
48
12
40
Binary, Other
11+
118
82
105
p
0.0008
p == 0.0030
(c) Gender and Years Experience
(d) Years of Experience and Field of Study
Table Contingency tables of participant counts for different demographic pairs. p- values from Chi-square tests of independence; lower values are interpreted as evidence for dependence between variables.
A.3.1 Consolidating Demographics Groups
To balance our analysis given large proportion of men participants, we collapsed all responses from women, non-binary, and other participants into a single category. Similarly, we collapsed participants identified as Asian, Black, Hispanic or Latino, and Middle Eastern into a single category of non-white because of large proportion of white participants. See detailed racial categories below. Field of study was collected as free response and manually classified into four categories by the authors. We collected the research fields that our participants studied in the form of free responses. Out of the 816 responses, 644 (79%) responses included field information. We classified 635 free responses into four field categories: computer science, social science & humanities, natural science & engineering, and biology& & medicine, and 9 responses were classified other and excluded from the analysis. Computer science group had 257 participants and also included interdisciplinary fields with computer science, such education technology, except biotechnology. Social science & humanities had 152 participants and included psychology, behavioral science, education, sociology, and more. Natural science & engineering had 132 participants, and included math, chemistry, physics, environmental science, electrical engineering, and more. Biology & medicine had 94 responses, and included cognitive science, bioinformatics, biotechnology, public health, neuroscience, and more. We included interdisciplinary fields between biology and computer science in the biology & medicine group instead of the computer science group to balance the number of participants in each field.
After consolidating the demographic groups, we inspected the correlation between certain demographic groups: a series Chi-square tests of independence (chisq. test) between all pairs of the five demographic groups, with multiple comparisons p-value correction using Holm-Bonferne (p. adjust). Through a series of Chi-square tests and Hool--onnferoon correction, we found that most variables appear independent, except for three pairs with significant p-values: race and years experience, gender and field of study, and years of experience and field of study.
18

---

Published as a conference paper at COLM 2025
Gender
Race
Year
Publication
Field
Gender
0.7578
1.0000
0.1433
0.0008
Race
0.0101
1.0000
1.0000
Year
1.0000
0.0030
Language
/
1.0000
Field
Table 2: p-values from Chisquare tests of independence between demographic groups. All p-values were corrected via Foolm-Bonferron.. Significant values (p < 0.05) are in bold and indicate dependence between pairs of variables. We previously also had another variable representing researcher experience nurbber of publications-bae but found that it was highly correlated with Years of research experience (Chisq Independence test; p-value = 3.3E-15).
A.3.2 Participants' Country of Origin
Country
Percentage
Number of Participants
United States America
23.80%
154
Germany
6.34%
41
Canada
4.33%
28
Italy
4.17%
27
India
4.17%
27
United Kingdom of Great Britain
3.25%
21
and Northern Ireland
Spain
2.47%
16
China
2.47%
16
Australia
2.47%
France
2.32%
15
Nigeria
2.16%
Brazil
2.01%
13
Sweden
1.85%
12
Russian Federation
1.85%
12
Netherlands
1.85%
12
Table 3: Top 15 countries of origin reported by participants
A.3.3 Participants' Racial Identities
Among all participants who provided their racial identities, 51.4% identified as white Caucasian, 17.8% identified as Asian, 3.6% identified as Black or African Amer- ican, 8.1% identified Hispanic or Latino, 2% identified as Middle Eastern. Additionally, 8.9% selected Prefer not disclose and 8.1% selected Prefer not to disclose, which were excluded from the demographic analysis.
A.4 Quantitative Methods
In practice, addition to the regression model described in the main text, we always attempt second model fit that includes an interaction lerm-fyDemme mymm--fgDmmrraphicc x UsageType.j. We conduct likelihood ratio test using anova() in R between each model against null model which has no demographic variable (i.e., the null hypothesis of no demographic effects) and choose the interaction model if the interaction term is statistically significant; otherwise, we defer to the simpler model without an interaction term. In our analysis, we find that rarely is the interaction term needed.
19

---

Published as a conference paper at COLM 2025
Asian 17.8%
44
Black
3.5%
White / Caucasian
127
20
Hispanic 8.1%
2.0%
22
Prefer not disclose
20
Prefer to self dest-desrrbe
Figure 6: Among all participants who provided their racial identities, 51.4% identified as white/ white/Caucasian 17.8% identified as Asian, 3.6% identified as Black or African American, 8.1% identified as Hispanic Latino, 2% identified as Middle Eastern.
B Quantitative Results
Gender
Race
Language
Years
Field
All
RQ2 Frequency
3666
3162
3864
3864
3810
4896
RQ5.1 Risk
2973
2597
3110
3110
3066
3410
RQ5.2 Benefits
2315
2016
2440
2440
2403
2738
RQ5 3 Ethics
3191
2744
3354
3354
3314
3665
RQ5,4 4 Disclosure to Peers
3061
2631
3207
3207
3168
3493
RQ5.4 Disclosure Reviewers
3028
2637
3177
3177
3139
3450
Table 4: Number of answers to survey questions across all participants, broken out by demographic and question. Each question (row) had six sub-questions; participants did not have to answer all questions.
Type of Usage
Risk
Benefit
Ethics
Disclosure to Peers
Disclosure to Reviewers
Information Seeking
-0.303
0.513
0.261
0.127 (0.0005)
0.078 (0.041)
Editing
-0.276
0.557
0.282
0.171
0.073 (0.041)
Ideation & Framing
-0.372
0.651
0.385
0.197
0.177
Direct Writing
-0.401
0.575
0.409
0.184
0.136 (0.0004)
Data Cleaning Analysis
-0.437
0.68
0.335
0.229
0.186
Data Generation
-0.45
0.665
0.383
0.262
0.245
Overall Usage
-0.401
0.62
0.389
0.232
0.183
Table 5: Kendall's tau correlation between the frequency of LLM usage and the perception of that usage. Each cell includes the tau coefficient with the Holm-Bonferroni corrected p-value in parenthesis. Coefficients not followed by parenthesis all had p0.000..
20

---

Published as a conference paper at COLM 2025
Man-(Woman, NonBin., Oth.)
NonWhite- White
11+ yrs- 0-3 YTS
P
P
Benefit (1-5) Ethics (1-5)
0.420
.0001
-0.4187
0.0004
0.351
0.0017
Bio-CS
p
CS-Soc.Sci
P
Pesrs (1-5) Disclos.Reviewer. (1-5)
-0.644
0,0001
0,513
0,0002
-0.622
0.0007
0.451
0.0043
Table 6: Post-hoc tests for significant pairwise differences in LLM perception ratings between demographic groups across all LLM usage types. This table reports the rating differences between demographic levels and associated p-values from emmeans. For example, in the column for race and the row for Benefit, the result shows that Non-White researchers, average, report 0.42 points higher ratings than White researchers on perceived benefits of LLM usage. Only statistically significant results are included.
C Qualitative Themes
Theme
Description
Example
Reproducibility
Whether research is repro-
"Berause reszarch should be reproducible, and this is only possible with the use of open-source LLMs."
haabbl//rpplcaale/eeeee veeiifiblle by other researchers
...an open-source modelidata could be used and verified by external researchers and, hence, more trustworthy." "
Transparency
Whether the model release is trans- parent, including code, training dataset, evaluation dataset, etc.
"Gratter transporency on the models and training data would increase my conft- dence in the mudels' accurncy."
As reserchers, not just sufficient use the service as black box. would bike to know more about how the models were trained and what data went into it. It builds and expands the knowledge."
Availability
Whether the model is widely avail- able, released to public, and has no/low barriers for researchers cess
" "orld don't 't have similar access to Al that night result in systemic discrimina- tion."
Lising for-profit solutions produces costs that require funding, but open- options are typicully less 'randy--tu-sed use and often require schup and potentially also Incal/arstilbble computing resourres.
Accountability
Who should be responsi- ble/ accountablee for the model and its use
"I don't think makes difference whether I'm using as tech or ChatGPT, it's still my responsibility to dilgently cleck the outputs and the responsibliity any risks is on me us the user."
paid, close-source product could be less reliable, but there somoone to sue if things awry."
Privacy
Whether data is kept private and stored securely
"Open "Open-saurce saurce entities are more reliable, and transparent. The risk of using capyy righted data less. The risk of stenling my personal sensitive data less."
"I trust open-source software more than yrrprittry/commrccil corp if have to hanille sesstta/senstivee duta."
Incentives
What kinds of incentives drive the cre- ation release model
"LLMs are fundamentally dependent on using people's actual creative and artistic withont thiir consent. The motives
I.I crettor have no effect that. Motives similarly have no effect LLMs' woreliabiltt. Motioes do not change whether M. are acconnbable for their mistakes, brcause 1J Ms comnot he accountabie."
"I can't trust the objectives and implementations of closed commercial corpora- tine." "
Neutrality
Whether tonls are considered neutral who created /owns the tool does not matter
"The tecloolooy is the same (whether it's provided by commerccial an open source entity), so my perceptions are same.""
"I don't know enough about what's going for not) 'bekindd the scenes LLMs regardless of where they'm from my percoption of thene vemains that they'r 'e unethical and useful."
Ethics & Bias
Whether the model is biased or uses unethical methods and data
"..a for profit company night produce models which are facouring certain kind of upinion, line of thought, or products. These might equiculently good/bod non profit model, but poses this inferent biss by whometer is curventlyy financially invested into the entity"
"The fuoddemental issues of bins, fallaacination, ethical issuss of invention, the nond for reniew, etc, will not change whether the LLM is open-source or closed."
Quality
Whether the model has good perfor- mance (outputs are useful and of high
'd rather use open-source LLMs, but acknotoladge that their performance still behind commerrcial cial models which makes it diffcull to use them.
"
quality)
would concerned the commercial aspects would affect the results served. Perhops we would be directed more to poid sources."
Table 7: Themes found in Q64: Would your perceptions of the benefits and risks of using LLMs be affected by whether the LLM is part of an open-source or non-profit entity versus a commercial corporation? Why why not?
21

---

Published as a conference paper at COLM 2025
Theme
Description
Example
Language Equity
Bridging language gaps support non-native English speakers
"I am native Engglish speaker, L. fuip with the language harrier. "Because I'm not Native American, I've received a number of negative comments from reviewers, and always wondered how if I'm not onz. Today, with the LLM tools, I can understand the terms I use that aren't 't Native American, and I'm able improve my writing" " "Fur reseorhers wnthout progrummingg skills, LLMs allow data awalyets without programming. What they need to lourn is how make gond natural language prompts." "
write a Natioe Americane
Other Equity
Removing barriers for researchers (not language), such as support- ing neuro-divergent researchers, searchers with limited resources, or researchers without programming ex- perience
"mpproe understandability for non-spacialists"
"For honest researchers in resource rescurce-nnstraainn constraiind deweloping countrrees, with little research funding, auailability and use of LLMs is gome-changer leveling the playing field with other rescarchers in more fortunate climes.
Efficiency
Saving time and resources during the research process
" Exthrriing work efficreny and the cupubiitt of information gathering and ex- traction, thereby anafding researcurss focus more ouergy on craative tasks." "Spedding up the academic writing/vading will speed up the research cycle of the community."
Routine Task Assistance
Completing routine or repetitive lasks, freeing the researcher to fecus on higher level tasks
think the main benefit could be saving time that would otherwiie to spent on relatioely 'mechamionl" tasks, such as literature soarch, writing code for data cleaning and analysts, creating nice figures for publication and similar." "Anthhng repetitine would benefit from L.Mss"
Search
Search and information retrieval tasks, which might include literature review
"Information sooking actions (like search) soen to often depend on seriat reformu- lations of concepts, rephrasinngs.
find the right thing. Even if LLMs hallucinate sources, this can be very helpful in those intermediary steps that are part of soarching for things. There's fusst too much out there."
to
"LLMs give you enough information to continue scurching online" "
Literature Review
Helping with literature related tasks, such as finding related work, writing
"Eaiing the otherwise time-conssnningg and bardensome processes in re- soarch. Tiskk like lit seurching, mading through contnct determine study extracting data from puts..." big pile noise. Publications still nather moisy piie of information. LLM's cut through that to some degree."
literature reviews, and summarizing literature
"The internet
somewhat less notey, but
Editing
Editing tasks, such as rephrasing sen- tences
"Mostly rephrasing, reuriting, condensation, buileting"
"Editing and langunge perfection, something like an advenced Grammariy." "
Overcoming Writer's Block
Helping researchers to begin writing (to put something the blank page)
"The nujor benefit of using LLMs cames in actioe when we are stack, for exumple not knownnng the exact word torm, not finding the answer some the ideas
about how it can be used or how can be applied."
loosing the fear of blank sheet paper when you don't know how start your
ariiee...
Broadening Perspectives
Helping researchers discover perspec- tives and diversify their sources
"LLM are remarkable important as they reduce generic data and improve novelty of work"
Allmays available "collengue" to discuss ideas with and get feeback from."
Programming
Supporting programming tasks, in- cluding debugging and writing new code
"I basically code using CoPilot GPT now, often for research give code." " "Definitely sives time on coding applications where I'm not muvare of certain routines already exist). Very helpful to get syntax correctly. Gat fendback on coding errirs..."
packages (for exumpic python puckages
Brainstorming
Helping brainstorm, organize ideas, and get feedback ideas
"LLMs are a great tool help you create hupotheses, as a way to brainstorm, where wrong ideus and therefore you cannut suffer with any patential misteading information, as you are expected to have domnin expertise amyway." "Having always aovilable writing componion help with idcation, divergent thinking, encouragement, and general advice."
there really
Table 8: Themes found in Q65:
Given your ratings above on the benefits/usefulness of these different ways of using LLMs, can you explain what you see to be the main benefitsinsefulness for the academic community and for you as researcher?
22

---

Published as a conference paper at COLM 2025
Theme
Description
Example
Hallucination & Misinformation
Production and spread of incorrect information in- vented by model
"Sometines creates complirated hallucinations that expert Üsink that what it writes it true althongh it is not."
"Putting more falschoods into [the internet's shardd mensory crime."
Inaccuracy
Incorrect conclusions and analyses
"There risk of less experienced scientists using these technologgies as they are unable to check if the outputs are correct as easily as someone with more experienceintaation..
"Th he risks proportional to prior knowuledge of the subject." "
Biases
Model's outputs could con- tain biases and stereotypes
Promattoons of somar papers more than others- fortther narginalzzing voines of those whoo are afrendy less disconered and less cited despite similar quality papers."
"I worry about biosed LLMs influencing the research directions we choose and the conclusions "
Lack of Disclo-
Attribution or disclosure of LLM usage
"Not acknowiedging the use of the Al universities have totally different palicies."
sure
"The adconcement of this technology will make these models a kind of co-author, to the point where will not know the real contribution of the human component.
Plagiarism
Risks of plagiarism when us- ing models to generate pa- per text
"Blind trust in system that is hard to understanud whiick could lead to accidentai plagiarions it's oasy anderstand thicch information LLMs base their outputs
"plogiarism at scale, the commuriit doesn't have enough time to check all the existing popers"
Disrespecting Authorship
Copyright and other con- cerns related to ownership of training data and model outputs
". its use to profit off of text whose authors weren't it compensated is pretty fuched up.
"There issues of integrity where the data that trsined on doeen't necessority belong to the mudel makers, and the model's output doesn't belong the user."
Fabrication
Using LLMs to fabricate data and research results
"The risk of reporting 'results' based on syuthotic data withont actually having conducted any experinent." "
"LLMs are tools for automated plogiarism and data fabrication that pase au existential threat to network of trust essential for the integrity of acadomic work and the proper attribution of credit."
Decreasing Creativity
Effects of model use on the creativity and originality research work
....lmss are going to make people less creative over time. this needs of course more thinking and midener, but to me, yipl may not start thinking collaborating human to faunau find vaiuable H2H coliaborative idens, but rather M2H idens, which might wiss the kumman touch."
he main general risk is to flatten on "mverage", which the worst thingg that may happen a researcher (and it is already happening for arts such music, since this would block inmovation.
Pollution of Research Ecosystem
Decreusing quality re- search that leads to all pollution of information and community
"The hage number of poor quabity papers out there are airendy making science more difficult, and can LLMs making this much worse."
"We need better judgment, slower science, and more thoughtful and ambitious work right now, the opposite. Otherwise, risk ridding science of special attributes just crank out more papers."
Decreasing Diligence
Over-reliance on and trust in models leading to de- creasing diligence of re- searchers
"Speed, copu/paste attitude, less research propositions, less reading of the full article, not enough tracting...
"The main risks are related human tenalnny to for lazy and appreciate concenience too much. would easy to overtrust LLM output the inherent opacity the technology invites people to do so. It is hard to check why particular output is produced and people have antural talent expinin and justify flangs, even completeey urong thangs when more convenient.
Deskilling
Loss research skills due to reliance models
"...srt of like with self-drizing cars which, if we ever get ones that work a bet, people will stop lourning how drice, which will be ind when the things actually get confuseel and harud you the wheell!
"Lorrinn how relay infor information is important skill that ones career. The process of writing the information gives you the way to check whether what you hive done actually makes sense provides value.
cultioated throughout
Table 9: Themes found in Q66: Given your ratings above on the risksfethical considerations of these different ways of using LLMs, can you explain what you see to be the main riskslethical considerations for the academic community and for you as researcher?
23


Reviews:
[{"review_id": "O2tmyNSqcw", "reviewer": "Reviewer_iVRx", "summary": "The paper presents a survey on how LLMs are used as tools for research. There have been previous studies on this topic, but the authors claim that those have either focused on a specific domain or relied on small sample sizes, while the current study relies on researchers from different fields of knowledge and with varied profiles, and uses a larger sample size of 816. The survey participants were sourced from a \"large research platform\", which remains unspecified.\n\nUsage of LLMs is categorized under six types, namely Information Seeking, Editing, Ideation & Framing, Direct Writing, Data Cleaning & Analysis, and Data Generation. For each type, the participants are asked about their LLM usage frequency and perception, using a five-point Likert scale.\n\nThe main findings are that most (81%) respondents use LLM in some way, mainly for search and for text rewriting, and the existence of statistically significant differences between demographics.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- While there are some issues with the participant's data (see below), it is good to have some quantitative data on this issue.\n- The data is made available.", "reasons_to_reject": "- The participants are self-selected: The recruitment email was sent to over 100k people, with only 1226 responses having been collected. The ones that choose to participate, whatever their motivation, may not be representative of the whole.\n- The data is self-reported: Participants can very well admit to using LLMs for small adjustments to writing, but never to, say, fully write a paper or a review, but we can't really know whether they are being honest. Who would admit to unethical behaviour on a survey? Even if you, the study authors, claim it is anonymous, the participant may nonetheless be afraid to risk disclosing this information. \n- The demographics may not be representative: The participants come from different fields of study, but the distribution is nevertheless quite skewed towards STEM domains, with only 24% from Social Sciences and Humanities. The demographic profiles of the participants are also skewed towards white (61%), male (79%), etc.\n- The data may show an outdated picture: The data was collected over a six month period from November 2023 to April 2024, so it's at least one year old. The capabilities of LLMs are progressing so quickly, leading to big shifts in perception on what LLMs can do, that this may be a bit too old to have an accurate account of how LLMs are currently perceived.", "questions_to_authors": "Why don't you explicitly name the research platform from where you sourced the participants? Is it an anonymization issue? But, if so, why didn't you do the same thing you did when naming the IRB that reviewd the study and explicitly used \"omitted for anonymity\" (cf. line 107)?\n\nAs mentioned above, the participant demographics are skewed, but this may simply be a reflection of a skew that is (almost certainly) already present in the members of the research platform that you used as a source. Do you any any demographic statistics on the membership of this platform?\n\nSome other feedback below (as there's no specific section for this):\n- 115: Having \"Non-Native English\" under \"Native Language\" sounds very strange. I'd expect either (a) Native Language: English / Other, or (b) Language: Native English / Non-native English.\n- 651: \"Prefer not to disclose\" appears twice\n- Figure 6: \"Prefer to self-describe:\"", "extra_scores": {}}, {"review_id": "t6LDt0m4Yq", "reviewer": "Reviewer_5NYA", "summary": "This paper is among the first to conduct a large-scale survey on LLM usage as a research tool. The survey contains questions on self-reported demographics, LLM usage, and perceptions. The authors contacted 107,346 verified authors with a final response rate of 0.8%. They find that disadvantaged groups in academia report higher LLM usage and perceived benefits, while women, non-binary, and senior researchers have greater ethical concerns.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "As the authors state, this is among the first studies that provides much-needed evidence about how LLMs are currently used by researchers. The authors made a great effort in gathering and processing this data, and it is nice to see that they have attached their data and code (I believe they will also open source it upon acceptance?).", "reasons_to_reject": "Given the survey’s low response rate, selection and non-response biases are potential concerns that may overestimate actual LLM usage. Additionally, the high proportion of computer scientists among respondents (as acknowledged in the limitations) likely skews results toward disciplines already inclined toward technology adoption, affecting generalizability. It would strengthen the discussion if authors could clarify whether any anonymized metadata on non-respondents exists, despite understandable GDPR constraints, to assess how representative the sample is.", "questions_to_authors": "See reasons to reject.", "extra_scores": {}}, {"review_id": "yNWRvOCzj3", "reviewer": "Reviewer_u4YT", "summary": "The study conducted a systematic survey of 816 researchers to quantify the current usage of large language models (LLMs) in scientific research. Results showed 81% of researchers have adopted LLMs, primarily for information retrieval and text editing, with less use for data analysis and generation. Findings revealed non-white, junior, and non-native English speaking researchers use LLMs more frequently and perceive greater benefits, while female and non-binary researchers expressed stronger ethical concerns. Computer science researchers showed highest acceptance, with overall preference for open-source/non-profit models. The research suggests LLMs may enhance research equity but require addressing ethical issues and establishing disciplinary norms. It provides the first empirical evidence on LLM adoption patterns and perception differences across demographic groups in academia.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Approximately 81% of surveyed researchers utilize LLMs, primarily for information retrieval and text editing tasks, while employing them less frequently for data analysis and content generation.\n\n2. Non-white, junior, and non-native English speaking researchers demonstrate higher LLM adoption rates and more positive perceptions of benefits versus risks, whereas female and non-binary researchers report heightened ethical concerns.\n\n3. Although LLM usage spans all disciplines, computer scientists exhibit greater transparency in usage reporting and fewer ethical reservations compared to other fields, with researchers overall favoring open-source/non-profit LLM platforms.", "reasons_to_reject": "1. The paper lacks sufficient novelty and offers limited reference value.\n\n2. While the study surveyed 816 researchers, it failed to adequately distinguish between different academic levels (e.g., senior scholars vs. junior researchers) and disciplines. In reality, researchers at different career stages and from various fields may have substantially different motivations for using LLMs.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is the result and analysis of a large-scale survey (816 participants) for researchers, inquiring over the use of LLMs in their research process. Some of the more interesting findings are the high prevalence as well as the difference perception of their utility and concerns across demographic traits

The discussion was mostly centered on the drawbacks of relying on self-reported metrics, as well as the impact this self-selection had on the demographic distribution and the answers per se. Many of these are intrinsic to the nature of this work, and the authors have done a good job in replying to those concerns

This is certainly an interesting paper which will fuel good discussions at COLM and beyond. It should be reminded that this is a snapshot - redoing this survey now (a year later) might yield very different results. At the same time, it is this snapshot which might prove impactful in the future when studying the trajectory of LLM usage in research. So many things are changing at the same time, that having some point analysis might become very influential

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept