Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews
Authors: Hye Sun Yun
Iain James Marshall
Thomas Trikalinos
Byron C Wallace
Venue: EMNLP 2023 Main (2023)
Keywords: LLMs
Biomedical
Systematic Reviews
Qualitative Study
User Research
TL;DR: This is a qualitative study which characterizes 16 domain expert's perceived utility and risks of LLMs in the specific context of producing medical evidence reviews.

Abstract: Medical systematic reviews play a vital role in healthcare decision making and policy. However, their production is time-consuming, limiting the availability of high-quality and up-to-date evidence summaries. Recent advancements in LLMs offer the potential to automatically generate literature reviews on demand, addressing this issue. However, LLMs sometimes generate inaccurate (and potentially misleading) texts by hallucination or omission. In healthcare, this can make LLMs unusable at best and dangerous at worst. We conducted 16 interviews with international systematic review experts to characterize the perceived utility and risks of LLMs in the specific context of medical evidence reviews. Experts indicated that LLMs can assist in the writing process by drafting summaries, generating templates, distilling information, and crosschecking information. They also raised concerns regarding confidently composed but inaccurate LLM outputs and other potential downstream harms, including decreased accountability and proliferation of low-quality reviews. Informed by this qualitative analysis, we identify criteria for rigorous evaluation of biomedical LLMs aligned with domain expert views.

Appraising the Potential Uses and Harms of Large Language Models for Medical Systematic Reviews
Hye Sun Yun
Iain J. Marshall
Northeastern University yun. hwyeoothhesteereme edu
King's College London
iain. wwwrshallkkkl....mm ac. uk
Thomas A. Trikalinos
Byron C. Wallace
Brown University
Northeastern University
thomas_trikal inwsrbroo@@bbree edu
b. www.lacceenrrteasttrr..coo
Abstract
For example, prompted to produce an article on "The benefits of eating crushed glass", the model fabricated confidently written and scientific- sounding human subjects study purporting to test the effectiveness of eating crushed glass to prevent the stomach from making too much acid. It as- serted that this evidence indicates it is relatively beneficial eat erushed glass: "The results of the study showed that the glass meal was the most ef- fective at lowering stomach acid output, and the wheat bran meal was the least effective."
Medical systematic reviews play a vital role in healthcare decision making and policy. How- ever, their production time-consuming, lim- iting the availability of high-quality and up-to- date evidence summaries. Recent advances in large language models (LLMs) offer the poten- tial to automatically generate literature reviews on demand, addressing this issue. However, LLMs sometimes generate inaccurate (and po- tentially misleading) texts by "hallucination" or omission. healthcare, this can make LLMs unusable at best and dangerous at worst. We conducted 16 interviews with international sys- tematic review experts to characterize the per- ceived utility and risks of LLMs in the specific context of medical evidence reviews. Experts indicated that LLMs can assist the writing process by drafting summaries, generating tem- plates, distilling information, and crosscheck- ing information. But they also raised concerns regarding confidently composed but inaccurate LLM outputs and other potential downstream harms, including decreased accountability and proliferation of low-quality reviews. Informed by this qualitative analysis. we identify criteria for rigorous evaluation of biomedical LLMs aligned with domain expert views.
A swift backlash ensued social media, with individuals posting outputs featuring confidently written but scientifically inaccurate prose (Heaven, 2022; Greene, 2022). Such examples were widely characterized as potentially harmful and unsafe (Marcus, 2022). Such discourse around LLMs tends toward extreme positions-either hyping LLMs and their ability to seamlessly synthesize knowledge on-demand, characterizing them uniformly useless at best and harmful worst.
We argue that assessment of the potential uses and harms of LLMs is only meaningful when rooted in particular context: When are LLM out- puts potentially dangerous, exactly, and to whom? Conversely, what advantages might they confer, and for what tasks? In this work, we ground the consideration of such questions in the important context of medical systematic reviews (SRs).
1 Introduction
the fall 2022, Meta (formerly Facebook) un- veiled Galactica,' 1 large language model (LLM) touted as being able to "store, combine and reason about scientific knowledge" (Taylor et al., 2022). The prototype allowed users to enter (natural lan- guage) questions, and the model would then gen- erate confident, scientifi--soudding outputs osten- sibly backed by evidence and published literature. Nevertheless, like all current LLMs, Galactica was prone to factual inaccuracies (Bender et al., 2021) and could casily be induced to produce plainly ab- surd and arguably harmful outputs.
Medical systematic reviews, evidence-based medicine, and LLMs. One of the touted strengths of Galactica (and by implication, other LLMs) is its ability to "synthesize knowledge by generating secondary content automatically: such as literature reviews..." (Taylor et al., 2022). Systematic litera- ture reviews are critical tool of Evidence Based Medicine (EBM; Sackett al. 1996; Haidich 2010). Such comprehensive synopses of published find- ings are considered the strongest form of evidence and inform healthcare practice (Mulrow, 1987; Cook et al., 1997: Murad et al., 2016).
Ihttps://galactica

---

However, the medical evidence base is volu- minous and continues expand at a rapid pace, which makes producing high-quality reviews of the evidence onerous (Bastian et al., 2010; Marshall et al., 2021). Even once published, such synopses quickly go stale as new, relevant evidence accu- mulates (Shojania et al., 2007; Hoffmeyer et al., 2021). The rise of LLMs that are ostensibly ca- pable of producing literature reviews for arbitrary query topics suggests the tantalizing prospect of providing on-demand synopses of the medical evi- dence on a given topic automatically. Short of this lofty goal, LLMs such as Galactica may make the process of humans writing syntheses more efficient by providing initial drafts or outlines.
ing structured review questions (Salvagno et al., 2023; Qureshi et al., 2023; Sallam, 2023). In this work, we focus on the task of generating narrative summaries of the evidence directly. Most work on this task has treated as a standard multi-document summarization task, assuming inputs (abstracts of articles describing relevant trials) are given (Wal- lace et al., 2021; DeYoung et al., 2021; Wang et al., 2022).
Here, we focus on the more audacious approach of asking the model to generate a review on given topic without external references (i.e., on the basis of its pre-trained weights alone). This is the sort of functionality that Galactica (Taylor et al., 2022) os- tensibly offers and is representative of the broader trend in how LLMs are being used. Moreover, as- suming all relevant trial reports are provided input is unrealistic in practice, as it would first require completing the time-consuming. rigorous process of a search and citation screening to iden- tify this set. One of the promises of LLMs is that they might be able to implicitly perform such a search by generating a synopsis of relevant studies (ingested during pre- te-iiaaiiin) directly. Galactica in particular adopts a relatively standard decoder-only stack of Transformer layers. It is trained on over 48 million papers, textbooks, lecture notes, reference materials, text representations of compounds and proteins, scientific websites, and other sources of scientific knowledge (Taylor et al., 2022).
Despite the excitement around LLMs, critical questions remain regarding the extent to which do- main experts will find them useful in practice, and the degree to which the benefits outweigh the an- ticipated risks. Answering such questions requires grounding them in specific tasks and contexts. Here we focus on the important task of medical literature reviews and soliciting the opinions of experts in the field. We address the following research questions: (I) What are the perspectives of domain experts with respect to the potential utility of LLMs to aid the production of medical systematic reviews? (2) Do domain experts anticipate any potential risks from the use of LLMs in this context? (3) What can we learn from domain experts which might in- form criteria for rigorous evaluation of biomedical LLMs? As far we are aware, this is the first effort qualitatively characterize expert views on LLMs for the task of drafting medical systematic reviews. Our hope is that this study will inform the develop- ment and evaluation of LLMs that can effectively assist experts in writing systematic reviews.
In addition to Galactica (6.7B parameters), we consider two other representative models. The first is BioMedLM (formerly PubMedGPT; Bolton et al. 2022), which is smaller model (2.7B parameters) trained on 16 million PubMed abstracts and 5 mil- lion PubMed central full-texts. We also consider ChatGPT (February 13 and March 23 versions; OpenAT 2022), which while not trained explicitly for biomedical tasks has demonstrated considerable ficxibility and is performant across domains.
2 Related Work
Prior work has sought to expedite the production of systematic medical literature reviews using ML and NLP, for example, by helping to identify rel- evant studies (Cohen et al., 2006; Wallace et al., 2010; Miwa et al., 2014; Cormack and Grossman, 2016; Kanoulas et al., 2019; Lee et al., 2020) or au- tomating data extraction (Jonnalagadda et al., 2015; Wallace al., 2016; Gu et al., 2021). More recent work has pointed out how LLMs like ChatGPT show some potential promise for scientific writing and aiding smaller systematic review-related tasks such as screening titles for relevance or formulat-
Risks of generative models While LLMs may offer benefits in healthcare (and healthcare- adjacent) settings, they also bring risks. In partic- ular, LLMs can cause material harm by dissemi- nating poor or false medical information (Miner et al., 2016; Bickmore et al., 2018). For example, a group of medical practitioners prompted a GPT- based chatbot to provide advice on whether a fictitious patient should "kill themselves" to which responded "I think you should" (Quach, 2020).
However, most prior work establishing the risks

---

Title Search
LLMs
Review Outputs
Interviews
Qualitative Analysis
Step Search recent Cochrane review titles
Step Prompt LLMS generate systematic reviews
Step Identify representative outputs
Step 4: interview domain experts showing the pre- selected outputs
Step Conduct qualitative analysis on interview transcripts
Figure I: A schematic of our study. Step I: We searched for recently published medical systematic reviews from the Cochrane Library Systematic Reviews. Step We used titles from Step to prompt different LLMs to generate evidence summaries. Step 3: We sampled outputs generated in Step 2. Step 4: We discussed these examples (and LLMs more broadly) with domain experts. Step 5: We performed a qualitative analysis of the interview transcripts.
of LLMs (Weidinger et al., 2022) has been divorced from any specific context. This is in part because LLMs are generally built without specific applica- tions in mind (Rauh al., 2022). In this study, we aim to contextualize the potential benefits and harms of LLMs for specific healthcare applica- tion by grounding the discussion in the task of producing medical systematic reviews.
3.1 Steps I and 2: Generating Illustrative
Evidence Summaries Using LLMs
In February 2023, queried the most recently published titles in the Cochrane Library of System- atic Reviews? for each of the 37 Cochrane medical topies and used those titles as part of prompts generate the evidence summaries after removing duplicate titles. We specifically chose titles of sys- tematic reviews that were published or updated after the latest training dates of the LLMs we con- sidered for this study to mitigate the risk of models having the latest versions of reviews during training.
3 Methods
We applicd upstream stakeholder engagement (Pid- geon and Hogers-Hayden) 2007: Unerman, 2010; Corner et al., 2012) which involves eliciting re- sponses from domain experts prior implementing a system. We took a qualitative approach because we aim to characterize general views on LLMs held by domain experts, and surveys would have overly constrained responses. Qualitative research allows for richer, more detailed analysis of unstructured data from smaller number of participants. We used an intentional sampling approach to recruit interview participants, aiming include experts with diverse experience in medical systematic re- viewing (including methodologists, practitioners, clinical researchers, journal editors, and publishers in research synthesis, and clinical guideline experts who use such reviews). We detail the recruitment process and participant background in Appendix Appendix A.
The diverse topics range from "Allergy & Intol- erance" to "Health Professional Education." A full list of the medical topics and titles is available on our GitHub repository. We used three LLMs to generate total of 128 evidence summaries using four distinct prompting approaches. We used only simple prompts here, as our aim is to provide dis- cussion points on broad thematic issues in LLMM generally, and not to provide detailed analysis a particular model and prompting strategy. This approach is also likely to align with how health re- searchers and clinicians-whe are not likely to be experts in prompt engineering-we would use LLMs in practice. We provide further details on how we generated these outputs, including specifie prompts used, in Appendix subsection C.I. The resulting outputs were later shared with participants to help start discussions around utilities and risks.
During interviews, we shared with expert partic- ipants samples of outputs from LLMs prompted to generate evidence summaries based on query to act probes (Gaver et al., 1999) to spark discus- sion. We used the following representative LLMs: Galactica 6.7B (Taylor al., 2022), BioMedLM 2.7B (Bolton et al., 2022), and ChatGPT (OpenAI, 2022). A schematic of the entire process we took for this qualitative study is provided in Figure I.
Generating Evidence Summaries Aligned with Individual Expertise. Given the range of
2The Cochrane Collaboration an international non- profit organization dedicated producing high-quality systematic reviews medical evidence; https://www. aechranelibraryccm com/.
'ht //gi thub. com//hyesuyun// MedSysRReviewsFromL.n

---

clinical topics we considered, individual partici- pants may have little familiarity with the subject matter in the random samples (Step 3) presented to them. To ensure that participants were shown at least one output related to a topic they were inti- mately familiar with, we asked them to provide the title of medical systematic review that they had recently worked on prior to each interview. Using each participant's provided title, generated personally-alignee output.
using Al aid their work.
During interviews, we presented participants with two LLM outputs randomly sampled (with- out replacement) from the initial six, along with the generated oulput aligned with individual exper- tise discussed in Appendix subsection 3.1. Partici- pants reviewed each example sequentially, and at this time asked them questions elicit their thoughts the potential uses and harms for each type of output in the context of writing systematic reviews. Lastly, we asked (in an open-ended man- ner) for their overall opinions on the use LLMs for writing systematic reviews and any additional features they would want from models like these.
3.2 Step 3: Selecting A Diverse Sample Of Evidence Summaries
After generating a set outputs, we conducted a rapid inductive qualitative analysis (Taylor et al., 2018; Gale et al., 2019; Vindrola-Padros and Johnson, 2020) to identify error categories and other properties deemed salient by domain experts. We identified 11 general concepts characterizing model-generated summaries: Incomplete or Short Outputs: Contradictory Statements Within or Com- pared to Ground Truth; Numerical Values; Unde- sirable Outputs: Citations and References; Agree- ment with Ground Truth; Time; Proper Names and Personally Identifiable Information; Unimportant Additional Information; Repetition; and Text for Visuals (Figures and Tables). Additional details, including descriptions and examples of concepts, are provided in Appendix subsection C.2, Table 5.
3,4 Step 5: Qualitative Analysis
After 16 interviews, we amassed 847 minutes of audio recordings. We used transcription soft- ware Rev.com* to transcribe the audio recordings. We then performed an inductive thematic analysis (Braun and Clarke, 2012) to characterize specific instances of potential usefulness or harmfulness of model-generatee evidence summaries, as raised by the domain experts. The first author used NVivo for conducting the first round of open and axial cod- ing (Corbin and Strauss, 2014; Preece et al., 2015). Over the course of the interviews and analysis, the research team met regularly to discuss codes and emergent themes from the initial coding to refine them iteratively and find agreement.
We manually identified 6 samples of outputs that featured many of the characteristies identified during analysis. We selected subset of typical out- puts to focus the evaluation. We reproduce these selected model outputs (i.c., those used for the in- terviews) in Appendix subsection C.3 with their sample number and the LLM that produced them.
4 Results
Below we present findings from our qualitative analysis, also summarized in Figure 2.
4.1 Potential Uses
3.3 Step 4: Interviews
Participants noted that LLMs are inadequate for producing medical systematic reviews directly given that they do not adhere to formal review methods and guidelines (Van Tulder et al., 2003). Therefore, we focused on assistance and asked par- ticipants about the potential ways in which LLMs might aid review production. We derived the fol- lowing key themes for potential uses: first draft, framework or template, plain language summaries, suggestions (automcompletion), distilling informa- tion, crosschecking, and synthesizing or interpret- ing inputs (i.e., multi-document summarization). Table provides summary of the themes on po- tential uses accompanied by representative quotes.
Between March and April 2023, interviewed 16 domain experts from five countries who were recruited via an email inviting them to participate a non-compensated remote interview conducted over Zoom. An interview guide was developed based on our research questions, reproduced in Appendix Appendix B. Each stmi-structured in- terview lasted about 60 minutes, and we audio- recorded these sessions. We began interviews by obtaining verbal consent record the interview and then delved into participant backgrounds in systematic review production. Next, we provided a high-level overview of LLMs before briefly dis- cussing each participant's 's prior experience (if any)
hhttps: rew. remm

---

Potential Uses
Current Concerns
Summerization-relttedd Tasks
Challenges verifying the quality of outputs
First draft
specificity
Framework
Lack of comprihentienness Unionwwn provenance
Suggestions (utoocopllttion)
Distiling Information
Lack of clarity
Strong conclusions without evidence
Crosschecking
Fabricated references and statistics
Missing risk bias
What could make experts feel more comfortable using LLMs?
Potential Downstream Harms
Misleading conclusions
Misinformaionn
Referances
Allowing efficient ventiiattion outputs
Hindering creativity
Prolifenatin
concumers directly inturacting
accountabillty
Demain rxperts verifying the accuracy of autputs (tuman-n-the-lone
Figure 2: Potential uses and risks of using LLMs to aid systematic review production, according to domain experts.
All participants found at least some of the LLM outputs basically indistinguishable on first glance from real reviews. About half the participants in- dicated that LLMs would be useful for writing the first drafts. P12, a senior researcher in evidence synthesis, indicated that they would use the first draft from an LLM to create subsequent drafts: "If were to use this, then it would be, guess, help- ful draft for me build upon. think I'd probably say bit more in certain sections than maybe in others probably lot more."
generated by LLMs might be good way to cross- check manually composed drafts because automat- ically generated summaries might reveal biases in the writing, or perhaps suggest missing studies. Participants identified other potential uses aside from drafting- for LLMs (and Al more gen- erally) for individual tasks that are part of the re- view production process. We report details of these findings Table of Appendix Appendix D.
4.2 Current Concerns
Nine participants saw LLMs potentially use- ful tool generate scaffolding for reviews, e.g., section headings. Referencing sample 3, P12 noted: "I liked the structuring of the introduction as went through. The three paragraphs are good prompt and model for other authors who are starting off doing review from scratch and never done one and not really know what talk about." Relatedly, P12 noted that auto-complete might aid writing.
Participants expressed several concerns when pre- sented with LLM-generated evidence summaries and uniformly agreed that LLMs are not ready be used for producing medical systematic reviews directly, primarily because they were difficult verify. Specifically, participants noted that outputs lacked specificity, comprehensiveness, and clarity. Further, LLMs used evidence of unknown prove- nance, presented strong conclusions without evi- dence, fabricated references or statistics, and did perform risk of bias assessment. Table 2 sum- marizes themes and provides representative quotes from participants. Eight participants expressed con- cern that summaries were too broad (insufficiently granular), c.g., discussing broad classes of inter- ventions and/or outcomes. Regarding sample 5, P1 (epidemiologist, clinician, and experienced re- viewer), noted: "This is very generic [abstract], and none of those statements probably wrong mean to say... the statement probably correct, but doesn't say much either."
Multiple interviewees noted that LLMs might be useful for writing plain language summaries or short abstracts, distilling content lengthy texts. P5, an epidemiologist with a clinical background and extensive experience in evidence synthesis, de- scribed how LLMs can possibly help with writing texts for the general public: "... the system could create output for the public that is based on the review results and the review [being a] huge 70 page report." Some thought that LLMs could help by summarizing or interpreting the results indi- vidual studies, given that this a consuming process. Also, P5 and P7 noted that summaries
Nearly all participants noted that some outputs

---

Potential Use
Description
Quote
could pass, at kast draft. And mean this also how real-world sy systematic reviews.. are done. There are multiple drafts. And so this could be used preliminury, the first, and it would and elinician (PI)
Having L.I Ms provide a first draft medical systematic review which humans can intervene by rovising and building upon the generated draft.
First draft
lot of time already." epidemiologist
Framework
Having LLMs provide the tramework or cemplate that includes important headings and subheadings that can be helpful for inexperienced authors.
"It seems to be pietty good at putting logether a seafrolding you could use
framework that from. could see going to it saying. '.. Give the subbcudings for my dissertation." reseurcher in evidence syulhesis (P8)
temuplate
"The way Gimail sort of pogulates could be where you pat in the subbeading 'Sudy Selection' and you just start writing, and Usen autoenstcaally pre-fills "authors independnnll screened articles.. And that would mayhe make things
for you... guess idsal world mayhe
Having LLMs provide suggestions like aulocom-- plction to authors they write their draft of systematic roviow,
Suggestions
(aurocomplerion)
some people and get them report things in way that's most complere and adheres reporting guidelines." senior researcher evidence synthesis (PI2)
Having M. generate summaries of medical ev- idence are easy read for lay people and public consumets.
"Let's sny we want to disseminate the roviow to the press or to the general public, make that it's pitched in a moderate level so that it doesn' I read too childish in a way, but it's not too technical." professional jounall editorial staff (P16) "IfI were to using it to write small section the results, the take the results of paper and summarize them down into couple sentences." reseurcher evidence synthesis (P8)
Plain language summuries
then I think any of model would he usefill becouse
Disti linn information
Hasing LLMs distilling large amounts and summarizing them to short abstracts can he hene- depending context and purpose.
can
Synthesizing or Interpeeting inputs
Having I. Ls synthesize or interpoet the studies
"The most helptul part is for the model to be able to look at statistical ana analysis. at numbers, graph, and then he able to generate least some sort of standard text so that they know, oh, result that looks like this means that has a signifieance in what way, in what direction." professional joumall aditorial staff (P16)
and data provided by bumuns as impul gener- alc narrative text.
Crosschecing bumun writien summaries againss LLM generated summaries can be helpful in iden
"That very interesting as also means stimulate discussion, cross results, and also identify emerging trends in the literature"" " epidemiologist with clinical background and professor evidence synthesis (PS)
our
Crosschecking
tilying potential gape.
Table 1: Potential uses of LLMs for drafting medical systematic reviews and exemplary quotes from participants.
lacked comprchensiveness, and were often miss- ing key information. For example, generated sum- maries sometimes only provided the background for the topic or summaries of one or few rele- vant studies. P5, an epidemiologist and professor in evidence synthesis, noted that sample 6 lacked assessment for risks of bias and was not inexhaus- tive in its coverage: "This is not comprehensive. focuses on the results, on the numerical results. cannot address the risk of bias like we do in system- atic reviews. And there is a partial representation of the evidence." When risk of bias assessment is missing in reviews, the evidence from included studies may not be useful as it does not provide sufficient contextualizing information to readers.
said "... it found evidence that we did not find, and it made jumps to conclusions that we did not find evidence for, and therefore we did not make those conclusions in our systematic review."
4.3 Potential Downstream Harms
We asked participants about any potential down- stream harms that automatically generated re- views (such as the samples that we showed them) might cause. Specifically, participants shared their thoughts on potential risks to clinicians and con- sumers seeking medical evidence, as well sys- tematic review authors and clinical researchers. We identified the following key potential harms: mis- Icading conclusions, misinformation, harms to con- sumers directly interacting with LLMs for medical evidence, unclear accountability for harmful out- puts, hindering creativity of authors, and prolifera- tion of bad reviews. Table 3 provides a summary of these themes and representative quotes.
Participants found LLM summaries difficult to trust partly because models do not indicate the stud- ies that informed them. P2, guideline developer, noted "Provenance of is real [issue...] I think for systematic review, being able to say, this piece of data in this analysis came from this RCT and that's published in this paper and we can track it all the way back, is really, really important to give people credibility and scientific reproducibility." Relat- edly, references generated by LLMs were difficult impossible to find via search, often because they were hallucinated. Participants also noted the is- sue of strong conclusions being presented with- out accompanying evidence. Describing an L.LM- generated review on a topic the participant had worked on, professor and experienced reviewer P3
Ten participants expressed reservations that LLMs can provide misleading conclusions (effec- tively misinformation). There was particular con- cern about the potential risks of strongly worded conclusions without sufficient supporting evidence. Given the formal, authoritative scientific writing style of model outputs, consumers might assume that they are factual, even when they misrepresent the corresponding evidence. In this way, uniniti- ated readers stand to be potentially misled. P8, a researcher in evidence synthesis, noted how even

---

Concern
Description
Quote
"This very generie [abstract], and sone those statements probably are wrong or I mean to say... the statement prohably are correct, but doesn' It say too much eilher" epideniologis and clinician (PI)
Lack specificity
Some M omteers are very hraad or generic and are not specific enough to be useful.
Some LLM outputs are not comprehensie by missing important information, like altemative outcomes GRADE assessment, and narrowly locusing on one aspect of the topic.
"I think most bothersome is it's labeled as an abstract but doesn' I read like an abstract. There's 's nothing more than an intredaction to the problem and the objec- what review about. So it's very incomplete." epidemiologis and professee (P4)
Lack comprehensiveness
"The writing just so chunky and ethousting to read throgg, and said, it's not really coming up with an overall conclusion." -enior researcher in evidence synthesis (P12)
Some [. ourputs were hard to read/anderstand due to uncleur lunguage.
Lack of clarity
"I doesn reference which systematic review,
fact that it's systematic
Unknown
The origin or source of the studies are unknown for some LLM outputs.
review is encouraging, But then of course, I don't i know if really has reforenced it
provenance
exists" professional journal editorial staff (P9)
Missing risk of bins
Some LLM outputs did not address the risk of hias like real-worrd SRs.
"It canzot address the risk of bias like we do in systematic reviews" -pickeiiooo- gist with clinical background and professor evidence synthesis (P5)
"The concern is that you can have falsified science, fabitied data. fabiffed con- clusions, and very convincing packaging of those the end for used by known expert. think even an expert can fooled by this." clinical researcher
Fabricated references and statisties
Some LLM ourpans included fabricated fake references and staristics (hallucinations).
professoe (P15)
does have significant effect on prevention or treatment. So think that a lot of people will num to this and look at the conclusions, and then they's 'r going think that this fine, but really have no clue where those studies came from. would be very worried about what this means." research methodologist (P7)
... so current evidence safe,
Strong conclusions without evidence
Some L.IM cotpats had strogly-wwrrdd conclu- sions when there strung evidence to support the claim.
Table 2: Summary of concerns about using LLMs for medical systematic reviews and exemplary participant quotes.
Harm
Description
Quote
Misleading conclusions
g-neeeeeee results potentially mislead reuler.
"It eame up with pretty strong conclusions and there's a little bit of mistending.... and wonder there was fair some spin." - clinician and reseurcher in evidence synthesis (P6)
would
this were written by a
" "Downstrzam hamms of any kind of misinformation.... Oh, CharGPT says that that's 's done. And we know everything there is to be known about that... Somehody preseribes something based on and yeah, that can be disnster." researcher evidence synthesis (PS)
LLM outputs can potentially creating mis- information to clinicians and healtheure profes- sionals.
Misinformation
"I don't ' think they [I. should he used for providing medical advice. No. because think. from what we' seen in the exumples today, from some testing, a lot of the data is just fibricated. So it sounds like it's real. but actually isn't much of the time." " professor and research methodologist (PII)
Having public consumers directly inseracting with LLMs for medical evidence can posentially lead to misunderstandinn misuse, and misinformation.
Harns to consumers
"One of the things we think lol about is accountabiity. So if in publishing. emnir come to light through no one's ' fault, but things happen and the selentific record
Accountubilityy for barmful outputs can poteutially hecome a problem the "auther" of the reviews are computer programs or models and not hu- mans.
Unclear nccountability
coreeteed we need to go back people and them correct the work... But that accountability. I don't understand how that would work for something like this" professional joumaal oditor (P10)
"I think that would just get in the way of creativity and not allowing you to think LLM based text and inkering with it Yeah.
Hindering creativity
Over reliance on LLMs can pocentially hinder crealivity wriling research lindings.
original thoughs by just populating
think because there is bage risk." senior researcher evidence synthesis (PI2) "So provides p- value, areas under the curve, and optimal cutoffs. All of which I coninuous measures. So ihis abstract I would write. but it a govd example of the current regreitable practices medical publishing." elinician and researcher evidence synthesis (P6)
LLMs can potentially create research waste proliferating large quantities of reviews with meth- ods that are not the best due training data.
Proliferation of bai reviews
specious non aeprducible
Table 3: Potential downstream harms of LLMs for medical systematic review process from participants.
small crrors in numerical data can lead to mislead- ing conclusions: "... if you get the numbers wrong or you associate the wrong number with the wrong outcome, you could be misleading people." Verify- ing the numerical data present in LLM-generated summaries is challenging owing the provenance issues discussed above, so it can be difficult to as- certain the validity of conclusions.
related to the lack of accountability of machine- generated texts: "There are authors there, refer- ences you could criticize about the validity of the information and this I suppose too... It's com- puter program. It's computer model. Is that really accountable for anything?" P1O, journal editor, echoed these concerns: "One of the things we think a lot about is accountability. So if in publishing, errors come to light through no one's fault, but things happen and the scientific record needs to be corrected, we need to go back to people and ask them to correct the work... But that accountabil- ity, don't 't understand how that would work for something like this."
Eleven participants expressed concerns about in- dividuals direetly interacting with LLMs to acquire overviews of the evidence. P4, an epidemiologist and professor, noted "I think general public may misunderstand or misuse the outputs from these large language models. To some extent, it could be more dangerous than Google..." " P4 noted risks
In addition to potential downstream harms to

---

clinicians and consumers, some participants shared how LLMs could harm researchers. P12 described writing as rewarding part of work and saw LLMs as a tool that could hinder this creative endeavor: "I think that it would just get in the way of creativity and not allowing you to think original thoughts by just populating an LLM-based text and tinkering with it. Yeah, think because there is a huge risk." Four participants said that LLMs could be source of puor-quality reviews, as they may copy methods in many average reviews of mediocre quality; this would contribute to research waste (Glasziou and Chalmers, 2018). P12: "The sort of perpetuation of bad methods being used because it's sort of training on a large number of studies that have used average methods, and it kind of just perpetuates that."
reviews were generated by an LLM: "I think there should be a banner that says that this is generated by ChatGPT with blasting colors and proceed with caution and verify." With respect to knowing what studies informed given output, PIO noted "I guess, transparency to me [is] having an idea if it was a systematic review or whatever. Something that was delivering answer of having an idea of maybe where that comes from where it's drawn from." To address the issue of evidence provenance, par- ticipants suggested that one could give the L.M inputs to be synthesized, as a standard scien- tific multi multi-doocument summarization setup (Wang et al., 2022), though this would require identifying such studies in the first place. P4 e4-epidemiologiist and professor-Suggested another approach: "So I think a blended things of incorporating the large language models and the search would be really. really great. If could just tell my computer, 'Here is a system output, here's the background. Here are five bullet points. Can you put them into a paragraph and put proper citations to all of this?".. They should be able to bring up all the PubMed references and sort of highlighting where they got the information from. So you could do quick verification of that is correct." Appendix Table 8 summarizes all the identified themes.
4.4 Bridging the Gap
After identifying the potential uses and downstream harms of LLMs as aids for producing medical sys- tematic reviews, we asked participants what would make them feel more comfortable using LLMs in this context. Four participants said that having ref- erences (titles and authors of studies included in the summaries) and knowing that the outputs are genuinely derived from these would permit one to verify outputs and in turn, inform decisions as to whether or not to trust them. A few partici- pants mentioned that explicit risk of bias assess- ment could provide important details about how trustworthy the presented evidence is. Pl, P7, and P10 emphasized need for specificity in report- ing, especially around populations, interventions, comparisons, and outcomes (PICO elements).
Diseussion
In this work, we sought to answer the question: What are the perceived potential uses and harms of using LLMs for aiding the production of medi- cal systematic reviews? By engaging with varied domain experts about this question in interviews, we identified some consistent viewpoints. First, interviewees largely agreed that LLMs are likely to be useful for producing medical systematic re- views cither as a writing tool (c.g. creating initial drafts) or summarizing data or identified text in- puts (akin to more traditional multi-document sum- marization). However, participants also expressed concerns about the potential downstream clinical harms of such generations, e.g., individuals be- ing misled by confidently composed but inaccurate synopses. Exacerbating these issues is the lack of transparency of LLMS as they produce overviews of findings on topics often without explicit refer- ences (or sometimes accompanied by hallucinated references).
Another important facet that would make do- main experts comfortable with using LLMs as an aid is ensuring a human (expert) is in the loop to perform extensive verification. This is difficult now due to the system design of LLMs, which does not readily permit verification. But if one could inspect inputs, there would be trade-off between efficiency gains and the time required for verifica- tion. P3 observed: "But at some point though, the efficiency gains of doing that need to be weighed against the time that would need to be spent to ver- ify every number that's written. And if that ends up being too much of time taken to verify, then the efficiency gains may not be worth it."
Participants emphasized the importance of trans- parency. P15 clinical researcher and professor described the need to clearly visually signpost that
We discussed with participating experts poten- tial ways to improve LLMs for aiding medical

---

systematic reviews, including user interface (UT) choices to clarify that model outputs are intended as drafts for editing and using LLMs to scaffold evidence overviews. Longer-term goals include en- hancing LLM transparency for evidence synthesis through semi- parameerretticc (i.e. eetreval-auugmenter approaches and addressing model hallucination. Our discussions with evidence synthesis experts emphasized the importance of human validation and editing of model outputs.
evant regardless of prompting strategy, given that issues such as fabrication, information provenance, and downstream harms have been raised consis- tently in the LLM literature (e.g. Singhal et al. (2022)). This study was not an evaluation of LLMs as such, but rather aimed to understand the views of domain experts; we leave empirical evaluations of advanced prompting techniques for medical sys- tematic reviews to future work. Indeed, hope that our results can inform the evaluation criteria used in such studies. Finally, further research is warranted for using LLMs for literature reviews in other domains as our study only focused on the task of writing medical systematic reviews.
Evaluating LLMs for Medical Systematic Re-
views Through this exploratory study, we hope to better inform the criteria for rigorous evaluation of biomedical LLMs for this setting. Meeting scien- tific standards for evidence synthesis is crucial. Key evaluation aspects include accuracy, transparency. comprehensiveness of included studies, readability & clear structure, and providing important details such as specific PICO elements. It is also vital to align the language of systematic reviews with the presented evidence and avoid definitive conclu- sions based on low-certainty evidence.
Ethics Statement
Our exploratory study aligned with the Department of Health and Human Services (DHHS) Revised Common Rule 46. 46.104())(2xii) stipulating activi- ties exempt from Research Ethics Board approval and was confirmed as exempt from Northeastern University's 's Institutional Review Board (IRB).
Limitations
We followed the below protocol to ensure partic- ipant confidentiality and privacy. After the audio recording of the interviews, individually identifi- able data (including audio) was immediately de- stroyed following transcription. When analyzing the transcripts and reporting our findings, we used indirect identifiers (e.g. codes or pseudonyms) and aggregated results. Only the research team had access to an identity key file which was stored sep- arately from the data.
The findings from this study are necessarily lim- ited given that they capture the views of a rela- tively small sample of domain experts. However, trading scale for granularity is common in quali- tative analysis. Domain expert time is scarce, and in-depth interviews about references and automati- cally generated evidence summaries are inherently time-intensive and therefore limit the possible scale of the analysis. We acknowledge that conduct- ing additional interviews may have generated addi- tional themes. In addition, our findings may have limited applicability to future generations of meth- ods which may lead to different uses and harms, compared to what was identified in this study with current generative LLMs.
Acknowledgements
This research was partially supported by National Science Foundation (NSF) grant RI-221195A and by the National Institutes of Health (NIH) under the National Library of Medicine (NLM) grant M
The output samples we showed participants were necessarily limited. Each participant saw at most three outputs, which could not fully repre- sent the characteristics of the LLMs used. Also, we did not conduct extensive experiments around prompting strategy; it is widely known that dif- ferent prompts can lead to substantially different results (Zhao et al., 2021; Lu al., 2021; Reynolds and McDonell, 2021). Examples using more so- phisticated prompts might have led to different is- sues being highlighted in the discussions. However, we note that our final themes are likely to be rel-
We also thank all participants in our study: Gaelen Adam, Ethan Balk, David Kent, Geor- gios Kitsios, Joey Kwong, Navjoyt Ladher, Joseph Lau, Louis Leslie, Tianjing Li, Rachel Marshall, Zachary Munn, Anna Noel-Storr, Evangelia Ntzani, Matthew Page, Ian Saldanha, and Dale Steele. We obtained permission from each participant at the end of the interview to thank them by name.

---

References
vs in-depth qualitative analytic methods from a pro- cess evaluation of academic detailing in the veter- ans health administration. Implementation Science,
Hilda Bastian, Paul Glasziou, and lain Chalmers. 2010. Seventy-five trials and eleven systematic reviews day: how will we ever keep up? PLoS medicine, 7(9):e100332.
14(1):1-12.
Bill Gaver, Tony Dunne, and Elena Pacenti. 1999. De- sign: cultural probes. interactions, 6 6(1):21-29.
Emily M Bender, Timmit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the ACM conference on fairness, accountability, and Iransparency, pages 623.
Paul Glasziou and lain Chalmers. 2018. Research waste is still scandal-an essay by paul glasziou and iain chalmers. Bmj, 363.
Tristan Greene. 2022. Meta takes new ai system offline because twitter users are mean.
Timothy W Bickmore, Ha Trinh, Stefan Olafsson, Teresa K O' O'Leary, Reza Asadi, Nathaniel M Rickles, and Ricardo Cruz. 2018. Patient and consumer safety risks when using conversational assistants for medi- cal information: an observational study of siri, alexa, and google assistant. Journal of medical Internet research, 2011
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specifie lan- guage model pretraining for biomedical natural lan- guage processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1-23.
Anna-Bettina Haidich. 2010. Meta-analysis in medical research. Hippokratia, 14(Suppl 1):29.
Elliot Bolton, David Hall, Michihiro Yasumaga, Tony Lee, Chris Manning, and Percy Liang. 2022. Stan- ford crfm introduces pubmedgpt 2.7b.
Will Douglas Heaven. 2022. Why meta's latest large language model survived only three days online.
Virginia Braun and Victoria Clarke. 2012. Thematic analysis. American Psychological Association.
Julian PT Higgins, Jelena Savović, Matthew J Page, Roy Elbers, and Jonathan AC Sterne. 2019. Assess- ing risk of bias in randomized trial. Cochrane hand- book for systematic reviews of interventions, pages 205-228.
Aaron M Cohen, William R Hersh, Kim Peterson, and Po- Yin Yen. 2006. Reducing workload in systematic review preparation using automated citation classifi- cation. Journal of the American Medical Informatics Association, 13(2):206-119,
Bodil Dalsgaard Hoffmeyer, Mikkel Zola Andersen, Siv Fonnes, and Jacob Rosenberg. 2021. Most cochrane reviews have not been updated for more than 5 years. Journal of evidence-basee medicine.
Deborah J Cook, Cynthia D Mulrow, and R Brian Haynes. 1997. Systematic reviews: synthesis of best evidence for clinical decisions. Annals of internal medicine, 126(5)3376-33)
Brian E Howard, Jason Phillips, Kyle Miller, Arpit Tan- don, Deepak Mav, Mihir R Shah, Stephanie Holm- gren, Katherine E Pelch, Vickie Walker, Andrew A Rooney, et al. 2016. Swift-review: a text-mining workbench for systematic review. Systematic reviews, 5:1-16.
Juliet Corbin and Anselm Strauss. 2014. Basics of Qual- itative Research: Techniques and Procedures for De- veloping Grounded Theory, fourth edition. Sage pub- lications.
Siddhartha R Jonnalagadda, Pawan Goyal, and Mark D Huffman. 2015. Automating data extraction in sys- tematic reviews: a systematic review. Systematic reviews, 4(1):1-16.
Gordon V Cormack and Maura R Grossman. 2016. Engineering quality and reliability in technology- assisted review. In Proceedings of the 39th Inter- national ACM SIGIR conference on Research and Development in Information Retrieval, pages 75-84.
Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2019. Clef 2019 technology assisted reviews in empirical medicine overview. In CEUR workshop proceedings, volume 2380, page 250.
Adam Corner, Nick Pidgeon, and Karen Parkhill. 2012. Perceptions of geoengineering: public attitudes, stakeholder perspectives, and the challenge of 'up- stream' engagement. Wiley Interdisciplinary Reviews: Climate Change, 3(5x4511446.
Eric W Lee, Byron C Wallace, Karla Galaviz, and Joyce Ho. 2020. MMiDaS-AE: multi-modal miss- ing data aware stacked autoencoder for biomedical abstract screening, In Proceedings of the ACM Con- ference on Health, Inference, and Learning, pages 139-150.
Jay Young Iz Beltagy, Madeleine van Zuylen, Bai- ley Kuchl, and Lucy Lu Wang. 2021. Ms2: Multi- document summarization of medical studies. arXiv preprint 2rXiv:2104.06444 104.06486.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel. and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786.
Randall C Gale, Justina Wu, Taryn Erhardt, Mark Boun- thavong. Caitlin M Reardon, Laura J Damschroder, and Amanda M Midboe. 2019. Comparison of rapid

---

Gary Marcus. 2022. Close! galactica is dangerous be- cause it mixes together truth and bullshit plausibly & at scale. you thought that nobody would notice care, and then took your own judgments about risk as the only ones that were relevant. it's that last part that comes across as arrogant. https: httts::/ttwiteett tter. com/ GaryMar wttpccusss/tt/////tt//tsstsss (aAaUu//1554222256662226666 cessed: 2023-2-9.
to bringing us closer to systematic review automa- tion? Systematic Reviews, 12(1):72.
Maribeth Rauh, John Mellor, Jonathan Uesato, Po-Sen Huang, Johannes Welbl, Laura Weidinger, Sumanth Dathathri, Amelia Glaese, Geoffrey Irving. Iason Gabriel, al. 2022. Characteristics of harmful text: Towards rigorous benchmarking of language models. arXiv preprint arXiv:2206.0823
Ac-
Jain J Marshall, Joël Kuiper, and Byron C Wallace. 2014. Automating risk of bias assessment for clinical tri- als. In proceedings of the 5th ACM Conference on Bioinformatiis, Computational Biology, and Health Informatics, pages 88-95.
Laria Reynolds and Kyle McDonell. 2021. Prompt pro- gramming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Comput- ing Systems, pages 1-7.
lain James Marshall, Veline L'Esperance. Rachel Marshall, James Thomas, Anna Noel-Storr, Frank Soboczenski, Benjamin Nye, Ani Nenkova, and By- C Wallace. 2021. State of the evidence: survey of global disparities in clinical trials. BMJ Global Health, 6(1 6(00:504
David Sackett, William MC Rosenberg, JA Muir Gray, R Brian Haynes, and W Scott Richardson. 1996. Ev- idence based medicine: what it is and what isn't.
Malik Sallam. 2023. Chatgpt utility in healthcare edu- cation, research, and practice: systematic review the promising perspectives and valid concerns. Healthcare, volume I,, page 887. MDPI.
Adam S Miner, Amoll Milstein, Stephen Schueller, Roshini Hegde, Christina Mangurian, and Eleni Linos. 2016. Smartphone-baset conversational agents and responses to questions about mental health, interpersonal violence, and physical health. JAMA internal medicine, (6655)=61-6255
Michele Salvagno, Fabio Silvio Taccone, Alberto Gio- vanni Gerli, et al. 2023. Can artificial intelligence help for scientific writing? Critical care, 27(1):1-5.
Makoto Miwa, James Thomas, Alison O' M'Mara-Eves, and Sophia Ananiadou. 2014. Reducing systematic review workload through certainty- based screening. Journal of biomedical informatics, 51:242-253.
Kaveh G Shojania, Margaret Sampson, Mohammed T Ansari, Jun Ji, Steve Doucette, and David Moher. 2007. How quickly do systematic reviews go out of date? a survival analysis. Annals of internal medicine, 147(4):224-233
Cynthia D Mulrow. 1987. The medical review article: state of the science. Annals of internal medicine, 106(3):485-488.
Karan Singhal. Shekoofeh Azizi, Tao Tu, S Sara Mah- davi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022. Large language models encode clinical knowledge. arXiv preprint 2:X.::2212.113:..
M. Hassan Murad, Noor Asi, Mouaz Alsawas, and Fares Alahdab. 2016. New evidence pyramid. BMJ Evidence-Baan Based Medicine, 21(4):125-127
OpenAl. 2022. Chatgpt: Optimizing language models for dialogue.
Beck Taylor, Catherine Henshall, Sara Kenyon, Ian Litchfield, and Sheila Greenfield. 2018. Can rapid ap- proaches to qualitative analysis deliver timely, valid findings to clinical leaders? a mixed methods study comparing rapid and thematic analysis. BMJ open. 8( (10):e019993.
Mourad Ouzzani, Hossam Hammady, Zbys Fedorow- icz, and Ahmed Elmagarmid. 2016. Rayyan-a web and mobile app for systematic reviews. Systematic reviews, 5:1-10.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:221 1.09085.
Nick Pidgeon and Tee Rogers-Hayden. 2007. Opening up nanotechnology dialogue with the publics: risk communication 'upstream engagement"? Health, Risk Society, 1221191-210.
Jennifer Preece, Helen Sharp, and Yvonne Rogers. 2015. Interaction design: beyond tion. John Wiley & Sons.
Jeffrey Unerman. 2010. Stakeholder engagement and dialogue. In Sustainability accounting and account- ability, pages 105-122 122. Routledge.
interac-
Katyanna Quach. 2020. Researchers made an openai gpt-3 medical chatbot as an experiment. told a mock patient to kill themselves. The Register.
Maurits Van Tulder, Andrea Furlan, Claire Bombardier, Lex Bouter. Editorial Board of the Cochrane Col- laboration Back Review Group, et al. 2003. Up- dated method guidelines for systematic reviews in the cochrane collaboration back review group. Spine, 28( 12): 1290-1299.
Riaz Qureshi, Daniel Shaughnessy, Kayden AR Gill,
Karen Robinson, Tianjing Li, Eitan Agai. 2023. Are chatgpt and large language models "the answer"

---

Cecilia Vindrola- IndroolaPPadros and Ginger Johnson. 2020. Rapid techniques in qualitative research: a critical review of the literature. Qualitative health research, 30(10):1596- 1694.
published systematic reviews at the time of recruit- ing, members of evidence-based medicinc/medical systematic review communities that the authors of this paper were also part of, or were recommended by the participants themselves. We emailed a total of 40 domain experts for recruitment.
Byron C Wallace, Johl Kuiper, Aakash Sharma, Mingxi Zhu, and lain J Marshall. 2016. Extracting pico sen- tences from clinical trial reports using supervised distant supervision. The Journal of Machine Learn- ing Research, 17(1):4572-4596.
A.2 Background
We interviewed 16 participants who worked searchers or methodologists (11), academics (9), journal editors (5), clinicians (4), and guideline developer (1) and came from the USA (9), UK (3), Australia (2), China (I), and Greece (I). All participants had contributed to multiple system- atic reviews and meta-analyses as authors, advi- sors, or reviewers. Eight had contributed to more than 100 reviews, four to 25-100 reviews, three to 10-25 reviews, and one participant to <10 re- views. The systematic review topies that partici- pants had previously worked on spanned a diverse range of subjects. including nutrition, vaccines, mental health, medical nursing, ophthalmology, pe- diatrics, women's health, cardiovascular diseases, toxicology, drug therapy, and sexual well-being Table 4 reports participant characteristics.
Byron Wallace, Sayantan Saha, Frank Soboczenski, and Iain J Marshall. 2021. Generating (factual?) narrative summaries of rots: Experiments with neural multi- document summarization. AMIA Summits on Translational Science Proceedings, 2021:605.
Byron C Wallace, Kevin Small, Carla E Brodley, Joseph Lau, and Thomas A Trikalinos. 2012. Deploying an interactive machine learning system in an evidence- based practice center: abstrackr. In Proceedings of the 2nd ACM SIGHIT international health informat- ics symposium, pages 819-824.
Byron C Wallace, Kevin Small, Carla E Brodley, and
Thomas A Trikalinos. 2010. Active learning for biomedical citation screening. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 173- 182.
Lucy Lu Wang, Jay De Young, and Byron Wallace. 2022. Overview of MSLR2022: A shared task on multi- document summarization for literature reviews. In Proceedings of the Third Workshop on Scholarly Doc- ument Processing, pages 175-180, Gyeongju, Repub- lic of Korea. Association for Computational Linguis- tics.
Only one participant (P4) had some experience using an LLM (ChatGPT) for tasks related to sys- tematic reviews. Six participants have used Chat- GPT but not for systematic review work. A pro- fessional journal editor (P10) had not used any AI systems for their work. Most participants had used some sort of Al tools to aid systematic review- ing for tasks such as abstract and article screening. and for assessing study risk of bias (Higgins et al., 2019). These specialized tools offer some degree of AT-based assistance for things like elassifying abstracts and extracting data; none attempt to gen- erate review drafts using LLMs.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang. John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, al. 2022. Taxonomy of risks posed by language models. 2022 ACM Conference on Fairness, Ac- countabiliry, and Transparency, pages 214-229.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In- ternational Conference on Machine Learning, pages 12697-12706. PMLR
B Interview Question Guide
A Participant Recruitment and Background
The below questions were asked during the semi-structured interviews with the domain experts to gain their perspective regarding the uses and harms of LLMs for medical systematic reviews. The questions were divided into broad categories: Background, Previous Experience Using AI Thoughts on Outputs from LLMs, and General Thoughts on LLMs for the Task. Following the nature of semi-structured interviews, we asked follow-up questions or skipped some questions when appropriate given the content and context of
A.1 Recruitment
Between March and April 2023, we emailed do- main experts in evidence synthesis inviting them participate in a non-compensated remote inter- view conducted over Zoom. We took an intentional sampling approach, reaching out to experts with diverse levels of experience with medical system- atic reviews. These included: authors of recently

---

Participants
6. How important do you think this ror/risk/harm for you?
Location
USA
Do you think it involves any physical and/or mental health harm?
UK
Australia
China
Greece
General Thoughts on LLMs for the Task
Background
How do you think large language models could be used for the process of writing sys- tematic reviews?
Rescarcher/Mcthhoollngit
Professor
Journal Editor
Clinician
2. Do you think large language models can be deployed for general public use to access sys- tematic reviews? Or do you think they should only be used after having a domain expert review the outputs?
Guideline Developer
Number SRs
100+
26-100
10-25
Do you have any concerns using large lan- guage models for writing systematic reviews?
Past Al Experience
Abstrackr (Wallace al. 2012)
RobodReviewer (Marshall al.,
4. Do you think there are any downstream harms?
2014)
DistillerSR
Rayyan (Ouzzani al., 2016)
5. In ideal world, what would you want from AI system that can help you write sys- tematic review and feel more comfortable in using systems like these?
ChaiGPT (OpenAI, 2022)
SWIFT-Review (Howard 2016)
Cochrane automation tools
Table 4: Backgrounds of interviewed domain experts.
LLM-Generated Medical Evidence Summary Outputs
the interview.
C.1
Generating LLM Outputs
Background
We collected the most recently published Cochrane reviews for each medical topic at the time of our search on February 23, 2023. We removed dupli- cates from the collected 37 reviews, resulting in 32 reviews. We generated total of 128 outputs using the collected titles as prompts Galactica (with two prompting approaches), BioMedLM,, and Chat- GPT. Full lists of the 37 medical topics, 32 review titles, and the 128 LLLM-generated text outputs are available on our our GitHub repository.
1. How many medical systematic reviews have you done or contributed to?
2. What kinds of contributions have you made to reviews for biomedical literature?
Previous Experience Using AT
1. Have you ever used AT to help you screen articles or draft systematic reviews?
For Galactica, we used the python package from GitHub and used their sample code for generating paper documents with top_p=0.1 and max_length=2048. For BioMedLM, we used the Huggingface" model with the parameters max_length=1022 and top_k=50. For ChatGPT (Feb 13 version), we used the web demo provided by OpenAl". We generated the ChatGPT outputs before the API became available.
2. Tell me about your experience using Al for this specific task?
Thoughts on Outputs from LLMs
1. What stood out to you about the model- generated text?
2. What did you like most about the output? What attributes did you find to be useful?
3. What did you like least about the output? What attributes did you find bothersome?
The prompt templates generate reviews using LLMs are provided below. Our prompts
4. Are there parts of this output that can help you with writing systematic reviews?
'https://githhb. www.//pppeswithcoe/eeeeeeoeeeeee hhttps://huggingface co/ stt/sfffoodddrffl BiaMedLM
5. Are there any potential errors or risks or harms you see in the output?
hhttps://chtt.opee

---

are intentionally simplistic, our goal is to gather qualitative feedback on broad thematic areas. Our approach also permits an ecologically valid evaluation of how LLMs might be used by domain experts. We used two different document generation prompts for Galactica as we discovered that they produced very different outputs, where the prompt that starts with # generates in-text references reliably than the prompt that starts with Title. (Review Title} was replaced with actual titles of Cochrane reviews. The full code for generating the outputs also available on our GitHub repository.
producing systematic reviews, P15 who is clin- ical researcher and professor said, "[LLMs can] do the hard work, searching the literature, find- ing the right papers, and the papers are machine readable, extract the data in reproducible ways into tables, and then it will be up for the expert to con- duct the right analysis of the methods. Use the data in ways that are answering the questions of the systematic review. So that's where the utility in accelerating those painful steps of compil- ing the literature, finding the papers, obtaining the data from table spreadsheets, whatever they may be, and producing analyzable tables. dunno if that's aspirational, but that's really where see the value." Table 8 gives definition for each category of what could make experts feel more comfortable using LLMs for medical systematic review process, and exemplary quotes.
Galactica
Title: (Review Title)An\n
(Review Title)An\n
BioMedLM
Title: (Review Title}
ChatGPT
Give me a review on (Review Title}
C.2 LLM Outputs Analysis Results
Table provides the 11 general concepts identi- fied during the qualitative analysis of the LLM- generated outputs, accompanied by description and an example for each concept.
C.3 Sample LLM Outputs
A full list of outputs shown to participants during the interviews is publicly available at this website. Table provides the pre-selected outputs on various medical topics.
D Additional Codes and Representative Quotes
The tables below provide additional codes and rep- resentative quotes from participants that have been identified part of the study. Table 7 gives defi- nition for each category of potential uses of LLMs that are sonmammmaiization tasks for medical sys- tematic reviews, and exemplary quotes. Partici- pants found LLMs to have potential uses when comes to automating some of the mundane tasks of

---

Concept
Description
Example
Occurrences
cycle bresthing technique cystic brosis Abstract: The active eyele of breathing technique (ACBT) relatively technique which aims reduce airway abstrnction and to improve ventilation in patients with cystic fibrosis. This urticle describes the ACBT, theorcriaal
Outputs that either under 100 words, include only hasic intrddaction, missing conclusions. missing references for the citations, incomplete
Incomplete or Short Outpuls
effects on airway obstruction, lung fanction and clinical outcomes in patients with cystic fi-
Model-geweran geweraaed conclution:
Contradictory
Mindfulness based interventions have shown posilive fects on menial well being in students and junior doctoes. Hlamun-uritten conclusion:
Statemenis Wäthin or
Outputs report contradictory ties or do mot align with the conclusions the human-written abstract
107
Compared Ground Truth
effectiveness remains unconfirmed.
our target population
The meta-analysi found anteratal dietary supple- mentation with myo ioosint was associated with sig- nificunlly lower risk of developing GDM compured placebo or standard care (RR 0.69, 95% CI 0.0.52.0.92, p= 0,D1).
Outputs with any numerical values such p-values, risk ralio, difference, AUC, and ROC
Numerical Values
103
Ourputs that reviews (abstracts single studies, clinical trial information, review
Hailaciiated reforence:
Undosirable Ourputs
Effect of mirabegron overactive bladdr symptoms Japanese parients: 12-week, open-lahel, multicentre study. Kajiwara
and responses), hallocinated (fake) references, or mis- spellings
Ciaation & References
Outputs with references, footnotes, cita- tions, URL
hitps://ww ema curope ow/en/nwws prest-rele dlesses/nwws22230
Agrecment with Ground Truth
Outpuls with similar conclusions or have reported the same namher of studies ground
tients (1050 vascular and 453 endoviscular) Human writte: We included RCTs with 3393 partici- panis aalyysdd...
Model generated: We included
pa-
27
Ourputs that mention the year of the snad- or references
Methods: systemaric search was performed in multipls databuses up
Time
2021.
Proper Names & Fensonalyy Mentiitable Information
Outpuls names suihors, reviewers, emails, or jouraal names
Autbors: G, Zhang Z, Ziang H, ctal.
Tite: Medication review in hospitalised patients to ro- duce mochidity and mortality: A systematic review and analysis... For many years, lapuroscopic createctomy (LDP) has been performed as an alternatvee benign lesions non-parasitic
Unimportant Additional Information
Outputs with additional information that is not directly related to the topic or not useful supplemental data
pan-
surgical method henign eysts...
# Single visit
Outputs repeated sentences con- secutively repeated references without any other
Multiple visits
Repetition
e Single visit
Multiple visits
Figure 4: A, Forest plot showing the overall response
Visuals (Figures Tables)
Ourputs with eaptions figures tables
B, Forest plor showing the overall survival for necadjuvant treatment
nenadjivant
Table 5: Descriptions and examples 11 general concepts identified during the qualitative analysis the LLM- generated outputs. The number of occurrences refers the number of times these concepts were found in the outputs.

---

Model
Output
Gensionl dakktte mellins (GDM) is common complication of pregraccy that can lead alverse maiznil and ocosstal oungoniss Myn-inosital is mitraaly rxcarring sugat alcuhol that hos hern propused ae potentii distury supplemen for proseting GDM, Tleee ic review interatal chetary sugllmmemation with myn-inosine for preversigg
diaheses:
Tidk: Antinncal cietary supplementaioon controlled cizls
myo-inosiul for preseningg gedational disbesss: a sysiematic and meta -ntlaii randomized
Authore G. Zhang Z.
Zhong II, et
Publistte is. Jounal of Ubsectics and Gynapeology Ressurch (2021)
Overview: This review simel to evaluit the effectivence and suftt of intennal dictury suplamemttion with mys-inncise for prevaningg GDM. The authers conducted a systematic search of miltipp dushases and stemihed right candonized comtinled mas incoviig 1,511 pregtant wourn. The studles comperal the of myo-inciiul epplements to placoho cr standied and rosessal the incidence of GDM as the
ChaGGT
anteome.
The aitiiit found that inienatal 0.60 95%
appleseatdiin mvo-innissl
associated with sigaifleaaly
of developing
GDM companad to plactbu sombard presemn delivery in mşu inuetel goup compared betwam grops in trrms other securdary noltames incluling Nirthooight, made delivery, or nocuial vaterms The review did not idlmnrl any serious atverse events assocased wit mye inocted supleeeerctionn
P- The suthess also found signifiznnt reluction contol group (RK 0,39, 95% 010,15-0885 0.02) No signficam diffeeences were found
Coschsion. The authers conclided that attenatal dietayy supelencmation with myo-inocitel max be effective is reducing the risk of GDM and pecteras delivery, Howeever, further studies are needed to costinm these fiddigg and determine the optimal dosage and duratinn of myo inositol supplememation. complications proterm
Intaveetrinnlr hasmumhage (IVI1) incepullopsihy premccarity (EDP) neurological dusbilities Stem based nteeventions hove emerged is potental thenpy for these conditions Here is resiew of the evidence sem well-hased intrrmation for the proveniion and becammen [VI EOP in protemn infors:
lesd vignificant
Titl:: Stem cell based intervnnions for the prevention and treamment of inmaeedraaiia hncmoothuge and encephalogathy of premonurity in prstersz infares systematic review and mete- inalsss
Background: The zim of this systematic coveew and meta andyyss as was to evaluzze the sabery and efficacy of stem cell based intervations for the prevestion and teesmmentt of IVH and EOP in pecteri infants. Methods; A call-tuned inervention with placcho standand care for presention was
performeed suhiple electronic dasabases up to March 2011. Randonized controlled titls (RC comparing IVH
pretern infart inclued includdd sudies, primary outzome was incidenor
risk
ChaIGT
access quality were monaalty, senredeoekpmeenall onteames, and sbvens events
BOP and secondary ouccomes
Resuin: A unal of RCTs invlving 1153 presern infants were included in the meta analysis Sien cell- intervntioons included mesenchymal sem cells (MSCS) acural stem cells, and unbiical cood bloed cells. The disk of bass was high in most of the studes. The mato analyss slowed no signidicar difierence the incidence of IVII or EOP between the stem coll-based intervention group and the control group inik ratio [KK] (192, 99% confidence interval [CT] 0,78 189). There was also to signiheart differencs morallit (XR 0.92, 95% CI (.33-1.33) adverie events (KR 0.9% US% CI 0.75 1.21) betwon Bhe
However, siem coll-hesal intevvntions wer asocisted with sigilicantly rolucod risk of comhhal pulyy develepmental delay 0.58, 95% 0.40-.555.
du have siprificant offect the presention IVH or EOP in prsterm infutss However, these intereentoon may imppove loeg-term neurddeeeopmmitt outcomes these infunts Further ligh-qualit RCTs ane roaded to confirm these findings and to dutermine the opeimal timing doss, and route of adminioteiion stem cell-hoseed intervmtions.
Conclusion: camest evilence sugress
-leusll interventios
for treating overactive Stadder in adults
Introslaction
Overativv bludder (OAB) syaptom complex uristry segeney, csullly with frequency and necturia, with without inceutinance. the absence of innnary infoction
paihnlogy [1) has neuntad 10%
98 man (2). DAB
sigsiftcant moshidity. roduced quality life, sigmificai healiheare costs [3], is major bealthear problem many countries
The cument phammazologicaal trcalmen: of DAB inclules mimuseaniic agenis bets-3 almmooppr agansts, and alpha- bicckers (4) chinical practics.
these drugs are used for the ueatmenn of OAB, ahbough their efteaay has been questioned 15, 6). Purttermrre, long torm treatment with diese drugs ofuen associated with adverne effecs (7. 8). Therelone, other therapies are needed to mor OAB.
Acupuacoure is trdiioaal Chinese medicine (TLM) technque that bas been med for weeting OAB for more than 2000 years. Acupuneture is used traat DAB by siimulating specifie sespoints 19] The acupoins used acupuncture for neating OAB have heen imessignnd by diiiccal trials [10].
Galactica
These demosstrate acupuscture effective for ueating OAB, zithough efficacy been questioned [19].
this stnly we performed a syseestii review candomieel contilled (RCT))
the officacy safety scspancturn for teeating OAB
adults...
Refenemazs
[]] An international umggnecoloicall association Vittematio continence society (IUS) joint repoet on the terminoloyy for female pelvic floor dysfinction, Baylen
[2] Prevalence of overactive bladder and other lower unnary tract symptoms: results of the UPIC study, Irwin
[3] Economic aspects of ovenative bladte: the Dunish burdes of incoutinence stayy (DANBIO), Milsom
[4] Clnicall gusdeim for ovenacive bladder gyndrome, Yanagucho...
Tel:: Brain natriunciic pepiide Vateminal besin natrisnctic peplide for the diagnacis of harmalynamicaly significnt puieni ductos ancrrious
presem secrates
Abrrna: Back grondd and sime: The sim of hiss stuly was to evalnnte the digguescc acrunayy of plee bnin nattomnii popiike (BNP) and N Nateminal
tras natriunttic pepeide (NT proBINe) (NTppou
presern necnates wish patent ductus antrioses U'DA). Methods: Thiry four peeterm nconates with a clinical
diagancs of PDA were included the study. The pulients were diided into two incouded 17 presern measured using enzyse-linkel linkel nomenoeeeeetttoo assay (ELISA) Resllt: Plasna BNP and NT-poRNP levels were significamlly higher is presern secnates IsPDA thmn those without PDA ip 0.001. RNP and NT level hula a good diagsostic accursyy for the delectins LuPA prsterm nesoutes (seca under the curve [AUC]] 1851. p c0000 and AUC: 0.366, p< 0.001, repeemell).. The uptimal cut-off valuec of plaems INP and ye livels for the detection baPDA ware 585 pphml. bessitivity 76.5%, specifiity 92.4%) and 589 polnf. (sencifivity 70.6% spacificity 82.4%), nopucivelyy Conchsiine Plasma ENP and NT-pmBBP levels can be used a dingrostic tool for bsPDA in presenn neorates.
acconding their clinicll and aehocadiognghiic resalls. Group I with hamoodmunicaaly significant PDA (IsPDA), and Group included
Golactica
with PLM BNY and
NT-pooENP
reeivve opecating chiracteriric (BUC) curves showed that plasma
Tilk: Theatment of demal onhodmntic complictions Balasssenia
Abstract: Summary Thalassaemia is commoo interited disorder Southeast Asia. Thelnssammia paticuts oftea presest wide ragge dennal and oethoduncic complication. These may he resull their disesse proezss the effects iron chelation therapy or effecte of their insstments. Management of thalassemmia patients with dental and onhodomic problems requrres a miltdissiplinury appeeoch. This paper aims review the dental and or orbuudmiic complisaiion of thalesssmii and the incaiment apresche to those complication.
Galactica
Tile: Calcium and vitasin for incresting bone nincral Seasity peomenopuusal women candomised controlled tial study parformal South India Won Chui BMD and found that adlition calcium vitamin D regular calcium suppieeentaiion incressed the lunbar spine BMD only tie vitamis D-upplemenni grop, bui in the cilcim--rgelamnntt group
(201 I
evalusie the affect of cakiuum vitamin suplementation the
The same cunclusoo was drawn in another sndy by Wen Sun Choi "et alt. 12011) [24] as compared to the women who only received calcium upplementtion Howner, a stily un totd of 236 premenggaveal women performmal by Koon Sik. You and Won Sin Choi (2009) [26] cuncluded that cakiuu suplommentiin did not help improving the BMD. This andy included 123 patient undergoing assisted reproductive technology (ARI) and itss womes with an underiying medical condtion ic palycystie owalls. The mait limitation this study that they adminisened celcium supplaments BMD smpro
BioMedLM
The doses 500 mglday: however, the optimum review caliims suplemmmmtiion showed there assufficiest evidence support the benefiis calcium supleeeenttiion health [19], However there was a statstically and claniaally significant improvenent the BMD a perlod 1:2 years of calciua sugplenestution. However, spplenentttionnco vitunin abong with calsium spplomerttion would be more effective messure than calsium supelematcttion incresse BMD [19). The recommmmeed duily allowance (RDA) of calcium is less thmn 500 mphhy and the musemmentted duily allowamee (RDA) vitamin is less thin 300 Iilday for Indis wamen [19) For the mejoriy of South Aviass, the vikamin D iauke in spite of a higher imtike of foodstulls is for below the sufficient level (19)
frequency for calium suplemmettaion
remains still undelined.
sisgle stal from Nigeti assessed the effect of the supplentemation of 300 ma calcium per day with without 400 IU vitumic per day on the BMD of 368 women. 20 years apo, and found significmi improvement BMD of femar and obis after supplementtioon vitamin irrespectve the saleium spplecscatution (27).
Table 6: Examples of LLM LLM--enerated reviews shown to participants during the interviews.

---

Potential Use
Description
Quote
LLMs for refining research questions and topic in beginning systematic review process
".. for the initial discussions where the topic refinement is done, so topic refine- ment, horizon scunning. scoping, port available, and crosschock data" epidemiologist with clinical background and professor in evidence synthesis (P5) "The contructing goverament agency olten gives about week to reply with a proposal, a competitive proposal, and now might actually be useful... And might be actually a way, way generate funding proposals... But they often have general understanding the topic
Rewerrh question refinement
LLMs for penerating competirive fund- ing proposals for contracting government agencies
Generate fundiig proposals
two phases. is sert of undersiund the potential challenges and controversies in specilic area. And then the third probably less useful, but you could model what are the challenges and controversies in this area?" clinician and researcher in evidencv synthesis (P6)
next phase
rhink potentially with nssisting with search terms well, developing your search strateyy and sugpesring synonyms. And maybe it can even draft first search strategy. which you could review discuss
Generating search stringskarategies
L.Mss for penerating search strings search strategies
an information scientist as well." - professor and reseurch methodologist (PI1)
"And because with computer could tiralesely identify potential lozation of the information in a puper and then that can be highlighted and then the human can then verify the veracity of such information and approve such data be extracted. would expedite things. in some sense
LLMs for extracting impertant and rel- data from text of stadies that useful for systematic reviews
Data extmaction
analogy massive language model ourput and then verified it by human." epidemiologist and clinician (PI)
's's also seen it used elearly enough for writing code and staff like that. People have acked it how to data R, and it's cranked out some decent formulas." reseurch methodollgiit (P7)
M. for generaring R python cade conducting analysis
Generating analysis code
"I dunno would he able to check consistency because that does, some of numbers appear multiple places. So you' 'll have figures, you 'll have it in resalts, you'll il have in the abstract, the interpretations going to he in the conclusion. So there could in an arricle. Could do something around consistency and making sure that these numbers comsistent." former professional journal editor and guideline developer (P2)
LLMs for checking bias or inconsisten- hummn--wiittn drafts systematic reviews
Binn/comsistency reviewer
places
number going to appear
"There are only so many ways you can generate forest plot, and that's soet of seems the of task that might accessible and probubly betler some bured person duing it al the last minute hoping nobudy ever sees it. So that would that comes mind"" clinicion and researcher evidence synthesis (P6) "Mayhe ChatGPT to generate guidance and see how concordant or discordant it is... could be any blood pressuee control reommenddaioos and whether it's concordant, discordant things that affecting people's i life. impactful treatments or interventions or public health preventive measures that are going impact people's while those large language models be able respond to prompts that are consistent concondant with the majer guidelines that should hased cystematic reviews?" epidemiologiat and professer (P4)
Alternative for graphs
LLMs generating alterativv texi graphs
are the most
Generating guidelines
LLMs for generating medical guidelines.
"The reality of most systematic reviews only consider articles published in English. but we often recognize that there be content that's missed is result of limiting language.
LLMs for finding and including nen- English studies if I.I.M tainndd on non English text.
Including non-Fnglish studies
yeah, you might these langunge learning models" " professional journal editor (P10) "I'd like to say could have, might very helpful ton-Einglish speaking writers of English reviews in this example, to write coherent reviews." professor and methodologist (P13) "I don't know if this possible, but you actually pul in the studies and then could do some soet of narrative summary those sradies in tenms of where the studies were conducted, what interventions were assessed,
thought that that
strength
Helping non-native Riglish writers
English writers to write English systematic re- views.
LLMs for helping
L.Mss for creating annotated bihliogra- phy when studies details are provided as
Annolated Bibliography
outcomes were assessed, those types of details, like a slight, almost like an annotated bibliographe that could potenrially be useful well." professor and research methedalogist (PII)
inpur.
Table 7: Potential uses of LLMs (aside from just summarization) for medical systematic review process and exemplary quotes from participants.

---

Criterion
Description
Quote
By having LLMS produce summarics of known provenance (i.c. knowing that text genuinely derived from the presented references, which in turn uinely reflect well-chosen urticles), people can have increwed trust in outputs the system.
"This the black box. Obviously, this would be useless unless you had a citation to the specilic roview, the spocifie paper that chose." clinician and reseurcher synthesis (P6)
Known provenance
existing
"When doesn't it know, it makes stuff up. And that has to he checked, course. And the question is whether that checking will easier hard, harder than doing it yourself. And I'm guessing that at ceriain point it might be eusier. but I'm not sure." clinicisn, professor, and researcher (P14)
Al systems should allow humans to efti- ciently and eusiy verify quality inputs and outputs.
Allowing efficient verilication of oulputs
Domain experts verifying the accuracy outputs the- loop)
Medical domain experts or subject maller experts needed be able to fully cross correct.
specilies of cither the discase or the treatment or the test. Right now I lack that aspect. So do know exactly whether certain things make and would. if just read even the things that have questioned about, I would not know clinician (PI)
domain expert
knowledgel able]
outputs safe
whether is right wrong" opidemiologist
Giving more guided procppts (specilic populations, interventions, comparisons, and outcomes) or carefully selected stud- les as inputs can incresse contidence in LLMss for medical systematic viows.
review. The analyses. Have all that data available and then limit the writing of the absrmet review.
"I think you could systematic review full and or sections
Providing more guidance to LLMs
ideniiiied during meihodological process then as an editor would be much happier for that to come to my journa and then review in that. But yeah, that would profossional journal editer and guideline developer (P2)
confidence. I think." former
Table Table summarizing what could make experts feel more comfortable using LLMs for the medical systematic review process and exemplary quotes from participants.


Reviews:
[{"review_id": "xvt5fUupgK", "reviewer": "Reviewer_w8DF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive, thoughtful review and for acknowledging how our findings provide valuable perspectives on the benefits, harms, and risks of using large language models (LLMs) for medical systematic reviews. Below we address the main concerns raised about our work.\n\n**Comparative Analysis of LLMs**\n\nWe agree that conducting a comprehensive quantitative comparative analysis of LLMs in this application space would be a valuable exercise. We also agree that the performances of different LLMs will likely differ due to various factors (training data, model size, architecture, etc.). However, this qualitative work is not intended to provide such comparison; rather, here we are looking at LLMs as a general new technology with associated risks and benefits which we believe will be similar for all current state-of-the-art models. To this end, we used 3 representative LLMs to generate examples to catalyze discussion in our interviews with experts, as many have not used or seen outputs from LLMs for the medical systematic review generation task. We highlight that our main conclusions call attention to issues (and opportunities) across all current generation LLMs.\n\nAs we do not aim to provide a direct head-to-head comparison of models, we have explicitly noted this in the Limitations section: “Since this study was not a performance evaluation of LLMs but an exploratory study to understand the views of domain experts, further empirical studies are needed to actually evaluate the effectiveness of LLMs for medical systematic reviews.” In sum: comparing different LLMs for this task would be a useful undertaking, but it is not the focus of this qualitative work which aims to assess expert perceived potential utility and risks of LLMs—as a general technology—for systematic reviews.\n\n**Small Number of Samples**\n\nWe used a qualitative method, aiming to discover a diversity of themes (rather than aiming for statistical representativeness). Qualitative research allows for richer, more detailed analysis of unstructured data from a smaller number of participants. Our sample size is typical for this methodology.\nDomain expert time is scarce, and in-depth interviews about references and automatically generated evidence summaries are inherently time-intensive and therefore limit the possible scale of the analysis. We acknowledge that conducting additional interviews might have generated additional themes.\n\n**Very Basic Prompting Technique**\n\nWe agree that our basic prompting strategy might have failed to leverage the full potential of LLMs for generating medical systematic reviews. However, our focus here was not on performing a comparative analysis of prompting strategies; our aim was merely to provide illustrative examples to familiarize the experts with typical outputs, shortcomings, and strengths of current generation LLMs. Our conclusions highlight broad thematic issues found empirically in representative LLMs. We anticipate that improved prompts will not greatly affect these broad thematic areas.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors have engaged in insightful interviews with domain experts to meticulously evaluate the advantages and potential risks linked to the utilization of LLMs in the domain of medical systematic reviews. These interviews have yielded invaluable perspectives on a spectrum of aspects, ranging from the potential applications of LLMs in medical systematic reviews to concerns and potential harms tied to their unwarranted application. Additionally, the interviews offer constructive insights into strategies for enhancing the reliability of LLMs, thereby contributing to bridging the gap associated with their implementation. In each of these cases, authors have backed up their claims with examples from domain experts' comments.\n\n2. The authors have identified 11 general concepts characterizing model-generated summaries.\n\n3. The paper includes all the relevant details regarding recruitment of domain experts and interview process which is a great contribution to the research domain as it can be used as reference for further related studies.", "reasons_to_reject": "1. In this study, three distinct LLMs named Galactica, BioMedLM, and ChatGPT have been selected by the authors. The differences between these LLMs are outlined in the related work section of the paper. Despite their notable distinctions in training data and size, the evaluation of all these LLMs follows a uniform approach. A more effective approach would involve assessing the outputs of each LLM separately, as this could provide valuable insights into their relative performance in generating medical systematic reviews. It is plausible that certain LLMs might exhibit higher risk factors, while others could excel in generating coherent systematic reviews. In essence, this study would benefit significantly from a comprehensive comparative analysis between the LLMs, allowing for a more nuanced understanding of their respective capabilities, limitations, and potential benefits.\n\n2. The number of samples presented to each domain expert appears to be relative inadequate to draw definitive conclusions about the abilities and constraints of LLMs in generating systematic reviews. Additionally, during the expert interviews, the inclusion of human-written systematic reviews, without indicating their human origin, could offer valuable insights. This approach would allow observation of how domain experts react to these reviews, shedding light on the deficiencies of LLM-generated systematic reviews and thereby allowing a more comprehensive understanding of the lacking of the LLM generated review.\n\n3. The prompting technique used in this study is very basic and fail to leverage the full potentials of LLMs. Carefully curated prompts can gain better results in generating better systematic reviews.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper offers a comprehensive and in-depth qualitative analysis that delves into the systematic review generation capabilities of three distinct Language Model Models (LLMs). The approach adopted by the authors involves conducting interviews with domain experts, aiming to uncover the potential utilities and risks associated with LLMs in the context of producing medical system reviews. Additionally, expert opinions have been gathered to explore strategies for enhancing the reliability of LLMs for medical systematic reviews.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aZ8lA68I68", "reviewer": "Reviewer_UT4H", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive, thoughtful reviews and for acknowledging the novelty and contribution of our work. Below we focus on addressing the main concerns about our work.\n\n**Limited Prompting Experimentation**\n\nWe will provide more details on the prompt selection for each model in Appendix C.1. In regards to the lack of prompting experimentation, we are aware that our very basic prompting strategy might have failed to leverage the full potential of LLMs for generating medical systematic reviews. In this study, we did not focus on doing a head-to-head comparison of different prompting strategies but used basic prompts to generate outputs to demonstrate the full capabilities of the benefits and harms these LLMs can do. \n\n**Small Number of Samples and Participant Size**\n\nQualitative methods do not aim to provide statistical generalisability; and instead rely on in-depth, unstructured data from a smaller number of diverse participants, aiming to uncover a full range of themes. Our sample size is typical for the research method. We acknowledge, as with any qualitative research, that additional interviews may have generated new ideas or themes for analysis. Similarly, our goal in providing LLM examples was to familiarize our participants (most of whom did not have deep LLM experience) with typical outputs, limitations, and opportunities of these models. We do not aim to provide a definitive accuracy evaluation of these specific models; instead, we aim to learn about general issues in their application. We highlight that our conclusions focus on generally applicable issues, and complement other quantitative work on individual models/approaches.\n\n**Missing References**\n\nThank you for bringing these contemporaneous related works to our attention. We will add them to the Related Work section of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is novel as this study is the first attempt to characterize \"expert\" views on LLMs for the drafting of medical systematic reviews. \n2. This study has the potential of enhancing the reliability of LLM-generated evidence summaries. \n3. The article has a clear narrative and makes it clear to the reader what message the article is trying to convey.", "reasons_to_reject": "1. The authors tested three LLM models with various prompts, but details on selecting prompts for specific models and the process of prompting experimentation are lacking.\n\n2. The authors only shared a small number of samples (potentially just three) during the interview with participants, and one of these samples was chosen based on the participant's background. This limited number of samples and the small participant size could impact the representativeness of the findings.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper explores the potential of using LLMs to generate medical evidence reviews as well as highlights the risk of LLMs generating inaccurate or misleading content. To address this, the authors interviewed systematic review experts, who shared that LLMs could aid in drafting summaries and templates but expressed concerns about inaccuracies and reduced accountability. The paper contributes by outlining evaluation criteria for biomedical LLMs, aligning with expert perspectives.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[These are contemporaneous work, hence not required to be cited]\n\n1. Qureshi, R., Shaughnessy, D., Gill, K.A.R. et al. Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation?. Syst Rev 12, 72 (2023). https://doi.org/10.1186/s13643-023-02243-z\n\n2. Sallam, M. ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare 2023, 11, 887. https://doi.org/10.3390/healthcare11060887"}}, {"review_id": "FhPCl2pRxL", "reviewer": "Reviewer_VXn3", "summary": "", "questions": "Question A: Would you be able to release the interview data their codification, or other additional resources related to the interview data and its analysis without compromising the ethics protocol and participant consenting?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive, thoughtful review, and address the main concerns below. \n\n**Making Interview Transcripts and Their Codification Available**\n\nWe agree that releasing the interview transcripts and their codification would be ideal and would provide improved transparency for our analysis. However, as anticipated by the reviewer, this would compromise the protocol we presented to the IRB prior to engaging in this exercise. We designed our protocol this way to allow participants to talk freely without fear of views being attributed to their organizations (since many held senior positions in academia, guidelines, and evidence synthesis). That said, we will talk with our IRB to see if we can release redacted transcripts and their codification (after seeking permission from individual participants); but we cannot promise that these will be released at this time (as it will depend on their determination). \n\n**Limited References to Technology Assisted Review (TAR)**\n\nIn the paper, we focused on presenting our work as the first qualitative assessment of LLMs for the task of drafting medical systematic reviews with experts, which is complementary to prior work on technology assisted reviews. However, we are grateful for the additional pointers and will add a discussion of these prior efforts to further contextualize our contribution.\n \n**Presentation Improvements**\n\nThank you for suggesting concrete ways for us to improve the presentation of our work. We will make sure to spell out all acronyms when used for the first time in the paper and to organize the in-text citation lists in chronological order.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Likely to be an influential study going forward because the paper addressed the perceived utility and risks of the breakthrough innovation of LLMs to assist authoring medical evidence reviews\n\n* Excellent and insightful methodology: experts were interviewed\n\n* The paper is excellently written, and this writing demonstrated authors' methodological excellence in the review topic as well (e.g., LLMs, systematic reviews, and qualitative analysis)", "reasons_to_reject": "* Interview transcripts, their codification, or other additional resources related to the interview data and its analysis do not seem to be released together with the paper. They would have highlighted the methodological rigor, improved result transparency, and encouraged future work.\n\n* The extent of new insights related to the perceived utility of technology assisted reviewing (TAR) is quite limited compared to the prior work, noting that TAR protocols, guidelines, and shared tasks have been published and organised for at least 5-10 years", "questions_to_authors": "Question A: Would you be able to release the interview data their codification, or other additional resources related to the interview data and its analysis without compromising the ethics protocol and participant consenting?", "paper_topic_and_main_contributions": "This study guided the future of technology assisted reviews in health and medicine by interviewing international systematic review experts to characterize the perceived utility and risks of large language models (LLMs) in the specific context of medical evidence reviews", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "This study seems to miss many Technology Assisted Review (TAR) references. For example, you may find the following to be helpful:\n\nDi Nunzio, Giorgio Maria, and Evangelos Kanoulas. \"Special Issue on Technology Assisted Review Systems.\" Intelligent Systems with Applications (2023): 200260.\n\nKanoulas, Evangelos, Dan Li, Leif Azzopardi, and Rene Spijker. \"CLEF 2019 technology assisted reviews in empirical medicine overview.\" In CEUR workshop proceedings, vol. 2380, p. 250. 2019.\n\nCormack, G.V. and Grossman, M.R., 2016, July. Engineering quality and reliability in technology-assisted review. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval (pp. 75-84).", "Typos_Grammar_Style_and_Presentation_Improvements": "* Introduce all acronyms (e.g., LLMs in the abstract)\n* In-text citation lists should be organised from the oldest to the latest (and alphabetically within a given year) (e.g., \"(Murad et al., 2016; Cook et al., 1997; Mulrow, 1987)\" should be \"(Mulrow, 1987; Cook et al., 1997; Murad et al., 2016)\"."}}]

Author rebuttal:

Meta-review: This paper provides a qualitative study of how experts in systematic reviews perceive the potential uses and drawbacks of using LLMs to conduct systematic literature reviews in medical research. All reviewers agree that this is a well-written and well-motivated paper tackling an important issue in a robust way. The reviewers agree that the insights derived from the qualitative data are novel and insightful and likely to be impactful for future medical researchers and practitioners. I concur with the reviewers, particularly given the prevalence of medical domain experts (and domain experts from law, education, policy, etc) using LLMs to support research, it is critical to understand how such practitioners perceive the capabilities and limitations of LLMs for such tasks. 

Reviewers wanted to see a comparative analysis of the performance of different LLMs, and raised concerns about the number of samples, both of which the authors addressed in their rebuttal. The authors should add additional context about the qualitative data collection and analysis methods used to further justify the approach taken.

Reviewers raised concerns about the prompting approach taken, which the authors satisfactorily responded to by justifying their approach - that experts in systematic reviews who are turning to LLMs to support their work may not necessarily have expertise in prompt engineering, and thus a qualitative approach that maintains fidelity to the practitioners’ typical work practices is a reasonable approach to take (Salvagno et al., 2023), rather than one that may adopt state of the art practices in prompt engineering, which medical practitioners may not themselves adopt. To address this, I recommend that the authors better contextualize that the qualitative approach taken here is designed to have ecological validity with respect to medical practitioners’ use of LLMs for systematic reviews and the tradeoffs of such use within the social context of medical research practice, rather than making claims about the technical capability of LLMs divorced from such social context (where, for instance, using SotA prompting techniques and conducting comparative analyses of various model capabilities would be more appropriate). 

Reviewers raised concerns about the sample size, however, for qualitative interview studies such as this, such sample sizes are well within the bounds of acceptable sample sizes. Prior research suggests that sample size is not a sufficient grounds on which to evaluate rigor of qualitative research and should be contextualized with respect to other approaches to establishing validity of the results (Caine, 2014).


https://ccforum.biomedcentral.com/articles/10.1186/s13054-023-04380-2
Decision: Accept-Main