Dr ChatGPT tell me what I want to hear: How different prompts impact health answer correctness
Authors: Bevan Koopman
Guido Zuccon
Venue: EMNLP 2023 Main (2023)
Keywords: ChatGPT
LLM
Health Misinformation
Prompt Knowledge
Consumer Health
TL;DR: Investigation of how different prompt impact the effectiveness of LLM (ChatGPT) for health information seeking, including risk of bias in retrieve-the-generate scenarios.

Abstract: This paper investigates the significant impact different prompts have on the behaviour of ChatGPT when used for health information seeking. As people more and more depend on generative large language models (LLMs) like ChatGPT, it is critical to understand model behaviour under different conditions, especially for domains where incorrect answers can have serious consequences such as health. Using the TREC Misinformation dataset, we empirically evaluate ChatGPT to show not just its effectiveness but reveal that knowledge passed in the prompt can bias the model to the detriment of answer correctness. We show this occurs both for retrieve-then-generate pipelines and based on how a user phrases their question as well as the question type. This work has important implications for the development of more robust and transparent question-answering systems based on generative large language models. Prompts, raw result files and manual analysis are made publicly available at \url{https://github.com/ielab/drchatgpt-health_prompting}.

Dr ChatGPT tell me what I want to hear: How different prompts impact health answer correctness
Bevan Koopman
Guido Zuccon
CSIRO and The University of Queensland Brisbane, Australia
The University of Queensland Brisbane, Australia g wwcconuuq.
bevan. kowppkkmmmomii....aaa
Abstract
some domains this may not matter, in this paper we investigate a domain where this could have serious consequences - health. In particular, we consider the case a health consumer (i.e., non profes- sional) using ChatGPT to understand whether a treatment X has a positive effect condition Y.
This paper investigates the significant impact different prompts have on the behaviour of ChatGPT when used for health information seeking. As people more and more depend on generative large language models (LLMs) ChatGPT, critical to understand model behaviour under different conditions, espe- cially for domains where incorrect answers can have serious consequences such health. Using the TREC Misinformation dataset, empirically evaluate ChatGPT to show not just its effectiveness but reveal that knowl- edge passed in the prompt can bias the model to the detriment of answer correctness. We show this occurs both for retrieve-then-generae pipelines and based on how user phrases their question as well as the question type. This work has important implications for the development of more robust and transparent question-answert systems based on gener- ative large language models. Prompts, raw result files and manual analysis are made publicly available https://gi comb
Our evaluation is done using the TREC Misin- formation track, resource that contains complex health questions, often pertaining misinforma- tion regarding the efficacy of treatments, and that are typical subset of information seeking tasks undertaken by health consumers. Even though the risks of searching for health information online are well documented (Benigeri and Pluye, 2003; White and Horvitz, 2009; Zuccon et al., 2015), users still continue to seek health information through online mediums (Fox, 2011). Research on the effective- ness of using traditional search engines for health answers does exist (Zuccon al., 2015), but the parallel for LLMs does not. This paper aims at investigating this by considering two main experi- mental conditions and one specific, popular LLM, ChatGPT, currently being relied upon by end-users:
Question-only: ChatGPT is asked to provide an answer to health question without further information provided to the model; and
1 Introduction
Prompt-based generative large language models, such as ChatGPT, can be used to answer com- plex natural language questions, often with im- pressive effectiveness. Where previously, search engine was the main tool available, now users can choose to use large language model (LLM). While queries to search engines are typically short and ad-hoc, prompts to LLM can be far longer and richer. This can be double edged sword: on the one hand, far more information is available to the model to generate good answer; the other hand, LLMs suffer from hallucinations (Alkaissi and Mcfarlane, 2023) and incorrect, biased or mis- leading prompt information (e.g., obtained through a retrieved-then tebeereeeertte pipeline) may derail the model's ability to give a good answer. While in
Evidence-biased: ChatGPT is asked to provide an answer to health question after being pro- vided a prompt with information from a web search result (document) containing informa- tion about treatment X and condition Y. This provides insights into typical retrieve-then- generate setup, whereby search engine pro- vides results in prompt to the LLM.
We investigate cases where the document pro- vided in the prompt either supports the use of the treatment (supporting evidence) or where the docu- ment dissuades from the use of the treatment (con- trary evidence). Given these settings, we can then evaluate our two main research questions:

---

RQI- Question-Only Effectiveness: How effec- live ChatGPT at answering complex health information questions typical health con- sumers would ask? How does question phras- ing impact this effectiveness?
including using search engines and symptom check- ers, has therefore attracted substantial amount of research (Toms and Latter, 2007; White and Horvitz, 2009; Miller and Pole, 2010; White, 2013; Stanton et al., 2014; Lopes and Ribeiro, 2015; Sem- igran et al., 2015; Pogacar et al., 2017). The effec- tiveness of these technologies has also been moni- tored and assessed through several studies (Semi- gran et al., 2015; Zuccon al., 2015; Jimmy al., 2018; Cross al., 2021), and new technologies have also been deployed; e-g., the health informa- tion cards now commonly displayed by major com- mercial search engines (Gabrilovich, 2016; Jimmy et al., 2019b,a). Health chatbots based on pre- LLMs technology have also been proposed and investigated (You et al., 2023); however, extensive evaluation and understanding of chatbot solutions that rely on current state-of-the-art methods con- versational generative LLMs lacking.
RQ2- Evidence-Biased Effectiveness: How does biasing ChatGPT by prompting with supporting and contrary evidence influence answer correctness?
The paper is divided in two parts around these two questions. For question-only effectiveness (RQI), we show that ChatGPT is actually quite effective answering health related questions (ac- curacy 80%), but this effectiveness hides a system- atic bias based on both how the question is phrased and whether the correct answer to the question is a "Yes" or "No" that majorly degrades effectiveness.
When biasing the prompt with supporting or con- trary evidence (RQ2), we find that: (1) the evidence often is capable of overturning the model answer about a treatment, showing prompts have real im- pact on answer correctness; and (2) both supporting and contrary evidence can have a detrimental effect on answer correctness, reducing accuracy to %. Previous work has shown that engincering prompts has an important impact on the effective- ness of LLMs like ChatGPT (Liu et al.. 2023). This paper adds to that understanding by contribut- ing that it not just the "form" of the prompt that matters (e.g., the clarity of the instructions contained in the prompt, or the stance expressed in the prompt), but also that the correctness of information contained in the prompt can highly influence the quality of the output of the LLMs. This is important when LLMS are integrated in retrieve-then-genee -gnnerate pipeline (Chen al., 2017; Karpukhin al., 2020; Lewis et al., 2020), where information related to the question is first identi- fied from a corpus (e.g., the Web), and this is then passed to the model via the prompt to inform the model's 'ss output.
2.2 ChatGPT for health question-anssering.
In this paper, we empirically study the effectiveness of ChatGPT in answering consumer health ques- tions. Limited prior work looked how well Chat- GPT performs on medical exam questions (Gilson et al., 2022: Kung et al., 2023). Instead, we con- sider questions the general public may ask, which are quite different to medical exams. In addition. these previous studies did not consider the impact of prompting with evidence (our RQ2).
Gilson al. (2022) presented a preliminary eval- uation of ChatGPT on questions from the United States Medical Licensing Examination (USMLE) Step and Step 2 exams. The prompt only included the exam question, and ChatGPT answer was evalu- ated in terms of accuracy of the answer, along with other aspects related to logical justification of the answer and presence of information internal and external to the question. ChatGPT effectiveness was comparable to a 3rd year medical student.
Nov et al. (2023) compared ChatGPT responses to those supplied by a healthcare provider in 10 typical patient- patient-provider interactions. 392 laypeo- ple were asked to determine if the response they received was generated by either ChatGPT or healthcare provider. The study found that it was difficult for participants to distinguish between the two, and that they were comfortable using chatbots to address less serious health concerns - support- ing the realistic setting of our study.
2 Related Work
2.1 Health information seeking on the Web.
People rely on information on the web to make health decisions (Fox, 2011). Researching how they seek health information and advice the web.
Also referred to as read-then-generact or retrieval-
enhanced generation.

---

Benoit (2023) evaluated ChatGPT in the diag- nosis and triage of medical cases presented as vi- gnettes" and found ChatGPT displayed a diagnostic accuracy of 75.6% and a triage accuracy of 57.8%. De Angelis et al. (2023) recognised that Chat- GPT can be used to spread misinformation within public health topics. In our study, we empiri- cally demonstrate this to be the case, including the systematic biases present in the current Chat- GPT model that exacerbate this risk. addition, we show how misinformation can be injected into ChatGPT's prompt and this definitively impacts answer correctness, with potentially dire conse- quences.
[question_text]
Answer <Yes>, <No?, and provide an explanation afterwards.
Figure I: GPTChat prompt format for determining gen- eral effectiveness (RQ1) on TREC Misinformation top- ics.
ness. We have similar observations in that in RQ2 we vary the documents provided evidence in the prompt and find that two documents can have widely different effects on the answer of the model despite having the same stance about the topic of the question.
2.3 Prompt construction for LLMs.
RQ1 Question-only Effectiveness
One of the aims of our study is to evaluate the impact of ChatGPT answer correctness with re- spect to different prompt strategies. The effect of prompting LLMs is attracting increasing attention, especially with respect to the so called practice of "prompt enginecring"; i.c., finding an appropri- ate prompt that makes the language model solve a target downstream task. Prompt engineering, as opposed to fine-tuning, does not modify the pre- trained model's 's weights when performing a down- stream task (Liu et al., 2023). Prompt engineering is commonly used enable language models to ex- ecute few-shot or zero-shot learning tasks, reducing the need to fine-tune models and rely on supervised labels. Furthermore, a "prompt-leaming" approach can be applied as follows: during inference, the in- put x is altered using template to create a textual prompt x', which is then provided as input to the language model generate the output string y. The typical prompt-leariing setup involves construct- ing prompts with unfilled slots that the language model fills to obtain final string à which is then used to produce the final output y (Liu et al., 2023).
Our first research question relates to how effective ChatGPT is in answering complex health informa- tion questions. Measuring this serves two purposes: (i) it answers the basic question of how effective is this LLM in health information seeking tasks; and (ii) it provides the baseline effectiveness of the model when relying solely on the question; i.c., without observing any additional knowledge in the prompt. We also consider ChatGPT effectiveness on Yes vs No type questions and how question phrasing can impact ChatGPT.
3.1 Methods
We use 100 topics from the TREC 2021 and 2022 Health Misinformation track (Clarke et al., 2021) with associated ground truth. Each topic relates to the efficacy of a treatment for a specific health issue. The topic includes natural language ques- tion, e.g., "Does apple cider vinegar work to treat ear infections?", and ground truth answer, either 'Yes' or 'No". Ground truth answers were assigned based on current medical evidence.
In our study, we do not perform prompt-learning we instead use the prompt to pass external knowl- edge to the model and measure how this changes the answers it generates. Related to this direction, but in the context of few-shot learning, Zhao et al. (2021) observed that the use of prompts containing training examples for LLM (in particular GPT-3) provides unstable effectiveness, with the choice of prompt format, training examples and their order being major contributors to variability in effective-
We issue each question to ChatGPT part of the prompt shown in Figure 1, which instructs the model to provide a Yes/No answer and associated explanation. We then evaluate the correctness of the answer by comparing the ChatGPT answer with the TREC Misinformation Track ground truth.
In addition to asking for a Yes/No answer, we also run another experiment instructing the model to provide Yes/No/ option to determine the impact of adding the unsure option. For this, the prompt in Figure 1 was modified to add the Un- sure option. Since TREC Health Misinformation only provides Yes/No answers, Unsure is treated
A clinical vignette is a brief patient summary that includes relevant history, physical exam results, investigative data, and treatment.

---

an incorrect answer in our evaluation. The Un- sure option was investigated to understand how often ChatGPT responded with Unsure if given the option. Furthermore, our results will reveal that Un- sure answers could potentially lead users to adopt false health claims with serious consequences. For this reason well, we treat Unsure as incorrect. Processing the response from ChatGPT was done as follows: the ChatGPT response started with the terms "Yes", "No", or "Unsure" then these were recorded automatically. (Most cases fit this condition.) For the few cases where the response did not start with the terms "Yes", "No" or "Un-
How ChatGPT answered:
No
Unsure
Yes
Prompt type: Yes/No
80
60
40
20
Count
sure", manually assessed these responses. The manual assessment was done by two annotators (authors). While there were no disagreements, an- notators flagged some cases for joint discussion and adjudication.
Prompt type: Yes/No/Unnure
80
60
Questions in the TREC Misinformation dataset are in the form "Can X treat Y?". Our initial re- sults, discussed below, revealed a systematic bias in ChatGPT behaviour dependent on whether the ground truth was a Yes or No answer. To further investigate this effect we conducted an additional experiment whereby we manually rephrased each question to its reversed form: "Can X treat Y?" becomes "X can't treat Y At the same time inverted the corresponding ground truth match. Note that this was just rephrasing from the affir- mative to the negative, without changing the con- tents. The hypothesis here was that the phrasing of the question to match either Yes or No answer has a significant impact on ChatGPT behaviour and effectiveness.
40
20
0
Correct
Incorrect
ChatGPT Answer
Prompt
Overall Accuracy
Yes/No
0.80
Yes/No/Unnsue
0.33
Figure 2: Effectiveness of CharGPT when prompting for "Yes/No" and "Yes/No/Unsure" answers to TREC Misinformation questions.
To control for randomness in ChatGPT re- sponses, we repeated the above experiments 10 times and averaged the results (see the Limitations section for further discussion).
sure" answers from qualitative standpoint, we found that while ChatGPT answered Unsure, the explanation given not neutral. Consider the question "Will drinking vinegar dissolve a stuck fish bone?", which has ground truth answer of No. The ChatGPT response was "Unsure. Drink- ing vinegar may help break down the fish bone, but it's not a guaranteed solution.". This was clearly not a definitive No and could well drive user to take an unfounded and potentially dangerous treatment (Shiclds et al., 2016). Further qualita- tive assessment revealed that other answers specifically formulated to cover the whole spectrum of opinions on the topic. Covering all opinions would potentially fuel anchoring bias (Azzopardi, 2021: Lau and Coiera, 2007: White, 2013); iee. the
3.2 Results
The effectiveness of ChatGPT is shown in Figure Overall accuracy was 80%. ChatGPT answered "Yes" and "No" a similar number of times (55% vs 45%) and its eтror rate was similar between the two. When allowing for an "Unsure" option, accuracy much lower at 33% and the model hardly ever gave "No" answer. This lower accuracy is also by-product of our evaluation which treated "Un- sure" as an incorrect answer when comparing with the TREC Health Misinformation ground truth. We felt this was appropriate as when examining "Un-

---

be compared far more effective, answering Yes correctly most often, then answering Unsure, but never answering No. contrast, for No answers (left subplot), Chat- GPT was far less effective and answered Unsure the vast majority of the time. This behaviour repre- sents systematic bias in ChatGPT on whether the answer to a user's question is a Yes No.
Correct
Incorrect
GroundTruth: No
GroundTruth: Yes
40
30
There is an important difference to user on getting a Yes or No answer correct. Recall that TREC Misinformation most often contained ques- tions of the form "Can X treat Y?". If the ground truth is Yes, ChatGPT will say Yes or Unsure (it never answered No to Yes questions). The re- sulting user could seek the treatment X or remain unsure. Either way they will not be following an un- founded medical claim. Now consider the opposite case where the ground truth is a No. Such ques- tions often pertain to false health claims (e.g.. "Will drinking vinegar dissolve stuck fish bone?"). For such questions, ChatGPT is highly likely answer Unsure. From qualitative assessment of ChatGPT's explanation, we know that Unsure answers often contain some rationale for Yes and some for a No; a user with any preconceived bias on the topic may adopt their viewpoint. If they adopt the Yes viewpoint then they may well seek the treatment for false medical claim (e.g., electing to drink the vinegar to dissolve the stuck fish bone). Seeking treatments based on false health claims can have adverse health effects (indeed, ingestion of vinegar can in fact be fatal (Shields et al., 2016)).
Count
20
10
No Unsure Yes No Unsure Yes
Answer given by ChatGPT
Figure 3: If the correct answer to health question is Yes, ChatGPT is much more likely to get it correct. ChatGPT much more likely to respond with Unsure to questions where the correct answer is No.
user perceives the answer as affirming their precon- ceived answer/stance. Anchoring bias is shown to be a major challenge when supporting consumers seeking health information (White, 2013).
Besides the decision to treat Unsure as Incorrect, we did not perform an in depth analysis of these explanations (e.g., to verify whether the claims made in the explanation were true or were halluci- nations (Ji al., 2022; Bang et al., 2023)). because we lack the required medical expertise. We plan to investigate this aspect in future work through engagement with medical experts. A brief analysis of these explanations, however, revealed that they often contain remarks about the presence of limited scientific evidence (or even conflicting evidence) with respect treatment option for a condition. The answers also often contain a suggestion to con- tact a health practitioner for further review of the advice.
Having discovered systematic difference ChatGPT behaviour for Yes or No questions, we turn to results for manually reversing the phrasing of questions and ground truth (e.g.. "Can X treat Y?" becomes "X can't an't treat Y?"). Figure 4 shows the effectiveness of ChatGPT on answering the re- versed queries. (This figure Figure 2, which was obtained for the original ques- tions). Overall, ChatGPT was far less effective for the manually reversed questions compared to the original phrasing. Accuracy dropped from 0.80 to 0.56 for the Yes/No prompt and 0.33 to 0.04 for the Yes/No/Unsure prompt. We observe from Figure 4 that ChatGPT very seldom answered Yes at all to any of the questions. This is in contrast to the sults obtained for the original questions (Figure 2) where Yes answers were quite common. Recall that we simply rephrased the question from an af- firmative to a negative (as well as switching the associated ground truth). While the core content
TREC Misinformation ground truth was either Yes or No in answer to the question. On analysis of ChatGPT answers for the Yes/No/Unsure prompt, we discovered significant difference in ChatGPT behaviour based on whether the ground truth an- swer was a Yes No. Figure 3 divides the results according to whether the ground truth was Yes or No. For Yes answers (right subplot), ChatGPT was

---

4 RQ2 E92d-EEienneeee Evidence-biased Effectiveness
How ChatGPT answered:
No
Unsure
Yes
Our second research question investigates the im- pact on answer correctness of biasing ChatGPT by prompting with (i) supporting evidence; and (ii) contrary evidence. Measuring this allows us determine the impact of providing external informa- tion via the prompt. It helps also understand how ChatGPT might perform in a retrieve-then-gent generate pipeline.
Prompt: Yes/No (Reversed question)
100
75
50
25
4.1 Methods
0
Count
Supporting and contrary evidence taken indi- vidual documents from the TREC Misinformation track ground truth (qrels). Documents judged as ("Supportive") were selected as supporting evi- dence. Documents judged as ("Dissuades") were taken as contrary. The process to issue the evidence biased prompt to ChatGPT was as follows:
Prompt: Yes/No/Unsure (Reversed question)
100
75
50
25
1. For each TREC topic, selected maxi- mum of 3 supportive and 3 contrary evidence documents from the TREC qrels,
0
Correct
Incorrect
ChatGPT Answer
2. For each document, we generated the prompt Figure 5, including both the question and the evidence text in quotes;
Prompt
Overall Accuracy
Yes/No
0.56
3. Although the prompt explicitly asked for a Yes/No answer, we found that in many cases ChatGPT did not use these words, but did provide the answer in another way (e.g., "In- haling steam can help alleviate the symptoms of a common cold"). We manually mapped these responses to Yes, No, or Unsure, All an- swers were assessed by two of the authors and discussion took place to resolve disagreement.
Yes/No/Unsure
0.04
Figure 4: Effectiveness of ChatGPT when prompting after rephrasing TREC Misinformation questions to re- versed form (c.g.. "Can X treat Y?" becomes "X can't treat Y?"). When comparing the original questions from Figure we note that simply rephrasing the ques- tion strongly degraded answer correctness.
4. Once ChatGPT provided an answer, we then evaluated the correctness of the answer by comparing to the TREC Misinformation Track ground truth.
the question was unchanged, the rephrasing had a huge impact on ChatGPT responses. Previous work has shown that there is considerable variation in how people pose health questions to search en- gines (Zuccon et al., 2015; Koopman et al., 2017). This variation leads to significant impact in search engine effectiveness. Our results suggest this is also the case for ChaiGPT. However, for ChatGPT we find that this effect is quite insidious in that arises from how the question phrased rather than the actual keywords authored by the user.
We again added Yes/No/Unnsure prompting option, similar to RQL. We used the 35 topics from TREC 2021 Health Misinformation track that contained document-level relevance assessments."
We took first three documents according the official TREC qrel file. All topics
least supporting evidence passages. 23 topics had least contrary evidence passages. topics had only contrary evidence passages. topics had only contrary evidence passage. topics contrary evidence passages.
'Document-lewe level qrels are not available for TRFC 2022 at the time of writing.

---

[question_text]
web search for this question has returned the following evidence, which I provide you in quotes:
ChatGPT answered:
No
Unsure
Yes
"[document_text]""
Prompt type: Yes/No
You MUST answer to my question with one of the following options ONLY: <Yes>, <No>. Your answer MUST NOT be based just the web result I provided: you should consider the web result along with your knowledge. Please also provide explanation for your answer.
100
50
Count
Figure ChatGPT Prompt used determine what impact supportive or contrary document has on answer correctness.
Prompt type: Yes/No/Unsure
100
The documents were web pages from the clean version of the C4 dataset. Some docu- ments exceeded the maximum token limit Chat- GPT and had to be trimmed. We used the NLTK word_tokenize method to count the number of tokens. Then, through experimentation, we identi- fied that limit of 2,200 NLTK tokens was about the maximum we could use to concatenate with the remainder of the prompt and issue to ChatGPT without encountering problems with the input size limit. An analysis of document length showed that the median was 1,845 tokens, and trimming only occurred on three occasions.
50
Correct
Incorrect
ChatGPT Answer
Prompt
Overall Accuracy
Yes/No
0.63
Yes/No/Unnsue
0.28
Figure 6: Effectiveness of ChatGPT when prompting with either a supporting or contrary evidence.
Figure 9 of the appendix contains two actual prompts we used, one with support and one with contrary evidence. Experiments were repeated 10 times and results averaged control from random- ness in ChatGPT.
Correct
Incorrect
4.2 Results
200
Count
The effectiveness of ChatGPT with evidence- biased prompting is shown Figure 6. Overall accuracy was 63% - far less than the 80% of the simpler, question-only prompt. In addition. when evidence-biasing in the prompt, ChatGPT much more likely to give "Yes" answer (62% Yes 55% Yes in RQI). If the prompt allowed for "Unsure" answer then ChatGPT answered Unsure 62% of the time, reducing accuracy 28%. Figure shows a breakdown of effectiveness af- ter evidence-biasing, Specifically, how the answer changed compared to RQI 's question-only condi- tion: "Flipped" indicates that ChatGPT's answered opposite (Yes to No No to Yes) after evidence-
100
Flipped Unchanged
Unsure
Status after evidence biasing
Figure 7: How evidence-biased prompting changed ChatGPT answers when compared with question-only prompting.
biasing; "Unchanged" means the answer matched RQI's question-only condition: "Unsure" indicates where ChatGPT did not provide a Yes/No answer. We observe that when ChatGPT changed answer (i.c., Flipped), it was wrong 68% of the time. conclusion, evidence-biasing can flip the answer
*C4 contains BB english extracts from the April 2019 snapshot of Common Crawl.

---

Incorrect & Supporting Evidence
1
Incorrect
Incorrect & Contrary Evidence
Incorrect
Yes
2
Correct & Supporting Evidence
Correct
Correct
No
Correct & Contrary Evidence
Ground truth answer
Question-only correctness (RQ1)
Evidence-biased condition
Evidence-biased correctness (RQ2)
Figure 8: Sankey diagram showing the breakdown of all results. From the left, topics divided by ground truth answer (Yes or No); next topics are divided according to RQI question prompting (Correct or Not Correct); next the prompt evidence biased (Supporting and Contrary); finally, post evidence-biand breakdown is shown.
and it's typically for the worse.
then peen-generate pipelines to reduce hallucinations and improve answer correctness.
Figure 8 provides a detailed analysis of the an- swer behaviour of CharGPT with respect to the two conditions: question-only and sidencce-bissed This analysis provides number of notable insights. First, consider the incorrect answers provided in the evidence-biased condition (label the Sankey diagram). These were cases where providing the question only in RQI was incorrect and providing evidence in RQ2 was also incorrect. In these cases, providing supporting evidence to to overturn an incorrect answer did not flip it to correct answer. In fact, providing the evidence (contrary sup- portive) had little impact: nearly all the Incorrect cases from the RQI column flowed to the Incorrect column of RQ2.
Conclusions
We have examined the correctness of ChatGPT, rapidly emerging generative large language model, in answering complex health information questions regarding the effectiveness of treatment for con- dition. We did this two settings: when only the question is presented to ChatGPT (question-only) and when the question is presented along with evi- dence suddence-biasedy i.c., a web search result retrieved when searching for the question. Impor- tantly, we controlled the stance used by the ques- tion, and whether the evidence or contrary the treatment. This in turn allowed us understand the effect of question phrasing and of providing evidence the prompt. ChatGPT an- swer accuracy was 80% when relying solely on the question being posed, and no additional evidence. However, (i) ChatGPT displays systematic bias based on whether the question has Yes or No answer, (ii) simply rephrasing the question to its negative form strongly degrades effectiveness, and (iii) providing evidence in the prompt degrades an- swer accuracy, even if the evidence is correct.
in supporting
Now consider the case where adding evidence caused the answer to flip from correct (RQ1) to in- correct (RQ2) (label @). This actually occurred more often when supporting evidence was pro- vided, rather than contrary evidence, but the dif- ference was minimal. The key takeaway here is that providing evidence degrades answer accuracy. even when that evidence is supporting.
In conclusion, prompting with evidence did not help flip incorrect answers to correct and actually caused correct answers to flip to incorrect. This may serve as warning for relying on retrieve-
We specifically highlight the last result: inject- ing evidence into the prompt, akin to retrieve- then-

---

generate pipeline, can heavily influence the answer more importantly, it reduced ChatGPT accuracy to only 63%. This finding at odds with the idea of relying on retrieve-then-geneee gegeratee pipelines to reduce hallucinations models like ChatGPT.
only considered the single-turn setting. The ability to hold multi-turn conversations would support, for example, providing multiple evidence items, demand for a binary decision instead of an unsure position, and clarifying aspects of the answer that may be unclear.
Large language models seem set to be increas- ingly used for information seeking tasks. If such tasks drive important health related decisions (such as what treatment to take) then it paramount we better understand these models. This understand- ing includes the obvious aspect of answer correct- ness, but also how different prompting can sway the model. An understanding of such aspects both helps mitigate potential harms and helps inform the design of better models.
When producing answer, we instructed Chat- GPT to also explain the reason for providing the specific advice. These explanations often included claims about research studies and medical prac- tices; we did not validate these claims. In addi- tion, we did not ask to attribute such claims to sources (Bohnet et al., 2022; Menick et al., 2022; Rashkin et al., 2021): correct attribution appears to be critical requirement to improve the quality, in- terpretability and trustworthiness of these methods. Finally, in our analysis revealed that users may be affected by anchoring bias (Azzopardi, 2021) when examine ChatGPT's answers, espe- cially those for which the model declares itself Unsure. We acknowledge that user studies should be instructed to further understand this aspect.
Prompts, raw result files and manual analysis are made publicly available at: https: //gi:://hhhuuu com/ielab/ dr wrrahhttthhaaeeeeee
Limitations
Our study has number of limitations earmarked for future work. We did not analyse the charac- teristics of the evidence inserted in prompts that triggered divergent answers, despite having identi- cal stance, nor did we study the effect of different prompt formats (including the extraction of key passages from the evidence documents used) aspects that are known to lead to variability in ef- fectiveness in prompt learning (Zhao et al., 2021). )..
References
Hussam Alkaissi and Samy Mcfarlane, 2023, Artificial hallucinations in chatgpt: Implications in scientific writing. Cureus,
Leif Azzopardi. 2021. Cognitive biases in search: view and reflection cognitive biases in information retrieval. In Proceedings of the 2021 conference on human information interaction and retrieval, pages 27-37.
We did not perform fine-tuning of the LLM to specialise it to answering health questions from consumers. While we acknowledge that providers offering such a service health consumers may investigate fine-tuning their LLMs, instead wanted to evaluate the common case of people di- rectly posing their health questions to ChatGPT, which is currently realistic scenario.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity, arXiv preprint arXiv:2302.044 04023.
In ChatGPT, like in other LLMs, answer genera- tion is stochastic. We could have controlled for this by setting the temperature parameter of the model zero, thus enforcing deterministic answers when the LLM was provided with the same prompt. We instead resorted to study the variability of answer generation (and the associated effectiveness) across multiple runs of the same question using the default temperature value from ChatGPT. We did this be- cause a user of ChatGPT does not have control of the temperature parameter, and thus wanted to accurately reflect this real scenario.
Mike Benigeri and Pierre Pluye. 2003. Shortcomings of health information on the internet. Health promotion international, 1 18(4):311-3) -366.
James RA Benoit. 2023. Chatgpt for clinical vignette generation, revision, and evaluation. medRxiv, pages 2023-02.
Bernd Bohnet, Vinh Q Tran, Pat Verga, Roce Aharoni, Daniel Andor, Livio Baldini Soares. Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, et al. 2022. Auributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08D0
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open- domain questions. In Proceedings of the 55th Annual
A key feature of ChatGPT is its interactivity: it supports multi-turn conversations. We, instead,

---

Meeting of the Association for Computational Lin- guistics ( (Volume I: Long Papers), pages 1870-1879.
Bevan Koopman, Guido Zuccon, and Peter Bruza, 2017. What makes an effective clinical query and querier? Journal of the Association for Information Science and Technology. 68(1 18:55)22577
Charles LA Clarke, Saira Rizvi, Mark D Smucker, Maria Maistro, and Guido Zuccon. 2021. Overview of the tree 2021 health misinformation track. In
Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Blepaño, Maria Madriaga, Rimel Aggabao. Giezel Diaz- Candido, James Maningo, al. 2023. Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLoS digitul health, 2(2):e0000198.
TREC.
Sebastian Cross, Ahmed Mourad, Guido Zuccon, and Bevan Koopman. 2021. Search engines vs. symptom checkers: comparison of their effectiveness for online health advice. In Proceedings of the Web Conference 2021, pages 206-216.
Annie YS Lau and Enrico W Coiera. 2007. Do peo- ple experience cognitive biases while searching for information? Journal of the American Medical Infor- maticy Association, 14(5):999-605
Luigi De Angelis, Francesco Baglivo, Guglielmo Arzilli, Gaetano Pierpaolo Privitera. Paolo Ferragina, Al- berto Eugenio Tozzi, and Caterina Rizzo. 2023. Chat- gpt and the rise of large language models: The new ai-driven infodemic threat in public health. Available al SSRN 4352931.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmenne auggeeeeeee generation for knowledge-intensivee ulp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-944 9974.
Susannah Fox. 2011. Health topics: 80% of internet users look for health information online. Pew Inter- net & American Life Project.
Evgeniy Gabrilovich. 2016. Cura te ipsum: answering symptom queries with question intent. In Second WebQA workshop, SIGIR, volume 2016.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt. and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35.
Aidan Gilson, Conrad Safranek, Thomas Huang. Vimig Socrates, Ling Chi, Richard Andrew Taylor, and David Chartash. 2022. How well does chatgpt do when taking the medical licensing exams? the im- plications of large language models for medical edu- cation and knowledge assessment. medRxiv, pages 222
Carla Teixeira Lopes and Cristina Ribeiro. 2015, Elfects of terminology on health queries: An analysis by user's health literacy and topic familiarity. In Current issues in libraries, information science and related fields, volume 39, pages 145-184. Emerald Group Publishing Limited.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young. Lucy Campbell- Gillingham, Geoffrey Irving, et al. 2022. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203. 100003...143 11147.
Jimmy, Guido Zuccon. Gianluca Demartini, Bevan Koopman, et al. 2019a. Health cards to assist de- cision making in consumer health search. In AMIA Annual Symposium Proceedings, volume 2019, page 1091. American Medical Informatics Association.
Edward Alan Miller and Antoinette Pole. 2010. Di- agnosis blog: checking up on health blogs in the blogosphere. American Journal of Public Health, 100K8):5144-151.
Jimmy, Guido Zuccon, Bevan Koopman, and Gianluca Demartini. 2019b. Health cards for consumer health search. Proceedings of the 42nd International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, pages 35-44.
Oded Nov, Nina Singh, and Devin M Mann. 2023. Putting chatgpt's 's medical advice to the (turing) test. medRxiv, pages 2023-01.
Frances A Pogacar, Amira Ghenai, Mark D Smucker, and Charles LA Clarke. 2017. The positive and nega- tive influence of search results on people's 's decisions about the efficacy of medical treatments. In Proceed- ings of the ACM SIGIR International Conference on Theory of Information Retrieval, pages 209-216.
Jimmy, Guido Zuccon, Joao Palotti, Lorraine Goeuriot, and Liadh Kelly. 2018 Overview of the clef 2018 consumer health search task. CLEF 2018 Working Notes, 2125.
Vladimir Karpukhin. Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781.
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm. Michael Collins, Dipanjan Das, Slay Petrov, Gau- rav Singh Tomar, Iulia Ture, and David Reitter. 2021. Measuring attribution in natural language generation models. arXiv preprint arXiv:21 12. 12870.

---

document either Hannah Semigran, Jeffrey A Linder, Courtney Gi- dengil, and Ateev Mehrotra. 2015. Evaluation of symptom checkers for self diagnosis and triage: au- dit study, bmj. 351.
Supporting evidence
Can folic acid help improve cognition and treat dementia?
Lisa BE Shields, Cristin M Rolf, and John C Hun- saker III. 2016. Sudden death due to forced ingestion of vinegar. Forensic science international, 266:e233- e26.
A web search for this question has returned the following evidence, which I provide to you in quotes:
Isabelle Stanton, Samuel leong, and Nina Mishra. 2014. Circumlocution in diagnostic medical queries. In Proceedings of the 37th international ACM SIGIR conference on Research & development in informa- tion retrieval, pages 133-142.
"An important study appeared in JAMA Psychiatry in June 2016, providing addi tional evidence that high blood levels of vitamin B12 can slow the shrinking of the brain that commonly occurs after age 60..."
Elaine G Toms and Celeste Latter. 2007. How con- sumers search for health information. Health infor- mutics journal. 13(3):223-25
You MUST answer my question with one of the following options ONLY: <Yes>, <No>. Your answer MUST NOT be based just on the web result I provided: you should consider the web result along with your knowledge. Please also provide an explanation for your answer.
Ryen White. 2013. Beliefs and biases in web search. In Proceedings of the 36th international ACM SIGIR conference Research and development in informa- tion retrieval, pages 3-12.
Ryen White and Eric Horvitz. 2009. Cyberchondria: studies of the escalation of medical concerns in web search. ACM Transactions Information Systems (TOIS), 1-(4)
Contrary evidence
Yue You, Chun- Hun Tsai, Yao Li, Fenglong Ma. Christo- pher Heron, and Xinning Gui. 2023. Beyond self- diagnosis:
Can folic acid help improve cognition and treat dementia?
How chatbot-based symptom checker should respond. ACM Transactions on Computer- Human Interaction.
web search for this question has returned the following evidence, which I provide to you in quotes:
Zihao Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. 2021. Calibrate before Improv- ing few-shot performance of language models. In- ternational Conference Machine Learning, pages 12697-12706. PMLR.
"No evidence that folic acid with wi thout vitamin B12 improves cognitive function of unselected elderly people with or without dementia. Long-term supplementation may benefit cognitive function of healthy older people with high homocysteine levels..."
Guido Zuccon, Bevan Koopman, and Joao Palotti. 2015. Diagnose this if you can: On the effectiveness of search engines in finding medical self-diagnosis in- formation. In European Conference Information Retrieval (ECIR).
You MUST answer my question with one of the following options ONLY: <Yes>, <No>, Your answer MUST NOT be based just on the web result provided: you should consider the web result along with your knowledge. Please also provide an explanation for your answer.
A Sample Prompts for RQ2
Figure 9 contains two actual prompts used, one with support and one with contrary evidence.
Figure Evidence-biased prompts submitted Chat- GPT. Each contains the question and then evidence document from web search. supports the question is contrary to the questions. Questions, documents and ground truth are from the TREC Misinformation Track.


Reviews:
[{"review_id": "4MpsYmf1l6", "reviewer": "Reviewer_o8M9", "summary": "", "questions": "How is the evaluation done when considering ChatGPT effectiveness based on Yes vs. No type questions? ", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper.\n\n### Reviewer Question: “How is the evaluation done when considering ChatGPT effectiveness based on Yes vs. No type questions?”\n\nIn the experiments where ChatGPT was instructed to only produce a Yes/No answer, ChatGPT did successfully constrain the response to start with these tokens. Therefore, the response could be automatically assigned to the Yes/No classes.\n\nIn the experiments where ChatGPT was instructed to produce a Yes/No/Unsure answer, we adapted the following process:\n- If the ChatGPT response started with the terms “Yes”, “No”, or “Unsure” then these were recorded automatically. (Most cases fit this condition.) \n- For the few cases where the response did not start with the terms “Yes'', “No” or “Unsure”, we manually assessed these responses. \n- The manual assessment was done by two annotators (authors). While there were no disagreements, annotators flagged some cases for joint discussion and adjudication.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. The work is valuable because it provides a deeper understanding of current state-of-the-art LLM functioning with respect to the domain (in this case, healthcare). \nB. The empirical and detailed evaluation through two research Questions which are : \nRQ1 – Question-only Effectiveness of ChatGPT\nRQ2 – Evidence-biased Effectiveness of ChaptGPT", "reasons_to_reject": "The limitations, such as multiple talk turns, and the inspection of the relevance/authenticity of the supportive document for prompting, could have been reasons for concern. But, since the authors have mentioned it in future research directions, I do not see any strong reason for the manuscript to be rejected. ", "questions_to_authors": "How is the evaluation done when considering ChatGPT effectiveness based on Yes vs. No type questions? ", "paper_topic_and_main_contributions": "The manuscript “Dr. ChatGPT tell me what I want to hear: How different prompts impact Health answer correctness” investigates the impact of different prompts on the behavior of ChatGPT based on the TREC Misinformation dataset. The manuscript empirically investigates the bias induced in the model applied to the healthcare domain. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "Minor Typos and punctuation errors. ", "Justification_for_Ethical_Concerns": "NA"}, "extra_text": {}}, {"review_id": "00agiR1qd1", "reviewer": "Reviewer_XcW1", "summary": "", "questions": "Lack of robust validation could lead to overestimation of the proposed method's effectiveness. This includes insufficient testing, not using appropriate datasets or baselines for comparison, or cherry-picking results. The prompting approach is unclear and required detailed explanation with architecture.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper.\n\n### Reviewer Remark: “Lack of robust validation could lead to overestimation of the proposed method's effectiveness. This includes insufficient testing, not using appropriate datasets or baselines for comparison, or cherry-picking results.”\n\nWe are confused about this remark: to what extent was our testing insufficient? Our datasets were not appropriate? Which baseline should we have included in such a study? What results have been cherry-picked?\n- We believe our dataset is appropriate: it is based on a well developed public TREC (a NIST-lead, well developed international evaluation initiative) dataset on Health Misinformation. The dataset is based on high quality medical evidence (Cochrane systematic reviews)\n- Regarding baselines: other similar LLM services that have now become available including Google Bard and Bing Chat (based on ChatGPT), but they were not yet publicly available at the time of the study (and still not via API access). Yet, these are not baselines, but alternative systems. Our goal was not to benchmark a system A vs. a system B. Instead, it was to understand and quantity answers of the most widely used LLM, ChatGPT, that people are using for health advice. \n- Regarding cherry-picking. We are unsure to what result this comment is referring. We did highlight some specific cases (“Will drinking vinegar dissolve a stuck fish bone?”), but these were to exemplify general behaviour of the model. \n\n### Reviewer Remark: “The prompting approach is unclear and requires detailed explanation with architecture.”\n- The prompts used in our experiments are given in Figures 1 and 5; two sample prompts are provided in Figure 9 in the Appendix. \n- Based on the reviewer’s comment, we will further complement these by including a workflow/architecture diagram in the Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "One of the main strengths of any research paper is its level of innovation. If the paper presents new methods, techniques, or concepts that advance the field of NLP, it would be of great interest to the community.  If the research can be directly applied to real-world problems, this is also a considerable strength. This work shall support safety concerns in healthcare sector with the use of prompting, the need of hour.", "reasons_to_reject": "Lack of robust validation could lead to overestimation of the proposed method's effectiveness. This includes insufficient testing, not using appropriate datasets or baselines for comparison, or cherry-picking results. The prompting approach is unclear and required detailed explanation with architecture.", "questions_to_authors": "Lack of robust validation could lead to overestimation of the proposed method's effectiveness. This includes insufficient testing, not using appropriate datasets or baselines for comparison, or cherry-picking results. The prompting approach is unclear and required detailed explanation with architecture.", "paper_topic_and_main_contributions": "The research paper discuss the impact of different prompts on the behavior of ChatGPT, particularly when used for health information seeking. The paper explores how the model's responses can be influenced by the way a question is asked, as well as the type of question, and how it can potentially provide incorrect answers in crucial domains like health.\n\nThe researchers examine two main scenarios:\n\n\"Question-only\": Where the model is asked a health question with no further context provided, and\n\"Evidence-biased\": Where the model is asked a health question after being provided with additional information (or evidence) obtained from a web search result.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7Achr4pND5", "reviewer": "Reviewer_96ee", "summary": "", "questions": "Question A: line 461: “a couple of occasions” how many to be precise?\n\nQuestion B: footnote3: are there any statistics?\n\nQuestion C: Were there quotes inside the evidences? \n\nQuestion D: lines 378-408: You mention that ChatGPT is very sensitive to question phrasing with negation. Could it be possible that the negation in the prompt itself (MUST/MUST NOT) contributes to the final performance as well?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the clear and specific questions. We will include the relevant statistics and comments we provide in the answers as part of a new version of the paper.\n\n### Reviewer Question: “Question A: line 461: “a couple of occasions” how many to be precise?”\n- This was on three occasions.\n\n### Reviewer Question: “Question B: footnote3: are there any statistics?”\n- All topics had at least 3 supporting evidence passages.\n- 23 topics had at least 3 contrary evidence passages.\n- 2 topics had only 2 contrary evidence passages.\n- 6 topics had only 1 contrary evidence passage.\n- 4 topics had 0 contrary evidence passages.\n\n### Reviewer Question: “Question C: Were there quotes inside the evidences?”\n- Out of 184 evidence passages, 77 contain quotes (42%).\n- We understand the reason for asking this question: the evidence passed in the prompt is encapsulated in quotes, and quotes within the evidence may impact this format. Our anecdotal experience with ChatGPT is that this has little to no influence, but we did not systematically investigate this.\n- While we could have intentionally removed quotes from passages, this might create a somewhat artificial setting, in that real users may very well paste into the prompt evidence that contains quotes. Our intention was to simulate as closely as possible how real users may use and interact with ChatGPT.\n\n### Reviewer Question: “Question D: lines 378-408: You mention that ChatGPT is very sensitive to question phrasing with negation. Could it be possible that the negation in the prompt itself (MUST/MUST NOT) contributes to the final performance as well?”\n- We apologise for what seems to us a misunderstanding. We did not change the text in the prompt that refers to the part “You MUST answer to my question”: that stayed the same.\n- We instead only changed the question text (marked in the prompt of Figures 1 and 5 as [question_text]). So, if the question in the TREC dataset was “Does inhaling steam help treat common cold?” became “Inhaling steam does not help treat common cold” . We will include all these in the Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Authors raises important topics of ChatGPT behaviour: higher sensitivity to question phrasing rather than actual keywords authored, negative influence of provided evidences for question answering.\n- Well argumented decisions, detailed procedures, multiple experiments\n- Important notion of preconceived bias of the user about unsure answer", "reasons_to_reject": "- Lack of in-depth analysis of influence of evidence to ChatGPT response\n- Lack of statistics and analysis of evidences", "questions_to_authors": "Question A: line 461: “a couple of occasions” how many to be precise?\n\nQuestion B: footnote3: are there any statistics?\n\nQuestion C: Were there quotes inside the evidences? \n\nQuestion D: lines 378-408: You mention that ChatGPT is very sensitive to question phrasing with negation. Could it be possible that the negation in the prompt itself (MUST/MUST NOT) contributes to the final performance as well?", "paper_topic_and_main_contributions": "This paper presents an analysis of ChatGPT performance on health-related general-public questions with and without evidence. The authors use the TREC Misinformation dataset of treatment-related questions with provided evidences, both supportive and contrary, to evaluate ChatGPT capacity of responding these questions correctly.\n\nThis paper is a first research analysis of using LLM for health answers by general public. Since ChatGPT became popular and widely used and the topic of health is sensitive and risky, it is important to evaluate how ChatGPT can perform and argument its answers. The work contains detailed analysis of different scenarios and limitations of the current model.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the impact of different prompts on the behavior of ChatGPT based on the TREC Misinformation dataset. They report the bias induced in the model applied to the healthcare domain.

All reviewers agreed that the paper studies an important topic, and "support safety concerns in healthcare sector". Reviewers asked some specific questions about the study method and the statistical significance of the results, which I believe the authors sufficiently addressed in their rebuttals. Note that one reviewer who rated the paper most negatively did not update their score, but did comment that "I think the paper is good to go now."

I recommend accepting the paper into the Main track, and encourage the authors to add a figure illustrating the workflow, extending the related work session, and providing clarification to the human eval.
Decision: Accept-Main