Is the Answer in the Text? Challenging ChatGPT with Evidence Retrieval from Instructive Text
Authors: Sophie Henning
Talita Anthonio
Wei Zhou
Heike Adel
Mohsen Mesgar
Annemarie Friedrich
Venue: EMNLP 2023 Findings (2023)
Keywords: question answering
hallucination
evidence retrieval
dataset creation
generative language models
chatgpt
TL;DR: We present a challenging task and dataset to ChatGPT to investigate its hallucination problems: Given a text document and a non-factoid question, recognize whether the text provides an answer to the question or not.

Abstract: Generative language models have recently shown remarkable success in generating answers to questions in a given textual context. 
However, these answers may suffer from hallucination, wrongly cite evidence, and spread misleading information.
In this work, we address this problem by employing ChatGPT, a state-of-the-art generative model, as a machine-reading system.
We ask it to retrieve answers to lexically varied and open-ended questions from trustworthy instructive texts.

We introduce WHERE (WikiHow Evidence REtrieval), a new high-quality evaluation benchmark of a set of WikiHow articles exhaustively annotated with evidence sentences to questions that comes with a special challenge: All questions are about the article's topic, but not all can be answered using the provided context.
We interestingly find that when using a regular question-answering prompt, ChatGPT neglects to detect the unanswerable cases.
When provided with a few examples, it learns to better judge whether a text provides answer evidence or not.
Alongside this important finding, our dataset defines a new benchmark for evidence retrieval in question answering, which we argue is one of the necessary next steps for making large language models more trustworthy.

Is the Answer in the Text? Challenging ChatGPT with Evidence Retrieval from Instructive Text
Sophie Henning 1.27 Talita Anthonio 1.37 Wei Zhou
Heike Adel Mohsen Mesgar' Annemarie Friedrich
Bosch Center for Artificial Intelligence, Renningen, Germany
"Ludwig Maximilian University of Munich, Germany
University of Stuttgart, Germany *University of Augsburg, Germany Hochschule der Medien Stuligart, Germany sophie. henning@de bosch. com talita anthoni heyaooooooyyaooo..oooooooooooo.m com
annemarie. friedri ch@informatik, uni -wugsaauguuuoooo de
Equal contribution
Abstract
wikiHow
[I] Some of you want be Disney Character at Halloween, but some of you don't know how to dress like Disney character. [...]
Generative language models have recently shown remarkable success in generating an- swers to questions a given textual context. However, these answers may suffer from hal- lucination, wrongly cite evidence, and spread misleading information.
[4] == Mickey Mouse ===
[5] Sew his cars with black,
How Dress
[6] Sew some buttons (white) and
this work, we ad- dress this problem by employing ChatGPT, state-of-the-art generative model, as a machine- reading system. We ask to retrieve answers lexically varied and open-ended questions from trustworthy instructive texts.
up Disney
short (red). [7] Wear a black shirt. [8] Wear a yellow boots. [...]
Character
[11] === Peter Pan
[12] Have green elf hat. [13] Put the feather the elf hat. [14] Dye your hair orange. [15] Wear lime green shirt, pants and shoos. [...]
Should I dress up Disney Character for Halloween?
answer
We introduce WHERE (WikiHow Evidence REtrieval), a new high-quality evaluation benchmark of a set of WikiHow articles ex- haustively annotated with evidence sentences to questions that comes with special challenge: All questions are about article's s topic, but not all can be answered using the provided context. We interestingly find that when us- ing regular question-anse annweeing prompt, Chat- GPT neglects to detect the unanswerable cases. When provided with a few examples, it learns better judge whether a text provides answer evidence. Alongside this important finding, our dataset defines a new benchmark for evidence retrieval in question answering, which argue is one of the necessary next steps for making large language models more trustworthy.
evidence
Which Disney Character costume is the most popular wear?
evidence
Figure Evidence retrieval for questions related instructive articles from WikiHow. For the question in the upper box, a system should ideally identify the sentences annotated as evidence in the text. For the question in the lower box, it should not retrieve wrong sentences as "evidence."
In this paper, we focus on use case where gen- erative LM is queried for advice on a range of per- sonal issues, including health interpersonal rela- tionships, or difficult tasks. This is challenging scenario for LMs because questions are often open- ended and non-factoid, and require well-informed instructions as answers. As illustrated in Figure 1, we explicitly query generative LMs to retrieve ev- idence sentences for answering a question from a trustworthy instructive text. Our challenging setup requires two competencies the model side: (i) identifying whether or not the question is answer- able using only the provided text input, and (ii) retrieving evidence from the trustworthy source, which could, e.g., support generated answer.
1 Introduction
Generative language models (LMs) are trained to generate an output text given an input text. While such models have recently shown a remark- able performance in various NLP tasks (Touvron et al., 2023a; Radford et al., 2019; Brown et al., 2020), they are known to suffer from hallucination, i.e., they often generate text that lacks evidence (McKenna al., 2023; Ji al., 2022). This may lead to the spread of misinformation (Dong et al., 2022: Carlini et al., 2021), and thus reduce the systems' trustworthiness.
Existing question answering datasets, e.g., Wiki- HowQA (Deng et al., 2020) and SQuaD (Rajpurkar

---

al., 2016, 2018), do not fit our evaluation setup. The WikiHowQA QA dataset (Deng et al., 2020) uses the titles as questions, and does not cover the sen- tence retrieval aspect. SQuaD contains unanswer- able questions but focuses on factoid questions. To make our evaluation setup challenging and sound, we create new high-quality test set. We col- lect set of diverse and open-ended questions for WikiHow articles via crowd-sourcing, and perform double annotation of evidence sentences in the arti- cles. We use our dataset to perform a challenging evaluation of ChatGPT, a successor of InstructGPT (Ouyang et al., 2022b), which has been pretrained on a huge amount of texts including instructive web texts, in a systematic manner.
2.1 Dataset Creation
We export WikiHow² articles for the following cat- egories: Arts and Entertainment, Home and Gar- den, Health and Relationships and Travel. We collect articles for all categories in October 2022 and additional articles for Home and Garden and Arts and Entertainment in December 2022.
Question collection. To collect questions, we set up Human Intelligence Task (HIT) on Ama- zon Mechanical Turk. We display the title of the article (e.g., "How to Dress up as Disney Charac- ter"), the first paragraph of the article, and a set of keywords generated from the full article, and ask the crowd-workers to write six questions for which they would expect to find answers in the article (see A Appendix D). Since annotators never see the full article, they can only make educated guesses about which questions may be answered by the full article and thus sometimes write questions that cannot be answered given the complete article. We encour- age workers to start their questions with different question words (what, why, how, can, should, who). Our crowd-workers must be Master Workers, live in the UK or US, and have HIT approval of at least 95%.
Our contributions are follows: (1) We cre- ate and publish WHERE (WikiHow Evidence REtrieval), a new high-quality evaluation test set of lexically diverse and open-ended questions for instructive articles taken from WikiHow. Evidence sentences are annotated with high agreement in all documents. WHERE contains both questions with evidence in the article and questions without. (2) We evaluate ChatGPT this dataset in zero- and few-shot settings. Our experiments show that despite decent results on retrieving evidence for questions with evidence in the text, ChatGPT fails to recognize questions for which the text does not provide any evidence. When provided with a few no-evidence examples in the prompt, it refuses to answer if there is no evidence, but at the expense of recall of sentence retrieval. (3) We make our dataset and code publicly !vailable.
Answer evidence annotation. We tokenize doc- uments into sentences using NLTK (Bird et al., 2009) and rule-based corrections, e.g., for enumer- ations. We then use the web-based annotation plat- form INCEpTION (Klie et al., 2018) to mark all sentences in an article that provide evidence for answering question. We double-annotate 570 questions from 95 documents in two teams: team is composed of several authors of this paper, the other team of paid annotators with engineering backgrounds and prior experience in NLP annota- tion tasks, who participate training phase. We allow annotators to discuss difficult cases within each team.
2 WikiHow Evidence Retrieval Dataset
Our goal is to collect questions with and without answer evidence the text that are lexically, syn- tactically, and semantically diverse. In a pre-study, we find it hard to achieve this goal when the con- tent of the article is known to the writer. We resort to crowdsoureing for question writing, and iden- tify good cases by double-annotating the sentences deemed evidence for the answer. Our reasoning is that if agreement on this task is low, the ques- tion is either somewhat ill poo-p or too close to the overall topic of the article.
Agreement. We treat the annotations of one team as the gold standard and the other team's as the system. They agree with procision/rccall/FI of 0/.6/57.3/63.3 on whether a document provides no evidence for question at all. This corresponds to a k-score (Cohen, 1960) of 0.43, which can be interpreted as moderate agreement according to Landis and Koch (1977).
To ensure a high-quality evaluation set, we com-
ht ppttps:/ www. wwwwhww.www..o.. wik.how.ccom/Special Export *e thank WikiHow for granting us permission to redis- tribute the texts.
httreess//ttttoo.ommmbbrrrrrrrrr findings2823

---

Statistic
Value
Question Cat. Count
Question Cat. Count
# documents
91
how
41 16.1 which
22
8.7
# questions
254
what
61
24.0 who
3.1
with evidence document
129
when
2.4
why
2.4
no evidence document
125
where
5.1 yes/no
97 38.2
# questions/docummnt
2.8:1:.11
# evidence sentenccs/question
10.0+12.2
Table 2: Distribution of question types in the test set.
# sentences/document
71.5++:2..5
# tokensdoccument sentence
16.4+889
# tokens/question
13.6:13.5
shot setting, we prompt ChatGPT using a template that asks the model to output a list of sentence IDs that can provide evidence for the answer. The list is empty if the model does not retrieve any evidence (see Appendix A). In the few-shot set- ting, we configure the message history such that ChatGPT sees five training examples from the test instance's category, e.g., Health, using the same prompt template as in the zero-shot setting but in- cluding gold responses. The training instances con- sist additional question-arie article pairs annotated by the paid annotator team only. Every five-shot training set contains exactly two no-evidence ques- tions and three with-evidence questions, which are about different documents.*
Table 1: Corpus statistics. "questions with evidence.
pute question-level precision, recall, and Fl for the binary task of deciding whether a sentence provides evidence for answering a question. We keep only the questions with an Fl score of at least 0.3, and the questions that both teams consider to not have evidence. Annotators often disagree if the question is somewhat unclear or if the text contains only evidence of part of question's 's aspects. By de- sign, filtering using positive threshold for Fl removes any questions that only one team consid- ered to have evidence in the article. Sentence-level K. 0.60, which is generally considered a solid score in semantic annotation tasks. For creating our gold standard, we take the union of the sen- tences marked by both teams as relevant evidence, disagreements on the filtered cases are mostly due to different decisions on how much context to include.
To accommodate for ChatGPT's context win- dow size of 4096 tokens, we only use chunks of the training articles in the few-shot setting. Ap- pendix provides details.
Since ChatGPT may generate anything even though ask it to output list, we need post-process the model outputs before evaluation. We first attempt to parse the model output using Python's eval function. this fails, we try to match lists within the string (as ChatGPT some- times provides explanations or just copies the se- lected sentences in addition to the list) using var- ious regular expressions. If none of this works, which happens only three times in the entire ex- periment, we manually parse the model output, A small number of questions is rejected by OpenAl's content filter. For those, we assume the empty list as model output as ChatGPT does not retrieve any evidence from the article.
2.2
Dataset Analysis
Table shows the corpus statistics for our final test set. About half of the questions can be answered based on the evidence in the corresponding docu- ment (with-evidence questions), the other half can- not (no-evidence questions). Examples are shown in Figure The instructive texts in WikiHow are kept simple as indicated by the short average sen- tence length. However, documents are long, which adds another challenge to our setup. For with evidence questions, on average, about 14% of the sentences are part of the evidence.
Table 2 illustrates the distribution of question types: questions are indeed varied. Out of the yes/no questions. 81% % ask for specific facts (is/are there, will/do/does)) and 19% ask for suggestions or recommendations (shouldJcam/could).
Experiments and Analysis
Evaluation metries. For no-evidence questions, we report recall for correctly recognizing that an article does not contain any evidence for answer- ing a question. For with-evidence questions, we compute precision, recall, and Fl score of the rele-
3 Method
We evaluate ChatGPT (version of March 2023, built upon GPT-3.5) in two settings. In the zero-
"We also attempted manual prompt engineering, leading to similar results.

---

vant class ("evidence sentence") per question, and report macro-average over questions.
With-evidence questions mac.R
Recall no-evidence
Model
mac.P
mac.F1
Baselines. The random baseline for with- evidence questions predicts that a sentence con- tains relevant evidence in 14% of the cases, which corresponds to the average percentage of sen- tences marked as evidence per question in the test set.Similarly, it predicts "no-evidence" for a ques- tion with a probability of 49.2%. The results re- ported for "human scores" correspond to the agrec- ment scores.
random baseline
14.0
14,0
11.3
49.2
ChatGPT O-shot
37.9
52.6
37.7
7.2
ChatGPT 5-shot
38.2
42.8
35.5
60.3
+ 2.0
2.1
+ 0.9
+ 5.5
"human" scores
64.4
77.0
64.2
63.3"
Table 3: Results on our evaluation set. For 5-shot, we report average and standard deviation of 5 different randomly sampled per-category 5-shots results. "Har- monic of the two recall scores, estimated all 570 aanbtl-d questions.
Results. Table 3 shows the zero- and few-shot performance of ChatGPT, separately evaluated on with-evidence and no-evidence questions. In the zero-shot setting, ChatGPT outperforms the base- line for the with-evidence questions by large margin, but fails to recognize when there no evidence for a question in an article. When pre- sented with five training instances of which two are no-evidence questions, ChatGPT almost reaches human performance on recognizing no-evidence questions. This, however, comes at the cost of decreased performance on the questions with ev- idence. On with-evidence questions, ChatGPT is still far from human performance both zero- and few-shot settings.
Table in Appendix C). In addition, we needed to manually parse 30 model outputs (compared to one when using ChatGPT).
Impact of pre-training data. It is well possible that some of the WikiHow articles (or earlier ver- sions thereof) have been part of ChatGPT's training data. For our evaluation setup, however, this is not an issue, since the documents are contained in the model input anyway. We intentionally did not crawl questions from the web e.g., the WikiHow com- ments sections, but let crowd- workers write them methodology mitigating bias towards the articles content. Hence, it is unlikely that ex- isting LLMs have seen these specific questions or their answers as part of their pre-training.
based
Analysis. In the zero-shot setting, OpenAl's content filter identifies the content of 10 of the test cases harmful and declines to generate outputs for them. We do not consider these instances to be harmful, e.g., they are about instructions on how to find emergency procedures in hotels. For the few-shot configurations, these cases vary from 11 16 instances, depending the few-shot prompts. This illustrates that improving content filters is an important future research direction.
Related Work
Extractive QA datasets. The SQuAD datasets (Rajpurkar et al., 2016, 2018) are the largest datasets for extractive QA. The questions are fac- toid and can be answered through short, single answer spans. In contrast, our dataset includes non-factoid questions based on instructional text and requires models to identify set of answer spans across long documents. The MultiSpanQA (Li al., 2022) and MASH-QA (Zhu al., 2020) datasets also contain QA pairs with several answer spans. However, MultiSpanQA is automatically de- rived from Natural Questions (Kwiatkowski et al., 2019) and MASH-QA also contains automatically created QA pairs. Only few datasets contain unanswerable questions, e.g., the latest version of SQuaD. However, its questions have been cre- ated given text passage, whereas the questions of W WHERE based only on the keywords and a summary of an article. This leads to larger lexical variety between the question and the text and, thus, creates a more realistic and challenging setting.
Table 4 breaks down the evaluation results by the question types identified in Table Comparing how-, what- and yes/no questions, we find that the latter are the easiest in both the zero- and the few- shot setting. In terms of recognizing that there is no evidence for question, how questions are consistently more difficult than what and yes/no questions.
Applicability beyond ChatGPT. While this pa- per focuses ChatGPT, WHERE can be used evaluate any other large LM whose training data does not contain WHERE, We also evalu- ated Llama 2-Chat (Touvron et al., 2023b) in the zero-shot setting, finding to perform consider- ably worse than ChatGPT in the same setting (see

---

Type (#w/#n)
With-evidence questions mac.R mac.F1
Recall no-evidence
previous studies also reveal short-comings of GPT models, for instance, their hallucination problem (Ouyang et al., 2022a; Li et al., 2023; Zhang et al., 2023). To address this issue, we propose to ana- lyze the usage of ChatGPT in a setting in which we force it to retrieve evidence from text instead of or complex question answering (Tan al., 2023),
Setting
mac.P
how
O-shot 5-shot
38.1
55.0
35.1
0
(21/20)
38.6
40.2
33.8
42.0
5.0
+ 3.1
3.9
7.6
what (41/28)
O-shot 5-shot
31.4
48.7
32.1
10.0
35.9
39.6
31.0
56.0
+ 2.9
2.1
+ 3.8
+ 8.2
directly generating an answer.
when (0/6)
0-shot S-shot
0
43,3
+ 14.9
Conclusion and Outlook
where (7/6)
O-shot 5-shot
48.3
51.4
46.9
16.7
In this work, we have introduced a challenging benchmark to evaluate ChatGPT on the task of re- trieving evidence sentences from instructive text to answer a question. We have presented new high-qualite dataset from WikiHow consisting of crowd-sourced questions and expert evidence an- notations. A special real-world challenge of our dataset is the inclusion of questions that are not answerable given an article. When evaluating Chat- GPT on our dataset, we found that it fails on de- tecting no-evidence questions if not provided with targeted examples in the prompt. Our results on the new benchmark highlight important shortcomings of generative large language models that need be addressed by future research: besides hallucinat- ing answers, there is no free lunch with regard calibration in eeitence-retrieval setups.
35.2
52.3
35.9
60.0
8.2
1 20.7
11.9
1 9.1
which (13/9)
O-shot 5-shot
58.6
47.0
41.1
47.7
39.7
68.9
4.4
+ 3.8
12.2
who
0-shot 5-shot
21.4
100
34.7
0
(2/6)
67.7
95.0
75.2
80.0
5.0
+
+
7.5
why (2/4)
O-shot S-shot
12.5
50
20.0
25.0
20.0
40.0
26.7
30.0
T 11.2
22.4
14.9
+ 11.2
yes/no (43/54)
O-shot S-shot
42.4
51.5
41.0
9.3
39.4
41.2
36.8
69.3
4.9
5.0
+
2.6
+ 7.6
Table Results of ChatGPT on our evaluation set, sep- arated by question type. For 5-shot, we report average and standard deviation of 5 different randomly sampled per-category 5-shots results. in the dataset. #w: number of with-evidence questions, #n: number of no-evidence questions.
questions of this type
Limitations
Since we accessed ChatGPT via the OpenAl API. results might not be reproducible if the model behind the API is exchanged with new version. Moreover, previous work has shown that ChatGPT is stable only to an extent of 79% for complex question answering tasks (Tan et al., 2023).
Evidence retrieval. Our work builds on a long line of NLP work addressing the fine ane-grained re- trieval of supporting facts from text. This is common basis for tasks like question answering (Murdock et al., 2012), fact checking, or slot filling (Petroni al., 2021), and has also been used sup- port other tasks, c.g., question generation (Lewis et al., 2020), document retrieval (Akkalyoncu Yil- maz et al., 2019) or natural language inference (Vladika and Matthes, 2023).
Another limitation of using ChatGPT via the API is that we do not have access to the model parame- ters and probability distributions. This reduces the amount of analysis we perform the model results.
Analysis of ChatGPT. Since the publication of ChatGPT, many papers focus on analyzing the ap- plicability of the system (Liu et al., 2023b) for dif- ferent tasks and domains, i.a., question answering for education (Frieder et al., 2023) and in medi- cal environments (Nov et al., 2023). This large attention indicates the importance of the system for real-world applications and is the main motiva- tion for our work. Despite promising performance in several benchmark datasets, for instance logi- cal reasoning comprehension (Liu et al., 2023a)
The dataset we present in this work is double- annotated and, as result, of high quality but rather small compared to other question-answering datasets (but other datasets are often crowd- sourced, created using heuristics, and/or contain less varied questions).
Ethics Statement
Data source and licensing. We ensured that we may re-publish the WikiHow texts under CC BY- NC-SA-3 3.0 by obtaining written permission from

---

WikiHow. Some of the topies of the selected arti- cles are about health and relationships, yet, there is no personal information involved.
In 30th USENIX Security Symposium, USENIX Se- curity 2021, August 11-13, 2021, pages 2633-2650. USENIX Association.
Crowd-sourcing and annotation. We collect set of questions via Amazon Mechanical Turk. We pay dollar per HIT, but increase to dol- lars/HIT in the subsequent tasks, given the average completing time (between and 10min).
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Mea- surement, 20(
Yang Deng, Wai Lam, Yuexiang Xie, Daoyuan Chen, Yaliang Li, Min Yang, and Ying Shen. 2020. Joint learning of answer selection and answer summary generation in community question answering. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAT Symposiun on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7651- 7658. AAAI Press.
The paid annotators participating in our project were completely aware of the goal of the anno- tations and even helped designing the annotation scheme. They gave explicit consent to the publica- tion of their annotations. The main annotator was paid considerably above country's minimum wage. For this type of semantic annotation lask not involving any personal data, our institution did not require obtaining an IRB Review.
Yue Dong, John Wieting, and Pat Verga. 2022. Faithful to the document or to the world? mitigating hal- lucinations via entity-linked knowledge in abstrac- tive summarization. In Findings of the Asvociation for Computational Linguisties: EMNLP 2022, pages 1067-1082, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Acknowledgements
We thank Dragan Milchevski and Zhe Feng for helpful discussions on the fho-shot setting.
Simon Frieder, Luca Pinchetti, Ryan-Rhys Grif- fiths. Tommaso Salvatori, Thomas Lukasiewiez. Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023. Mathematical capabilities of ChatGPT. CoRR, abs/2301.13867.
References
Zeynep Akkalyoneu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019. Cross-domain modeling of sentence-level evidence for document retrieval. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing DOMNLP-IJCNLP pages 3490- 3496, Hong Kong, China. Association for Computa- tional Linguisties.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu. Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang. Wenliang Dai, Andrea Madotto. and Pascale Fung. 2022. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55:1 - 38.
Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho. and Iryna Gurevych. 2018. The INCEpTION platform: Machine-assisted and knowledge-oriented interactive annotation. In Proceedings of the 27th International Conference Computational Linguistics: System Demonstrations. pages 5-9, Santa Fe, New Mexico. Association for Computational Linguistics.
Steven Bird, Ewan Klein, and Edward Loper. 2009, Nat- ural Language Processing with Python. O' O'Reilly Media, Inc.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Matcusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, abs/2005. 14165.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illi Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wo Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452-466.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, ÚI- far Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models.
J Richard I Landis and Gary G Koch. 1977. An appli- cation of hierarchical kappa-type statistics in the as- sessment of majority agreement among multiple ob- servers. Biometrics, pages 363-374.

---

Patrick S. H. Lewis. Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kütteer, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe generation for nowwledg-intensivv NLP tasks. In Advances in Neu- ral Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022b. Training language models to follow instruc- tions with human feedback. In NewrIPS.
Kiela. 2020.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis. Majid Yazdani, Nicola De Cao, James Thore,, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online. Association for Computational Linguisties.
Haonan Li, Martin Tomko, Maria Vasardani, and Tim- othy Baldwin. 2022. MultiSpanQA: A dataset for multi-span question answering. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2250-1260, Seattle. United States. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Language models are unsupervised multitask learners.
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian- Yun Nie. and Ji-Rong Wen. 2023. HaluEval: A large- scale hallucination evaluation benchmark for large language models. CoRR, abs/2305.1 11477.
Pranav Rajpurkar, Robin Jia, and Percy I. Liang. 2018. Know what you don't know: Unanswerable questions for squad. CoRR, abs/1806.03822.
Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023a. Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4. CoRR, abs/2304.03439.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguisties.
Yiheng Liu, Tianle Han, Siyuan Ma. Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Dajiang 7hu, Xiang Li, Ning Qiang. Dingang Shen, Tianming Liu, and Bao Ge. 2023b. Summary of ChatGPT/GPT- 4 research and perspective towards the future of large language models. CoRR, abs/2304.01852.
Yiming Tam, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yon- grui Chen, and Guilin Qi. 2023. Evaluation of Chat- GPT as a Question Answering System for Answering Complex Questions. CORR, abs/2303.07992.
Nick McKenna, Tianyi Li, Liang Cheng, Moham- mad Javad Hosseini, Mark Johnson, and Mark Steedman. 2023. Sources of hallucination by large language models on inference tasks. CoRR, abs/2305. 14552.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. CORR. abs/2302.139)
J William Murdock, James Fan, Adam Lally, Hideki Shima, and BK Boguraev. 2012. Textual evidence gathering and analysis. IBM Journal of Research and Development, 56(3.4):8-I.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi. Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton- Ferrer, Moya Chen, Guillem Cucurull, David Esiobu. Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller. Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn. Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Di- ana Liskovich. Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten. Ruan Silva, Fric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan,
Oded Nov, Nina Singh. and Devin Mann. 2023. Putting ChatGPT's medical advice to the (Turing) test. CoRR, abs/2301. 10035.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022a. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744. Curran Associates, Inc.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroli L. Wainwright, Pamela Mishkin, Chong

---

Melanie Kambadur, Sharan Narang, Aurélien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307 .09288.
Juraj Vladika and Florian Matthes. 2023, Sebis SemEval-2023 task 7: A joint system for natural lan- guage inference and evidence retrieval from clinical trial reports. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-202) pages 1863-1870, Toronto, Canada. Association for Computational Linguistics.
Muru Zhang, Ofrr Press, William Merrill, Alisa Liu, and Noah A. Smith. 2023. How language model hallucinations can snowball. CoRR, htn/2305.11534
Ming Zhu, Aman Ahuja, Da-Cheng Juan, Wei Wei, and Chandan Reddy. 2020. Question answering with long multiplc-span answers. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pages 1840-3849, Online, Association for Computa- tional Linguistics.

---

Supplementary Material
System
A Prompt Template
User (m))
Figure 2 shows the complete message histories passed through ChatGPT generate system out- puts.
Assistant (n1)
Our prompt template for ChatGPT is as follows:
System message: Your task is to select sentences from document that answer given question.
(ma)
Assistant (ya)
System
User message (question, document): Select sentences from the document below that answer the question below. It may also be the case that none the sentences answers the question. In the document, each sentence is marked with an ID. Output the IDs of the relevant sentences as list, e.g., "[1,2,3]", and output "[]" if no sentence is relevant. Output only these lists.
User (zr)
User (2t)
Zero-shot
5-shot
Figure 2: Message history passed as input to ChatGPT in zero- and few-shot setting. He. test instance, dy training instance i, ground truth for x;.
With-evidence questions mac.P mac.R mac.F1
Recall no-evidence
Model
question:"<<quettonn
random baseline
14.0
14.0
11.3
49.2
Document: <ddccmmnnt"
ChatGPT O-shot
37.9
52.6
37.7
7.2
Llama 2 Ohohot
21.9
23.7
18.6
6.4
ChatGT S-shot
38.2
35.5
60.3
B Sampling Chunks for Few-Shot Instances
+ 2.0
2.1
+ 0.9
5.5
"human" scores
64.4
77.0
64.2
63.3*
To fit ChatGPT's maximum input size, only pro- vide chunks of the few-shot training instances. We use the tiktoken library to count the number of tokens needed for prompting ChatGPT and create chunks accordingly. The largest test input instance requires 2235 tokens, i.c., 1861 tokens remain for the few-shot instances and the model output, called "completion" by OPenAL. In the ground-truth an- swers, the maximum number of completion tokens is 189. To enable some flexibility of the model, reserve 300 tokens for completion, which equals the amount of tokens needed to encode list of sentence IDs from 0 to 99 including. Hence, 1561 tokens remain to encode the 5 training instances, i.e., approximately 312 tokens per training instance (including the ground-truth answer). We fill these tokens from the training instances by sampling context window of at most 350 tokens around ran- dom (relevant case of question with evidence) sentence.
Table 5: Results on our evaluation set. For 5-shot, we report average and standard deviation of 5 different randomly sampled per-category 5-shots results. *Har- monic mean of the two recall scores, estimated on all 570 double-annotated questions. Ilama 2: Llama 2- Chat (13B).
C Additional Results
Here, we provide an extended version of Table 3 that includes scores for Llama 2-Chat (13B) (Tou- vron et al., 2023b) in the zero-shot setting. Com- pared to CharGPT, LLama 2 is equally bad at detect- ing no-evidence questions and recognizes relevant sentences for answering with-evidence questions worse
D Crowdsourcing
We show the interface that we used for collecting the questions Amazon Mechanical Turk in Fig- ure 3, The interface contains two parts. On the left. we show croke-workers two components based on the WikiHow article: the summary and the key- words. The summary is the first paragraph from a
"To entire documents, we would needed to use the 32k version GPT-4, which up to 60x more expensive than ChatGPT.
hnttps://githut com/ con/openai//e /tiktoken

---

WikiHow article, which typically functions as an introduction to the article's 's topic. The keywords contain set of keywords that we generated based on the WikiHow article using the Wordcloud' pack- age. On the right, we provide a form where crowd- workers can submit six questions. Crokd-workers are encouraged to make use of our suggestions on how to start question and to start each question with different word. Furthermore, in the first three input fields, we ask crowd-workers tion about a certain topic. In Figure 3, these topics are, from top to bottom: Researching Upgrade Op- tions, Planning Your Honeymoon and Upgrading on the go. These are the section headers from the WikiHow article, which we use to collect questions that are relevant to the article.
ask a ques-

---

Ask question about Plenning Your Honeymoon Use different start word.
Ask question about Upgrading on the Go. Use different start word.
How can/should When, Why, What, Where, Which, Who Do N.. Can 1. is it.... Should
quesiion about Researching Upgrade Options
a question Use different start word.
Ask question. Use different start word.
Ask question. Use differemt start word.
to
Questions
Question 5
Question
Question
Quiestion
Quiestion
Question
make, esk, offer, give, may mention, try. sign, take, upgrade, will, call, bring, love. help. Include. look, book. want, arrive, dress, onloy, sharc, add. gift. stort, plan. provide, quallfy, search, talk, know, request. reserve. say show, feel, tip. collect, maintain. leed, allow, encourage spend, use, register. complete. check conaider
honeymonn. upgrade, free, car staff, card, company, travel, perk hotel, special. small, incentive, sure, mile, flight, good, reservation, compact. frequent, flier, credit. rental, wey. likely, sirport, book, chance, resteurant, seat gift trip, aitline progrem lot, friendlly, wedding well, couponn destination, demand, saston agont, persnn people, extra, important. tip. everyone
encure
With littlc rosearch, some stratogic planning, and willingeess to share your new
some upgradee upgredes. Just maintain a kind, anthusi astic itrtudde and got ready for an upgradad honeymoon of lifetime!
The key pwordss (verts and other words] be are related flow Get Free Honeymoon Upgrades. Use them in your questions.
Everyene loves upgrades triss, honeymoon to enjoy thoyy.ttmm. them.
How to Set Free Honeymocn Upgrades
go, you're
Summary
Keywords
Verbe:
Other
Figure 3: The interface used our crowdsourcing set-up for collecting questions. The left part the interface shows the summary and keywords based the WikiHow article and the right shows the form that workers used to submit their questions.

---

E
Dataset Example
Title: How to Find Trusted Advice on Covid-19
Questions:
1. Is there a central authority that I should check with for COVID-19 advice?
[5, 6, 7]
2. What groups does the most research on COVID-19? no-evidence
3. How can T know I'm hearing the truth about COVID-19 when listening to someone?
[18, 23, 27, 28, 32, 38, 39]
4. Who should I follow to find out about the latest COVID-19 variants? no-evidence
Article Text:
[1] Whether you't re surfing the web, texting friend, or tuning into the nightly news, you're ro''r probably hearing lot of different things about the COVID-19 outbreak.
[2] It's difficult to get finger on the pulse of what's going on during the current state of the world, but there are several ways to make the fact-checking process a bit easier.
[3] If you know where to look, and subsequently, where to stay away from, you can stay informed as the COVID-19 situation continues to develop.
[4] ==Steps==
[5] == Reliable Organizations===
[6] Consult WHO and the UN for reliable, global updates.
[7] The World Health Organization (WHO) and the United Nations (UN) are constantly studying
and reporting on COVID-19 cases all over the world.
[8] These organizations' websites offer plenty of resources and articles that you can peruse, which can help keep you up-to-date on the latest news and best practices for staying safe during the pandemic. [9] You can find list of common COVID-19 mythbusters here:
[10] Visit the COVID-19 "hubs" on various social media and internet platforms.
[11] Sites like Facebook, Apple News, Google, Snapchat, and Twitter have all created special "hubs," or
featured sections of information pertaining to the COVID-19 outbreak.
[12] You'll '1l have use bit of your own discretion as you go through the different news bytes however, many of these platforms try to prioritize more reliable news sites.
[13] These are the casiest ways to stay up-to-date with the newest COVID-19 developments.
[14] Stop by the Johns Hopkins site for accurate reports of case numbers.
[15] It can be bit gloomy to think about how many COVID cases there are in the world currently.
[16] However, if you want a more exact count, the Johns Hopkins Center for Systems Science and Engineering runs dashboard that calculates the current number of global cases.
[17] You can find this site here:
[18] Follow trusted experts on social media for factual information.
[19] Platforms like Twitter can rife with misinformation if you're re following the wrong people, but they
can be a great source of news and factual information if you't 're following medical experts.
[20] Doctors and members of the medical community are great people to listen to during the pandemic.
[21] ===Unrelablle Information Sources===
[22] Don't put too much stock in random social media posts.
[23] The current climate has left lot of people nervous and anxious about the coming days, which
is perfectly understandable.
[24] can be easy believe what you see on social media, but take the time to scrutinize posts carefully and discredit any info you find about home-brewed cures or solutions for COVID-19.
[25] Search for studies only published by reliable groups

---

[26] If you're browsing through different studies, look over the specific publication details.
[27] While many studies are authoritative, there can be different factors that impact study's credibility, like the funding source, or where the study was first published.
[28] Generally, try to get your information from studies published in reputable journals.
[29] Fact-check new information against reputable organizations.
[30] Studies are teaching us new things every day, but they aren't It rewriting the rules when it comes to
COVID-19, either.
[31] Even if new information is released in study, you shouldn't toss out everything you 've learned out the window, either.
[32] Try to fact-check new information against reputable organizations, so you can get an idea of what believe.
[33] For instance, you can cross-check new studies with WHO.
[34] Report misinformation as you find it online.
[35] It can be really frustrating to find blatantly incorrect things online.
[36] Thankfully, most sites and platforms give you the option to report misinformation, which can help
make the online world safer place.
[37] You can figure out the best way report misinformation here:
[38] Tips
[39] "hhe best way stay informed is to cross-check new information over several reliable sources.
[40] *If a source follows strict fact-checking process, it's probably a safe source of information.
[41] === ==aanings==
[42] Don't fall for the narrative that certain demographics are more "likely" to believe false facts or
spread misinformation.
Discussion:
An example for a clear case is provided for question 1: sentence [6] "Consult the WHO and UN for reliable, global updates." clearly provides evidence for the answer to question 1.
Answering question 3 based on the text is possible, but requires more reasoning. For example, sentence [18] can be considered as evidence in the sense of "if an expert is trusted, you will get factual information from them, and know that you're 're hearing the truth about COVID-19." Sentence [23] is borderline case (which this author annotator would not have marked), but we see the interpretation that one should rather distrust nervous and anxious people.
Both annotator teams agree that the article does not provide any evidence for questions 2 and


Reviews:
[{"review_id": "jweskN2BAt", "reviewer": "Reviewer_Vngq", "summary": "", "questions": "1. Which backbone of ChatGPT is evaluated?\n2. How do you split the document into numbered sentences, as shown in Figure 1?\n3. l. 116, if the annotators are asked to write questions that can be answered by the document, then where are the no-evidence questions coming from?\n5. l. 253 - l. 258, are there any examples to illustrate the failed cases?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper, and for valuing our new resource. As the other reviewers mention, in addition to contributing a new dataset and methodology to construct such datasets, the important main finding of our paper is that ChatGPT almost completely fails to recognize the no-evidence questions in the zero-shot setting.\n\nComments on the reasons to reject you list:\n\n1.\tThanks for suggesting to report further baselines (e.g., a fine-tuning baseline) to the experiments. We also ran a preliminary check on Llama 2 (13B), finding it to perform considerably worse than ChatGPT, and are happy to add these scores to the paper. However, please note that our point is not that ChatGPT outperforms a random baseline, but that the **random baseline actually outperforms ChatGPT when it comes to recognizing questions that cannot be answered based on a text** (last column of Table 3)! Contrasting the 0-shot and the 5-shot rows, we see that ChatGPT can be taught to realize this, but at the cost of correctly answering the with-evidence questions.\n\n2.\tWe agree that the experiments in our paper are relatively straightforward. However, the development of the methodology was not, and necessitated a number of experiments and annotation steps. We ensured question diversity in the first step of the dataset collection using a carefully designed crowdsourcing interface (see Figure 3 in the Appendix) and then doubly-annotated the dataset in a second step to filter the dataset for high-agreement cases. We argue that our **short** paper contains a sufficient amount of discussion and analysis of the issue of realizing whether a text does or does not contain evidence for a question in the context of real-world problems and LLMs. \n\n3.\tOn the size of the dataset, please see the discussion in our Limitations section. It is true that for three question types (when, who and why) there are fewer instances, but for the other diverse question types (how, where, which, yes/no), we collected a larger number of questions. We believe that the dataset itself and the associated methodology provide enough grounding for future research. We will follow your suggestion to add more examples to the paper in order to illustrate the lexical diversity of our dataset. Please note that the entire dataset will also be released. Our sentence-level agreement (Cohen’s kappa) is 0.6, which is considered as moderate (almost substantial) agreement according to Landis and Koch (1977). In the context of semantic annotation tasks, this is usually considered as sufficient agreement.\n\n4.\tThank you for this feedback, the questions you provide are easy to clarify and we will gladly add the requested information to the paper.\n\nAnswers to your questions:\n\n1.\tWe use ChatGPT with GPT-3.5 (version of March 2023) as its backbone.\n\n2.\tThe sentence tokenization is based on NLTK and rule-based corrections (e.g., for enumerations).\n\n3.\tFor each article, annotators only see the title and the first paragraph (see Section 2.1). Additionally, they get a set of keywords generated from the full article, but they never see the full article. Hence, they can only make educated guesses about which questions may be answered by the full article (cf. l.111-l.117 and Figure 3) and thus sometimes write questions that cannot be answered given the complete article.\n\n4.\tWe will use the extra page in the camera-ready version to include these examples in the paragraph starting at line 252 and potentially add more to the appendix.\n\nMissing References:\n\nThank you for this suggestion. As our short paper focuses on the machine reading component, we mainly discussed this in the related works section. We will gladly add some references on evidence retrieval", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of using ChatGPT (or other LLMs) for evidence retrieval is sound.\n2. The evaluation of ChatGPT provides discoveries and insights that may benefit future works.\n3. The paper present new resources with a new dataset.", "reasons_to_reject": "1. The conclusion is not well-supported. There's no persuasive baselines since the only one being compared is a random baseline. \n2. The work is fairly simple, the author provides with limited analysis of the results and no extra discussions or ablation studies. \n3. The final dataset comprises only 254 questions from 91 documents and includes only a few samples for 3 question types, I am not convinced that the dataset is high-quality and lexically diverse.\n4. The paper needs careful proofreading and revising, as many details are missing or explained unclearly. Please refer to the question section.", "questions_to_authors": "1. Which backbone of ChatGPT is evaluated?\n2. How do you split the document into numbered sentences, as shown in Figure 1?\n3. l. 116, if the annotators are asked to write questions that can be answered by the document, then where are the no-evidence questions coming from?\n5. l. 253 - l. 258, are there any examples to illustrate the failed cases?", "paper_topic_and_main_contributions": "This paper investigates ChatGPT's ability for evidence retrieval. The contribution includes developing a new evaluation dataset that comprises questions with evidences and questions without. In addition, the author prompts ChatGPT to retrieve evidences from a given document to the question. The model also needs to recognize when there's no evidence for answering the questions. Experimental results including a zero-shot and 5-shot setting demonstrate that ChatGPT outperform a random baseline.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The paper focuses on evidence retrieval. However, there's no discussion of related works on this task.", "Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\nl.181, \"of\" → \"if\""}, "extra_text": {}}, {"review_id": "0Jc20pKglp", "reviewer": "Reviewer_cu6q", "summary": "", "questions": "How do you see the application of this evaluation method to other large language models beyond ChatGPT?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper, and for appreciating our new resource and the insights on “capabilities and limitations of ChatGPT” that our evaluation provides.\n\nWe agree with you that our dataset is relatively small compared to other QA datasets, but it is of high quality (please also see the discussion in the Limitations section). The dataset is large enough for the focused contributions of this short paper: demonstrating the capabilities and weaknesses of ChatGPT-like models when retrieving evidence from instructive texts. \n\nRegarding your question on the application of our evaluation method to other LLMs beyond ChatGPT: Any LLM can be tested with our dataset (given that it is not comprised in the LLM’s training data). The methodology that we propose to collect high-quality testing data can be used to create further data, also in other genres/domains. Constructing high-quality test sets has become an important goal, as the training data of LLMs is often not disclosed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses a pertinent issue in the NLP community, which is the evaluation of large language models like ChatGPT in real-world scenarios.\n2. The creation of a new dataset specifically tailored for this task is commendable. The dataset's focus on lexically diverse and open-ended questions makes it a valuable resource for future research.\n3. The systematic evaluation provides insights into the capabilities and limitations of ChatGPT, which can guide future improvements in the model.", "reasons_to_reject": "The dataset, while of high quality, is relatively small compared to other question-answering datasets.", "questions_to_authors": "How do you see the application of this evaluation method to other large language models beyond ChatGPT?", "paper_topic_and_main_contributions": "The paper primarily focuses on evaluating ChatGPT's performance in the context of evidence retrieval from WikiHow articles. The main contributions of the paper are:\n\n1. Creation and publication of a new high-quality evaluation test set comprising lexically diverse and open-ended questions for instructive articles.\n2. sourced from WikiHow. This dataset includes both questions with evidence in the article and questions without.\n3. A systematic evaluation of ChatGPT on this dataset in both zero- and few-shot settings. The results highlight ChatGPT's strengths and weaknesses in recognizing evidence-based and no-evidence questions.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2noH4aGCHI", "reviewer": "Reviewer_zJHM", "summary": "", "questions": "Would the results be affected if WikiHow articles were included in ChatGPT's training data?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper, and for calling our dataset “well-designed and carefully constructed” and recognizing its complementariness to existing datasets.\n\nComments on weaknesses / reasons to reject:\n\n- Concern on the usefulness of our findings on ChatGPT for the research community: ChatGPT (GPT-3.5)’s outputs are often used as a teacher to train open-source model (see, e.g., https://arxiv.org/pdf/2304.12244.pdf, https://lmsys.org/blog/2023-03-30-vicuna/). Thus, analyzing the output of models like ChatGPT is necessary for the research community, with our finding on ChatGPT’s behavior with respect to no-evidence question contributing an important aspect to the bigger picture.\n- For a discussion on the random baseline, please see our response to Vngq: Our point is that **the random baseline actually outperforms ChatGPT** on the task of recognizing whether or not a text provides evidence for answering a question.\n\nOther comments:\n- Thank you for the great suggestion to add evaluation results per question type. We will add them in the camera-ready version.\n- We agree that it can be expected to some degree that ChatGPT fails to return no evidence. However, the prompt we use in the experiments explicitly tells it to output “[]” if there is no evidence in the text. From a human and an application perspective, noticing that one does not know something is crucial, and should also be required from LLMs. We have established that ChatGPT fails this requirement with respect to missing evidence in instructional texts.\n- Thank you for your suggestion to add ChatGPT’s outputs in the appendix. We will make the outputs publicly available.\n\nQuestion on effects of WikiHow articles potentially being included in ChatGPT’s training data: As the articles are anyways included in the prompt, we believe that whether or not some of the articles have been part of ChatGPT’s training data does not affect our results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed dataset can complement existing ones with a wide range of unanswerable/non-factoid questions.", "reasons_to_reject": "The findings are limited to one generative language model, ChatGPT, and might only be valid with the ChatGPT version the paper experimented with.", "questions_to_authors": "Would the results be affected if WikiHow articles were included in ChatGPT's training data?", "paper_topic_and_main_contributions": "This paper proposes a new dataset for question-answering, which contains unanswerable questions. Unlike previous similar datasets, the proposed dataset also includes non-factoid questions and requires the system to identify evidence sentences in a given text or return no answer (i.e., no-evidence). The paper uses WikiHow articles as input texts and asks human annotators to generate various question types (e.g., how, what, when, etc.). The evaluation is focused on ChatGPT with two settings: zero-shot and few-shot. The paper finds that ChatGPT fails to recognize no-evidence questions in the zero-shot setting. This issue can be alleviated in the few-shot setting but at the cost of recall of with-evidence questions.\n\nStrengths\n- The problem setting is interesting (i.e., unanswerable and non-factoid questions). \n- The dataset was well-designed and carefully constructed.\n \nWeaknesses\n- While the dataset would be used in subsequent work, I am unsure whether the findings about ChatGPT's behaviors are useful for the research community.\n- The random baseline is weak.\n\nComments\n- In the analysis (lines 261-265), it would be better if the paper also shows the statistics per question type.\n- The case when ChatGPT fails to return no evidence can be expected because it (as an LLM in general) is pre-trained to predict the next token given the context input.\n- The paper would consider adding the actual ChatGPT's outputs in the appendix.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree this work proposes a useful evaluation resource, and a surprising negative result in how poorly ChatGPT performs on “no-evidence” questions.
While the analysis is limited and the findings are only for ChatGPT, for a short paper this work provides useful resources and informative insights to practitioners using ChatGPT for question answering in ambiguous settings.
Decision: Accept-Findings