Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
Authors: Sukmin Cho
Jeongyeon Seo
Soyeong Jeong
Jong C. Park
Venue: EMNLP 2023 Findings (2023)
Keywords: Open-Domain Question Answering
Large Language Model
Prompt Engineering

Abstract: Large language models (LLMs) enable zero-shot approaches in open-domain question answering (ODQA), yet with limited advancements as the reader is compared to the retriever. This study aims at the feasibility of a zero-shot reader that addresses the challenges of computational cost and the need for labeled data. We find that LLMs are distracted due to irrelevant documents in the retrieved set and the overconfidence of the generated answers when they are exploited as zero-shot readers. To tackle these problems, we mitigate the impact of such documents via Distraction-aware Answer Selection (DAS) with a negation-based instruction and score adjustment for proper answer selection. Experimental results show that our approach successfully handles distraction across diverse scenarios, enhancing the performance of zero-shot readers. Furthermore, unlike supervised readers struggling with unseen data, zero-shot readers demonstrate outstanding transferability without any training.

Improving Zero-shot Reader by Reducing Distractions
from Irrelevant Documents in Open-Domain Question Answering
Sukmin Cho
Jeong yeon Seo
Soyeong Jeong Jong C. Park
School of Computing
Korea Advanced Institute of Science and Technology
(nelllpic, yena. seo, star starsuzi, wttpsiiiiiiik/aa/a///aa jongpark ac. kr
Abstract
Q: Who is the bad guy in Lord of the Rings?
Genersted
Large language models (LLMs)) enable zero- shot approaches in open-domain question swering (ODQA), yet with limited advance- ments the reader is compared to the retriever. This study aims at the feasibility of zero-shot reader that addresses the challenges compu- tational cost and the need for laheled data. We find that LLMs are distracted due to irrelevant documents in the retrieved set and the overcon- fidence of the generated answers when they exploited as zero-shot readers. To tackle these problems, we mitigate the impact of such docu- ments via Distraction-aware Answer Selection (DAS) with a negation-basee -based instruction and score adjustment for proper answer selection. Experimental results show that our approach successfully handles distraction across diverse scenarios, enhancing the performance of zero- shot readers. Furthermore, unlike supervised readers struggling with unseen data, zero-shot readers demonstrate outstanding transferability without any training.
retriever
Irrolvvant Docs
Lord Voldemort
0.98
Sauron
0.96
Relevant
0.40
Final Lord Voldemort
Figure I: An overview of distraction from the irrelevant documents when exploiting LLM as zero-shot reader.
and Grave, 2021). While the supervised readers show remarkable performance on ODQA, they are hampered by two weaknesses: the high computa- tional cost involved in training and the necessity of annotated query-document datasets. These limi- tations impede the transferability of readers to di- verse tasks and domains. To solve this, we aim to validate the feasibility of using an LLM as a reader, leveraging its inherent advantages while reducing the aforementioned limitations.
However, the performance of an LLM in various tasks is easily distracted by irrelevant documents (Li al., 2022; Shi et al., 2023), underscoring the importance of resolving these challenges in ODQA. The tendency of LLM to generate incorrect an- swers becomes apparent when reading retrieved sets that include irrelevant documents. These docu- ments, while related to the query, may lack the nec- essary information to provide an answer, leading to the occurrence of hallucination. This emphasizes the need for proper handling of such documents to fully harness the potential of LLM, thereby achieving reliable performance paper addresses the requisite of hallucination mit- igation to validate the possibility of an LLM as a zero-shot reader.
1 Introduction
Open domain question answering (ODQA) a task for answering questions with the evidence documents fetched from a large corpus (Voorhees and Tice, 2000). A retrieve-read framework has achieved remarkable performance in ODQA by fine-tuning the language models with labeled datasets (Lee et al., 2019; Karpukhin et al., 2020; Izacard and Grave, 2021). The emergence of large language models (L.MMs) has enabled the exploration of zhro approaches in this frame- work, with less emphasis on the reader compo- nent (Sachan et al., 2022; Chuang et al., 2023; Levine et al., 2022).
reader. This
Utilizing an LLM a reader provides an ad- vantage in the generalization ability with the rich world knowledge, unlike conventional small-sized supervised readers (Karpukhin et al., 2020; Izacard
In this paper, we propose Distraction-aware Answer Selection (DAS), handling the challenges posed by irrelevant documents and overconfident scores as shown in Figure 1. First, we provide
Corresponding author

---

models with an "unanswerable" instruction, allow- ing them to abstain from answering. Then, we adjust the answer scores by reflecting the query generation score as the relevance between the given query-document pairs. These approaches reduce the impact of irrelevant documents and improve the selection of the correct answer from the relevant document.
Distraction from Noisy Input Recent work ad- dresses the negative impact of noisy inputs when exploiting an LLM diverse tasks. LLMs are eas- ily distracted by the noisy input having incorrect or irrelevant information on machine reading compre- hension tasks (Li et al., 2022; Su et al., 2022; Shi et al., 2023). However, the ODQA task increases the complexity, where large-scale document sets ap- pear within unrelated documents. Given the impact of distracting sentences in QA (Khashabi et al., 2017; Jia and Liang, 2017; Ni et al., 2019), our approach aims to alleviate them.
We evaluate our proposed method on represen- tative ODQA benchmarks with two publicly open LLMs, FLAN-TS (Chuang et al., 2023) and OPT- IML-MAX (lyer et al., 2022). This results in sub- stantial performance improvements achieved by ours compared to naïve LLM across all scenar- ios. Note that ours effectively alleviates the hal- lucination induced by irrelevant documents by hancing the robustness against the number of docu- ments that are read. Furthermore, an LLM with our method exhibits excellent transferability compared to the supervised reader, offering the untapped po- tential of an LLM as a zero-shot reader. Our contributions in this paper are threefold:
Method
3.1 Preliminaries
To adopt the LLM into the reader, we define a two-step answering pipeline consisting of answer candidate generation and final answer selection.
Answer Candidate Generation The LLM M generates answer candidate based on the given query q, the evidence document d; in retrieved doc- uments D and the reading comprehension instruc- tion Pre via greedy decoding. This process results in an answer candidate set { d:)
We tackle the distraction incurred by irrelevant documents and overconfident scores when exploiting an LLM as a zero-shot reader in ODQA tasks.
Final Answer Selection We select the final document-answer pair p" = (a", ,d") from an an- swer candidate set based on the generation prob- ability (ar q. di, Pre) as the answer score. The document-ansser pair with the highest probability is chosen as the most likely correct answer.
We introduce Distraction-aware Answer Selection (DAS) for zero-shot reader, with the unanswerable instruction and the score adjustment eliciting its deductive ability.
We empirically verify the efficacy of our pro- posed upproach in effectively mitigating hallu- cination and unlocking the feasibility of zero- shot readers with a generalization ability.
3.2 Problem Definition
We address selecting the incorrect answer as the final one as caused by distraction from the irrel- evant documents dx. The irrelevant documents present challenge as they cannot be used to infer the correct answer, misleading the LLM to generate incorrect but plausible answers ax. The presence of such answers av in the answer set A can result in obstacles during the final answer selection.
2 Related Work
Zero-shot Approach in ODQA The advent of LLM has shown the potential that can be used in two stages without parameter updates. For the re- trieval stage, an LLM is exploited as re-ranker via query generation or document permutation (Sachan et al., 2022; Cho et al., 2023; Sun et al., 2023) or ex- panded query to diverse pseudo queries for improv- ing the performance of supervised retrievers (Liu et al., 2022; Yu et al., 2023; Chuang et al., 2023). For the reader stage, Levine et al. (2022) attempted utilize an LLM as a zero-shot reader, address- ing the irrelevant documents through a re-ranker. this study, we focus on a fully zero-shot reader without an additional module.
Another challenge arises from the overconfident scores, making it difficult to discern the incorrect answers AN from the documents dx- The LLM, being nuto-regressive model, tends to produce text sequences with high probabilities when using greedy decoding. Consequently, it becomes hard accurately determine the correct answer a* based on the generation probabilities, especially when it also includes incorrect answers like aN.

---

Top.20
Top-100
Retriever
NQ
TQA
WeeQ
SQD
NQ
TQA
WebQ
SQD
FLAN-TS-LL
23.37
52.68
16.19
19.40
17.86
46,12
15.83
15.79
W/ DAS
(+40.85)
31,51
(+22.59)
64,54
(434495)
20.14
(+105%)
39.39
(489 (489.55)
33.84
(+40 3%)
68.86
(463.65)
25,90
46.71
(1188))
BM25
OPA-
20.21
53.21
25.38
22.93
16.32
46.57
18.71
34
w/ DAS
24.72
$6.95
24,10
32.37
29.76
59.87
24.10 10
37.74
(+42.3%)
(+7,0%%)
(e3.1%)
(641.2%)
(+82.416)
(125.016)
(125.86)
(+10016)
FLAN-TS-XL
22.43
47.44
20,50
12.85
15,90
39.17
16:55
10.30
w/ DAS
(408.49)
(+35.9%)
(451,59)
(3138%)
(474.299)
(452.
(4231
DPR
OPT-IML-MAX
23.28
50.24
21.58
16.03
16.65
43.67
19:42
14.47
DAS
(+18,79)
$6.61
26.98
21,97
(+97,995)
59
Table I: EM accuracy of the final answer among the answer candidates generated from the top-k retrieved documents. The best scores are marked in bold. The number in parentheses means the improvement percentage from DAS.
BM25
DPR
Reader Model
Train Set
NO
TOA
SQU
RQA
DEg!
Multi
41.
56.8
29.8
FiD-koe
NQ TQA
45.1
26.9
54.1
64.5
27.5
34.1
29.8
33.2
FiD-arge
NQ TQA
50.8
59.2
36.2
34.0
40
30.9
69.0
31.
34.4
10
50
100
10
50
100
FLAN-TS-XL DAS
34.0
57.2
433
353
Table Comparison of ours against the supervised readers the test set under the condition of exploiting DPR. denotes the performance from its paper.
Figure 2: EM accuracy depending on the number of the documents retrieved by BM25 and DPR on TQA.
3.3 Distraction-aware Answer Selection
vance. This is formulated as follows:
We present simple yet effective Distraction-awarg Answer Selection (DAS) for a zero-shot reader. We aim to reduce the negative impact of irrelevant doc- uments in two-step answering pipeline. Initially, we offer an option to refuse responses to irrelevant documents via unanswerable instruction. To improve the final answer selection, incorporate the relevance of the query-document pair into the scoring process.
(a" ""''d" = arg (d,4))
argmax max Pu(a'la.ds.prr Prc) Parlgdd,
where Pag denotes the query generation instruction.
The query generation score from the given docu- ment is computed as:
1 Σ log P(gxlae),
log P(gld)
(3)
19
Document Selection (D.S.) We utilize the unan-
swerable instruction enhance the deduction ca- pability by giving the option not to respond. We exclude responses that belong the unanswerable response U as follows:
4 Experimental Setup
Dataset We experiment on Natural Ques- tion (NQ) (Kwiatkowski et al., 2019), Trivi- aQA (TQA) (Joshi et al., 2017), WebQuestions (WebQ) (Berant et al., 2013) and SQuAD (Ra- jpurkar et al., 2016) (SQD). For annotated ev- idence documents for query, the development sets of each dataset are used.
s' [(a,, di)la, & U, (a., d.) S}
(1)
We construct an unanswerable response set U {" ("Unanswerable", "Answer not context"). The answers U are judged unanswerable as if the reader rejects to respond to the irrelevant docu- ments.
We employ the representative sparse retriever, BM25 (Robertson and Zaragoza, 2009), and the dense one, DPR (Karpukhin et al., 2020).
Retriever
Answer Selection (A.S.) Then, we adjust the an- swer score by multiplying the query generation score in consideration for the query-document rele-
Following the settings from Karpukhin et al. (2020), the English Wikipedia dump from Dec 20, 2018, is used.

---

Reader
Correct Answer
Incorrect Answer
Total Answer
100
FI-X-
5.50 (3.59)
94 50 (005)
100
W/DAS
221,211989)
10.27 10.27 (78.07%)
13.16
OPA-MMI-MX
(5,13
96.87 (44.87))
100
60
W/DAS.
288 (10,9716)
23.14 (89.029)
26,00
40
Table 3: Average number of answers in the candidate set The number in parentheses represents the proportion relative to the total number in S.
20
w/o DAS
DAS
40
Figure 4: Distribution of answer pairs p" based on document-query relevance and answer correctness.
30
BM
20
10
10M
As the size of the retrieved set increases the like- lihood of including relevant documents, the reader should be robust to irrelevant documents. Neverthe- less, the presence of disctration becomes apparent as indicated by the performance decline without DAS, as shown in Table 1 and Figure 2, when processing more documents. This challenge is ad- dressed by mitigating the negative impact of irrele- vant documents. Our approach achieves an average enhancement of 17% in EM when reading 100 doc- uments compared to 20. This shows the robustness of our approach in handling the problem stemming from the irrelevant documents.
Figure 3: EM accuracy depending the model size. The exploited models are the families of FL.NN-T5.
Language Model We select two publicly open LLMs: 1) FLAN-T5 (Chung et al., 2022) is the family of T5 (Raffel et al., 2020) with instruction tuning; 2) OPT-IML (Iyer et al., 2022) is the fine- tuned version of OPT (Zhang et al., 2022) by in- struction meta learning. We exploit FLAN-T5-XL. containing 3B parameters and OPT-IML-MAX- 1.3B in our main experiments.
Also, find that when reading 100 documents, the use of documents collected through BM25 has more positive impact on the performance of the reader compared to documents from DPR. This finding is noteworthy, especially considering that DPR generally performs better in retriever tasks. When employing zero-shot reader, it cannot be definitively concluded that improved performance of the retriever will necessarily lead to enhanced reader performance. More details are in Appendix C.
Metrics In our evaluation, we employ the exact match (EM) accuracy metric to assess whether the reader generates the same answer the annotated answer, after applying normalization techniques such as punctuation removal. We adhere to the same normalization process utilized in previous works (Chen et al., 2017; Lee et al., 2019).
Implementation Details The reading compre- hension instruction "Read the following context and answer the question". We add "If you don't t know the answer, return unanswerable" " for the unanswerable instruction, as mentioned in Sanh et al. (2022). Also, we compute the query gener- ation score, following settings from Sachan al. (2022). More details are in Appendix B.
Comparison against Supervised Reader We di- rectly compare with the supervised readers on the aforementioned datasets and an additional held-out dataset, RealTimeQA (RQA) (Kasai et al., 2022). As shown in Table 2, the zero-shot reader with ours shows robust performance compared to su- pervised readers, DPR (Karpukhin et al., 2020) and FiD (Izacard and Grave, 2021), which perform poorly on unseen data such as SQuAD and RQA.
5 Result
5.1
Main Result
Table demonstrates the significant performance improvements achieved by DAS regardless of re- trievers, LLMs, and datasets. Our method achieves an increase in EM of 64% on average against the default, with a remarkable improvement 231 %.
We highlight their potential as a valuable alterna- tive that avoids the limitations and costs associated with supervised readers.

---

5.2 Analysis
Limitations
Our analysis is conducted on NQ with the top 100 documents retrieved by DPR with FLAN-TS-XL. Detailed analysis are in Appendix D.
Our methodology utilizes two-step pipeline to enhance the performance of an LLM as zero-shot reader, addressing hallucination issues and lever- aging its functionality. While ours fully elicit the inherent ability of the zero-shot reader from LLM, its effectiveness dependent on the capabilities and characteristics of the LLM. For example, the prompt sensitivity of an LLM is one of the impor- tant aspects to consider, as different prompts may lead to varying results. Also, the performance of an LLM is size-dependent. Although our experi- ments have yielded consistent results in numerous cases, further investigation is required to evaluate our approach with larger LLMs. Despite these limi- tations, the zero-shot approach holds great promise in terms cost-effectiveness and leveraging abun- dant world knowledge. As future advancements in LLMs are anticipated, we expect even greater im- provements in performance over the state-of-the-art supervised readers.
Impact of Model Size We conduct experiments to assess the impact of model size on performance. As shown in Figure 3, the results demonstrate that even with smaller models, ours maximizes the per- formance of an LLM as zero-shot reader. This indicates that our approach enables LLMs to func- tion effectively as zero-shot readers, even without the need for extensively large parameter sizes.
Answer Candidate Set We examine the effects of applying DAS on the answer candidate set S as presented in Table 3. Our findings highlight a remarkable shift in the distribution of answers, with changes of 16.43%p and 5.84%p observed in each reader. Substantial increases in the ratio of correct answers demonstrate that ours effectively mitigates the inclusion of incorrect answers from irrelevant documents.
Ethics Statement
Final A Answer Pair Figure illustrates an anal- ysis of the distribution the final answer pair p*. The results provide evidence that ours successfully selects documents that are relevant to the given query and enable the extraction of a higher number of correct answers from the relevant documents. Additionally, ours shows a reduction of approxi- mately 5% in the rate of incorrect answers gener- ated from irrelevant documents.
We acknowledge the possibility bias or offen- sive answer sets in utilizing an LLM as zero- shot reader. Since this paper primarily focuses on the mitigating impact of irrelevant documents in ODQA without parametric updates, addressing the issue of bias and offensive language within an LLM is beyond the scope of our paper. We are aware that ongoing research and efforts are being made by researchers address these concerns and improve the ethical aspects of LLMs. It is expected that future advancements and research in the field will contribute to addressing these biases and ensuring an ethical use of LLMs.
6 Conclusion
In this paper, we propose Distraction-awart Answer Selection (DAS) to address the irrelevant documents in the retrieved set when an LLM is used as zero-shot reader. To validate its capability, we define hallucination caused by irrelevant docu- ments and overconfident answer scores ODQA setting. Ours aims to mitigate the impact of these aspects by incorporating unanswerable instruction and adjusting answer scores for better answer se- lection. Experimental results demonstrate the effec- tiveness of our proposal in handling hallucination across various scenarios. thereby improving the performance of ODQA benchmarks. Our approach, utilizing an LLM, showcases strong generalization capabilities across diverse datasets, distinguishing it from supervised readers and highlighting the po- tential of zero-shot reader.
Acknowledgements
This work was supported by an Institute for Infor- mation and communications Technology Promo- tion (ITPP) grant funded by the Korea government (No. 2018-0-00582, Prediction and augmentation of the credibility distribution via linguistic analy- sis and automated evidence document collection). This work was also supported by the Artificial intel- ligence industrial convergence cluster development project funded by the Ministry of Science and ICT (MSIT, Korea) & Gwangju Metropolitan City.

---

References
re-ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Devel- opment in Information Retrieval, SIGIR '21, page 1803-1807, New York, NY, USA. Association for Computing Machinery.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1533-1544, Seattle, Wash- ington, USA. Association for Computational Linguis- tics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 6769-678) (7811 Online. Association for Computational Linguistics.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1870-1877 1879, Vancouver, Canada. Association for Computational Linguistics.
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ro- nan Le Bras, Akari Asai, Xinyan Yu, Dragomir R. Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2022. Realtime QA: what's 's the answer right now? CoRR, abs/2207.13333.
Sukmin Cho, Soyeong Jeong, Jeongyeon Seo, and Jong C Park. 2023. Discrete prompt optimization via constrained generation for zero-shot re-ranker. arXiv preprint arXiv:2305. 13729.
Daniel Khashabi. Tushar Khot, Ashish Sabharwal, and Dan Roth. 2017. Learning what essential in ques- tions. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLI. 2017). pages 80-89, Vancouver, Canada. Association for Computational Linguistics.
Yung-Sung Chuang, Wei Fang, Shang-Wee Li, Wen-tau Yih, and James Glass. 2023. Expand, rerank, and retrieve: Query reranking for open-domain question answering. arXiv preprint arXiv:2305. 17080.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Erii Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma. et al. 2022. Scaling instruction- finetuned language models. arXiv preprint arXiv:2210. 11416.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452 466.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shus- ter, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. 2022. Opt-iml: Scaling language model instruc- tion meta learning through the lens of generalization. arXiv preprint arXiv:2212. 12017.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 6086-6096, Florence, Italy. Association for Computational Linguisties.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ationfor Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computa- tional Linguistics.
Yoav Levine, Ori Ram, Daniel Jannai, Barak Lenz, Shai Shalev-Shwartz, Ammon Shashua, Kevin Leyton- Brown, and Yoay Shoham. 2022. Huge frozen lan- guage models as readers for open-domain question answering. In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
Robin Jia and Percy Liang. 2017. Adversarial exam- ples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2021-2031, Copenhagen. Denmark. Association for Computational Linguistics.
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2022. Large language models with controllable working memory. arXiv preprint arXiv:2211.05110
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Pupers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Linqing Liu, Patrick Lewis, Sebastian Riedel, and Pon- tus Stenetorp. 2022. Challenges in generalization in open domain question answering. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2014-2029, Seattle, United States. Asso- ciation for Computational Linguistics.
Jia HueHuei Ju, Jheng-Honn Yang, and Chuan-Ju Wang. 2021. Text-to-text multi-view learning for passage

---

Jianmo Ni. Chenguang Zhu, Weizhu Chen, and Julian McAuley. 2019. Learning to attend on essential terms: An enhanced retriever-reader model for open- domain question answering. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pages 335-344, Minneapolis, Min- nesota. Association for Computational Linguistics.
Thomas Wolf, and Alexander M Rush. 2022, Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. arXiv preprint arXiv:2302.0000
Cicero Nogucira dos Santos, Xiaofei Ma. Ramesh Nalla- pati, Zhiheng Huang, and Bing Xiang. 2020. Beyond [CLS] through ranking by generation. In Proceed- ings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 1722-1727, Online. Association for Computational Linguistics.
Dan Su, Xiaoguang Li, Jindi Zhang. Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022. Read before generate! faithful long form question answering with machine reading. In Findings of the Association for Computational Linguisticx: ACI. 2022, pages 744- 756, Dublin, Ireland, Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-tess trans- former. J. Mach. Learn. Res., 21(1).
Weiwei Sun, Lingyong Yan, Xinyu Ma. Pengjie Ren, Dawei Yin. and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large lan- guage models as re-ranking agent. arXiv preprint 2304.0900099442
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't t know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guisticy (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, Association for Computational Linguistics.
Nandan Thakur, Nils Reimers, Andreas Rücklé Ab- hishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguisties.
Ellen M. Voorhees and Dawn M. Tice. 2000. The TREC- 8 question answering track. In Proceedings of the Second International Conference on Language Re- sources and Evaluation (LREC" Athens, Greece. European Language Resources Association (ELRA).
Stephen Robertson and Hugo Zaragoza. 2009. The prob- abilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333-389.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du Andrew M. Dai, and Quoe V Le. 2022. Finetuned language mod- els are zero-shot learners. In International Confer- ence on Learning Representations.
Devendra Sachan, Mike Lewis. Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. Improving passage retrieval with zero-shot question generation. In Proceedings of the 2022 Conference Empirical Methods in Nat- ural Language Processing, pages 3781-3797, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen. Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing: System Demonstrations, pages 38 38-45, Online. Association for Computational Linguistics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu. Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao,
Wenhao Yu, Dan Iter, Shuobang Wang. Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. International Confer- ence for Learning Representation (ICLR).

---

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.
Opt: Open pre-trained transformer language models. arXiv preprint arXiv:205.001068.

---

A
Related Work
the answers extracted from Wikipedia documents. In our experiment, development and test set of NQ contain 6,515 and 3,610 queries, respec- tively.
We describe the related work on unanswerable in- struction and query generation score in our pro- posed method, Distraction-aware Answer Selection (DAS).
TriviaQA (TQA) (Joshi et al.. 2017) was for reading comprehension dataset consisting of question-answer--evideee triplets. The queries fetched from the quiz websites and the corre- sponding evidence documents are collected from the Wikipedia documents via the Bing search en- gine. In our experiment, a development set and a test set of TQA contain 6,760 and 1 1,,13 queries, respectively.
A.1 Unanswerable Instruction
The unanswerable queries were introduced to en- sure the effective discernment of query-document relevance (Rajpurkar et al., 201 8). This approach incorporated in the pre pre-training of LLMs when models cannot find the answer within the provided document (Wei et al., 2022; Sanh et al., 2022; Iyer et al., 2022). We revisit these approaches in a zero- shot setting to confirm the fcasibility of the unan- swerable instruction for filtering out irrelevant doc- uments in the retrieved set.
WebQuestions (WebQ) (Berant et al., 2013) collected the queries from Google Suggest API and its answer from the entities in Freebase. The evidence documents were defined as the highest- ranked documents from BM25 having the answer (Lee et al., 2019). We use a development set WebQ consisting of 361 questions.
A.2 Document Ranking with Query Generation Score
The query generation score is a widely used mea- sure of query-document relevance when ranking the documents (Nogueira dos Santos et al., 2020; Ju et al., 1). Recently, LLMs serve as zero-shot re-rankers with outstanding performance gain by computing the measure (Sachan al., 2022; Cho al., 2023). To this end, highlight the capacity of LLMs to ascertain the relevance between the query-document pair when exploiting them as a zero-shot reader.
SQuAD (SQD) (Rajpurkar et al., 2016) was based manually annotated queries from Wikipedia documents. While SQuAD wasn't de- signed for ODQA tasks, it was widely used for evaluating reader performance. development set and test of SQuAD contain 8,886 and 10,570 queries, respectively.
B Experimental Setup
B.1 Dataset
B.2 Instruction & Template
In our main experiments, utilize development set of four representative ODQA datasets for em- ploying annotated evidence documents to analyze the impact of query-document relevance. We apply a filtering process to exclude some data that do not contain evidence documents. For fair comparison against the supervised readers, DPR (Karpukhin et al., 2020) and FiD (Izacard and Grave, 2021)* we use the test set of each dataset which has already been preprocessed in Sachan al. (2022) 3.
As LLMs are sensitive to instruction and templates when adopting the downstream tasks without pa- rameter updates, we carefully select via iterative validation. The reading comprehension instruc- tion is "Read the following context and answer the question" and the unanswerable instruction is "Read the following context and answer the ques- tion. If you don't 't know the answer, return unan- swerable". When transmitting a query, a docu- ment, and an instruction to LLMs, we use the input template following the setting from Chung et al. (2022) and Iyer et al. (2022). The input templates are "(IJAntnContex: [D))nQuestion: (Q)" for FLAN-T5 and "[I)uunCCnttex: (D)AnQuestion: (Q) Ansseer: for OPT-IML-MAX where I,D and Q denotes an instruction, document, and a question, respectively.
Natural Question (NQ) (Kwiatkowski et al., 2019) was specifically crafted for ODQA tasks. It comprises queries from Google search engines and
We evaluate FiD publicly opened repository.
model checkpoints from their
'https: //gi www/ hetm/DDDeSgghahaaaann unsupervised-a mesupervised-passge--eee passage-reranking

---

Retriene
Reader
TQA
Top-10
Webk)
SqU
TQQ
Top-20
WebQ
5QD
Top-50
WebQ
SQD
Tup-100
FLAN-TS-XL
234
53.8
15.1
30.8
22.4
52.7
16.2
19.4
19.5
50.1
18.4
17.1
17.9
46.1
15.8
153
W/ DAS
28.5
61.2
21.6
35.8
51.5
64.5
20.1
39.4
32.8
67.1
24.1
43.7
33,8
68.9
26.0
46.7
BM25
OPT- MAX
208
54.1
21.0
23.7
20.2
53.1
23.4
22.9
17.6
50.2
20.1
20.4
16.3
66.5
18.7
18.3
W/ DAS
268
54.5
22.7
29.7
28.7
$7.0
24.1
32.4
29.6
59.1
27.3
55.7
20,8
59.9
24.1
37.7
FI-AN-TS-XL
258
$1.0
19.8
13.4
20.4
47.4
20.5
12.9
18.7
42.9
19.4
11.2
15.9
36.2
16.6
10.3
w/ DAS
37.2
62.2
26,0
22.6
57.8
64.5
27.0
26.7
37.9
67.0
25.5
51.1
38.0
68.2
25.2
34.1
DPR
OPA-IMI-MAX
26.1
52.0
23.7
15.5
23.5
50.2
21.6
16.0
19.1
46.7
23.0
15.5
16.6
43.7
19.4
14.5
W/ IMS
335
54.8
26.1
19.5
33.7
36.5
27.0
22.0
33.4
583
21.0
25.9
33,0
29.1
25.5
28.5
Table 4: Exact match accuracy of the final answer among the generated answers from top-k retrieved documents for the open-domain question answering benchmarks.
Document
Document
selections contribute positively to enhancing the performance of LLM. However, in the casc of OPT- IML-MAX, the impact of document selection is found to be insignificant. This observation suggests that OPT-IML-MAX, despite its ability distin- guish irrelevant documents based on instructions, falls short compared FLAN- T5 in effectively addressing the hallucination.
Cur
NQ-Uev
FLAN-TS-XL
1.58
1.09
0.01
91.29
WAD
OPT-IML-MAX
151
0.01
7.36
w/DAS
TQA-Dev
FIAN-TS-XI,
361
1.54
0.01
13.08
78.86
3.00
W/DAS
323
MAX
3.79
1.32
0.2
13.16
75.62
6.29
W/DAS
79.17
D Analysis
Table 5: Average number of answers in the answer candidate set including unanswerable response set U. Cor and Inc denote correct and incorrect answer, respectively. NR means no response to the documents.
D.1 Aanlaysis of Unanswerables
As shown in Table 5, we conduct an analysis of the model's responses to documents, including those that are excluded from the answer candidate set S during the document selection process. While our method successfully reduced the number of responses from irrelevant documents, observed slight decrease in relevant documents. However, the primary focus our methodology on increas- ing the portion of correct answers by minimizing the number of incorrect answers originating from irrelevant documents. This aspect is key to our ap- proach and contributes to the overall improvement of reader performance.
B.3 Environment
We conduct all experiments on A100 80GB GPUs. We use BEIR (Thakur et al., 2021) framework' for the retriever, BM25 and DPR. We employ FLAN- T5 and OPT-IML-MAX with 3B and 1.3B pa- rameters publicly open on the Huggingface model hub (Wolf et al., 2020).
C Detailed Results
We provide more comprehensive results in terms of both top-10 and top-50 documents, as illustrated in Table and Figure 6. In the absence of our proposed methodology, there is a noticeable decline in performance as the number of documents read increases. However, when employing DAS, we observe a reduction in the impact of hard-negative documents within the document set, resulting in enhanced reader capability. DAS effectively mitigates the adverse effects of such documents and maximizes the overall performance of reader. In an ablation study, Figure 6 showcases the in- fluence of document selection (D.S.) and answer selection (A.S.) within our proposed method. Both
D.2 Analysis of Overconfident Score
We conducted a verification to determine whether the answer score was indeed overconfident. As de- picted in Figure 5, when DAS not utilized, the incorrect answer exhibits a remarkably high gener- ation probability, making indistinguishable from the correct answer. However, upon implementing DAS, the scores are normalized, resulting in dis- cernible distribution disparity between correct and incorrect answers.
D.3 Case Study
We present two curated examples in Table 6 to illus- trate the effectiveness of our proposed approach mitigating hallucination compared to naïve LLMs.
'http://beir.ai/ 'https://huggingte http://acccccccce//eeeeeeee co/models

---

HAS.
Lenaet Apever Inomee Anrvat
Comun Arenvr
Incemect Answer
90
43-
39-
Skarz
Figure 5: The analysis of answer The left plot is for FLAN-T5-XL. without HAS and the Right plot is with HAS. Both experiments are on NQ development set with evidence documents retrieved by DPR.
In these examples, the naïve LLMs erroneously provide the answer "Straits of Mackinac" in related contexts to "Lake Michigan-Huron" when given the query about "The Great Lakes". How- ever, by employing our method, the correct answers are extracted from the relevant documents. This highlights the ability of our approach to alleviate hallucination and facilitate the accurate selection of appropriate answers based on contextual infor- mation.
Additionally, we showcase two error cases in Table 6. these cases, reader generates the cor- rect answer based on the relevant document, but our approach produces plausible alternative answers. For instance, in response to the question "What is the deepest depth in the oceans?", the reader correctly identifies "Challenger Deep" based other relevant document not included in annotated evidence set. While this answer technically in- correct according to EM evaluation, it is difficult to perceive it as entirely incorrect when assessed qualitatively.

---

50
BM25
DPR
BM22
DPR
9-0-0
0
6-9-9
10
1020
100
FLAN-TS-XL.
OPT-MME-MAX-1.D
1QA
70
30
0
1020
50
100
1020
50
100
1020
100
1020
50
100
ILAN-15-XL
OPT-IML MAX 1.30
Well)
BM22
Tope
B5125
DPR
0
1020
50
100
10.00
50
100
1020
100
1020
30
100
FLAN-TS-XL
SQUAD
30
BM25
BM25
30
8-0-0-6---0
0-0-0
FLAN-TS-XL.
MIH-MM-MAX--..
Figure 6: accuracy depending on the number of the retrieved documents.

---

Case
Case
Query
Where do the great lakes meet the oceans?
Who was the creator of Victoria's 's Secret?
Gold Answer
the Saint Lawrence River
Roy Raymond
w/o DAS
Final Document
Lake Michigan-Huroo because they are one hydrological body of water connected by Straits of Mackinac, The straits are wide and deep; the water levels (...)
Traci Paige Johnson an American ani- mator, television producer. and voice ac- tress, most known for creating the Nick Jr. preschool television series, (...)
Final Answer
Straits of Mackinac
Traci Paige Johnson
w/ DAS
Final Document
The Great Lakes series of intercon- nected freshwater lakes located primarily in the upper mid-east region of North America. the Canada-Uninne Uniudd States border, which connect to the Atlantic Ocean through the Saint Lawrence River.
Victoria's 's Secret is American designer, manufacturer, and marketer of women's 's lin- gerie, womenswear, and beauty products. (...) Victoria's Secret founded Roy Raymond, and his wife Gaye Raymond, in San Francisco, California, (...)
Final Answer
the Saint Lawrence River
Roy Raymond
Error Case 1
ErrorCase
Query
Who plays mrs. potato head in toy story?
What is the deepest depth the occans?
Gold Answer
Estelle Hamis
Mariana Trench
w/o DAS
Final Document
(..)) After Mr. Potato Head saves three Pizza Planet Aliens (Jeff Pidgeon) from falling out of Pizza Planet truck, his wife, Mrs. Potato Head (Estelle Harris) adopts them, making her husband upset. (...)
In the Challenger Deep, he and Lt. Don Walsh of the United States Navy first people to explore the deepest part of the world's ocean, and the deepest location the surface Earth's 's crust. the Mariana Trench, located the western North Pacific Ocean. (...)
the
Final Answer
Estelle Harris
Mariana Trench
w/ DAS
Final Document
Pop singer Melanie Martinez released a
The Challenger Decp, located just outside the Trench Unit, is the deepest point in the Earth's 's oceans, deeper than the height of Mount Everest above sea level. (...)
song called "Mrs. Potato Head" on her de- but album "Cry Baby". Mr. Potato Head is also in the Disney/Pixar "Toy Story films" voiced by Don Rickles. (..)
Final Answer
Don Rickles
Challenger Deep
Table Examples of hallucination alleviation and error cases. FLAN-TS-XL is exploited reader on the Natural Question dataset.


Reviews:
[{"review_id": "btzoSnQkbL", "reviewer": "Reviewer_CcX8", "summary": "", "questions": "See \"reasons to reject\"", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful and helpful comments to increase the quality of our manuscript. The following discussion refers to our submitted manuscript as our paper. Additional explanations and materials are for enhanced clarity, which will be reflected in the revised version. The positions of the graphs in Figure 5 have been swapped. We apologize for your confusion but it can be easily fixed.\n___\n### **1. Hallucination Induced by Easily Distracted and Overconfident LM (Q4)**\n\nFirst, we want to point out a misunderstanding of our research motivation, the hallucination of LLM, as mentioned in lines 51-65. \n\nHallucination generally refers to the generation of plausible yet untrue text. Its reasons are diverse on specific tasks but usually inherent in the mechanism of LLM [lines 119-130]. Previous work on open-domain question answering (ODQA) points out that LLMs are easily distracted to produce incorrect answers by the irrelevant context from retrieved sets [1,2,3] and their overconfidence (i.e., high log probability of sequences) from the decoding strategy [4,5,6]. Lines 51-59 are supported by studies about the impact of irrelevant context leading to accuracy degradation. Figure 5 indicates that the overconfidence of LLMs on output makes it hard to identify the correct ones between generated answers. \n\nIn addition, we empirically confirm that both causes from input and output are related through our analysis of Tables 3, 5, and Figure 5 of our paper. Table 3 illustrates a high frequency of incorrect answers among the answer candidates, and Table 5 specifies that these errors stem from irrelevant documents. Then, Figure 5 describes overconfidence in answer scores on a default setting, which means that LLMs have trouble identifying correct results. Our proposed method, HAS, successfully alleviates such problems depicted in all analyses.\n\n|                    | NQ | TQA | SQuAD |\n|:--------------------------------------:|:-----:|:-----:|:-----:|\n|  FLAN-T5-XL w/o Context (Lower Bound) | 12.09 | 25.30 | 5.86 |\n|  FLAN-T5-XL w/ Top-100 Docs    | 15.90 | 39.17 | 10.30 |\n|  FLAN-T5-XL w/ HAS        | 37.96 | 68.22 | 34.12 |\n| FLAN-T5-XL w/ Gold Document (Upper Bound) | 54.53 | 75.27 | 73.84 |\n\nFor clear understanding, the table above shows that naïvely augmenting the retrieved document enhances the performance marginally, leaving much room for improvement. In terms of utilizing LLMs, this underscores the impacts of hallucination rather than augmenting with external knowledge. Ours is much more aligned with the upper bound. \n\nThis highlights that *ours effectively filtered out the negative impact of such a context, resulting in alleviating hallucination in LLMs*. \n\n___\n### **2. Necessity of Incorporating LLM into the Traditional Retrieve-then-Read Pipeline (Q1 & Q3)**\n\nWe categorized this section into two research questions mentioned in the reasons to reject. \n\n**1) LLMs themselves have impressive abilities in answering questions requesting background knowledge.**\n\nAlthough the open-domain data is included in the training datasets, LLMs are still challenged under the closed-book setting for knowledge-intensive tasks. Our work is aligned with the concurrent studies that retrieval augmentation provides a promising path toward improving performance [7,8,9]. Furthermore, the langchain already incorporates the retriever modules to locate evidence information, highlighting the significance of retriever augmentation for LLMs [10]. Our proposed method, HAS, is suggested to utilize the retrieve-then-read framework in handling the hallucination mentioned above and eliciting the full performance of LLMs. \n\n**2) LLMs with the proposed methods do not outperform small supervised model solutions, considering their increased inference cost.**\n\nExploiting LLMs brought us to the expectation of a zero-shot approach and generality of LLMs [lines 37-50, 117-118]. Regarding perspective, comparing a supervised model directly with LLMs is not straightforward, given the substantial training costs and the need for ongoing updates to maintain the performance on out-domain datasets.\n\nWe conducted additional experiments on out-domain datasets with the instruction-tuned model, FLAN-T5-base, and large, which have the same size as FiD readers. These experiments are conducted on the supplementary dataset, RealTimeQA [11], based on current world knowledge.\n\n\n|     | Param | Train Set | SQuAD | RealTimeQA | Avg. |\n|:-----------------:|:-----:|:---------:|:-----:|:----------:|:----:|\n|  FiD - Base  | 250M |  NQ  | 34.1 |  29.8  | 32.0 |\n|         |   |  TQA  | 27.5 |  33.2  | 30.4 |\n|  FLAN-T5-Base |   |  X  | 10.5 |  20.9  | 15.7 |\n|   w/ HAS   |   |     | 36.4 |  31.6  | 34.0 |\n|  FiD - Large  | 780M |  NQ  | 36.2 |  34.0  | 35.1 |\n|         |   |  TQA  | 31.5 |  34.4  | 33.0 |\n| FLAN-T5-Large |   |  X  | 10.8 |  20.1  | 15.5 |\n|   w/ HAS   |   |     | 41.5 |  34.7  | 38.1 |\n| FLAN-T5-XL w/ HAS | 3B |  X  | 43.5 |  35.8  | 39.7 |\n\n\nThe results follow a tendency shown in Table 2 of the paper with the FLAN-T5-XL. The small supervised models perform well over in-domain datasets (i.e., train datasets), but their performance does not remain consistent over out-domain datasets such as SQuAD and RealTimeQA [lines 249-254]. Using LLMs alone does not enhance performance. Notably, even with the smaller scale of the instruction-tuned model, our proposed approach is found to harness its capacity for generalization in ODQA.\n\n___\n### **3. Other Possible Prompt Designs such as Chain-of-Thoughts (CoT) (Q1)**\n\nWe wish to clarify that CoT and our proposed method, HAS, take a different approach. While CoT has gained attention for simply mimicking the reasoning process of a human with performance improvement, it seems inappropriate to solve the problems we described [3,12,13]. In particular, [3] demonstrates that CoT cannot alleviate the negative impact of the irrelevant context even in arithmetic reasoning. The following table proves that the CoT design cannot solve the problem induced by hallucination when top-100 documents from DPR are transmitted to FLAN-T5-XL.\n\n\n|   | NQ-Dev | TQA-Dev |\n|:------------:|:------:|:-------:|\n| FLAN-T5-XL | 15.90 | 39.17 |\n|  w/ CoT  | 13.00 | 34.52 |\n|  w/ HAS  | 37.96 | 68.22 |\n| w/ CoT & HAS | 37.04 | 65.21 |\n\nCompared to this, our proposed method, HAS, is a *negation-based prompting* that lets the model not answer when models are unsure. To the best of our knowledge, we have empirically validated the effectiveness of such a prompt design on negation. In contrast, the current prompt design is targeted at generating self-supervised evidence. \n\n### **4. Validation on Much Larger Models (Q2)**\n\nWe validate multiple models under 10 billion parameters available at our academic resources. Also, please note that we showed the result of FLAN-T5-XXL having 11 billion parameters in Figure 3. The table below gives detailed results showing a similar tendency on even over 10B models for clear understanding. \n\n|      | 10 Docs | 50 Docs | 100 Docs |\n|:-----------:|:-------:|:-------:|:--------:|\n| FLAN-T5-XXL | 29.7 | 23.0 | 19.9 |\n|  w/ HAS | 36.9 | 37.8 | 37.7 |\n\nThe problem of hallucination, as previously defined, is apparent in all cases involving models over 100B parameters [3, 8]. However, it is essential to emphasize that our focus does not extend to black-box models such as GPT-3, primarily due to our constrained access to token log probabilities. This limitation was articulated in the Limitation section in lines 303~ 307, aligned with the viewpoint expressed in the previous work [3,8].\n\n___\n#### **References**\n\n[1] LI, Daliang, et al. Large language models with controllable working memory. In: Findings of the Association for Computational Linguistics. 2023. p. 1774-1793.\n\n[2] SU, Dan, et al. Read before generate! faithful long form question answering with machine reading. In: Findings of the Association for Computational Linguistics. 2023. p. 744-756.\n\n[3] SHI, Freda, et al. Large language models can be easily distracted by irrelevant context. In: International Conference on Machine Learning. PMLR, 2023. p. 31210-31227.\n\n[4] LIN, Stephanie; HILTON, Jacob; EVANS, Owain. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022.\n\n[5] MIELKE, Sabrina J., et al. Reducing conversational agents’ overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 2022, 10: 857-872.\n\n[6] XIONG, Miao, et al. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. arXiv preprint arXiv:2306.13063, 2023.\n\n[7] MALLEN, Alex, et al. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. p. 9802-9822.\n\n[8] SI, Chenglei, et al. Prompting gpt-3 to be reliable. In: International Conference on Learning Representation. 2023.\n\n[9] KANDPAL, Nikhil, et al. Large language models struggle to learn long-tail knowledge. In: International Conference on Machine Learning. PMLR, 2023. p. 15696-15707.\n\n[10] LangChain AI. 2023. Available online: https://python.langchain.com/docs/modules/data_connection/\n\n[11] KASAI, Jungo, et al. RealTime QA: What's the Answer Right Now?. arXiv preprint arXiv:2207.13332, 2022.\n\n[12] LI, Junlong; ZHANG, Zhuosheng; ZHAO, Hai. Self-prompting large language models for open-domain qa. arXiv preprint arXiv:2212.08635, 2022. \n\n[13] TURPIN, Miles, et al. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. arXiv preprint arXiv:2305.04388, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper proposes using LLMs as readers in retrieve-read with a simple and effective HAS to handle irrelevant information and hallucination caused by inaccurate retrievers. Despite no significant gains over previous methods, the zero-shot setting greatly reduces training costs and annotation needs.\n2.\tExperiments validate the efficacy of HAS, showing clear improvements over not using HAS across datasets and retrievers.", "reasons_to_reject": "1.\tHAS adds document and answer selection modules using LLMs to filter retriever results. The increased inference costs were not considered. With a large number of inference, LLM costs may exceed training a small model. HAS does not outperform small supervised model solutions, sometimes underperforming. Given LLM abilities to select relevant documents, ideas like COT prompt design could be explored to have LLMs filter then answer to achieve HAS effects. In this case, what is the advantage of employing LLMs in this task?\n2.\tMore popular LLAMA and ChatGPT models were not tested. The smaller FLAN-T5 and OPT-IML have less zero-shot capacity than current best open LLM models, so HAS may perform better on stronger models.\n3.\tIt may be unnecessary to follow the traditional retrieve-read framework for LLMs in open-domain QA. LLMs could ingest open-domain data during pretraining, supplement with lang-chain, and benchmark current LLM open-domain performance.\n4.\tI think the title is somehow misleading. Hallucination is indeed an issue for LLMs. However, this paper is not to alleviate hallucination in LLMs, it tries to filter out the irrelevant documentations during retrieval procedure for open-domain QA tasks, which is not to improve the LLMs themselves. Also this brings limitation to the study.", "questions_to_authors": "See \"reasons to reject\"", "paper_topic_and_main_contributions": "This paper proposes to use LLMs as the reader module in the traditional retrieve-read framework for open-domain question answering, under a zero-shot setting. Compared to previous small-sized supervised readers, the zero-shot approach eliminates the need for downstream task training and annotated query-document datasets. However, using LLMs also introduces hallucination issues, exacerbated when retriever results contain irrelevant documents. To address this, the authors propose Hallucination-aware Answer Selection (HAS) to alleviate the impact of irrelevant documents and overconfident LLMs, which is to prompt the LLMs to filter out the irrelevant documents via unanswerable instruction. Experiments show that HAS achieves average EM improvements of 67% and up to 231%. However, performance does not significantly exceed previous fine-tuning methods like DPR and FID.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "61U72rvfMF", "reviewer": "Reviewer_ubV7", "summary": "", "questions": "- lines 207-210 mention that Sanh et al. 2022 also have an unanswerable instruction in their prompt. If so, what part of the unanswerability forms the novelty of this work? If not, could the author(s) kindly elaborate on the differences between the prior work and the proposal in this draft?\n- Why does the absolute number of correct answers decrease after applying HAS to the FLAN-T5 (or OPTI-IML-MAX)? Does this imply the model is susceptible to removing some correct documents? Is there a pattern for when the correct documents are removed?\n\n**Reference**:\n\nSanh et al. 2022: Multi-task prompted training enables zero-shot task generalization. ICLR 2022", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in reviewing our paper and providing valuable comments. The subsequent contents pertain to the response to your comments and explanation of our work, including additional results. The positions of the graphs in Figure 5 have been swapped. We apologize for your confusion and it can be easily revised. The responses will be reflected in the revision of our work. We reorganized your comments into three sections for clarity.\n\n___\n### **1. Clear Description of Our Proposed Method, HAS (Reason to Reject)**\n\nWe explained the two-step mechanism of HAS in Section 3.3 with a concise overview [lines 166-174] and in Appendices B.3 and B.4. We will revise the description of the method for a clearer understanding, including the contents of the Appendix.\n\n HAS consists of two stages: document and answer selection. The document selection is for not responding to the irrelevant documents in the retrieved sets. By adding the unanswerable instruction, *the negation-based prompting*, to the original instruction [line 205-209], documents and answers from the answer candidate set $U$ are filtered out when their outputs are *“unanswerable”* or *“Answer not in context”*. Subsequently, the answer selection alleviates the overconfidence of generation probability by reflecting the relevance between the question and the document [lines 181-186]. The answer, having the highest value from Equation (2) and the query generation score described in Appendix B.4, is chosen as the final answer to the question. \n\nFor example, let the relevant context $d$ and the irrelevant documents $d_{n1}$ and $d_{n2}$ in the retrieved document set $D$ for the given query $q$. For the document selection stage, if LLM generates $a_{n1}$ as *“unanswerable”* from the $d_{n1}$, the pair $(a_{n1},d_{n1})$ is excluded to the answer candidate set $A$. Then, the answer candidate set $A$ is $\\lbrace(a_{n2},d_{n2}), (a,d)\\rbrace$. For the answer selection stage, we calculate the answer score for each pair based on Equation (2). Considering the relevance of context, the score of $(a,d)$ is higher than the score of $(a_{n2}, d_{n2})$. \n\n___\n### **2. The Novelty of HAS & Differences from Previous Approaches (Reason to Reject & Q1)**\n\nEach component of the HAS framework brings novel insights to questionable and underexplored domains, as described in Appendix A. Our novelty comes from bridging this gap by empirically demonstrating both negation-based prompting & score regulation and effective mitigation of hallucination induced by both input and output.\n\n\nAlthough the integration of unanswerable instructions within the pre-training of LLMs has been acknowledged [1], it remains one of the instructions describing diverse tasks aimed at enhancing generality, often underestimating its potential for negation. [2,3] also pointed out the negative impact of irrelevant or hallucinated contexts when exploiting LLMs for diverse tasks, underscoring the necessity of negation.  To the best of our knowledge, the score regulation based on query generation is first utilized for the alleviation of overconfidence in LLMs, while its utilization is usually on document ranking.\n\n\n___\n### **3. Distribution Shift of Answer Candidate Set $U$ (Q2)**\n\nHAS excludes irrelevant documents as much as possible and utilizes only answers from relevant documents. Negating irrelevant contexts involves a trade-off with the number of accurate responses, but we confirmed through Table 1 and Figure 1 that this gives rise to positive results on ODQA. \n\nFor further analysis, Table 5 within the Appendix highlights that most of the incorrect answers are from irrelevant contexts, leading to a significant proportion of answer candidate set $U$. HAS has efficiently diminished the count of incorrect answers originating from irrelevant contexts while preserving the number of accurate answers from relevant contexts.\n\n\n\n___\n#### **References**\n\n[1] Sanh et al. 2022: Multi-task prompted training enables zero-shot task generalization. ICLR 2022\n\n[2] SHI, Freda, et al. Large language models can be easily distracted by irrelevant context. In: International Conference on Machine Learning. PMLR, 2023. p. 31210-31227.\n\n[3] SI, Chenglei, et al. Prompting gpt-3 to be reliable. ICLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Performance increases as the number of retrieved documents increases in the model, implying that the model is indeed more robust to adding more irrelevant documents.\n- Experiments with multiple models and across multiple datasets highlight the generality of the proposed method.", "reasons_to_reject": "- **Novelty and Clarity**: Inferring how the abstention section works is hard when the description is restricted to just four lines (L175-179). It is unclear what answers form a part of the unanswerable set. Is it just a single answer \"unanswerable\"? Or is there some other mechanism used? Further, by the authors' own admission, the technique of abstention and score regulation has also been explored in prior works in the question-answering setting. This calls to question the novelty of the proposed method beyond the fact that it has been applied to zero-shot models.\n", "questions_to_authors": "- lines 207-210 mention that Sanh et al. 2022 also have an unanswerable instruction in their prompt. If so, what part of the unanswerability forms the novelty of this work? If not, could the author(s) kindly elaborate on the differences between the prior work and the proposal in this draft?\n- Why does the absolute number of correct answers decrease after applying HAS to the FLAN-T5 (or OPTI-IML-MAX)? Does this imply the model is susceptible to removing some correct documents? Is there a pattern for when the correct documents are removed?\n\n**Reference**:\n\nSanh et al. 2022: Multi-task prompted training enables zero-shot task generalization. ICLR 2022", "paper_topic_and_main_contributions": "In this work, the author(s) introduce Hallucination-aware Answer Selection (HAS), as part of a two-stage pipeline to mitigate the effects of LLMs hallucinating incorrect answers in the presence of irrelevant documents. To mitigate the challenge of irrelevant documents provided by the retriever model, the author(s) propose the use of an \"unanswerable\" instruction that prompts the language model to abstain from answering if the context does not provide enough information. Such contexts are subsequently removed from final answer selection. In the second stage, the author(s) propose to tackle the challenge of overconfident answer predictions from LLMs by regulating the score with the probability of generating the query given a document. Intuitively, if the document is irrelevant, the probability of the query would be low and hence the model down weights the score assigned to such an answer. Applying the two-stage pipeline on two instruction-tuned models, FLAN-T5 and OPT-IML-MAX, the proposed HAS method shows significant improvements over the vanilla models. Further analysis also indicates that the model is more robust to the addition of irrelevant documents, and works at any scale across multiple datasets.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 221: improvement of 231% → improvement of 231% on SQUAD.\n- line 262: kindly mention that this is the percentage improvement in the number of correct answers for FLAN-T5 and OPT-IML-MAX respectively."}}, {"review_id": "l3zmPbhvgF", "reviewer": "Reviewer_Y5qm", "summary": "", "questions": "1. What’s the difference between hallucination and incorrect answers in your work? Since you just use exact match (EM) accuracy as metric.\n    \n2. In Table 2, why the EM results for FLAN-T5-XL w/ HAS is different from the EM results in Table 1?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful and valuable feedback on our work in the best regard. The positions of the graphs in Figure 5 have been swapped. We apologize for your confusion and it can be easily revised. In this response, we have restructured your comments into three distinct sections.\n\n___\n### **1. Feasibility of HAS on Much Larger Models (Reason to Reject)**\n\nWe validated multiple models with fewer than 10 billion parameters within the limit of available academic resources. Our findings include the results obtained from FLAN-T5-XXL, which has 11 billion parameters, presented in Figure 3. The table below elucidates these outcomes, highlighting a consistent trend with Figure 2 across models exceeding the 10 billion parameters.\n\n\n|  DPR  | 10 Docs | 50 Docs | 100 Docs |\n|:-----------:|:-------:|:-------:|:--------:|\n| FLAN-T5-XXL | 29.7 | 23.0 | 19.9 |\n|  w/ HAS | 36.9 | 37.8 | 37.7 |\n\nThe issue of hallucination defined previously is evident in all instances of utilizing LLMs [1,2]. Our scope does not fully encompass black-box models like GPT-3, which are constrained in obtaining token log probabilities. We mentioned this in the Limitation section [lines 303~307] as we acknowledge the same perspective of the review. We consider a universal approach encompassing diverse model types as one of our future research directions.\n\n___\n### **2. Relation between Hallucination and Exact Match (EM) (Q1)**\n\nOur research is motivated by hallucination [lines 51-65] and its impact is on the accuracy degradation of ODQA when augmenting the retrieved documents to LLMs. \n\nWe find that LLMs augmented with retrieved sets confront hallucination on ODQA as caused by irrelevant documents on the input side and overconfident scores on the output side [lines 148-164]. These two facets of hallucination culminate in the eventual selection of an erroneous answer for the given question, thereby substantiating the observed decline in accuracy for ODQA tasks, as shown in Table 1 and Figure 2.\n\nOur analysis of these phenomena, as illustrated in Tables 3 and 5 and Figure 5, demonstrates the relevant mechanics. The findings in Tables 3 and 5 highlight that the existence of irrelevant documents within retrieved sets frequently prompts LLMs to generate inaccurate outputs, consequently resulting in a significant proportion of incorrect answers within the set of candidates. Meanwhile, Figure 5 visually represents the phenomenon of overconfidence in incorrect answers, which exhibits a distribution comparable to that of accurate answers. The coexistence of these two phenomena makes selecting the correct answer as the final output by LLMs challenging, hindering the advantage of retriever augmentation.\n\n___\n### **3. About Table 2 (Q2)**\n\nThe difference between the exact match (EM) scores comes from the data division. The results in Tables 1 and 2 were based on the validation and test split, respectively. Because the validation split annotated the relevant documents for each query, the subsequent analyses are possible based on the results in Table 1, whereas Table 2 used the test split for a fair comparison with other supervised models. In the revision, we will provide a clear explanation of the distinction between the two results. \n___\n#### **References**\n\n[1] SHI, Freda, et al. Large language models can be easily distracted by irrelevant context. In: International Conference on Machine Learning. PMLR, 2023. p. 31210-31227.\n\n[2] SI, Chenglei, et al. Prompting gpt-3 to be reliable. ICLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They empirically verify the efficacy of this simple approach in mitigating hallucination. The method achieves an increase in EM of 64%.\n  \n2. Conduct sound experiments to analyze including the impact of model size, and Answer Candidate Set.\n", "reasons_to_reject": "1. Just mainly conduct experiments on FLAN-T5-XL (3B) and OPT-IML-MAX (1.3B). It’s better to explore larger models since your topic is about LLM.", "questions_to_authors": "1. What’s the difference between hallucination and incorrect answers in your work? Since you just use exact match (EM) accuracy as metric.\n    \n2. In Table 2, why the EM results for FLAN-T5-XL w/ HAS is different from the EM results in Table 1?", "paper_topic_and_main_contributions": "This paper captures the hallucination aspect arising from the presence of irrelevant documents and the generated overconfident answer scores when exploiting an LLM as a reader.\n\nTo reduce this problem, they define a two-step answering pipeline consisting of answer candidate generation and final answer selection.\n\n1. they provide models with an unanswerable instruction, allowing them to refuse response to irrelevant documents.\n    \"If you don’t know the answer, return unanswerable\"\n    \n2. adjust the answer scores by multiplying the query-document relevance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a method to improve results in open-domain QA. Typically, open-domain QA is done by 1) retrieving relevant docs, 2) running a "reader" that reads the question + the retrieved documents and outputs the answer. The proposed method focuses on filtering out noisy retrieved documents. It allows the reader to output "no answer" if the retrieved document is not useful. 

The model is implemented using FLAN-T5-XL-3B and using OPT-IML-MAX-1.3B. Results show performance gains on NQ, TQA, WebQ, SQuAD. 

Reviewers found the result to be generally sound. 
Some reviewers complained about the choice of the base models, which I don't think is a fair criticism; 3B is a reasonable size + not every researcher can run the largest available model.
Others commented on the novelty of the work, and yes, the idea of giving the model the ability to say "I don't know" is not new. The details of how it is implemented are a little different from prior work, though.


Nit: I agree with R1 that using the word hallucination in this context is inaccurate. Hallucination usually means generating output not supported by the input. Here, the output is supported by one of the retrieved documents, it just happened that it was the wrong document.
Decision: Accept-Findings