PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering
Authors: Wookje Han
Jinsol Park
Kyungjae Lee
Venue: EMNLP 2023 Main (2023)
Keywords: Long-Form QA
Large Language Models
Presuppositions
TL;DR: This paper proposes a unified QA system that handles long-form information-seeking questions by extracting presuppositions and exploiting them as working memory.

Abstract: Information-seeking questions in long-form question answering (LFQA) often prove misleading due to ambiguity or false presupposition in the question. While many existing approaches handle misleading questions, they are tailored to limited questions, which are insufficient in a real-world setting with unpredictable input characteristics. In this work, we propose PreWoMe, a unified approach capable of handling any type of information-seeking question. The key idea of PreWoMe involves extracting presuppositions in the question and exploiting them as working memory to generate feedback and action about the question. Our experiment shows that PreWoMe is effective not only in tackling misleading questions but also in handling normal ones, thereby demonstrating the effectiveness of leveraging presuppositions, feedback, and action for real-world QA settings.

Pre PoMe: for Long Form Question Answering
Exploiting Presuppositions as Working Memory
Wookje Han
Jinsol Park
Kyungjae Lee" LG AI Research
Columbia University wookje. han@columbia edu
Carnegie Mellon University jinsolpêcs. cmu.edu
kyungjae lee@lgresearch.... ai
Abstract
Misleading Questions
Question:
Who the war between Ethiopis and Italy?
Information- seeking questions in long-form question answering (LFQA) often prove mis- leading due to ambiguity or false presuppo- sition the question. While many existing approaches handle misleading questions, they are tailored to limited questions, which are in- sufficient in a real-world setting with umpre- dictable input characteristics. In this work, we propose Pr.MeMe, a unified approach capable of handling any type of cntemattion-seeiinn question. The key idea of PreM involves extracting presuppositions in the question and exploiting them working memory to gener- ate feedback and action about the question. Our experiment shows that PreWoMe effective not only in tackling misleading questions but also in handling normal ones, thereby demon- strating the effectiveness of leveraging presup- positions, feedback, and action for real-world QA settings.
Desirable Answer:
There were two wars between Ehhiopis and Italy The first war (1888-009)
was won by Ethropis, and the second (1935-1837) was won by Italy.
Question:
Who wan the Nobel Peace Prize for DNA structure?
(b)
Desirable Answer:
DNA structure, Instead, Jamess Watson, Francis Crick, Maurice Wilkins the Nobel Prize Physiology Medicine for their discovery of the DNA structure.
Noboddy the Nobel
Two types of misleading questions: (a) hav- ing ambiguity, and (b) false presupposition. Highlights show essential cues to avoid misleading.
Figure I:
won the Nobel peace prize for DNA structure"), which should be corrected before answering the question (Yu et al., 2022; Kim et al., 2022).
Desirable answers to mislcading questions re- quire discerning and resolving misleadings. A pre- vious work by Amplayo et al. (2022) focuses ambiguous questions, by introducing query refine- ment prompts that encourage LMs consider mul- tiple facets of the question. Another work (Kim et al., 2021) tackles questions containing FP, by ex- tracting and verifying presuppositions. While such works assume only one specific type of misleading factor, in real scenario, the users ask various types of questions, which can be mislcading or normal (i.e., non non-misleading). This makes prior single- type-tailored approaches hard to be deployed in real-world QA settings.
1 Introduction
Answering informationn-sekinng long-form ques- tions has recently shown significant progress (Fan et al., 2019; Krishna et al., 2021; Xu et al., 2023). However, users' questions in the real-world may often mislead language models (LMs) output misinformation. For example, users ask questions with false presuppositions (FP) (Kim et al., 2022: Yu et al., 2022) which can induce hallucinations LMs believe the presuppositions are true (Maynez et al., 2020). Also, users often ask questions that do not have a single and clear answer, i.e., an am- biguous question, which is difficult for LMs to identify (Min et al., 2020; Stelmakh et al., 2022). Figure 1 shows examples of two types of mis- leading questions: (a) an ambiguous question that requires considering prior knowledge ("two differ- ent wars between Italy and Ethiopia")) to answer properly, and (b) question with FP ("somebody
In this paper, we emphasize the role of presuppo- sitions in handling any type questions in a unified way. Presuppositions are cru- cial building blocks of questions and play an essen- tial role in understanding the meaning of the ques- tion (Ge, 2011; Duăí and Číhalová, 2015). This suggests that exploiting presuppositions is promis- ing in handling misleading questions, while gen- erally applicable to any type of question. Based on this, we propose new approach, PreWoMe
ietoommttionseekinng
'Throughout this paper, we use the term misleading ques- tions for questions that are ambiguous or have FP.
"Corresponding author

---

(Presupposition as Working Memory), that han- dles any type of information-secking question in unified way without any parameter updates.
clear single
answwer to quastion?
LLM
(8) Presupposition Extraction
The main idea of PreWoMe is to extract pre- suppositions and use them as working memory to generate analysis and directions for answering the question. Our contributions can be summarized as follows:
LLM
(b) Feedback, Action Generation
(F.4)
We analyze the performance of recent large lan- guage models (LLMss, GPT-4 and GPT-3.5, on misleading questions. To the best of our knowl- edge, we are the first to explore the performance of LLMs this large on misleading questions.
(c) Answer Generation
Figure Overall pipeline of PreWoMe. q is the given question, (p1, PP2.....PP. are the suppositions. (F) is feedback and action.
pre-
We propose PreWoMe, for real-world QA settings that 1) capable of handling any type of anformation-seeking ques- tion, and 2) does not entail any parameter up- dates, making it easily adaptable to LLMs.
approach designed
Self-correctngg Approaches While LLMs occa- sionally struggle to generate accurate answers with- out hallucination, recent works suggest that LLMs can enhance accuracy by self- ceffecorr their re- sponses through iterative prompting (Madaan et al., 2023; Shinn et al., 2023; Saunders et al., 2022; Press et al., 2022; Weng et al., 2023). Such ap- proaches are based on the idea that verifying cor- rectness is easier than directly generating an accu- rate answer. Inspired by this idea, hypothesize that 1) LLMs are capable of generating and veri- fying the generated presuppositions implicit in a given question, and 2) guiding LLMs to generate answers after the verification will yield better re- sults than generating answers directly.
We propose using presuppositions as working memory when answering information-seeking questions.
2 Background
Presupposition Presuppositions are conditions that are believed to be true by speakers in a dis- course (Yu et al., 2022). Thus, in order for an utter- ance to be appropriate, the presuppositions should be true. For example, if someone said care for my sister, then it can be assumed that the speaker has sister. If not, the utterance would be inappropriate. In this paper, we suggest using presuppositions to solve an information-sseking QA task.
3 PreWoMe
Chain-of-thoughe Prompting (CoT) Wei et al. (2022) explores how thinking with intermediate steps greatly improves LM's performance. Chain- of of-thought Prompting effectively enhances the rea- soning abilities of LMs, particularly in mathemat- ical and logical reasoning tasks. This technique involves guiding the model through series of in- terconnected thought processes. Inspired by this concept, we propose a novel approach that lever- ages presupposition, feedback, and action, serving as intermediate steps, which boosts the robustness of LMs in handling open-ended questions.
We introduce PreWoMe, a new approach that an- swers any type of long-form information-seeking question in a unified way. PreWoMe follows three intermediate steps, as illustrated in Figure 2.
3.1 Presupposition Extraction
To exploit presuppositions working memory, Pre- WoMe first generates presuppositions contained in the question g. Motivated by the few-shot learning capability LLMs (Brown et al., 2020), we feed g with N (question, presupposition) pairs as few-shot examples within designed prompt (See Appendix A). The LLM generates multiple presuppositions (P1. P2, Pz) in q using this input (Figure 2(a)). In addition, we append manual presupposition PA: "There is a clear and single answer the question", ". as shown in Figure 2, which is a com- mon presupposition of all normal questions. If pA recognized as False in the next step, the question
Working Memory Working memory (Miller et al., 2017) is the immediate information buffer that is accessed while performing conscious tasks (Li al., 2022). We make presuppositions func- tion as working memory by extracting and feeding them into the model along with the question as intermediate steps.

---

Type
(a) Ambignous
(b) False Presapposition
Question
Who
between Ethiopia Italy?
only Indian win Oscar for music?
Presuppositions
1) There was a war between ethiopia and italy.
1) There is categury for music in the Oscars.
2) Some country won the war.
There is only one Indian who won Oscar
A) There is clear and single answer to the question.
A) There is a cleur and single unswer to the question.
Peedback
There were two wars between Italy and Ethiopia, different
question contains false presuppocition there is only one Indian who won the Oscar for music.
winners Therefore, the question contains false presuppusttion
that there clear and single answer
question.
Action
Your answer should include the winner of the war betweon Ethiopia and Itlly each respectively in detail.
Correct the false presuppastion that there is only one Osear win-
ner Indian and respond based on the corrected presuppositio..
Table 1: Examples of misleading questions and their intermediate steps. Boldface in feedback and action are parts that distinguish different types of questions.
is classified as misleading due to its ambiguity.
model with N pairs of (question, feedback-action, answer) as few-shot examples within a designed prompt (See Appendix A).
3.2 Feedback and Action Generation
Motivated by Yao et al. (2022) that explores the ef- fectiveness of reasoning processes and task-specific actions, we make LLMs generate feedback and action for q (Figure 2(b)) by exploiting extracted presuppositions as working memory. We concate- nate q with extracted presuppositions and PA and feed it as part of an input to the L.M. We use N (question, presupposition, feedback-action triplets as few-shot examples within designed prompt (See Appendix A).
Experiments and Results
Datasets We evaluate PreWoMe three dis- tinct types of questions. For misleading questions with ambiguity, we utilize ASQA (Stelmakh et al., 2022), a LFQA dataset specifically curated for answering ambiguous questions. For misleading questions with False Presupposition (FP), we use the subset of (QA)² (Kim et al., 2022) that are la- beled as harboring FP. For normal questions, we use BioASQ (Krithara et al.. 2023), considering that questions in BioASQ are unlikely to be am- biguous or have FP biomedical experts have carefully curated the questions. Detailed statistics for each dataset are Appendix C.I.
Table shows examples of the generated feed- back and action for two types of misleading ques- tions. The feedback is the description of whether the given input contains FP or not. It can be seen that both types of misleading questions can be han- dled in unified way- by indicating which presup- position is false. If the LLM determines that PA is false, the feedback provides the hint that question q is ambiguous. If other presuppositions are deter- mined be false, then the corresponding feedback states that question contains FP. For normal ques- tions, the feedback informs that all presuppositions are valid (See Appendix B). The feedbacks can be viewed as a classifier that determines the type of input question.
Evaulation For ASQA, we report Rouge-L (Lin, 2004), D-F1 (Disambiguated F1), and DR (Disambiquation-ROUGGE) following Stelmakh et al. (2022). Detail of each metric is in Appendix C.2. However, rather than using the f-measure, propose using recall-measure of Rouge-L in- stead. This is because we notice that ASQA does not provide answers to all aspects of ambiguity, and thus using the f-measure against ASQA cannot fully reflect the comprehensiveness of the gener- ated answer, which is also discussed in Amplayo et al. (2022) (See Appendix C.3). For (QA)² and BioASQ, we report fl-measure of Rouge- 1, Rouge- L, and BleuRT (Sellam et al., 2020).
The action is generated coherently with the feed- back, and serves as an instruction to answering the question. Table shows that for ambiguous ques- tions, the action asks to specify the answer in detail for multiple facets, and for questions with FP, the action asks to correct the FP.
Models We evaluate Pre two LLMs and GPT-4 (gpt-4) (OpenAI, 2023) provided by OpenAl with default hyperparameters and temperature set to O.
GPT-3355 (8pt-3..5-trie
3.3 Answer Generation
When generating the final answer a, the feedback and action (F, A) from the previous step act as a guideline that the language model can refer to. We concatenate g and (F, A) and feed it into the
4.1 Main Experiment
We compare PreWoMe with the vanilla LLM, which involves providing only a question without

---

(a) ASQA D-FI
(b) (QA) R-L
(c) BioASQ R-L
Method
R-L
DR
R-1
B-RT
R-1
B-RT
GPT4
Vanilla
43.66
34.71
38.93
24.77
22.77
0.42
18.34
20.48 22,42
0,45
Co'T
49. 16
34.59
41.24
26.12
24.57
0.43
20.16
0.46
Query Refinement
49.57
35.32
41.84
Step-by-Step w. TD
26.31
24.7
0.44
PreWoMe (Ours)
51.04
35.11
42.34
30.04
28.30
0.45
19.85
22.12
0.48
w/o. Presup
42.68
33.41
37.76
24.67
22.89
0.42
17.96
20.23
0.46
w/o. (F.4)
43.20
34.31
38,50
25.35
23.27
0.43
20.87
22.19
0.46
GPT-3.5
Vanilla PreWoMe
44.83
30.52
36.99
24.37
22.49
0.42
19.65
22.43
0.46
39.58
28.45
33.55
26,54
24.80
0.43
17.72
20.35
0.46
w. GPT- (F.4)
50.66
31.63
40.03
28.54
26.56
0.44
19.90
22.50
0.49
Table 2: Results on GPT-4 and GPT-3.5, "CoT" is Kojima et al. (2022). "Query Refinement" is proposed in Amplayo et al. (2022). "Step-by-Step w. TD" is method using task decomposition (TD) proposed in Kim et al. (2022). "w/o. Presup" is generating feedback-action without using presuppositions. "w'o. (F.A)) is using only presupposition withou! feedback and action. "w. GPT-4 (F,A)" is using feedback-action from GPT-4 to generate answers on GPT-3.5.
any feedback action (F.A). We use six question- answer pairs few-shot examples (N = 6) for both Woe and vanilla LLM. We also com- pare PreWoMe with Thoougt Prompting ("CoT"), following the prompt format of Kojima et al. (2022). As additional baselines, we consider two approaches that each target a single type of misleading question: Amplayo et al. (2022) pro- posed using refined prompts to make LLMs explic- itly consider multifaceted aspects of questions to handle ambiguous questions ("Query Refinement" in Table 2). Kim et al. (2022) proposed a prompt that combine CoT (Kojima et al., 2022) with Task Decomposition (TD) prompting (Khot et al., 2022) to handle questions with FP ("Step-by-Step w. TD" in Table 2). To validate the capability of PreWoMe on misleading questions, we compare PreWoMe against these two additional baselines for each cor- responding target dataset.
For GPT-3.5, we found that the vanilla LLM shows better performance some datasets than PreWoMe. Through manual analysis, observed that GPT-3.5 lacks the ability to generate high- quality feedbacks and actions, compared to GPT-4 (See Appendix D). Thus we conducted experiments where we replaced feedback and action (F,A) with those generated by GPT-4 and generated the answer on GPT-3.5. The results outperformed not only the vanilla GPT-3.5, but also the vanilla GPT-4, demon- strating that 1) misleading (F,A) can induce noise, 2) high-quality (F,A) is transferable between dif- ferent models, and 3) can positively impact L.LMs to the extent where GPT-3.5 outperforms GPT-4. For further validation different N-shot (N 1, 8), we conducted additional experiments, which can be found in Appendix E.
4.2 Human Evaluation
As shown in Table 2, we found that for GPT-4, our approach performs well on misleading ques- tions as well normal questions, outperforming the vanilla LLM across all datasets. Additionally, PreWoMe outperforms CoT on misleading ques- tions (ASQA and (QA)²) and yields results compa- rable to those of CoT for BioASQ. It is noteworthy that CoT itself demonstrates better performance than the vanilla LLM, supporting that utilizing intermediate steps improves QA performance in LLMs. In addition, PreWoMe outperforms both of the additional baselines. Especially considering that Query Refinement was optimized for ASQA and Step-by-Ste w. TD was designed for (QA)?. these findings suggest that our approach versatile in handling arbitrary types of misleading questions.
For human evaluation, we randomly sample total of 90 questions from three datasets and provided two generated answers, one from PreWoMe and one from the vanilla LLM, with the golden answer to three human evaluators. For each criterion overall impression (OI), information completeness (CP), and correctness (CR) we ask human evalu- ators to choose a better answer between two gen- erated answers or mark them as a tie, inspired by Stelmakh et al. (2022). Then, we adopt majority voting policy for each question based the votes given by each human evaluator. Figure 3 shows that Pre WoMe outperforms the vanilla LLM for OL The notable difference between PreWoMe and vanilla LLM in CP for ASQA and CR for (QA)² suggests that WoMe capable effectively covering

---

(+) A504
[B] [QW
[C] Bio450
We also report the final performance of Pre- WoMe and vanilla LLM when given correctly and incorrectly classifying feedback (Table 3 Perfor- mace). Specifically, we report D-FI for ASQA and Rouge-L. for BIOASQ and (QA)² of Wo. and vanilla GPT-4. It can be seen that regardless of the question type, performance gains increase when feedback correctly classifies the question type.
26.7%
16.7%
10.3%
26.77
56.7%
20.3%
33.0%
33.3%
46.7%
50.0%
27.0%
43.3%
35.
48. 31
32.0%
53 33.3%
30.3%
01
CP
04
08
CR
01
6A
Figure 3: Human evaluation results for each dataset. compare Pre Wewiithh vanilla LLM (GPT-4) over 90 questions in total.
Importance of Presuppositions Do presupposi- tions really contribute to generating better feedback and action? We explore this question by omitting presupposition extraction step and generating feed- back and action with GPT-4 just based on the ques- tion. As shown in Table 2 (w/o. Presup), WreW without presuppositions extraction shows poor per- formance, highlighting the role of presuppositions as working memory. One interesting observation is that the performance of PreWoMe without presup- position is worse than the vanilla LLM, indicating that misleading (F,A) becomes noise, consistent with findings in subsection 4.1.
Prediction Ambig,
Performauce
Normal
Conrect
Incomeet
Normal
74.9%
18.5%
6,6%
23.19/21.26
19.03 18.25
Galden
Ambig.
38,9%
39.70
43.65
FP
11.6%
14.0%
74.4%
30.45/22.38
21.93/24.02
Table 3: Prediction shows distribution of predicted ques- tion type of WreWoMe using generated feedback. Per- formance indicates result (PreWoMe vanilla L.LM) with correct and incorrect predictions.
more aspects of ambiguity and correcting FP when facing misleading questions. The prompt used for human evaluation is provided in Appendix F.
Im Importance of Feedback, Action Similarly, we also explore the importance Feedback and Ac- tion. We omit the feedback and action generation step and just feed the questions with the presuppo- sitions that are generated in the previous step into GPT-4. Table 2 (w/o. (F,A)) shows that using pre- suppositions without feedback/action generation step shows lower performance compared the full PreWoMe for mislcading questions. This result demonstrates the importance of feedback/action. and also that generating presuppositions itself is not sufficient to handle misleading questions.
4.3 Ablation Studies
For ablation studies, we first conduct amalysis on the LLM's capability of handling each intermediate step, then verify the importance of each intermedi- ate step within PreWoMe.
Presupposition Analysis To analyze the LLM's capability of generating the correct presuppositions, we randomly sample 150 questions and manually annotate whether the corresponding 395 generated presuppositions were indeed assumed by the ques- tion. The accuracy of presuppositions was ex- tremely high (391 correct out of 395 total x 99%). demonstrating GPT-4's capability of extracting ac-
Conclusion
curate presuppositions.
We propose PreWoMe, new approach that handles any type of question in a unified way by exploiting presuppositions as working memory. WoMe uses intermediate steps to guide the model to gen- erate presuppositions along with useful feedback and action for itself. Our experiments show that Pre WoMe boosts the performance of LLMSs on dif- ferent types of questions without parameter updates. This is achieved even though the model is given no information about the type of incoming ques- tion, demonstrating that Pre is effective for a world setting and will be a significant stepping stone to future works tackling real-world QA tasks.
Feedback Analysis As mentioned in subsec- tion 3.2, feedback works as a classifier that de- termines the type of the given question. We extract the question type predicted by the feedback using a rule-based approach and report the distribution in Table 3 (Prediction). Feedback effectively detects normal questions and questions with FP (74.9% and 74.4%). However, ambiguous questions have low detect ratio, consistent with Min et al. (2020). This indicates that there is more room to optimize for detecting ambiguity in the question, which we leave for future work.

---

Limitations
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in neural information processing systems, 35:22199- 22213.
As we discussed in subsection 4.3, we note that misconstructed feedback and action can be a noise for LLMs, which leads them to generate incorrect answers in the answer generation step. This phe- nomenon, similar hallucination snowballing, was also pointed out by Zhang et al. (2023). Future work may improve PreWoMe by exploring how to prevent hallucination snowballing.
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. 2021. Hurdles to progress in long-form question answering. arXiv preprint (xXix:210300633:
Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. Bioasq- qa: manually curated corpus for biomedical ques- tion answering. Scientific Data, 10(1):170.
Also, our work focuses on closed-book question- answering and explores performance only on closed-book question-answering tasks. Expanding PreWoMe's approach tasks such as question- answering with context, text summarization, etc will be also an avenue of future work.
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2022. Large language models with controllable working memory. arXiv preprint arXiv:2211.05110
Chin-Yew Lin. 2004. Rouge: A package for automatic
References
evaluation of summaries. In Text summarization branches out, pages 74-81.
Reinald Kim Amplayo, Kellie Webster, Michael Collins, Dipanjan Das, and Shashi Narayan. 2022. Query refinement prompts for closed-book long-form ques- tion answering. arXiv preprint a2 17525.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprinl arXiv: 1907.119111110
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1871-999..
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedhack. arXiv preprint arXiv:2303.
Marie Duží and Martina Číhalová. 2015. Questions, an- swers, and presuppositions. Computación y Sistemas, 19(4):647-659.
Joshua Maynez, Shashi Narayan, Berdd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguistics.
Angela Fan, Yacine Jernite, Ethan Perez, David Grang- ier, Jason Weston, and Michael Auli. 2019. Eli5: Long form question answering. arXiv preprint arXiv:1907.09/90.
Liping Ge. 201 I. Pragmatic functions of presupposition advertising english. Asian Culture and History, 3(1):153.
George A Miller, Galanter Eugene, and Karl Pri- bram. 2017. Plans and the structure of behaviour. In Systems Research for Behavioral Sciencesystems Research, pages 369-382. Routledge.
Tushar Khot, Harsh Trivedi, Matthew Finlayson. Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sab- harwal. 2022. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02-00.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Ambigqa: Answering ambiguous open-domain questions. arXiv preprint arXiv:2004.1064
Najoung Kim, Phu Mon Htut, Samuel R Bowman, and Jackson Petty. 2022. (2: Question answer- ing with questionable assumptions. arXiv preprint mrXiv:2212.1003::
OpenAL. 2023. Gpt-4 technical report.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Mcasuring and narrowing the compositionality gap in language models. arXiv preprint arX/v:2270.03350.
Najoung Kim, Ellie Pavlick, Bureu Karagol Ayan, and Deepak Ramachandran. 2021. Which lin-
guist invented the lightbulb? presupposition ification for mestion-answering arXiv preprint arXiv:2101.0033
Pranav Rajpurkar, Robin Jia, and Percy Liang, 2018.
Know what you don't know: Unanswerable questions for squad. arXiv preprint 1806

---

William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.0880
It can be seen in Table 5 that the feedback in- forms that the question does contain any FP. this case, the action functions as an elaboration of the question.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. Bleurt: Learning robust metrics for text gener- ation. In Proceedings of ACI.
C Dataset
C.1 Dataset Statistics
Noah Shinn, Federico Cassano, Beck Labash, Ash- win Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023, Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303. 10.00..11366 11366, 14.
ASQA ASQA includes train, development, and test split each of which contains 4,353, 948, and 1,015 questions. For evaluation, we use the de- velopment split of ASQA because the test split of ASQA is not open publicly.
Tvan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. Asqa: Factoid ques- tions meet long-form answers. arXiv preprint arXiv:2204.66092
(QA)² (QA)² consists of adaptation and evalu- ation splits, each of which contains 32 and 570 questions, respectively. For each split, half of them questions that include FP, and the other half are questions that do not. For evaluation, we use half of the evaluation split - only questions that contain the FPP.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint vrXiv:2201.19900
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. 2023. Large language mod- els are better reasoners with self verificiiiia CoRR, abs/2212.09560
BioASQ BioASQ consists of two distinct tasks (Task A, and Task B) and periodically releases datasets for each task. We use BioASQ9B, which is task B released in 2021, and concatenated the 5 batches (9B1_golden, 9B2_golden, .., and 9B5 (o__gol in the test split for evaluation. We gard ideal_answer _answer in the dataset as a gold reference. Following Stelmakh et al. (2022), we compare pre- dictions against all answers in ideal _aswer and report the maximum score. The concatenated test split includes 497 question-ans pairs.
Fangyuan Xu, Yixiao Song, Mohit lyyer, and Eunsol Choi. 2023. critical evaluation of evaluations for long-form question answering, arXiv preprint arXiv:2305.18201.
Shunyu Yao. Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran. Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint 200::2220.03669.
Xinyan Velocity Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. Crepe: Open-domain question answering with false presuppositions. arXiv preprint 22.1:2211.112555
C.2 ASQA Metric
Stelmakh et al. (2022) proposed several metrics for evaluating ASQA.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah Smith. 2023, How language model hallucinations can snowball, arXiv preprint arXiv:2305. 133.44
ROUGE-L Stelmakh et al. (2022) used a f- measure of ROUGE-L (Lin, 2004) score which is metric for evaluating generated text.
Disambig-F1 Stelmakh et al. (2022) used Disambig-Fi score evaluate the informativeness of generated text. The ambiguous question can be disambiguated to multiple pairs of disambigu- ous questions and short answers (%, a:). For each ambiguous question g, they feed system-generated long form answer as a context, along with each dis- ambiguated question (7:) to Roberta Model (Liu et al., 2019) that pretrained on SQUADv2 (Ra- jpurkar et al., 2018) to predict short answer. Then they calculated the token-level Fl score between the predicted short answer from Roberta and the gold short answer a; Then, Disambig-Fl score a single ambiguous question is given by averaging
A Designed Prompts and Few-shots
Table 4 shows specific designed prompts and the structure of few-shots used in subsection 4.1.
B Normal Question Example
Like Table 1, we give an example of assumptions, feedback, and action normal question (ques- tion without little imperfections). Numbering A in the list of presuppositions stands for the addi- tional presupposition РA. which is manually added by poos-procsssing extracted presuppositions as ex- plained in subsection 3.1.

---

the calculated Fl score of all corresponding disam- biguated questions. The final Disambig-Fi score is calculated by averaging Disambig-F1 scores of all questions in the split.
tion, by observing that the ambiguity is considered in the answer of the vanilla GPT-3.5. In Table 7 (b), GPT-3.5 was misled by the generated action. Specifically, GPT-3.5 answered that there was no fight between Muhammad Ali and Michael Dokes which is in fact not truc. The last row of (b) indi- cates that GPT-3.5 has knowledge about the fight between two boxers, which was not exploited due to faulty action.
Overall DR Score Stelmakh et al. (2022) pro- posed a novel metric for evaluating ASQA: DR Score. Specifically, the DR Score is given by cal- culating the geometric mean of Rouge-L and Dis- ambig Fl as expressed as follows:
D = v(Disambig F1)= * Rouge.
E Experiment on different number of few-shots
C.3 Recall-measure Rouge-L
this part, we explain the reason for using the recall-measure of Rouge-L instead of the f- measure with an actual example. The ASQA dataset (Stelmakh et al., 2022) consists of ambigu- questions and their disambiguated versions with corresponding answers. However, we have no- ticed that the disambiguated question-answer pairs ASQA do not always cover all aspects of am- biguity inherent in the ambiguous question. Thus, even though PreWoMe generates an answer for the ambiguous question with factually correct disam- biguations, if that disambiguation and answer were not provided by ASQA, using the fl-measure will penalize the factually correct answers just because they were not part of the ASQA dataset.
In this part, we show additional experiments con- ducted using N 4, numbers of few-shot ex- amples (Table 8, Table 9). The overall trend is consistent with Table 2, demonstrating that Pre- WoMe robust to different numbers of few-shot examples.
F Human Evaluation Prompt
In this part, release the prompt that given human evaluators in Table 10.
While we do not give higher scores for our fac- tually correct disambiguated answers, we thought it is fair to use a recall-measure to stop getting scores deducted because of them. Table gives example. The generated answer of PreWoMe includes an explanation of "Stuck in the Middle" by Tai Verdes and "Stuck in the Middle" by Boys Like Girls which are reasonable answers to the am- biguous question. However, as ASQA does not include any disambiguated question-answer pairs that cover such aspects, using the -1-measure will give even lower score compared to the prediction without those explanations.
D Quality of Feedback and Action
As discussed in subsection 4.1 and 4.3, poorly gen- erated feedback and action make LLMs underper- form. Table 7 shows some examples of such cases. In Table (a), it can be that the generated action is already itself the answer. This makes GPT-3.5 generate the answer by merely copying the action, which does not cover the ambiguity of the question. However, we can notice that GPT- 3.5 can, in fact, discern the ambiguity of the ques-

---

Prompt
You helpful assistant that analyzes the following question. Your task extract assumptions implicit in given question. You must notice that considering the intention of the question will be helpful to extract hidden assumption of the given question.
Few-shot
Question: When did the great depression began before world war I?
Presuppositions
(I) There was period called the Great Depression.
(2) There was a conflict called World War 1.
(3) The Great Depression began before World War 1.
Step I: Presupposition Extraction, P.A added the post- pott-pocessing..oo
Prompt
You helpful assistant that provides feedback the question and guideline for answering the question. You will he given question and the assumptions that implicit in the question. Your task first, provide feedback on the question based whether it contains any false assumption and then provide guideline for answering the question.
Few-shot
When did the great depression began hefore world war I? Presuppositions
Question:
(I) There was period called the Great Depression.
(2) There
conflict called World War I.
(3) The Great Depression began before World War 1.
(4) There clear and single answer the question.
Feedback: The question contains false assumption that the Great Depression began before World War 1.
Action Correct the false assumptions that the Great Depression began before World War and respond based
on the corrected assumption.
Step Feedback Action generation
Method
PreWoMe
Vanilla
Prompt
You are helpful assistant that answers the given ques- tion. The feedback the question and action needed for answering the question will also be given. Write coherent and detailed answer the question below.
You helpful assistant that answers the given ques- tion. Write coberent and detailed answer to the ques- tion below.
Few-shot
Question: : When did the great depression began before world war 1?
Question: : When did the great depression began before world war 1?
Feedback: The question contains false assumption that the Great Depression began before World War 1. Action Correct the false assumptions that the Great Depression began before World War and respond based on the corrected assumption.
The Great Depression did not begin before World War
I. The Great Depression began in 1929, which was after World War (1914-1918),
The Great Depression did not begin before World War I. The Great Depression began in 1929, which was after World War (1914.1918).
Suep Answer generation.
Table 4: Designed prompts and one few-shot example used in each step. PreWoMe takes all three steps, while the vanilla L.M only takes the Answer Generation step. In our subsection 4,1, we concat N 6 few shots that each have the structure shown in the table above.
Type
Normal Questions
Question
Is the TFRI gene dispensable for erythropoiesis?
Presuppositions
1) There is a gene called TFR1.
2) It is possihle for gene to be dispensahle essential for biological process.
A) There is clear and single answer to the question.
Feedback
The question does not contain any false presuppositions. All the presuppositions are valid.
Action
Answer the question by discussing the role of TFRI gene erythropoiesis and whether it essential or dispensable for the process. Provide evidence from scientific research to support the answer.
Table 5: Examples of normal question from (Krithara al., 2023) and the results of intermediate steps throughout our pipeline.

---

Ambiguous Question
Who sings the song for stuck in the middle?
Disambiguated Question-Answet Pairs
Q1) Which band sings the 1973 song "Stuck in the Middle with You"? >> Stealers Wheel
Q2) Who sings the lead vocals in the 1973 song "Stuck in the Middle with You"? -> A2) Gerry Rafferty
Q3) Who sings harmony in the 1973 song "Stuck in the Middle with You"? -> A3) Joc Egan
Q4) Who sings the 2004 song "Stuck in the Middle"? A4) Clea
Q5) Who sings the theme song for the TV show Stuck in the Middle? > A5) Sonus
Generated answer on Ambiguous Question
There multiple songs titled "Stuck in the Middle" sung by different artists. Some of them include:
1. "Stuck in the Middle with You" by Stealers Wheel: This song was released in 1972 and was written by Gerry Rafferty and Joe Egan. It became hit single and is often associated with the 1992 Quentin Tarantino film, Reservoir Dogs.
2. "Stuck in the Middle" by Tai Verdes: This song released in 2020 and part of Tai Verdes' debut album, "TV. Thh song gained popularity on social media platforms like TikTok.
3. "Stuck in the Middle" by Boys Like Girls: This song is from their 2006 self-titled debut album. The hand is American
pop-rock group formed in 2005.
These are just few examples, and there may be other songs with the same similar titles sung by different artists.
Table 6: Example from ASQA (Stelmakh et al., 2022) that shows why using fl -maasrre for Rouge-L is inappro- priate. The disambiguated question-answer pairs do not include all factually correct answers (Tai Verdes, Boys Like Girls), therefore penalizing generated answers even though they are more comprehensive and complete.
Type
Ambiguous Questions
Question
Who played zordon in the original power rangers?
Generated Feedback
The question contains true assumption.
Generated Action
The actor who played Zordon the original Power Rangers scries is David J. Fielding.
Generated Answer
The actor who played Zordon the original Power Rangers series David Fielding.
Answer by vanilla GPT-3.5
In the original Mighty Morphin Power Rangers television series, Zordon was voiced by actor David J. Fielding, However, Zordon's 's face was portrayed by actor Robert L. Manahan, who was credited as "Zordon's 's Voice" in the show's 's credits.
Golden Answer
In the 1993 American superhero television series Mighty Morphin Power Rangers, fictional character Zordon is initially voiced and portrayed by David Fielding, and later voiced by Robert L. Manahan. In the 1995 American superhero film Mighty Morphin Power Rangers: The Movic, Zordon is portrayed by Nicholas Bell.
(0) Example of poarly constructed feedback and action for the ambiguoas question.
Type
Ambiguous Questions
Question
Who won muhammad ali vs michael dokes?
Generated Feedhack
The question contains a false assumption that there was fight between Muhammad Ali and Michael Dokes.
Generated Action
Correct the false assumption that there was a fight between Muhammad Ali and Michael Dokes and respond based on the corrected assumption.
Generated Answer
Muhammad Ali and Michael Dokes never fought against each other in professional boxing match. Therefore, there is winner to report.
Answer by vanilla GPT-3.5
Muhammad Ali won the fight against Michael Dokes, The fight took place on September 10, 1977, at the The Forumin Inglewood, Californa. Ali won the fight by a unanimous decision after 10 rounds.
Golden Answer
In April 1977. Muhammad Ali fought two exhibitions of three rounds each against Jody Ballard and future World Hcavyweight champion Michael Dakes, both to a No-Decision.
(b) Example of paorly constructed feedback and action for the question that includes FP.
Table 7: Examples of feedback and action that mislead LLMs.

---

(QA) Roage-1.
BloASQ Rougo-1.
Model
Mothod
Rouge-1.
D-P1
DR
Rouge-1
BleuKT
Ronge-1
BleuKT
GPT-4
Vanilla
41.63
34.61
37.96
24
22,42
0.42
18.53
20. 00.68
PreWoMe
45.68
35.76
40.42
27.28
25.33
0.44
18.96
21.2
0.47
GPT-3.5
Vanilla
43:45
30.74
36.55
23.99
22.13
0.41
19.94
22.35
0.46
PreWoMe
41.22
28.
26.78
17:31
19:79
w. GPT (F.4)
45.83
30.79
37.57
27.13
25.42
0.44
18.96
21.82
0.47
Table 8: Result of PreWoMe and vanilla GPT-4 and GPT-3.5 using N few-shot examples. The last row (w. GPT-4 (F,A)) refers to the system that generates answers on GPT-3.5 using feedback and action generated by GPT-4.
Model
Method
Rouge 1.
(a) ASQA
D.F1
DR
Ronge
(b) (QA)
Rouge
BleuRT
Rouge 1
Rouge I.
BloASQ
BleuRT
GPT-4
Vanilla
42.51
34.98
38.56
25.72
23.68
0.43
18.82
20.95
0.45
PreWoMe
42.07
35.82
38.82
33.53
31688
0.46
21.55
24.15
0.46
GPT-3.5
Vanilla
35.64
20.68
23.26
PreWoMe
40.88
28.73
34.27
28.60
25.67
0.44
19.61
22.
0.46
GPT-4 (F.4)
45.39
31.38
37.74
30.38
28.11
0.45
21.65
24.35
0.46
Table 9: Result of PreWoMe and vanilla GPT-4 and GPT-3.5 using few-shot examples. The last row (w. GPTT4 (F,A)) refers to the system that generates answers GPT-3.5 using feedback and action generated by GPT-4.
you shown one question golden answer, ran generared answers Your goul evaluate which unswer better for each of the ihree criteria (Overall Impression, Completeness, Correciness) by referring the given golden
cystems.
For the Overall Impression criterion, your job is to select an answer (or mark it as a tie) that gives you more satisfaction. Fluency, correctness, consistency. formalling be considered factor.
sulficiecyy information, Completeness criterion, your job the given golden answer for the evaluation. Correttness criterion, your job wrong information).
answer mark tie) considering whether answer provides enough information. can refer
answer (or
considering
not include any hallucination (factually
You refor the given golden for evaluation.
Table 10: Prompt given to human evaluator


Reviews:
[{"review_id": "tYG3zur9EC", "reviewer": "Reviewer_M53a", "summary": "", "questions": "1. Qualitatively, how accurate are the generated presuppositions? Is there any correlation between correctly generated presuppositions and answer quality?\n2. This method seems rather tailored to false presupposition detection, it is unclear to me intuitively whether and why it helps with ambiguity detection. Do the ambiguities usually have to do with one or more of the presuppositions in the question? Can this be quantified in some way, or clarified qualitatively through an example?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer M53a.\n\nWe sincerely appreciate all your comments and suggestions. We respond to specific review comments below.\n\n&nbsp;\n\n> **Weakness #1**: The method is somewhat similar to chain-of-thought reasoning, albeit it provides more structure to the reasoning chain than prior work. This begs the question – how does PreWoMe compare with allowing LMs to flexibly use their own “working memory” in any way they see fit (e.g. how does it compare with a “think step-by-step” baseline)?\n\n**Answer**: \n\nTo compare PreWoMe with another baseline using working memory, we conducted additional experiments on GPT-4 with Chain of Thought (CoT) prompting [1].\n\n| ASQA         | Rouge-L | Disambig-F1 | Disambig-Rouge |\n| ------------ | ------- | ----------- | -------------- |\n| Baseline     | 43.66    | 34.71       | 38.93          |\n| CoT          | 49.16    | 34.59       | 41.24          |\n| PreWoMe      | **51.04**| **35.11**   | **42.34**      |\n\n| QA$^2$         | Rouge-1 | Rouge-L | BleuRT  |\n| ------------ | -------  | -------  | ------  |\n| Baseline     | 24.77    | 22.77    | 0.42    |\n| CoT          | 26.12    | 24.57    | 0.43    |\n| PreWoMe      | **30.04**| **28.3** | **0.45**|\n\n| BioASQ         | Rouge-1 | Rouge-L | BleuRT |\n| ------------ | ------- | ------- | ------ |\n| Baseline | 18.34   | 20.48    | 0.45   |\n| CoT      | **20.16**   | **22.42**   | 0.46   |\n| PreWoME  | 19.85   | 22.12   | **0.48**   |\n\n\nThe result shows that even though CoT boosts the baseline performance, PreWoMe outperforms CoT for misleading questions and shows comparable results on ideal questions.\n\nWe claim PreWoMe and CoT have different strengths.\nCoT has been demonstrated to effectively improve the reasoning abilities of LMs, such as mathematical and logical reasoning. \nMeanwhile, compared to CoT which does not consider the question's presuppositions, PreWoMe exploits presupposition, feedback, and action as working memory, and this makes LMs much more robust for open-ended QA in the wild. This is also substantiated by the above results.  \n\n&nbsp;\n  \n> **Weakness #2**: It seems from the paper that all the questions evaluated in ASQA are ambiguous, and all the questions evaluated in (QA)$^2$ have a false presupposition. This seems to be a somewhat contrived evaluation setup that presumably is unable to test for the ability of the feedback stage to properly identify when an ambiguity or false presupposition is present or absent (the model is rewarded for always saying that the question is ambiguous / contains a false presupposition).\n>\n> **Question:** In the n-shot demonstration setup, how are the n in-context examples sampled? If they are sampled from the same dataset, then the model can effectively \"cheat\" because it will only ever see in-context examples of ambiguous questions (if the test sample is ambiguous) or questions with false presuppositions (if the test sample has a false presupposition).\n\n**Answer**:\n\nTo clarify, we assume that the type of input question is not known in advance. Thus, we randomly sampled the 6 shots from multiple datasets (3 shots from ASQA, and 3 shots from QA$^2$), and used those same few shots for every input question regardless of datasets (ASQA, QA$^2$, and BioASQ).\nThese few-shot settings also improved the baseline on ideal and out-of-domain questions (BioASQ).\n\n&nbsp;\n\n> **Weakness #3**: The claim that PreWoMe outperforms GPT4 on ideal questions (L211-212) is also a bit suspect (or at least requires more explanation), given that the method does not substantially differ from baseline question answering in this setting, and the empirical differences are not large.\n\n**Answer**:\n\nAs shown in the table above, CoT using \"working memory\" also improved the baseline on ideal questions, while it is not designed for factoid QA such as BioASQ. CoT promotes LMs to generate intermediate steps of factual knowledge before generating a final answer. \nSimilarly, PreWoMe leverages intermediate steps on ideal questions. We conjecture that such intermediate steps elicit world knowledge of LMs, even on biomedical factoid questions in BioASQ. \n\n&nbsp;\n\n> **Question #1**: Qualitatively, how accurate are the generated presuppositions? Is there any correlation between correctly generated presuppositions and answer quality?\n\n**Answer**: \n\nTo analyze the accuracy of the generated presuppositions, we randomly sampled 150 questions and manually annotated whether the corresponding 395 generated presuppositions are correct, or not. The accuracy of generated presuppositions was extremely high (Correct 391/total 395, ~99%) demonstrating GPT-4’s capability of generating accurate presuppositions. Since the number of incorrect presuppositions is not statistically significant, we didn't calculate correlations. \n\nInstead, we found the relationship between the generated feedback and the quality of answers. As mentioned in our paper, the feedback acts as a classifier that verbally identifies the type of misleading questions. \nWe observed the accuracy on correct and incorrect feedbacks (generated by PreWoMe). For BioASQ and QA$^2$, we report Rouge-L, and for ASQA we report Disambiguation-Rouge. \n\n| | Correct Feed (Ours / Base) | Incorrect Feed (Ours / Base) |\n| ------- | ---------------------- | ------------------------ |\n| BioASQ  | 23.19 / 21.26   | 19.03 / 18.25 |\n| ASQA    | 39.70 / 34.04  | 43.65 / 41.51    |\n| QA$^2$  | 30.45 / 22.38  | 21.93 / 24.02    |\n\nIt is noticeable that PreWoMe tends to show higher performances especially when it successfully generates correct feedbacks. \n\n&nbsp;\n\n> **Question #2**: This method seems rather tailored to false presupposition detection, it is unclear to me intuitively whether and why it helps with ambiguity detection. Do the ambiguities usually have to do with one or more of the presuppositions in the question? Can this be quantified in some way, or clarified qualitatively through an example?\n\n**Answer**:\n\nMultiple steps in PreWoMe contribute to detecting the ambiguity in the question. \nWe provide some qualitative examples: \n\n&nbsp;\n\n**Example 1:**\nThe example below shows feedback/action generation of PreWoMe contributing to detecting ambiguity. PreWoMe effectively detects that the question does not specify which sports World Cup it is referring to, therefore determining that this is an ambiguous question, and providing appropriate action. \nCompared to this, the baseline answers by assuming that the world cup refers to the most famous world cup (FIFA world cup).\n\n\n| Question | Who is hosting the next world cup 2022?|\n| -------- | -------------------------------------------------------------- |\n| PreWoMe  Presuppositions |(1) There is an event called the World Cup (2) The next World Cup is in 2022 (3) The World Cup is hosted by a country (4) There is a clear and single answer to the question.\n|PreWoMe Feedback | The question **does not specify which sport's World Cup is being referred to**. Therefore, the question contains the false assumption that there is a clear and single answer to the question.\n|PreWoMe Action| Your answer should include the host countries for different sports' World Cups in 2022 and provide information about the events and their respective host countries.\n|PreWoMe Prediction| There are several World Cups taking place in 2022 across different sports. Here are some of the major ones: **(1) FIFA World Cup (Soccer)**: The 2022 FIFA World Cup will be hosted by Qatar ………….**(2) ICC Women's Cricket World Cup**: The 2022 ICC Women's Cricket World Cup will be held in New Zealand ………….**(3) FIBA Basketball World Cup (Women)**: The 2022 FIBA Women's Basketball World Cup will be hosted by Australia …………. \n|\n| Baseline | The next FIFA World Cup in 2022 will be hosted by Qatar. ...                                                                                    \n| \n\n&nbsp;\n\n**Example 2:**\nThe example below shows generated presuppositions of PreWoMe contributing to detecting ambiguity. PreWoMe generates presuppositions that comprehensively covers the question, by saying that there is a ‘movie’ or ‘song’ called \"I’ll be home for Christmas\". Compared to this, the baseline just assumes that the question is referring to a song.\n\n| Question | When was i'll be home for christmas released?\n| -------- | -------------------------------------------------------------- |\n| PreWoMe  Presuppositions |(1) There is a **movie or a song** called \"I'll Be Home for Christmas\".  (2) The movie or song was released at some point in time.  (3) There is a clear and single answer to the question.\n|PreWoMe Feedback | The question contains the false assumption that there is a clear and single answer to the question, as **\\\"I'll Be Home for Christmas\\\" could refer to a song or a movie**.\n|PreWoMe Action| Your answer should include the release date of both the song and the movie titled \"I'll Be Home for Christmas\n|PreWoMe Prediction| \"I'll Be Home for Christmas\\\" can refer to both a song and a movie. **(1) Song**: \"I'll Be Home for Christmas\\\" is a popular Christmas song written by …………. **(2) Movie**: \"I'll Be Home for Christmas\" is a 1998 American Christmas family comedy film ………….\n|\n| Baseline | \"I'll Be Home for Christmas\" is a Christmas song written by …                                                                                \n| \n\n\n———————————\n\n**Reference**\n\n[1] Kojima, Takeshi, et al. \"Large language models are zero-shot reasoners.\" Advances in neural information processing systems 35 (2022): 22199-22213.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Method is simple and works empirically well when using GPT-4.\n2. Human studies were run that confirm that the answers generated by PreWoMe were of higher quality than the answers generated by the baseline.\n3. Analysis highlights the importance of generating presuppositions.", "reasons_to_reject": "1. The method is somewhat similar to chain-of-thought reasoning, albeit it provides more structure to the reasoning chain than prior work. This begs the question – how does PreWoMe compare with allowing LMs to flexibly use their own “working memory” in any way they see fit (e.g. how does it compare with a “think step-by-step” baseline)?\n2. It seems from the paper that all the questions evaluated in ASQA are ambiguous, and all the questions evaluated in (QA)^2 have a false presupposition. This seems to be a somewhat contrived evaluation setup that presumably is unable to test for the ability of the feedback stage to properly identify when an ambiguity or false presupposition is present or absent (the model is rewarded for always saying that the question is ambiguous / contains a false presupposition).\n    1. In the n-shot demonstration setup, how are the n in-context examples sampled? If they are sampled from the same dataset, then the model can effectively “cheat” because it will only ever see in-context examples of ambiguous questions (if the test sample is ambiguous) or questions with false presuppositions (if the test sample has a false presupposition).\n3. The claim that PreWoMe outperforms GPT4 on ideal questions (L211-212) is also a bit suspect (or at least requires more explanation), given that the method does not substantially differ from baseline question answering in this setting, and the empirical differences are not large.", "questions_to_authors": "1. Qualitatively, how accurate are the generated presuppositions? Is there any correlation between correctly generated presuppositions and answer quality?\n2. This method seems rather tailored to false presupposition detection, it is unclear to me intuitively whether and why it helps with ambiguity detection. Do the ambiguities usually have to do with one or more of the presuppositions in the question? Can this be quantified in some way, or clarified qualitatively through an example?", "paper_topic_and_main_contributions": "This work proposes to treat ambiguous questions and questions with false presuppositions in a unified manner, specifically by first using a LM to generate any presuppositions within the question, and manually appending to the presuppositions: “There is a clear and single answer to the question.” In the next stage, the LM is expected to generate feedback on which presupposition is correct/incorrect (if the manually appended presupposition is deemed incorrect, the question is deemed ambiguous), followed by an action describing how best to answer the question. Experiments on three domains demonstrated the empirical effectiveness of this method over a vanilla LM baseline that simply generates answers to questions.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "meFNKbNYSQ", "reviewer": "Reviewer_4fGt", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 4fGt.\n\nWe sincerely appreciate all your comments and suggestions. We respond to specific review comments below.\n\n&nbsp;\n\n> **Weakness #1**: There's no evaluation or analysis provided for the intermediate generated presupposition, feedback, and action. It would be interesting to see LLM's capabilities in handling such subtasks. Besides, if the intermediate feedback and action are wrong or misleading, there is no point to feed them to the answer generation step.\n\n**Answer**:\n\nTo check the LLM's capabilities in handling subtasks, we evaluated the performance of intermediate steps.\n\n*First*, we analyze the accuracy of the generated presuppositions.\nWe randomly sampled 150 questions and manually annotated whether the corresponding 395 generated presuppositions are correct, or not. The accuracy of presuppositions was extremely high (Correct 391/total 395, ~99%) demonstrating GPT-4’s capability of extracting accurate presuppositions.\n\n*Second*, as mentioned in our paper, feedback can function as a classifier that predicts the type of input question. The table below reports the performance of the feedback in classifying the input question, which is also reported in our paper (Table 3).\n\n|               | Prediction.Ideal | Prediction.Ambig | Prediction.FP |\n| ------------- | ---------------- | ---------------- | ------------- |\n| Golden.BioASQ | **74.9%**            | 18.5%            | 6.6%          |\n| Golden.ASQA   | 54.6%            | **35.9%**            | 9.5%          |\n| Golden.QA$^2$ | 11.6%            | 14.0%            | **74.4%**         |\n\nFinally, we agree that feeding incorrect feedbacks to the answer generation step may induce noise, which is also mentioned in our limitation section. \nTo analyze this, we observed the accuracy on correct and incorrect feedbacks (generated by PreWoMe). For BioASQ and QA$^2$, we report Rouge-L, and for ASQA we report Disambiguation-Rouge. \n\n| | Correct Feed (Ours / Base) | Incorrect Feed (Ours / Base) |\n| ------- | ---------------------- | ------------------------ |\n| BioASQ  | 23.19 / 21.26   | 19.03 / 18.25 |\n| ASQA    | 39.70 / 34.04  | 43.65 / 41.51    |\n| QA$^2$  | 30.45 / 22.38  | 21.93 / 24.02    |\n\nIt is noticeable that PreWoMe tends to show higher performances especially when it successfully generates correct feedbacks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is unified to both ambiguous questions and FP questions, which is novel.\n2. The idea of extracting presupposition from the question and generating answers by leveraging it is technically sound.\n3. The paper is well-written. Experimental details are explained clearly.", "reasons_to_reject": "1. There's no evaluation or analysis provided for the intermediate generated presupposition, feedback, and action. It would be interesting to see LLM's capabilities in handling such subtasks. Besides, if the intermediate feedback and action are wrong or misleading, there is no point to feed them to the answer generation step.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a new approach for misleading factoid questions (i.e., with false presuppositions or are ambiguous). In specific, the method extracts presuppositions from the question using LLMs, and prompts LLMs to generate [feedbacks, actions] based on the question and presupposition (as working memory). LLMs then are used to generate the answer based on the [question, feedback, action] triplets. Experiments are conducted with misleading questions and answerable factoid questions. Results show the new approach outperforms the baseline by considering only the question.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GVCiWO3jK8", "reviewer": "Reviewer_QSHM", "summary": "", "questions": "* In line 208 – does “few-shot examples (N=6)” for PreWoMe mean having 6 examples for all the steps, or just the final answer generation step?\n\n* What are the details for human evaluation?  What’s the instruction, who are the evaluators and how much do they agree with each other on the choices?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer QSHM.\n\nWe sincerely appreciate all your comments and suggestions. We respond to specific review comments below.\n\n&nbsp;\n\n> **Weakness #1**: Baseline: The authors only compared against few-shot baselines. The query refinement method proposed in [1] should be a reasonable baseline to compare against, at least for the ASQA. Another baseline is the step-by-step baseline from [2] (table 9). As these two methods are both one-step (whereas the model proposed involves prompting the model three times), it’d be interesting to analyze the computational cost v.s. performance tradeoff.\n\n**Answer**:\n\nWe conducted the experiment with two additional baselines. The results are as follows:\n\n| ASQA                        | Rouge-L | Disambig-F1 | Disambig-Rouge |\n| --------------------------- | ------- | ----------------- | -------------------- |\n| PreWoMe                     | **51.04**   | 35.11             | **42.34**                |\n| Query Refinement Method [1] | 49.57   | **35.32**             | 41.84                |\n| Step-by-step [2]            | 41.62   | 33.18             | 37.16                |\n| Baseline                        | 41.63  | 34.61             | 37.96                |\n\n\n| QA$^2$                      | Rouge-1 | Rouge-L | BleuRT |\n| --------------------------- | ------- | ----------------- | -------------------- |\n| PreWoMe                     | **30.04**   | **28.3**    | **0.45**   |\n| Query Refinement Method [1] | 25.52   | 23.63   | 0.41              |\n| Step-by-step [2]            | 26.31   | 24.7    | 0.44   |\n| Baseline                        | 24.54  | 22.42  | 0.42   |\n\nThe results show that PreWoMe outperforms both of the additional baselines, even though [1] was specifically designed for ASQA and [2] was targeted for QA$^2$. This demonstrates that PreWoMe is effective in solving arbitrary misleading questions.\n\nNext, we also compare the computation costs for outputs of PreWoMe and baselines.  As the computation costs of autoregressive LMs are linear to their output tokens, we reported the average length of output tokens for each system in the table below.  Specifically, we analyze the cost of generating intermediate steps before generating the final answer (i.e. presupposition/feedback/action steps for PreWoMe, and intermediate steps for [1] and [2], \nexcluding the tokens of final answers). \n\n| ASQA         |  # Avg Output Token \n| ------------ |  ----------- \n| PreWoMe      |  106.78       \n| Query Refinement Method [1]          |  113.86      \n| Step-by-step [2]          |  16.92   \n\n| QA$^2$       | # Avg Output Token |\n| ------------ |  ------- \n| PreWoMe | 107.75    \n| Query Refinement Method [1]          | 151.38   \n| Step-by-step [2]          | 84.21    \n\nSurprisingly, even though PreWoMe includes multiple steps for generating the answer, the cost of PreWoMe is lower than that of Query Refinement Method [1] for both ASQA and QA$^2$. Nevertheless, the performance of [1] was lower than our PreWoMe.\n\nStep-by-step method [2] demands significantly lower costs compared to others. In Particular, the cost was minimal for the ASQA dataset. We observed the step-by-step method [2] does not tend to generate intermediate steps such as rationale on ASQA. Also, the performance of [2] for ASQA is even lower than the baseline.\n\n&nbsp;\n\n> **Weakness #2**: Results w/o presupposition and having different numbers of few-shot examples are reported. However as there are two steps before generating the final answer, it’d be interesting to see if just having the list of presuppositions and a general prompt to ask the model to generate the final answer is enough, or the second step of prompting the LLM to get a specific feedback for the presupposition is needed.\n\n**Answer**:\n\nTo check the effect of feedback/action generation step on the final answer, we experimented with PreWoMe by just generating the presuppositions, and skipping the feedback/action generation step (denoted as *w/o. F/A* below). The experiment results “without presuppositions and only feedback/action generation” are shown in Table 2 (denoted as *w/o. Presup*) of our paper. \n\n| ASQA         | Rouge-L | Disambig-F1 | Disambig-Rouge |\n| ------------ | ------- | ----------- | -------------- |\n| Full PreWoMe | **51.04**   | **35.11**       | **42.34**          |\n| w/o. F/A  | 43.2    | 34.31       | 38.5           |\n| w/o. Presup     | 42.68   | 33.51       | 37.76          |\n\n| QA$^2$       | Rouge-1 | Rouge-L | BleuRT |\n| ------------ | ------- | ------- | ------ |\n| Full PreWoMe | **30.04**   | **28.3**    | **0.45**   |\n| w/o. F/A  | 25.35   | 23.27   | 0.43   |\n| w/o. Presup     | 24.67   | 22.89   | 0.42   |\n\n| BioASQ       | Rouge-1 | Rouge-L | BleuRT |\n| ------------ | ------- | ------- | ------ |\n| Full PreWoMe | 19.85   | 22.12   | **0.48**   |\n| w/o. F/A  | **20.87**   | **22.19**   | 0.46   |\n| w/o. Presup     | 17.96   | 20.23   | 0.46   |\n\nIt can be seen that using only presuppositions without feedback/action generation step shows lower performance compared to the full PreWoMe for misleading questions. However, it can be seen that it performs slightly better than PreWoMe without presuppositions (that is, just using feedback/action). As mentioned in our paper, this quantitatively demonstrates the importance of presuppositions, and also that incorrect feedback/action can potentially induce noise.\n\n&nbsp;\n\n> **Question #1**: In line 208 – does “few-shot examples (N=6)” for PreWoMe mean having 6 examples for all the steps, or just the final answer generation step?\n\n**Answer**:\n\nWe have used 6 shots throughout all steps of PreWoMe.\nTo elaborate, PreWoMe is designed for a realistic setting, where input question types are unknown. Thus, we use few shots sampled from multiple datasets, and use those same few shots regardless of the type of input question or datasets.\nAn example of how the few shots for each step are constructed is illustrated in Table 4 of our paper’s appendix.\n\n&nbsp;\n\n> **Question #2**: What are the details for human evaluation? What’s the instruction, who are the evaluators and how much do they agree with each other on the choices?\n\n**Answer:**\n\nWe hired three undergraduate students as human evaluators.\nFor evaluation, we provided the evaluators with a question, a golden answer, and two generated answers (one from the baseline, and one from PreWoMe). Then, we asked them to select one of the two answers based on three criteria - overall impression (OI), information completeness (CP), and correctness (CI) - as mentioned in the paper.\nThe order of two generated answers was randomly shuffled for every question.\n\nThis is the prompt that was given to the human evaluators;\n\n*In this task, you will be shown one question and its golden answer, and two generated answers from different systems. Your goal is to evaluate which answer is better for each of the three criteria (Overall Impression, Completeness, Correctness) by referring to the given golden answer. For the Overall Impression criterion, your job is to select an answer (or mark it as a tie) that gives you more satisfaction. Fluency, correctness, consistency, sufficiency of information, or even formatting can be considered as a factor. For the Completeness criterion, your job is to select an answer (or mark it as a tie) considering whether the answer provides enough information. You can refer to the given golden answer for the evaluation. For the Correctness criterion, your job is to select an answer (or mark it as a tie) considering whether the answer does not include any hallucination (factually wrong information). You can refer to the given golden answer for the evaluation.*\n\nThe table below reports the Fleiss Kappa score, which shows how much the human evaluators agree with each other.  \nIt can be seen that all scores range between 0.4 and 0.6, which is interpreted as having a ‘moderate agreement’ between the evaluators (based on the widely used interpretation table proposed in [3]).\n\n|    | ASQA | QA$^2$ | BioASQ |\n| -- | ---- | ------ | ------ |\n| Overall Impression | 0.56 | 0.55   | 0.48   |\n| Completeness | 0.57 | 0.53   | 0.4    |\n| Correctness | 0.48 | 0.54   | 0.5    |\n\n\n----------------------\n\n**Reference**\n\n[1] Amplayo, Reinald Kim, et al. \"Query Refinement Prompts for Closed-Book Long-Form Question Answering.\" arXiv preprint arXiv:2210.17525 (2022).\n\n[2] Kim, Najoung, et al. \"QA$^2$: Question Answering with Questionable Assumptions.\" arXiv preprint arXiv:2212.10003 (2022).\n\n[3] Landis, J. Richard, and Gary G. Koch. \"The measurement of observer agreement for categorical data.\" biometrics (1977): 159-174.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper proposes a new prompting approach for generating long-form answers.\n* The author conducted both automatic and human evaluation on multiple datasets (ASQA, QA^2 and BioASQ), covering different types of questions.\n* The paper is generally well-written and easy to follow.", "reasons_to_reject": "My major concern is the lack of baseline and ablation study here. \n\n* Baseline: the authors only compared against a few-shot baselines. The query refinement method proposed in [1] should be a reasonable baseline to compare against, at least for the ASQA dataset. Another baseline is the step-by-step baseline from [2] (table 9). As these two methods are both one-step (whereas the model proposed involves prompting the model three times), it’d be interesting to analyze the computational cost v.s. performance tradeoff.\n\n* Ablation: results w/o presupposition and having different numbers of few-shot examples are reported. However as there are two steps before generating the final answer, it’d be interesting to see if just having the list of presuppositions and a general prompt to ask the model to generate the final answer is enough, or the second step of prompting the LLM to get a specific feedback for the presupposition is needed.\n\n[1] Reinald Kim Amplayo, Kellie Webster, Michael Collins, Dipanjan Das, and Shashi Narayan. 2022. Query refinement prompts for closed-book long-form question answering.\n\n[2] Najoung Kim, Phu Mon Htut, Samuel R Bowman, and Jackson Petty. 2022. Θ 2: Question answering with questionable assumptions.", "questions_to_authors": "* In line 208 – does “few-shot examples (N=6)” for PreWoMe mean having 6 examples for all the steps, or just the final answer generation step?\n\n* What are the details for human evaluation?  What’s the instruction, who are the evaluators and how much do they agree with each other on the choices?", "paper_topic_and_main_contributions": "This paper proposes a multi-step prompting method for closed-book long-form QA, which includes three steps: extracting presupposition statements from the question; generating feedback and action given the extracted presupposition and finally generating the answer given the question, feedback and action. Experiments are conducted on three datasets, covering ambiguous questions, questions with false-presuppositions and a biomedical dataset. Automatic evaluation and human evaluations show that the proposed method outperforms the baseline few-shot approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper suggests a multi-step prompt-based approach for handling open-domain questions, where the questions may contain false-presuppositions or be ambiguous. Specifically, the authors propose to first prompt an LLM for extracting the presuppositions in the question, then prompt it again to decide whether those presuppositions are correct and how to address them in the answer, and finally, given this feedback, generate an answer. The experimental results are sound and the problem is timely and important. Given the above, this paper can be a good fit for the conference.
Decision: Accept-Main