Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models
Authors: Gangwoo Kim
Sungdong Kim
Byeongguk Jeon
Joonsuk Park
Jaewoo Kang
Venue: EMNLP 2023 Main (2023)
Keywords: Question Answering
Large Language Model
Ambiguous QA
Open-domain QA
TL;DR: we introduce Tree of Clarifications: It recursively constructs a tree of interpretations for the ambiguous questions via few-shot prompting leveraging external knowledge; finally to generate a long-form answer.

Abstract: Questions in open-domain question answering are often ambiguous, allowing multiple interpretations. 
One approach to handling them is to identify all possible interpretations of the ambiguous question (AQ) and to generate a long-form answer addressing them all, as suggested by Stelmakh et al., (2022). While it provides a comprehensive response without bothering the user for clarification, considering multiple dimensions of ambiguity and gathering corresponding knowledge remains a challenge.
To cope with the challenge, we propose a novel framework, Tree of Clarifications (ToC):
It recursively constructs a tree of disambiguations for the AQ---via few-shot prompting leveraging external knowledge---and uses it to generate a long-form answer.
ToC outperforms existing baselines on ASQA in a few-shot setup across the metrics, while surpassing fully-supervised baselines trained on the whole training set in terms of Disambig-F1 and Disambig-ROUGE. Code is available at https://github.com/gankim/tree-of-clarifications.

Tree of Clarifications:
Answering Ambiguous Questions
with Retrieval-Augmenter Large Language Models
Gangwoo Kim
Sungdong 23m 2,34
Byeongguk Jeon'
Joonsuk Park 2,3,5 Jaewoo Kang"
Korea University! KAIST AI
NAVER Cloud?
NAVER Al Lab
University of Richmond
{gangwoo_ kimm, akjeon1211, kangj)@korea.ac. ac. kr
sungdong. kim@navercorp.c.m com
parkejoonsuk. org
Ambiguous Question (AQ)
Abstract
"What country has the medals in Olympic history?
Questions in open-domain question answering often ambiguous, allowing multiple inter- pretations. One approach to handling them is identify all possible interpretations of the ambiguous question (AQ) and to generate long-form answer addressing them all, as sug- gested by Stelmakh et (2022). While provides comprehensive response without bothering the user for clarification, considering multiple dimensions of ambiguity and gather- ing corresponding knowledge remains a chal- lenge. To cope with the challenge, we propose novel framework, TREE OF CLARIFICATIONS (ToC): It recursively constructs a tree of disam- biguations for AQ-via few-shot prompt- ing leveraging external knowledge- and uses it to generate a long-form answer. ToC out- performs existing baselines on ASQA in a few- shot setup across all metries, while surpass- ing superveue baselines trained on the whole training in terms of Disambig-FI and Disambig-ROU ROUGE. Code is available at getthh..oomgaakkknnee-e-----rria
Tree of fiatiois
Question
* Pruned
Information
Clarification
Retrieval
DQ$
DQ
DQ
"What country the most total medall In Olympic history?
Question
Question Clarification
Clarification
Passages
DQ11 DQ DQ
DQ21 DQ22
DQza
DQ24
"What country has the most medals in winter Olympichistory?
"What country has the gold medals in Olympic history?"
Long Form Answer
"The United States has the most total medals Norway has won most medals in winter Olympic."
Answer Generation
Figure I: Overview of TREE OF CLARIFICTIONS. (1) relevant passages for the ambiguous question (AQ) are retrieved. (2) leveraging the passages, disambiguated questions (DQs) for the AQ are recursively generated via few-shot prompting and pruned as necessary. (3) long- mor-formmo answer addressing all DQs is generated.
1 Introduction
open-domain question answering (ODQA), users often ask ambiguous questions (AQs), which can be interpreted in multiple ways. To handle AQs, several approaches have been proposed, such providing individual answers to disambiguated questions (DQs) for all plausible interpretations of the given AQ (Min et al., 2020) or asking clarifi- cation question (Guo et al., 2021). Among them, we adopt that of Stelmakh al. (2022), which pro- vides comprehensive response without bothering the user for clarification: The task is to identify all DQs of the given AQ and generate long-form answer addressing all the DQs (See Figure 1).
tiple dimensions ambiguity. For example, the AQ "what country has the most medals in Olympic history" in Figure can be clarified with respect to the type of medals-gold, silver, or bronze- or Olympics-sumee
or winter; and (2) substantial knowledge is required to identify DQs and respec- tive answers. For example, it requires knowledge to be aware of the existence of different types of medals and the exact counts for each country.
To address the challenges and provide long- form answer to AQ, we propose a novel framework, TREE OF CLARIFICATIONS (ToC): recursively constructs tree of DQs for the AQ-via via few-shot
There are two main challenges to this task: (1) the AQ may need to be clarified by considering mul-
Corresponding author

---

prompting leveraging external Knowledge-and uses it to generate long-form answer. More specif- ically, first, relevant passages for the AQ are re- trieved. Then, leveraging the passages, DQs for the AQ are recursively generated via few-shot prompt- ing and pruned as necessary. Lastly, a long-form answer addressing all DQs is generated. The tree structure promotes exploring DQs in targeting par- ticular dimensions of clarification, addressing the first challenge, and the external sources offer ad- ditional knowledge to cope with the second chal- lenge.
et al. (2023) suggest optimal prompts specifically engineered for the task. Kuhn et al. (2022) prompt LLMs to clarify ambiguous questions selectively. However, the studies do not utilize external infor- mation to ensure the factual correctness of the dis- ambiguations, thereby potentially increasing the risk of hallucinations from LLMs. Moreover, the results could be bounded by inherent parametric knowledge of LLM. Coneurrently, Lee et al. (2023) automatically generate clarifying questions re- solve ambiguity.
Our framework involves the recursive tree archi- tecture, inspired by several prior studies. Min et al. (2021) propose the rrc-dden algorithm to au- toregressively rerank passages in ambiguous QA. Gao et al. (2021) iteratively explore additional inter- pretations and verify them in round-trip manner. Concurrently, extending chain of thoughts (Wei et al., 2022) prompting, Yao al. (2023) apply the tree architecture to reasoning tasks for deduc- tive or mathematical problems. On the contrary, ToC recursively clarifies questions and introduces self-verification method to prune unhelpful DQs.
Experiments demonstrate that our proposed use of LLMs with aitrivval-aggmentatio and guid- ance to pursue diverse paths of clarification results in the new state-of-the-art on ASQA (Stelmakh al., 2022>a long-form QA benchmark for AQs. TOC outperforms existing baselines on ASQA in a few-shot setup across all metries. In addition, this 5-shot performance surpasses that of the fully- supervised baselines trained on the whole training set by 7.3 and 2.9 in terms of Disambig-F1 and Disambig-ROUGE. respectively.
The main contribution of this work is proposing a novel framework, TREE OF CLARIFICATIONS (ToC), for generating long-form answers to AQs in ODQA, advancing the state-of-the-ar on the ASQA benchmark. ToC introduces two main in- novations:
3 Tree of Clarifications
We introduce a novel framework, TREE OF CLARI- FICATIONS (ToC), as illustrated in Figure 1. We first devise retrieval-augmennte clarification (RAC; Sec. 3.1), basic component that clarifies AQ and generates DQs based on relevant passages. ToC explores various fine-grained interpretations, rep- resented as a tree structure (TS; Sec. 3.2) by re- cursively performing RAC and pruming unhelpful DQS. Lastly, it aggregates the tree and generates a long-form answer addressing all valid interpreta- tions.
It guides LLMs to explore diverse paths of clarification of the given AQ in a tree structure with the ability to prune unhelpful DQs.
To the best of our knowledge, it is the first to combine retrieval systems with LLM for generating long-form answers to AQs.
2 Related Work
3.1 Retrieval-Augmented Clarification (RAC)
We first retrieve relevant Wikipedia documents for the AQ by using two retrieval systems, Col- BERT (Khattab and Zaharia, 2020) and Bing search engine!. ColBERT is a recent dense retriever that has effective and efficient zero-shot search qual- ity. Following Khattab et al. (2022), we use the off-the-shelff model pre-trained on MS-Mare (Ba- jaj et al., 2016). We additionally include the Bing search engine to promote the diversity of retrieved Wikipedia passages. Finally, we obtain over 200 passages by combining passages retrieved by each system.
A line of studies (Min al., 2020, 2021: Gao et al., 2021; Shao and Huang, 2022) extends retrieve-and- read frameworks dominant in ODQA task (Chen et al., 2017; Karpukhin et al., 2020; Lewis al., 2020; Izacard and Grave, 2021) to clarify AQ and generate DQs with corresponding answers to them. However, their approaches require fine-tuning mod- els on the large-scale train set. On the other hand, our framework enables LLM to generate compre- hensive response addressing all DQs via few-shot prompting.
Recent studies introduce LLM- methods to generate a long-form answer to the AQ. Amplayo
https//wwww.miccssttccmmmmmmm

---

After collecting a passage for the AQ. rerank and choose top-k passages and augment them to prompt. We use SentenceBERT (Reimers and Gurevych, 2019) pre-trained on MS-Marco as the reranker backbone. For in-context learn- ing setup, we dynamically choose k-shot examples with the nearest neighbor search² and add them to the prompt. We initiate with the instruction of Am- playo et al. (2023) and revise it for our setup. Given the prompt with relevant passages and AQs, LLM generates all possible DQs and their corresponding answers?.
Model
D-F1
R-L
Fully- supey-supe
TS-Large Closed-Book
7.4
33,5
15.7
T5-Large JPR PaLM w/ Soft Prompt Tuning"
26.4
43.0
33.7
27.8
37.4
32.1
Few-shot Prompting (5-shor)
PaLM GPT-3"
25.3
34.5
29.6
25.0
31.8
28.2
Tree of Clarifications (ToC; Ours)
GPT-3 + RAC
31.1
39.6
35.1
GPT-3 RAC TS
32.4
40.0
36.0
GPT-3 RAC TS w/ Pruning
33.7
39.7
36.6
from Amplayo et al. (2023)
3.2 Tree Structure (TS)
Table 1: Evaluation results for long-form QA ambigu- ous questions from the development set of ASQA (Stel- makh et al., 2022). Baselines are either fully-supervised or 5-shot prompted. Note, ToC framework consists of retrieval- saureete-- clarification (RAC) and tree struc- ture (TS).
To effectively explore the diverse dimensions of ambiguity, we introduce recursive tree structure of clarifications. Starting from the root node with AQ, it progressively inserts child nodes by recur- sively performing RAC, each of which contains disambiguated question-ansee
pair. In each ex- pansion step, passages are reranked again regarding the current query. It allows each step to focus on its own DQ encouraging TOC to comprehend a wider range of knowledge. Exploration of a tree ends when it satisfies termination conditions; it reaches the maximum number of valid nodes the max- imum depth. We choose the breadth-first search (BFS) by default, hence the resulting tree could cover the broader interpretations..
if the given answer could be correct answer AQ.
Answer Generation Once constructing the tree of clarifications, ToC aggregates all valid nodes and generates comprehensive long-form answer to AQ. It selects the disambiguations retained nodes of the resulting tree with the relevant pas- sages. the number nodes is insufficient, we undo the pruning steps from closer nodes to the root node in BFS order. Passages that contain the answers of valid nodes are prioritized. It finally generates long-form answer, encoding AQ, lected disambiguations, and relevant passages
Pruning with Self-Verification To remove un- helpful nodes, we design pruning method, in- spired by current studies for verfiveeriinttion (Ka- davath et al., 2022: Cole al., 2023). Specifically, we check the factual coherency between the an- swers target node and the AQ the root node. By doing so, we discard the generated DQs that ask different or irrelevant facts from the original one. For example, given an AQ "Who will host the next world cup 2022?", generated disambiguation "DQ: Who hosted the world cup 2018? A: Russia" is factually consistent question-answer pair but it changes the original scope of the AQ We perform self- verificatin by prompting LLMs to determine whether the current node would pruned or not. Prompted with AQ, the answer to the target DQ, and the answer-containing passage, LLM identifies
Experiment
4.1 Experimental Setup
Datasets All baselines and our framework are evaluated on ASQA (Stelmakh et al., 2022), It is a long-form QA dataset built upon the 6K ambiguous questions identified from AmbigNQ (Min et al.. 2020). More details are in Appendix A.1
We use three evaluation metrics, following Stelmakh et al. (2022). (1) Disambig-F1 (D-F1) measures the factual correct- of generated predictions. It extracts short an- swers to each DQ and computes their F1 accuracy. (2) ROUGE-L (R-L) measures the lexical overlap between long-form answers from references and predictions. (3) DR score is the geometric mean of
Evaluation Metries
"Soe Appendix A.3 for detailed implementation *See Appendix C.2 for example prompts
'It suboptimal to adopt depth-first search since would encounter unambiguous questions more frequently. See Appendix for failure cases.
"See Appendix C.3 for more detuiled case studies
'Sec Appendix C.4 for an example prompt

---

Model
D-F1
R-L
Filtration
#(QQs)
Answer-F1
GPT-3 (Baseline)
24.2
36.0
29.5
w/o Pruning (None)
12,838
40.9
GPT-3 w/ RAC
31.1
39.6
35.1
W Pruning
Disambiguations
30.5
37.3
33.7
+ Deduplication
10,598
40.1
Bing Search Engine
28.5
37.4
32.7
Self-Verification
4,239
59.3
Retricval Systems
25.6
35.1
30.0
Table 3: Ablated results with and without pruning meth- ods. The number of retained DQs after pruning and Answer-Fl are reported.
Table 2: Ablation study on all components of retrieval- augmented clarification (RAC).
two scores, which assesses the overall performance. For validating intermediate nodes, we additionally use Answer-Fl that measures the accuracy of gen- erated short answers disanbiguation. Further details are in Appendix A.2.
1.3 and 0.9. When pruning the tree with our pro- posed self-verification (TS Pruning), the model achieves state-of tfate-of-the-ar performance in D-F1 and DR score, surpassing the previous few-shot base- line by 8.4 and 7.0. Notably, it outperforms the best model in fully-supervised setup (T5-large with JPR) by 7.3 and 2.9. In the experiment, T5-Large in a closed-book setup achieves comparable per- formance with LLM baselines in ROUGE-L score despite its poor D-F1 scores. It reconfirms the ob- servation from Krishna et al. (2021) that shows the limitations of the ROUGE-L metric.
Baselines Stelmakh et al. (2022) propose fine- tuned baselines. They fine-tune T5-large (Raf- fel et al., 2020) to generate long-form answers on the whole train set. Models are evaluated in the closed-book. setup combined with JPR (Min et al., 2021), task-specific dense retriever for am- biguous QA by enhancing DPR (Karpukhin al.. 2020). On the other hand, Amplayo et al. (2023) propose a prompt engineering method to adapt LLMs to the ASQA benchmark. They employ PaLM (Chowdhery et al.. 2022) and Instruct- GPT (Ouyang et al., 2022) that learn the soft prompts or adopt in-context learning with few-shot examples. They conduct experiments in the closed- book setup. Note that they share the same back- bone with our models, GPT-3 with 175B parame- ters doeettddvvniii-022
Integrating retrieval systems largely con- tributes to accurate and diverse disambigua- tions. Table 2 displays the ablation study for mea- suring the contributions of each proposed compo- nent. When removing disambiguations from few- shot training examples, the ROUGE-L score sig- nificantly degraded, which shows the importance of the intermediate step to provide the complete answer. Integrating retrieval systems (i.e., Bing search engine and ColBERT) largely improves the model performance, especially in the D-FI score. It indicates using external knowledge is key to en- hancing the factual correctness of clarification. We report intrinsic evaluation for each retrieval system in Appendix B.
4.2 Experimental Results
ToC outperforms aully-supervised and few-shot prompting baselines. Table shows the long-form QA performance of baselines and TOC on the de- velopment set of ASQA. Among baselines, using the whole training set (Fully-supervised) achieves greater performances than Few-shot Prompting in all metrics. It implies that long-form QA task is challenging in the few-shot setup. In the closed-boot setup, GPT-3 shows competitive per- formances with Ta-ge with JPR in D-FI score, showing LLM's strong reasoning ability over its inherent knowledge.
Our pruning method precisely identifies help- ful disam Disambiguations from the tree. Table shows intrinsic evaluation for generated disambiguations, where all baselines are evaluated with Answer-F1 score that measures the Fl accuracy of the answer to the target DQ. Compared to the baseline, the valid nodes that pass self-verification contain more accurate disambiguations, achieving much higher Answer-Fl score (+18.4). On the other hand, solely using deduplication does not advance the accuracy, indicating the efficacy of our proposed self-verification method.
Among models, LLM with RAC outper- forms all other baselines in D-F1 and DR scores. It indicates the importance of leveraging external knowledge in clarifying AQs. Employing the tree structure (TS) helps the model to explore diverse interpretations, improving D-FI and DR scores by

---

5 Discussion
long-form answer. Our framework explores diverse dimensions of interpretations of ambiguity. Experi- mental results demonstrate ToC successfully guide LLMs to traverse diverse paths of clarification for a given AQ within tree structure and generate com- prehensive answers. We hope this work could shed light on building robust clarification models, which can be generalized toward real-world scenarios.
Ambiguity Detection ToC is designed to clar- ify AQs without bothering users; hence does not explicitly identify whether the given question is am- biguous not. It tries to perform clarification even if the question cannot be disambiguated anymore, often resulting in generating duplicate or irrele- vant DQs". However, we could presume a question to be unambiguous biguated In ToC, when it fails to disambiguate the given question or all generated disambiguations are pruned, the question could be regarded as un- ambiguous.
can no longer be disam-
Limitations
Although ToC is a model-agnostic framework that could be combined with other components, our study is limited in demonstrating the generalizabil- ity of different kinds or sizes of LLMs. In addition, the experiments
Computational Complexity Although ToC re- quires multiple LLM calls, its maximum number is less than 20 times per question. Exploration of the tree ends when it obtains the pre-defined num- ber of valid nodes (10 in our experiments). Since the clarification process generates from two five disambiguations for each question, it satisfies the termination condition in a few steps without the pruning method. Failing to expand three times in row also terminates the exploration. Pruning steps consume smaller amount of tokens since they encode single passage without few-shot exem- plars. Compared to the existing ensemble methods such as self-comsistency (Wei et al., 2022) which cannot be directly adopted to the generative task, ToC achieves a state-of-the-art performance with a comparable number of LLM calls.
only conducted mark, ASQA (Stelmakh et al., 2022). Although ToC enables LLM explore diverse reasoning paths by iteratively prompting LLM, the cost of multiple prompting is not negligible.
bench-
We tried the recent prompting method, chain of thoughts (Wei et al., 2022), but failed to en- hance the performance in our pilot experiments. It might indicate the disambiguation process requires external knowledge, which shows the importance of document--rouunee or retrieval-augmented sys- tems. Future work could suggest other pruning methods that identify unhelpful DQs more effee- tively. The performance could be further enhanced by using the state-of-the-art reranker in the an- swer sentence selection task, as proposed by recent works (Garg al., 2020; Lauriola and Moschitti, 2021).
The key idea of ToC could be potentially generalized to other tasks and model architectures. It has a model-agnostic structure that could effectively explore diverse paths of recursive reasoning, which would be helpful for tasks that re- quire multi-step reasoning, such as multi-hop QA. Future work might investigate the generalizability of ToC to diverse tasks, datasets, and LM architec- tures.
Generalizability
Acknowledgements
The first author, Gangwoo Kim, has been sup- ported by the Hyundai Motor Chung Mong-Koo Foundation. This research was supported by the National Research Foundation of Korea (NRF- 2023R A2C3004176 553-2033-00262000 MSIT (Ministry of Science and ICT), Korea, under the ICT Creative Consilience program (ITTP-2022- 2020-0-01819) supervised by the ITTP (Institute for Information & communications Technology Planning & Evaluation), and the Electronics and Telcoommunications Research Institute (RS-2023- 00220195).
the
6 Conclusion
In this work, we propose a novel framework, TRFE OF CLARIFICATIONS It recursively builds tree of disambiguations for the AQ via shw- prompting with external knowledge and utilizes it to generate
"See Appendix
failure cases
"The idea is aligned with the annotation process of Am- bigQA (Min et al., 2020), in which the target question classified as ambiguous if multiple distinct answers to it were observed.

---

References
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.
Reinald Kim Amplayo, Kellie Webster, Michael Collins, Dipanjan Das, and Shashi Narayan. 2023. Query refinement prompts for closed-book long-form ques- tion answering. Proceedings of the Annual Meet- ing of the Association for Computational Linguisties.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781
Payal Bajaj. Daniel Campos, Nick Craswell, Li Deng. Jianfeng Gao, Xiaodong Liu, Rangan Majumder, An- drew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine read- ing comprehension dataset. 30th Conference on Neu- ral Information Processing Systems (NIPS 2016), Barcelona, Spain.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-searcht predict: Composing retrieval and language mod- els for knowledge-intensivy nlp. arXiv preprint arXiv:2212.14022.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1870-1879.
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pages 39- 48.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. 2021. Hurdles to progress in long-form question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 4940-4957.
Jeremy R Cole. Michael JQ Zhang, Daniel Gillick, Ju- lian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. 2023. Selectively answering ambiguous questions. arXiv preprint arXiv:2305. 1466.
Yifan Gao, Henghui Zhu, Patrick Ng. Cicero dos San- tos, Zhiguo Wang. Feng Nan, Dejiao Zhang, Ramesh Nallapati, Andrew Arnold, and Bing Xiang. 2021. Answering ambiguous questions through generative evidence fusion and round-trip prediction. In Pro- ceedings of the 59th Annual Meeting of the Asso- ciation Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 3263- 3276.
Lorenz Kuhn, Yarin Gal, and Sehastian Farquhar. 2022. Clam: Selective clarification for ambiguous ques- tions with large language models. arXiv preprint arXiv:2212.07769.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452- 466.
Siddhant Garg, Thuy Vu, and Alessandro Moschitti. 2020. Tanda: Transfer and adapt pre- pre-trained trans- former models for answer sentence selection. In Proceedings of the AAAI conference on artificial in- lelligence, volume 34, pages 7780-7788.
Ivano Lauriola and Alessandro Moschitti. 2021. An- swer sentence selection using local and global context in transformer models. In European Conference Information Retrieval, pages 298-312. Springer.
Meiqi Guo. Mingda Zhang, Siva Reddy, and Malihe Alikhani. 2021. Abg-coqa: Clarifying ambiguity in conversational question answering. In 3rd Confer- ence on Automated Knowledge Base Construction.
Dongryeol Lee, Segwang Kim, Minwoo Lee, Hwan- hee Lee, Joonsuk Park, Sang-Woo Lee, and Kyomin Jung. 2023. Asking clarification questions to han- dle ambiguity open-domain qa. arXiv preprint arXiv:2305.
Gautier Izacard and Édouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chupter of the Associ- ation for Computational Linguistics: Main Volume, pages 874 880.
Patrick Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9474.
Jeff Johnson, Matthijs Douze, and Hervé Jégou, 2019. scale similarity search with gpus. IEFE Transactions on Big Data, 733)5335-547.

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke 7ettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint 1rXXiv1907.11 16692.
of pre-trained transformers. Advances Neural formation Processing Systems, 33:5776-577 5788.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le. Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-2445
Sewon Min, Kenton Lee, Ming-
Beig--Wei Wei Chang, Kristina Toutanova, and Hannaneh Hajishirzi. 2021. Joint passage ranking for diverse multi-answer retrieval. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 6997-7008.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao. and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models, arXiv preprint arXiv:2305. 10601.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Ambigqa: Answering biguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods Nat- ural Language Processing (EMNLP), pages 5783- 5797.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems. 35:27730-27744
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, (1(455585555
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't t know: Unanswerable ques- tions for squad. In Proceedings of the 56th Annual Meeting of Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784-789.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bre Sentence embeddings using siamese nert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992.
Zhihong Shao and Minlie Huang. 2022. Answering open-domain multi-answer questions via a recall- then-verify framework. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1825- 1838.
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming- Wei Chang. 2022. ASQA: Factoid questions meet long-form answers. In Proceedings of 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 8273 8288, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression

---

A Experimental Setup Details
Retrieval System
AC@
AC@ 30
AC@
A.1 Ambiguous QA Datasets
CvlBERTv2 w/ Reranker
56.4
68.4
73.4
56.8
69.0
73.4
All baselines and our framework are evaluated on ASQA benchmark (Stelmakh al., 2022). It is long-form QA dataset built on the sub- set of ambiguous questions identified from Am- bigNQ dataset (Min et al., 2020). It contains open- domain questions collected from Natural Ques- tions (Kwiatkowski et al., 2019). ASQA consists of 6,316 16 ambiguous questions and their long-form an- swers with disambiguations, split into 4,353, 948, and 1,015 train, development, and test set, respec- tively.
Bing Seurch Engine
43.3
58.3
73.5
w/ Reranker
62.7
68.0
72.8
Combined w/ Reranker
64.2
77.4
80.1
Table 4: Intrinsic evaluation of retrieval systems. An- swer coverage at (AC@K) measures the proportion of disambiguated answers that are covered by top-k re- trieved passages.
prune the target node or not, rerank and pick the most relevant passage among those containing the answer in the target node. In the answer generation process, we took ten valid disambiguations in BFS order and five answer-containing passages. We use API served by OpenAl to employ GPT-3 as our backbone. We set max tokens as 300 and top-p as 1.0.
A.2 Evaluation Metrics
Following Stelmakh et al. (2022), we use three eval- uation metries on ASQA. First, ROUGE-L (R-L) measures the lexical overlap between long-form an- swers from references and -enerrtee predic- tions. Since the benchmark provides two ground- truth answers, report the maximum ROUGE-L score. Disambig-F1 (D-F1) measures the factual correctness of generated predictions. A reading comprehension model, RoBERTa (Liu et al., 2019) trained on SQuADv2 (Rajpurkar& et al., 2018), finds short answers to the ground-truth DQs from the generated long-form response. Then, Fl accuracy of the detected answer is calculated to check if the long- form answer contains accurate information. Disambiguation-ROGG ROUGE (DR) score is computed the geometric mean of ROUGE-L and Disambig- Fl to measure the overall performance. We addi- tionally use Answer-F1 to validate the disambigua- tions. It computes the maximum Fl accuracy of answers to a single DQ. We use ground-truth dis- ambiguations provided by AmbigNQ (Min et al., 2020).
B Additional Experiment
B.1 Intrinsic Evaluation for Retrieval Systems
We randomly sample 100 examples from ASQA dataset and report intrinsic evaluation results for retrieval systems. Since single AQ has multi- ple ground-truth DQs and their answers, is not trivial check how many answers are covered by retrieved passages. Inspired by Min et al. (2021). we devise an evaluation proxy, answer coverage, for measuring the quality of retrieved passages ambiguous QA tasks. We consider the retrieval as successful if the retrieved passages contain one of the answers to the target DQ. We calculate the proportion of success among DQs for a single AQ to check overall answer coverage.
A.3 Implementation Details
Table 4 compares retrieval systems in answer coverage (AC@k) of top-k passages. Bing search engine without reranker performs worst among baselines AC@ 10 and @30. However, with reranker, its performances are greatly enhanced, outperforming ColBERT baselines. When combin- ing two retrieval systems, it shows the best perfor- mances across all evaluation metrics; hence two results are complementary. It achieves 80.1 in AC@ 100 scores, which indicates the passage set has sufficient information properly explored.
A large portion of our implementation is based on the DSP library (Khattab et al., 2022). To dynami- cally find few-shot examples with the nearest neigh- bor search, we use pre-trained MiniLM (Wang et al.. 2020) to obtain hidden representations of questions and compute similarity scores with Faiss library (Johnson et al., 2019). We add 5-shot train- ing examples the prompt, following Amplayo al. (2023). was the optimal number in our pilot experiment.
For prompting LLM to perform RAC, we use top-5 relevant passages. To determine whether to

---

Qualitative Analysis
C.1 Prompt Format
We add format descriptions to our prompt follow- ing Khattab et al. (2022). Table displays the for- mat specifically designed to generate disambigua- tions for given question based external doc- uments. The format description is augmented to prompts both RAC and the answer generation. By using it, we encouarge LLM to comply with the format.
C.2 Question Clarification
Table 6 shows an example of RAC for the AQ. Retrieval systems provide the external knowl- edge. Leveraging it, LLM generates disambiguated question-answer pairs. In RAC, long-form answers are also generated follow the format but do not use them in the later steps.
In Table 7, we observe the cases where ToC en- counters unambiguous questions and fails to clarify them. It often asks different or irrelevant facts from them of original AQ.
C.3 Self Verification
Table 8, 9 show examples of self-verification prompt. We prompt LLM verify the current answer is factually coherent with AQ based on the relevant passage. generates 'True' or 'False' to determine whether the node would be discarded or not. We do not provide few-shot training examples or formats.
C.4 Answer Generation
Table 10 depicts an example of answer generation prompt. We use similar prompt to that of RAC except disambiguations are given inputs. It en- codes up to disambiguations and five relevant passages.

---

Follow the following format.
Context:
$[sources that may contain relevant content)
Question: S/ambiguous question to be disambiguated]
Disambiguations: $(the disambiguated pairs of questions and answers, each is separated by new ]
DQ i: S(())th disambiguated question that clarifies the ambiguous question]
DA i: $(short factoid answers separated by semi-colon (:) to (i)-th disambiguated question, often between 1 and 5 words)
S(a thorough, detailed answer that explains the multiple interpretations of the original question and includes the appropriate disambiguations, at least three sentences.)
Answer:
Table Format description for both RAC and the answer generation.
I will provide ambiguous questions that can have multiple answers based their different possible interpretations. Clarify the given question into several disambiguated questions and provide short factoid answers each question. Subsequently, summarize them into detailed long-form answer of at least three sentences. Here some examples.
Context:
[1] Fred and George Weasley Fred and George Weasley fictional characters in the "Harry Potter" book series written by J. K. Rowling. The characters are the identical twin brothers ... The twins were played by identical twin brothers James and Oliver Phelps in the film adaptations
[2] James and Oliver Phelps James Andrew Eric Phelps and Oliver Martyn John Phelps (born 25 February 1986) are English actors and identical twin brothers. They are known for playing Fred and George Weasley in the "Harry Potter" film series from 2001 to 2011
[5] Chris Rankin plays of "Bugsy Malone" and "The Lion, The Witch and The Wardrobe". His professional acting career began when he won the role Percy Weasley September 2000 after his character's s absence from "Harry Potter and the Goblet"
Question: Who played the weasley brothers in harry potter?
Disambiguations:
DQ 1: Who played the fictional characters Fred and George Weasley in the "Harry Potter" book series?
DA 1: James and Oliver Phelps
DQ 2: Who are the English actors and identical twin brothers known for playing
Fred and George Weasley the "Harry Potter" film series?
DA 2: James and Oliver Phelps
DQ 3: Who is the actor that plays Percy Weasley in the Harry Potter series?
DA 3: Chris Rankin
The Weasley brothers in the Harry Potter series were played by identical twin brothers James and other Oliver Phelps. The Phelps brothers are English actors who have appeared in films and TV shows together as a duo. Chris Rankin is the actor who played Percy Weasley in the Harry Potter series. Rankin is a British actor who has appeared in theatre, film, and TV.
Answer:
Table 6: Example prompt and output in RAC. Few-shot training examples and format descriptions omitted for simplicity.

---

I will provide ambiguous questions that can have multiple answers based on their different possible interpretations. Clarify the given question into several disambiguated questions and provide short factoid answers to each question. Subsequently, summarize them into a detailed long-form answer of at least three sentences. Here are some examples.
Follow the following format. Context: [1] 1991 Major League Baseball All-Star Game The 1991 Major League Baseball All-Star Game
[5] Venues of the 1996 Summer Olympics would serve as host to the Peach Bowl from
Question: When did Toronto host the MLB All-Star Game in 1991?
Disambiguations:
DQ 1: When was the 1991 Major League Baseball All-Star Game played?
DA 1: July 9, 1991
DQ 2: What was the outcome of the 1991 Major League Baseball All-Star Game?
DA 2: American League defeated the National League
Answer:
The 1991 Major League Baseball All-Star Game
I will provide ambiguous questions that can have multiple answers based their different possible interpretations. Clarify the given question into several disambiguated questions and provide short factoid answers each question. Subsequently, summarize them into detailed long- formform& answer of at least three sentences. Here
examples.
Follow the following format.
Context:
[1] Highest-paid NBA players by season Highest-paid NBA players by scason The highest-paid NBA players by season over...
[5] Hagd NBA players by season Highest-paid NBA players... ...
Question: Who was the highest-paid NBA player in the 2017-2018 season?
Disambiguations:
DQ 1: Who was the highest-paid NBA player in the 2017-2018 season by salary?
DA 1: LeBron James
DQ 2: Who was the highest-pail NBA player in the 2017-2018 season by total earnings?
DA 2: LeBron James
Answer: LeBron James was the highest-paid NBA player in the 2017-2018 season
Table 7: Failure case where the model encounters and clarifies unambiguous questions. Few-shot training examples and format descriptions are omitted for simplicity.

---

Correct Case
DQ: Who was selected to host the 2018 FIFA World Cup?
I will provide question, relevant context, and proposed answer to it. Identify whether the proposed answer could be correct answers not with only 'True' or 'False'
Context:
2018 and 2022 FIFA World Cup bids FIFA's headquarters in Zurich. Russia was chosen host the 2018 World Cup, and Qatar chosen host the 2022 World Cup. This made Russia the first Eastern European country host the World Cup, while Qatar would be the first Middle Eastern country host the World Cup. Blatter noted that the committee had decided to "go to new lands" and reflected desire "develop football" by bringing it to more countries. In each round a majority of twelve votes was needed. no bid received 12 votes in a round, the bid with the fewest votes
Question: Who is hosting the next world cup 2022?
Proposed Answer: Russia
False
Correct Case
DQ: Which player has won the most World Series in baseball?
I will provide a question, relevant context, and proposed answer to it. Identify whether the proposed answer could be correct answers or not with only 'True' or "False"
Context:
World Series ring on World Series rings. The New York Yankees Museum, located in Yankee Stadium, has an exhibit with replicas of all Yankees' World Series rings, including the pocket watch given after the 1923 World Series. Yogi Berra won the most World Series rings with 10, as player. Frankie Crosetti won 17 as a player and as coach. Yogi Berra Museum and Learning Center. World Series ring A World Series ring is an award given to Major League Baseball players who win the World Series. Since only one Commissioner's Trophy is awarded to the team, a World Series ring is
Question: Who's 's won the most world series in baseball?
Proposed Answer: Yogi Berra
True
Table Correct cases pruning method. Few-shot training examples formats are not augmented to prompt. Generated texts colored green.

---

Incorrect Case
DQ: Who the highest goalscorer in world football in single game?
I will provide a question, relevant context, and proposed answer to it. Identify whether the proposed answer could be correct answers or not with only 'True' or "False'
Context:
List of footballers with the most goals in a single game This is list of players with the most goals in football game. The list only includes players who have scored the most multiple goals in first class or fully professional matches for country or club. current world record for an international is held by Archie Thompson, who scored 13 goals against American Samoa in Australia's 's 31-0 victory during the 2002 FIFA World Cup qualification. David Zdrilic scored 8 goals.In November 2022, Shokhan Nooraldin Salihi scored 15 goals in the match of Al-Hilal against Sama in the 2022-23 Saudi Women's Premier League. In this match, AI-Hilal beat Sama 18-0.
Question: Who has the highest goals world football?
Proposed Answer: Archie Thompson
False
Incorrect Case 2
DQ: When was episode 113 of Dragon Ball Super released in the US?
I will provide a question, relevant context, and proposed answer to it. Identify whether the proposed answer could be correct answers or not with only 'True' or "False"
Context:
Dragon Ball Super I would be available in the United States in summer 2017. Bandai has also announced the updated "Dragon Ball Super Card Game" that starts with one starter deck, one special pack containing booster packs and promotional Vegeta card and booster box with 24 packs. It was released July 28, 2017. A line of six "Dragon Ball Super" Happy Meal toys were made available at Japanese McDonald's 's restaurants in May 2017. The average audience TV rating in Japan was 5.6% (Kanto region). The maximum audience rating 8.4% (Episode 47) and the lowest rating was 3.5% (Episodes 109-110).
Question: When is episode 113 of dragon ball super coming out?
Proposed Answer: November 5, 2017
False
Table 9: Incorrect cases self- velf-verification. Generated texts are colored green.

---

I will provide ambiguous questions that can have multiple answers based on their different possible interpretations. Clarify the given question into several disambiguated questions and provide short factoid answers to cach question. Subsequently, summarize them into a detailed long-form answer of at least three sentences. Here are some examples.
Context:
[1] Game of Thrones | Game of Thrones Game of Thrones is an American fantasy drama television series created by David Benioff and D. B. Weiss. and its seventh season ended on August 27, 2017.
The series will conclude with its eighth season
[2] Game of Thrones Game of Thrones is an American fantasy drama television series created by David Benioff and for HBO. It is an adaptation of "A Song of Ice and Fire".... Set on the fictional continents of Westeros and Essos, "Game of Thrones" has a large ensemble cast
[5] A Game of Thrones (comics) A Game of Thrones (comics) A Game of Thrones is the comic book adaptation of George R. Martin's 's fantasy novel "A Game of Thrones", ... It is intended follow the story and atmosphere of the novel closely, at rate of about a page of for each page of text, and
Question: What kind of series game of thrones?
Disambiguations:
DQ I: What the genre of the American television series Game of Thrones?
DA 1: fantasy drama
DQ 2: What is the genre of the comic book series A Game of Thrones?
DA 2: fantasy
DQ 10: What is the genre of the board game A Game of Thrones?
DA 10: strategy
A Answer: There are multiple works that share the title Game of Thrones. The first is a television series that is a fantasy drama, the second is comic book series that is fantasy, the third is a book series that is fantasy, and the fourth is board game that is strategy game.
Table 10: Example prompt for the answer generation process. Generated texts colored green.


Reviews:
[{"review_id": "6qSlf9csv5", "reviewer": "Reviewer_T734", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s valuable comments. In particular, we are delighted that the reviewer agrees the proposed method is straightforward and reasonable for the target task. We are also happy that the reviewer assesses the paper as well-written and confirms our experiments demonstrate the effectiveness of the method and show the helpful ablation studies.\n\n***Response to Reason to Reject***\n\n1. *The paper only presents experiments with only one LLM model (GPT-3) on one dataset (ASQA).*\n\nAlthough we could not provide enough discussion due to the short page limit, the key idea of ToC could be potentially generalized to the other tasks and model architectures. It has a model-agnostic structure that could effectively explore diverse paths of recursive reasoning, which would be helpful for tasks that require multiple reasoning, such as multi-hop QA. \n\n  However, in this work, we focus on handling the ambiguity in open-domain QA by leveraging strong reasoning and generation ability of recent LLMs.  \n- Dataset: To the best of our knowledge, ASQA [1] is the most recent and only benchmark that evaluates long-form responses to the ambiguous questions. \n- Model: For comparing our closed-book baseline, we chose the same architecture (i.e., GPT-3 text-davinci-002) with the previous work [2], except for the unreleased model, PaLM 1 540B. Fine-tuning the smaller scale of LMs (e.g., T5, LLaMa, Vicuna, and MPT) would be less effective due to the small training set as the baselines show.\n\nWe would continue to investigate the generalizability of ToC to diverse tasks, datasets, and LLMs for future work.\n\n[1] Stelmakh et al., EMNLP 2022, \"ASQA: Factoid questions meet long-form answers.\"\\\n[2] Amplayo et al., ACL 2023, “Query Refinement Prompts for Closed-Book Long-Form QA”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written. The novel idea of using a recursive tree structure to methodically explore ambiguities seems like a simple and reasonable method for long-form ambiguous QA. The paper also does important ablations showing the contributions of different components of their framework. different Strong Results: TOC Outperforms fully-supervised models with only 5 examples, and achieves new SOTA results on the ASQA benchmark. \n", "reasons_to_reject": "The paper only presents experiments with only one LLM model (GPT-3) on one dataset (ASQA).  ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a framework called \"Tree of Clarifications\" (TOC) for answering ambiguous open-domain questions where they recursively construct a tree to explore different interpretations (disambiguations) of the ambiguous question. TOC uses retrieval to find relevant passages and augment them to prompts for LLMs. It prunes unhelpful or irrelevant disambiguations using a self-verification method. The experiments on the ASQA benchmark show TOC outperforms previous baselines and fully supervised models trained on the whole dataset. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9dr0xRMh9o", "reviewer": "Reviewer_UCkq", "summary": "", "questions": "- Do you think the improvements you get from RAC + TS come from adding the disambiguation tree, or from the fact that you are performing even more retrieval than with RAC alone? The biggest jump in the ablation seems to be the implicit RAG from RAC.\n- Why not consider a diversity-based reranking of the retrieved passages, since your goal is to handle ambiguity?\n- Why the different metrics between Table 2 and Table 3? \n- Would this framework apply to other recursive question answering problems, such as multi-hop qa?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s detailed and helpful feedback. In particular, we are delighted that the reviewer agrees the proposed method is straightforward and sensible for the target task. We are also happy that the reviewer assesses the paper as clear and detailed and confirms our experiments demonstrate the effectiveness of the method and show helpful ablation studies.\n\n***Response to Reasons to Reject***\n\n1. *Limited scope. Prompting and prompt chaining is a powerful framework, but mostly interesting when it enables general-purpose problem solving. This paper is limited to the ambiguous question problem.*\n\nAlthough we could not provide enough discussion due to the short page limit, the key idea of ToC could be potentially generalized to the other tasks and model architectures. It has a model-agnostic structure that could effectively explore diverse paths of recursive reasoning, which would be helpful for tasks that require multiple reasoning, such as multi-hop QA. \\\nIn this work, we focus on one of the most common and challenging issues, the ambiguity in open-domain QA. Our work holds significance in that it tackles the crucial issue where the prompting methods for general-purpose such as chain-of-thoughts or self-consistency [1] were ineffective or not applicable.\n\n2. *Source of improvements. Unclear whether the improvements can be attributed to the tree structure, or to the inclusion of more information via retrieval. The paper should include a comparison with retrieval-augmented methods.*\n\nIn Table 1, we demonstrate how much retrieval contributes to the improvement by isolating RAC and provide the comparison with other retrieval-augmented methods, “T5-large w/ JPR” from the previous work [3]. In our pilot experiment, we found that simply increasing the number of retrieved passages does not improve the performance, and then fixed the number of passages as 5. On the other hand, tree structure further enhances the retrieval diversity and relevance, enabling context-specific search for each interpretation. Unlike prior studies' efforts to improve recall scores in retrieval tasks, TS is designed especially to broaden the coverage of retrieved passages.\n\n3. *Missing a key analysis*\n  - *Whether multiple levels of disambiguation are needed*\n\nThe tree structure (TS) is motivated by the previous work [4] that reports a single ambiguous question may pose multiple types of ambiguity. Moreover, improvement from the pruning method could also implicate the effectiveness of using multiple levels. TS with pruning can explore the deeper level of the tree while the TS-only baseline reaches the maximum node limit earlier in the shallower level. In Table 1, TS with pruning shows better performance in F1 and DR scores than the TS-only baseline.\n\n  - *how often multiple levels of disambiguation are necessary*\n\nThanks for the great suggestion but it would be not trivial to precisely measure it with an analysis. We will try to handle it by providing statistics about the number of nodes or depth in resulting trees and reporting them in our final draft.\n\n  - *LLMs ability to faithfully integrate multiple branches of the disambiguation tree*\n\nIn Table 3, we provide the accuracy of the disambiguation trees. With our pruning technique, intermediate trees are enhanced, leading to the improvement of the final answer. It would be one of the pieces of evidence showing that the answer generation module faithfully refers to the disambiguation trees. When removing the intermediate tree from the input prompt, its performance largely degrades.\n\n  - *the extent to which retrieval helps disambiguate (as opposed to the fact that retrieval clearly helps answer the questions). The most meaningful contribution is the notion of retrieval-augmented clarification, but I do not think the experiments isolate/evaluate RAC effectively.*\n\nIn Table 1, we demonstrate how much retrieval contributes to the improvement by isolating RAC from TS. RAC largely advances the model performance and TS further enhances it.\n\n4. *Missing latency/efficiency measurements. Using a recursive LLM framework comes with computational costs; this should be measured by the authors. It's difficult when using an API, but even a sense of the typical numbers of API calls would be helpful.*\n\nAlthough ToC requires the multiple generation process by LLM, its maximum number is less than 10 per question. Exploration of the tree ends when it obtains 10 valid nodes and the clarification process generates 2-5 disambiguations for each question. Failing to expand three times in a row also terminates the exploration. Pruning steps consume a smaller amount of tokens since they encode a single passage without few-shot exemplars. Compared to the existing ensemble methods such as self-consistency [1] which even cannot be directly adopted to the generative task, ToC achieves the state-of-the-art performance with a comparable number of LLM calls.\\\nRegarding the concern about complexity, we will add these details to our final draft.\n\n5. *It seems like no details for standard PaLM and GPT prompting were included. The prompt examples seem to all be for the RAC+TS prompting.*\n\nIn Table 1, the performances for standard PaLM and GPT without retrieval are reported by the previous work [2] so please refer to it for the prompts. For our closed-book setup in Table 2, we use the same prompt as that in Table 6 except “Context” and “Disambiguations” sections.\n\n***Answers to Questions***\n- Q1: *Do you think the improvements you get from RAC + TS come from adding the disambiguation tree, or from the fact that you are performing even more retrieval than with RAC alone? The biggest jump in the ablation seems to be the implicit RAG from RAC.*\n\nAs mentioned in Response 2 (*Source of improvements*), simply increasing the number of passages is ineffective. It could introduce noisy and redundant distractors for the limited input length of LLM, as reported by recent works [5,6]. Repeatedly performing retrieval with similar queries would not expand the coverage of the passage set. On the other hand, tree structure enhances the diversity and coverage of retrieved passages. It would be the key advantage of ToC and where the most improvement comes from. \n\n- Q2: *Why not consider a diversity-based reranking of the retrieved passages, since your goal is to handle ambiguity?*\n\nIn each node, passages are reranked regarding each DQ, which enhances the diversity of passages. Thanks for the interesting idea. Adopting diversity-specific techniques might improve the performance.\n\n- Q3: *Why the different metrics between Table 2 and Table 3?*\n\nWe use D-F1 to evaluate the long-form answers to the ambiguous questions. In Table 3, we show the accuracy of intermediate pairs of disambiguated questions and their short answers. “Answer-F1” is used to validate how accurate the short answers are. Appendix A.2 provides the details about it. We will add a footnote for it.\n\n- Q4: *Would this framework apply to other recursive question answering problems, such as multi-hop qa?*\n\nFor handling multi-hop QA tasks, current studies on the chain of reasoning are introduced and they could be viewed as a special case of the tree structure with depth-first search. Tree structure might be applied to compare and validate multiple reasoning chains. Thanks for the interesting idea.\n\n[1] Wei et al., 2022, “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”\\\n[2] Amplayo et al., ACL 2023, “Query Refinement Prompts for Closed-Book Long-Form QA”\\\n[3] Stelmakh et al., EMNLP 2022, \"Asqa: Factoid questions meet long-form answers.\"\\\n[4] Min et al., EMNLP 2020, “AmbigQA: Answering Ambiguous Open-domain Questions”\\\n[5] Shi et al., 2023, “REPLUG: Retrieval-Augmented Black-Box Language Models”\\\n[6] BehnamGahder et al., 2023, “Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Authors introduce a simple and sensible framework for disambiguating questions, where both question answering and question diambiguation are grounded with retrieved information.\n- The paper achieves significant improvements over standard LLMs.\n- Provides clear details on the prompts used for the RAC+TS framework.\n- Includes useful ablations\n- The paper contributes evidence that LLMs are effective at decomposing problems.", "reasons_to_reject": "- Limited scope. Prompting and prompt chaining is a powerful framework, but mostly interesting when it enables general-purpose problem solving. This paper is limited to the ambiguous question problem.\n- Source of improvements. Unclear whether the improvements can be attributed to the tree structure, or to the inclusion of more information via retrieval. The paper should include a comparison with retrieval-augmented methods.\n- Missing a key analysis. While many of the experiments and comparisons are helpful to understand how ToC compares to existing approaches to ASQA, I'd like to see more analysis of the separate components of ToC. There's noo analysis on whether multiple levels of disambiguation are needed, how often multiple levels of disambiguation are necessary, or the LLMs ability to faithfully integrate multiple branches of the disambiguation tree, or the extent to which retrieval helps disambiguate (as opposed to the fact that retrieval clearly helps answer the questions). The most meaningful contribution is the notion of retrieval-augmented clarification, but I do not think the experiments isolate/evaluate RAC effectively.\n- Missing latency/efficiency measurements. Using a recursive LLM framework comes with computational costs; this should be measured by the authors. It's difficult when using an API, but even a sense of the typical numbers of API calls would be helpful.\n- It seems like no details for standard PaLM and GPT prompting were included. The prompt examples seem to all be for the RAC+TS prompting.", "questions_to_authors": "- Do you think the improvements you get from RAC + TS come from adding the disambiguation tree, or from the fact that you are performing even more retrieval than with RAC alone? The biggest jump in the ablation seems to be the implicit RAG from RAC.\n- Why not consider a diversity-based reranking of the retrieved passages, since your goal is to handle ambiguity?\n- Why the different metrics between Table 2 and Table 3? \n- Would this framework apply to other recursive question answering problems, such as multi-hop qa?\n", "paper_topic_and_main_contributions": "The authors introduce a recursive approach for clarifying ambiguous questions. Given an ambiguous question, relevant information is retrieved and an LLM is prompted to generate possible disambiguated questions. The process is repeated for the disambiguated questions. Answers are then combined to form a passage that answers the question, addressing ambiguity. The method demonstrates improvements over pure LLMs for ambiguous question answering.\n\nOverall I think this is a very interesting idea, limited mainly by the lack of analysis on why ToC is effective.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dcxbbevRbZ", "reviewer": "Reviewer_tfvE", "summary": "", "questions": "How do you ensure all interpretations are captured?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your valuable comments. Specifically, we are delighted that you consider our method as a reasonable and novel solution for the challenge of ambiguity in open-domain QA. We are also pleased that you mentioned our experimental results strongly show the effectiveness of our method.\n\n***Response to Reason to Reject***\n\n1. *The only limitation of this work is that it requires a tree of clarification process which makes the proposed solution complicated and it is a challenge to deploy such approach to production.*\n\nAlthough ToC requires multiple generation processes by LLM, its maximum number is less than 10 per question. Exploration of the tree ends when it obtains 10 valid nodes and the clarification process generates 2-5 disambiguations for each question. Failing to expand three times in a row also terminates the exploration. Pruning steps consume a smaller amount of tokens since they only encode a single passage without few-shot exemplars. \n\nCompared to the existing ensemble methods such as self-consistency [1] which even cannot be directly adopted to the generative task, ToC achieves the state-of-the-art performance with a comparable number of LLM calls. It also outperforms the vanilla baseline with a single LLM call, significantly. For production, one could find the optimal way in such a trade-off between the clarification quality and computational cost.\n\nRegarding the concern about complexity, we will add these details to our final draft.\n\n[1] Wei et al., 2022, “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a novel framework called \"Tree of Clarifications\" (ToC) to address the challenge of ambiguity in open-domain question answering.\n2. The experimental results demonstrate that ToC outperforms existing baselines on the ASQA benchmark in a few-shot setup across various metrics.\n3. The proposed approach guides LLMs to explore diverse paths of clarification of the given AQ in a tree structure with the ability to prune unhelpful DQs. It is the first to combine retrieval systems with LLM for generating long-form answers to AQs.", "reasons_to_reject": "The only limitation of this work is that it requires a tree of clarification process which makes the proposed solution complicated and it is a challenge to deploy such approach to production.", "questions_to_authors": "How do you ensure all interpretations are captured?", "paper_topic_and_main_contributions": "The paper introduces a novel framework called \"Tree of Clarifications\" (ToC) to address the issue of ambiguity in open-domain question answering. Ambiguous questions often have multiple interpretations, making it challenging to provide a comprehensive answer. ToC constructs a tree of disambiguations for the ambiguous question using few-shot prompting with external knowledge and uses it to generate a long-form answer that covers all possible interpretations. Experimental results show that ToC outperforms existing baselines on ASQA.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nQcYV69cVY", "reviewer": "Reviewer_GVet", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s feedback. In particular, we are delighted that the reviewer considers our method as novel and convincing for the target task. We are also happy that the reviewer assesses the experimental results as robust.\n\n***Response to Reason to Reject***\n\n1. *The result is limited to a single dataset, ASQA. This work would be more convincing if it includes results on other datasets, to show that ToC is generalizable to other settings.*\n\nAlthough we could not provide enough discussion due to the short page limit, the key idea of ToC could be potentially generalized to the other tasks and model architectures. It has a model-agnostic structure that could effectively explore diverse paths of recursive reasoning, which would be helpful for tasks that require multiple reasoning, such as multi-hop QA. \nHowever, in this work, we focus on handling the ambiguity in open-domain QA by leveraging strong reasoning and the generation ability of recent LLM.  To the best of our knowledge, ASQA is the most recent and only benchmark that could evaluate long-form responses to the ambiguous question.\nWe would continue to investigate the generalizability of ToC to diverse tasks, datasets, and LLMs for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method, retrieval-augmented tree structure for clarification, is novel and convincing. Also, the reported result is solid.", "reasons_to_reject": "The result is limited to a single dataset, ASQA. This work would be more convincing if it includes results on other datasets, to show that ToC is generalizable to other settings.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces Tree of Clarification (ToC), which is a retrieval-based method that uses tree structure to clarify ambiguous questions. ToC method helps LLM to disambiguate questions, thereby achieving state-of-the-art result in ASQA dataset.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to tackle ambiguous open-domain questions by iteratively building a tree of clarifications, which holds interpretations for the original question. Then, the long-form answer is generated conditioned on these interpretations.

Reviewers all like the proposed approach for the problem, and are impressed with the improvements over the baselines.
Some reviewers are favorable towards the ablations in the paper but reviewer UCkq mentions that analysis is missing w.r.t the different components of the system.
Reviewers also mention that the latency slowdown of using the proposed approach is not analyzed, and that the authors only experiment with one dataset.

Improvement: I noted that you use SentenceBERT for ranking answer sentences. The specific research area is called Answer Sentence Selection, and you can find the state of the art below:

Siddhant Garg, Thuy Vu, Alessandro Moschitti. TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection. AAAI 2020.

Ivano Lauriola and Alessandro Moschitti: "Answer sentence selection using local and global context in transformer models", ECIR 2021.
Decision: Accept-Main