Asking Clarification Questions to Handle Ambiguity in Open-Domain QA
Authors: Dongryeol Lee
Segwang Kim
Minwoo Lee
Hwanhee Lee
Joonsuk Park
Sang-Woo Lee
Kyomin Jung
Venue: EMNLP 2023 Findings (2023)
Keywords: Clarification Question
Open-domain Question Answering

Abstract: Ambiguous questions persist in open-domain question answering, because formulating a precise question with a unique answer is often challenging. Previous works have tackled this issue by asking disambiguated questions for all possible interpretations of the ambiguous question. Instead, we propose to ask a clarification question, where the user's response will help identify the interpretation that best aligns with the user's intention. We first present CAmbigNQ, a dataset consisting of 5,653 ambiguous questions, each with relevant passages, possible answers, and a clarification question. The clarification questions were efficiently created by generating them using InstructGPT and manually revising them as necessary. We then define a pipeline of three tasks---(1) ambiguity detection, (2) clarification question generation, and (3) clarification-based QA. In the process, we adopt or design appropriate evaluation metrics to facilitate sound research. Lastly, we achieve F1 of 61.3, 25.1, and 40.5 on the three tasks, demonstrating the need for further improvements while providing competitive baselines for future work.

Asking Clarification Questions to Handle Ambiguity in Open-Domain QA
Dongryeol Lee
Segwang Kim 2-
Minwoo Lee'
Hwanhee Lee
Joonsuk Park456
Sang-Woo Lee
Kyomin Jung"t
'Dept. of ECE, Seoul National University, ?Samsung Electronics Mobile eXperience, CChun--Ang University, "NAVER AI Lab, SNAVER Cloud,
"University of Richmond, 'KAIST AI
(dr1123, ksk5693, minwoolee, kjung)êsnu.. ac kr, Srranheelee@cann ac. kr parkējoonsuk. org, sang. woo. wwweneeerccrrppccmmmmm com
Abstract
Ambiguous questions persist in open-domain question answering, because formulating pre- cise question with unique answer is often chal- lenging. Previous works have tackled this issue by generating and answering disambiguated questions for all possible interpretations of the ambiguous question. Instead, we propose to ask a clarification question, where the user's s response will help identify the interpretation that best aligns with the user's intention. We first present C CAMBIGNO, dataset consisting of 5,653 ambiguous questions, each with rel- evant passages, possible answers, and clari- fication question. The clarification questions were efficiently created by generating them us- ing InstructGPT and manually revising them as necessary. We then define a pipeline of three tasks-(1) ambiguity detection, (2) clarification question generation, and (3) clarification -based QA. the process, we adopt or design appro- priate evaluation metrics to facilitate sound re- search. Lastly, we achieve Fl of 61.3, 25.1, and 40.5 on the three tasks, demonstrating the need for further improvements while providing competitive baselines for future work.
Who pleyed
Who played
Tom Riddle Harry Potter?
Tom Ridcle Harry Potter?
Did you mean Who played young Tom Riddic in Harry Poher and the Chamber of Secrete? Aromas Chrician
ch version: young
chid scries teonager
scries G 7
Did you mesn.... Who played the child version of "omm Ridd in Haryy Potter and the Hart Blood Prinos?
Answer: Hera Fiennes-Tiit
Answer: Horo Flonnos- Titnn
Did you tean
Who playoo the teonage vension of tom ficidd Harry Porter and
Princo?
(a)
(b)
Figure 1: Two possible approaches to handling ambigu- ous questions (AQs) open-domain question answer- ing (ODQA): (a) presenting disambiguated questions (DQs) with answers (following Min et al. (2020)), and (b) asking clarification question (CQ) and displaying an answer based on the user's 's response to the CQ (ours).
question (DQ; disambiguated variation of the given AQ) for each possible interpretation (Min et al., 2020; Gao et al., 2021; Stelmakh et al., 2022). While such DQ-based approaches are an impor- tant step toward resolving ambiguities in ODQA, imagine how it would be deployed in real life; with- out knowing the user's intention, the QA system would have to list all possible answers to the user, as shown in Figure I(a). This is not suitable in most real-world scenarios where QA systems com- municate with their users through speech or small- screen devices (Zamani et al., 2020b; Croft, 2019; Culpepper et al., 2018).
1 Introduction
In open-domain question answering (ODQA), ques- tions can often be interpreted in several ways, each with distinct answer (Min et al., 2020; Zhang and Choi, 2021). For example, consider the question at the top of Figure 1. Though the question seems un- ambiguous, "young Tom Riddle" can mean "young version in series 2n, "child version in series 6", or "teenager version in series 6" Such ambiguity needs to be resolved to correctly find the answer sought by the user.
Instead, we propose to prompt the user with a clarification question (CQ), as shown in Figure 1(b). More specifically, given AQ, the goal is to ask CQ consisting of the possible interpretations options (e.g. "teenager in series 6")) along with a category summarizing the options (e.g. "version"). Ideally, the user's response to the CQ would help identify the interpretation that best aligns with the
Previous studies propose to handle ambiguous questions (AQs) by generating disambiguated
Equal contribution.
Corresponding authors. Work done while he was in Scoul National University.

---

user's intention, and the corresponding answer can be presented to the user. This CQ-based approach is not only applicable the aforementioned real- world scenarios, but also shown to be preferred by users according to our preference test. This is also consistent with the finding that asking CQs can improve user experience with "limited bandwidth" interfaces (Zamani et al., 2020a).
that though they were shown be helpful for generating DQs. predicted answers for AQ do not help improve the CQ-based approuch overall (Section 6).
2 Related Work
Clarification Question Datasets To resolve question ambiguity, CQ datasets have been released in various domains. In the information-sseeking do- main, CQ datasets for conversation (Aliannejadi et al., 2019, 2020; Guo et al., 2021; Wu et al., 2022) or web search (Zamani et al., 2020b) have been collected from crowdsourcing or real users' follow- up search queries. In the question-answert main, datasets that focus on specific topics (Rao and Daumé III. 2018; Braslavski et al., 2017; Sacidi et al., 2018) or knowledge- weaseddeebaseeeeeeeee (Xu et al., 2019) has been proposed. To the best of our knowledge. are the first release CQ dataset for ODQA.
To support research on CQ-based approaches to handle AQs in ODQA, we present Clarifying Am- biguous Natural Questions (CAMBIGNQ). CAM- BIGNQ is a dataset consisting of 5,653 AQs from AMBIGNQ (Min et al., 2020), each accompanied by relevant passages, possible answers, and newly created CQ. The CQs were first generated using In- structGPT (Ouyang al., 2022) through in-context few-shot learning, then manually vetted and edited as necessary by human editors. Such human- machine collaboration for constructing corpora has been shown to significantly reduce the time and the cost from fully manual approaches (Wang et al., 2021; Wu et al., 2021).
do-
Dataset Construction Leveraging LLMs Manu- ally constructing datasets from scratch is laborious and costly, which can be prohibitive depending the nature of the dataset. Also, access to real users' data is strictly limited to a certain community. To mitigate these issues, approaches leveraging LLMs to construct datasets have recently been used in var- ious domains such as dialogue (Bae et al., 2022), domain- -adaainnda (Dai et al., 2022), and in gen- eral (Ding et al., 2022). However, such an approach has not been used to construct CQ datasets, except for ours. We used InstructGPT (Ouyang et al., 2022) generate CQs through in-context few-shot learning, and then manually vetted and edited them as necessary to construct our dataset.
We also define a pipeline of three tasks to handle AQs in ODQA (1) ambiguity detection, (2) clari- fication question generation, and (3) clarification- based QA. In the process, we adopt or design ap- propriate evaluation metrics to facilitate sound re- search. The experiments show that though they were shown to be helpful for generating DQs, pre- dicted answers for AQ do not help improve the CQ-based approach overall. Lastly, we achieve Fl of 61.3, 25.1, and 40.5 on the three tasks, demon- strating the need for further improvements while providing competitive baselines for future work.' Our main contributions are threefold:
We propose to use CQs as a practical means to handle AQs in ODQA. Consistent with the findings by Zamani et al. (2020a), our human preference test shows that the use of CQ is preferred over that of DQs (Section 5).
Clarification Question Evaluation There are several options for evaluating the quality of CQs, First leveraging widely-used automatic text eval- uation metrics, such as BLEU (Papineni et al.. 2002) or ROUGE (Lin, 2004). However, due to the poor correlations berween such scores and hu- man evaluation, Zamani et al. (2020b) strongly discourages the use of such metrics for evaluation. Second is human evaluation. While it typically provides reliable estimate of how people would think of the given CQs, it can be time-consuming
We present CAMBIGNO, dataset support CQ-based handling of AQs in ODQA. was built efficiently by leveraging a well-curated resource, AMBIGNQ, as well as the power of InstructGPT and human editors (Section 4).
We define pipeline of tasks and appropri- ate evaluation metrics for handling AQS in ODQA (Section 3). The experiments show
"uu et al. (2019) presents a CQ dataset for KBQA that is also open-domain, but the setting is much more restricted than ODQA in that the goal is to find the appropriate knowleds ledge- base entry between exactly two entries about the same entity. Once an entry is determined, answering the question involves a simple table lookup, rather than full-on QA
'The data and code will he available at https://github

---

Ainbiguaus
Ambiguity
Relevent Passagea
?
?
?
Relavart Passages
Unambiguous Question
Ambiguous Question
Clarificaion Question Generator
Clarffiation Question
Answer
DA Model
Relevani
Response
QA Model
Answer
Passages
Clrification Question
Figure 2: Overview of proposed approach to ODQA. Given question, it is first checked for ambiguity ((a) ambiguity detection). If it is not ambiguous, is processed a normal QA setup ((b) QA; outside the scope of this work). Otherwise, an extra step of eliciting a response to clarification question precedes QA ((c) CQ generation clarification-based QA). Yellow blocks represent the user input, and blue blocks, the system output.
and costly. As a third option, evaluation methods using external neural models have recently been introduced (Rei et al., 2020; Mchri and Eskenazi, 2020; Lee et al., 2021). This approach improves the first, without the burden of the second. In this work, we design evaluation methods that suit our tasks, leveraging external neural models to provide a more comprehensive and accurate assessment.
"Which [eategory]: [option1]] [optiong]. [optional,lollionn)). or [optionn]]"
Here, [calegory] is category to which all op- tions belong, such as "version" in Figure 1. If the options can not be grouped into a single cate- gory, "one" is used as placeholder for the category. Also, where suitable, additional words like preposi- tions can precede "Which," e.g. "In which context". There should be [option,] for each possible in- terpretation of the AQ. Also, only the options are considered valid responses to the given CQ.
3 Task Overview
We propose to handle AQS in ODQA by asking CQs shown in Figure 2. There are three tasks: (1) ambiguity detection, (2) clarification question generation, and (3) clarification-bass based QA.
Evaluation We evaluate the quality of the gen- erated CQ in two levels granularity. First, we compare the generated CQ against the refer- ence CQ using the standard BLEU-4 metric and BERTSCORE (Zhang et al., 2019).
3.1
Task 1: Ambiguity Detection
Task Given a question and relevant passages, the goal is to determine whether the question is am- biguous not, shown in Figure 2(a). A question is considered ambiguous if it can be interpreted in multiple ways, each of which associated with unique answer, resulting in multiple possible swers. question unambiguous if can only be interpreted one way, thus one possible answer.
Second, we evaluate the category and the op- tions separately for more fine-grained evaluation. For the category, exact match (EM) and BLEU-1 are computed since the category is typically very short. For the options, partial match evaluation metric by Li al. (2022), whose goal to measure the similarity between a predicted set and a reference set of strings. Since the exact alignment of the strings between the sets is unknown, it measures the similarity-based on the longest common substring (LCS)-betwedt all pairs across the sets and keeps the highest score. Here, multiple strings from the predicted set be aligned with the same string in the reference set. In this work, we impose constraint that limits the alignment of a reference option to most one pre- dicted option, since each option should represent a unique interpretation the AQ. Thus, we find the
adopt and adjust the
For this binary classification task, use standard metrics, such as accuracy, recall, pre- cision, and F1.
Evaluation
3.2
Task 2: Clarification Question Generation
Task Given AQ and relevant passages, the goal is to generate CQ such that valid response the CQ associated with exactly one of the multiple answers to AQ.
CQ is typically formatted as follows:

---

optimal assignment that maximizes the similarity score using the Hungarian algorithm (Kuhn, 1955)
Split
CQ
Category
Options
#
Len.
Len.
Avg. #
Len.
and compute precision, recall, and Fl as follows:
Train
4,699
13.6
2.8
2.9
3.3
Validation
461
15.9
2.5
3.3
3,8
Test
493
17.8
2.8
3.4
4.1
maxprec. Eper, (in(p,((1))
(1)
Table I: Statistics of CAMBIGNO. Each clarification
max: fe ErER, sim(r, (r)),
(2)
question (CQ) consists of one category and multiple options. The length is reported in the number of words.
E, max, pree Σ.
Σ, max Ra
4 The CAMBIGNQ Dataset
prec
rec
4.1 Dataset Construction
where P; and R2 is the set of predicted and ref- erence options for i-th sample, sim(-) ) is the LCS- based similarity measure, and P. - R. is the optimal one-to-one mapping computed from the Hungarian algorithm. Fl is harmonic mean of precision and recall, as usual.
We present Clarifying Ambiguous Natural Ques- tions (CAMBIGNQ), a new dataset consisting of 5,653 AQ, each with relevant passages, possible answers, and a CQ. CAMBIGNQ was constructed from the AQs in AMBIGNQ (Min et al., 2020), which provides each AQ with relevant passages, as well the DQ and answer pairs reflecting the possible interpretations and respective answers of the AQ. To build CAMBIGNQ. we replaced each set of DQs with CQ. In other words, the CQ is an integrated version the set of DQs. Representing each DQ as single phrase option can be cumber- some to do manually. Thus, we collect high-quality of CQs by leveraging InstructGPT, using two- step framework: Generation via InstructGPT and Manual Inspection and Revision.
Please refer to Appendix A.I I for more details.
3.3 Task Clarification- QA
Task Given an AQ, relevant passages, and a CQ, the goal is generate a unique answer for every valid answer to the CQ-i.e., an option-whie is associated with an interpretation of the AQ Each answer is generated by calling a QA model on an AQ revised by CQ, which is the concatenation of AQ, category, and single option:
*QQ, Which [category): [optiony]".
Note, because each AQ revised by CQ is unique interpretation of the AQ with distinct answer, the relevant passages first need to be reranked before generating an answer.
Generation via InstructGPT To take advantage of the few-shot learning capability of InstructGPT, first manually annotate a small number of CQs for AQs. These edited CQs are then used as "few- shot" examples along with brief instructions and both the AQ and corresponding DQs. We sampled six examples considering the diversity of category and nunber of options. The final prompt is in the form of a concatenation of the six examples, instructions, target AQ, and target DQs in the fol- lowing form:
The procedure is similar to that of option evaluation for CQ generation, in that it uses the partial match method with the Hungarian algo- rithm to determine the optimal alignment between predicted and reference answers.
Evaluation
The only difference is that max;" prec and max for each aligned pair of predicted and reference answers are computed differently, because in QA, the correct answer may be expressed in multiple ways, e.g.. "Michael Jordan", "MJ", and "Jordan". Thus, a predicted answer is compared with all vari- ations of the same answer, and the max score is used for that pair. Then, precision, recall, and Fl are calculated as before, with the newly computed max,roe and max
"instruction, AQ_I, I, DQs_I, CQ_,, instruc- tion, AQ_6, DQs_6, CQ_ 6, instruction, -arget DQs "Qrgee"
Manual Inspection and Revision The recruited annotators were asked to read instructions and re- vise 25 CQs accordingly as qualification test. Then the editors who passed the qualification test were asked to examine, and revise as necessary, the CQs generated by InstructGPT.
Please refer to Appendix A.2 for more details.
They were asked to follow the following pro- tocol to ensure the quality of the final CQs: First, check whether the AQ had at least two distinct inter-
WW utilized cross encoder Mv2 model (Wang et al., 2020) fine-tuned on MSMARCO, https://hugh /huggingg face. co/ wwcroosssncooeo /ms- mar-o-aar-----oo marco-MiniLMe MiniLM--------------- L-

---

Target
Question
Example
Category
AQ
Who is Catch Me If You Can based on?
Only
GPT CQ
Which one: the 2002 film, hook, or the musical?
(12.9%)
Edited CQ
Which version: the 2002 film, the book, or the musical?
Options
AQ
When did the 420 note come out?
Only
GPT CQ
Which scries: F, or E?
(19.7%)
Edited CQ
Which series: F, variant, or
Category
AQ
Who plays Will on The Bold and Beautiful?
& Options
GPT CQ
Which time period: first, replacement, or 2013?
(31.4%)
Edited CQ
Which one: first actor, actor that replaces the wardens, actor that hegan playing 2013?
Whole
AQ
Who
all-time passing leader in the NFL?
Question
GPT CQ
Does the leader include regular season stats, or stats from the playoffs as well? In which context: in the regular seasons, including the playofts as well?
(7.8%)
Edited CQ
AQ
Who the current chairman of African Union commission?? Which chairman: 4th, 3rd, or 2nd?
None
GPT CQ
(26.7%)
Edited CQ
Which chairman: 4th, 3rd, or 2nd?
Table 2: Examples of manual revisions made clarification questions (CQs) generated by InstructGPT for ambiguous questions (AQs). The human editors were provided with disambiguated questions (DQs) for reference. Red and blue words represent the words before and after revision, respectively. The remaining 1.5% was marked as "unambiguous" by the editors, meaning only one interpretation, and thus one answer, exists for the given question. These were excluded from our dataset.
pretations and corresponding DQs; Second, check whether the CQ generated by InstructGPT is in the correct format (See Section 3.2); Third, check whether each option accurately represents its cor- responding DQ and the category is correct term describing the set of options. The editors had three actions to choose from-they could either: mark the AQ as not ambiguous, i.e. there only a sin- gle interpretation and answer (occurred in 1 1.5% of cases), revise the CQ (occurred in 71.8% of cases), or leave the CQ as is (occurred in 26.7% of cases). (See Table 2 for example revisions.) The high re- vision rate suggests that the few-shot generation with InstructGPT is far from perfect, and manual editing was necessary.
on which components of the CQs generated by InstructGPT were revised by human editors. Of the entire dataset, about 8% were due to the in- valid format of the CQs. This means that although InstructGPT was provided with six example CQs in the prompt, it is not always enough to enforce the format. Additionally, one common type of re- vision made the category was converted to or from "one", meaning InstructGPT often tried group ungroupable options or chose not to group options that can be grouped into a single category. A common revision made to the options was to split what InstructGPT generated as single option. Errors like this also lead to mismatch between the number of DQs, interpretations, and that of the options. Overall, there seems be room for further prompt engineering to minimize errors, but we believe manual revisions are a necessary com- ponent for constructing high-quality datasets as of yet. Please refer to Appendix B for more details.
For inter-annotator agreement, we use the valida- tion set which was annotated by two annotators, following Min et al. (2020). The kappa coeffi- cient (Cohen, 1960) is 0.623, which can be consid- ered a "substantial agreement." (McHugh, 2012).
4.2 Dataset Analysis
Experiment 1: CQ DQ Preference
The entire dataset consists of 5,653 data points, as shown in Table 1. The training set was sourced from that of AMBIGNQ, while the validation and test sets were randomly split from the develop- ment set of AMBIGNQ. Each AQ in the dataset has over three interpretations on average, which in turn means that each CQ has over three options on average. The average length of the CQs varies from one split another, with a general trend of longer CQs having more options.
We first conduct human preference test to in- vestigate the question: Is our CQ-based approach preferred over a DQ-based one handle AQs in ODQA? This is to check if is worthwhile to pur- sue the CQ-based approach.
Setup We randomly sampled 100 AQs from the development set. Then, for each AQ, we asked three annotators to show their preferences for "CQ", "DQ", or "Equal", along with the rationale. That is, given an AQ, we ask people to compare "being pre-
The first column in Table 2 shows the statistics

---

cQ
Split DQ
Input in addition AQ
Acc.
Prc.
Rec.
0.59
0.33
No Answers for AQ
63.9
61.9
60.7
61.3
Predicted Answers for AQ
56.5
59.7
24.1
34.3
Figure 3: Percentage of questions where the majority of people preferred "CQ", "DQ" and "Split", respectively, "Split" denotes that there was no majority response.
Table 3: Evaluation results for the Ambiguity Detection task. The No Answers case uses BERT-BASE-CAR determine whether given question ambiguous not. The Predicted Answers case makes use of answers predicted by SPANSEOGEN and classifies the question unambiguous only exactly answer predicted.
sented with answers to all possible interpretations (DQs) of the AQ" vs "first answering a CQ and then being presented with an answer fitting their in- tention." We then report the majority preference for each of the questions. Please refer to Appendix C for more details.
2. No Answers for AQ: not predicting answers the AQ, and thus not using them in the subsequent tasks
Figure 3 demonstrates that answering AQs using CQ is preferred over DQ. The prominent reasons stated by annotators for favoring "CQ" are its ease of use, conciseness, in- teractivity, and ability provide clear guidance. Conversely, annotators who preferred "DQ" men- tioned its advantage as being more specific and clearer in addressing the given question.
Results and Analysis
In the remainder of the section, we present the experimental setup and results for each task. Please refer to Appendix D for more details.
6.1 Task 1: Ambiguity Detection
Setup Since our dataset consists only of AQs. i.e., questions with multiple interpretations and an- swers, combine it with unambiguous questions, i.e., questions with a single interpretation and an- swer, from AMBIGNQ for this task.
Note, CQ was unanimously preferred 23 times. and DQ. 5 times. Also, unanimity in favor of CQ was observed across AQs regardless of the number of interpretations-on options- o unanimity in favor of DQ only occurred for AQs with up to three interpretations. In other words, CQ can be preferred regardless of the number interpreta- tions, while DQ is not preferred when many inter- pretations are possible. This is intuitive given that more interpretations result in more text for people to process for the DQ-based response to AQ.
For No Answers for AQ case, we use the BERT model (Devlin et al., 2018) with a sim- ple fully connected layer on top for the binary classification task. The model is trained on the combined dataset for 96 epochs. The model also takes in a prompt of the form "question (SEP) rele- vant "ast_ppasssss" input and outputs "Ambiguous" or " "Unambiguous".
For Predicted Answers for AQ case, we use BART-based model called SPANSEQGEN, the best- performing model for predicting answers for AQ by Min et al. (2020), and finetuned it the AM- BIGNQ dataset. This model takes in prompt of the form "question [SEP] passragat_pssagss""" as in- put and predicts all plausible answers. We classify a question as "Ambiguous" if the model outputs more than one plausible answer and "Unambigu- ous," otherwise.
6 Experiment 2: Handling AQ with CQ
Given that the CQs are preferred over DQs, we now study the question: Do predicted answers for AQ help improved the end-to-end performance of the CQ-based approach? Since predicted answers for AQ have been shown be helpful for previous DQ-based approaches (Min et al., 2020; Gao et al., 2021), we want to verify if they are also helpful for the CQ-based approach. 4 For this, we experiment with two settings:
Results and Analysis Table summarizes the result of ambiguity detection of two models BERT (No Answers) and SPANSEQGEN (Pre- dicted Answers). SPANSEOGEN exhibits similar precision BERT BASE (59.7 vs 61.9) but signifi- cantly lower recall (24.1 60.7). This is because most questions are classified "Unambiguous." since the average number of answers generated by SPANSEQGEN is 1.24. Consequently, this results in a much higher precision when compared to the
1. Predicted Answers for AQ: running QA model on the AQ and incorporating the pre- dicted answers as input to the subsequent tasks
'Note, while both CQ-based and DQ-based approaches seek to solve the issue of AQs in ODQA, the performances of the models are not directly comparable. This is because the problems are formulated diffcrently-the former generates a CQ and an answer aligned with the user's intention, whereas the latter generates DQs and answers for them.

---

Category
Options
Input addition to AQ and RPs
BLEU-4
BERTSCORE
EM
BLEU-1
Pre.
Rec.
F1
Avg. #
No Answers for AQ Predicted Answers for AQ Ground Truth Answers for AQ
7.9
88.9
20.2
47.3
37.4
18.2
24.5
2.0
7.9
88.9
22.8
44.0
36.9
19.0
25.1
2.0
15.4
89.6
25.2
46.9
34.3
34.4
34.3
3.7
Table 4: Evaluation results for the Clarification Question (CQ) Generation task, where generated CQs are compared against the reference CQs. Each CQ was generated from an ambiguous question (AQ), relevant passages (RPs). and either No Answers, Predicted Answers, or Ground Truth Answers for the AQ. The ground truth answers case represents an ideal scenario in which the QA system perfectly identifies all possible answers for the AQ.
NQ-pretrained BART
CQ-finetumed BART
CQ used to clarify the AQ
Pre.
Rec.
F1
# Ans.
Prc.
Rec.
F1
# Ans.
CQ generated with No Answers for AQ CQ generated with Predicted Answers for AQ
47.9
25.2
33.0
1.5
54.4
31.1
39.6
1.6
49.6
26.2
34.3
1.5
55.4
32.0
40.5
1.6
CQ generated with Ground Truth Answers for AQ
39.7
37.5
38.6
2.0
47.5
49.5
48.5
2,5
Ground Truth CQ
47.5
39.8
43.3
2.0
58.0
53.8
55.8
2.5
Table 5: Evaluation results for the Clarification-basse QA task. Answers found by QA model for the AQs clarified with CQs are compared against the ground truth answers for the AQs. Three variations of model-generated CQs. derived from the CQ Generation task, are used to clarify AQs. The Ground Truth CQ case is an ideal scenario which Ground Truth CQs used clarify the AQs. The # Ans. is the average number of unique answers predicted for each AQ.
recall for the same case. This result indicates that classifying AQs by predicting all plausible answers is a challenging task for the Seq2Seq model.
the two realistic scenarios (No Answers for AQ. Predicted Answers for AQ), the quality of the gen- crated CQs does not vary significantly in terms of the CQs themselves, the category, or the options. This suggests that incorporating plausible answers as input to the CQ generation process does not im- prove the quality of the generated CQs in realistic scenarios. However, when ground-truth answers are provided as input to the CQ generation process (Ground Truth Answers for AQ), a significant im- provement in the quality of the generated CQs is observed, but the quality is seemingly insufficient with a large room for improvement.
6.2 Task 2: Clarification Question Generation
Setup For this task, we only use ground truth AQS to isolate the task from ambiguity detection. Please refer to Section 6.4 for experiments which errors do propagate from one task to the next.
For No Answers for AQ we first train a BART model for 18 epochs, that takes "AQ as input and generates CQ output. During inference, this model was used with prompt of the form "AQ [SEP] rele- vant_passages".
[SEP]
In some cases, predicted CQs that are semanti- cally correct were counted as incorrect. For exam- ple, the model generated the CQ "Which chairman: 2017 or 2012?" for example 5 in Table 2. Although deemed incorrect, a manual examination of rele- passages revealed the 4th and 3rd chairmen took office in 2017 and 2012, respectively. This illustrates the challenge of directly comparing a pre- dicted CQ and the respective reference CQ. Thus, the absolute score in Table 4 may not be as mean- ingful as the relative scores. Also, evaluating CQs in a downstream task may be necessary to better assess the qualities of the CQs, which we do in the clarification-bart QA task.
For Predicted Answers for AQ, we train another BART model for 41 epochs, that takes "AQ [SEP] possible _atiblrr_sswersssseeeee [SEP]
as input and generates a CQ as output. During infer- ence, the model takes input with gessible_answed answers predicted by SPANSEQGEN.
We also consider Ground Truth Answers for AQ case. This case is an ideal version of the Predicted Answers for AQ case, where the ground truth answers are used as possible _assible_answer.. answers. Hence, this case allows us to examine the effect of providing the correct set of plausible answers.
additional setting, the
6.3 Task 3: Clarification-baseed QA
Results and Analysis CQ generation with three variations of inputs are presented in Table 4. The results indicate that in
The evaluation results of
Setup We use NQ-pretrained BART for the reader model which was trained on Natural Ques-

---

tions (NQ) dataset (Kwiatkowski et al., 2019a). The model takes in an AQ clarified by cQ-which is the concatenation of AQ, category, and option- and reranked relevant passages as input and pre- dicts an answer for AQ clarified by CQ. (See Sec- tion 3.3 for more details on AQ clarified by CQ)
Ambig. Detect.
CQ Gen.
Prc.
Rec.
F1
No Answers
43.2
19.9
27.3
No Answers
Pred Answers
42.8
19.6
26,9
No Answers
22.5
8.3
12.1
Pred Answers
Pred Answers
24.7
9.0
13.1
Table 6: End-to-end Evaluation Results: The perfor-
mances are measured at the end of the pipeline, i.c., clarification-based QA.
In addition to the NQ-pretrained model, we also finetuned the NQ-pretrained reader model (CQ- finetuned BART) our proposed dataset for 8 epochs. During finetuning, the model also takes in an AQ clarified by CQ as imput. The target label is the corresponding answer for each option.
6.4 End-to-End
Setup We now conduct experiments to check RQ2- predicted answers for AQ help im- prove the CQ-based approach to handle AQ end-to- end. We consider four combinations of setting for ambiguity detection and CQ generation:
During the inference, we employed three varia- tions of model-generated CQs, derived from Sec- tion 6.2 CQ Generation task. Morcover, we con- sider an ideal scenario wherein the Ground Truth CQ is available and used to clarify the AQ.
Pred Answers Pred Answers: running QA model on the AQ and incorporating the pre- dicted answers in both tasks
2. No answers-Nee No Answers: not running the QA model on AQ
Results and Analysis The evaluation results of clarification-baseed QA using four variations of in- put and different reader models are presented in Table 5. Two ideal settings (CQ generated with Ground Truth Answers for AQ and Ground Truth CQ) exhibit lower precision scores. On the other hand, they outperform the other two variations (CQ generated with No Answers for AQ and CQ gen- eraled with Predicted Answers for AQ) in terms of recall, resulting in higher FI scores, well. One reason for this that the CQs generated by Ground Truth Answers for AQ and Ground Truth CQs contain more options (1.5 more on average) which leads to predicting more answers than the other two variations, resulting in higher recall and lower precision scores.
3. Pred Answers-No Answers: running the QA model on the AQ but using the predicted an- swers as input for ambiguity detection only
4. No Answers-Pred Answers: running the QA model on the AQ but using the predicted an- swers as input for CQ generation only
The end-to-end performances are measured at the end of the pipeline, i.e., clarification-batt QA.
Results and Analysis As shown in Table 6, the use of the BERT model (No Answers) for ambigu- ity detection and prompting without answers (No Answers) in the input for CQ generation yields the highest F1 score of 27.3. However, the combination of using the BERT model (No Answers) for ambigu- ity detection and utilizing predicted answers by the SPANSEOGEN model (Pred Answers) in the input for CQ generation resulted in an Fl score 0.4 lower than the best combination. Note, the No Answers- Pred Answers setting is not only (slightly) worse than the best approach, but is also inefficient as it requires running both BERT and SPANSEQGEN models during inference.
The average numbers of options in Table 4 and those of unique answers in Table 5 indicate that both NQ-pretrained BART and CQ-finetuned BART struggle to generate distinct answers for dis- tinct options. For instance, in the CQ generated with Ground Truth Answers for AQ case, where the average number of options for CQs is 3.7, only 2.5 distinct answers were generated for each AQ. In other words, both models tend to produce the same answer for the given AQ even if the specified options are different. This phenomenon, referred to as the "collapse" of the models has also been rc- ported in previous studies such as (Zhang and Choi, 2021). suggests that deep learning models can be insensitive to subtle differences in the input-whre different options are chosen for the same AQ, the input would be identical except for the option.
No Answers-No No Answers and Pred Answers- Pred Answers are the only settings in which only single model is used for ambiguity detection and generating input for CQ generation. Among these, the quality of the generated CQs varies significantly. More specifically, the results show that in the Pred Answers-Pree Answers scenario, the poor perfor- mance of the ambiguity detection stage propagates

---

the remainder of the pipeline. This suggests that incorporating plausible answers as input to the CQ generation process prior to generating the CQs is not a desirable approach in the CQ framework. Fi- nally, the end-to-end performance of all four cases still has large room for improvement, showing the challenging nature of CQ-based approach to handling AQs in ODQA, as well the need for resources like CAMBIGNQ.
addressed through our human validation process. Furthermore, the data annotators were fairly com- pensated for their work, and the details of payment can be found in Appendix B.
A Acknowledgements
K. Jung is with ASRI, Seoul National University, Korea. This work has been financially supported by SNU-NAVER Hyperscale AI Center. This work was partly supported by Institute of Information & communications Technology Planning & Eval- uation (ITP)) grant funded by the Korea govern- ment(MSIT) INO.2021-0-01444 Artificial Intelli- gence Graduate School Program (Seoul National University)]
7 Conclusion
We proposed a CQ-based approach to handle AQs ODQA. Along with presenting a new dataset, we defined a pipeline of tasks, designing appropri- ate evaluation metrics for it. Experiments show the promising, yet challenging nature of the tasks. We hope that our dataset will serve as valuable resource for research in this area and support the development of more user-friendly QA systems.
References
Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2020. Convai3: Generating clarifying questions for open- domain dialogue systems (clariq). arXiv preprint arXiv:2009. 11352.
Limitations
As shown in our results, both clarification question generation and clarification-basee question answer- ing evaluations can still underestimate the perfor- mance of the generated clarification questions due to various factors. One reason is that the reference clarification questions are one of many possible an- swers, not the only correct answer. Another reason is that the intrinsic evaluation, which depends the overlap between the texts, may not properly handle semantically correct predictions. Addition- ally, the extrinsic QA model for clarification- based question answering may fail to perform reasoning. These limitations highlight the need for further re- search in the field to improve evaluation methods for clarification question generation tasks.
Mohammad A Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2019. Asking clar- ifying questions in open-domain information- conversations. In Proceedings of the 42nd Interna- tional ACM SIGIR Conference on Research and De- velopment in Information Retrieval, SIGIR' 19, page 475-484, New York, NY, USA. Association for Com- puting Machinery.
Sanghwan Bae, Donghyun Kwak, Sungdong Kim, Dong hyun Ham, Soyoung Kang, Sang-Wov Lee, and Woo Chul Park. 2022. Building a role specified open-domain dialogue system leveraging large-scale language models. ArXiv, htss/2055.0017..
Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017. What do you mean exactly? analyzing clarification questions in eqa. In Proceedings of the 2017 conference on conference human information interaction and retrieval, pages 345-348
Ethics Statement
Our proposed datasets will not pose any ethical problems as they have been constructed from the publicly available AMBIGNQ (Min et al., 2020) dataset, which itself is derived from the Natural Questions dataset (K (Kwiatkowski et al., 2019b).
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological mea- surement, 20( 3037).37-66
W Bruce Croft. 2019. The importance of interaction for information retrieval. In SIGIR, volume 19, pages 1-2.
Additionally, the use of the InstructGPT model for the generation of data was done by utilizing the official website of OpenAI'. All models used in the experiments from the publicly available web- site or Github. While there is a possibility of bias or toxicity in the generated text, such issues are
J Shane Culpepper, Fernando Diaz, and Mark D Smucker. 2018 Research frontiers in information retrieval: Report from the third strategic workshop on information retrieval in lomne (swirl 2018). In ACM SIGIR Forum, volume 52, pages 34 90. ACM New York, NY, USA.
'https://openal

---

Zhuyun Dai. Vincent Y Zhao. Ma. Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B Hall, and Ming-Wa Wei Chang. 2022. Promptagator: Few-shot dense retrieval from 8 examples. arXiv preprint wrXiv:2209.17555
Chin-Yev Lin. 2004. Rouge; A package for automatic evaluation of summaries. In ACL 2004.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How lan- guage models use long contexts. arXiv preprint arXiv:2307.0317
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. 2022. When not to trust language models: Inves- tigating effectiveness and limitations of paramet- ric and non- con-paraettiic memories. arXiv preprint arXiv:221..05111
Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing. Shafiq R. Joty, and Boyang Li. 2022. Is gpt-3 a good data annotator? ArXiv, abs/2212. 10450.
Yifan Gao, Henghui Zhu, Patrick Ng. Cicero dos San- tos, Zhiguo Wang. Feng Nan, Dejiao Zhang. Ramesh Nallapati, Andrew O Amoll, and Bing Xiang. 2021. Answering ambiguous questions through generative evidence fusion and round-trip prediction. In Pro- ceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3263- 3276.
Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica. 22(3):276-282.
Shikib Mehri and Maxine Eskenazi. 2020. Usr: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 68 1877
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering am- biguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 5783- 5797, Online. Association for Computational Lin- guistics.
M. Guo, Mingda Zhang, Siva Reddy, and Malihe Alikhami. 2021, Abg-coqu: Clarifying ambiguity conversational question answering. Conference on Automated Knowledge Base Construction.
Harold W. Kuhn. 1955. The hungarian method for the assignment problem. Naval Research Logistics (NRL). 52.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroli L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv, abs/2203.022 021155.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins. Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wi Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019a. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics. 7:452-466.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In ACL.
Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions us- ing neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2737-2746, Melbourne, Aus- tralia. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wai Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe V. Le, and Slav Petrov. 2019h. Nat- ural questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:453-466.
Ricardo Rei, Craig Alan Stewart, Ana C. Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. In EMNLP.
Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Joongbo Shin, and Ky- omin Jung, 2021. Kpqa: A metric for generative question answering using keyphrase weights. In NAACI.
Marzieh Saeidi. Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2087-2097, Brussels, Belgium. Association for Computational Linguistics.
Haonan Li, Martin Tomko, Maria Vasardami, and Tim- othy Baldwin. 2022. Multispanqa: A dataset for multi-span question answering. In NAACL.

---

Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wes Wei Chang. 2022. Asqa: tions meet long-form answers. arXiv preprint
A Details of partial match with the Hungarian algorithm
Factoid ques-
arXiv:2204.0609..
A.1 Alignment for Clarification Question Generation task
Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce label- ing cost? gpt-3 can help. In Conference Empirical Methods Natural Language Processing.
The similarity function sim y) is defined as fol- lows:
Wenhui Wang. Hangbo Bao, Shaohan Huang, Li Dong. and Furu Wei. 2020. Minilmv2: Multi-head self-attention relation distillation for compress- ing pretrained transformers. arXiv preprint arXiv:2012. 15828.
sim(a; u) wew.leen((.CC)) ( (4)
In Equation 1 and 2, similarity scores between prediction and reference are calculated by dividing the length of the longest common subsequence by the length of the predicted option and reference option, respectively.
Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liangbo He. 2021. A survey of suman-in--heelooy for machine learning. Future Gener. Comput. Syst. 135:364-381.
To match cach predicted option to corre- sponding reference option, the Hungarian algo- rithm (Kuhn, 1955) applied, and optimal corre- spondence mapping function fü for i-th option is obtained.
Zeqiu Wu, Ryu Parish, Hao Cheng, Sewon Min, Prithvi- raj Ammanabrolu, Mari Ostendorf, and Hannaneh Hajishirzi. 2022. Inscit: information-seeking con- versations with mixed-initiative interactions. arXiv preprint 22X002207.00746..
Jingjing Xu, Yuechen Wang, Duyu Tang, Nan Duan, Pengcheng Yang, Qi 7eng. Ming Zhou, and Xu Sun. 2019. Asking clarification questions in knowledge- based question answering. In Proceedings the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP--JCNILP), pages 1618-1629, Hong Kong, China. Association for Computational Linguistics.
A.2 Alignment for Clarification-bssed Question Answering
In the evaluation for CBQA, the reference answers are not a single string but rather a list of strings that may represent single answer. this sense, the max? pree and max for i-th example is calculated differently as follows:
Hamed Zamani. Susan Dumais. Nick Craswell, Paul Bennett, and Gord Lueck. 2020a. Generating Clar- ifying Questions for Information Retrieval, page 318-428. Association for Computing Machinery. New York, NY, USA.
max prec EppER, max, sim(p, f((p)))
(5)
where, f(()) is the single representation (e.g. "MJ" for the example from Section 3.3) of the ref- erence answer and prediction p is aligned to single reference answer set consisting of total J representations.
Hamed Zamani, Gord Lueck, Everest Chen, Rodolfo Quispe, Flint Luu, and Nick Craswell, 2020b. Mim- ies: A large-scale data collection for search clari- fication. In Proceedingy of the 29th ACM interna- tional conference on information & knowledge man- agement, pages 3189-310 3196.
Michael Zhang and Eunsol Choi. 2021. SituatedQA: In- corporating extra-lingee lingusiin contexts into QA. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 7371- 7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
max TeC = Ever, max, sim(ry, ft (r)),
(6)
where, r is not the single string but list of J strings that may represent single answer. (e.g. ["MJ", "Michael Jefferey Jordan", "Jordan") for the example from Section 3.3) and all representa- tions in the list are aligned the same prediction by f (r).
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.09675
"The Hungarian algorithm, used for assigning tasks to workers in a one-to-one manner with the objective of minimiz- ing the cost, is adapted in our study to maximize the cost by altering the setting.

---

B Details of data collection and dataset
cluded 20 validation examples that were annotated by co-author in each set. The annotators were notified of the existence of the validation examples and asked to re-annotate the samples if the correct percentage of the correctly annotated validation examples did not meet a pre-determined threshold. For the payment of the annotators, the co-authors first conducted annotations for an hour to estimate the average number of annotations that could be completed within an hour. Based on this estimation, rate of 15 dollars per example was established to ensure that the annotators would be paid at least 133% of the minimum wage.
B.1 Details of data generation by LLM
We use OpenAl (fext-davinci-003) API model for the generation. For the hyperparameters of Instruct- GPT, we limited the maximum length of the output token 100 and used 0.7 for temperature. We set top_p and to 1. As mentioned in Section 4, the prompt in the form of concatenation of the six examples, instructions, target AQ, and target DQs. Each example used for the prompt is described in Table 9.
B.2 Details of Manual Editing
B.3 Details of Inter-Annotator Agreement
Ten undergraduate students fluent in English were recruited through the university's online commu- nity, and seven of them successfully passed the qualification test. The recruited annotators were provided with detailed description of task def- initions, instructions, and examples as shown in Figure During the recruitment process, all ap- plicants were informed that their annotations would be utilized for academic purposes and would be in- cluded in published papers. This information was explicitly stated in the recruitment announcement and instructions to ensure transparency. The an- notators then asked to review 25 examples that had been previously annotated by co-authors and revise 25 CQs generated by the InstreutGPT model.
We conducted an evaluation of the output from two annotators and report the BLEU score and EM score. The BLEU score and EM score for the entire CQ are 65.8 and 39.3, respectively. When considering the category and options separately, the BLEU score for the category is 76.5 with an EM score of 56.8, while the BLEU score for options is 66.0 with an EM score of 63.9. All scores provided have been micro-averaged.
We conduct an analysis of instances in which there was disagreement between the two annota- tors. The primary cause of these disagreements can be attributed variations in the specificity of cat- egories, the options provided. For example, in Figure there were different opinions within the category of "version" and the first option "young in series 2". The other annotator suggested alter- natives such as "version of Tom Riddle" for the category and "the young Tom Riddle in Harry Pot- ter and the Chamber of Secrets" for the first option. These alternatives accurate in capturing the in- tended meaning, but they differ in their surface form.
Seven annotators who passed the qualification stages were then selected to participate in the man- ual editing stages. As shown in Figure 4, the an- notators were provided with Ambiguous Question and Disambiguated Questions on the left side of the page. To assist the annotation process, we used a process to identify the longest common subse- quence between the AQ and DQs, a spaCy con- stituency parser to identify different constituent parts of the DQs, and highlighted these parts. On the right side of the page, the InstructGPT-- generated CQ was provided, and the annotators were given the option to revise, pass (no revision), report (single interpretation and answer) the given CQs.
B.4 Details of human editing
We provide deeper analysis of human editing made on examples generated by InstructGPT, as shown in Table Specifically, for instances where human annotators made partial revisions, focusing on either the category or the options alone (referred to "Category Only" and "Options Only" in Table 2), compute the BLEU score. Interestingly, both the "Category Only" and "Options Only" cases exhibit BLEU scores of 37.0 and 53.6, respectively. Additionally, in cases where the model generated invalid forms (referred to "Whole Question" Table 2), the BLEU scores between the model's
We used streamlit?, an open-source app frame- work for creating web apps for data science and machine learning, to construct the interface. The InstructGPT-generaan examples were divided into sets of 500 examples, and for quality control, in-
"https://stream"

---

predictions and the human revisions yield 36.7. is important note that while BLEU scores may not capture semantic similarity, they do provide valuable insights into the disparity between human- labeled data and model-generated data.
number of DQs) and user preference, we proceeded by partitioning our development dataset into five distinet groups based on the number of interpre- tations. These groups were categorized as data with two, three, four, five, and more than five in- terpretations. Subsequently, single example was randomly selected from each group, resulting in the formation of one batch comprising five instances. Each batch was then assigned to an annotator for annotation, with total of 20 batches being pro- cessed in this manner. An example annotation interfaces is shown in Figure 8.
B.5 Details of dataset
Table 10 provides an overview of the most fre- quently used categories within each split of the dataset. It is evident that the top five categories con- sistently uppear in all three sets, although their spe- cific order may vary. This suggests that the dataset was well split into three parts. Moreover, Figure illustrates the top 50 categories from the entire dataset, providing a broader perspective. Over- all. we have 593 distinct categories, with 412 of these categories occurring only once. This obser- vation aligns with the previously discussed Inter- Annotator Agreement in Section B.3, where vari- ations in specificity among annotators contribute the presence of unique categories. For instance, examples such as "Jurrassic world" "movie" or "Will Turner" vs "character" illustrate this inherent variability. This existence of multiple representa- tions for single categories or options is inherent to our task, and it is considered a natural occurrence. The set of categories can be expanded as needed. Additionally, we provide the number of options in our dataset in Figure 7. Since our dataset is built upon the AMBIGNQ, the distribution of options is comparable to the previous study.
C.2 Detailed analysis on test result
Figure 9 provides the results obtained for varying numbers of interpretations. The percentages of annotators favoring "CQ" remain relatively stable across different numbers of interpretations.
D Training Details
Training Detail of Ambiguity Detection The Ambiguity Detection task utilized combined dataset consisting of 9,996, 977, and 977 instances in the train, validation, and test sets, respectively, For the model, the BERT-base-cased model was finetuned with batch_sizz 16, accumulation __sepp I, learning rate 2e-5, and early_stop patopp_patenceeeeee We use released checkpoint for pretrained SPANSE- QGEN model We used one A5000 GPU for finetuning and it took approximately 4 hours. The training epochs were determined according to the validation performance based on accuracy.
Details of human preference test
C.1 Details of test setup
Training Detail of Clarification Question Gener- ation The two BART 1112 were finetuned on our CAMBIGNQ with the split as described in Table 1. Both models share the same hyperparameter during finetuning, which are batch_sie 10, accumulation __tep_etpppppp 2, learning rate Ie-5, and early_stop uuop_patience 10. The training epochs were determined according the valida- tion performance based the BLEU score of the whole CQ. We used one A6000 GPU for both fine- tuning and it took approximately hours for No Answers case and 4 hours for Predicted Answers and Ground Truth Answers cases.
We use Amazon Mechanical Turk for the human preference test. To ensure the quality of responses, we restricted the workers whose nations are the US, CA. AU, NZ, UK and whose HIT minimum hits are over 5,000, and HIT rates are higher than 97%. Additionally, we enforced a requirement for annotators to provide at least one sentence explain- ing the reason for their choices. Instances where annotators failed to provide reason, provided reason consisting of few words or presented rea- son irrelevant our task were deemed as "rejected" cases. Annotators were informed of this rule and compensation for the MTurk workers was set at more than $10 per hour.
hhttps://huggingfface cttp://aaccccccccaeeaaeeeeeeaaae co/bert- Seseert-baee-csseee "https://nlp..e wawwingthnnnnoooooooo edu/ aaaaambbiggaa anbpg/////mmeess//////////////// barbiggn-bartt-lre--------
In order to investigate the potential correlation between the number of interpretations (i.e., the
" hhttps://hugginggeeee
1he total length of input was truncated to 1,024 tokens due to the maximum input length of the model
Phttps: //wwww mturk.

---

Reader Model
Pre.
Rec.
F1
Acc.
ChatGPT Prc.
Rec.
Fl
Accuracy
CQ finetuned BART
58.0
53.8
55.8
35.8
Zero-shot
8.0
64.5
14.3
50.8
InstructGPT
7.4
60.0
13.1
43.2
Four-shot
11.3
64.0
19.2
49.9
Table 7: Evaluation results for the Clarification-based QA task employing two different reader models. Both cases utilize the truth CQs to clarify the AQ. The Ace. represents accuracy which evaluates whether the model's 's response includes any gold answer.
Table 8: Evaluation results of conversational setting em- ploying ChatGPT. ChatGPT receives the input framed as an interactive dialogue between the user and model, outlined in the subsequent sequence: 1) User asking AQ, 2) Model offering ground truth CQs, 3) User selecting an option, and 4) Model generating the corresponding answer for selected option. Zero-shot and Four-shot denote the number of examples presented to the model within the prompt.
Training Detail of Clarification- Question Answering For Clarification-Baani Based Question An- swering, the NQ-pretrained BART model was finetuned with batch_size 10, accumulation 2, learning rate le-5, and early_stop seyy_sto__pattte pot_eeaeeece 10. We used one A6000 GPU for finetuning and it took approximately 2 hours.
number of unique answers is challenging. Follow- ing Liu et al. (2023) and Mallen et al. (2022), adopt accuracy, which evaluates whether the prediction includes any gold answer.
E Inference Employing Large Language Models
Results and Analysis The evaluation results of clarification-bute QA, utilizing InstructGPT as the reader model, and our task's reformulation within an interactive dialogue framework with ChatGPT, are presented in Table 7 and Table 8, respectively. It is noteworthy that the low precision in both results from InstructGPT, ChatGPT in zero-shot config- uration, and ChatGPT in four-shot configuration can attributed to the model's 's tendency to gener- ate answers at the sentence level. These responses average 27.3 words, 25.4 words, and 19.9 words respectively. In contrast, the gold answers are more concise, averaging 2.6 words, leading to the ob- served low precision scores.
Our primary experiments, which leveraged the BART-large models as our baselines, demonstrated suboptimal performance across different settings. To evaluate the efficacy of recent Large Language Models (LLMs) in our task, designed additional experiments incorporating LLMs within our frame- work. these supplementary experiments, we only consider the most ideal case from Section 6.3 where Ground Truth CQs are available.
Setup We leveraged two distinct variations of InstructGPT (Ouyang et al., 2022), provided by OpenAl (namely, text-davinci-003, gpt-3.5-turbo), for our additional studies. Initially, we employed the text-davinci-003 model (InstructGPT) as the reader model in Section 6.3, replacing the previ- ously used BART-large models. Subsequently, we reformulated our task an interactive dialogue be- tween the user and the QA models, comprising the following sequence: 1) User asking AQ, Model offering CQs, 3) User selecting an option, and 4) Model generating corresponding answer for a given option. Within this conversational framework, we utilized the gpt-3.5-turbo model (ChatGPT) and conducted inference under two settings: zero-shot and four-shot.
Utilizing InstructGPT reader model showed improved performance compared to our baseline, which uses CQ fine-tuned BART as a reader model. Additionally, reformulating our task as an interac- tive dialogue and incorporating ChatGPT shows improved recall and accuracy. However, it is evi- dent that there is substantial potential for further enhancement which underscores both the challeng- ing nature of our tasks and the need for further research.
We evaluate both models using the conventional metrics of precision, recall, and Fl score. Fur- thermore, due the fact that both models gener- ate responses at the sentence-level, quantifying the
h/mppp:://nn.... caa.p.cs..aase. w.s.washington. www.ggtnneeuuueeee edu/ambi htt.:eddaaaggggmmmdeeeeeeeee g/ aq-bart-large waw.aaaree22-.2221 24-8.zip

---

Generate the claritying question for ambiguous question that gives options corresponding disambiguated question.
Instruction
ambiguous question: Why did the louis cardinals move to arizona?
disambiguated question what ability caused the st louis cardinals move to arizona?
Example_1
disambiguated question what physical issue caused the st louis cardinals move to arizona?
disambiguated question what fan issue caused the st louis cardinals move arizona?
clarifying question: Which type reason; Ability, physical issue, or fan issuc?
amhignous question: Who the current chairman african union commission?
disambiguated question I: who is the 4th chairman african union commission?
Example_2
disambiguated question 2: who is the 3rd chairman of african union commission?
disambiguated question 3: who is the 2nd chairman of african union commission?
clarifying question: Which chairman: 4th, 3rd, or 2nd?
ambiguous question: W ho won the final hoh big brother 20?
disambiguated question 1: who won the final hoh the american reality show big brother 20?
Example_3
disambiguated question 2: who won the final vote in the british reality show celebrity big brother 20?
clarifying question: Which version: the american reality show, or the british reality show celebrity?
ambiguous question: How long do contestants get to answer on jeopardy?
disambiguated question I: how long do contestants get to answer a typical question on jeopardy?
disambiguated question 2: how long do contestants get answer final jeopardy question on jeopanly?
Example 4
disambiguated question 3: how long do contestants get to answer on jeopardy 's online test?
disambiguated question 4: how long do contestants have to answer during the first two rounds of jeopardy? clarifying question: For which type of questions: typical question, final jeopardy question, jeopardy's online test, or during the first two rounds of jeopard?
ambiguous question: Who is the longest serving manager in the premier league?
disambiguated question I: who is the longest serving manager in the premier league of all time in terms of time?
Example_5
disambiguated question who is the longest serving manager in the premier league of all time in terms of number of games?
clarifying question: In terms of what: time, the number of games?
ambiguous question: Who sang the original do you love me?
disambiguated question who is the band that sang the original do you love me in 1962? disambiguated question who is the singer that sang the original do you love in for the contours in 1962?
disambiguated question who characters that sang original you love the fiddler the roof?
Example_6
di daabbiggated question who singers that sang the original do you love me the 1971 fiddler on the roof film?
clarifying question: Which one: the hand 1962, the singer the contours 1962, the characters in the fiddler on the roof, or the singer the 1971 fiddler the roof film?
Table 9: The few-shot examples used for the prompt of the InstructGPT. These examples are concatenated with the instruction in certain order as mentioned in Section 4.

---

Interface
For eoch of and DGs the loft the Interfase,
CQ annotation instruction (Eng)
"CO sengasape whsdh's
right comeepancien 322 generated by LLM ploose CO carefully realting the following inirustioma.
Definition
the Nota Procesding ceeariccts the Open Domain Questor Aroverng domen..
given pararnted Vi tevked
after
busk is penarate Clattynng Sunnton 100) given Art biguas Cumetion (AD) and Beweeiggatt Quealeen
proclem""
is weinwitton Probler
cornopancing
Anciiquess Quastion (KG) a pertion that cin murtan inaepirnaa to and cultiipe pinu bis answers.
"teport buttons think phan a lingle snswer plase di "Separt"
Ambrgaess Qumstion: Whe the current chaimean ahrican union commrisiion?
EXI Abow un dndegann browowr ze youn nallihhe anmortta foom "Dareett teoreeguuate da untiven (Fig the anndine thar are eatn the ine dicac hav un coe in and unturiguaanon anover Gransa part of the DGs the part Fal onfireet the ong nut 422 twen hes the car novp ny arreet telbrr
You should click either "Revise" "No Problem' "Report" button for each question and then move to the next question.
trehe Clar Duestiee
Escludo D
Disaniigpated
who chaimman africon union
Answor 2: Houssa Falú, Housse Fald Matamntt
Exclude
Disentiggatted Question who the chaimann atriaan union commissiion
Answer2: 2e Wouerraaa Clarie Dtamine Zurea Nikorazana Blamiré Durna
1. Andoggaaze Duee
beclude D
2. Dhanntgurre Quaition (DC)
Phantiiaaatee Queation who io the 2ndd chaimman of atrican untor commissioo 2
2. CO serertte LLke The output of our model.
growoned by: your work default output be charged to the changes
byy wdttng AQ Orange seer each becare unwmiilgoco sou so th has only ona
Subrrissiin Buttone:
Charlying (us) a carotion thtt "Cartts AQ anit autlons tua keot to oomsuppening ng Don Generityy sach option the oninge part of each cormepanding DoL part
Move buttone "ores" button will lead you the previous ditse "naett button liad thw daes "ood unttoeee w lovi the that never routaed checked
Caaggery
aull Nitich the memammn vt, and, dec?
7. This prompt una Use curee by aeed CD lo
(a) CQ revision instruction page 1,
(b) CQ revision instruction page 2.
betlue D0
Instructions
Srarliggated gunatio how leng contiettaann git auseeee of puclpppaaon quatiio angegpe
coneful with the proppctttion words before "withh (e.g. "Whiih year "As of whih year?
Anmwer 2:10
%sx of" caww: EB
Catlude DO
Thankiggted Queatie 2: contrrtees
Anower 2:
Bluae Sigutted gustiic fo dadaaaana baik gh
Phoza optione the givan order (pechaption option comppende each DO: Ja "Whish year: 2012, 2010 or 20117? 2012 coreesoond bot corrosoonds D00 (G5)
"Wirich good category word use difficalt determine the categery word (e.g. "Wich one leader, party, or (ectioxr") "casss where optiora can not calegeriaed!
canscory
The categery word should be repeated the optione. (wg.. "Which year: 2010 or 30122" 2010 20177' not.l
but "hich
Each opdin should n disanbiiuazed quection For mudiile diandiquatedd sucstions with the sama meaning and the me arooess stark vactude"" for dherr bigueted quediions to ramoved the and the number of unstecked dicombrqusted ausstions and the number of apdiono should be the same."
a. For the case below, DC1 and DO hawe the same arawat Howoves since DO fannd DQ have difeew mean rask you should not ude any D1)
gooddattoow consessras ger
Exd Eus
Blar endt Quantied t fon lng do Iestrds get brewer typics quetiin pepordy?
Asswerts to fv sscods
(c) CQ revision instruction page 3.
(d) CQ revision instruction page 4.
Figure 4: The instructions provided to the recruited annotators for CQ revision.

---

Revise Clarified Question
cQ generated by LLM. Which skkeover: the Russion Red Army, unmanked armed men, with Russian group??
Ari Antkiguuus gpous Question When was the ukraine takeo over russia?
Rovia Quentinn
Wisch takeoven: Russion armed
Exclude
Reaise
Obambigurtedd Qvestion
over army??
Anewer 1920
neot
Exclude 2
Dhambigureed Question 2: when was the ukrsine coineas partamentary buildding taken unmuried mane
Febraay
Exclude 3
Olambigureed
Anewer 18 March 2014
Exckude 00
Diaakigureed Question & when was the ukraine volass doeessk Juhansk taken
Throughout
(a) Interface page for CQ revision example 1.
Revise Clarified Question
CQ generated LUM Which oner animaated short the Trazey Ullman Show, hour show?
Ambiguess Queation: When simpsona first inlevisio??
Plesse Rovie Oennten
Reviee
Dlsaniiguated Question when the simpoons first television aniimated
short tiicyy uliman show
prev
April
nest
Report
Exclude D0
Disanbiguated Question when first halt
December 17, 3969
Currenty Questions Which animated Trecey Uliman Show, as half hour prime time show?
Excluded DQ Indec:
(b) Interface page CQ revision example
Figure 5: Interface of qualification and manual editing stage for CQ revision.

---

Split
Categories (in order frequency)
Train
version, year, type, information, time version, type, time, year, information version, type, information, year, time
Validation
Test
Table 10: Most frequent categories in CAMBIGNO.
Top-50 Categories
500
400
200
142
100
8
B
1010
Varsoon
yyer
Information
"dharicto""
'sarnee
"age
"paroos"
"team"
"podie
"rellaso"
Siagor
'tapee'
"tangaage'
'eacore'
Individull
"book"
'steffomm!
pusition
spnderr
'Ann
Voloe
Figure 6: Top-50 categories in CAMBIGNQ.
2
3
4
5
6+
Opt. #
0.49
0.29
0.10
0.05
0.07
Figure Number of options distribution CAMBIGNQ.

---

Thank for partcipating work. used
Natural Language Processing.
Suppose you question
engine. Howevei, quastion multple possible answers.
These answers are shown you 2 different ways (Options and Bi, and your job is to tell us which option you profer as user the search engine.
Please select equai
difficut
Option (1)
clarificatio question, you select reporse, and (3) the cornect shown you.
Option possible
you,
(a) Instructions given MTURK workers.
Who
Suppose reading antwerij) from smartphone
B
you
Did you meat....
Who plays daays of our lives 20107
2015,
2010::18, 2018?
expleining your
rejocted automatically.
Did you mean...
Who plays days our lives
Did mean...
Who plays
(b) MTURK interface page example.
Figure Interface for preference test for MTURK workers.
interpretations
3 interpretations
Vote
Vola
0.55
(a) Preference test result of two interpretations examples.
(b) Preference test result of three interpretations examples.
interpretations
5 interpretations
Split
Split
Majority Vote
0.55
0.15
0.50
Majority Voto
0.60
0.15
0.25
(c) Preference test result of four interpretations examples.
(d) Preference test result of five interpretations examples.
interpretations
Split
Majority
(e) Preference result of examples with than five interpretations.
Figure 9: The preference test results for each group with a different number of interpretations.


Reviews:
[{"review_id": "bitOc3EZcv", "reviewer": "Reviewer_Sdsh", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for acknowledging as strengths the clarity of our description, the dataset, and the initial performance of baselines of our proposed task. \n\nWe appreciate the comment that you do not see a reason to reject our work. To address the concerns about the reproducibility of our work, we will make our dataset and models publicly available after acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an interesting set of problems, which the paper presents one type of solution to.", "reasons_to_reject": "I can't see any reason to reject this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "When confronted with an ambiguous question, rather than have the alternatives spelled out as a list of separate\nquestions, it is more natural to produce a clarification question with concisely expressed options. In this paper the\nauthors describe the addition to an existing dataset of ambiguous questions, relevant passages, and possible answers, a\nclarification question of the preferred type. This was done using InstructGPT with some few-shot examples to generate\npossible candidates which were then manually checked and revised.\n\nThey distinguish two subtasks relevant to the creation of this dataset: ambiguity detection and clarification question\ngeneration, and one which uses the results of these subtasks, namely finding answers to the clarification question.\n\nAnnotators were asked to state a preference between a number of the original verbose individual questions and the more\ncompact alternative version. This latter was usually preferred, and was invariably preferred when more than three interpretations were possible.\n\nThe paper presents preliminary results for ambiguity detection: under one condition, a BERT-based classifier is trained\nto distinguish ambiguous from non-ambiguous questions (trained and tested on the original dataset) and under another a\nBART-based model is used to predict answers, with the input classified as ambiguous if more than one is predicted. The\nfirst classifier achieved 61.3 F1, the second 34.3.\n\nThere are also preliminary results for clarification question generation. Various BART models are trained to predict the\nclarification question given (1) the ambiguous one and relevant passages, or (2) the ambiguous question, answers as\npredicted by one of the models just described as well as the relevant passages. For comparison, in one condition (3) the\npredicted answers were replaced by the actual answers, as a kind of best case. Naturally, the latter case produced the\nbest results, with no significant variation between 1 and 2.\n\nThe final set of experiments tests QA performance on clarification questions, testing four conditions: ambiguity\ndetection with and without predicted answers and clarification questions created with or without the predicted\nanswers. Perhaps surprisingly the best results were obtained with no answers for either stage.\n\nThe main contributions of the paper are (1) a clear description of the problem (2) provision (public?) of a dataset and (3) initial results for the accuracy of automation of some of the stages, and utility in a QA setting.\n", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I could not see any typos. The paper is clear and well written."}, "extra_text": {}}, {"review_id": "JKArhgoYSJ", "reviewer": "Reviewer_1HwZ", "summary": "", "questions": "1. While CQ improves user preference (as shown in experiment 1), 33% of people preferred not to generate a CQ. Could this indicate the generated CQs could be misleading or incorrect? Why do human annotators do not prefer those CQs? It would be beneficial to understand the underlying reasons for this preference.\n\nAnd at the very least, generating CQs should ideally (at least) not alter human preference, as more information is provided compared to only AQ and DQ provided. However, the fact that human preference decreases by 33% when generating CQs raises questions about the necessity and effectiveness of producing these CQs.\n\n2. The paper relies on large language models such as ChatGPT to generate CQs. I am curious about the ability of a model like ChatGPT to **directly** solve AQ, instead of using generated CQs to train small language model? Can ChatGPT solve AQ in a form similar to chain-of-thought, for example, a chain consisting of (1) ask CQ (2) generate DQ (3) answer the AQ. To summarize, what is the ChatGPT performance on such task? Could ChatGPT perform better than this complex pipeline?\n\n3. line 394 stated *“since predicted answers for AQ have been shown to be helpful for DQ based approaches”*, however, as shown in Table \n3 and Table 6, it seems *No Answers for AQ* makes better performance than *Predicted Answers for AQ* on the ambiguity detection and end-QA task. Does it mean your observation contradict to the observation from existing literature? Can you explain this discrepancy? Or please correct me if I misunderstand this. \n\n4. Since *category* name are mostly short, so, does the use of BLEU or EM score as evaluation metrics reflect the correctness of generated CQs accurately? How does it align with human evaluations? And also same question as the evaluation of CQs, did authors perform human evaluation or semantic-based evaluation on CQs, since the CQs is much longer than *category*?\n\n5. As shown in Table 5, it seems adding this CQ generation, as an intermediate process, does not help much on end-to-end QA performance, i.e., predicting answers for AQ. So, why do authors think generating CQ is necessary if people still mostly care about the end-to-end QA performance? \n\n6. The proposed method makes good ablation study on using or not using CQ in the question answering generation process. However, the paper does not compare with SoTA performance on AmbigQA. \n\nLeaderboard: https://nlp.cs.washington.edu/ambigqa/leaderboard.html\n\n\nMinor: \n\n7. For CQ generation evaluation, how many references are provided? Do authors construct multi-reference for evaluation to improve the robustness? \n\n8. Do authors try different prompts (as shown in line 239 and line 289) to generate CQ using ChatGPT?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for acknowledging as strengths the novel approach to handling AQs under realistic scenarios, our dataset, and the clarity of our writing.\n\n### **Response to Reject-1**\n>**Doubtful Motivation**: While CQ improves user preference (as shown in Experiment 1), 33% of people preferred not to generate a CQ, which raises questions about the necessity and effectiveness of producing these CQs.\n\nPlease see our **Answer to Question-1**.\n\n\n### **Response to Reject-2**\n>**Insufficient Comparison with SoTA**: The paper lacks comparison with state-of-the-art methods, which is essential to gauge the true effectiveness of the proposed approach.\n\nAs per your suggestion, we have conducted additional experiments using InstructGPT—a model trained for NLP tasks rather than conversation like ChatGPT—as a reader in place of BART in Section 6.3. In this experiment, we employed ground-truth CQs instead of model-generated CQs to measure the best possible performance in the ideal setting, as Reviewer SP89 suggested. \nThe two tables below indicate that the InstructGPT showed improved performance compared to our baseline (for which we also report CQ fine-tuned BART results in the same setup). However, there remains substantial room for improvement, underscoring the challenging nature of our task. Please note that the low precision in InstructGPT’s results can be attributed to the model generating answers at the sentence level, which averages 27.3 words, compared to the gold answers averaging 2.6 words. We also report the Exact Match metric, which checks whether the gold answer is included in the generated sentences.\n\n\nInitially, we chose BART-large for its free availability and public checkpoints, making it a suitable starting point for future research.\nFurthermore, direct comparisons with SOTA methods in AmbigQA are not feasible due to differences in tasks (Please see answer to Questions-6 for details).\n|               | |      InstructGPT    |      |          |\n|---------------|:-----------:|:--------:|:----:|:--------:|\n|               |     Pre.    |   Rec.   | F1   |    EM    |\n| AQ revised by |             |          |      |          |\n| Ground Truth CQ         |     7.4     | **60.0** | 13.1 | **43.2** |\n\n|               |      | **CQ finetuned BART** |      |      |\n|---------------|:----:|:---------------------:|:----:|:----:|\n|               | Pre. |          Rec.         |  F1  |  EM  |\n| AQ revised by |      |                       |      |      |\n| Ground Truth CQ         | 58.0 |          **53.8**         | 55.8 | **35.8** |\n\n\n### **Response to Reject-3**\n>**Questions about Methodology**: Several questions arise concerning the methodology, including the choice of evaluation metrics, the contradiction in observations, and the real necessity of CQ generation for end-to-end performance. These need to be clearly addressed to strengthen the paper (see questions for more details).\n\nThank you for your comprehensive feedback. We will address questions individually: \n- The choice of evaluation metrics (Please see **Answer to Question-4**)\n- The contradiction in observation (Please see **Answer to Question-3**)\n- The real necessity of CQ generation for end-to-end performance (Please see **Answer to Question-5**)\n\n\n### **Answer to Question-1**\n>1. While CQ improves user preference (as shown in experiment 1), 33% of people preferred not to generate a CQ. Could this indicate the generated CQs could be misleading or incorrect? Why do human annotators do not prefer those CQs? It would be beneficial to understand the underlying reasons for this preference.\nAnd at the very least, generating CQs should ideally (at least) not alter human preference, as more information is provided compared to only AQ and DQ provided. However, the fact that human preference decreases by 33% when generating CQs raises questions about the necessity and effectiveness of producing these CQs.\n\nJust to clarify, the alternatives being compared are: **“1 AQ + n DQs + n answers for the DQs” vs “1 AQ + 1 CQ + 1 answer based on the user response to the CQ”.** **No additional information is provided in CQ cases compared to DQ cases**. We will make this more clear in the paper. \n\nYes, 33% of people indeed preferred DQs over CQs. However, as shown in Fig 3, 59% of people preferred CQs over DQs. As discussed in Sec 5, people generally preferred CQ when there are many possible interpretations, but when the options are few, some people preferred DQs. This is because the benefits of using less screen space and reducing the amount of text to be read by people are magnified as the number of DQs increases.\n\n\n### **Answer to Question-2**\n>2. The paper relies on large language models such as ChatGPT to generate CQs. I am curious about the ability of a model like ChatGPT to directly solve AQ, instead of using generated CQs to train small language model? Can ChatGPT solve AQ in a form similar to chain-of-thought, for example, a chain consisting of (1) ask CQ (2) generate DQ (3) answer the AQ. To summarize, what is the ChatGPT performance on such task? Could ChatGPT perform better than this complex pipeline?\n\nFirstly, our problem setting consists of 1) a user asking an ambiguous question, 2) the model responding with a clarification question, 3) the user selecting an option, and 4) the model generating a distinct answer based on the selected option. **This is not a sequential reasoning process but an interactive one based on user responses, making it unsuitable for chain-of-thoughts.**\nWe conducted additional experiments with ChatGPT per your comment, by prompting our task as a conversation between the user and models. We found that ChatGPT struggled to generate distinct answers in both zero-shot and few-shot scenarios which again showcases the challenging nature of our tasks and the need for further research. Similar to experiments conducted by using InstructGPT, ChatGPT yields low precision due to the sentence level generation, averaging 25.4 words in the zero-shot and 19.9 words in the four-shot scenario.\n\n|            | **ChatGPT** |          |      |          |\n|------------|:-----------:|:--------:|:----:|:--------:|\n| Ground Truth CQ |     Pre.     |    Rec.   |  F1  |    EM    |\n| zero-shot  |     8.0     | **64.5** | 14.3 | **50.8** |\n| four-shot  |     11.3    | **64.0** | 19.2 | **49.9** |\n\n### **Answer to Question-3**\n> 3. line 394 stated “since predicted answers for AQ have been shown to be helpful for DQ based approaches”, however, as shown in Table 3 and Table 6, it seems No Answers for AQ makes better performance than Predicted Answers for AQ on the ambiguity detection and end-QA task. Does it mean your observation contradict to the observation from existing literature? Can you explain this discrepancy? Or please correct me if I misunderstand this.\n\nWe did mean that predicted answers for AQ were helpful for DQ-based approach. But this paper proposes a CQ-based approach, which is a different task setup to deal with AQ. That is, unlike DQ-based approaches in which all possible DQs and respective answers are found for the given AQ, CQ-based approach involves generating a CQ and finding 1 answer to AQ based on the user response to the CQ. We believe terms like “CQ-based approach” and “DQ-based approach” are causing confusion. We will clarify this in the paper.\n\n### **Answer to Question-4**\n> 4. Since category name are mostly short, so, does the use of BLEU or EM score as evaluation metrics reflect the correctness of generated CQs accurately? How does it align with human evaluations? And also same question as the evaluation of CQs, did authors perform human evaluation or semantic-based evaluation on CQs, since the CQs is much longer than category?\n\nIn light of supporting future work on the new task we propose, we chose to design automatic evaluation metrics so that other researchers can easily and fairly compare against our baseline results. Thanks for your suggestion to use a semantic-based evaluation metric, which still aligns with our goal and provides an additional means of evaluation. As per your suggestion, we measured BERTSCORE (Zhang et al., 2020), and these results and additional analysis will be included in the revised manuscript. The error analysis in Section 6.2 also gives insight to the future researcher that automatic evaluation might capture the relative quality of generated CQ but not implicit quality.\n| Input in addition to AQ and RPs | CQ | | Category | | Options | | | |\n|---------------------------------|:------:|:---------:|:--------:|:------:|:-------:|:----:|:----:|:------:|\n| | BLEU-4 | BERTSCORE | EM | BLEU-1 | Pre. | Rec. | F1 | Avg. # |\n| No answers for AQ | **7.9** | **88.9** | 20.2 | 47.3 | 37.4 | 18.2 | 24.5 | 2.0 |\n| Predicted answers for AQ | **7.9** | **88.9** | 22.8 | 44.0 | 36.9 | 19.0 | 25.1 | 2.0 |\n| Ground Truth answers for AQ | **15.4** | **89.6** | 25.2 | 46.9 | 34.3 | 34.4 | 34.3 | 3.7 |\n\n\n### **Answer to Question-5**\n> 5. As shown in Table 5, it seems adding this CQ generation, as an intermediate process, does not help much on end-to-end QA performance, i.e., predicting answers for AQ. So, why do authors think generating CQ is necessary if people still mostly care about the end-to-end QA performance?\n\nAs previously discussed, the preference test shows that CQ can improve user experience, with 59% of people preferring CQ over DQs. Accordingly, the room for improvement in the end-to-end performance can be seen as a reason to conduct further research in this direction rather than discarding this setup.\n\n### **Answer to Question-6**\n> 6. The proposed method makes good ablation study on using or not using CQ in the question answering generation process. However, the paper does not compare with SoTA performance on AmbigQA.\n\nWe cannot compare against the SOTA performance on AmbigQA, because the tasks are different— although both tasks were proposed to solve the same issue of dealing with AQs. In its essence, AmbigQA (DQ-based approach to deal with AQ) takes in an AQ and relevant passages and 1) first predicts plausible answers, 2) then generates DQs (to be displayed to the user with each answer). On the other hand, we (CQ-based approach to deal with AQ) take in an AQ with relevant passages and 1) first generate CQ, 2) then generate an answer for the AQ based on the response to CQ, where the CQ and the response to CQ ideally reveal the single correct interpretation of the AQ.\n\n### **Answer to Question-7**\n> 7. For CQ generation evaluation, how many references are provided? Do authors construct multi-reference for evaluation to improve the robustness?\n\nWe chose a simple CQ format to enhance user experience and ease the construction process. Although more complex formats are possible, our current format was found to be effective.\n\n### **Answer to Question-8**\n> 8. Do authors try different prompts (as shown in line 239 and line 289) to generate CQ using ChatGPT?\n\nWe experimented with various prompts in a pilot study and selected the ones that performed best to be used in the main experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**-- Novel Approach:** The authors introduce a new method by shifting from conventional disambiguated question generation to a clarification questions-based method, aligning more closely with real-world applications.\n\n**-- New Dataset:** The creation of the CAmbigNQ dataset to support research in ambiguity in QA represents a valuable asset for the community.\n\n**-- Writing Quality:** The paper is clear, well-written, and effectively uses examples to explain the concepts, enhancing readability and understanding.\n", "reasons_to_reject": "**-- Doubtful Motivation:** While CQ improves user preference (as shown in Experiment 1), 33% of people preferred not to generate a CQ, which raises questions about the necessity and effectiveness of producing these CQs.\n\n**-- Insufficient Comparison with SoTA:** The paper lacks comparison with state-of-the-art methods, which is essential to gauge the true effectiveness of the proposed approach.\n\n**-- Questions about Methodology:** Several questions arise concerning the methodology, including the choice of evaluation metrics, the contradiction in observations, and the real necessity of CQ generation for end-to-end performance. These need to be clearly addressed to strengthen the paper (see questions for more details).\n", "questions_to_authors": "1. While CQ improves user preference (as shown in experiment 1), 33% of people preferred not to generate a CQ. Could this indicate the generated CQs could be misleading or incorrect? Why do human annotators do not prefer those CQs? It would be beneficial to understand the underlying reasons for this preference.\n\nAnd at the very least, generating CQs should ideally (at least) not alter human preference, as more information is provided compared to only AQ and DQ provided. However, the fact that human preference decreases by 33% when generating CQs raises questions about the necessity and effectiveness of producing these CQs.\n\n2. The paper relies on large language models such as ChatGPT to generate CQs. I am curious about the ability of a model like ChatGPT to **directly** solve AQ, instead of using generated CQs to train small language model? Can ChatGPT solve AQ in a form similar to chain-of-thought, for example, a chain consisting of (1) ask CQ (2) generate DQ (3) answer the AQ. To summarize, what is the ChatGPT performance on such task? Could ChatGPT perform better than this complex pipeline?\n\n3. line 394 stated *“since predicted answers for AQ have been shown to be helpful for DQ based approaches”*, however, as shown in Table \n3 and Table 6, it seems *No Answers for AQ* makes better performance than *Predicted Answers for AQ* on the ambiguity detection and end-QA task. Does it mean your observation contradict to the observation from existing literature? Can you explain this discrepancy? Or please correct me if I misunderstand this. \n\n4. Since *category* name are mostly short, so, does the use of BLEU or EM score as evaluation metrics reflect the correctness of generated CQs accurately? How does it align with human evaluations? And also same question as the evaluation of CQs, did authors perform human evaluation or semantic-based evaluation on CQs, since the CQs is much longer than *category*?\n\n5. As shown in Table 5, it seems adding this CQ generation, as an intermediate process, does not help much on end-to-end QA performance, i.e., predicting answers for AQ. So, why do authors think generating CQ is necessary if people still mostly care about the end-to-end QA performance? \n\n6. The proposed method makes good ablation study on using or not using CQ in the question answering generation process. However, the paper does not compare with SoTA performance on AmbigQA. \n\nLeaderboard: https://nlp.cs.washington.edu/ambigqa/leaderboard.html\n\n\nMinor: \n\n7. For CQ generation evaluation, how many references are provided? Do authors construct multi-reference for evaluation to improve the robustness? \n\n8. Do authors try different prompts (as shown in line 239 and line 289) to generate CQ using ChatGPT?", "paper_topic_and_main_contributions": "The paper proposes an innovative approach to handle ambiguous questions (AQ) in open-domain QA by introducing clarifying ambiguous natural questions (CAmbigNQ), a dataset consisting of 5,653 AQs with relevant passages, possible answers, and clarification questions (CQ). Unlike previous methods that resolved ambiguities by **directly** generating disambiguated questions (DQs), the authors present a novel method that first ask CQ before generating an answer. The pipeline consists of three tasks: ambiguity detection, clarification question generation, and clarification-based QA. Experimental results are presented, emphasizing the need for further enhancements.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a", "Typos_Grammar_Style_and_Presentation_Improvements": "n/a"}, "extra_text": {}}, {"review_id": "n6blR5W7cv", "reviewer": "Reviewer_SP89", "summary": "", "questions": "- One key contribution is to generate clarification questions from disambiguated questions. The InstructGPT with few-shot learning shows limited performance. Performance of some intuitive models would be necessary to be reported, such as rule-based approach which extracts new words in disambiguated questions compared to the ambiguous questions and then consider them as \"options\". Other approaches to be considered is to fine-tune BERT for example. Did you conduct them?\n\n- The clarification question follows a fixed template, i.e. “Which [category]: [option1], [option2], ..., or [optionn]?” In human communication, there could be other ways which may be more natural and fluent given a certain context. Why did you choose this template, and how to compare with other possibilities?\n\n- What do you mean by \"reranked related paragraphs\" in line 242 and 504? (Reranked by what?)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for acknowledging as strengths the compelling motivation for the proposed task formulation to handle AQs, a systematic presentation of experiment results, and the release of the dataset and the code.\nFollowing your suggestion, we will provide more analyses of results along with additional experiments:\n\n### **Response to Reject-1**\n>Some experimental design is not convincing. For example, in Sec 6.1, they use BERT model for one setting and BART model for the other, so the difference between these two results may not come from whether the predicted answers are useful or not, since it may come from the different capacity of these two models. Another one is that in Table 5, the experiment with ground truth CQ is required, because this will identify whether the low performance in Table 5 is caused by the QA modeling or the generated CQ quality.\n\nWe appreciate your suggestion for additional analyses and experiments. We first would like to clarify that the choice of different models in Section 6.1 is necessary, given the difference in the approaches to ambiguity detection. BERT (for classification) is used when ambiguity is directly detected through binary classification. BART (for text generation) is used when ambiguity is detected by first generating possible answers and checking whether the number of answers is greater than 1. While we think the former is more natural, the latter is used in related works, so we report the performance as well.\n\nOur initial upper bound case–using CQ generated with ground-truth (GT) answers for AQ–represented a tighter upper bound. We agree with your idea to use GT CQ as a more relaxed upper bound to explain the low performance observed in Table 5. Additional experiments prompting GT CQs with the reader model reveal suboptimal results, which indicates that the low performance in Table 5 is due to the QA modeling. Also, using the SOTA model in the pipeline still shows insufficient performance (See Response to Reject-2 for details) which shows the inherent challenge of our task and further emphasizes the need for qualified datasets and approaches like ours. We’ll include additional experiments in the revision to this effect. \n\n| | CQ finetuned BART | | | |\n|-----------------------------------------------|:-----------------:|:--------:|:--------:|:------:|\n| | Pre. | Rec. | F1 | # Ans. |\n| CQ used to clarify the AQ | | | | |\n| CQ generated with Ground Truth Answers for AQ | 47.5 | 49.5 | 48.5 | 2.5 |\n| Ground Truth CQ | **58.0** | **53.8** | **55.8** | 2.5 |\n\n### **Response to Reject-2**\n>The benchmark experiments are conducted with BERT, BART which don't represent a SOTA performance. These experimental results are not convincing that these tasks are difficult for more SOTA models such as XLNET, GPT-3, GPT-3.5, GPT-4.\n\nWe conducted additional experiments using InstructGPT as the reader model in Section 6.3. We also framed our task as a conversation between the user and the models, using ChatGPT for this purpose. \n\nDespite employing state-of-the-art (SOTA) models, the performance remains suboptimal, which emphasizes the challenging nature of our task. While we acknowledge that not comparing our method with SOTA models may weaken our claims that these tasks are difficult, we want to emphasize our primary focus of our paper: \n1. Proposing CQs as an alternative to DQs for handling ambiguity in ODQA\n2. Providing a qualified dataset for future research \n3. Introducing a framework for generating and evaluating CQs.\n\nGiven these objectives, we chose BART-large for its free availability and public checkpoints. Nevertheless, we will discuss the suboptimal performance of SOTA models in the extra page made available upon acceptance to further underscore the challenging nature of our task.\n\n|               |      | CQ finetuned BART |      |          |\n|---------------|:----:|:-----------------:|:----:|:--------:|\n|               | Pre. |        Rec.       |  F1  |    EM    |\n| AQ revised by |      |                   |      |          |\n| Ground Truth CQ         | 58.0 |      **53.8**     | 55.8 | **35.8** |\n\n|                    ||                InstructGPT             |||\n|---------------|:----:|:-----------:|:----:|:--------:|\n|               | Pre. |     Rec.    | F1   |    EM    |\n| AQ revised by |      |             |      |          |\n| Ground Truth CQ         |  7.4 |   **60.0**  | 13.1 | **43.2** |\n\n\n|            | ChatGPT |          |      |          |\n|------------|:-------:|:--------:|:----:|:--------:|\n|Ground Truth CQ|   Pre.  |   Rec.   |  F1  |    EM    |\n| zero-shot  |   8.0   | **64.5** | 14.3 | **50.8** |\n| four-shot  |   11.3  | **64.0** | 19.2 | **49.9** |\n\nPlease note that the low precision in the SOTA model’s results can be attributed to the models generating answers at the sentence level. Specifically, InstructGPT averaged 27.3 words, ChatGPT in a zero-shot setting averaged 25.4 words, and ChatGPT in a four-shot setting averaged 19.9 words, compared to the gold answers averaging 2.6 words. We also report the Exact Match metric, which checks whether the gold answer is included in the generated sentences.\n\n### **Response to Reject-3**\n> They only consider one fixed template for asking clarification question, which limits the scope of this work.\n\nWe have considered several CQ formats and decided to use the current format as the only format, as it is the simplest format containing all absolutely necessary information (options) that we could think of. We believe this format not only makes the problem easier but also minimizes the amount of text that users need to read. And since it’s the simplest format containing all necessary information, it is actually more general and applicable than a more detailed and tailored format.\n\n### **Answer to Question-1**\n>One key contribution is to generate clarification questions from disambiguated questions. The InstructGPT with few-shot learning shows limited performance. Performance of some intuitive models would be necessary to be reported, such as rule-based approach which extracts new words in disambiguated questions compared to the ambiguous questions and then consider them as \"options\". Other approaches to be considered is to fine-tune BERT for example. Did you conduct them?\n\n\nOur objective with respect to the dataset construction was to somehow construct a quality dataset of CQs, rather than to find and propose an optimal way to do so. Thus, we do not think comparing with other means to construct the datasets is high on the priority list given the limited space. But, we did try several approaches during a pilot study including a rule-based approach, which turned out to be quite noisy. We found the current approach of using InstructGPT with manual editing to be sufficient for constructing the dataset we aimed for. \n\n\n### **Answer to Question-2**\n>The clarification question follows a fixed template, i.e. “Which [category]: [option1], [option2], ..., or [optionn]?” In human communication, there could be other ways which may be more natural and fluent given a certain context. Why did you choose this template, and how to compare with other possibilities?\n\nAs stated in our response to Response to Reject-3, we expect the simplicity of our CQ format to enhance user experience while making the construction process easier. While other formats might offer fluency or naturalness, they could add complexity to the construction process without increasing informativeness, as this is this simple format contains all absolutely necessary information (options).\n\n\n### **Answer to Question-3**\n>What do you mean by \"reranked related paragraphs\" in line 242 and 504? (Reranked by what?)\n\nRelevant passages are retrieved based on the ambiguous questions. For both L242 and L542 cases, AQs are revised by CQs and subsequently disambiguated. Passages are then reranked based on the revised AQs. We used the publicly available sentence cross-encoder model 'MiniLM,' pretrained on the MSMARCO dataset, for this task. This information will be included in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Construct a new open-domain QA dataset for resolving ambiguity by asking clarification questions. This dataset can be used for following work to improve the modeling performance.\n- Demonstrate the advantage of asking clarification questions than disambiguated questions. This effort encourages future work towards generating clarification question.\n- Report solid benchmark performance. These experiments reveal the challenge of these tasks.\n- Evaluate InstructGPT's ability on generate clarification questions from disambiguated questions through in-context few-shot learning. The insufficient performance demands better methods.", "reasons_to_reject": "- Some experimental design is not convincing. For example, in Sec 6.1, they use BERT model for one setting and BART model for the other, so the difference between these two results may not come from whether the predicted answers are useful or not, since it may come from the different capacity of these two models. Another one is that in Table 5, the experiment with ground truth CQ is required, because this will identify whether the low performance in Table 5 is caused by the QA modeling or the generated CQ quality.\n- The benchmark experiments are conducted with BERT, BART which don't represent a SOTA performance. These experimental results are not convincing that these tasks are difficult for more SOTA models such as XLNET, GPT-3, GPT-3.5, GPT-4. \n- They only consider one fixed template for asking clarification question, which limits the scope of this work.", "questions_to_authors": "- One key contribution is to generate clarification questions from disambiguated questions. The InstructGPT with few-shot learning shows limited performance. Performance of some intuitive models would be necessary to be reported, such as rule-based approach which extracts new words in disambiguated questions compared to the ambiguous questions and then consider them as \"options\". Other approaches to be considered is to fine-tune BERT for example. Did you conduct them?\n\n- The clarification question follows a fixed template, i.e. “Which [category]: [option1], [option2], ..., or [optionn]?” In human communication, there could be other ways which may be more natural and fluent given a certain context. Why did you choose this template, and how to compare with other possibilities?\n\n- What do you mean by \"reranked related paragraphs\" in line 242 and 504? (Reranked by what?)", "paper_topic_and_main_contributions": "This paper constructs an open-domain QA dataset (CAMBIGNQ) with clarification questions for resolve the ambiguity. The dataset is built upon an existing dataset (AMBIGNQ). The only difference between these two datasets are that the newly constructed one asks clarification question while the existing one asks disambiguated questions for each possible interpretation. This paper reports the human evaluation on the preference between these two ways for resolving ambiguity in QAs. The results support their arguments that this new setting is more preferable. \n\nIn addition to the dataset, this work establishes benchmark performance on three tasks: ambiguity detection, clarification generation and clarification-based question answering. The experimental results reveal the difficulty of these tasks, thus show the need of this new dataset  for future development.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes a novel dataset for resolving question ambiguity in open-domain QA and proposes an innovative approach: instead of directly generating disambiguated questions (DQs) for every possible interpretation, Clarification Questions (CQs) are generated to be asked to the user before generating the correct answer.  The dataset builds upon the existing AMBIGNQ and adds appropriate clarification questions for each prompt. CQs are generated automatically, then evaluated and revised by human annotators. Human evaluation of the preference between the two settings (i.e. DQ vs CQ) supports the argument that CQ is preferable. Experiments with BART, however, do not report better performance, which indicates the tasks are challenging. The reviewers initially had different positions but ended up in coming closer on soundness after the author rebuttals.   

**Pros.** 

The paper presents a novel approach to a still challenging task in QA; 

It contributes a novel dataset, useful for future developments in dealing with AQ in QA; 

It establishes a benchmark performance on three tasks: ambiguity detection, clarification generation and clarification-based question answering, for easier and better comparison in future works;  

It demonstrates the need for better models/ methos to handle ambiguity in QA; 

It is fairly well written, it states clearly its objectives, and authors seem to have covered the related literature well.  

**Cons:**

The paper lacks an explanation for the use of different models in some experiments (see reviewer SP89 comments). Authors, however, provided satisfactory details in their rebuttal.   

experiments with ground-truth CQs are missing, which would better complement and explain the result in Figure 5.  As, authors already run such experiments, they can easily be added to the paper; 

The benchmark experiments do not include SOTA models, which makes the claim that the targeted tasks are still a challenge less convincing. For their rebuttal authors performed experiments with InstructGPT and ChatGPT that confirm the claim. This should be included in the paper. 

 


It is recommended that the authors integrate the promised additional materials into the final paper.
Decision: Accept-Findings