Model Analysis & Evaluation for Ambiguous Question Answering
Year: 2023

Abstract:
Ambiguous questions are a challenge for Question Answering models, as they require answers that cover multiple interpretations of the original query. To this end, these models are required to generate long-form answers that often combine conflicting pieces of information. Although recent advances in the field have shown strong capabilities in generating fluent responses, certain research questions remain unanswered. Does model/data scaling improve the answers’ quality? Do automated metrics align with human judgment? To what extent do these models ground their answers in evidence? In this study, we aim to thoroughly investigate these aspects, and provide valuable insights into the limitations of the current approaches. To aid in reproducibility and further extension of our work, we open-source our code here.

heading: 1 Introduction
text: Question Answering (QA) has been subject to great progress in the past years, largely thanks to the representational capabilities of modern architectures like the Transformer (Vaswani et al., 2017), but also due to the curation of large, high-quality datasets that enabled the effective training of these models (Joshi et al., 2017; Kwiatkowski et al., 2019).
At the same time, the presence of ambiguous questions has been a challenging aspect of QA. In order to answer such questions, models are required to generate long answers with fluency and cohesion, which is often referred to in the literature as LongForm Question Answering (LFQA). To tackle this, Min et al. (2020) curated the AmbigQA dataset, which contains disambiguations for various questions that were present in popular benchmarks. Extending this work, Stelmakh et al. (2022) selected a subset of the questions and crowd-sourced gold answers that cover all possible interpretations of each question, resulting in the ASQA dataset.
∗Equal contributions †Correspondence email: dinos.ppk@gmail.com
Recently, Krishna et al. (2021) performed a case study on ELI5 (Fan et al., 2019), one of the largest collections available for LFQA, and pointed out several issues that complicate the development and evaluation of suitable models. In particular, the authors questioned whether the retrieved documents are considered by the generative models when producing an answer, and the correlation of the common evaluation metrics with human judgment.
In this work, we aim to investigate whether the baselines set on ASQA by Stelmakh et al. (2022) suffer from the issues pointed out by Krishna et al. (2021), but also to analyze the modeling choices that contribute to performance. Concretely, we set out to answer the following research questions:
RQ1 Does scaling the size of the generative models affect the quality of the generated answers?
RQ2 Can an intermediate round of fine-tuning on non-ambiguous LFQA collections improve performance in ambiguous QA?
RQ3 When comparing models head-to-head, does human judgment reflect the difference in the automated evaluation metrics?
RQ4 Do models base their answers on the retrieved evidence, or could they be hallucinating?

heading: 2 Methodology
text: We design a standard retrieval-augmented system, to identify the dimensions that contribute to disambiguating questions and generating factual answers.

heading: 2.1 The LFQA Pipeline
text: Evidence Retrieval The first step of the pipeline is to identify the documents that will form the basis of the generated answers. Given a question q, we employ a retrieval method R that fetches the top-k relevant documents {di}ki=1. For a document di to be considered relevant, it needs to cover at least
4570
one aspect of q. To completely resolve the ambiguity, the passages1 in the index should suffice to collectively answer all aspects of a given question.
Answer Generation Once the evidence has been collected, we feed the retrieved passages to a generative model to summarize them in a concise answer that disambiguates the question at hand. As is the standard practice in contemporary literature, we opt for a sequence-to-sequence model G that follows an encoder-decoder architecture, which first creates a dense representation of the concatenation of the question and the passages, and then produces the answer by attending to this latent representation:
answer = G([q; d1; . . . ; dk])
with [· ; ·] being the concatenation of ≥ 2 passages.

heading: 2.2 Modeling Choices
text: We expect more sophisticated pipelines to provide better answers for ambiguous questions, and thus we make a distinction based on the complexity of the interaction between the two components.
Naive We implement the QUESTION baseline, which repeats the ambiguous question a few times in order to match the typical length of the answers in the dataset. This is a lower bound on the task, as we are not truly answering the question at hand.
Retrieval-Only In this case, we rely exclusively on a retriever to fetch the top-k passages as a response to the ambiguous question. We experiment with different values of k to evaluate whether using more passages leads to an answer that covers more of the disambiguated questions.2
Sequence-to-Sequence A generative language model is often used to produce a concise response that summarizes all of the disambiguating answers. We analyze three scenarios:
• Closed Book: In the most extreme approach, we assume that the model is not conditioned on the results of a retriever, but rather only on the question itself, and relies on its parametric knowledge (Roberts et al., 2020) to respond. We expect this to significantly harm performance, as the available context to provide an accurate answer is limited.
1We use the terms document and passage interchangeably. 2Employing some form of result diversification has the potential to improve the performance of the retrieval component, but we leave this direction as future work.
• Random Retrieval: In order to verify whether the model grounds its answers on the retrieved documents (Krishna et al., 2021), we design a controlled scenario where we randomly sample passages from our index as evidence.
• Open Book: In the most realistic setting, the model treats the top-k results of a retriever as context to respond appropriately. We expect stronger retrieval methods to lead to more comprehensive answers, as the generative model will be conditioned on more diverse and relevant information.

heading: 3 Experimental Setup
text: We aim to assess whether LFQA systems can generate concise answers that disambiguate the provided questions. In this section, we present the datasets and models used, as well as the evaluation metrics.

heading: 3.1 Datasets
text: We use the ASQA dataset to train and evaluate our systems. It is a subset of the AmbigQA dataset, with long-form answers and additional context for each of the selected samples. More specifically, it contains 6,316 ambiguous questions, with each one being paired to a set of disambiguated questions, the corresponding short answers, and the Wikipedia passages where the answers were found. For each question, the dataset curators crowd-sourced a longform answer that resolves the ambiguity by summarizing all short answers. The annotators provided one reference answer for all train (4,353) samples, and two for all dev (948) and test (1,015) samples.3
Although ASQA is a useful resource for LFQA, when going through the dataset for preliminary analysis, we found cases where the ambiguity was to identify when “last/this year” refers to. We argue that training a model on such samples is counterintuitive, as we generally assume that the information is coming from a fixed snapshot of a knowledge base. We provide a few examples that cover similar types of questions in Appendix A.
Additionally, given the limited size of the ASQA dataset, we follow one of the proposed research directions posed by Stelmakh et al. (2022) and investigate the impact of intermediate fine-tuning on a larger LFQA collection, before training on ASQA. In
3An illustration of ambiguity can be observed in a seemingly straightforward query like “Who was the ruler in France in 1830?”, which presents a challenge due to the existence of two rulers during that period.
particular, we use a processed version of the ELI5 dataset4 that addresses some of the issues raised by Krishna et al. (2021) (226,147 train / 3,020 dev samples), as well as the NLGEN set of the MS MARCO QA dataset (153,725 train / 12,467 dev samples).

heading: 3.2 Models
text: The typical QA pipeline comprises a retrieval and a generative model, which can be pre-trained separately and then fine-tuned on the downstream task. Although this can be done in an end-to-end fashion, we chose to keep the retriever frozen, to avoid re-indexing of the support passages during training, and only train the generative model. We provide detailed training information in Appendix B.
Retrieval Models We experiment with both sparse (lexical) and dense (neural) methods, in order to investigate whether the type of the question encoder has an impact on the relevancy of the retrieved passages. We chose BM25 (Robertson et al., 1995) for the former, and DPR (Karpukhin et al., 2020) for the latter, as they were both supported by the Pyserini (Lin et al., 2021) toolkit and they constitute two of the most explored options in their corresponding fields. In both cases, we use the prebuilt indices of Wikipedia provided by Pyserini to have a common knowledge base that matches the one used by the dataset curators.
Generative Models We use two of the most popular Transformer-based encoder-decoder models, namely T5 (Raffel et al., 2019) and BART (Lewis et al., 2020). More specifically, we experiment with three variants of these models, in increasing parameter count: BART-base (140M params), T5-base (220M params), and BART-large (400M params). By doing so, we aim to verify whether increasing the capacity of the generative model corresponds to an increase in answer quality.

heading: 3.3 Metrics
text: Automated Evaluation Evaluating the performance of generative models is one of the most challenging aspects for LFQA. For recall-oriented QA systems, the most common metric used is ROUGEL (Lin, 2004), which identifies the longest common sub-sequence between the generated answer and a reference (gold) answer. Sequences are penalized proportionally to their length to prevent generating a longer output to artificially increase the overlap
4Available on HuggingFace: lfqa & lfqa_support_docs.
with the reference text. However, a recent study by Krishna et al. (2021) revealed that ROUGE-L did not always correlate with human judgment, pointing to the need for a more diverse evaluation setup. Specifically for ambiguous LFQA, Stelmakh et al. (2022) proposed two metrics to quantify the disambiguation ability of the model:
1. STR-EM (String Exact Match): the fraction of disambiguated answers that the model includes verbatim in its output.
2. DISAMBIG-F1: the fraction of answers that can be deduced with a text comprehension model, using the predicted long answer and the disambiguated question:
DISAMBIG-F1 = 1
N
∑
k
1
n(k)
∑
i
ϕ(y (k) i , ŷ (k) i )
where N is the number of evaluation samples, n(k) the number of disambiguations for the k-th question, y(k)i its i-th ground-truth short answer, ŷ (k) i the predicted short answer5, and ϕ a function that computes the token-level F1 score between them.
Finally, they define another metric, namely DR, which is the geometric mean of DISAMBIG-F1 and ROUGE-L, as an overall estimate of the performance in both disambiguation and answer overlap.
Human Evaluation To better align the model’s performance with the overall satisfaction of the human users that interact with the system, we follow Stelmakh et al. (2022) in creating an anthropocentric evaluation pipeline. Our approach differs in that it performs head-to-head comparisons between different models, aiming to draw conclusions about the design choices that lead to a well-performing LFQA system. For a pair of answers, we compare their comprehensiveness (COMP); whether the answer suffices to understand both the source of ambiguity in the question, and the relation between the individual answers, their fluency (FLUE); whether the answer is coherent and fluent from a human reading stance, and the overall human impression (OVER); which of the answers is prefered overall.

heading: 4 Results
text: Automated Evaluation We evaluate our models on the development set of ASQA using the au-
5Generated from a RoBERTa (Liu et al., 2019) model trained on SQUAD2.0 (Rajpurkar et al., 2018), with the predicted long answer and the disambiguated question as input.
tomated metrics introduced in Section 3.3. Table 1 displays the results for the naive baseline, the retrieval-only experiments using both BM25 and DPR, and the closed/open book experiments with DPR. In cases where we use a retriever, we fetch the top-k relevant documents, for k ∈ {1, 3, 5}. As the level of information contained in the retrieved passages has a direct impact on the answer quality, we perform a short study on the upper bound of retrieval in Appendix C.
Naturally, the naive QUESTION baseline performs the worst, with a ROUGE-L score of 15.3 and a DISAMBIG-F1 score of 0.1, leading to an overall DR score of 1.4. Focusing on the retrievalonly experiments, we observe that DPR consistently outperforms BM25 in all metrics, which is anticipated as semantic matching allows us to retrieve passages that answer different aspects of the question. Remarkably, although the closed-book variants surpass the retrieval-only methods in terms of ROUGE-L, we see the opposite trend for all other metrics. The open-book variants outperform the rest, confirming our hypothesis that augmenting the generative model with a retriever is crucial for performance. We also notice that the increase in performance closely follows the growth in parameter count, implying that [RQ1] model scaling improves the quality of the answer.6 Surprisingly, the impact of fine-tuning on larger collections on performance is marginal, as the disambiguation metrics only narrowly improve. For completeness, we report the intermediate fine-tuning results for the closed-book experiments in Appendix D. In general, we deem that [RQ2] naively fine-tuning on non-ambiguous LFQA datasets has limited added
6To gain further insight into some of the common errors, we showcase a few example model outputs in Appendix E.
value, as the models are not inclined to identify any uncertainty in the meaning of the questions asked.
Human Evaluation We perform a head-to-head comparison between a selection of approaches to determine which modeling aspects contribute more/less to the performance. For each pair, we asked 2 assessors to compare the models blindly using the human evaluation metrics defined in Section 3.3. We report our results in Table 2, where we observe that most comparisons follow the trends we described previously using the automated metrics. Noticeably, when comparing T5-base DPR@3 vs BART-base DPR@3 and DPR@1 vs BART-base DPR@1, human annotators seem to equally prefer both models overall. However, the difference in DR for these pairs is merely a few decimal points, which justifies the inability to distinguish between them. Hence, we confirm that [RQ3] the overall human impression aligns with automated metrics.
Random Retrieval Finally, we evaluate whether the generated answers are grounded on the retrieved passages. Table 3 showcases the performance for the models used in our open-book experiments, and we generally see a decrease in performance when compared to fetching relevant documents. Interestingly, BART-base with random re-
trieval performs better compared to its closed-book variant. As this is the smallest model we tested, we speculate that for LFQA, under-parameterized architectures may be relying more on the question and not the corresponding evidence. Despite this, [RQ4] models for ambiguous LFQA appear to be well grounded in the retrieved evidence overall.

heading: 5 Conclusion
text: In conclusion, by exploring the LFQA field in answering ambiguous questions, we find that the ASQA dataset is a good foundation to develop and evaluate models. We notice that larger generative models produce better answers, with semantic matching for retrieval having a positive impact. To compensate for the small size of the dataset, we experiment with intermediate fine-tuning on larger collections and find that doing so only marginally improves the results. By comparing the performance when using relevant versus random documents, we show the models’ dependency on the provided context. Our human evaluation confirms the general trends we observed using the automated metrics, giving credit to their disambiguating ability.
Limitations
We see two main limitations in our study. Primarily, given the clear trend of larger generative models producing higher quality answers, an obvious question is to investigate whether this continues to be the case indefinitely, or whether it saturates after a critical amount of parameters. Despite this, due to hardware restrictions, we were unable to experiment with models larger than BART-large. Additionally, considering that the field of ambiguous QA inherently requires complementary pieces of evidence, there is no doubt that diversification methods are bound to yield better results in terms of disambiguation quality. In this work, however, we limited ourselves to using a typical neural retriever, shifting our focus toward the factuality and the fluency of the generated answers.
Ethics Statement
As we use a publicly available Wikipedia index as our knowledge base, it is possible that the generated answers may contain some form of bias that reflects the information submitted by the anonymous editors of the website. To prevent this, follow-up work could examine how to detect misinformation or hate speech in indexed passages, before using them as evidence for the generative models.

heading: Acknowledgements
text: This work was carried out on the Dutch national e-infrastructure with the support of the SURF Cooperative. We would also like to thank Evangelos Kanoulas for his valuable feedback and guidance during this project.

heading: C Upper Bound for Retrieval
text: To investigate the efficiency of our retrieval models, we perform an analysis for the upper bound of the retrieved passages’ relevancy to the corresponding gold answers. In particular, for each of the systems used, we count the number of relevant short answers retrieved using an exact string match with the dataset’s gold answers. In Table 4, we notice that even the best retrieval setup at our disposal, namely DPR@5, only fetches ∼ 44% of the relevant answers on average or up to 58% in cases where it identifies at least one relevant passage. This emphasizes the need for a multi-hop retrieval system like JPR (Min et al., 2021) in order to fully utilize the power of the first component of the pipeline. It is evident that a higher upper bound for retrieval will increase the overall performance of the systems that tackle ambiguous LFQA, making it a high priority for follow-up work.

heading: D Effect of Intermediate Fine-Tuning for the Closed Book Experiments
text: Table 5 displays the impact of the intermediate finetuning on a larger LFQA collection in the closedbook setting, using automated evaluation metrics. Similarly to the open-book setting, BART-large ELI5 performs the best, which confirms the hypothesis that larger models are able to benefit the most when given more training data.

heading: E Answer Demonstration
text: We demonstrate a brief selection of answers generated by our best-performing model (BART-large ELI5) in Table 6. We observe that the main reason for our model’s inability to disambiguate all parts of the question is that the passages retrieved with DPR simply do not cover all possible interpretations. In most cases, the first passage retrieved suffices to answer at least one of the disambiguated questions, and the rest passages are either not relevant or answer the same question again. Contrary to the results of Stelmakh et al. (2022), our model does not hallucinate or repeat itself. We hypothesize that since their model (T5-large) is almost twice as big as BART-large in terms of parameter count, it can better utilize its context but at the same time be overconfident in its answers.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Yes, after the conclusion (Section 5)
3 A2. Did you discuss any potential risks of your work? Yes, in the ethics statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? Yes, in Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Yes, Sections 3.1 and 3.2
3 B1. Did you cite the creators of artifacts you used? Yes, Sections 3.1 and 3.2
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We assume models and datasets on HuggingFace have a permissive license to use for research
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We discuss possible misuses of our models in the Ethics section, and we use artifacts purely for research purposes, without discussing it however in the paper’s text
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No, but we mention this as a follow-up work in the Ethics statement
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We use English-only datasets, but the original authors of these works have not specified such information
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes, Section 3.1
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Yes, Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Yes, Appendix B
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No, we ran our experiments for a single seed due to computational budget limitations
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No, but we use the default settings for the packages used
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Yes, Section 4
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No, the instructions were given orally as it was a small cohort
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No, but the assessors were fellow classmates who volunteered to do it
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No, but they were fully aware of the purposes of this research
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No, they are Master’s students from Europe, both male and female

heading: Future Work
text: 2Employing some form of result diversification has the potential to improve the performance of the retrieval component, but we leave this direction as future work. • Random Retrieval: In order to verify whether the model grounds its answers on the retrieved documents (Krishna et al. , 2021), we design a controlled scenario where we randomly sample passages from our index as evidence. • Open Book: In the most realistic setting, the model treats the top-k results of a retriever as context to respond appropriately. We expect stronger retrieval methods to lead to more comprehensive answers, as the generative model will be conditioned on more diverse and relevant information