WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering
Year: 2023

Abstract:
Answering non-factoid questions (NFQA) is a challenging task, requiring passage-level answers that are difficult to construct and evaluate. Search engines may provide a summary of a single web page, but many questions require reasoning across multiple documents. Meanwhile, modern models can generate highly coherent and fluent, but often factually incorrect answers that can deceive even non-expert humans. There is a critical need for highquality resources for multi-document NFQA (MD-NFQA) to train new models and evaluate answers’ grounding and factual consistency in relation to supporting documents. To bridge this gap, we present WIKIHOWQA,1 a new multi-document NFQA benchmark built on WikiHow, a website dedicated to answering “how-to” questions. The benchmark includes 11,746 human-written answers along with 74,527 supporting documents. We describe the unique challenges of the resource, provide strong baselines, and propose a novel human evaluation framework that utilizes highlighted relevant supporting passages to mitigate issues such as assessor unfamiliarity with the question topic. All code and data, including the automatic code for preparing the human evaluation, are publicly available.

heading: 1 Introduction
text: Non-factoid questions (NFQs) requiring long, passage-level answers, such as explanations or opinions, pose challenges for current questionanswering systems. While a few datasets exist for NFQA (Cohen and Croft, 2016; Hashemi et al., 2019; Soleimani et al., 2021), users are currently limited to seeing a summary of the most relevant document in a snippet on a search result page (SERP), which can be insufficient for complex questions that have scattered answers across multiple documents or require sophisticated reasoning
1https://lurunchik.github.io/WikiHowQA/
to generate an answer. However, even relatively straightforward questions like “how to fix my computer issue” can have multiple solutions, requiring the user to manually search through multiple retrieved documents to find the one that applies to their situation. Complex questions such as “how to feel calm and relaxed” may require an aggregated summary of the most popular methods from multiple relevant sources. There are also questions for which answers have yet to be written, requiring a QA system to treat relevant documents as initial sources of information and then to reason out an answer based on them. For example, the question “how to prepare to buy a house in [neighbourhood], [city]” would require the system to retrieve relevant documents about buying houses in general and specific articles about the local house market and the neighbourhood, and then construct a more sophisticated answer through reasoning rather than just summarizing multiple documents.
To address these challenges and move towards more advanced QA systems that can provide indepth and comprehensive answers to a wide range of questions, we propose a new benchmark for the task of multi-document non-factoid QA (MD-
5291
NFQA). This task involves using multiple relevant documents to generate a complete and coherent answer to a given NFQ. We focus our benchmark on the INSTRUCTION category of NFQs, which often begin with “how to”. These questions are underrepresented in current multi-document QA (Bolotova et al., 2022) datasets, despite their popularity, as evidenced by a 140% increase in “how to...” searches on Google since 20042 and making up over half of the most searched queries.3 Our benchmark aims to fill this gap by specifically targeting INSTRUCTION questions.
WikiHow is a web-resource for INSTRUCTION questions that contains over 235,000 articles on a wide range of topics. These articles provide comprehensive step-by-step instructions and are written by a community of experts and reviewed by an average of 16 people. The WikiHow website has proven to be a valuable resource for machine learning tasks (Koupaee and Wang, 2018; Yang et al., 2021; Bhat et al., 2020; Zellers et al., 2019; Ladhak et al., 2020; Boni et al., 2021; Zhang et al., 2020; Cohen et al., 2021; Anthonio et al., 2020), and in our work, we leverage WikiHow to create a high-quality benchmark specifically designed for MD-NFQA within the INSTRUCTION question category. Our benchmark consists of 11,746 questions from the INSTRUCTION category, each paired with a corresponding human-written answer, sourced from a diverse range of WikiHow articles. Each QA pair is supported by corresponding parsed relevant HTML pages from which the answer can be derived (Fig. 1). We evaluate several baseline models on the new benchmark that could serve as lower and upper bounds for model performance.
Human annotation is often used as the standard for evaluating long-form answers, but research has shown that individuals without specific training can only distinguish between human-generated and auto-generated text at a level equivalent to random chance (Clark et al., 2021). This presents a challenge for evaluating the new benchmark for NFQA, particularly when the answers are lengthy or the topic is unfamiliar (Krishna et al., 2021) or not interesting (Bolotova et al., 2020). Given that standard metrics struggle to detect factual inconsistencies, such as number swapping, negation, etc. (Kryscinski et al., 2019), we delve into human sensitivity to such discrepancies by conducting a
2https://bit.ly/telegraph-google-how-to 3https://bit.ly/most-asked-questions-on-google
series of crowdsourcing experiments. Inspired by the elaborate yet resource-intensive manual evaluation framework of abstractive models proposed by Dou et al. (2022), we introduce a simple human evaluation framework that leverages highlighted relevant passages to enhance the quality of NFQA assessments. Our findings demonstrate that incorporating highlighted relevant supporting passages into the evaluation process not only aids evaluators in understanding the context of each question and answer but also contributes to the factual accuracy of the evaluation. As a result, we integrate this method into the evaluation of the newly proposed benchmark.
The new benchmark for MD-NFQA serves as a valuable resource for the development of more advanced QA systems that can provide in-depth and comprehensive answers grounding information in supporting documents. By including parsed passages from relevant HTML pages and providing human judgments, our benchmark also has the potential to facilitate research in evaluating the factual correctness of long-form answers. All data is publicly available4 on the dataset website.1

heading: 2 Related Work
text: This section covers datasets for related MDNFQA tasks such as long-form question answering (LFQA) and query-based multi-document summarization (QF-MDS), highlighting the scarcity of resources available. We examine the strengths and limitations of these resources, as well as the differences between these tasks and MD-NFQA. LFQA: The task of LFQA, introduced by Fan et al. (2019), involves providing long answers to open-ended questions. While the associated ELI5 benchmark includes supporting “document” to generate answers, the LFQA task, unlike MDNFQA task, does not assume that an answer is scattered across multiple documents. Moreover, while MD-NFQA exclusively targets NFQs, ASQA LFQA dataset (Stelmakh et al., 2022) addresses the need for long-form answers to ambiguous factoid questions. Similarly, Natural Questions dataset (Kwiatkowski et al., 2019) contains both long and short answers for factoid questions (Xu et al., 2022).
The ELI5 dataset (Fan et al., 2019) for LFQA includes 272,000 questions from the “Explain Like I’m Five” Reddit web forum, where questions and
4Under RMIT University DTA license for research use
answers must have a score of at least two to be included. The top-voted answer for each question is considered the correct answer, and the supporting content for each question is generated by extracting sentences with high tf-idf similarity from the top 100 web pages that match the question from the Common Crawl corpus. In contrast, our proposed MD-NFQA dataset uses texts parsed from relevant HTML pages chosen by the author of the corresponding article on WikiHow, rather than relying on automated methods for curating reference content. In addition to a lack of grounding in supporting content, Krishna et al. (2021) found significant Train/Valid overlap in the ELI5 dataset, and highlight challenges with both automatic answer evaluation and human annotation for this benchmark. In our proposed benchmark, we address these issues through the use of a novel human-evaluation framework utilizing supporting documents and by avoiding question overlap (Sec. 5 and Sec. 3).
Nakano et al. (2021) used questions from the ELI5 dataset and collected new answers from human annotators who were instructed to search for related documents and use them to construct their answers. They trained the WebGPT model on that dataset to answer long-form questions by mimicking the way humans research answers to questions online: it searches and navigates the web to find relevant pieces of information and concludes an answer based on them by citing sources for factual accuracy. Authors report that the model answers are preferred by assessors 69% of the time to the best human-written answer from Reddit when evaluating on ELI5. While the WebGPT dataset is valuable, as of this moment, the authors have only released the model’s answers and questions, not the supporting documents.
Finally, it’s worth noting that while REASON and EVIDENCE-BASED categories of NFQs prevail in the ELI5 (Bolotova et al., 2022) and WebGPT datasets, our dataset focuses specifically on the INSTRUCTION category.
QF-MDS: In contrast to MD-NFQA, which involves generating detailed passage-level answers to NFQs, QF-MDS (Tombros and Sanderson, 1998) focuses on creating concise summaries in response to specific queries, which may include factoid questions or queries not in question form, such as [Entity], [Event], etc. This requires a different set of skills and approaches compared to MD-NFQA, as QF-MDS summaries do not involve making con-
clusions or inferences based on the provided information, while MD-NFQA requires higher level reasoning and synthesis to provide complete and accurate answers. While there are a few resources available for MDS (Litkowski, 2004; Angelidis and Lapata, 2018; Liu et al., 2018; Dang, 2006; Fabbri et al., 2019; Ganesan et al., 2010; Wang and Ling, 2016; Yasunaga et al., 2019; Koupaee and Wang, 2018; Lu et al., 2020) and QFS (Zhong et al., 2021; Nema et al., 2017; Zhao et al., 2021), they are scarce for QF-MDS.
The QMDSCNN and QMDSIR datasets target QF-MDS task (Pasunuru et al., 2021). The first dataset is derived from CNN/DailyMail having real summaries with simulated queries, while the second dataset is derived from a search engine query log and has simulated summaries with real queries.
Another automated approach for curating large datasets for query-focused summarization tasks is AquaMuse (Kulkarni et al., 2020). This dataset supports both abstractive and extractive QF-MDS tasks. Queries and long answers from the Natural Questions dataset (Kwiatkowski et al., 2019), and a pre-processed version of the Common Crawl corpus were used (Raffel et al., 2020). Long answers from Natural Questions and the Common Crawl corpus are encoded into sentence embeddings. Then a similarity search is performed over the corpus and long answers to find candidate documents from the corpus for QF-MDS tasks (similar to ELI5).
Boni et al. (2021) proposed HowSumm, a QFMDS dataset automatically constructed from WikiHow content by utilizing the referenced articles as the summarization source, the corresponding elements of WikiHow articles as the target summaries, and titles used as the queries. There are two types
of QF-MDS tasks, one for methods (HOWSUMMMETHOD) and one for steps (HOWSUMM-STEP). While this work is similar to the dataset we present, HowSumm is directed toward the QF-MDS task, does not discuss train-test overlap, and only includes the source URLs in their dataset.

heading: 3 Resource Description
text: This section presents our new resource, including its construction, text statistics, comparison with other benchmarks, and thorough quality analysis.

heading: 3.1 Data Collection
text: Fig. 2 illustrates an example of a WikiHow article accompanied by a high-quality, human-written article summary. These summaries are equivalent to shorter passage-level answers to a question. To build WIKIHOWQA, we first downloaded over 236,000 articles published on WikiHow before January 2022 using the MediaWiki API. We then filtered out articles without references, resulting in a collection of 126,711 articles. Among these articles, only about 20% had human-written article summaries, which we used as target answers. We downloaded the HTML content of all cited URLs using the Wayback Machine to provide the version closest to when the article was created or modified. The final HTMLs were saved from the snapshot versions. We simplified the HTML source code, and processed the content in two ways: (1) by extracting text content or (2) by converting it to Markdown format. The latter format preserves formatting such as tables and lists. Finally, we rejected articles with missing HTML snapshot links, empty HTML, or empty extracted text. Each instance in our new dataset consists of a question, a human-written article summary treated as the target answer, and a list of parsed texts from the relevant HTML documents cited by the article’s author. Fig. 1 shows a simplified example from WIKIHOWQA. Avoiding Overlap in Train-Test: When studying state-of-the-art model performance on ELI5 (Fan
et al., 2019), Krishna et al. (2021) observed little to no evidence that analyzed models grounded answer generation in the retrieved documents. They attributed this to a significant overlap (around 43.6%) in the training, validation, and test sets of ELI5. Similarly, Lewis et al. (2021) identified train-test overlap in various QA datasets. To avoid this issue in our new resource, we explicitly split questions in a way that no same-topic or paraphrased questions ended up in different splits. To do so, we clustered all questions prior to splitting and assigned all questions belonging to the same cluster to the training split. We only assign clusters to the training split so that the validation and test splits will have more varied questions, to avoid a potential evaluation bias due to the presence of many similar questions.5

heading: 3.2 Dataset Statistics
text: The WIKIHOWQA dataset include a diverse range of topics, 19 in total, with the most frequent being “health”, “home and garden”, “pets and animals”, and “computer and electronics”.6 Table 1 compares WIKIHOWQA to other QF-MDS and LFQA datasets. ELI5 is the largest dataset in terms of questions and has the lowest average words in reference documents (recall that the single reference document is an aggregate of top-k passage retrieval and hence the absence of some of the reported statistics within Table 1). The number of answer words is higher than other datasets except for HOWSUMM-METHOD. AQUAMUSE has
5Implementation details can be found in App. B. 6The full category distribution is provided in App. A
fewer questions, although the reference sources and answer content are most similar to the statistics of HOWSUMM-STEP and WIKIHOWQA. The HOWSUMM-METHOD dataset statistics are unique, it has 11 source documents per instance on average, and the length of answers are greater with an average of 31 sentences per instance. HOWSUMMSTEP also has a high number of source documents while the remaining statistics follow a similar trend to the other datasets described. While our proposed dataset WIKIHOWQA shares many common statistical characteristics of other datasets, the construction process, source-target mappings and task applications differ as discussed in Sec. 2.

heading: 3.3 Quality Verification
text: Since the relevant passages in our WikiHowQA datasets are sourced from web pages cited by the WikiHow article author, they may not contain the exact words or phrases from target answers. Therefore, to verify the quality of our benchmark, we assess the feasibility of constructing an answer from the given supporting documents. As this is a crucial aspect for a MD-NFQA resource, we evaluate our benchmark using both automatic metrics and thorough human evaluation. Automatic metrics: The relevance of the supporting documents in our WikiHowQA dataset is first evaluated using the Novel N-Gram Percentage metric (See et al., 2017) as a measure of word intersection between answers and documents. This metric, commonly used for summarization datasets, allows for a strong upper bound comparison. Results presented in Table 2 show that the passage answers in our dataset correspond to the content of the supporting documents with a relatively high n-gram percentage score, similar to that of HOWSUMMMETHOD. We also include results for the Document Understanding Conference (DUC) as an upper bound as it is a high quality human crafted summarization dataset (as reported by Fabbri et al.).
Secondly, we report the average coverage, density and compression metrics for WIKIHOWQA (Grusky et al., 2018). These metrics are commonly used to characterize the quality and difficulty of summarization tasks. We follow Fabbri et al. (2019) who adapted these measures for the MDS use case. The coverage, density and compression scores for WIKIHOWQA are 0.89, 1.86 and 81.46 respectively. It is important to note that Tejaswin et al. (2021) and Bommasani and
Cardie (2020) propose additional metrics to compare datasets, some of which are a reformulation of those presented by Grusky et al. (2018).
The above metrics give a general indication of the dataset quality, although even when examples from our dataset are deemed lower quality, for example when they have low semantic similarity, we find that there are instances where an answer can typically still be constructed from the supporting documents. Fig. 1 demonstrates that answers can be successfully constructed from documents even if they have low semantic similarity. For instance, in the first sentence of an answer for the question “How to cook pork tenderloin” (Fig. 1), although the target length of 55 minutes was not mentioned in any document, it can be inferred by adding up roasting time of 30 minutes on one side and 25 minutes on the other side. Similarly, the suggestion “flipping it occasionally” in the last sentence can be rephrased from “turning every 1 1/2 to 2 minutes”. Human evaluation: To verify our observations and ensure the quality of our dataset, we conducted a crowdsourcing study7 with the goal of verifying the feasibility of answer construction. Each participant task in the study consisted of a QA pair, one supporting document, and three evaluation components. The first evaluation required annotators to classify the question into the appropriate category from a provided list. This step served as an attentiveness check. Annotators were presented with four categories in a random sequence, one
7 All our crowdsourcing studies were reviewed and approved by the Human Research Ethics Committee of RMIT University. We use Surge AI as our data labeling platform, which provides a workforce of highly skilled and educated native speakers, ensuring high-quality data labeling at scale, allowing for higher quality labeling compared to traditional platforms such as Mechanical Turk.
of which was the original category derived from the WikiHow website, while the remaining three were randomly selected from the other 18 categories. This task was designed to ensure that the annotator was paying attention and had read the question carefully. The tasks were rejected if the classification was incorrect. The second evaluation required annotators to read the answer and the supporting document, and use a provided highlighting tool to mark any words or phrases in the supporting document that were directly relevant to the given answer, or could be used to reason or conclude it. In the final step of the process, annotators evaluated the relevance of the provided documents to the answer on a five-point Likert scale (from 0, “Very Irrelevant”, to 4, “Very Relevant”). In total, 31 randomly selected QA pairs and their associated supporting documents were annotated, resulting in 104 participant tasks, with only four being rejected. Each assessment was carried out by three workers, receiving compensation of $0.7 per completed task. The study8 involved 22 fluent English speakers. Fig. 3 illustrates a randomly selected result of this annotation process, as captured in the (simplified) annotation interface used in the study.
The results showed that the workers deem the supporting documents to be mostly “Somewhat Relevant” in relation to the given answers, with a mean document relevance score of 2.59. The innerannotator agreement is moderate, with a Cohen’s Kappa (Artstein and Poesio, 2008) value of 0.51, indicating a reasonable consistency in the annotators’ assessments of document relevance. In terms of related text selections within the documents, the mean overlap score between assessor pairs was 0.65, calculated using the overlap coefficient (Vijaymeena and Kavitha, 2016):
Overlap(H1, H2) = |H1 ∩H2|
min(|H1|, |H2|) where H1 and H2 are the sets of unique words from
8Interface with exact annotation instructions in App. C.
two annotations. This is consistent with previous similar research (Qu et al., 2019; Bolotova et al., 2020) and indicates that the annotators had good agreement with each other on the parts of the documents that could be used for associated answer construction. Overall, these findings demonstrate that the provided documents are relevant and contain the necessary information for the answers.

heading: 4 Baseline Models
text: This section describes WIKIHOWQA baseline models. The task is to generate an answer to a NFQ grounded in a set of relevant documents from which the answer can be reasoned or concluded.
As a reference point for performance evaluation, we use the Retrieval-Augmented Generation (RAG) (Lewis et al., 2020b) architecture, which is a common choice for abstractive summarization and QA tasks (Fan et al., 2019). Since our dataset provides a small predefined set of relevant supporting documents for each question, we use a retriever to filter out passages from those documents that are less relevant to the question, to help fit more relevant passages into the limited input of a generator. Then, a generator is tasked with generating an answer conditioned on input passages. We experimented with DPR (Petroni et al., 2021) as the retriever and BART-large (Lewis et al., 2020a) as the generator.
When building model input, we first ranked passages from supporting documents based on their relevance to the question using DPR, then truncated them to retain the maximum number of complete passages in the ranking that fit withing the maximum input length of BART. Passages were then reordered based on their source documents, and combined into a single input string of the format “<q>question</q><d><p>doc 1 passage 1</p><p>...</p></d><d><p>doc 2 passage 1</p>...</d>”. Finally, BART is trained9 to
9Training and decoding parameters are in App. D.
generate gold answers conditioned on this context. To establish a stronger baseline, we also evaluated the performance of the GPT-3 (Brown et al., 2020) variant text-davinci-00310 in a zero-shot setting. As a model at least 400x times larger than BART, it forms an upper bound for model performance. We compared two prompt kinds for GPT-3, one with the question and another with the question and relevant passages retrieved via DPR.
Standard automatic evaluation metrics for abstractive QA, Rouge-X and BertScore, are reported in Table 3. However, these scores are known to poorly reflect actual model performance (Deutsch and Roth, 2021; Krishna et al., 2021). While recent QA-based evaluation metrics for summarisation seem to better correlate with human judgements (Deutsch et al., 2021; Scialom et al., 2021), their adaptation to NFQA is not straightforward and remains an important area of research, as answers in NFQA are expected to contain facts not mentioned in supporting documents. Instead, we report the results obtained through our human evaluation framework as a more accurate measure of model performance (Sec. 5.2).

heading: 5 How To Evaluate How-To Answers
text: To further ensure the reliability of MD-NFQA evaluation, we introduce a simple human evaluation framework, which we then employ to assess the performance of baselines in our benchmark.

heading: 5.1 Evaluating Human Evaluation
text: Initially, we employed a conventional human evaluation approach, as described in previous studies (Fan et al., 2019; Krishna et al., 2021), where assessors are presented with both model-generated and gold standard answers in a randomized order and asked to select their preferred response. However, our findings indicated that, even when presented with gold standard questions and evaluated by high-performing assessors, simpler modelgenerated answers were frequently (48% of the time) preferred over reference answers, despite containing factual inaccuracies. Manual inspection revealed that choosing between two well-formulated answers was challenging for participants, in line with previous research of Krishna et al.; Clark et al.. To quantify the ability of annotators to identify factual inconsistencies in answers, we conducted a
10https://beta.openai.com/docs/models/gpt-3
crowdsourcing7 evaluation experiment.11 To facilitate this, we generated a set of modified answers by deliberately introducing factual errors into a subset of the WIKIHOWQA. This was achieved by randomly selecting 5 QA pairs from each of the 19 question categories available on WikiHow and systematically incorporating various types of inaccuracies, as outlined in Table 4, into 4 out of the 5 answers, resulting in an average of 4.5 modifications per answer. We then manually reviewed the deteriorated answers to ensure they maintained both contextual relevance and grammatical correctness. Following this, we conducted two separate evaluation rounds involving a total of 34 workers, who assessed all 95 QA pairs. It’s important to note that no worker participated in both rounds, ensuring an unbiased evaluation. The first trial aimed to establish the baseline performance of the standard evaluation framework, in which assessors evaluate answers without the assistance of any on-screen relevant information. Each HIT included one QA pair where the answer may had been deteriorated. Similar to Sec. 3.3, HITs began with the attentiveness test of question category classification. Assessors then indicated their familiarity with the question and evaluated the usefulness of the answer on a five-point Likert scale (from “Very Unfamiliar” / “Very Useless” to “Very Familiar” / “Very Useful”). Finally, they were tasked with highlighting spans within the answer that they found either useful or misleading, thereby providing a more nuanced understanding of the answer’s perceived value. The second trial featured the same evaluation tasks, except this time assessors were provided with highlighted relevant passages for each sentence in the answer, accessible by clicking on the sentence. Top-ranked passages according to DPR (Sec. 4) were selected for each sentence, ensuring the overall length is under 5000 characters to fit on the screen. Passages were displayed in their original order, grouped by source documents. Passage highlighting came from the model proposed by Bolotova et al. (2020), designed to simulate user gaze during NFQA evaluation. Assessors could freely examine the passages while determining the usefulness of the answer and selecting useful or misleading spans in the answer.
The choice of this presentation form was based on several factors. Firstly, it allowed for a more in-
11For interfaces and implementation details refer to App. E
Table 4: Examples of deterioration types
Deterioration Type Original Sentence Deteriorated Sentence
Number Swap Sauté the onions for 5 minutes. Sauté the onions for 15 minutes. Sentence Negation Turn the vehicle off and open the hood. Don’t turn the vehicle off and open the hood. Antonyms Swap Adjust your iron to hot for linen. Adjust your iron to cold for linen. Entity Swap As a rabbi, you’ll train in a branch of Judaism. As a rabbi, you’ll train in a branch of Christianity.
teractive and engaging experience for the assessors, enabling them to delve deeper into the context of each sentence. Secondly, it provided a clear visual cue to the assessors about the relevance of each sentence, thereby facilitating a more accurate and efficient evaluation process. Given that assessors often face challenges due to their unfamiliarity with the topic or potential distractions, we aimed to enhance their ability to assess with higher accuracy by providing relevant passages as supporting information. This approach, designed to be simple yet effective, prioritized the provision of relevant information over more elaborate feedback mechanisms like clicks or gaze tracking, which are potential areas for future research. After comparing different forms of passage presentation through trial runs and discussions among the authors, we found that the form offering easy accessibility by clicking on the sentence was the most effective.
As a measure of the ability to spot factual inconsistencies, we calculated the inconsistency detection rate (IDR), defined as the average percentage of identified deteriorations. A deterioration was marked as identified if it was highlighted as part of a misleading span. Results demonstrate a statistically significant difference (Student’s t-test, t − statistic = 2.57, p < 0.01) in IDR between the first and the second trials, with the IDR scores of 0.21 and 0.35. In addition to this, we also calculated the number of False Positive IDRs. False positives were defined as answers that were marked as being misleading by an assessor, even though they were not. The first trial included four false positives, and the second trial two.
While question familiarity scores were consistent between the two trials, IDR improvement was persistent and statistically significant (Student’s t-test t − statistic = 2.18, p < 0.05) in the second trial for both high and low familiarity. This suggests that participants were able to assess more effectively with an access to relevant information sources, regardless of their familiarity with a particular question. As shown in Fig. 4, there was a general improvement in detec-
Figure 4: Inconsistency detection rate (IDR) by type
tion of almost all deterioration types in the second trial. Notably, the detection rate for sentence negations and number swaps increased by 131% and 158%, respectively. These increases were statistically significant, as confirmed by Student’s t-tests (t − statistic = 4.2, p < 0.01 for sentence negations; t − statistic = 3.18, p < 0.01 for number swaps). A detailed explanation of the answer deterioration process, including examples, as well as the interfaces utilized in the human study trials, is provided in App. E.

heading: 5.2 How Good Are The Baselines, Really?
text: We evaluated the effectiveness of lower- and upperbound models from Sec. 4 using our human evaluation framework. Specifically, we compared models through A/B testing (Krishna et al., 2021) while incorporating highlighted relevant passages as reference sources of information. After classifying the question category as an attentiveness task, assessors were instructed to evaluate the usefulness of a pair of answers – one gold and one model – and select the more useful answer. Answers were presented in random order without disclosing the source. We provided ranked relevant passages for each sentence in each answer separately, to allow participants to make more informed decisions about factual consistency.
In this experiment, 28 workers evaluated gold answers to 100 random test questions against answers from DPR + BART, text-davinci-003, and DPR + text-davinci-003. To eliminate potential bias, we ensured that workers did not evaluate the
same question twice. Each HIT was compensated with $0.5.
Human evaluation results are available in Table 3. All differences are statistically significant (Student’s t-tests, p < 0.01). In comparison with the initial attempt at human evaluation, the simpler DPR + BART model scored lower, while still being preferred over gold answers in some cases. Unlike the automatic evaluation, human evaluation supports our initial observation that the quality of answers from text-davinci-003 is much higher than from BART. When prompted with supporting passages, text-davinci-003 generates significantly better answers based on human judgement (Table 3), which highlights the value of reference documents in WIKIHOWQA. While text-davinci-003 mainly generated grammatically plausible and logicaly sound answers, our manual analysis revealed that it frequently failed at factual consistency and overall coherency, especially for questions that require a step-by-step instruction. Coupled with the fact that assessors still preferred answers from text-davinci003 in some of these cases, the proposed evaluation framework should be further improved to guide assessors better in their judgement. The problem is twofold; first, provided relevant passages do not always include the required information to fact-check the answer; second, the presentation and fluency of model answers misleads assessors even when the overall instruction is impossible to follow. This requires an additional skill-set for assessors, and we leave the research of a better evaluation framework specifically tailored to instructions for future work.

heading: 6 Conclusion
text: In this work we have presented a new dataset and benchmark for multi-document non-factoid question answering, WIKIHOWQA, sourced from the WikiHow website and consisting of passage-level answers to “how to” questions. Our released test collections fills a critical gap in currently available resources and evaluation testbeds for multidocument non-factoid QA, a widely occurring information task. We have also presented baseline performance benchmarks, and introduced and employed an information-augmented human evaluation framework that improves the reliability of QA annotations. Still, manual failure analysis revealed remaining challenges in the evaluation of convincing but factually incorrect model answers.

heading: Acknowledgments
text: We extend our deepest gratitude to Mr. Luke Gallagher for his invaluable contributions to this study. His insightful comments and contributions have greatly enriched our work. We also wish to acknowledge the financial support provided by the Australian Research Council (Grant No. DP180102687), which has been instrumental in facilitating this research.
Limitations
In this section we discuss possible limitations of our work, and present interesting avenues for future investigation. First, not all documents in WIKIHOWQA are equally useful; some documents are overall less relevant, some contain very relevant bits alongside generally irrelevant information. More accurate passage or span selection may be required for models to generate better answers.
Another limitation is the focus on highlighted relevant passages as the method of aiding human evaluators in assessing factual correctness. While this approach helped to identify some factual inconsistencies and was proven to also make assessments faster in previous research (Bolotova et al., 2020), additional techniques should be considered for their potential to deliver further improvements.
Unfortunately, due to limitations of the evaluation interface used, we were not able capture how frequently annotators clicked on sentences to see aiding passages. Click ratio data could have yielded valuable insights into when and how often assessors referred to the provided information.
While we considered a range of answer deterioration types, it does not fully align with the kinds of hallucinations that neural NFQA models may produce in the wild. Further research is needed to analyse the robustness of these models in real-world scenarios. One direction is to adopt the comprehensive evaluation framework of abstractive neural models proposed by (Dou et al., 2022) for NFQA.
In our baselines, we experimented with text parsed directly from the HTML source code. However, it is important to note that we also provide the option of using the Markdown files, which preserve formatting information such as tables and lists. This could be useful for certain cases where formatting is important. Furthermore, some reference HTML pages contain pagination and long comment sections that we did not consider when scraping the data.
Finally, the evaluation of passage-level QA remains a challenging task, both for human and automatic evaluation frameworks. Difficulties include accurate assessment of factual correctness and overall consistency, especially when answers require complex reasoning based on multiple sources. Advancements in automatic evaluation of similar tasks, such as summarisation, could be adopted for NFQA.

heading: A WIKIHOWQA dataset
text: Fig. 13 presents the distribution of categories and subcategories in a sunburst diagram. The inner ring represents the main categories, with the outer rings displaying the corresponding subcategories. The size of each segment in the graph represents the percentage of questions belonging to that category or subcategory, with the corresponding number of questions.

heading: B Data Processing Details
text: Fig. 5 illustrates a typical human-written summary on WikiHow. Often, target answer summaries end with a sentence encouraging a reader to study the whole article, such as “To learn more, such as how to wash unfinished leather or use a washing machine, keep reading the article!”. To get a shorter, more answer-like versions of summaries, we filtered out these sentences using a set of patterns.
To simplify HTML source code of downloaded related documents, we used ReadabiliPy, a Python wrapper for Mozilla’s Readability.js12 package13. We then processed HTMLs in two ways: (1) by running it through html2text14 and saving the output in Markdown15 format; and (2) by utilising newspaper3k16 to extract text content directly. The former preserves formatting information useful when working with tables and lists, while the latter produces raw text that is simpler to use and analyse. Additionally, we split each Markdown file into a list of passages, and used library-link to merge all lists and tables into one passage. All titles were treated as separate passages to filter them out if necessary.
To avoid question overlap, HDBSCAN (Campello et al., 2013) was used as the clustering algorithm due to its robustness to noise. Questions were embedded using Multilingual Universal Sentence Encoder (MUSE) (Lee and Chen, 2017). “min_cluster_size” and “min_samples” were set to 2 for HDBSCAN, with the default values used for other parameters.
12https://github.com/mozilla/readability 13https://github.com/alan-turing-institute/
ReadabiliPy 14https://github.com/aaronsw/html2text 15https://daringfireball.net/projects/markdown 16https://newspaper.readthedocs.io/en/latest/

heading: C Dataset Validation Study
text: Figure 6 presents the interface used in our human evaluation study to assess the feasibility of constructing answers from the provided supporting documents in our WIKIHOWQA. The example assessment shown in the figure is a random selection from the validation data collected from the workers. The detailed instructions are shown in Figure 6. The short task description presented to the workers was as follows: “Welcome to our study! We are interested in understanding how well "supporting documents" can be used to answer a question. Thank you for participating in the study. Please review the participant information form before proceeding: (App. G).”

heading: D Model Training
text: Table 5 show significant hyper-parameters used to train the BART model and Table 6 contains the hyper-parameters for decoding. The number of training epochs was controlled through early stopping based on the validation loss (typically 2-4 epochs).

heading: E Evaluating Human Evaluation Study
text: In this section, we first provide more details on the answer deteriorating process and then describe the interfaces used in both human study trials on evaluation assessors’ ability to identify factual inconsistencies.
E.1 Answer Deteriorations Here we describe the methods used to create a diverse set of incorrect answers for our human evaluation experiment. Specifically, we detail the various types of factual inconsistencies we introduced, inspired by text transformations proposed by Kryscin-
ski et al. (2020) in their research on factually consistent models for abstractive text summarization. These include number swap, sentence negation, antonyms swap, entity swap, and paraphrasing.
• Number Swap: Replacing a numerical value in the original sentence with a different value, as shown in the provided example (Table 4) where the sauté time was changed from 5 minutes to 15 minutes.
• Sentence Negation: Rephrasing the original sentence using negations and different grammar structures, altering the meaning of the sentence and making the answer factually inconsistent, as shown in the provided example (Table 4) where the instruction to cool the pouch before removal was negated.
• Antonyms Swap: Replacing an adjective in the original sentence with its antonym, as shown in the provided example (Table 4) where the heat setting on the iron was changed from hot to cold.
• Entity Swap: Replacing a proper noun or named entity in the original sentence with a different entity, potentially altering the context of the sentence and making the answer factually inconsistent.
Note that the above methods were used to generate a large number of incorrect answers, but they were not always successful in producing grammatically or semantically correct answers. Therefore, we also manually reviewed and edited each generated answer to ensure that they were both contextually plausible and grammatically correct. For all deteriorations types, to identify adjectives, entities, numbers auxiliary verbs and Lexical verbs, we used the Spacy NLP library.
For Antonym Swap, for each identified adjective in the text an antonym is produced using the WordNet Lexical database, then the adjective in text was replaced with the randomly selected antonym. Due to the noise, and randomness, sometimes antonyms were chosen which did not best fit the sentence. Whilst manually overlooking the deterioration, these were sometimes removed or changed to a better option. For Number Swap, the digit entity is replaced at random in the code, however after manually overlooking the deterioration, the numbers were adjusted to make more contextual sense, or be harder to distinguish. For Sentence Negation, lexical verbs had “don’t” or “doesn’t” placed before them, whilst auxiliary verbs were replaced with their negation. This method at times failed to make grammatical or common sense after the deterioration for complex sentences, containing multiple clauses. In these cases, an entire sentence could have been slightly adjusted in order to accommodate the negation. For Entity Swap, named entities, often identified manually, were replaced to appropriately fit the context, but also make the answer incorrect.
E.2 Study Trials
Figure 8 illustrates an interface of the first trial run for the study with a random annotation example which happens to have an example of a zero IDR score where an assessor failed to find any deterioration. The exact instructions are presented in the Figure, while the overall task description is as follows:
“Welcome to the question-answering evaluation study! In this study, you will be asked to evaluate the quality of question-answer pairs. All of the questions will be of the "instruction" type, meaning that they will start with the phrase "How to". You will be presented with a question and asked to classify the category it belongs to. Then, you will be asked to rate your familiarity with the ques-
tion topic. Finally, you will be shown an answer to the question and asked to rate its usefulness. Please take your time to carefully read each question and answer, and consider your responses before submitting them. Your feedback is important and will help us to improve the quality of questionanswering systems. Thank you for participating in the study. Please, check the participant information form:(App. G)”
Figures 9 and 10 (top and bottom parts) shows an interface of the second trial run of the study with a random annotation example. The exact instructions are presented in the Figure, while the overall task description was the same as in the first trial.
Figure 7 illustrates the percentage of recognized deterioration within various question categories, and compares the results of the two trials. Most categories demonstrate an increase in detection in the second trial, with the exception of Education and Communication, Family Life, Finance and Business, Philosophy and Religion, Relationships and Youth. It is interesting to note that most of these exception categories can be grouped as relating to social life and human experience, topics for which answers are vague and often open-ended. The other two exception categories, Education and Communication, and, Finance and Business, also happened to be the two hardest categories to evaluate. It can be concluded that for the exceptions, deterioration of correctness is especially difficult to identify, or correctness of an answer is vague.
We also calculated the amount of False Positive IDRs. False positives were defined as answers that were marked as being misleading by an assessor, even though they were not. The first trial included four false positives, and the second trial two.
E.3 A/B Human Evaluation
Figures 11 and 12 demonstrate the interface used for A/B evaluation of our lower and upper-bound models, using a random annotation example for DPR + text-davinci-003 model evaluation. Figure 11 shows Answer A, which is the answer generated by the model, and Figure 12 contains Answer B, which is an answer from the dataset. The specific instructions are provided in the figures, while the overall task description is as follows: "In this study, you will be evaluating answers to "how-to" questions. You will be presented with a question and asked to classify the category it belongs to. Then, you will be given two answers, labeled A
and B, and asked to rate their usefulness and select which answer is more helpful/useful to the person asking the question. Please take your time to carefully read each question and answer and consider your responses before submitting them. Your feedback is important and will help improve the quality of question-answering systems. Thank you for participating in the study. Please, check the participant information form: (App. G).

heading: F Benchmarking Baselines via Human
text: Evaluations

heading: G Participant Information Form
text: INVITATION TO PARTICIPATE IN A RESEARCH PROJECT PARTICIPANT INFORMATION Thank you for taking part in this credibility assessment task. Before starting, please read this document carefully and be confident that you understand its contents before deciding whether to continue with your participation. The full participant information form can be accessed here.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Left blank.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 3, introduction
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: introduction
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. section 3
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section3, 5

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? appendix

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? section 3

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 3

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Section 3

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section 3, 5

heading: Future Work
text: Limitations
In this section we discuss possible limitations of our work, and present interesting avenues for future investigation. First, not all documents in WIKIHOWQA are equally useful; some documents are overall less relevant, some contain very relevant bits alongside generally irrelevant information. More accurate passage or span selection may be required for models to generate better answers. Another limitation is the focus on highlighted relevant passages as the method of aiding human evaluators in assessing factual correctness. While this approach helped to identify some factual inconsistencies and was proven to also make assessments faster in previous research (Bolotova et al. , 2020), additional techniques should be considered for their potential to deliver further improvements. Unfortunately, due to limitations of the evaluation interface used, we were not able capture how frequently annotators clicked on sentences to see aiding passages. Click ratio data could have yielded valuable insights into when and how often assessors referred to the provided information. While we considered a range of answer deterioration types, it does not fully align with the kinds of hallucinations that neural NFQA models may produce in the wild. Further research is needed to analyse the robustness of these models in real-world scenarios. One direction is to adopt the comprehensive evaluation framework of abstractive neural models proposed by (Dou et al. , 2022) for NFQA. In our baselines, we experimented with text parsed directly from the HTML source code. However, it is important to note that we also provide the option of using the Markdown files, which preserve formatting information such as tables and lists. This could be useful for certain cases where formatting is important. Furthermore, some reference HTML pages contain pagination and long comment sections that we did not consider when scraping the data. Finally, the evaluation of passage-level QA remains a challenging task, both for human and automatic evaluation frameworks. Difficulties include accurate assessment of factual correctness and overall consistency, especially when answers require complex reasoning based on multiple sources. Advancements in automatic evaluation of similar tasks, such as summarisation, could be adopted for NFQA