SlowGuess/ACL_2020_OCR huggingface dataset:
Jon Ander Campos<sup>1</sup>, Arantxa Otegi<sup>1</sup>, Aitor Soroa<sup>1</sup>, Jan Deriu<sup>2</sup>, Mark Cieliebak<sup>2</sup>, Eneko Agirre<sup>1</sup>
<sup>1</sup>University of the Basque Country (UPV/EHU)
$^{2}$ Zurich University of Applied Sciences (ZHAW)
1{jonander.campos, arantza.otegi, e.agirre, a.soroa}@ehu.eus
2{jan.deriu, mark.cieliebak}@zhaw.ch
The goal of this work is to build conversational Question Answering (QA) interfaces for the large body of domain-specific information available in FAQ sites. We present DoQA, a dataset with 2,437 dialogues and 10,917 QA pairs. The dialogues are collected from three Stack Exchange sites using the Wizard of Oz method with crowdsourcing. Compared to previous work, DoQA comprises well-defined information needs, leading to more coherent and natural conversations with less factoid questions and is multi-domain. In addition, we introduce a more realistic information retrieval (IR) scenario where the system needs to find the answer in any of the FAQ documents. The results of an existing, strong, system show that, thanks to transfer learning from a Wikipedia QA dataset and fine tuning on a single FAQ domain, it is possible to build high quality conversational QA systems for FAQs without indomain training data. The good results carry over into the more challenging IR scenario. In both cases, there is still ample room for improvement, as indicated by the higher human upperbound.
The overarching objective of our work is to access the large body of domain-specific information available in Frequently Asked Question sites (FAQ for short) via conversational Question Answering (QA) systems. In particular, we want to know whether current techniques are able to work with limited training data, and without needing to gather data for each target FAQ domain. In this paper we present DoQA, a task and associated dataset for accessing domain-specific FAQs via conversational QA<sup>1</sup>. The dataset contains 2,437 information-seeking question/answer dialogues on three different domains
How can I store chopped onions in the fridge without the smell?
I regularly store chopped onion in my refrigerator (or at least halves & quarters).
17 I either use tight-sealing plastic containers or zip-top bags.  
You may want to double-bag in zip-tops to be sure to avoid a smell.
One problem you may be having is onion-ness getting on the outside of the container. Be sure the outside is all clean and dry - no point in having a nicely sealed packet of onion when the outside can get all stinky anyway.
USER: How can I store chopped onions in the fridge without the smell?
EXPERT: You may want to double-bag in zip-tops to be sure to avoid a smell. (Follow up).
USER: I used a plastic container the last time and the whole fridge smelled of onion, why is that?
EXPERT: One problem you may be having is onion-ness getting on the outside of the container. (Follow up).
USER: Have you had good experience with using a double bag like you suggested?
EXPERT: Yes, I regularly store chopped onion in my refrigerator (or at least halves & quarters). (Don't follow up).
USER: I will be chopping 4-6 onions because I'm serving a large crowd, do you still think that will be okay?
EXPERT: I don't know sorry. (Don't follow up).
Figure 1: A dialogue about cooking. On top, the original post, comprising a topic and an excerpt of the answer passage. In italics, dialogue acts (cf. Section 3).
(10,917 questions in total). These dialogues are created using the Wizard of Oz technique by crowdworkers that play the following two roles: the user asks questions about a given topic posted in Stack Exchange $^2$ , and the domain expert replies to the questions by selecting a short span of text from the long textual reply in the original post. The first question is prompted by the real FAQ question, which sets the topic of interest driving the user questions. In addition to the extractive span, we also allow experts to rephrase it, in order to provide an abstractive, more natural, answer. The dataset covers unanswerable questions and some relevant dialogue acts. We focused on three different domains: Cooking, Travel and Movies. These forums
<sup>1</sup>The DoQA dataset is available here: http://ixa.eus/node/12931
<sup>2</sup>https://stackoverflow.com/
7302
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7302-7314 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
are some of the most active ones and contain knowledge of general interest, making it easily accessible for crowdworkers. DoQA contains two scenarios: in the standard scenario the test data comprises the questions and the target document from which the answers need to be extracted; in the information retrieval (IR) scenario the test data contains the questions, but the target document is unknown, and the system needs to select the documents which contain the answers among all documents in the collection.
Previous work on conversational QA datasets include CoQA (Reddy et al., 2018) and QuAC (Choi et al., 2018). The main focus of CoQA are reading comprehension questions, which are produced with access to the target paragraph. The topic of the questions are delimited by the paragraph, which leads to specific questions about details in the paragraph. Choi et al. (2018) observed that a large percentage of CoQA answers are named entities or short noun phrases. In QuAC, the topic of the conversation is set by a title and first paragraph of a Wikipedia article about people. The user makes up questions about the person of interest. Note that, contrary to our setting, there is no real information need in any of those datasets, which can lead to less coherent conversations: any question about the paragraph or person of interest is valid, respectively.
DoQA makes the following contributions. Firstly, contrary to made-up reading comprehension tasks, DoQA reflects real user needs, as defined by a topic in an existing FAQ. Good results on DoQA are of practical interest, as they would show that effective conversational QA interfaces to FAQs can be built. Secondly, for the same reason, the conversations in DoQA are more coherent, natural and contain less factoids than other datasets, as shown by our analysis. Thirdly, the IR scenario and the multiple domains make DoQA more challenging and realistic. Table 1 summarizes the characteristics of DoQA.
Although one could question the small size of our dataset, our goal is to test whether current techniques are able to work with limited training data, and without needing to gather data for each target FAQ domain. We thus present results of an existing strong conversational QA model with limited and out-of-domain data. The system trained on Wikipedia data (QuAC) provides some weak results which are improved when fine-tuning on
DoQA	QuAC	CoQA
Real information need	▲
Naturalness	▲
Dialogue coherence	▲
Non-factoid questions	▲
Unanswerable questions	▲	▲
Dialogue acts	▲	▲
Multi-domain	▲	▲
IR scenario	▲
Table 1: Summary of the characteristics of DoQA compared to QuAC and CoQA. ▲ for positive.
the FAQ dataset. Our empirical contribution is to show that a relatively low amount of training in one FAQ dataset (1000 dialogues on Cooking) is sufficient for strong results on Cooking (comparable to those obtained in the QuAC dataset with larger amounts of training data), but also on two other totally different domains with no in-domain training data (Movies and Travel). In all cases scores over 50 F1 are reported. Regarding the IR scenario, an IR module complements the conversational system, with a relatively modest drop in performance. The gap with respect to human performance is over 30 points, showing that there is still ample room for system improvement.
Conversational QA systems stem from the body of work on Reading Comprehension, whose goal is to test the capacity of a system to understand a document by answering any question posed over its content. Recent work on the field has resulted in the creation of multiple datasets (Rajpurkar et al., 2016; Trischler et al., 2017; Nguyen et al., 2016; Kočisky et al., 2018; Dunn et al., 2017). These datasets are typically composed of multiple question/answer pairs, often along with a reference passage from which the answer is curated. Whereas the questions are always in free text form, some datasets represent the answers as a contiguous span in the reference passage, while others contain free form answers. The former are usually referred as extractive, whereas the latter are called abstractive. All in all, in these QA datasets the queries are unrelated to each other, and thus there is no dialogue structure involved.
Iyyer et al. (2017) propose to answer complex queries by decomposing them into sequences of single, co-referent queries. The question sequence can be seen as different turns in a dialogue, and each question refers and refines previous ones. The
7303
authors present the SequentialQA dataset, which comprises 6K question sequences posed over the content of Wikipedia tables. In the case of our task, it is the user who makes several questions in sequence.
More similar to our work, CoQA (Reddy et al., 2018) and QuAC (Choi et al., 2018) are two conversational QA datasets comprising QA dialogues that fulfill the information need of a user by answering questions about different topics. Similarly to our, both datasets are built by crowdsourcing, where one person (the questioner) is presented with a topic and has to pose free-form questions about it. Another person (the answerer) has to select an answer to the question by choosing an excerpt from the relevant passage describing the topic. Some of the questions in both datasets are unanswerable, and access to previous questions and answers are needed in order to answer some of the questions.
CoQA contains 127k questions with answers, obtained from 8k conversations about passages from broad domains, ranging from children stories to science. The answers are also excerpts from the relevant passage, but answerers have the choice of reformulating them. The authors report that  $78\%$  of the answers had at least one edit. Although reformulating answers can yield to more natural dialogues, Yatskar (2018) showed that span based systems can in principle obtain a performance up to 97.8 points F1, showing that editing the answers does not yield to systems with better quality. In CoQA, both questioner and answerer have access to the full passage, which guides the conversation towards the specific information conveyed in it.
QuAC is a dataset that contains 14k information-seeking question answering dialogues. The dialogues in QuAC are about a specific section in Wikipedia articles about people. The answerer has access to the full section text, whereas the questioner only sees the section's title and the first paragraph of the main article, which serves as inspiration when formulating the queries. QuAC also contains dialogue acts in each turn, which are useful when collecting the dialogues, as they can be used by the answerer to indicate to questioner whether to continue making questions about the last answer or drift to other aspects of the topic. We will compare CoQA and QuAC in more detail in Section 4.
Previous conversational QA datasets provide the relevant document or passage that contain the answer of a query. However, in many real world
scenarios such as FAQs, the answers need to be searched over the whole document collection. In related question answering research, Chen et al. (2017) and Watanabe et al. (2017) combine retrieval and answer extraction on a large set of documents. In (Talmor and Berant, 2018) the authors propose decomposing complex questions into a sequence of simple questions, and using search engines to answer those single questions, from which the final answer is computed. We find that requiring the system to search for relevant documents and passages is more realistic, and DoQA is the first conversational QA task incorporating this scenario.
In contemporary work, Castelli et al. (2019) present a question answering dataset for the technical support domain which focuses on actual questions posed by users and has a real-world size with only 600 training instances. It also requires systems to examine 50 documents per query. Our work has similar motivations for setting up more realistic tasks, and is complementary in the sense that we cover non-technical domains and conversational QA.
Community Question Answering has been also the focus of two related tasks (Nakov et al., 2016, 2017), where, given a new question and a collection of pre-existing questions and answers, the systems need to rank the answers that are most useful for answering the new question.
This section describes our conversational QA dataset collection process which consists of an interactive task designed for two crowdworkers in Amazon Mechanical Turk (AMT).
3.1 FAQ Post Selection
We collected topic-answer pairs for the three different domains from the Stack Exchange data dumps. We focused on the Cooking<sup>3</sup>, Travel<sup>4</sup> and Movies<sup>5</sup> domains, as they are active forums and contain knowledge of general interest, making it easily accessible and attractive for crowdworkers. Note that the posts in Stack Exchange (as in most FAQ sites) comprise broad questions which often require lengthy answers. We refer to the question in the post as topic and to the long answer in the post as passage (not to be confused with the actual ques
<sup>3</sup>https://cooking.stackexchange.com/
4https://travel.stackexchange.com/
5https://movies.stackexchange.com/
tions/answers in the collected dialogues). Figure 1 shows an example of a topic and its corresponding passage for the Cooking domain. More details on post filtering and selection can be found in Appendix A.
3.2 Crowdsourcing Task
For the annotation process, we defined a HIT in AMT as the task of generating a dialogue about a specific topic between two workers (the specifications of the defined HIT can be found in Appendix B). One of the workers (the user) asks questions to the second one (the domain expert) about a certain topic from a Stack Exchange Cooking, Travel or Movies thread. The worker who adopts the user role has access to a small paragraph that introduces the topic. Having this information, he must ask free text questions. The first question of every dialogue must be the title of the topic that appears in the title of the Stack Exchange thread. The domain expert has access to the whole answer passage and he/she answers the query by selecting a span of text from it. In order to make the dialogue look more natural, the domain expert has the opportunity to edit the answer, but note that if he does so the answer will not match the content of the text span anymore. Therefore, and following Yatskar (2018), we motivate minimal modifications by copying the selected text span directly into the answer field in the web application. In addition to the span of text, the expert has to give feedback to the user with one of the following dialogue acts: an affirmation act, which is required when the question is a Yes/No question (yes, no or neither); an answerability act, which defines if the question has an answer or not (answerable or no answer). When no answer is selected, the returned string is "I don't know"; and a continuation dialogue act, which is used for leading the user to the most interesting topics (follow up or don't follow up). The last dialogue act is used to minimally guide the user in his/her questions, where the expert can encourage (or discourage) the user to continue with questions related to his last questions using follow up (or alternatively don't follow up). These dialogue acts are the same as in QuAC, but we discarded the maybe follow up act from the continuation act because we felt it was not intuitive enough.
Dialogues are ended when a maximum of 8 question and answer pairs is reached, when 3 unanswerable questions have been asked, or when 10 min
Cooking	Travel	Movies
Train	Dev.	Test	Test	Test
Questions	4,612	911	1,797	1,713	1,884
Dialogues	1,037	200	400	400	400
Unique sections	546	162	400	400	400
Tokens / question	10.79	10.14	10.66	10.45	9.45
Tokens / answer	13.19	13.10	12.58	13.47	12.40
Dialogue turns	4.47	4.55	4.49	4.28	4.71
Extractive %	69.68	67.18	66.95	65.44	74.15
Abstractive %	30.32	32.82	33.05	34.56	25.85
Yes/No %	20.22	21.07	22.20	25.10	18.05
I don’t know %	27.55	27.33	29.71	22.83	29.41
Table 2: Statistics of the different domains of DoQA.
utes time limit is reached. The purpose of these limits is to avoid long and repetitive dialogues, because real threads of the selected domains are very focused on a certain topic. Dialogues are only accepted if they have a minimum length of 2 question and answer pairs and if they have at least one answer that is not "I don't know sorry".
The data collection interface is based on Co-CoA $^6$ , which we modified. The interfaces for the user and expert are shown in Appendix C.
3.3 Dataset Details
Following usual practice, we divided the main Cooking dataset into a train, development and test splits. For the other two domains, Travel and Movies, we only have the test split. Statistics for all the domains and splits are shown in Table 2.
The splits of the Cooking dataset have very similar characteristics, so we can expect them to be valid representatives of the whole Cooking dataset. In the test splits we do not allow more than one dialogue about the same section, as it can end up producing inaccurate evaluation of the models.
3.4 Collecting Multiple Answers
In order to estimate the performance of a human in the task, we collected additional answers for the test splits for the three domains in a second round, after having completed the dialogues. For each question in the dialogues collected in the first round, we show to the worker the previous questions and answers in the dialogue (if available), and he has to provide an answer span. The interface for the collection of multiple answers can be seen in Appendix D.
$^{6}$ https://github.com/stanfordnlp/cocoa (He et al., 2017)
7305
3.5 Information Retrieval Scenario
In the usual setting for this kind of tasks, the system is given the question and the passage where the answer is to be extracted from. In a realistic scenario, however, relevant answer passages that may contain the answer will need to be retrieved first. More specifically, if a user has an information need and asks a question to a conversational QA system on a FAQ, the system can search for similar questions which have already been answered, or the system can directly search in existing answer passages. In other words, there are two ways to check automatically if the forum contains a relevant answer passage to a new question: (1) question retrieval, where relevant or similar questions are searched (and thus, the answer for this relevant question is taken as a relevant answer), and (2) answer retrieval, where relevant answers are searched directly among existing answers.
We added information about both relevant cases to the main Cooking dataset, in the form of the 20 most relevant answer passages for each dialogue in the dataset. We followed a basic approach to get these relevant answer passages. We created two separate indexes using an IR system<sup>7</sup> for the two mentioned approaches, question and answer retrieval. For the former, we indexed the original topics posted in the forum; and for the latter, we indexed the answer passages for each post in the forum. Then, for each dialogue in the development and test splits, the top 20 documents were retrieved using the first question of the dialogue. Given that the dialogues are about a single topic, we only use the first question in the dialogue, and then use the retrieved passages for the rest of questions in the dialogue as well.
The question retrieval approach yields very good results (0.94 precision at one), as expected, as the crowdworker doing the questions has access to the topic when asking the first question and usually did minor edits. The results for answer retrieval are more modest, 0.54 precision at one. The results section shows the results of the conversational QA system when relying on the passages returned by the IR module.
Overall statistics In this section we present an quantitative and qualitative analysis of DoQA and we compare them to similar conversational datasets
DoQA	QuAC	CoQA
Questions	10,917	98,407	127,000
Dialogues	2,437	13,594	8,399
Tokens / question	10.43	6.5	5.5
Tokens / answer	12.99	14.6	2.7
Dialogue turns	4.48	7.2	15.2
Extractive %	69.13	100	66.8
Abstractive %	30.87	-	33.2
Yes/No %	21.01	25.8	-
I don’t know %	27.47	20.3	1.3
Table 3: Statistics of DoQA compared to QuAC and CoQA.
like QuAC and CoQA, stressing its similarities and differences.
Table 3 shows the overall statistics of DoQA, together with the statistics of QuAC and CoQA. As can be seen, DoQA has the smallest amount of questions and dialogues. However, other features makes it very interesting for the research of conversational QA. For instance, the average tokens per questions and answers (10.43 and 12.99, respectively) are closer to real dialogues if we compare to the other datasets. Specially CoQA has very short questions and answers on average, suggesting that CoQA is closer to factoid QA than dialogue, as human dialogues tend to be longer and convoluted, not just short answers. DoQA has the lower ratio of questions per dialogue, which is expected, as most of the dialogues are about a very specific topic and the user is satisfied and gets the answer without the need of long dialogues. CoQA ends up on having almost all of its questions answerable, facing the same issues as SQuAD 1.0 (Rajpurkar et al., 2016) that motivated the addition of unanswerable questions in SQuAD 2.0 (Rajpurkar et al., 2018).
We also have the results of a short survey that workers had to respond to at the end of each HIT. On the one hand, the user had to give feedback on how satisfied was with the answers of the expert in a scale of 1-5. The average satisfaction was 3.9. On the other hand, the expert had to give feedback on how sensible were the questions and the helpfulness of the answers. The average scores obtained were 4.27 and 4.10, respectively, which makes the AMT task satisfactory.
Naturalness One of the main positive aspects of our dataset is the naturalness of the dialogues that other similar datasets like QuAC do not have. The answers of DoQA come from a forum where the answer text is directed to a person who posted the
7Solr https://lucene.apache.org/solr/
7306
question, and does not come from a much formal text like Wikipedia, as it is the case of QuAC. The naturalness and casual register of the former it is more adequate than the formal register of the latter for a conversational QA system. The dialogue in Figure 1 is a clear example of such naturalness, where the expert answers to the user with casual and directed expressions like "You may want" and "you may be having". To verify whether dialogues in DoQA are more natural than the ones in QuAC, we sample randomly 50 dialogues in DoQA Cooking domain and QuAC and performed A/B testing to determine which of the two dialogues is more natural. This test showed that  $84\%$  of the times DoQA dialogues are more natural.
This naturalness is probably caused because a dialogue in DoQA is started by a user with a very specific aim or topic to solve in mind, and thus, follow-up questions are very related to previous answers, and all the questions are set within a context. In contrast, dialogues in QuAC do not show so clear objective and questions seem to be asked randomly. Dialogues in DoQA are ended when the initial information need of the user is satisfied and this adds naturalness to dialogues.