SlowGuess/ACL_2020_OCR huggingface dataset:
Further analysis of the samples showed that answers in DoQA seem to be more spontaneous because they have more orality aspects, such as higher level of expressivity ("Normally when I try they end up burned not crispy!", "My biggest worry here would be...", "hey let's not be hasty"), opinions ("I came across a suggestion to cover the lid...", "I'd recommend simply adding...", "It sounds like fermentation to me") and humor ("well yeah but booze is booze"). Contrarily, answers in QuAC are more hermetic and do not show any features of orality or spontaneity that a dialogue should have. All these features make DoQA dialogues look more natural.
We also analyzed the remaining  $16\%$  cases where DoQA dialogues appear less natural. In most of these dialogues there were responses that did not really answer the question. The following question (Q) and answer (A) pairs are good examples of it: (Q) "Is the taste going to be significantly different?" (A) "there is cornstarch in confectioner's sugar"; (Q) "how about reheating?" (A) "When you defrost it, do so in your fridge leaving it overnight so that it defrosts gradually"; (Q) "Can I use my potatoes or carrots if they already have some roots?" (A) "The green portions of a potato are toxic". In some of these cases the correct answer for the respective
question is not in the answer text provided to the expert. If this was the case, the expert should answer "I don't know", instead of giving a nonsense answer.
Question types Table 4 includes the most frequent two initial words of the questions in the Cooking dataset along with their percentages of occurrences and some examples. Most of the questions start with what and how (16.6% and 15.1% of the questions, respectively), which are also the most frequent in QuAC and CoQA. Contrary to them, the questions in the Cooking dataset do not refer to factoids, with the exception of "How long" questions. The questions in DoQA require long and complex answers. In contrast to this, in CoQA and QuAC many of the most frequent initial words such as who, where, and when indicate factoid questions. In order to confirm this fact, we manually inspected 50 random questions from the Cooking domain and QuAC datasets. This analysis revealed that  $66\%$  of the questions are non-factoid in the DoQA Cooking domain, showing that most of the questions are open-ended. These amount is larger than in QuAC, as in our analysis for QuAC we found that only  $36\%$  of the questions are non-factoid. These values differ slightly from those reported by Choi et al. (2018), as they say that about half of questions are non-factoid.
Context or history dependence The manual analysis also shows that  $61\%$  of the questions are dependent on the conversation history, as many questions have coreferences to previous questions or answers in the dialogue. For example, "What are other methods to sharpen a knife?", "How long should I cook it in the microwave?", "Can you explain the science behind this cooking procedure?". Moreover, we could note that less than  $1\%$  ask further advice or tips about the current topic, confirming that these conversations are about specific topics where the user is satisfied with the expert answers after a few questions.
Dialogue coherence Related to the just mentioned fact that the user does not usually ask any other tips, users in DoQA do not tend to switch topics in a dialogue. In order to confirm it, we performed another A/B testing to the same 50 dialogues samples of the DoQA Cooking domain and QuAC to determine which of the two dialogues is more coherent, that is, which dialogue has a smoother flow. This test revealed that in  $64\%$  of
7307
Bigram prefix	%	Example
What
(16.6%)	is	30.8	What is the purpose of adding water to an egg wash?
are	8.0	What are other methods to sharpen a knife?
How
(15.1%)	do	24.0	How do you properly defrost frozen fish?
long	21.9	How long should I cook it in the microwave?
Is
(10.5%)	there	52.8	Is there a special tool available for cracking open a pistachio?
it	19.8	Is it safe to cook with rainwater?
Do
(7.6%)	you	70.7	Do you have any advice for storing green onions?
I	16.1	Do I have to peel the apples?
Can
(5.5%)	I	52.8	Can I put them back in the oven to reheat?
you	25.3	Can you explain the science behind this cooking procedure?
I
(5.0%)	have	19.6	I have been told that frying it would make it tastier, but is it healthier to grill or fry?
am	15.3	I am cooking for somebody who doesn’t eat shellfish, so is the fish sauce safe?
Why
(3.5%)	is	22.1	Why is it important to increase the fermentation time?
does	21.7	Why does my custard pudding taste like raw eggs?
Table 4: Most frequent initial words and bigrams in questions (Cooking domain).
the cases dialogues of DoQA are more coherent than QuAC. Only in  $10\%$  of the cases dialogues of DoQA are less coherent, with the remaining  $26\%$  equally coherent. We analyzed the  $10\%$  and saw that they contain similar questions one after the other, or repeated answers in the same dialogue.
Summary Table 1 summarizes the positive characteristics of DoQA compared to the similar datasets like QuAC and CoQA.
5 Task Definition
Given a textual passage and a question, traditional QA systems find an answer to the question within the passage. Conversational QA systems are more complex, as they need to deal with a sequence of possibly inter-dependent questions. That is, the meaning of the current question may depend on the dialogue history. For this reason, a dialogue history comprised by previous question/answer pairs is also provided to the system. In addition, some dialogue acts have to be predicted as an output: yes/no answers, which are required for affirmation questions, and continuation feedback, which might be useful for information-seeking dialogues.
We denote the answer passage as  $p$ , the dialogue history of questions and respective ground truth answers as  $\{q_1, a_1, \ldots, q_{k-1}, a_{k-1}\}$ , current question as  $q_k$ , the answer span  $a_k$  which is delimited by its starting index  $i$  and ending index  $j$  in the passage  $p$ , and dialogue act list  $v$ . The dialogue act list contains  $\{yes, no, -\}$  values for predicting affirmation and  $\{follow-up, don't follow-up\}$  for continuation feedback.
6 Baseline Models
We present two strong baseline models to address our task. Although the state-of-the-art evolves quickly, our choice has the benefit of simplicity and strong performance.
BERT We took the fine-tuning approach for QA of BERT, which predicts the indexes  $i$  and  $j$  of the  $a_{k}$  answer span given  $p$  and  $q_{k}$  as input. This baseline has shown strong performance on QA datasets such as SQuAD (Devlin et al., 2018).
BERT+HAE The previous baseline does not model dialogue history. We used BERT with History Answer Embedding (HAE) as proposed by Qu et al. (2019) as a baseline that deals with the multi-turn problem, as this is the publicly available system that performs best in the QuAC leaderboard<sup>8</sup>. The system introduces dialogue history  $\{q_{1},a_{1},\dots q_{k - 1},a_{k - 1}\}$  to BERT by adding a history answer embedding layer, which learns whether a token is part of history or not.
Evaluation metrics Given the similarity between QuAC and DoQA, we use the same evaluation metrics and criteria used in QuAC. F1 is the main evaluation metric and is computed by the overlap at word level of the prediction and reference answers. As the test set contains multiple answers for each question we take the maximum F1 among them. Note that when computing F1 QuAC filters
accessed on August 20, 2019
7308
Cooking	Travel	Movies
Setting	Model	F1	HEQ-Q	F1all	F1	HEQ-Q	F1all	F1	HEQ-Q	F1all
Native	BERT	40.1	35.1	38.3	36.2	34.8	34.8	36.1	33.5	35.0
BERT+HAE	47.8	43.0	45.9	44.0	37.4	42.9	42.8	37.1	41.9
Zero-shot	BERT	40.2	34.7	38.9	34.0	30.1	33.1	38.2	33.2	37.4
BERT+HAE	46.2	42.0	44.5	42.7	37.1	42.3	45.4	41.4	44.8
Transfer	BERT	43.3	37.8	42.4	40.6	33.6	40.1	41.8	36.3	41.3
BERT+HAE	53.2	48.3	51.4	50.8	42.1	50.6	51.6	44.3	51.5
Transfer all	BERT	43.1	37.0	42.0	40.6	33.4	40.5	42.0	34.5	41.6
BERT+HAE	53.4	46.9	52.7	51.6	43.3	50.9	52.1	45.2	51.7
Human	-	100.0	86.6	-	100.0	87.4	-	100.0	88.8
Table 5: Results of the baseline systems in the three DoQA domains (columns) in all four settings (rows). See text for explanation of each row. Note that Travel and Movies results are obtained without any Travel or Movies training data.
out answers with low agreement among human annotators. An additional F1-all is provided for the whole set. We also report HEQ-Q (human equivalence score on a question level) which measures the percentage of questions for which system F1 exceeds or matches human F1.
Experimental Setup We carried out experiments using the extractive information in DoQA, leaving the abstractive information for the future. The parameters we used to train the baseline models are the ones proposed in the original papers. We tested the models in four settings. In the native setting the Cooking DoQA train and dev data are used, the first for training and the second for early stopping. In the zero-shot setting we use QuAC training data for training and early stopping. In the transfer setting we use QuAC and Cooking DoQA for training. Finally, in the transfer all setting we additionally use the test data from the other two domains for training.
We also experimented on the IR scenario, using the provided IR rankings (see Section 3.5), which contain the top 20 passages for each dialogue. In the first experiment, Top-1, we just use the top 1 passage and apply the baseline BERT model. In a second experiment, Top-20:BERT, the passages are fed to the BERT model and the passage that contains the answer with highest confidence score is selected. Note that we discard passages that produced "I don't know" type of answers. In a third experiment, Top-20:BERT*IR, we select the passage with highest combined score according to BERT and the search engine.
All the reported results have been achieved using the BERT Base Uncased model.
Results Table 5 summarizes our results. In the bottom row we give the human upperbound. The three metrics used for evaluation behave similarly, so we focus on one (e.g. F1) for easier discussion. We report all three for completion and easier comparison with related datasets. In all settings and domains the BERT+HAE model yields better results than BERT, showing that DoQA is indeed a conversational dataset, where question and answer history needs to be modelled.
Regarding the different settings, we first focus on the Cooking dataset. The native scenario and the zero-shot settings yield similar results, showing that the 1000 dialogues on Cooking provide the same performance as 13000 dialogues on Wikipedia from  $\mathrm{QuAC}^9$ . The combination of both improves performance by 7 points ("Transfer" row), with small additional gains when adding Movies and Travel dialogues for further fine-tuning ("Transfer all" row). Note that the performance obtained for Cooking in the "Transfer" or "Transfer all" setting is comparable to the one reported for QuAC, where the training and test are from the same domain<sup>10</sup>.
Yet, the most interesting results are those for the Travel and Movies domains, which do not have access to in-domain training data on Travel or Movies. In this case, the native and transfer results with
9When randomly subsampling QuAC to the same size as DoQA the results on the cooking domain fall down to 36.5.
10 BERT+HAE obtains 62.4 in QuAC (Qu et al., 2019), 9 points higher than in DoQA Cooking, but note that QuAC contains more reference answers per question than DoQA, and thus the resulting F1 scores are higher. When evaluating BERT+HAE using a single reference answer in both datasets, the score is 45.9 on QuAC and 47.8 on the Cooking dataset of DoQA.
7309
Model	F1	HEQ-Q	F1-all
Answer retrieval
Top-1	37.2	33.3	35.8
Top-20:BERT	32.7	29.6	31.0
Top-20:BERT*IR	36.1	32.9	34.4
Question retrieval
Top-1	42.2	36.76	41.1
Top-20:BERT	35.8	31.2	34.3
Top-20:BERT*IR	41.6	36.4	40.5
Table 6: Results on the IR scenario (Cooking domain). See text for explanation
no in-domain training are as high as those for Cooking. These results show that it is not necessary to train for each domain in a FAQ, and that training data from other FAQ domains is highly reusable.
The results obtained on out-of-domain test conversations (Movie and Travel) when trained on Wikipedia and Cooking are striking, as they are comparable to the in-domain results obtained for the Cooking test conversations. We hypothesize that when people write the answer documents in FAQ websites such as Stackexchange, they tend to use linguistic patterns that are common across domains such as Travel, Cooking or Movies. This is in contrast to Wikipedia text, which is produced with a different purpose, and might contain different linguistic patterns. As an example, in contrast to FAQ text, Wikipedia text does not contain first-person and second-person pronouns. We leave an analysis of this hypothesis for the future.
Table 6 presents the results of the experiments on the IR scenario. The simplest Top-1 approach is the best performing for both question and answer retrieval strategies. We leave the exploration of more sophisticated techniques for future work. The results using question retrieval are very close to those in Table 5. Given the large gap in the IR results in Section 3.5 for answer retrieval, it is a surprise to see a small 5 point decrease with respect to question retrieval. We found that there is a high correlation between the errors of the dialogue system and the answer retrieval system, which explains the smaller difference. In both retrieval strategies the results are close to the performance obtaining when having access to the reference target passage.
The goal of this work is to access the large body of domain-specific information in the form of Frequently Asked Question sites via conversational QA systems. We have presented DoQA, a dataset for accessing Domain specific FAQs via conversational QA that contains 2,437 information-seeking dialogues on the Cooking, Travel and Movies domain (10,917 questions in total). These dialogues are created by crowdworkers that play the following two roles: the user asks questions about a certain topic posted in Stack Exchange, and the domain expert who replies to the questions by selecting a short span of text from the long textual reply in the original post. The expert can rephrase the selected span, in order to make it look more natural. In contrast to previous conversational QA datasets, our dataset responds to a real information need, is multi-domain, more natural and coherent. DoQA introduces a more realistic scenario where the passage with the answer needs to be retrieved.
Together with the dataset, we presented results of a strong conversational model, including transfer learning from Wikipedia QA datasets to our FAQ dataset. Our dataset and experiments show that it is possible to access domain-specific FAQs using conversational QA systems with little or no in-domain training data, yielding quality which is comparable to those reported in QuAC.
For the future, we would like to exploit the abstractive answers in our dataset, explore more sophisticated systems in both scenarios and perform user studies to study how real users interact with a conversational QA system when accessing FAQs.
This research was partially supported by a Google Faculty Award, EU ERA-Net CHIST-ERA LIH-LITH funded by the Agencia Estatal de Investigación (AEI, Spain) project PCIN-2017-118 and the Swiss National Science Foundation (SNF, Switzerland) project 20CH21 174237, project DeepReading (RTI2018-096846-BC21) supported by the Ministry of Science, Innovation and Universities of the Spanish Government, the Basque Government (DL4NLP KK-2019/00045 and excellence research group), project BigKnowledge (Ayudas Fundación BBVA a Equipos de Investigación Científica 2018) and the NVIDIA GPU grant program. Jon Ander Campos enjoys a doctoral grant from the Spanish MECD.
Vittorio Castelli, Rishav Chakravarti, Saswati Dana, Anthony Ferritto, Radu Florian, Martin Franz, Dinesh Garg, Dinesh Khandelwal, Scott McCarley, Mike McCawley, Mohamed Nasr, Lin Pan, Cezar Pendus, John Pitrelli, Saurabh Pajar, Salim Roukos, Andrzej Sakrajda, Avirup Sil, Rosario Uceda-Sosa, Todd Ward, and Rong Zhang. 2019. The TechQA Dataset.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettle-moyer. 2018. QuAC: Question Answering in Context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174-2184.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805.
Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. 2017. SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine. CoRR, abs/1704.05179.
He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1766-1776.
Mohit Iyyer, Wen tau Yih, and Ming-Wei Chang. 2017. Search-based Neural Structured Learning for Sequential Question Answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1821–1831, Vancouver, Canada. Association for Computational Linguistics.
Tomáš Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328.
Preslav Nakov, Doris Hoogeveen, Lluis Márquez, Alessandro Moschitti, Hamdy Mubarak, Timothy Baldwin, and Karin Verspoor. 2017. Semeval-2017 task 3: Community question answering. arXiv preprint arXiv:1912.00730.
Preslav Nakov, Lluis Marquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alhakim Frei hat, Jim Glass, and Bilal Randeree. 2016. SemEval-2016 task 3: Community question answering. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 525-545, San Diego, California. Association for Computational Linguistics.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated Machine Reading CComprehension Dataset. arXiv, abs/1611.09268.
Chen Qu, Liu Yang, Minghui Qiu, W. Bruce Croft, Yongfeng Zhang, and Mohit Iyyer. 2019. BERT with History Answer Embedding for Conversational Question Answering. CoRR, abs/1905.05412.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don't Know: Unanswerable Questions for SQuAD. arXiv preprint arXiv:1806.03822.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. CoQA: A conversational question answering challenge. arXiv preprint arXiv:1808.07042.
Alon Talmor and Jonathan Berant. 2018. The Web as a Knowledge-Base for Answering Complex Questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–651, New Orleans, Louisiana. Association for Computational Linguistics.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-200, Vancouver, Canada. Association for Computational Linguistics.
Yusuke Watanabe, Bhuwan Dhingra, and Ruslan Salakhutdinov. 2017. Question Answering from Unstructured Text by Retrieval and Comprehension. CoRR, abs/1703.08885.
Mark Yatskar. 2018. A qualitative comparison of CoQA, SQuAD 2.0 and QuAC. arXiv preprint arXiv:1809.10735.
7311
A FAQ Post Selection
First, we downloaded the data dumps from September 2018 for cooking forum and September 2019 for travel and movies forums. We then removed threads with unaccepted answers. At this point we did a preliminary analysis of the cooking topic scores and the lengths of the answer passages. Regarding the scores, we realized that all topic scores were in the range  $[-6, 240]$ . After manually analysing some random samples, we concluded that even low scoring topics had a good quality, except for the ones with negative scores. Regarding the length of the answer passages, some of them were too long for our task (up to 2,960 tokens), as very long passages make the task very tedious. Taking all this into account, we applied the following filters to the topic-passage pairs for the three domains:
- Topics with score  $<= 0$  are removed, as we are not interested in badly asked questions.
- Topic titles with more than one question mark are removed. The reason behind this filter is that we are interested in having the topic titles as the first question of our dialogues and we are not interested in having more than one question per dialogue turn.
- The length of the answer passage has to be greater than 50 and shorter than 250 tokens. This way, we try to ensure that the answer passage is long enough for collecting dialogue, but not too long for avoiding tedious answer spotting.
- Answers that contain HTML tags such as hyperlinks, images, code, etc. are removed.
B Amazon Mechanical Turk HIT Specifications
In order to select the workers in AMT, we defined the HIT with the following specifications:
- HIT approval rate  $\geq 98\%$ .
Approved HITs  $\geq 1000$
- Location of the workers: English speaking countries.
We paid the workers $0.10 for doing the HIT and a bonus of $0.33 for each question or answer given
during the task except for the “I don’t know sorry” case where $0.05 was paid. This difference in the payment motivates the workers to force themselves to find the actual answer in the passage, because answering “I don’t know” is less demanding than searching for the correct answer span. The average price for each dialogue is $3.2.
C Dialogue Collection Interfaces
For dialogue collection, the worker carrying out the user role used the interface shown in Figure 2 and one with the expert role used the interface displayed in Figure 3.
D Multiple Answers Collection Interface
The interface used for multiple answers collection can be seen in Figure 4.
7312
Cooking Chat
Click here to show/hide the instructions
[12/05/19 16:41:02] <You entered the room.>
[12/05/19 16:41:04] <Your partner has joined the room.>
[12/05/19 16:43:18]<You>How long can I store mole?
[12/05/19 16:43:42]<Partner> In the freezer, more-or-less indefinitely (Please do follow up questions)
TITLE:
How long can I store mole?
BACKGROUND FOR FOLLOW-UP QUESTIONS:
I'd like to have a go at making some kind of mole, but all the recipes I have seem to make big batches. How long can I keep mole for, either in the fridge or freezer?
Enter your question here.
(maximum 150 characters)
ENDCHAT
Figure 2: Dialogue collection interface for the user.
[12/05/19 16:41:02] <You entered the room.>
[12/05/19 16:41:03] <Your partner has joined the room.>
[12/05/19 16:43:18] <Partner> How long can I store mole?
[12/05/19 16:43:42] <You> In the freezer, more-or-less indefinitely (Please do follow up questions)
If the answer of the question is in the text:
Select one of these 3 options:
Just text span "Yes,..." No...
Select span of text from below (max. 200 characters)
In the freezer, more-or-less indefinitely, depending on how good your freezer is. Certainly months to years.