Venue: NAACL 2019

# ComQA: A Community-sourced Dataset for

Complex Factoid Question Answering with Paraphrase Clusters

Abdalghani Abujabal\({}^{1}\), Rishiraja Saha Roy\({}^{2}\), Mohamed Yahya\({}^{3}\) and Gerhard Weikum\({}^{2}\)

\({}^{1}\)Amazon Alexa, Aachen, Germany

abujaba@amazon.de \({}^{2}\)Max Planck Institute for Informatics, Saarland Informatics Campus, Germany

{rishiraj, weikum}@mpi-inf.mpg.de \({}^{3}\)Bloomberg L.P., London, United Kingdom

yahya.mohamed@gmail.com

###### Abstract

To bridge the gap between the capabilities of the state-of-the-art in factoid question answering (QA) and what users ask, we need large datasets of real user questions that capture the various question phenomena users are interested in, and the diverse ways in which these questions are formulated. We introduce _ComQA_, a _large_ dataset of _real_ user questions that exhibit different _challenging aspects_ such as _compositionality_, _temporal reasoning_, and _comparisons_. ComQA questions come from the WikiAnswers community QA platform, which typically contains questions that are not satisfactorily answerable by existing search engine technology. Through a large crowdsourcing effort, we clean the question dataset, group questions into _paraphrase clusters_, and annotate clusters with their answers. ComQA contains \(11,214\) questions grouped into 4,834 paraphrase clusters. We detail the process of constructing ComQA, including the measures taken to ensure its high quality while making effective use of crowdsourcing. We also present an extensive analysis of the dataset and the results achieved by state-of-the-art systems on ComQA, demonstrating that our dataset can be a driver of future research on QA.

+
Footnote †: The main part of this work was carried out when the author was at the Max Planck Institute for Informatics.

+
Footnote †: The main part of this work was carried out when the author was at the Max Planck Institute for Informatics.

+
Footnote †: The main part of this work was carried out when the author was at the Max Planck Institute for Informatics.

+
Footnote †: The main part of this work was carried out when the author was at the Max Planck Institute for Informatics.

+
Footnote †: The main part of this work was carried out when the author was at the Max Planck Institute for Informatics.

+
Footnote †: The main part of this work was carried out when the author was at the Max Planck Institute for Informatics.

+
Footnote †: The main part of this work was carried out when the author was at the Max Planck Institute for Informatics.

## 1 Introduction

Factoid QA is the task of answering natural language questions whose answer is one or a small number of entities Voorhees and Tice (2000). To advance research in QA in a manner consistent with the needs of end users, it is important to have access to datasets that reflect _read_ user information needs by covering various _question phenomena_ and the wide _lexical and syntactic variety_ in expressing these information needs. The benchmarks should be large enough to facilitate the use of data-hungry machine learning methods. In this paper, we present ComQA, a large dataset of 11,214 real user questions collected from the WikiAnswers community QA website. As shown in Figure 1, the dataset contains various question phenomena. ComQA questions are grouped into 4,834 paraphrase clusters through a large-scale crowdsourcing effort, which capture lexical and syntactic variety. Crowdsourcing is also used to pair paraphrase clusters with answers to serve as a supervision signal for training and as a basis for evaluation.

Table 1 contrasts ComQA with publicly available QA datasets. The foremost issue that ComQA tackles is ensuring research is driven by information needs formulated by real users. Most large-scale datasets resort to highly-templatic synthetically generated natural language questions Bordes et al. (2015); Cai and Yates (2013); Su et al. (

Figure 1: ComQA paraphrase clusters covering a range of question aspects e.g., temporal and compositional questions, with lexical and syntactic diversity.



2016; Talmor and Berant, 2018; Trivedi et al., 2017). Other datasets utilize search engine logs to collect their questions (Berant et al., 2013), which creates a bias towards simpler questions that search engines can already answer reasonably well. In contrast, ComQA questions come from WikiAnswers, a community QA website where users pose questions to be answered by other users. This is often a reflection of the fact that such questions are beyond the capabilities of commercial search engines and QA systems. Questions in our dataset exhibit a wide range of interesting aspects such as the need for temporal reasoning (Figure 1, cluster 1), comparison (Figure 1, cluster 2), compositionality (multiple subquestions with multiple entities and relations) (Figure 1, cluster 3), and unanswerable questions (Figure 1, cluster 4).

ComQA is the result of a carefully designed large-scale crowdsourcing effort to group questions into paraphrase clusters and pair them with answers. Past work has demonstrated the benefits of paraphrasing for QA (Abujabal et al., 2018; Berant and Liang, 2014; Dong et al., 2017; Fader et al., 2013). Motivated by this, we judiciously use crowdsourcing to obtain clean paraphrase clusters from WikiAnswers' noisy ones, resulting in ones like those shown in Figure 1, with both lexical and syntactic variations. The only other dataset to provide such clusters is that of Su et al. (2016), but that is based on synthetic information needs.

For answering, recent research has shown that combining various resources for answering significantly improves performance (Savenkov and Agichtein, 2016; Sun et al., 2018; Xu et al., 2016). Therefore, we do not pair ComQA with a specific knowledge base (KB) or text corpus for answering. We call on the research community to innovate in combining different answering sources to tackle ComQA and advance research in QA. We use crowdsourcing to pair paraphrase clusters with answers. ComQA answers are primarily Wikipedia entity URLs. This has two motivations: (i) it builds on the example of search engines that use Wikipedia entities as answers for entitycentric queries (e.g., through knowledge cards), and (ii) most modern KBs ground their entities in Wikipedia. Wherever the answers are temporal or measurable quantities, we use TIMEX31 and the International System of Units2 for normalization. Providing canonical answers allows for better comparison of different systems.

Footnote 1: [http://www.timeml.org](http://www.timeml.org)

Footnote 2: [https://en.wikipedia.org/wiki/SI](https://en.wikipedia.org/wiki/SI)

We present an extensive analysis of ComQA, where we introduce the various question aspects of the dataset. We also analyze the results of running state-of-the-art QA systems on ComQA. ComQA exposes major shortcomings in these systems, mainly related to their inability to handle compositionality, time, and comparison. Our detailed error analysis provides inspiration for avenues of future work to ensure that QA systems meet the expectations of real users. To summarize, in this paper we make the following contributions:

* We present a dataset of 11,214 real user questions collected from a community QA website. The questions exhibit a range of aspects that are important for users and challenging for existing QA systems. Using crowdsourcing, questions are grouped into 4,834 paraphrase clusters that are annotated with answers. ComQA is available at: [http://qa.mpi-inf.mpg.de/comqa](http://qa.mpi-inf.mpg.de/comqa).
* We present an extensive analysis and quantify the various difficulties in ComQA. We also present the results of state-of-the art QA systems on ComQA, and a detailed error analysis.

\begin{table}
\begin{tabular}{p{95.4pt}|c|c|c|c} \hline
**Dataset** & **Large scale (\(>5k\))** & **Real Information Needs** & Complex Questions & **Question Paraphrases** \\ \hline
**CompQA** (This paper) & ✓ & ✓ & ✓ & ✓ \\ \hline Free917 (Cai and Yates, 2013) & ✗ & ✗ & ✗ & ✗ \\ WebQuestions (Berant et al., 2013) & ✓ & ✓ & ✗ & ✗ \\ SimpleQuestions (Bordes et al., 2015) & ✓ & ✗ & ✗ & ✗ \\ QALD (Ubeck et al., 2017) & ✗ & ✗ & ✓ & ✗ \\ LC-QuAD (Trivedi et al., 2017) & ✓ & ✗ & ✓ & ✗ \\ ComplexQuestions (Baet al., 2016) & ✗ & ✓ & ✓ & ✗ \\ GraphQuestions (Su et al., 2016) & ✓ & ✗ & ✓ & ✓ \\ ComplexWebQuestions (Tallnor and Berant, 2018) & ✓ & ✗ & ✓ & ✗ \\ TREC (Voorhees and Tice, 2000) & ✗ & ✓ & ✓ & ✗ \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 1: Comparison of ComQA with existing QA datasets over various dimensions.

 

Related Work

There are two main variants of the factoid QA task, with the distinction tied to the underlying answering resources and the nature of answers. Traditionally, QA has been explored over large textual corpora [21, 34, 56, 7] with answers being textual phrases. Recently, it has been explored over large structured resources such as KBs [8, 9], with answers being semantic entities. Recent work demonstrated that the two variants are complementary, and a combination of the two results in the best performance [20, 21].

**QA over textual corpora.** QA has a long tradition in IR and NLP, including benchmarking tasks in TREC [22, 10] and CLEF [34, 1]. This has predominantly focused on retrieving answers from textual sources [13, 1, 12, 14]. In IBM Watson [13], structured data played a role, but text was the main source for answers. The TREC QA evaluation series provide hundreds of questions to be answered over documents, which have become widely adopted benchmarks for answer sentence selection [23]. ComQA is orders of magnitude larger than TREC QA.

_Reading comprehension_ (RC) is a recently introduced task, where the goal is to answer a question from a _given_ textual paragraph [15, 16, 17, 18, 19]. This setting is different from factoid QA, where the goal is to answer questions from a large repository of data (be it textual or structured), and not a single paragraph. A recent direction in RC is dealing with unanswerable questions from the underlying data [16]. ComQA includes such questions to allow tackling the same problem in the context of factoid QA.

**QA over KBs.** Recent efforts have focused on natural language questions as an interface for KBs, where questions are translated to structured queries via semantic parsing [1, 12, 19, 18, 19, 20, 21, 22, 23, 24]. Over the past five years, many datasets were introduced for this setting. However, as Table 1 shows, they are either small in size [13, and ComplexQuestions], composed of synthetically generated questions (SimpleQuestions, GraphQuestions, LC-QuAD and ComplexWebQuestions), or are structurally simple (WebQuestions). ComQA addresses these shortcomings. Returning semantic entities as answers allows users to further explore these entities in various resources such as their Wikipedia pages, Freebase entries, etc. It also allows QA systems to tap into various interlinked resources for improvement (e.g., to obtain better lexicons, or train better NER systems). Because of this, ComQA provides semantically grounded reference answers in Wikipedia (without committing to Wikipedia as an answering resource). For numerical quantities and dates, ComQA adopts the International System of Units and TIMEX3 standards, respectively.

## 3 Overview

In this work, a factoid question is a question whose answer is one or a small number of entities or literal values [22] e.g., _"Who were the secretaries of state under Barack Obama?"_ and _"When was Germany's first postwar chancellor born?"_.

### Questions in ComQA

A question in our dataset can exhibit _one or more_ of the following phenomena:

* **Simple:** questions about a single property of an entity (e.g., _"Where was Einstein born?"_)
* **Compositional:** A question is compositional if answering it requires answering more primitive questions and combining these. These can be _intersection_ or _nested_ questions. Intersection questions are ones where two or more subquestions can be answered independently, and their answers intersected (e.g., _"Which films featuring Tom Hanks did Spielberg direct?"_). In nested questions, the answer to one subquestion is necessary to answer another (_"Who were the parents of the thirteenth president of the US?"_).
* **Temporal:** These are questions that require temporal reasoning for deriving the answer, be it explicit (e.g., _'in 1998'_), implicit (e.g., _'during the WWI'_), relative (e.g., _'current'_), or latent (e.g. _'Who is the US president?'_). Temporal questions also include those _whose answer 

is an explicit temporal expression (_"When did Trenton become New Jersey's capital?"_).
* **Comparison:** We consider three types of comparison questions: _comparatives_ (_"Which rivers in Europe are longer than the Rhine?"_), superlatives_ (_"What is the population of the largest city in Egypt?"_), and _ordinal_ questions (_"What was the name of Elvis's first movie?"_).
* **Telegraphic**Joshi et al. (2014): These are short questions formulated in an informal manner similar to keyword queries (_"First president India?"_). Systems that rely on linguistic analysis often fail on such questions.
* **Answer tuple**: Where an answer is a tuple of connected entities as opposed to a single entity (_"When and where did George H. Bush go to college, and what did he study?"_).

### Answers in ComQA

Recent work has shown that the choice of answering resource, or the combination of resources significantly affects answering performance Savenkov and Agichtein (2016); Sun et al. (2018); Xu et al. (2016). Inspired by this, ComQA is not tied to a specific resource for answering. To this end, answers in ComQA are primarily Wikipedia URLs. This enables QA systems to combine different answering resources which are linked to Wikipedia (e.g., DBpedia, Freebase, YAGO, Wikidata, etc). This also allows seamless comparison across these QA systems. An answer in ComQA can be:

* **Entity:** ComQA entities are grounded in Wikipedia. However, Wikipedia is inevitably incomplete, so answers that cannot be grounded in Wikipedia are represented as plain text. For example, the answer for _"What is the name of Kristen Stewart adopted brother?"_ is Taylor Stewart, Dana Stewart\(\}\).
* **Literal value:** Temporal answers follow the TIMEX3 standard. For measurable quantities, we follow the International System of Units.
* **Empty:** In the factoid setting, some questions can be based on a false premise, and hence, are unanswerable e.g., _"Who was the first human being on Mars?"_ (no human has been on Mars, yet). The correct answer to such questions is the empty set. Such questions allow systems to cope with these cases. Recent work has started looking at this problem Rajpurkar et al. (2018).

## 4 Dataset Construction

Our goal is to collect factoid questions that represent real information needs and cover a range of question aspects. Moreover, we want to have different paraphrases for each question. To this end, we tap into the potential of community QA platforms. Questions posed there represent real information needs. Moreover, users of those platforms provide (noisy) annotations around questions e.g., paraphrase clusters. In this work, we exploit the annotations where users mark questions as duplicates as a basis for paraphrase clusters, and clean those. Concretely, we started with the WikiAnswers crawl by Fader et al. (2014). We obtained ComQA from this crawl primarily through a large-scale crowdsourcing effort, which we describe in what follows.

The original resource curated by Fader et al. contains \(763\)M questions. Questions in the crawl are grouped into \(30\)M paraphrase clusters based on feedback from WikiAnswers users. This clustering has a low accuracy Fader et al. (2014). Extracting factoid questions and cleaning the clusters are thus essential for a high-quality dataset.

### Preprocessing of WikiAnswers

To remove non-factoid questions, we filtered out questions that (i) start with _'why'_, or (ii) contain words like _(dis)similarities, differences, (dis)advantages_, etc. Questions matching these filters are out of scope as they require a narrative answer. We also removed questions with less than three or more than twenty words, as we found these to be typically noisy or non-factoid questions. This left us with about \(21\)M questions belonging to \(6.1\)M clusters.

To further focus on factoid questions, we automatically classified questions into one or more of the following _four classes_: (1) temporal, (2) comparison, (3) single entity, and (4) multi-entity questions. We used SUTime Chang and Manning (2012) to identify temporal questions and the Stanford named entity recognizer Finkel et al. (2005) to detect named entities. We used part-of-speech patterns to identify comparatives, superlattices, and ordinals. Clusters which did not have questions belonging to any of the above classes were discarded from further consideration. Although these clusters contain false negatives e.g., _"What official position did Mendeelev hold until his death?"_ due to errors by the tagging tools, 

most discarded questions are out-of-scope.

**Manual inspection.** We next applied the first stage of human curation to the dataset. Each WikiAnswers cluster was assigned to one of the four classes above based on the majority label of the questions within. We then randomly sampled \(15\)K clusters from each of the four classes (60K clusters in total with \(482\)K questions) and sampled a representative question from each of these clusters at random (60K questions). We relied on the assumption that questions within the same cluster are semantically equivalent. These 60K questions were manually examined by the authors and those with unclear or non-factoid intent were removed along with the cluster that contains them. We thus ended up with \(2.1\)K clusters with \(13.7\)K questions.

### Curating Paraphrase Clusters

We inspected a random subset of the \(2.1\)K WikiAnswers clusters and found that questions in the same cluster are semantically _related but not necessarily equivalent_, which is in line with observations in previous work [9]. Dong et al. dong2017multiview reported that 45% of question pairs were related rather than genuine paraphrases. For example, Figure 2 shows 10 questions in the same WikiAnswers cluster. Obtaining accurate paraphrase clusters is crucial to any systems that want to utilize them [1, 10, 11]. We therefore utilized crowdsourcing to clean the Wikianswers paraphrase clusters. We used Amazon Mechanical Turk (AMT) to identify semantically equivalent questions within a WikiAnswers cluster, thereby obtaining cleaner clusters for ComQA. Once we had the clean clusters, we set up a second AMT task to collect answers for each ComQA cluster.

**Task design.** We had to ensure the simplicity of the task to obtain high quality results. Therefore, rather than giving workers a WikiAnswers cluster and asking them to partition it into clusters of paraphrases, we showed them pairs of questions from a cluster and asked them to make the binary decision of whether the two questions are paraphrases. To reduce potentially redundant annotations, we utilized the transitivity of the paraphrase relationship. Given a WikiAnswers cluster \(Q=\{q_{1},...,q_{n}\}\), we proceed in rounds to form ComQA clusters. In the first round, we collect annotations for each pair \((q_{i},q_{i+1})\). The majority annotation among five annotators is taken. An initial clustering is formed accordingly, with clusters sharing the same question merged together (to account for transitivity). This process continues iteratively until no new clusters can be formed from \(Q\).

**Task statistics.** We obtained annotations for 18,890 question pairs from 175 different workers. Each pair was shown to five different workers, with \(65.7\%\) of the pairs receiving unanimous agreement, \(21.4\%\) receiving four agreements and \(12.9\%\) receiving three agreements. By design, with five judges and binary annotations, no pair can have less three agreements. This resulted in questions being placed in paraphrase clusters, and no questions were discarded at this stage. At the end of this step, the original \(2.1\)K WikiAnswers clusters became \(6.4\)K ComQA clusters with a total of \(13.7\)K questions. Figure 3 shows the distribution of questions in clusters.

To test whether relying on the transitivity of the

Figure 3: The distribution of questions in clusters.

Figure 2: A WikiAnswers cluster split into four clusters by AMT Turkers.

 

paraphrase relationship is suitable to reduce the annotation effort, we asked annotators to annotate 1,100 random pairs \((q_{1},q_{3})\), where we had already received positive annotations for the pairs \((q_{1},q_{2})\) and \((q_{2},q_{3})\) being paraphrases of each other. In \(93.5\)% of the cases there was agreement. Additionally, as experts on the task, the authors manually assessed \(600\) pairs of questions, which serve as honeypots. There was \(96.6\%\) agreement with our annotations. An example result of this task is shown in Figure 2, where Turkers split the original WikiAnswers cluster into the four clusters shown.

### Answering Questions

We were now in a position to obtain an answer annotation for each of the \(6.4\)K clean clusters.

**Task design.** To collect answers, we designed another AMT task, where workers were shown a representative question randomly drawn from a cluster. Workers were asked to use the Web to find answers and to provide the corresponding URLs of Wikipedia entities. Due to the inevitable incompleteness of Wikipedia, workers were asked to provide the surface form of an answer entity if it does not have a Wikipedia page. If the answer is a full date, workers were asked to follow dd-mmm-yyyy format. For measurable quantities, workers were asked to provide units. We use TIMEX3 and the international system of units for normalizing temporal answers and measurable quantities e.g., _'12th century'_ to lXX. If no answer is found, workers were asked to type ip in _'no answer'_.

**Task statistics.** Each representative question was shown to three different workers. An answer is deemed correct if it is common between at least two workers. This resulted in \(1.6\)K clusters (containing \(2.4\)K questions) with no agreed-upon answers, which were dropped. For example, _"Who was the first democratically elected president of Mexico?"_ is subjective. Other questions received related answers e.g., _"Who do the people in Iraq worship?"_ with Allah, Islam and Mohamed as answers from the three annotators. Other questions were underspecified e.g., _"Who was elected the vice president in 1796?"_. At the end of the task, we ended up with 4,834 clusters with 11,214 question-answer pairs, which form ComQA.

## 5 Dataset Analysis

In this section, we present a manual analysis of \(300\) questions sampled at random from the ComQA dataset. This analysis helps understand the different aspects of our dataset. A summary of the analysis is presented in Table 2.

**Question categories.** We categorized each question as either simple or complex. A question is complex if it belongs to one or more of the compositional, temporal, or comparison classes. \(56.33\%\) of the questions were complex; \(32\%\) compositional, \(23.67\%\) temporal, and \(29.33\%\) contain

\begin{table}
\begin{tabular}{l c r} \hline
**Property** & **Example** & **Percentage\%** \\ \hline \multicolumn{3}{c}{_Compositional questions_} \\ \hline Conjunction & “_What is the capital of the country whose northern border is Poland and Germany?”_ & \(17.67\) \\ Nested & “_When is Will Smith’s oldest son’s birthday?”_ & \(14.33\) \\ \hline \multicolumn{3}{c}{_Temporal questions_} \\ \hline Explicit time & “_Who was the winner of the World Series in **in 1994?**_ & \(4.00\) \\ Implicit time & “_Who was Britain’s leader **during WW1?**_ & \(4.00\) \\ Temporal answer & “_When did Trenton become New Jersey’s capital? ”_ & \(15.67\) \\ \hline \multicolumn{3}{c}{_Comparison questions_} \\ \hline Comparative & “_Who was the first US president to serve **2 terms?**_ & \(1.00\) \\ Superdrive & “_What ocean does the **longest** river in the world flow into?_ & \(14.33\) \\ Ordinal & “_When was Thomas Edisons **first** wife born?”_ & \(14.00\) \\ \hline \multicolumn{3}{c}{_Question formulation_} \\ \hline Telegraphic & “_Neyo first album?”_ & \(8.00\) \\ \hline \multicolumn{3}{c}{_Entity distribution in questions_} \\ \hline Zero entity & “_What public company has the most employees in the world?”_ & \(2.67\) \\ Single entity & “_Who is **Brad Walst’s wife?**_ & \(75.67\) \\ Multi-entity & “_What country in **South America** lies between **Brazil** and **Argentina?**_ & \(21.67\) \\ \hline \multicolumn{3}{c}{_Other features_} \\ \hline Answer tuple & “_Where was Ryton Manning born and **what year was he born?**_ & \(2.00\) \\ Empty answer & “_Who was Calgay’s first woman mayor?”_ & \(3.67\) \\ \hline \end{tabular}
\end{table}
Table 2: Results of the manual analysis of \(300\) questions. Note that properties are not mutually exclusive.

 

comparison conditions. A question may contain multiple conditions (_"What country has the **highest population in the year 2008?"_ with comparison and temporal conditions).

We also identified questions of telegraphic nature e.g., _"Julia Alvarez's parents?"_, with \(8\%\) of our questions being telegraphic. Such questions pose a challenge for systems that rely on linguistic analysis of questions Joshi et al. (2014).

We counted the number of named entities in questions: \(23.67\%\) contain two or more entities, reflecting their compositional nature, and \(2.67\%\) contain no entities e.g., _"What public company has the most employees in the world?"_. Such questions can be hard as many methods assume the existence of a pivot entity in a question.

Finally, \(3.67\%\) of the questions are unanswerable, e.g., _"Who was the first human being on Mars?"_. Such questions incentivise QA systems to return non-empty answers only when suitable. In Table 3 we compare ComQA with other current datasets based on real user information needs over different question categories.

Answer types.We annotated each question with the most fine-grained context-specific answer type Ziegler et al. (2017). Answers in ComQA belong to a diverse set of types that range from coarse (e.g., person) to fine (e.g., sports manager). Types also include literals such as number and date. Figure 4 shows answer types of the \(300\) annotated examples as a word cloud.

Question topics.We annotated questions with topics to which they belong (e.g., geography, movies, sports). These are shown in Figure 4, and demonstrate the topical diversity of ComQA.

Question length.Questions in ComQA are fairly long, with a mean length of 7.73 words, indicating the compositional nature of questions.

## 6 Experiments

In this section we present experimental results for running ComQA through state-of-the-art QA systems. Our experiments show that these systems achieve humble performance on ComQA. Through a detailed analysis, this performance can be attributed to systematic shortcomings in handling various question aspects in ComQA.

### Experimental Setup

Splits.We partition ComQA into a random train/dev/test split of 70/10/20% with 7,850, 1,121 and 2,243 questions, respectively.

Metrics.We follow the community's standard evaluation metrics: we compute average precision, recall, and F1 scores across all test questions. For unanswerable questions whose correct answer is the empty set, we define precision and recall to be \(1\) for a system that returns an empty set, and \(0\) otherwise Rajpurkar et al. (2018).

### Baselines

We evaluated two categories of QA systems that differ in the underlying answering resource: either KBs or textual extractions. We ran the following systems: (i) Abujabal et al. (2017), which automatically generates templates using questionnaire pairs; (ii) Bast and Haussmann (2015), which instantiates hand-crafted query templates followed by query ranking; (iii) Berant and Liang (2015), which relies on agenda-based parsing and imitation learning; (iv) Berant et al. (2013), which uses rules to build queries from questions; and (v) Fader et al. (2013), which maps questions to queries over open vocabulary facts extracted from Web documents. Note that our intention is not to assess the quality of these systems, but to assess how challenging ComQA is.

The systems were trained with ComQA data. All systems were run over the data sources for which they were designed. The first four baselines are over Freebase. We therefore mapped ComQA answers (Wikipedia entities) to the corresponding Freebase names using the information stored with entities in Freebase. We observe that the Wikipedia answer entities have no counterpart in Freebase for \(7\%\) of the ComQA questions. This suggests an oracle F1 score of \(93.0\). For Fader et al. (2013), which is over web extractions, we mapped Wikipedia URLs to their titles.

Figure 4: Answer types and question topics on \(300\) annotated examples as word clouds.

 



### Results

Table 4 shows the performance of the baselines on the ComQA test set. Overall, the systems achieved poor performance, suggesting that current methods cannot handle the complexity of our dataset, and that new models for QA are needed. Table 5 compares the performance of the systems on different datasets (Free917 uses accuracy as a quality metric). For example, while Abujabal et al. (2017) achieved an F1 score of \(51.0\) on WebQuestions, it achieved \(22.4\) on ComQA.

The performance of fader-etal:2013:EMNLP is worse than the others due to the incompleteness of its underlying extractions and the complexity of ComQA questions that require higher-order relations and reasoning. However, the system answered some complex questions, which KB-QA systems failed to answer. For example, it answered _"What is the highest mountain in the state of Washington?"_. The answer to such a question is more readily available in Web text, compared to a KB, where more sophisticated reasoning is required to handle the superlative. However, a slightly modified question such as _"What is the fourth highest mountain in the state of Washington?"_ is unlikely to be found in text, but be answered using KBs with the appropriate reasoning. Both examples above demonstrate the benefits of combining text and structured resources.

### Error Analysis

For the two best performing systems on ComQA, QUINT Abujabal et al. (2017) and AQQU Bast and Haussmann (2015), we manually inspected 100 questions on which they failed. We classified failure sources into four categories: compositionality, temporal, comparison or NER. Table 6 shows the distribution of these failure sources.

**Compositionality.** Neither system could handle the compositional nature of questions. For example, they returned the father of Julius Caesar as an answer for _"What did Julius Caesar's father work as?"_, while, the question requires another KB predicate that connects the father to his profession. For _"John Travolta and Jamie Lee Curtis starred in this movie?"_, both systems returned movies with Jamie Lee Curtis, ignoring the constraint that John Travolta should also appear in them. Properly answering multi-relation questions over KBs remains an open problem.

**Temporal.** Our analysis reveals that both systems fail to capture temporal constraints in questions, be it explicit or implicit. For _"Who won the Oscar for Best Actress in 1986?"_, they returned all winners and ignored the temporal restriction from _'in 1986'_. Implicit temporal constraints like named events (e.g., _'Vietnam war'_ in _"Who was the president of the US during Vietnam war?"_) pose a challenge to current methods. Such constraints need to be detected first and normalized to a canonical time interval (November 1st, 1955 to April 30th, 1975, for the Vietnam war). Then, systems need to compare the terms of the US presidents with the above interval to account for the temporal relation of _'during'_. While detecting explicit time expressions can be done reasonably well using existing time taggers Chang and Manning (2012), identifying implicit ones is difficult. Furthermore, retrieving the correct temporal scopes of entities in questions (e.g., the terms of the US presidents) is hard due to the large num

\begin{table}
\begin{tabular}{l|c c c} \hline  & **Avg. Prec** & **Avg. Rec** & **Avg. F1** \\ \hline Abujabal et al. (2017) & \(21.2\) & \(38.4\) & \(22.4\) \\ Bast and Haussmann (2015) & \(20.7\) & \(37.6\) & \(21.6\) \\ Berant and Liang (2015) & \(10.7\) & \(15.4\) & \(10.6\) \\ Berant et al. (2013) & \(13.7\) & \(20.1\) & \(12.0\) \\ Fader et al. (2013) & \(7.22\) & \(6.59\) & \(6.73\) \\ \hline \end{tabular}
\end{table}
Table 4: Results of baselines on ComQA test set.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} \hline
**Dataset** & **Size** & **compositional** & **Temporal** & **Comparison** & **Telegraphic** & **Empty Answer** \\ \hline
**ComQA** & \(11,214\) & \(32\%\) & \(24\%\) & \(30\%\) & \(8\%\) & \(4\%\) \\ \hline WebQuestions Berant et al. (2013) & \(5,810\) & \(2\%\) & \(7\%\) & \(2\%\) & \(0\%\) & \(0\%\) \\ ComplexQuestions Bao et al. (2016) & \(2,100\) & \(39\%\) & \(34\%\) & \(9\%\) & \(0\%\) & \(0\%\) \\ \hline \end{tabular}
\end{table}
Table 3: Comparison of ComQA with existing datasets over various phenomena. We manually annotated \(100\) random questions from each dataset.



ber of temporal KB predicates associated with entities.

**Comparison.** Both systems perform poorly on comparison questions, which is expected since they were not designed to address those. To the best of our knowledge, no existing KB-QA system can handle comparison questions. Note that our goal is not to assess the quality the of current methods, but to highlight that these methods miss categories of questions that are important to real users. For _"What is the first film Julie Andrews made?"_ and _"What is the largest city in the state of Washington?"_, both systems returned the list of Julie Andrews's films and the list of Washington's cities, for the first and the second questions, respectively. While the first question requires the attribute of filmReleasedIn to order by, the second needs the attribute of hasArea. Identifying the correct attribute to order by as well as determining the order direction (ascending for the first and descending for the second) is challenging and out of scope for current methods.

**NER.** NER errors come from false negatives, where entities are not detected. For example, in _"On what date did the Mexican Revolution end?"_ QUINT identified _'Mexican'_ rather than _'Mexican Revolution'_ as an entity. For _"What is the first real movie that was produced in 1903?"_, which does not ask about a specific entity, QUINT could not generate SPARQL queries. Existing QA methods expect a pivotal entity in a question, which is not always the case.

Note that while baseline systems achieved low precision, they achieved higher recall (21.2 vs 38.4 for QUINT, respectively) (Table 4). This reflects the fact that these systems often cannot cope with the full complexity of ComQA questions, and instead end up evaluating underconstrained interpretations of the question.

To conclude, current methods can handle simple questions very well, but struggle with complex questions that involve multiple conditions on different entities or need to join the results from sub-questions. Handling such complex questions, however, is important if we are to satisfy information needs expressed by real users.

## 7 Conclusion

We presented ComQA, a dataset for QA that harnesses a community QA platform, reflecting questions asked by real users. ComQA contains 11,214 question-answer pairs, with questions grouped into paraphrase clusters through crowdsourcing. Questions exhibit different aspects that current QA systems struggle with. ComQA is a challenging dataset that is aimed at driving future research on QA, to match the needs of real users.

## Acknowledgments

We would like to thank Tommaso Pasini for his helpful feedback.

## References

* Abujabal et al. (2018) Abdalghani Abujabal, Rishiraja Saha Roy, Mohamed Yahya, and Gerhard Weikum. 2018. Never-ending learning for open-domain question answering over knowledge bases. In _WWW_, pages 1053-1062.
* Abujabal et al. (2017) Abdalghani Abujabal, Mohamed Yahya, Mirek Riedewald, and Gerhard Weikum. 2017. Automated template generation for question answering over knowledge graphs. In _WWW_, pages 1191-1200.
* Bao et al. (2016) Jun-Wei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016. Constraint-based question answering with knowledge graph. In _COLING_, pages 2503-2514.
* Bast and Haussmann (2015) Hannah Bast and Elmar Haussmann. 2015. More accurate question answering on freebase. In _CIKM_, pages 1431-1440.
* Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In _EMNLP_, pages 1533-1544.
* Berant and Liang (2014) Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In _ACL_, pages 1415-1425.
* Berant and Liang (2015) Jonathan Berant and Percy Liang. 2015. Imitation learning of agenda-based semantic parsers. _TACL_, 3:545-558.
* Bordes et al. (2015) Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. _arXiv_.
* Bader et al. (2016)

\begin{table}
\begin{tabular}{l|c|c} \hline \hline
**Category** & **QUINT** & **AQQU** \\ \hline Compositionality error & \(39\%\) & \(43\%\) \\ Missing comparison & \(31\%\) & \(26\%\) \\ Missing temporal constraint & \(19\%\) & \(22\%\) \\ NER error & \(11\%\) & \(9\%\) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Distribution of failure sources on ComQA questions on which QUINT and AQQU failed.

 

* Cai and Yates (2013) Qingqing Cai and Alexander Yates. 2013. Large-scale semantic parsing via schema matching and lexicon extension. In _ACL_, pages 423-433.
* Chang and Manning (2012) Angel X. Chang and Christopher D. Manning. 2012. Sutime: A library for recognizing and normalizing time expressions. In _LREC_, pages 3735-3740.
* Cui et al. (2005) Hang Cui, Renxu Sun, Keya Li, Min-Yen Kan, and TatSeng Chua. 2005. Question answering passage retrieval using dependency relations. In _SIGIR_, pages 400-407.
* Dietz and Gamari (2017) Laura Dietz and Ben Gamari. 2017. TREC CAR: A data set for complex answer retrieval. version 1.4.(2017).
* Dong et al. (2017) Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017. Learning to paraphrase for question answering. In _EMNLP_, pages 875-886.
* Fader et al. (2014) Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In _KDD_, pages 1156-1165.
* Fader et al. (2013) Anthony Fader, Luke S. Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question answering. In _ACL_, pages 1608-1618.
* Ferrucci (2012) David A. Ferrucci. 2012. This is watson. _IBM Journal of Research and Development_, 56(3):1.
* Finkel et al. (2005) Jenny Rose Finkel, Trond Grenager, and Christopher D. Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In _ACL_, pages 363-370.
* Harabagiu et al. (2006) Sanda M. Harabagiu, V. Finley Lacatusu, and Andrew Hickl. 2006. Answering complex questions with random walk models. In _SIGIR_, pages 220-227.
* Harabagiu et al. (2003) Sanda M. Harabagiu, Steven J. Maiorano, and Marius Pasca. 2003. Open-domain textual question answering techniques. _Natural Language Engineering_, 9(3):231-267.
* Harabagiu et al. (2001) Sanda M. Harabagiu, Dan I. Moldovan, Marius Pasca, Rada Mihalcea, Mihai Surdeanu, Razvan C. Bunescu, Roxana Girju, Vasile Rus, and Paul Morarescu. 2001. The role of lexico-semantic feedback in open-domain textual question-answering. In _ACL_, pages 274-281.
* Herrera et al. (2004) Jesus Herrera, Anselmo Penas, and Felisa Verdejo. 2004. Question answering pilot task at CLEF 2004. In _CLEF_, pages 581-590.
* Joshi et al. (2014) Mandar Joshi, Uma Sawant, and Soumen Chakrabarti. 2014. Knowledge graph and corpus driven segmentation and answer inference for telegraphic entity-seeking queries. In _EMNLP_, pages 1104-1114.
* Kocisky et al. (2017) Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Melis, and Edward Grefenstette. 2017. The narrativeqa reading comprehension challenge. _CoRR_, abs/1712.07040.
* Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017. RACE: large-scale reading comprehension dataset from examinations. In _EMNLP_, pages 785-794.
* Magnini et al. (2004) Bernardo Magnini, Alessandro Vallin, Christelle Ayache, Gregor Erbach, Anselmo Penas, Maarten de Rijke, Paulo Rocha, Kiril Ivanov Simov, and Richard F. E. Sutcliffe. 2004. Overview of the CLEF 2004 multilingual question answering track. In _CLEF_, pages 371-391.
* Mohammed et al. (2018) Salman Mohammed, Peng Shi, and Jimmy Lin. 2018. Strong baselines for simple question answering over knowledge graphs with and without neural networks. In _NAACL-HLT_, pages 291-296.
* Prager et al. (2004) John M. Prager, Jennifer Chu-Carroll, and Krzysztof Czuba. 2004. Question answering using constraint satisfaction: Qa-by-dossier-with-contraints. In _ACL_, pages 574-581.
* Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In _ACL_, pages 784-789.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In _EMNLP_, pages 2383-2392.
* Ravichandran and Hovy (2002) Deepak Ravichandran and Eduard H. Hovy. 2002. Learning surface text patterns for a question answering system. In _ACL_, pages 41-47.
* Reddy et al. (2014) Siva Reddy, Mirella Lapata, and Mark Steedman. 2014. Large-scale semantic parsing without question-answer pairs. _TACL_, pages 377-392.
* Saquete et al. (2009) Estela Saquete, Jose Luis Vicedo Gonzalez, Patricio Martinez-Barco, Rafael Munoz, and Hector Llorens. 2009. Enhancing QA systems with complex temporal question processing capabilities. _J. Artif. Intell. Res._
* Saquete et al. (2004) Estela Saquete, Patricio Martinez-Barco, Rafael Munoz, and Jose Luis Vicedo Gonzalez. 2004. Splitting complex temporal questions for question answering systems. In _ACL_, pages 566-573.
* Sevenkov and Agichtein (2016) Denis Sevenkov and Eugene Agichtein. 2016. When a Knowledge Base Is Not Enough: Question Answering over Knowledge Bases with External Text Data. In _SIGIR_, pages 235-244.
* Su et al. (2016) Yu Su, Huan Sun, Brian Sadler, Mudhakar Srivarusa, Izzeddin Gur, Zenghui Yan, and Xifeng Yan. 2016. On generating characteristic-rich question sets for QA evaluation. In _EMNLP_, pages 562-572.
* Sun et al. (2018) Haitian Sun, Buwan Dhingra, Manzil Zaheer, Kathryn Mazaiitis, Ruslan Salakhutdinov, and William W Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. _EMNLP_ 

Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In _NAACL_, pages 641-651.
* Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In _ReptNLP@ACL_, pages 191-200.
* Trivedi et al. (2017) Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. 2017. Lc-quad: A corpus for complex question answering over knowledge graphs. In _ISWC_, pages 210-218.
* Unger et al. (2012) Christina Unger, Lorenz Buhmann, Jens Lehmann, Axel-Cyrille Ngonga Ngomo, Daniel Gerber, and Philipp Cimiano. 2012. Template-based question answering over RDF data. In _WWW_, pages 639-648.
* Usbeck et al. (2017) Ricardo Usbeck, Axel-Cyrille Ngonga Ngomo, Bastian Haarmann, Anastasia Krithara, Michael Roder, and Giulio Napolitano. 2017. 7th Open Challenge on Question Answering over Linked Data (QALD-7). In _SemWebEval_.
* Voorhees and Tice (2000) Ellen M. Voorhees and Dawn M. Tice. 2000. Building a question answering test collection. In _SIGIR_, pages 200-207.
* Wang and Nyberg (2015) Di Wang and Eric Nyberg. 2015. A long short-term memory model for answer sentence selection in question answering. In _ACL_, pages 707-712.
* Xu et al. (2016) Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016. Question answering on Freebase via relation extraction and textual evidence. In _ACL_.
* Yahya et al. (2013) Mohamed Yahya, Klaus Berberich, Shady Elbassuoni, and Gerhard Weikum. 2013. Robust question answering over the web of linked data. In _CIKM_, pages 1107-1116.
* Yang et al. (2014) Min-Chul Yang, Nan Duan, Ming Zhou, and Hae Chang Rim. 2014. Joint relational embeddings for knowledge-based question answering. In _EMNLP_, pages 645-650.
* Yang et al. (2015) Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In _EMNLP_, pages 2013-2018.
* Yao and Van Durme (2014) Xuchen Yao and Benjamin Van Durme. 2014. Information Extraction over Structured Data: Question Answering with Freebase. In _ACL_, pages 956-966.
* Yin et al. (2015) Pengcheng Yin, Nan Duan, Ben Kao, Junwei Bao, and Ming Zhou. 2015. Answering questions with complex semantic constraints on open knowledge bases. In _CIKM_, pages 1301-1310.
* Ziegler et al. (2017) David Ziegler, Abdalghani Abujabal, Rishiraj Saha Roy, and Gerhard Weikum. 2017. Efficiency-aware answering of compositional questions using answer type prediction. In _IJCNLP_, pages 222-227.