Abstract: Designing a reliable natural language (NL) interface for querying tables has been a longtime goal of researchers in both the data management and natural language processing (NLP) communities. Such an interface receives as input an NL question, translates it into a formal query, executes the query and returns the results. Errors in the translation process are not uncommon, and users typically struggle to understand whether their query has been mapped correctly. We address this problem by explaining the obtained formal queries to non-expert users. Two methods for query explanations are presented: the first translates queries into NL, while the second method provides a graphic representation of the query cell-based provenance (in its execution on a given table). Our solution augments a state-of-the-art NL interface over web tables, enhancing it in both its training and deployment phase. Experiments, including a user study conducted on Amazon Mechanical Turk, show our solution to improve both the correctness and reliability of an NL interface.

Introduction: Natural language interfaces have been gaining significant popularity, enabling ordinary users to write and execute complex queries. One of the prominent paradigms for developing NL interfaces is semantic parsing, which is the mapping of NL phrases into a formal language. As Machine Learning techniques are standardly used in semantic parsing, a training set of question-answer pairs is provided alongside a target database BIBREF0 , BIBREF1 , BIBREF2 . The parser is a parameterized function that is trained by updating its parameters such that questions from the training set are translated into queries that yield the correct answers. A crucial challenge for using semantic parsers is their reliability. Flawless translation from NL to formal language is an open problem, and even state-of-the-art parsers are not always right. With no explanation of the executed query, users are left wondering if the result is actually correct. Consider the example in Figure FIGREF1 , displaying a table of Olympic games and the question "Greece held its last Olympics in what year?". A semantic parser parsing the question generates multiple candidate queries and returns the evaluation result of its top ranked query. The user is only presented with the evaluation result, 2004. Although the end result is correct, she has no clear indication whether the question was correctly parsed. In fact, the interface might have chosen any candidate query yielding 2004. Ensuring the system has executed a correct query (rather than simply returning a correct answer in a particular instance) is essential, as it enables reusing the query as the data evolves over time. For example, a user might wish for a query such as "The average price of the top 5 stocks on Wall Street" to be run on a daily basis. Only its correct translation into SQL will consistently return accurate results. Our approach is to design provenance-based BIBREF3 , BIBREF4 query explanations that are extensible, domain-independent and immediately understandable by non-expert users. We devise a cell-based provenance model for explaining formal queries over web tables and implement it with our query explanations, (see Figure FIGREF1 ). We enhance an existing NL interface for querying tables BIBREF5 by introducing a novel component featuring our query explanations. Following the parsing of an input NL question, our component explains the candidate queries to users, allowing non-experts to choose the one that best fits their intention. The immediate application is to improve the quality of obtained queries at deployment time over simply choosing the parser's top query (without user feedback). Furthermore, we show how query explanations can be used to obtain user feedback which is used to retrain the Machine Learning system, thereby improving its performance.

Question: How was lexical diversity measured?
Answer: By computing number of unique responses and number of responses divided by the number of unique responses to that question for each of the questions