Panel on Web-Based Question Answering

Early TREC-style Question Answering Systems were characterized by the following features: (a) the answer of the question was known to be included in a given local corpus, (b) the size of the small corpus permitted preprocessing, including named entity extraction and parsing of all documents, and (c) the corpus consisted of well-written news documents. More recently, QA Systems have started to use the Web as a corpus, either by extracting answers from the Web rather than a local corpus or by learning lexical patterns from the Web which are then used to improve the system itself. Using the Web for Question Answering presents an interesting combination of opportunities and challenges. This panel will discuss how to leverage the opportunities while addressing the inevitable challenges. The Web as a repository of answers Scaling QA systems to the Web presents an extraordinary challenge. The collections used in TREC8-10 contained a couple of hundred thousand documents while Google indexes more than 3 Billion Web pages. While preprocessing (POS tagging, chunking, semantic entity annotation) the TREC corpus is feasible, doing the same with Google’s index is out of the question with current technology. Furthermore, the TREC corpus is limited to a short and fixed time period while new documents are constantly added to the Web. Web-based question answering systems therefore also need to deal with issues such as redundancy, recency, and reliability. While the Web presents a number of interesting challenges to Question Answering, it is important to observe that a number of existing systems successfully address many of them and achieve high-quality results. Many of the systems discussed so far use existing search engines such as Google as their backends. Even though Copyright © 2002, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. Google’s performance at getting documents containing the answers to natural language questions even without making changes to the questions (e.g., see Radev et al. 01), most systems deploy query modulation to convert each question to a query in the specific language of the search engine. In addition to query modulation and an interface to public search engines, all systems also employ some form of passage retrieval and passage ranking. To some extent, all systems are architecturally isomorphic. At the same time, all major systems exhibit some interesting properties that set them apart from each other. MIT’s START system (Katz 97) has been on line for a number of years and has traditionally focused more on questions about the MIT InfoLab and geography. Recently, it has expanded to include general domain questions. Ask Jeeves (www.ask.com) is the most popular search engine that allows natural language questions. User questions are then matched against a database of known questions and answers. Ask Jeeves is not, strictly speaking, a Q&A system, given that its results come as documents, rather than answers. U. Washington’s system, Mulder (Kwok et al. 01) was the first major Q&A system to exploit the Web as a source of answers. Mulder sends a question to several search engines and uses a natural language parser to extract likely answers. The Waterloo team (Clarke et al. 01) reduce the problem of question answering to a two-stage process: first, they extract relevant passages using IR techniques, then decide what passage is most likely to contain the answer by its frequency. Microsoft’s system (Dumais et al. 02) also looks at redundancy of n-grams to identify good answers. Michigan’s system, NSIR (Radev et al. 02), uses probabilistic phrase reranking based on part of speech From: AAAI Technical Report SS-03-07. Compilation copyright © 2003, AAAI (www.aaai.org). All rights reserved.