Web Search Engines Evolve to Meet Challenges

Finding information on the Web can challenge the most experienced searchers. Librarians teach patrons that choosing the correct Web search tool is half of the search battle. We recommend use of selective directory-type resources such as Yahoo! or LookSmart for general queries or to get acquainted with resources available on a topic, and large, crawler-built indexes like AltaVista or HotBot for more specific questions. But as professional researchers, we know that there are still a number of other key choices to be made along the road to an answer. For example, knowing which Web index to begin with and when to move on to a meta-search site such as Webcrawler, which will run the search against multiple sources simultaneously, are skills acquired from often frustrating trial-and-error experiences at the reference desk. Fortunately for ail of us, these popular search tools are evolving, based on extensive research into user needs and behaviors. Providers are finding unique ways to improve search results for the large Web indexes. In this column, we offer a summary of some important trends in this area. Smarter Tools Developers of Web indexes (or "search engines") are working to build more intelligence into search responses. The most promising new search tool in this regard is Ask Jeeves. In simple terms, Ask Jeeves simulates a reference interview. After the user enters a query in natural language, Jeeves provides a concise list of possible answers for the user to consider and explore. These answers are drawn from a large knowledge base of more than seven million items developed by professional researchers. Jeeves also provides a meta-search option that delivers answers from five other search engines. And Jeeves can get "smarter" over time by incorporating data about responses chosen by users. For example, in response to the question "Where do truffles grow?", Jeeves pointed to a brief encyclopedia article on truffles, which provided a good answer, and also displayed lists of hits from the major search engines. Jeeves alleviates user confusion by pointing first to the best answer and by organizing the remaining information in a compact and easy to navigate display. Because one significant factor in search failures is spelling errors, Jeeves also offers a spell-checking option for searches. AltaVista now offers a branded version of Ask Jeeves called "Ask AltaVista." If a natural language query is entered in AltaVista, several answers from the Ask Jeeves knowledge base are provided, along with a "More Answers" link that leads to a longer list of responses. Instead of results from five major search engines, only AltaVista search results follow the Ask Jeeves answers. When asked the very specific question "Where can I find information about diseases of Netherlands dwarf rabbits?" the AltaVista-branded service pointed to answers relevant for each of three major concepts identified in this question: rabbits, disease/disability, and Netherlands. The first answer on pet rabbits was linked to the House Rabbit Society Web site, which provided some good general health information. One other page from the AltaVista search results was useful. When the same question was entered directly into Ask Jeeves, similar answers were produced, but the additional hits from the other four search engines proved more useful in getting to an answer on the more specific aspects of this question than the AltaVista hits alone. Direct Hit, now available on HotBot, takes a different and somewhat cruder approach to applying intelligence to Web searching. This service uses the collective judgments of past searchers to inform current ones. The company's "Popularity Engine" tracks the sites that users select from the search results and also considers how much time users spend viewing these pages. The engine then uses this data to re-rank the HotBot listings, then offers searchers access to a list of the top ten most-visited sites for particular search words. …