Defining Evaluation Methodologies for Topical Crawlers

Topical crawlers are becoming important tools to support applications such as specialized Web portals, online searching, and competitive intelligence. As the Web mining field matures, the disparate crawling strategies proposed in the literature will have to be evaluated and compared on common tasks through well-defined performance measures. We have argued that general evaluation methodologies are necessary for topical crawlers. Such methodologies should (1) characterize classes of crawling tasks that support applications of different nature and difficulty, and (2) define performance measures for fair comparative evaluations of crawlers with respect to notions of precision, recall, and efficiency that are appropriate and practical for the Web. We have proposed a general framework for the evaluation of topical crawlers that relies on independent relevance judgments compiled by human editors and available from public directories. Such a framework synthesizes a number of methodologies in the topical crawlers literature and many lessons learned from several studies conducted by our group. It is our desire to generate a debate that can ultimately lead to some consensus in how to standardize the evaluation of topical crawlers, and possibly even to a crawling task in the Web Track of TREC. Background and Motivation Topical crawlers, also known as topic driven, focused, or preferential crawlers, are an important class of crawler programs that complement search engines. Search engines serve the general population of Web users. In contrast, topical crawlers are activated in response to particular information needs. These could be from an individual user (query time or online crawlers) or from a community with shared interests (topical or vertical search engines and portals). The crawlers underlying search engines are designed to fetch as comprehensive a snapshot of the Web as is possible; topical crawlers are designed to target portions of the Web that are relevant to the triggering topic. Such crawlers have the advantage that they may in fact be driven by a rich context (topics, queries, user profiles) within which to interpret pages and select the links to visit. Today, topical crawlers have become the basis for many specialized services such as investment portals, competitive intelligence tools, and scientific paper repositories. One research area that is gathering increasing momentum is the evaluation of topical crawlers. The rich legacy of information retrieval research comparing retrieval algorithms in the non-Web context offers many evaluation methods and measures that may be applied toward this end. However, given that the dimensions of the crawler evaluation problem are dramatically different, the design of appropriate evaluation strategies is a valid challenge. In a general sense, a crawler may be evaluated on its ability to retrieve “good” pages. However, a major hurdle is the problem of recognizing these good pages. In an operational environment real users may judge the relevance of pages as these are crawled allowing us to determine if the crawl was successful or not. Unfortunately, meaningful experiments involving real users for assessing Web crawls are extremely problematic. For instance the very scale of the Web suggests that in order to obtain a reasonable notion of crawl effectiveness one must conduct a large number of crawls, i.e., involve a large number of users. Crawls against the live Web also pose serious time constraints. Therefore crawls other than short-lived ones will seem overly burdensome to the user. We may choose to avoid these time loads by showing the user the results of the full crawl — but this again limits the extent of the crawl. Next we may choose indirect methods such as inferring ∗Contact author. Tel: +1-319-335-5708, Fax: +1-319-335-5374. Email: p dmini-srinivasan@uiowa.edu . Partially supported by National Library of Medicine grant No. RO1-LM06909 †Current affiliation: School of Informatics and Computer Science Department, Indiana University. Email: fil@indiana.edu . Partially supported by National Science Foundation CAREER grant No. IIS-0133124