Optimizing Recall/Precision Scores in IR over the WWW
Authors: Matthew Montebello
Venue: SIGIR
Year: 1998

The rapid growth of the World Wide Web (WWW) and the massive size of the information corpus available for access symbolizes the wealth and benefits of this medium. At the same time this immense pool of information has created an information overflow which requires users to revert to techniques and tools in order to take advantage of such a resource and enhance the effectiveness of online information access. Search engines were created to assist users to find information by employing indexing techniques and suggest appropriate alternatives to browse. These search engines have inefficiencies and are not focused enough to the needs of individual users and little has been done to ensure that the information presented is of a high recall and precision standard. ‘Recall’ measures how efficient the system is at retrieving the relevant documents from the WWW, while ‘precision’ measures the relevance of the retrieved set of documents to the users’ requirements. We present our experiences with a system we developed to optimize the recall/precision scores. We attempt to achieve this objective by employing a number of search engines and user profiling in tandem. Namely, we attempt to optimize: l recall by aggregating hits from major search engines and other previously developed retrieval systems, l precision by generating user profiles and predicting appropriate and focused information to specific users. Our system is able to easily and inexpensively accommodate future generations of web-based retrieval systems and technologies. Our contribution to the IR field is that we were able to incorporate several desirable characteristics from different techniques to optimize and personalize WWW searching. 1 Background and Motivations In recent years there has been a well-publicized explosion of information available on the Internet, and a corresponding increase in usage. The WWW’s sheer scale and its exponential growth renders the task of simply finding information, tucked away in some Web site, laborious, tedious, long-winded and time consuming. The fact that Permission to make digital/hard copy of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage, the cowriaht notice. the title of the Dublication and its date appear, and notice is gi;en that copying is by permission of ACM, Inc. To copy otherwise, to republish, to post on servers OF to redistribute to lists, requires prior specific permission and/or fee. SlGIR’98, Melbourne, Australia @ 1998 ACM l-58113-015-5 8/98 $5.00. a user’s time is valuable and that relevant information might not be accessed, imposes serious restrictions on the efficient use of the WWW and the benefits that users can expect from their interaction. Users are faced with the problem of search engines being too generalised[2, 31 and not focused enough[8, 91 to their real and specific needs. This triggered research to develop more sophisticated techniques and agent like systems that make use of the user profile to personalize the service they provide and add value to the information they presented [4, 5, 71. In Section 2 we briefly present the system we developed to reuse information generated by search engines and previously developed retrieval systems. Conceptually, it is similar to a meta-search engine, but with the major difference that it employs user profiling to specifically target documents for individual users. The system makes use of a number of machine learning techniques to extract features from documents and generate profiles. We point out how this web-based application has been designed and developed to incorporate different techniques and evolve in the eventuality that new retrieval systems or novel machine learning techniques need to be implemented in future. Finally, in Section 3 we present our conclusions and future work. 2 Architecture and Current Implementation Our goal is to achieve a high recall and high precision performance score on the information presented to the user. Recall measures how efficient the system is at retrieving the relevant documents from the WWW, while precision measures the relevance of the retrieved set of documents to the user requirements. In order to obtain a high recall execution we make use of the metasearch approach, namely, hits returned by a number of traditional search engines together with previously developed retrieval systems are blended, aggregated and utilised by our system. On the other hand, to achieve a high precision execution we employ machine learning techniques[6] to extract features from documents specific users find interesting, generate a profile and predict other documents that fit this profile. The task performed by the system is decomposed into a number of simpler tasks. Figure 1 shows the major components of the system: the WWW and the external systems at the bottom level, the underlying application software on the next level up, and the GUI at the top. All the external systems are considered to be black boxes and action is taken upon the information they output. Wrappers are used to manage the appropriate and proper handshaking between the diverse search engines and the other retrieving systems and the application layer. The system requires an administrator to manage the general needs and demands of a specific interest group of users. The administrator can initialise search terms tailored to any type of interest group and furthermore users will be able to suggest any other terms to add to the main search list. Documents relevant to the specific area of interest are retrieved and stored by the underlying application within the main index, and when a user logs-in he/she is able to benefit from the systems’ high recall fidelity. Having analysed the documents, individual users can bookmark and highlight specific items as interesting and appealing. These will be saved inside their personal database index. At this stage the underlying application plays another important role in attaining precise targeting of documents to individual users by generating a profile from the personal database index and predicting other documents from within the main index. Users can decide to add the suggested documents to their personal database index or remove them completely. As new and suggested documents are entered in the personal database index the user profile becomes more focused and finely tuned, as a result of which higher precision results will be achieved.