Digital Libraries Based on Full-Text Retrieval

Because digital libraries are expensive to create and maintain, Internet analogs of public libraries—reliable, quality, community services—have only recently begun to appear. A serious obstacle to their creation is the provision of appropriate cataloguing information. Without a database of titles, authors and subjects, it is hard to offer the searching and browsing facilities normally available in physical libraries. Full-text retrieval provides a way of approximating these services without a concomitant investment of human resources. This presentation will discuss the indexing, collection and maintenance processes, and the retrieval interface, to public digital libraries. Creating a publicly-available digital library presents some interesting challenges, particularly if the library is intended for serious professional work rather than casual browsing. First, the raw material: the collection must comprise text that can be placed in the public domain. Second, the selection of material: although a huge amount of public-domain text is present on the Internet, its quality is extremely uneven and only a tiny fraction is appropriate for inclusion in a library. Third, the format of material: it is helpful if the collection resembles the traditional form of typeset pages rather than raw electronic text. Fourth, cataloguing: appropriate bibliographic information in a usable format may be difficult to find and onerous to provide manually. Fifth, information retrieval: a uniform, easy-to-use, publicly-accessible interface is necessary. There are two essentially opposite approaches to solving these problems. One is to arrange for authors, or their representatives, to contribute information to the library, so that a traditional author/title/keyword catalogue can be built through which readers access the collection. Because authors have a vested interest in seeing their work fully represented and properly catalogued, they will presumably be prepared to invest time and effort to ensure that full information is supplied. An extreme example of this philosophy is the HYPATIA project (http://hypatia.dcs.qmw.ac.uk) which is creating a database of people, research interests, bibliography entries, and papers to which people offer information by subscribing individually and by academic department. The second approach, pursued in this paper, is to gather material from public repositories without any active participation by authors or their institutions. This has the potential to create a vastly larger collection, but requires a new mechanism for searching the library because conventional bibliographic information is not available. An extreme example of this philosophy is the well-known ALTAVISTA search engine which provides a full-text index to a vast universe of indiscriminately-garnered Web pages (http://altavista.digital.com). Although the second approach provides the basis for the search engines that are rapidly becoming our primary access point to the Web, few digital library projects have adopted this philosophy. One that has is the New Zealand Digital Libraries project. This is unique in several respects. First, it provides a full-text index of the entire contents of each document, whereas other digital libraries index on user-supplied document descriptions, abstracts, or other document surrogates. Second, it makes a minimum of assumptions about conventions adopted by document repositories. Other schemes rely for their information on the index file that is present by convention in most directories of technical reports, or on other information provided explicitly for indexing purposes. Third, the work directly addresses the problem of building the library in a geographically remote location with high Internet costs—an environment in which the benefits of networked library technology are especially striking. Finally, the scheme is extremely economical in local disk resources. Like many other digital library projects, we have chosen the domain of computer science technical reports to exemplify our approach. Much high-quality information already exists in digital form and is freely accessible on the Internet, and because the time value of information is high, computer science already relies more than most on pre-publication in the form of technical reports. Name Domain Reference Sites Documents Indexed on e-print archive Physics [Ginsparg 1994] several million user-supplied title, author, abstract NTRS NASA [Nelson et al. 1994] 3 000 000 abstracts HARVEST CS [Bowman et al. 1994] 380 37 000 limited info from several file types UCSTRI CS [Van Heyningen 1994] 185 14260 file of abstracts at ftp site DIENST CS [Davis & Lagoze 1994] 15 – bibliographic information provided WATERS CS [Maly et al. 1994] 15 2 000 citation and possibly abstract NCSTRL CS www.ncstrl. org 55 – bibliographic information provided Karlsruhe CS bib liinwww.ira. uka.de/bibliography 560 000 full text of bibliographic entry Table 1: Technical report indexes on the Internet This paper begins by briefly surveying both Internet technical report collections and Web search engines. These exemplify the two different philosophies to collection-building outlined above. We then go on to describe our prototype digital library. We describe the indexing and retrieval mechanism, including the full-text index, and summarize the types of search provided, the indexes that are needed to support them, and the query interface through which they are made accessible. We go on to discuss the process of building the collection, and the crucial issue of what information is stored centrally and what can remain distributed. We then briefly summarize how the information is gathered, consider the factors that limit the size of the library, and review future directions for networked digital libraries. Internet libraries for technical reports Several technical report searching and indexing systems have been developed in the past; some are summarized in [Tab. 1]. The physics e-print archive, operating since 1991, has already supplanted journals and pre-print mailings as the primary information dissemination point for several areas of physics. Documents are submitted by e-mail in TeX format, along with bibliographic information which is used for indexing. NASA has a large index to publications since 1962, but most references are to documents that are only available in physical form. Computer Science is well endowed with indexes. The HARVEST system contains a large number of documents, many of which are Web pages rather than technical reports. The documents are indexed on limited information extracted from the original files. UCSTRI allows searching of files of abstracts, which often appear in technical report FTP archives. While this information is limited in scope, the collection includes 185 sites and 14,000 documents. The DIENST and WATERS systems have been superseded by NCSTRL (see below). DIENST is a distributed library architecture, developed in a project involving several universities, that originally accommodated scanned images and indexed on text produced by OCR. WATERS included technical reports from several universities and provides bibliographic searches based on information provided by participating departments The Karlsruhe bibliography service indexes a large collection of computer science reference lists. Recently, the NCSTRL (for “Networked Computer Science Technical Reports Library”) project has arisen as a development of the DIENST and WATERS systems. NCSTRL is a collection of computer science technical reports made available for non-commercial and educational use. There are two levels at which organizations can participate. “Standard” participants run indexing, searching, and user interface software locally, whereas “Lite” participants maintain an ordinary FTP site of technical reports and supply bibliographic information to a special server. Lite connection is easiest but Standard connection allows a site to maintain its own search engines, to provide access to documents in multiple formats, and to provide customized user interfaces—thumbnail browsing, page zooming, page selection, support for logical document structure, etc. NCSTRL currently (July 1996) has 55 participants, of which nearly two-thirds are Lite. Standard NCSTRL sites use the DIENST software, an open, distributed, implementation of digital library servers. DIENST provides three basic services at each site: a repository, indexes and a search engine, and a user interface. The repository can hold documents in different formats (PostScript, TIFF, GIF, HTML) and can handle the same document in multiple formats. Sites can use home-grown software for any of these components. For example, they can supply their own browsers (and some do). DIENST makes a distributed library appear as a single collection at the top level, even though the repositories, search engines, and browsing interfaces are distributed.