Matching: Discrimination, Misinformation and Sudden Death

Introduction Sameness is a sometime thing. Libraries and other information-intensive organizations have long faced the problem of large collections of records growing incrementally. With the computerized records in a networked environment has come the recognition that duplicate records pose a threat to effective information retrieval. Yet what constitutes a match may be neither exact nor duplicate. Levels of discernment are required to permit matches on records that do not differ significantly and those which do. Initial Definitions What is matching? For this discussion, matching is the process by which additions to a large database are screened and compared with existing database records. Ideally this insures that duplicates are not added, nor erroneous replacements made of records pairs that are not really equivalent. A detailed review of the literature in this area is beyond the scope of this paper, but sources such as O'Neill, Rogers, and Oskins, (1993) and Hickey (1979) are good overviews of the problems of identifying duplicates and the implications for matching software. Which database? This project took place at OCLC Online Computer Library Center Inc., a nonprofit organization serving member libraries and related institutions throughout the world. OCLC's Extended WorldCat (XWC) is the database. It is the chief database capital of the organization, and it is 'owned' in a sense by the member libraries worldwide that use and contribute to it. At this writing it contains some 58 million records. What are the database contents? Individual records in XWC are complex bibliographic representations of physical or virtual objects--books, films, URLs, maps, slides, and much more. The records use the MARC communications format (Library of Congress, 2002). For example a record for a book might typically contain such fields for author, title, publisher, date, but many more in addition. The representation of any one object can be quite complex, containing scores of fields and subfields. Such a record may be brief, or several thousand characters long. The depth and richness of the records varies enormously. Why is matching a challenge? Does it not sound simple for computer software to compare records one to one and detect if they are identical? Perhaps so, but this is not the task of matching. Determination of what constitutes a match, under what conditions, involves a complex process of knowledge elicitation, requirements gathering, and even experimentation. Deliberating on sameness and difference in specific cases was not a trivial task. Two records describing the same intellectual creation or work (e.g. Shakespeare's Othello) can vary by physical form and other attributes. Two records describing both the same work and exactly the same form can differ from each other if the records were created under different rules of record description (cataloging). Two records intended to describe the same object can vary unintentionally if typographical or other entry errors are present in one or both. Thus sorting out significant from insignificant changes was a major task in developing requirements for the design of the software. This paper discusses the problems of matching encountered in the Metadata Capture Project at OCLC Online Computer Center. Metadata Capture is essentially the new implementation of a system for processing incoming records received from institutions, once pre-processed records by the existing infrastructure, and deciding which are already in the database. Matching decisions can result in adds or replaces of records in the database, or merely flags set to indicate library holdings. Tens of millions of records are processed through the existing record loading system every year. The task of Metadata Capture Project was to redesign the system software for interaction with the new Extended WorldCat database. Matching was one of the chief subprojects in the two year venture. …