Issues in Corpus Creation and Distribution: The Evolution of the Linguistic Data Consortium

The Linguistic Data Consortium (LDC) is a non-profit consortium of universities, companies and government research laboratories that supports education, research and technology development in language related disciplines by collecting or creating, distributing and archiving language resources including data and accompanying tools, standards and formats. LDC was founded in 1992 with a grant from the Defense Advanced Research Projects Agency (DARPA) to the University of Pennsylvania as host organization. LDC publication and distribution activities self-support from membership fees and data sales while new data creation is supported primarily by grants from DARPA and the National Science Foundation. Recent developments in the creation and use of language resources demand new roles for international data centers. Since our report at the last Language Resource and Evaluation Conference in Granada in 1998, LDC has observed growth in the demand for language resources along multiple dimensions: larger corpora with more sophisticated annotation in a wider variety of languages are used in an increasing number of language related disciplines. There is also increased demand for reuse of existing corpora. Most significantly, small research groups are taking advantage of advances in microprocessor technology, data storage and internetworking to create their own corpora. This has lead to the birth of new annotation practices whose very variety creates barriers to data sharing. This paper will describe recent LDC efforts to address emerging issues in the creation and distribution of language resources. 1. The Value of Language Resources Developing realistic models of human language that support research and technology development in language related fields requires masses of linguistic data: preferably hundreds of hours of speech, tens of millions of words of text and lexicons of a hundred-thousand words or more. Although independent researchers and small research groups now have the desktop capacity to create smallto medium-scale corpora, the collection, annotation and distribution of resources on a larger scale presents not only computational difficulties but also legal and logistical difficulties to challenge most research organizations whether they be educational, commercial or governmental. While some large corporate research groups routinely engage in mediumto large-scale corpus creation, these groups typically lack the necessary distribution infrastructure; resources created at considerable cost in those environments are seldom shared outside the immediate group. Published language resources benefit a broad spectrum of researchers, technology developers and their customers. The presence of community standard resources reduces duplication of effort, distributes production costs and removes a barrier to entry. As research communities mature, published resources are corrected, improved and further annotated. They provide a stable reference point for the comparison of different analytic approaches. Over the past two decades, the situation for language engineers has evolved from one in which concerns over intellectual property, usage agreements, publication standards and replication costs prevented resource sharing to the current state in which the value of shared resources is widely recognized. Based on the success of the DARPA “common task” methodology and the popularity of early shared databases such as the Brown text corpus and Texas Instruments’ TI 46 and TI DIGITS corpora, the LDC was created to foster the development, distribution, archiving and maintenance of language resources in electronic form. 2. The Linguistic Data Consortium The Linguistic Data Consortium was founded in 1992 with an initial grant from the Defense Advance Research Projects Agency (DARPA) and continuing funding from DARPA and the National Science Foundation (NSF). The University of Pennsylvania serves as the LDC’s host institution, providing space, infrastructure and logistical support. LDC staffers are University employees, and Penn enters into all legal arrangements on behalf of the consortium members and the research community at large. From the beginning, LDC has build fruitful links with groups in Europe, Asia and other parts of the world. In Europe, the principal partner has been ELRA but hundreds of European organizations are also LDC members. More than one hundred Asian organizations use LDC data and LDC has maintained links with the emerging GSK group in Japan. Because progress in several areas of language technology depends upon the accessibility of the consortium’s products, LDC is open to researchers around the world. Before LDC was created, an external planning committee set the membership fees that have not changed in 8 years. The membership fee for a university is roughly the cost of a new PC or attendance at an international technical meeting. The membership fee for a commercial organization is roughly the cost of a high-end workstation, certainly less than the cost to create a single small-scale corpus and an order of magnitude less than the cost of an average medium-scale corpus. As a matter of policy, no bona fide researcher is prevented from having access to LDC data by genuine inability to pay. Organizations join LDC on a yearly basis and gain perpetual rights to all corpora produced in the years in which they join. Current LDC members also have network access to LDC Online, a service that facilitates browsing and searching of indexed text, speech and lexical corpora. Since 1992, nearly 1000 organizations worldwide have used LDC data; more than 300 companies, universities and government research laboratories have joined the consortium; almost 700 others have purchased one or more corpora. Of all the organizations that use LDC data, about half are American. Europeans comprise a third of the user base with the remaining groups hailing from Asia, the Middle East, Africa and Australia. As required by the terms of LDC’s founding grant, membership fees and data sales provide funding to support the consortium’s ongoing activities in the publication, documentation, maintenance and distribution of databases as well as the negotiation of necessary legal arrangements plus a small amount of new database creation. Often, databases created elsewhere must be extensively transformed to make them suitable for electronic publication; this work is also carried out at the LDC with internal financing. Responding to demands from its constituent research communities, LDC has expanded its role from that of a specialized data publisher to include data collection, corpus creation, and research on the use and structure of language resources. LDC staff has grown accordingly. Twenty regular staffers manage the research, technical, collection, annotation, publication and customer service functions of LDC’s Philadelphia office. LDC also maintains a part-time workforce that varies from 10 to 35 staffers depending upon project workload. 3. LDC Data Publication The primary function of the LDC is the publication and archiving of data resources. LDC publishes most corpora on digital media, currently CD-ROM. The sheer volume of some corpora begs for distribution on a denser medium. LDC expects to publish its data on DVD or some future innovation once its market penetration is sufficient to be cost effective. Every corpus, unless prevented by intellectual property agreements, becomes available for network access via LDC Online as well. 3.1. Distribution on Digital Media Each year LDC publishes between 15 and 25 corpora. About half come from outside organizations that have collected and annotated data on their own but asked LDC to assist with the final formatting, intellectual property arrangements and distribution. The other half are corpora created by or with the help of LDC. The latter typically support government-sponsored technology evaluation projects. About two-thirds of LDC’s publications are speech corpora, the remaining third are text corpora and lexicons. At the time of writing, LDC had published 164 corpora including 106 speech corpora, 48 text corpora and 10 lexicons. Some of the most recent include: Topic Detection and Tracking Corpus – richly annotated broadcast news and newswire in English and Mandarin described further below Treebank 3 the latest update of the landmark hand-parsed corpus of written and conversational English Corpus of Spoken American English collected by the University of California, Santa Barbara Center for the Study of Discourse (John W. Du Bois, Director) and representing the American Component of the International Corpus of English (Charles W. Meyer, Director). BLLIP 1987-89 WSJ Corpus – a Treebank-style parsing of the three-year, 30 million word, Wall Street Journal archive from the ACL/DCI corpus developed by Eugene Charniak and his group at Brown University Speech Under Simulated and Actual Stress (SUSAS) created by the Robust Speech Processing Laboratory at Duke University (Professor John H. L. Hansen, Director) Taiwanese Putonghua 40 transcribed monologues and dialogues in Taiwanese accented Putonghua gathered by San Duanmu at the University of Michigan American English Spoken Dictionary – containing recorded pronunciations of 50,000 of the most common English words JURIS the database of the Justice Department Retrieval and Inquiry System containing almost 700,000 legal documents from the 1700's through the early 1990's covering Administrative, Case, Statutory and Tax Law, plus Executive Orders, Regulations, and International Agreements, etc. Voicemail Corpus – over 1800 messages contributed by IBM volunteers and collected and Figure 1: Organizations that use LDC by country. Marks indicate country but not exact location. transcribed by M. Padmanabhan, G. Ramaswamy, B. Ramabhadran, P. S. Gopalakrishnan and C. Dunn at IBM. Broadcast News audio and transcripts in English, Mandarin and Spanish News text corpora in English,