cometadata/datacite-titles-descriptions-related-identifiers huggingface dataset:
DOI: 10.35111/9130-0z09
title: TDT4 Multilingual Text and Annotations

description: <h3>Introduction</h3><br>
<p>TDT4 Multilingual Text and Annotations was developed by the Linguistic Data Consortium (LDC) with support from the DARPA TIDES (Translingual Information Detection, Extraction and Summarization) Program. This release contains the complete set of English, Arabic and Chinese news text (broadcast news transcripts and newswire data) used in the 2002 and 2003 Topic Detection and Tracking technology evaluations, along with topic annotations created for those evaluations. The audio corresponding to the broadcast news transcripts in this release can be found in TDT4 Multilingual Broadcast News Speech Corpus (<a href="http://catalog.ldc.upenn.edu/LDC2005S11" rel="nofollow">LDC2005S11</a>).</p><br>
<p>Topic Detection and Tracking (TDT) refers to automatic techniques for finding topically related material in streams of data such as newswire and broadcast news. Evaluation tasks in 2002 and 2003 included the segmentation of a news source into stories, the tracking of known topics, the detection of unknown topics, the detection of initial stories on unknown topics, and the detection of pairs of stories on the same topic.</p><br>
<h3>Samples</h3><br>
<p>To see an example of this corpus, please examine this <a href="desc/addenda/LDC2005T16.txt" rel="nofollow">sample</a>. This sample is an English translation from an Arabic news broadcast. The translation is the product of the IBM Arabic to English translation engine.</p><br>
<p>The World is a co-production of Public Radio International and the British Broadcasting Corporation and is produced at WGBH Boston.</p></br>
Portions © 2000-2001: Xinhua News Agency, Agence France Presse, New York Times, The Associated Press, SPH AsiaOne Ltd, An Nahar, Al Hayat, Nile TV, Public Radio International, Cable News Network, LP, LLP, American Broadcasting Company, National Broadcasting Company, Inc., China National Radio, China Television System, China Central TV, China Broadcasting System, © 2002, 2003, 2005 Trustees of the University of Pennsylvania <br><br>The World is a co-production of Public Radio International and the British Broadcasting Corporation and is produced at WGBH Boston.
DOI: 10.35111/0yyq-cj29
title: Fisher English Training Part 2, Transcripts

description: <h3>Introduction</h3><br>
<p>Fisher English Training Part 2 Transcripts represents the second half of a collection of conversational telephone speech (CTS) that was created at the LDC during 2003. It consists of time-aligned transcripts for the speech contained in <a href="http://catalog.ldc.upenn.edu/LDC2005S13" rel="nofollow">Fisher English Training Part 2, Speech (LDC2005S13)</a>.</p><br>
<p>The Fisher telephone conversation collection protocol was created at the LDC to address a critical need of developers trying to build robust automatic speech recognition (ASR) systems.<br />Previous collection protocols, such as CALLFRIEND and Switchboard-II and the resulting corpora, have been adapted for ASR research but were in fact developed for language and speaker identification respectively. Although the CALLHOME protocol and corpora were developed to support ASR technology, they feature small numbers of speakers making telephone calls of relatively long duration with narrow vocabulary across the collection. CALLHOME conversations are challengingly natural and intimate. Under the Fisher protocol, a large number of participants each calls an other participant, whom they typically do not know, for a short short period of time to discuss the assigned topics. This maximizes inter-speaker variation and vocabulary breath while also increasing formality.</p><br>
<p>Previous protocols such as CALLHOME, CALLFRIEND and Switchboard relied upon participant activity to drive the collection. Fisher is unique in being platform driven rather than participant driven. Participants who wish to initiate a call may do so however the collection platform initiates the majority of calls. Participants need only answer their phones at the times they specified when registering for the study.</p><br>
<p>To encourage a broad range of vocabulary, Fisher participants are asked to speak on an assigned topic which is selected at random from a list, which changes every 24 hours and which is assigned to all subjects paired on that day. Some topics are inherited or refined from previous Switchboard studies while others were developed specifically for the Fisher protocol.</p><br>
<h3>Data</h3><br>
<p>The first half of the collection (Fisher English Training Speech,Part 1) was released by the LDC in 2004 (LDC2004S13 for speech data,LDC2004T19 for transcripts). Taken as a whole, the two parts comprise11,699 recorded telephone conversations.</p><br>
<p>The individual audio files are presented in NIST SPHERE format, and contain two-channel mu-law sample data shorten compression has been applied to all files.</p><br>
<p>Data collection and transcription were sponsored by DARPA and the U.S. Department of Defense, as part of the EARS project for research and development in automatic speech recognition.</p><br>
<h3>Samples</h3><br>
<p>To see an example of this corpus, please examine this <a href="desc/addenda/LDC2005T19.txt" rel="nofollow">sample</a>.</p></br>
Portions © 2003-2005 Trustees of the University of Pennsylvania
DOI: 10.35111/qwyc-cw15
title: RT-04 MDE Training Data Text/Annotations

description: <h3>Introduction</h3> <p> This corpus was created by Linguistic Data Consortium to provide training data for the RT-04 Fall Metadata Extraction (MDE) Evaluation, part of the DARPA EARS (Efficient, Affordable, Reusable Speech-to-Text) Program. This data set has been created and distributed by Linguistic Data Consortium. This data was previously released to the EARS MDE community as LDC2004E31. </p> <p> The goal of MDE is to enable technology that can take raw Speech-to-Text output and refine it into forms that are of more use to humans and to downstream automatic processes. In simple terms, this means the creation of automatic transcripts that are maximally readable. This readability might be achieved in a number of ways: flagging non-content words like filled pauses and discourse markers for optional removal; marking sections of disfluent speech; and creating boundaries between natural breakpoints in the flow of speech so that each sentence or other meaningful unit of speech might be presented on a separate line within the resulting transcript. Natural capitalization, punctuation and standardized spelling, plus sensible conventions for representing speaker turns and identity are further elements in the readable transcript. LDC has defined a SimpleMDE annotation task specification and has annotated English telephone and broadcast news data to provide training data for MDE. </p> <p> In this release, some original annotations contained in LDC2004E31 have been re-mapped to new MDE elements to support better annotation consistency. In particular, the mapping affects Discourse Responses (DR), Discourse Markers (DM) and Backchannel SUs (BC). A description of the original mapping proposed by ICSI appears in 3) below, with complete documentation of the mapping rules contained in the docs/drmap-discussion directory. The scripts used to apply the mapping can be found in the docs/scripts/drmap directory. </p> <h3>Samples</h3> <p> For an example of this corpus, please review the following xml samples. </p><ul> <li> <a href="desc/addenda/LDC2005T24_bnews.xml" rel="nofollow">Broadcast News Annotations</a> </li> <li> <a href="desc/addenda/LDC2005T24_cts.xml" rel="nofollow">Telephone Conversation Annotation</a> </li> </ul> <p> The World is a co-production of Public Radio International and the British Broadcasting Corporation and is produced at WGBH Boston.</p></br>
Portions © 2004 Trustees of the University of Pennsylvania,© 2003 American Broadcasting Company,© 2003 National Broadcasting Company,© 2003 Public Radio International,© 2003 Cable News Network, Inc. All Rights Reserved,© 2003 National Cable Satellite Corporation <br><br> The World is a co-production of Public Radio International and the British Broadcasting Corporation and is produced at WGBH Boston.
DOI: 10.35111/7h63-pd43
title: HARD 2004 Text

description: <h3>Introduction</h3> <p>The HARD 2004 Text Corpus was produced by Linguistic Data Consortium (LDC), catalog number LDC2005T28 and ISBN 1-58563-372-0. </p> <p> This corpus contains source data for the 2004 TREC HARD (High Accuracy Retrieval from Documents) Evaluation. HARD 2004 was a track within the NIST Text REtrieval Conference (TREC), with the objective of achieving high accuracy retrieval from documents by leveraging additional information about the searcher and/or the search context, through techniques like passage retrieval and the use of targeted interaction with the searcher. The current corpus was previously distributed to HARD Participants as LDC2004E30. The topics and annotations that correspond to this release are distributed as LDC2005T29, HARD 2004 Topics and Annotations. This corpus was created with support from the DARPA TIDES Program and LDC. </p> <h3>Data</h3> <p> The corpus comprises eight English newswire and web text sources from January-December 2003. The sources are AFE: Agence France Presse - English APE: Associated Press Newswire CNE: Central News Agency Taiwan - English LAT: Los Angeles Times/Washington Post NYT: New York Times SLN: Salon.com UME: Ummah Press - English XIE: Xinhua News Agency - English</p> Volume of data for each source appears in the table below:Source Stories Total Tokens Avg. Token/Story ---------------------------------------------------------- AFE: 226,515 71,829,978 317 APE: 237,067 93,294,584 393 CNE: 3,674 797,194 217 LAT: 18,287 12,576,721 687 NYT: 28,190 16,673,028 591 SLN: 3,321 4,710,500 1,418 UME: 2,607 782,064 299 XIE: 117,854 24,016,670 203 Total: 637,515 224,680,739<p>Files are organized by source on a daily basis. Each file contains multiple documents identified by unique document IDs, in the form "SRCyyyymmdd.nnnn", where 'nnnn' is a sequential number starting from "0001" for each source/day. In addition, each document has some or all of the following components: - Keyword (optional), surrounded by tags - Date/time (optional), surrounded by tags - Headline, surrounded by tags - Main part, surrounded by tags. </p><p> Tags are used within this part to identify paragraph boundaries.</p> <p>For more information please visit the <a href="http://www.ldc.upenn.edu/Projects/HARD" rel="nofollow">HARD Project</a> website. </p> <h3>Samples</h3> <p> For an example of the data in this corpus, please review this <a href="desc/addenda/LDC2005T28.txt" rel="nofollow">sample</a>. </p>  </br>
 Portions © 2003 Agence France Presse, © 2003 The Associated Press, © 2003 Central News Agency Taiwan, © 2003 Los Angeles Times-Washington Post News Service, Inc., © 2003 The New York Times, © 2003 Salon.com, ©2003 Ummah Press Service, © 2003 Xinhua News Agency, ©2005 Trustees of the University of Pennsylvania
DOI: 10.35111/8sx8-1q92
title: HARD 2004 Topics and Annotations

description: <h3>Introduction</h3> <p>HARD 2004 Topics and Annotations contains topics and annotations (clarification forms, responses and relevance assessments) for the 2004 TREC HARD (High Accuracy Retrieval from Documents) Evaluation. HARD 2004 was a track within the NIST Text REtrieval Conference (TREC), with the objective of achieving high accuracy retrieval from documents by leveraging additional information about the searcher and/or the search context, through techniques like passage retrieval and the use of targeted interaction with the searcher. </p> <p>The current corpus was previously distributed to HARD Participants as LDC2004E42 and LDC2005E17. The source data that corresponds to this release is distributed as LDC2005T28, HARD 2004 Text. This corpus was created with support from the DARPA TIDES Program and LDC. </p> <h3>Data</h3> <p> Three major annotation tasks are represented in this release: Topic Creation, Clarification Form Responses, and Relevance Assessment. Topics include a short title, query plus context, and a number of limiting parameters known as metadata which include targeted geographical region, target data domain or genre, and level of searcher expertise. Clarification Forms are brief HTML questionnaires system developers submitted to LDC searchers to glean additional information about information needs directly from the topic creators. Relevance assessment consisted of adjudication of pooled system responses, and included document-level judgments for all topics, and passage-level relevance judgments for a subset of topics. The release is divided into training and evaluation resources. The training set comprises twenty-one topics and 100 document-level relevance judgments per topic. The evaluation set contains fifty topics, clarification forms and responses, document-level relevance assessment for all topics and passage-level judgments for half of the topics. HARD participants received the reference data over the course of the evaluation cycle in stages: (0) training topics, (1) evaluation topic descriptions without metadata, (2) clarification form responses, (3) topic descriptions with metadata, and (4) relevance assessments.</p> <h3>Samples</h3> <p> For an example of the data in this publication, please review the following samples: </p><ul> <li> <a href="desc/addenda/LDC2005T29" rel="nofollow">Topic</a> </li> <li> <a href="desc/addenda/LDC2005T29.html" rel="nofollow">CF</a> </li> <li> <a href="desc/addenda/LDC2005T29.001" rel="nofollow">Result</a> </li> </ul> </br>
© 2005 Trustees of the University of Pennsylvania.
DOI: 10.35111/cmrf-mr79
title: Arabic Treebank: Part 4 v 1.0 (MPG Annotation)

description: <h3>Introduction</h3> <p>This file contains documentation on the Arabic Treebank: Part 4 v 1.0 (MPG Annotation), Linguistic Data Consortium (LDC) catalog number LDC2005T30 and ISBN 1-58563-343-7. </p><p>The goal of the Arabic Treebank project is to support the development of data-driven approaches to natural language processing (NLP), human language technologies, automatic content extraction (topic extraction and/or grammar extraction), cross-lingual information retrieval, information detection, and other forms of linguistic research on Modern Standard Arabic in general, the LDC was sponsored to develop an Arabic POS and Treebank of 1,000,000 words. This corpus is the fourth part of that project. In this release, we provide annotation on part of speech (POS), gloss, and word segmentation.</p> <h3>Samples</h3> <p>To view a example of this corpus, please review this <a href="desc/addenda/LDC2005T30.txt" rel="nofollow">sample POS</a> file. </p> </br>
Portions © 2004 Assabah Press Group, © 2005 Trustees of the University Pennsylvania.
DOI: 10.35111/9fx9-gz10
title: BBN Pronoun Coreference and Entity Type Corpus

description: <h3>Introduction</h3> <p>This file contains documentation on the BBN Pronoun Coreference and Entity Type Corpus, Linguistic Data Consortium (LDC) catalog number LDC2005T33 and ISBN 1-58563-362-3. </p><p> This publication supplements the one million word Penn Treebank corpus of Wall Street Journal texts (LDC95T7). The corpus contains stand-off annotation of pronoun coreference, indicated by sentence and token numbers, as well as annotation of a variety of entity and numeric types. All annotation was done by hand at BBN using proprietary annotation tools. This corpus was developed by BBN to support the ACE and AQUAINT programs </p> <p> The corpus contains two components: </p><ul> <li> <p>Pronoun coreference. Stand-off annotation of pronoun coreference of the WSJ corpus is provided in a single file. Pronouns and antecedents are indexed by sentence and token numbers.</p> </li> <li> <p>Entity types. The corpus includes annotation of 12 named entity types (Person, Facility, Organization, GPE, Location, Nationality, Product, Event, Work of Art, Law, Language, and Contact-Info), nine nominal entity types (Person, Facility, Organization, GPE, Product, Plant, Animal, Substance, Disease and Game), and seven numeric types (Date, Time, Percent, Money, Quantity, Ordinal and Cardinal). Several of these types are further divided into subtypes. Annotation for a total of 64 subtypes is provided. </p> </li> </ul><h3>Samples</h3> <p>For an example of the data in this corpus, please examing the following samples:</p> <ul> <li> <a href="desc/addenda/LDC2005T33.qa" rel="nofollow">LDC2005T33.qa</a> </li> <li> <a href="desc/addenda/LDC2005T33_pron.txt" rel="nofollow">LDC2005T33_pron.txt</a> </li> <li> <a href="desc/addenda/LDC2005T33_sent.txt" rel="nofollow">LDC2005T33_sent.txt</a> </li> </ul> </br>
Portions © 1989 Wall Street Journal, © 2002 BBNT Solutions LLC., © 2005 Trustees of the University of Pennslyvania.
DOI: 10.35111/7vr2-b249
title: CSLU: Voices

description: <p>The Voices Corpus was created by Alexander Kain for his Ph.D. dissertation work on high resolution voice transformation. The corpus contains 12 speakers reading 50 phonetically rich sentences. The recording procedure involved a "mimicking" approach which resulted in a high degree of natural time-alignment between different speakers. The acoustic wave and the concurrent laryngograph signal were recorded for one "free" and two "mimicked" renditions of each sentence. Pitch marks, calculated from the laryngograph signal, and time marks, the output of a forced-alignment algorithm, have been added to the corpus.</p> <h3>Samples</h3> For an example of the data contained in this publication, please review the following samples. <ul> <li><a href="desc/addenda/LDC2006S01.lar" rel="nofollow">Concurrent laryngograph.</a></li> <li><a href="desc/addenda/LDC2006S01.pmv" rel="nofollow">Pitch marks derived from laryngograph signal.</a></li> <li><a href="desc/addenda/LDC2006S01.txt" rel="nofollow">Transcription.</a></li> <li><a href="desc/addenda/LDC2006S01.wav" rel="nofollow">Wave file of speech.</a></li> </ul> </br>
Portions © 2002 Center for Spoken Language Understanding, Oregon Health & Science University, © 2006 Trustees of the University of Pennsylvsania
DOI: 10.35111/a49p-4407
title: CSLU:  Stories v 1.2

description: <h3>Introduction</h3> <p>This file contains documentation on CSLU: Stories V1.2, Linguistic Data Consortium (LDC) catalog number LDC2006S14 and ISBN 1-58563-366-6. </p><p> CSLU: Stories contains extemporaneous speech collected from English speakers in the CSLU Multilanguage Telephone Speech data collection. Each speaker was asked to speak on a topic of his or her choice for one minute. Those utterances are collected in the Stories corpus. </p> <h3>Data</h3> <p>The Stories corpus comprises:</p> <ol> <li> Speech files for the 702 calls</li> <li> Time-aligned word level transcriptions (and corresponding comment files) for approximately 322 stories </li> <li> Word transcriptions (not time aligned) for 702 stories</li> <li>Time-aligned phonetic labels for 702 stories</li> </ol><h3>Samples</h3> <p>For an example of the data in this corpus, please listen to this <a href="./desc/addenda/LDC2006S14.wav" rel="nofollow">audio sample</a>. </p> </br>
Portions © 2002 Center for Spoken Language Understanding Oregon Health and Science University, © 2006 Trustees of the University of Pennsylvania
DOI: 10.35111/d0x5-q082
title: CSLU:  Spelled and Spoken Words

description: <h3>Introduction</h3> <p>This file contains documentation on the Spelled and Spoken Words Corpus, Linguistic Data Consortium (LDC) catalog number LDC2006S15 and ISBN 1-58563-382-8. </p><p> The Spelled and Spoken Words corpus consists of spelled and spoken words. 3,647 callers were prompted to to say and spell their first and last names, to say what city they grew up in and what city they were calling from, and to answer two yes/no questions. In order to collect sufficient instances of each letter, 1,371 callers also recited the English alphabet with pauses between the letters. Each call was transcribed by two people, and all differences were resolved. In addition, a subset of 2,648 calls has been phonetically labeled. </p><h3>Samples</h3> <p>For an example of the data in this corpus, please listen to this <a href="desc/addenda/LDC2006S16.wav" rel="nofollow">audio sample</a>. </p> </br>
Portions © 1994-2002 Center for Spoken Language Understanding, Oregon Health & Science University, © 2006 Trustees of the University of Pennsylvania
DOI: 10.35111/kkda-b418
title: CSLU: Spoltech Brazilian Portuguese Version 1.0

description: <h3>Introduction</h3> <p>CSLU: Spoltech Brazilian Portuguese Version 1.0, Linguistic Data Consortium (LDC) catalog number LDC2006S16 and ISBN 1-58563-383-6, contains microphone speech from a variety of regions in Brazil with phonetic and orthographic transcriptions. The utterances consist of both read speech (for phonetic coverage) and responses to questions (for spontaneous speech). The corpus contains 477 speakers and 8,080 separate utterances. A total of 2,540 utterances have been transcribed at the word level (without time alignments), and 5,479 utterances have been transcribed at the phoneme level (with time alignments). Protocol design, recording and transcription were performed by the Universidade Federal do Rio Grande do Sul and the Universidade de Caxias do Sul.</p> <h3>Data</h3> <p>The data has been recorded at 44.1 kHz (mono, 16-bit) and stored in RIFF format. The recording was conducted with a direct connection from the microphone to the sound card. The sound card was SoundBlaster-compatible. For the prompted sentences, the sentence was hidden from view when recording began, so that the speaker might utter the sentence more naturally. Verification of the recording quality was performed immediately after each utterance recording; the data-collection software allowed the speaker to re-record utterances in case the recording was not of sufficient quality. The acoustic environment was not controlled, in order to allow for background conditions that would occur in application environments.</p> <h3>Samples</h3> <p>For an example of the data in this corpus, please listen to this <a href="desc/addenda/LDC2006S16.wav" rel="nofollow">audio sample</a> and examine its <a href="desc/addenda/LDC2006S16.txt" rel="nofollow">transcript</a></p>.  </br>
Portions © 1994-2002 Center for Spoken Language Understanding, Oregon Health & Science University, © 2006 Trustees of the University of Pennsylvania