cometadata/datacite-titles-descriptions-related-identifiers huggingface dataset:
DOI: 10.35111/njz5-b969
title: Arabic Broadcast News Speech

description: <h3>Introduction</h3><br>
<p>Arabic Broadcast News Speech was developed by the Linguistic Data Consortium (LDC) and contains eight audio files totalling 10 hours of Arabic broadcast speech. The data was recorded by LDC from Voice of America (VOA) satellite radio news broadcasts in Arabic transmitted between June 2000 and January 2001. The corresponding transcripts for these speech files are available in <a href="../../../LDC2006T20">Arabic Broadcast News Transcripts (LDC2006T20)</a>.</p><br>
<p>This work was undertaken in the Networking Data Centers (NetDC) project (MLIS-5017, NSF IIS-9982201) in conjunction with the <a href="http://www.elra.info/en/">European Language Resources Association</a> (ELRA). ELRA collected 22.5 hours of Arabic broadcast data from Radio Orient (France) that is available in <a href="http://catalog.elra.info/product_info.php?products_id=13">NetDC Arabic Broadcast News Speech Corpus (ELRA-S0157)</a>. The goal of the NetDC project was to improve the infrastructure for language resources by designing and implementing new modes of cooperation between LDC and ELRA.</p><br>
<h3>Data</h3><br>
<p>The recordings were captured from a dedicated satellite receiver and stored as 16-bit PCM, 16-kHz, single-channel, in NIST SPHERE format. The duration of each recording is either 60 minutes or 120 minutes, depending on the VOA broadcast schedule. The date (YYYYMMDD), start-time, and end-time (HHMM EST) for each recording are indicated in the file names. The sample data are not compressed.</p><br>
<h3>Samples</h3><br>
<p>For an example of the speech in this corpus, please listen to this <a href="desc/addenda/LDC2006S46.wav" rel="nofollow">sample (WAV)</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2000, 2001, 2002, 2005, 2006 Trustees of the University of Pennsylvania
DOI: 10.35111/j0yk-ph77
title: Korean Propbank

description: <h3>Introduction</h3><br>
<p>Korean Propbank was developed by the Computer and Information Sciences Department at the University of Pennsylvania and is comprised of approximately 33,300 predicates annotated in 186,300 words of Korean text. The text used in Propbank comes from <a href="../../../LDC2002T26">Korean English Treebank Annotations (LDC2002T26)</a> and <a href="../../../LDC2006T09">Korean Treebank Version 2.0 (LDC2006T09)</a>. Each verb and adjective occurring in the Treebank has been treated as a semantic predicate and the surrounding text has been annotated for arguments and adjuncts of the predicate. The verbs and adjectives have also been tagged with coarse grained senses.</p><br>
<h3>Data</h3><br>
<p>This table gives a breakdown of the thousands of words and number of annotations contained in the corpus, broken down by source:</p><br>
<table style="margin-top: 30px; margin-bottom: 30px;" border="1" width="25%"><br>
<tbody><br>
<tr><br>
<td>Source</td><br>
<td>K-words</td><br>
<td>Predicates Annotated</td><br>
</tr><br>
<tr><br>
<td>Virginia Corpus</td><br>
<td>54.5</td><br>
<td>9,590</td><br>
</tr><br>
<tr><br>
<td>Newswire Corpus</td><br>
<td>131.8</td><br>
<td>23,700</td><br>
</tr><br>
<tr><br>
<td>Total</td><br>
<td>186.3</td><br>
<td>33,300</td><br>
</tr><br>
</tbody><br>
</table><br>
<p>There are two basic components to Korean Propbank:</p><br>
<ul><br>
<li><strong>The Verb Lexicon:</strong> A frames file, consisting of one or more frame sets, has been created for each predicate occurring in the Treebank. These files serve as a reference for the annotators and for users of the data. 2,749 such files have been created, totaling about ~10 MB of uncompressed data. The XML format and KSC 5,601 character set encoding are used in the frames file.</li><br>
<li><strong>The Annotation:</strong> There are two annotation files. The virginia-verbs.pb file has 9,588 annotated predicate tokens. These predicate tokens include all those occurring in 54.5 K-words of the Korean English Treebank Annotations, totaling ~791 KB of uncompressed data. The newswire-verbs.pb file has 23,707 annotated predicate tokens. These predicate tokens include all those occurring in 131.8 K-words of the Korean Treebank Version 2.0, totaling ~2,054 KB of uncompressed data.</li><br>
</ul><br>
<h3>Samples</h3><br>
<p>For an example of this corpus, please view this <a href="desc/addenda/LDC2006T03.txt">sample (TXT)</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2001-2002 CoGenTex, Inc., © 1994-2000 Korean Press Agency, © 1998-2006 Trustees of the University of Pennsylvania
DOI: 10.35111/40kx-0718
title: Levantine Arabic QT Training Data Set 5, Transcripts

description: <h3>Introduction</h3><br>
<p>Levantine Arabic QT Training Data Set 5, Transcripts, was developed by the Linguistic Data Consortuim (LDC) and contains transcripts for approximately 250 hours (1,660 calls) of Arabic Levantine telephone conversation collected between 2003 and 2005. The corresponding speech files may be found in <a href="../../../LDC2006S29">Levantine Arabic QT Training Data Set 5, Speech (LDC2006S29)</a>.</p><br>
<h3>Data</h3><br>
<p>This corpus is the combination of four former training data sets: LDC2004E21 and LDC2004E22, LDC2004E65 and LDC2004E66, <a href="../../../LDC2005S07">Arabic CTS Levantine Fisher Training Data Set 3, Speech (LDC2005S07)</a> and <a href="../../../LDC2005T03">Arabic CTS Levantine Fisher Training Data Set 3, Transcripts (LDC2005T03)</a>, and <a href="../../../LDC2005S14">Levantine Arabic QT Training Data Set 4 (Speech + Transcripts) (LDC2005S14)</a>. More than half of the speakers are Lebanese, the others are Jordanian, Palestinian, and Syrian. The table below shows the distribution of the speakers' national origin:</p><br>
<table><br>
<tbody><br>
<tr><br>
<td>Jordanian</td><br>
<td>559</td><br>
</tr><br>
<tr><br>
<td>Lebanese</td><br>
<td>1,853</td><br>
</tr><br>
<tr><br>
<td>Palestinian</td><br>
<td>355</td><br>
</tr><br>
<tr><br>
<td>Syrian</td><br>
<td>67</td><br>
</tr><br>
<tr><br>
<td>Levantine speakers of undetermined national origin</td><br>
<td>484</td><br>
</tr><br>
</tbody><br>
</table><br>
<p>The corpus package documentation includes callers' demographic information and tables indicating the topics given at the beginning of each call.</p><br>
<p>All transcription files are encoded in UTF-8.</p><br>
<h3>Samples</h3><br>
<p>Please view the following <a href="desc/addenda/LDC2006T07.txt">transcript sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2003-2006 Trustees of the University of Pennsylvania
DOI: 10.35111/02nk-p662
title: Korean Treebank Annotations Version 2.0

description: <h3>Introduction</h3><br>
<p>English-Arabic Parallel Treebank v 1.0 was developed by the Linguistic Data Consortium (LDC) and consists of 52,238 words in 224 files of individual Agence France Presse (AFP) news stories.</p><br>
<p>The AFP stories included here correspond to approximately the first 50K words of <a href="../../../LDC2005T02">Arabic Treebank: Part 1 v 3.0 (LDC2005T02)</a>. The English translation was provided by LDC, and was part-of-speech tagged and treebanked for this project.</p><br>
<h3>Data</h3><br>
<p>The guidelines followed for both part-of-speech and treebank annotation are essentially Penn Treebank II style, with two notable differences:</p><br>
<ol><br>
<li>POS: tokenization of hyphenated items ("New York-based" has been replaced by "New York - based" for example), and the addition of HYPH and AFX tags necessitated by this change in tokenization</li><br>
<li>TreeBank: the addition of the node label NML for sub-NP nominal constituents (replacing NX and most NP-internal NAC)</li><br>
</ol><br>
<p>More detailed addenda to the Penn Treebank II guidelines and a mapping from the original Arabic Treebank filenames to the current filenames used in this release can be found in the associated documentation.</p><br>
<h3>Samples</h3><br>
<p>For an example of the data in this corpus, please review this text <a href="desc/addenda/LDC2006T10.html" rel="nofollow">sample (TXT)</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
© 2001-2002 CoGenTex, Inc., © 2000 Korean Press Agency, © 2000-2005, 2006 Trustees of the University of Pennsylvania
DOI: 10.35111/7dp6-6613
title: English-Arabic Treebank v 1.0

description: <h3>Introduction</h3><br>
<p>English-Arabic Parallel Treebank v 1.0 was developed by the Linguistic Data Consortium (LDC) and consists of 52,238 words in 224 files of individual Agence France Presse (AFP) news stories.</p><br>
<p>The AFP stories included here correspond to approximately the first 50K words of <a href="../../../LDC2005T02">Arabic Treebank: Part 1 v 3.0 (LDC2005T02)</a>. The English translation was provided by LDC, and was part-of-speech tagged and treebanked for this project.</p><br>
<h3>Data</h3><br>
<p>The guidelines followed for both part-of-speech and treebank annotation are essentially Penn Treebank II style, with two notable differences:</p><br>
<ol><br>
<li>POS: tokenization of hyphenated items ("New York-based" has been replaced by "New York - based" for example), and the addition of HYPH and AFX tags necessitated by this change in tokenization</li><br>
<li>TreeBank: the addition of the node label NML for sub-NP nominal constituents (replacing NX and most NP-internal NAC)</li><br>
</ol><br>
<p>More detailed addenda to the Penn Treebank II guidelines and a mapping from the original Arabic Treebank filenames to the current filenames used in this release can be found in the associated documentation.</p><br>
<h3>Samples</h3><br>
<p>For an example of the data in this corpus, please review this text <a href="desc/addenda/LDC2006T10.html" rel="nofollow">sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2000 Agence France Presse, © 2006 Trustees of the University of Pennsylvania
DOI: 10.35111/4kh9-er55
title: Spanish Gigaword First Edition

description: <h3>Introduction</h3><br>
<p>Spanish Gigaword First Edition is a comprehensive archive of newswire text data that has been acquired over several years by the Linguistic Data Consortium (LDC) and contains over 750 million tokens spanning approximately 2.7 million documents. Although this is the first edition of the Spanish Gigaword Corpus, some of the data included here has been released previously in other LDC corpora.</p><br>
<p>The three distinct international sources of Spanish newswire in this edition, and the time spans of collection covered for each, are as follows:</p><br>
<ul><br>
<li>Agence France-Presse, Spanish Service (afp_spa) May 1994 - Dec 2005</li><br>
<li>Associated Press Worldstream, Spanish (apw_spa) Nov 1993 - Dec 2005</li><br>
<li>Xinhua News Agency, Spanish Service (xin_spa) Sep 2001 - Dec 2005</li><br>
</ul><br>
<h3>Data</h3><br>
<p>The overall totals for each source are summarized below. The "K-wrds" figures are simply the number in thousands of whitespace-separated tokens of all types after all SGML tags are eliminated.</p><br>
<table><br>
<tbody><br>
<tr><br>
<td>Source</td><br>
<td>K-wrds</td><br>
<td>#DOCs</td><br>
</tr><br>
<tr><br>
<td>AFP_SPA</td><br>
<td>393354</td><br>
<td>1382679</td><br>
</tr><br>
<tr><br>
<td>APW_SPA</td><br>
<td>263225</td><br>
<td>886998</td><br>
</tr><br>
<tr><br>
<td>XIN_SPA</td><br>
<td>94459</td><br>
<td>388561</td><br>
</tr><br>
<tr><br>
<td>TOTAL</td><br>
<td>751038</td><br>
<td>2658238</td><br>
</tr><br>
</tbody><br>
</table><br>
<p>Most of the text data (all of AFP_SPA, most of APW_SPA) were received at LDC via dedicated, 24-hour/day electronic feeds (leased phone lines in the case of APW_SPA, a local satellite dish for AFP_SPA). These 24-hour transmission services were all susceptible to "line noise" (occasional corruption of text content), as well as service outages both at the data source and at our receiving computers. Usually, the various disruptions of a newswire data stream would leave tell-tale evidence in the form of byte values falling outside the range of printable ASCII characters, or recognizable patterns of anomalous ASCII strings.</p><br>
<p>All the XIN_SPA data, and the portion of APW_SPA data beginning with 200406, were received as bulk electronic text archives via internet retrieval. As such, they were not susceptible to modem line-noise or related disruptions, though this does not guarantee that the source data are free of mishaps. More detailed information can be found in the included documentation.</p><br>
<h3>Samples</h3><br>
<p>For an example of the data in this publicaiton, please examine this <a href="desc/addenda/LDC2006T12.txt" rel="nofollow">sample file</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 1994-2005 Agence France Presse, © 1993-2005 The Associated Press, © 2001-2005 Xinhua News Agency, © 2006 Trustees of the University of Pennsylvania
DOI: 10.35111/h4ag-tj85
title: Korean Broadcast News Transcripts

description: <h3>Introduction</h3><br>
<p>Korean Broadcast News Transcripts was developed by the Linguistic Data Consortium (LDC) and consists of transcripts for 13 hours of Voice of America satellite radio news broadcasts in Korean. The broadcasts were recorded by the LDC at transmission time during a two week period between January 21, 2000 and February 7, 2000.</p><br>
<h3>Data</h3><br>
<p>This corpus contains 18 broadcast transcription text files. Ten of the broadcasts are 30 minutes long, and the other eight broadcasts are 60 minutes long. The file names indicate the date (YYYYMMDD) and the begin and end times (HHMM EST) of the original transmission.</p><br>
<p>The character encoding is Unicode UTF-8, and the file contents are structured using SGML. The markup strategy used here was defined by the National Institute of Standards and Technology (NIST) specifically for use in transcripts of broadcast news speech. The "docs" directory provides a working DTD file, a complete description (in the form of a PostScript file) of the document structure, tags and attributes, and a simple text file listing the 18 data file names in the corpus.</p><br>
<p>The transcripts have been manually time aligned at the phrasal level and annotated to identify boundaries between news stories and speaker turns; speaker names and gender are given where identifiable. These annotations are all provided via the SGML tags and their attributes.</p><br>
<p>A strong effort has been made to identify all unique speakers across the transcripts. However, there may be cases where an individual speaker has not been recognized and has been given a unique, anonymous identification.</p><br>
<p>The corresponding speech files for these transcripts are available in <a href="../../../LDC2006S42">Korean Broadcast News Speech (LDC2006S42)</a>.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2006T14.sgm">transcript sample (SGML)</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2000, 2006 Trustees of the University of Pennsylvania
DOI: 10.35111/f6ha-dk13
title: Iraqi Arabic Conversational Telephone Speech, Transcripts

description: <h3>Introduction</h3><br>
<p>Iraqi Arabic Conversational Telephone Speech, Transcripts was developed by Appen Pty Ltd, Sydney, Australia and contains transcripts for roughly 3000 mins of speech from Iraqi Arabic speakers taking part in spontaneous telephone conversations in Colloquial Iraqi Arabic.</p><br>
<p>This corpus was collected and transcribed in 2003 and 2004 by Appen Pty Ltd, Sydney, Australia. The corresponding speech files for these transcripts are available in <a href="../../../LDC2006S45">Iraqi Arabic Conversational Telephone Speech (LDC2006S45)</a>.</p><br>
<h3>Data</h3><br>
<p>A total of 478 conversation sides from 474 unique speakers are provided, and most of these transcripts contain both sides of a conversation (202 transcripts with both sides and 74 with just one side). The average duration per call is about six minutes, so each call side contains about three minutes of speech, on average.</p><br>
<p>The transcripts directory contains three subdirectories:</p><br>
<p style="padding-left: 30px;">devtest -- 12 two-sided transcripts<br />train1c -- 74 single-sided transcripts<br />train2c -- 190 two-sided transcripts</p><br>
<p>The single-sided transcripts represent just one side of a normal conversation with the speaker info (A or B) included in the filename, formulated like such: AR-[DATE]-[TIME]_[LINE]-[A or B].txt</p><br>
<p>The "devtest" set represents a relatively balanced (representative) sample drawn from the total pool of collected calls, based on a test-set selection process applied by the National Institute of Standards and Technology (NIST) and based on demographic, phone and audit information as provided by Appen.</p><br>
<h3>Samples</h3><br>
<p>For an example of the transcripts in this release, please review this <a href="desc/addenda/LDC2006T16.txt" rel="nofollow">sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2006 Trustees of the University of Pennsylvania
DOI: 10.35111/n8na-xw24
title: French Gigaword First Edition

description: <h3>Introduction</h3><br>
<p>French Gigaword First Edition is a comprehensive archive of newswire text data that has been acquired over several years by the Linguistic Data Consortium (LDC) and consists of over 650 million tokens spanning approximately 2.4 million documents.</p><br>
<p>The two distinct international sources of French newswire in this edition, and the time spans of collection covered for each, are as follows:</p><br>
<ul><br>
<li>Agence France-Presse (afp_fre) May 1994 - July 2006</li><br>
<li>Associated Press French Service (apw_fre) Nov 1994 - July 2006</li><br>
</ul><br>
<h3>Data</h3><br>
<p>The overall totals for each source are summarized below. The "K-wrds" figures are simply the number in thousands of whitespace-separated tokens of all types after all SGML tags are eliminated.</p><br>
<table><br>
<tbody><br>
<tr><br>
<td>Source</td><br>
<td>K-wrds</td><br>
<td>#DOCs</td><br>
</tr><br>
<tr><br>
<td>AFP_FRE</td><br>
<td>482904</td><br>
<td>1797139</td><br>
</tr><br>
<tr><br>
<td>APW_FRE</td><br>
<td>167405</td><br>
<td>622740</td><br>
</tr><br>
<tr><br>
<td>TOTAL</td><br>
<td>650309</td><br>
<td>2419879</td><br>
</tr><br>
</tbody><br>
</table><br>
<p>All text data are presented in SGML form, using a very simple, minimal markup structure; all text consists of printable ASCII, whitespace, and printable code points in the "Latin1 Supplement" character table, as defined by the Unicode Standard (ISO 10646) for the "accented" characters used in French. The Supplement/accented characters are presented in UTF-8 encoding.</p><br>
<p>Most of the text data (all of AFP_FRE, most of APW_FRE) were received at LDC via dedicated, 24-hour/day electronic feeds (leased phone lines in the case of APW_FRE, a local satellite dish for AFP_FRE). These 24-hour transmission services were all susceptible to "line noise" (occasional corruption of text content), as well as service outages both at the data source and at our receiving computers. Usually, the various disruptions of a newswire data stream would leave tell-tale evidence in the form of byte values falling outside the range of printable ASCII characters, or recognizable patterns of anomalous ASCII strings.</p><br>
<p>The portion of APW_FRE data beginning with 200406 was received as bulk electronic text archives via internet retrieval. As such, they were not susceptible to modem line-noise or related disruptions, though this does not guarantee that the source data are free of mishaps.</p><br>
<p>More detailed information can be found in the 0readme.txt file in the associated documentation.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2006T17.txt">text sample (TXT)</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 1994-2006 Agence France-Presse, © 1994-2006 The Associated Press, © 2006 Trustees of the University of Pennsylvania
DOI: 10.35111/azvq-yv94
title: TDT5 Multilingual Text

description: <h3>Introduction</h3><br>
<p>The TDT5 Multilingual Text corpus was developed by the Linguistic Data Consortium (LDC) and contains the complete set of English, Arabic, and Chinese newswire text, totalling 407,503 stories, used in the 2004 Topic Detection and Tracking technology evaluations.</p><br>
<p>The TDT5 corpora were created by LDC with support from the DARPA TIDES (Translingual Information Detection, Extraction and Summarization) Program. Topic Detection and Tracking (TDT) refers to automatic techniques for finding topically related material in streams of data such as newswire and broadcast news. The topic relevance annotations corresponding to this publication can be found in <a href="http://catalog.ldc.upenn.edu/LDC2006T19" rel="nofollow">TDT5 Topics and Annotations (LDC2006T19)</a>.</p><br>
<p>There were four TDT tasks defined for the 2004 evaluation: the tracking of known topics, the detection of unknown topics, the detection of initial stories on unknown topics, and the detection of pairs of stories on the same topic (links). Of these four tasks, the topic tracking task and the link detection task are considered to be "primary." Previous TDT evaluations also included a story segmentation task. This task applied only to broadcast news. Since TDT5 does not include broadcast news, there is no story segmentation task in the 2004 TDT Evaluation.</p><br>
<h3>Data</h3><br>
<p>The TDT5 corpus spans collections from April-September 2003 of English, Chinese, and Arabic news text. A total of 15 distinct news "sources" are included (where a "source" comprises data from a given news agency in a particular language; when an agency publishes in multiple languages, each language is considered a different "source"). In contrast to earlier TDT corpora, TDT5 has no broadcast/audio content, only printed news from wire and web sources.</p><br>
<p>TDT5 is a subset of the previous LDC release <a href="../../../LDC2005T28" rel="nofollow">HARD 2004 Text (LDC2005T28)</a>. In the HARD 2004 Text release, data is organized into one file per day per source. For the current TDT5 corpus, roughly half of the daily files are partitioned into chunks of some maximum size (around 30 K-words, on average), in order to support the "look-ahead" condition in the TDT5 evaluation plan and to keep sample file size relatively consistent.</p><br>
<h3>Samples</h3><br>
<p>The images below are samples of the text data contained in this corpus.</p><br>
<ul><br>
<li><a href="desc/addenda/LDC2006T18_arb.txt">Arabic (TXT)</a></li><br>
<li><a href="desc/addenda/LDC2006T18_eng.txt">English (TXT)</a></li><br>
<li><a href="desc/addenda/LDC2006T18_man.txt">Chinese (TXT)</a></li><br>
</ul><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2003 Agence France Presse, An-Nahar, The Associated Press, The Central News Agency, Los Angeles Times-Washington Post News Service, Inc., The New York Times, SPH AsiaOne Ltd, Ummah Press Service, Xinhua News Agency, © 2003, 2004, 2006 Trustees of the University of Pennsylvania