cometadata/datacite-titles-descriptions-related-identifiers huggingface dataset:
DOI: 10.35111/mesn-fv79
title: Web 1T 5-gram, 10 European Languages Version 1

description: <h3>Introduction</h3><br>
<p>Web 1T 5-gram, 10 European Languages Version 1 was created by Google, Inc. It consists of word n-grams and their observed frequency counts for ten European languages: Czech, Dutch, French, German, Italian, Polish, Portuguese, Romanian, Spanish and Swedish. The length of the n-grams ranges from unigrams (single words) to five-grams. The n-gram counts were generated from approximately one hundred billion word tokens of text for each language, or approximately one trillion total tokens.</p><br>
<p>The n-grams were extracted from publicly-accessible web pages from October 2008 to December 2008. This data set contains only n-grams that appeared at least 40 times in the processed sentences. Less frequent n-grams were discarded. While the aim was to identify and collect pages from the specific target languages only, it is likely that some text from other languages may be in the final data. This dataset will be useful for statistical language modeling, including machine translation, speech recognition and other uses.</p><br>
<h3>Data</h3><br>
<p>The input encoding of documents was automatically detected, and all text was converted to UTF8.</p><br>
<p>The following table contains statistics for the entire release.</p><br>
<p>File sizes (entire corpus): approximately 27.9 GB compressed (bzip2) text files</p><br>
<table><br>
<tbody><br>
<tr><br>
<td>Total number of tokens:</td><br>
<td>1,306,807,412,486</td><br>
</tr><br>
<tr><br>
<td>Total number of sentences:</td><br>
<td>150,727,365,731</td><br>
</tr><br>
<tr><br>
<td>Total number of unigrams:</td><br>
<td>95,998,281</td><br>
</tr><br>
<tr><br>
<td>Total number of bigrams:</td><br>
<td>646,439,858</td><br>
</tr><br>
<tr><br>
<td>Total number of trigrams:</td><br>
<td>1,312,972,925</td><br>
</tr><br>
<tr><br>
<td>Total number of fourgrams:</td><br>
<td>1,396,154,236</td><br>
</tr><br>
<tr><br>
<td>Total number of fivegrams:</td><br>
<td>1,149,361,413</td><br>
</tr><br>
<tr><br>
<td>Total number of n-grams:</td><br>
<td>4,600,926,713</td><br>
</tr><br>
</tbody><br>
</table><br>
<h3>Samples</h3><br>
<p>For an example of the data in this corpus please examine this <a href="desc/addenda/LDC2009T25.txt" rel="nofollow">sample file</a>.</p></br>
Portions © 2009 Google Inc., © 2009 Trustees of the University of Pennsylvania
DOI: 10.35111/abt0-qy36
title: Chinese Gigaword Fourth Edition

description: <h3>Introduction</h3> <p>Chinese Gigaword Fourth Edition, Linguistic Data Consortium (LDC) catalog number LDC2009T27 and isbn 1-58563-527-8, is a comprehensive archive of newswire text data that has been acquired over several years by the LDC. This edition includes all of the contents in <a href="http://catalog.ldc.upenn.edu/LDC2007T38" rel="nofollow"> Chinese Gigaword Third Edition (LDC2007T38)</a> as well as newly collected data. In addition, four entirely new sources have been added in the fourth edition, Central News Service, Guangming Daily, Peoples Liberation Army Daily, and Peoples Daily. </p> <p>The eight distinct international sources of Chinese newswire included in this edition are the following: </p><ul> <li>Agence France Presse (afp_cmn) </li> <li>Central News Agency, Taiwan (cna_cmn) </li> <li>Central News Service (cns_cmn) </li> <li>Guangming Daily (gmw_cmn) </li> <li>Peoples Daily (pda_cmn) </li> <li>Peoples Liberation Army Daily (pla_cmn) </li> <li>Xinhua News Agency (xin_cmn) </li> <li>Zaobao Newspaper (zbn_cmn) </li> </ul><p>The seven-letter codes in the parentheses above are used for the directory names and data files for each source, and are also used (in ALL_CAPS) as part of the unique DOC id string assigned to each news article.</p> <h3>Data</h3> <p> The original data received by the LDC from AFP, Peoples Liberation Army Daily, Xinhua, and Zaobao were encoded in GB-2312, those from CNA were in Big-5, and those from GMW, CNS, and Peoples Daily were in a combination of GB-2312 and GB-18030. To avoid the problems and confusion that could result from differences in character-set specifications, all text files in this corpus have been converted to UTF-8 character encoding. </p> <h3>New in the Fourth Edition</h3> <ul> <li>Two years worth of new articles (January 2007 through December 2008) have been added to the Xinhua, Agence France Presse, and CNA data sets.</li> <li>Four new data sources have been added - Guangming Daily, Central News Service, Peoples Daily and Peoples Liberation Army daily, covering a timespan from November 2006 through December 2008.</li> </ul><h3>Samples</h3> <p>Please view this <a href="./desc/addenda/LDC2009T27.jpg" rel="nofollow">sample</a>.</p> <h3>Sponsorship</h3> <p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p>  </br>
Portions © 2000-2008 Agence France Presse,© 1991-2008 Central News Agency (Taiwan),© 2006-2008 China Military Online, © 2006-2008 Chinanews.com, © 2006-2008 Guangming Daily, © 2006-2008 Peoples Daily, © 1998, 2000-2003 SPH AsiaOne, Ltd., © 1990-2008 Xinhua News Agency, © 2003, 2005, 2007, 2009 Trustees of the University of Pennsylvania
DOI: 10.35111/rfeg-z495
title: ACL Anthology Reference Corpus

description: <h3>Introduction</h3><br>
<p>ACL Anthology Reference Corpus is a digital archive of 10,291 research papers in computational linguistics sponsored by the Association for Computational Linguistics (ACL). Also <a href="http://acl-arc.comp.nus.edu.sg/" rel="nofollow">available from the ACL</a>, this release contains most of the papers that appear up to February 2007 in the web-based <a href="http://aclanthology.info/" rel="nofollow">ACL Anthology</a>, a dynamic repository that currently hosts over 16,500 articles drawn from a range of conferences and workshops as well as past issues of the <em>Computational Linguistics</em> journal. The ACL Anthology Reference Corpus is designed to be a standard, real-world digital collection testbed for experiments in bibliographic and bibliometric research.</p><br>
<p>The ACL is the international scientific and professional society for scholars working on problems involving natural language and computation. Membership includes the ACL quarterly journal, <em>Computational Linguistics</em>, reduced registration at most ACL-sponsored conferences, discounts on ACL-sponsored publications and participation in ACL Special Interest Groups. Since 1988, <em>Computational Linguistics</em> has been the primary forum for research on computational linguistics and natural language processing.</p><br>
<h3>Data</h3><br>
<p>The material in the ACL Anthology Reference Corpus was scanned at 600dpi grayscale for archival storage, down-sampled to 300dpi black-and-white, assembled into articles and stored in the "PDF Image with Hidden Text" format. Author and title metadata was extracted from the OCRed text and used to build HTML index pages. Older materials, such as conference proceedings from the 1960s and early volumes of <em>Computational Linguistics</em>, were manually digitized from microfiche slides.</p><br>
<p>ACL Reference Anthology includes:</p><br>
<ul><br>
<li>10,921 PDF files in the pdf/anthology-PDF tree.</li><br>
<li>13,551 files with metadata described in the metadata/anthology-XML tree</li><br>
<li>84,542 pages in the PDF files</li><br>
</ul></br>
Portions © 1963-2006 Association for Computational Linguistics, © 2009 Trustees of the University of Pennsylvania
DOI: 10.35111/g7a8-ez09
title: Arabic Treebank: Part 3 v 3.2

description: <h3>Introduction</h3><br>
<p>Arabic Treebank: Part 3 (ATB3) v 3.2 was developed at the Linguistic Data Consortium (LDC). It consists of 599 distinct newswire stories from the Lebanese publication An Nahar with part-of-speech (POS), morphology, gloss and syntactic treebank annotation in accordance with the Penn Arabic Treebank (PATB) Guidelines developed in 2008 and 2009. This release represents a significant revision of LDCs previous ATB3 publications: <a href="http://catalog.ldc.upenn.edu/LDC2004T11" rel="nofollow"> Arabic Treebank: Part 3 v 1.0 LDC2004T11</a> and <a href="http://catalog.ldc.upenn.edu/LDC2005T20" rel="nofollow"> Arabic Treebank: Part 3 (full corpus) v 2.0 (MPG + Syntactic Analysis LDC2005T20</a>.</p><br>
<p>The ongoing PATB project supports research in Arabic-language natural language processing and human language technology development. The methodology and work leading to the release of this publication are described in detail in the documentation accompanying this corpus and in two research papers, <a href="https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/lrec2008-enhancing-arabic-treebank.pdf">Enhancing the Arabic Treebank: A Collaborative Effort toward New Annotation Guidelines </a>and <a href="https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/lrec2010-consistent-flexible-integration.pdf">Consistent and Flexible Integration of Morphological Annotation in the Arabic Treebank</a>.</p><br>
<h3>Data</h3><br>
<p>ATB3 v 3.2 contains a total of 339,710 tokens before clitics are split, and 402,291 tokens after clitics are separated for the treebank annotation. This release includes all files that were previously made available to the DARPA GALE program community (Arabic Treebank Part 3 - Version 3.1, LDC2008E22). A number of inconsistencies in the 3.1 release data have been corrected here. These include changes to certain POS tags with the resulting tree changes. As a result, additional clitics have been separated, and some previously incorrectly split tokens have now been merged.</p><br>
<p>One file from ATB3 v 2.0, ANN20020715.0063, has been removed from this corpus as that text is an exact duplicate of another file in this release (ANN20020715.0018). This reduces the number of files from 600 files in ATB3 v 2.0 to 599 files in ATB 3 v 3.2.</p><br>
<h3>Sponsorship</h3><br>
<p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Sample</h3><br>
<p>The included data are available in many different formats and files, as described in detail in the corpus documentation. The following is a screenshot excerpt taken from one of the new integrated data files: <a href="desc/addenda/LDC2010T08.jpg" rel="nofollow">sample</a>.</p></br>
Portions © 2002 An Nahar, © 2003, 2004, 2005, 2007, 2008, 2009 2010 Trustees of the University of Pennsylvania
DOI: 10.35111/0bcd-4871
title: Arabic Treebank: Part 1 v 4.1

description: <h3>Introduction</h3><br>
<p>Arabic Treebank: Part 1 (ATB1) v 4.1 was developed at the Linguistic Data Consortium (LDC). It consists of 734 newswire stories from Agence France Presse (AFP) with part-of-speech (POS), morphology, gloss and syntactic treebank annotation in accordance with the Penn Arabic Treebank (PATB) Guidelines developed in 2008 and 2009. This release represents a significant revision of LDCs previous ATB1 publications: <a href="http://catalog.ldc.upenn.edu/LDC2003T06" rel="nofollow"> Arabic Treebank: Part 1 v 2.0 LDC2003T06</a> and <a href="http://catalog.ldc.upenn.edu/LDC2005T02" rel="nofollow"> Arabic Treebank: Part 1 v 3.0 (POS with full vocalization + syntactic analysis) LDC2005T02</a>. The ongoing PATB project supports research in Arabic-language natural language processing and human language technology development. The methodology and work leading to the release of this publication are described in detail in the documentation accompanying this corpus and in two research papers: <a href="https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/lrec2008-enhancing-arabic-treebank.pdf">Enhancing the Arabic Treebank: A Collaborative Effort toward New Annotation Guidelines</a> and <a href="https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/lrec2010-consistent-flexible-integration.pdf">Consistent and Flexible Integration of Morphological Annotation in the Arabic Treebank</a>.</p><br>
<h3>Data</h3><br>
<p>ATB1 v 4.1 contains a total of 145,386 tokens before clitics are split, and 167,280 tokens after clitics are separated for the treebank annotation.</p><br>
<h3>Sponsorship</h3><br>
<p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>For an example of the data in this corpus, please review the <a href="desc/addenda/LDC2010T13.PNG" rel="nofollow"> sample </a> file.</p><br>
<h3>Updates</h3><br>
<p>No updates have been issued as of this time.</p></br>
 Portions © 2000 Agence France Presse, © 2003, 2005, 2010 Trustees of the University of Pennsylvania
DOI: 10.35111/bpb8-ew63
title: Indian Language Part-of-Speech Tagset: Hindi

description: <h3> Introduction </h3> <p> Indian Language Part-of-Speech Tagset: Hindi, Linguistic Data Consortium (LDC) catalog number LDC2010T24 and isbn 1-58563-571-5, is a corpus developed by Microsoft Research (MSR) India to support the task of Part-of-Speech Tagging (POS) and other data-driven linguistic research on Indian Languages in general. It is created as a part of the <a href="http://research.microsoft.com/en-us/groups/mls/default.aspx" rel="nofollow">Indian Language Part-of-Speech Tagset (IL-POST)</a> project, a collaborative effort among linguists and computer scientists from MSR India, AU-KBC (Anna University, Chennai), Delhi University, IIT Bombay, Jawaharlal Nehru University (Delhi) and Tamil University (Tamilnadu). </p> <p> The goal of the IL-POST project is to provide a common tagset framework for Indian Languages that offers flexibility, cross-linguistic compatibility and reusability across those languages. It supports a three-level hierarchy of Categories, Types and Attributes. The corpus mainly consists therefore of two different levels of information for each lexical token: (a) lexical Category and Types, and (b) set morphological attributes and their associated values in the context. </p> <p> Hindi is the official language of India and a member of the Indo-Aryan language group. It is spoken mainly in the northern states of Rajasthan, Delhi, Haryana, Uttarakhand, Uttar Pradesh, Madhya Pradesh, Chhattisgarh, Himachal Pradesh, Jharkhand and Bihar as well as in much of central India and in communities in Africa, Australia, New Zealand, the Middle East, Europe and North America. Hindi is the first or second language of more than 500 million people. </p> <h3> Data </h3> This corpus contains 4859 sentences (98,450 words) of manually annotated Hindi text randomly collected from the Microsoft Hindi Research Corpus, sourced from the publisher <a href="http://www.webdunia.com/" rel="nofollow">WebDunia</a>. All annotated data is provided in both xml and text files. The xml files are contained in the "XML_files" folder and the text files in the "text_files" folder. Each data file contains between 900-5,000 words. The XML file contains metadata about the material, such as language, encoding and data size. <h3> Annotation Procedure </h3> <p> The Annotation Guidelines for Hindi, included in this release, contain a detailed description of the annotation methodology. The Annotation Tool Guideline 1.0, also included in this publication, describes the annotation interface developed for the IL-POST framework; the tool is not included in this corpus. </p><h3> Updates </h3> <p> Additional information, updates, bug fixes may be available in the LDC catalog entry for this corpus at <a href="http://catalog.ldc.upenn.edu/LDC2010T24" rel="nofollow">LDC2010T24</a>. </p><h3>Samples</h3>  </br>
 Portions © 2010 Microsoft Research Labs India, Pvt. Ltd., © 2010 Trustees of the University of Pennsylvania
DOI: 10.35111/d9yp-vk77
title: 2005 NIST Speaker Recognition Evaluation Training Data

description: <h3>Introduction</h3><br>
<p>2005 NIST Speaker Recognition Evaluation Training Data was developed at the Linguistic Data Consortium (LDC) and NIST (National Institute of Standards and Technology). It consists of 392 hours of conversational telephone speech in English, Arabic, Mandarin Chinese, Russian and Spanish and associated English transcripts used as training data in the NIST-sponsored 2005 Speaker Recognition Evaluation (SRE). The ongoing series of SRE yearly evaluations conducted by NIST are intended to be of interest to researchers working on the general problem of text independent speaker recognition. To that end the evaluations are designed to be simple, to focus on core technology issues, to be fully supported and to be accessible to those wishing to participate.</p><br>
<p>The task of the 2005 SRE evaluation was speaker detection, that is, to determine whether a specified speaker is speaking during a given segment of conversational speech. The task was divided into 20 distinct and separate tests involving one of five training conditions and one of four test conditions. Further information about the task conditions is contained in the <a href="https://www.nist.gov/sites/default/files/documents/2017/09/26/sre-05_evalplan-v6.pdf">The NIST Year 2005 Speaker Recognition Evaluation Plan.</a></p><br>
<h3>Data</h3><br>
<p>The speech data consists of conversational telephone speech with multi-channel data collected simultaneously from a number of auxiliary microphones. The files are organized into two segments: 10 second two-channel excerpts (continuous segments from single conversations that are estimated to contain approximately 10 seconds of actual speech in the channel of interest) and 5 minute two-channel conversations.</p><br>
<p>The speech files are stored as 8-bit u-law speech signals in separate SPHERE files. In addition to the standard header fields, the SPHERE header for each file contains some auxiliary information that includes the language of the conversation and whether the data was recorded over a telephone line.</p><br>
<p>English language word transcripts in .cmt format were produced using an automatic speech recognition system (ASR)with error rates in the range of 15-30%.</p><br>
<h3>Samples</h3><br>
<p>For an example of the data contained in this corpus, review this <a href="desc/addenda/LDC2011S01.wav" rel="nofollow">audio sample</a>.</p></br>
 Portions © 2004-2005, 2011 Trustees of the University of Pennsylvania
DOI: 10.35111/hv72-jf32
title: 2005 Spring NIST Rich Transcription (RT-05S) Evaluation Set

description: <h3>Introduction</h3><br>
<p>2005 Spring NIST Rich Transcription (RT-05S) Conference Meeting Evaluation Set was developed by LDC and NIST (National Institute of Standards and Technology). It contains approximately 78 hours of English meeting speech, reference transcripts and other material used in the <a href="http://www.itl.nist.gov/iad/mig/tests/rt/2005-spring/index.html" rel="nofollow">RT Spring 2005 evaluation</a>. Rich Transcription (RT) is broadly defined as a fusion of speech-to-text (STT) technology and metadata extraction technologies providing the bases for the generation of more usable transcriptions of human-human speech in meetings. LDC has also released <a href="http://catalog.ldc.upenn.edu/LDC2007S11" rel="nofollow">2004 Spring NIST Rich Transcription (RT-04S) Development Data LDC2007S11</a> and <a href="http://catalog.ldc.upenn.edu/LDC2007S12" rel="nofollow">2004 Spring NIST Rich Transcription (RT-04S) Evaluation Data LDC2007S12</a>.</p><br>
<p>RT-05S included the following tasks in the meeting domain:</p><br>
<ul><br>
<li>Speech-To-Text (STT) -convert spoken words into streams of text</li><br>
<li>Speaker Diarization (SPKR) -find the segments of time within a meeting in which each meeting participant is talking</li><br>
<li>Speech Activity Detection (SAD) - detect when someone in a meeting space is talking</li><br>
</ul><br>
<p>Further information about the evaluation is available on the <a href="http://www.itl.nist.gov/iad/mig/tests/rt/2005-spring/index.html" rel="nofollow">RT-05Spring Evaluation Website</a>. Please note the lecture meeting data is not included in this release.</p><br>
<h3>Data Description</h3><br>
<p>The data in this release consists of portions of meeting speech collected between 2001 and 2005 by the IDIAP Research Institutes Augmented Multi-Party Interaction project (AMI), Martigny, Switzerland International Computer Science Institute (ICSI) at University of California, Berkeley Interactive Systems Laboratories (ISL) at Carnegie Mellon University (CMU), Pittsburgh, PA NIST and Virginia Polytechnic Institute and State University (VT), Blacksburg, VA. Each meeting excerpt contains a head-mic recording for each subject and one or more distant microphone recordings.</p><br>
<p>Reference transcripts for the evaluation excerpts were prepared by LDC according to its <a href="http://www.ldc.upenn.edu/Catalog/docs/LDC2011S06/MeetingDataCarefulSpec-V1.2.pdf" rel="nofollow">Meeting Recording Careful Transcription Guidelines</a>. Those specifications are designed to provide an accurate, verbatim (word-for-word) transcription, time-aligned with the audio file and including the identification of additional audio and speech signals with special mark-up.</p><br>
<h3>Samples</h3><br>
<p>For an example of the data contained in this corpus, review this <a href="desc/addenda/LDC2011S06.wav" rel="nofollow">audio sample</a> and <a href="desc/addenda/LDC2011S06.txt" rel="nofollow">transcript sample</a>.</p></br>
 Portions © 2011 Trustees of the University of Pennsylvania