cometadata/datacite-titles-descriptions-related-identifiers huggingface dataset:
DOI: 10.35111/tc99-7t62
title: GALE Phase 3 and 4 Chinese Web Parallel Text

description: <h3>Introduction</h3><br>
<p>GALE Phase 3 and 4 Chinese Web Parallel Text was developed by the Linguistic Data Consortium (LDC). Along with other corpora, the parallel text in this release comprised training data for Phases 3 and 4 of the DARPA GALE (Global Autonomous Language Exploitation) Program. This corpus contains Chinese source text and corresponding English translations selected from weblog and newsgroup data collected by LDC and translated by LDC or under its direction.</p><br>
<h3>Data</h3><br>
<p>GALE Phase 3 and 4 Chinese Web Parallel Text includes 88 source-translation document pairs, comprising 67,514 tokens of Chinese source text and its English translation.</p><br>
<p>Data was manually selected for translation according to several criteria, including linguistic features and topic features. The files were formatted into a human-readable translation format and assigned to translation vendors. Translators followed LDC's Chinese to English translation guidelines. Bilingual LDC staff performed quality control procedures on the completed translations.</p><br>
<p>Source data and translations are distributed in TDF format. TDF files are tab-delimited files containing one segment of text along with meta information about that segment. Each field in the TDF file is described in TDF_format.txt. All data are encoded in UTF-8.</p><br>
<h3>Acknowledgement</h3><br>
<p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2017T02.cmn.txt">source sample</a> and <a href="desc/addenda/LDC2017T02.eng.txt">translation sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2011-2012, 2017 Trustees of the University of Pennsylvania
DOI: 10.35111/esge-nm35
title: First-Year Law Students' Court Memoranda

description: <h3>Introduction</h3><br>
<p>First-Year Law Students' Court Memoranda consists of 197 English law student writing samples of legal briefs annotated for certain characteristics along with accompanying survey responses by the student writers.</p><br>
<p>The briefs were created in a law school writing class at two law schools in the US Midwest during the 2011-12 academic year. Students who agreed to participate in this study uploaded their briefs to an online survey instrument and answered questions regarding their age, gender, level of education, most recent writing course and method of learning English. The study's purpose was to apply natural language processing approaches to determine any differences in the briefs' language attributable to the students' self-reported genders.</p><br>
<h3>Data</h3><br>
<p>The writings are the year-end memoranda of law to a court required in the two legal writing classes. All students were writing in the same genre and in many instances, on the same hypothetical legal case. The samples were imported into the General Architecture for Text Engineering (<a href="https://gate.ac.uk/">GATE</a>) and annotated by two human coders who identified large text segments specific to the legal genre in which the students wrote, such as text headings, citations, block quotes and footnotes.</p><br>
<p>Writing samples are presented as MS Word documents and annotations and survey responses are presented in XML format. The data has been anonymized to remove names and other identifying information about the student participants.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2017T03.docx">MS Word sample</a> and this <a href="desc/addenda/LDC2017T03.xml">XML sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2017 Brian N. Larson, © 2017 Trustees of the University of Pennsylvania
DOI: 10.35111/ewwh-ka74
title: BOLT Chinese Discussion Forum Parallel Training Data

description: <h3>Introduction</h3><br>
<p>BOLT Chinese Discussion Forum Parallel Training Data was developed by the Linguistic Data Consortium (LDC) and consists of 1,876,799 tokens of Chinese discussion forum data collected for the DARPA BOLT program along with their corresponding English translations.</p><br>
<p>The <a href="https://www.ldc.upenn.edu/collaborations/current-projects/bolt">BOLT</a> (Broad Operational Language Translation) program developed machine translation and information retrieval for less formal genres, focusing particularly on user-generated content. LDC supported the BOLT program by collecting informal data sources -- discussion forums, text messaging and chat -- in Chinese, Egyptian Arabic and English. The collected data was translated and annotated for various tasks including word alignment, treebanking, propbanking and co-reference.</p><br>
<h3>Data</h3><br>
<p>The source data in this release consists of discussion forum threads harvested from the Internet by LDC using a combination of manual and automatic processes. The full source data collection is released as BOLT Chinese Discussion Forums (<a href="../../../LDC2016T05">LDC2016T05</a>). Word-aligned and tagged data is released as BOLT Chinese-English Word Alignment and Tagging - Discussion Forum Training (<a href="../../../LDC2016T19">LDC2016T19</a>).</p><br>
<p>Data was manually selected for translation according to several criteria, including linguistic features and topic features. The files were then segmented into sentence units, formatted into a human-readable translation format and assigned to translation vendors. Translators followed LDC's BOLT translation guidelines. Bilingual LDC staff performed quality control procedures on the completed translations.</p><br>
<p>All data are presented as UTF-8. The following table shows the data volume of this package:</p><br>
<table border="1" cellpadding="5"><br>
<tbody><br>
<tr><br>
<td>Source Lang</td><br>
<td>Genre</td><br>
<td>Files</td><br>
<td>Source Tokens</td><br>
<td>Target Tokens</td><br>
</tr><br>
<tr><br>
<td>Chinese</td><br>
<td>Discussion Forum</td><br>
<td>1,541</td><br>
<td>1,876,779</td><br>
<td>1,557,873</td><br>
</tr><br>
</tbody><br>
</table><br>
<h3>Acknowledgement</h3><br>
<p>This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-11-C-0145. The content does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2017T05.cmn.xml">Chinese sample</a> and <a href="desc/addenda/LDC2017T05.eng.xml">English sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2017 Trustees of the University of Pennsylvania
DOI: 10.35111/hbvp-8v54
title: BOLT Egyptian Arabic SMS/Chat and Transliteration

description: <h3>Introduction</h3><br>
<p>BOLT Egyptian Arabic SMS/Chat and Transliteration was developed by the Linguistic Data Consortium (LDC) and consists of naturally-occurring Short Message Service (SMS) and Chat (CHT) data collected through data donations and live collection involving native speakers of Egyptian Arabic. The corpus contains 5,691 conversations totaling 1,029,248 words across 262,026 messages. Messages were natively written in either Arabic orthography or romanized Arabizi. A total of 1,856 Arabizi conversations (287,022 words) were transliterated from the original romanized Arabizi script into standard Arabic orthography.</p><br>
<p>The <a href="https://www.ldc.upenn.edu/collaborations/current-projects/bolt">BOLT</a> (Broad Operational Language Translation) program developed machine translation and information retrieval for less formal genres, focusing particularly on user-generated content. LDC supported the BOLT program by collecting informal data sources -- discussion forums, text messaging and chat -- in Chinese, Egyptian Arabic and English. The collected data was translated and annotated for various tasks including word alignment, treebanking, propbanking and co-reference.</p><br>
<h3>Data</h3><br>
<p>The data in this release was collected using two methods: new collection via LDC's collection platform, and donation of SMS or chat archives from BOLT collection participants. All data collected were reviewed manually to exclude any messages/conversations that were not in the target language or that had sensitive content, such as personal identifying information (PII).</p><br>
<p>A portion of the source conversations containing Arabizi tokens were automatically transliterated into Arabic script. Once the Arabizi source was transliterated into Arabic script automatically, LDC annotators reviewed, corrected and normalized the transliteration according to "Conventional Orthography for Dialectal Arabic" (CODA). All data is presented in XML.</p><br>
<h3>Acknowledgement</h3><br>
<p>This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-11-C-0145. The content does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2017T07.conv.xml">conversation sample</a> and <a href="desc/addenda/LDC2017T07.transli.xml">transliteration sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2017 Trustees of the University of Pennsylvania
DOI: 10.35111/9890-p128
title: Phrase Detectives Corpus

description: <h3>Introduction</h3><br>
<p>Phrase Detectives Corpus was developed by the <a href="https://www.essex.ac.uk/csee/">School of Computer Science and Electronic Engineering at the University of Essex</a> and consists of approximately 19,012 words across 40 documents anaphorically-annotated by the <a href="http://anawiki.essex.ac.uk/phrasedetectives/">Phrase Detectives game</a>, an online interactive "game-with-a-purpose" (GWAP) designed to collect data about English anaphoric coreference.</p><br>
<p>GWAPs for creating language resources are growing. In general, they employ non-monetary incentives, such as entertainment, to motivate participation and can be successful for large-scale persistent annotation efforts.</p><br>
<h3>Data</h3><br>
<p>The documents in the corpus are taken from <a href="https://www.wikipedia.org/">Wikipedia</a> articles and from narrative text in <a href="https://www.gutenberg.org/">Project Gutenberg</a>. Wikipedia articles and annotation files are presented as XML and Project Gutenberg source files are presented as plain text. All text is encoded as UTF-8. Annotations are comprised of a gold standard version created by multiple experts, as well as a set created by a large non-expert crowd (via the Phase Detectives game).</p><br>
<p>The data was annotated according to a prevalent linguistically-oriented approach for anaphora used in several tasks, including OntoNotes Release 5.0 (<a href="../../../LDC2013T19">LDC2013T19</a>), SemEval-2010 Task 1 Ontonotes English: Coreference Resolution in Multiple Languages (<a href="../../../LDC2011T01">LDC2011T01</a>) and The ARRAU Corpus of Anaphoric Information (<a href="../../../LDC2013T22">LDC2013T22</a>).</p><br>
<h3>Samples</h3><br>
<p>Please view the following <a href="desc/addenda/LDC2017T08.txt">source sample</a> and <a href="desc/addenda/LDC2017T08.xml">annotation sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2017 University of Essex, © 2017 Trustees of the University of Pennsylvania
DOI: 10.35111/n0mc-6m82
title: The EventStatus Corpus

description: <h3>Introduction</h3><br>
<p>The EventStatus Corpus was developed by researchers at <a href="https://engineering.tamu.edu/cse">Texas A&M University</a>, <a href="https://linguistics.stanford.edu/">Stanford University</a> and <a href="http://www.cs.utah.edu/">The University of Utah</a>. It consists of approximately 3,000 English and 1,500 Spanish news articles about civil unrest events annotated with temporal tags.</p><br>
<p>This corpus was designed to support the study of the temporal and aspectual properties of major events, that is, whether an event has already happened, is currently happening or may happen in the future. Since it focuses on a single domain (civil unrest events), it may be appropriate for tasks such as event extraction and temporal question answering.</p><br>
<h3>Data</h3><br>
<p>The relevant news articles were sourced from English Gigaword Fifth Edition (<a href="../../../LDC2011T07">LDC2011T07</a>) and Spanish Gigaword Third Edition (<a href="../../../LDC2011T12">LDC2011T12</a>). The civil unrest events include protests, demonstrations, marches and strikes. The data was annotated as PAST, ON-GOING or FUTURE and within each of those categories, as PLANNED, ALERT or POSSIBLE.</p><br>
<p>In addition to the annotated articles, file lists used in experiments for tuning and test are included. 10-fold cross-validations were performed, and the specific 10-fold splits of the test are included as well. All text is presented as plain text and encoded in UTF-8.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2017T09.txt">sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 1994-2010 Agence France Presse, © 1993-2010 The Associated Press, © 1997-2010 Central News Agency (Taiwan), © 1994-1998, 2003-2009 Los Angeles Times-Washington Post News Service, Inc., © 1994-2010 New York Times, © 2010 The Washington Post News Service with Bloomberg News, © 1995-2010 Xinhua News Agency, © 2017 Ruihong Huang, © 2003, 2005, 2006, 2007, 2009, 2011, 2013, 2017 Trustees of the University of Pennsylvania
DOI: 10.35111/q90m-zp97
title: BOLT English Discussion Forums

description: <h3>Introduction</h3><br>
<p>BOLT English Discussion Forums was developed by the Linguistic Data Consortium (LDC) and consists of 830,440 discussion forum threads in English harvested from the Internet using a combination of manual and automatic processes.</p><br>
<p>The DARPA <a href="https://www.ldc.upenn.edu/collaborations/current-projects/bolt">BOLT</a> (Broad Operational Language Translation) program developed machine translation and information retrieval for less formal genres, focusing particularly on user-generated content. LDC supported the BOLT program by collecting informal data sources -- discussion forums, text messaging and chat -- in Chinese, Egyptian Arabic and English. The collected data was translated and annotated for various tasks including word alignment, treebanking, propbanking and co-reference. The material in this release represents the unannotated English source data in the discussion forum genre.</p><br>
<h3>Data</h3><br>
<p>Collection was seeded based on the results of manual data scouting by native speaker annotators. Scouts were instructed to seek content in English that was original, interactive and informal. Upon locating an appropriate thread, scouts submitted the URL and some simple judgments about it to a database, via a web browser plug-in. When multiple threads from a forum were submitted, the entire forum was automatically harvested and added to the collection. The scale of the collection precluded manual review of all data. Only a small portion of the threads included in this release were manually reviewed, and it is expected that there may be some offensive or otherwise undesired content as well as some threads that contain a large amount of non-English content. Language identification was performed on all threads in this corpus (using <a href="https://github.com/CLD2Owners/cld2">CLD2</a>), and threads for which the results indicate a high probability of largely non-English content are listed in eng_suspect_LID.txt in the docs directory of this package.</p><br>
<p>The corpus is comprised of zipped HTML and XML files. The HTML files are a raw HTML file downloaded from the discussion thread. If the thread spanned multiple URLs, it was stored as a concatenation of the downloaded HTML files. The XML files were converted from the raw HTML.</p><br>
<p> </p><br>
<h3>Acknowledgement</h3><br>
<p>This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-11-C-0145. The content does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2017T11.html">html sample</a> and <a href="desc/addenda/LDC2017T11.xml">xml sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2017 Trustees of the University of Pennsylvania
DOI: 10.35111/x63k-xv09
title: 2015-2016 CoNLL Shared Task

description: <h3>Introduction</h3><br>
<p>2015-2016 CoNLL Shared Task, LDC Catalog Number LDC2017T13 and ISBN 1-58563-812-9, contains the Chinese and English training, development and test data for the <a href="http://cs.brandeis.edu/~clp/conll15st">2015</a> and <a href="http://www.cs.brandeis.edu/~clp/conll16st/">2016</a> CoNLL (Conference on Computational Natural Language Learning) Shared Task Evaluation which focused on shallow discourse parsing.</p><br>
<p>The <a href="http://www.conll.org/">Conference on Computational Natural Language Learning (CoNLL)</a> is accompanied every year by a shared task intended to promote natural language processing applications and evaluate them in a standard setting. Shallow discourse parsing is the task of parsing a piece of text into a set of discourse relations between two adjacent or non-adjacent discourse units. This task is called shallow discourse parsing because the relations in a text are not connected to one another to form a connected structure in the form of a tree or graph.</p><br>
<p>LDC has also released the following CoNLL Shared Task data sets:</p><br>
<ul><br>
<li>2006 CoNLL Shared Task - Ten Languages (<a href="../../../LDC2015T11">LDC2015T11</a>)</li><br>
<li>2006 CoNLL Shared Task - Arabic & Czech (<a href="../../../LDC2015T12">LDC2015T12</a>)</li><br>
<li>2008 CoNLL Shared Task Data (<a href="../../../LDC2009T12">LDC2009T12</a>)</li><br>
<li>2009 CoNLL Shared Task Part 1 (<a href="../../../LDC2012T03">LDC2012T03</a>)</li><br>
<li>2009 CoNLL Shared Task Part 2 (<a href="../../../LDC2012T04">LDC2012T04</a>)</li><br>
</ul><br>
<h3>Data</h3><br>
<p>This release consists of the tokenized, tagged, and parsed tags in English and Chinese. The English train, dev and test data are from Wall Street Journal material in Penn Discourse Treebank Version 2.0 (<a href="../../../LDC2008T05">LDC2008T05</a>); English blind test data are from wikinews. Chinese train, dev and test data are news material from Chinese Discourse Treebank 0.5 (<a href="../../../LDC2014T21">LDC2014T21</a>); Chinese blind test data are from wikinews.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2017T13.src.txt">source sample</a> and <a href="desc/addenda/LDC2017T13.anno.txt">annotation sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 1987-1989 Dow Jones & Company, Inc., © 1994-1998, 2006, Xinhua News Agency, © 2008, 2012 The Penn Discourse Treebank Group, © 2016 Attapol Rutherford, © 2001, 2004, 2005, 2007, 2008, 2009, 2010, 2012, 2013, 2014, 2017 Trustees of the University of Pennsylvania
DOI: 10.35111/ctjv-ez04
title: Ancient Chinese Corpus

description: <h3>Introduction</h3><br>
<p>Ancient Chinese Corpus was developed at <a href="http://en.njnu.edu.cn/schools">Nanjing Normal University</a>. It contains word-segmented and part-of-speech tagged text from <em>Zuozhuan</em>, an ancient Chinese work believed to date from the Warring States Period (475-221 BC). <em>Zuozhuan</em> is a commentary on the <em>Chunqui</em>, a history of the Chinese Spring and Autumn period (770-476 BC). This release is part of a continuing project to develop a large, part-of-speech tagged ancient Chinese corpus.</p><br>
<h3>Data</h3><br>
<p>Ancient Chinese Corpus consists of 180,000 Chinese characters and 195,000 segment units (including words and punctuation). The part-of-speech tag set was developed by Nanjing Normal University and contains 17 tags.</p><br>
<p>This release contains two text files: 268 paragraphs and 10,560 lines. A line is one sentence; paragraphs are separated by one empty line. Each word is tagged with its part-of-speech and separated by a space.</p><br>
<p>The files are presented in UTF-8 plain text files using traditional Chinese script.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2017T14.txt">sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2017 Xiaohe Chen, © 2017 Trustees of the University of Pennsylvania
DOI: 10.35111/gp5k-be63
title: English Web Treebank Propbank

description: <h3>Introduction</h3><br>
<p>English Web Treebank Propbank was developed by the <a href="http://clear.colorado.edu/start/index.html">University of Colorado Boulder - CLEAR (Computational Language and Education Research)</a> and provides predicate-argument structure annotation for English Web Treebank (<a href="../../../LDC2012T13">LDC2012T13</a>).</p><br>
<p>The goal of Propbank (or proposition bank) annotation is to develop annotations with information about basic semantic propositions. English Web Treebank Propbank provides semantic role annotation and predicate sense disambiguation for roughly 50,000 predicates, corresponding to all verbs, all adjectives in equational clauses and all nouns considered to be predicative. Mark-up is in the "unified" propbank annotation format, which combines representations in nouns, verbs and adjectives.</p><br>
<h3>Data</h3><br>
<p>The source data consists of weblogs, newsgroups, email, reviews and questions-answers. Human annotators followed the guidelines included with this release. Annotated propositions were automatically validated to ensure that (1) pointers to the tree nodes were valid, (2) Propbank labels were valid, and (3) Propbank annotation was consistent with the associated frameset.</p><br>
<p>Additionally, XML frame files were validated against the included dtd and were checked for frame internal consistency (e.g. misspelling, extraneous characters, general correctness). Data is presented in UTF-8 XML files.</p><br>
<h3>Samples</h3><br>
<p>Please view the following samples.</p><br>
<ul><br>
<li><a href="desc/addenda/LDC2017T15.src.txt">Source</a></li><br>
<li><a href="desc/addenda/LDC2017T15.prop.txt">Prop</a></li><br>
<li><a href="desc/addenda/LDC2017T15.frame.xml">Frame</a></li><br>
</ul><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2012 Google Inc., © 2011 Yahoo! Inc., © 2017 Colorado University - CLEAR, © 2012, 2017 Trustees of the University of Pennsylvania