cometadata/datacite-titles-descriptions-related-identifiers huggingface dataset:
DOI: 10.35111/289k-b489
title: GALE Phase 4 Chinese Weblog Parallel Sentences

description: <h3>Introduction</h3><br>
<p>GALE Phase 4 Chinese Weblog Parallel Sentences was developed by the Linguistic Data Consortium (LDC). Along with other corpora, the parallel text in this release comprised training data for Phase 4 of the DARPA GALE (Global Autonomous Language Exploitation) Program. This corpus contains Chinese source sentences and corresponding English translations selected from newsgroup and weblog data collected by LDC and translated by LDC or under its direction.</p><br>
<h3>Data</h3><br>
<p>GALE Phase 4 Chinese Weblog Parallel Sentences includes 231 source-translation document pairs, comprising 92,501 tokens of Chinese source text and its English translation.</p><br>
<p>Sentences were selected for translation in two steps. First, files were chosen using sentence selection scripts provided by GALE program participants <a href="http://www.sri.com/">SRI International</a> and <a href="http://www.ibm.com/us/en/">IBM</a>. The output was then manually reviewed by LDC staff to eliminate problematic sentences. Selected files were reformatted into a human-readable translation format and assigned to translation vendors. Translators followed LDC's Chinese to English translation guidelines and were provided with the full source documents containing the target sentences for their reference. Bilingual LDC staff performed quality control procedures on the completed translations.</p><br>
<p>Source data and translations are distributed in TDF format. TDF files are tab-delimited files containing one segment of text along with meta information about that segment. Each field in the TDF file is described in TDF_format.txt. All data are encoded in UTF-8.</p><br>
<h3>Acknowledgement</h3><br>
<p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>Please view the following samples.</p><br>
<ul><br>
<li><a href="desc/addenda/LDC2016T04.jpg">Chinese Source</a></li><br>
<li><a href="desc/addenda/LDC2016T04.sel.cmn.jpg">Chinese Selected Sentences</a> (Source data selected for translation)</li><br>
<li><a href="desc/addenda/LDC2016T04.sel.eng.txt">Selection English</a> (Translated data)</li><br>
</ul><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2016 Trustees of the University of Pennsylvania
DOI: 10.35111/3vm2-t509
title: BOLT Chinese Discussion Forums

description: <h3>Introduction</h3><br>
<p>BOLT Chinese Discussion Forums was developed by the Linguistic Data Consortium (LDC) and consists of 1,597,500 discussion forum threads in Chinese harvested from the Internet using a combination of manual and automatic processes.</p><br>
<p>The DARPA <a href="https://www.ldc.upenn.edu/collaborations/current-projects/bolt">BOLT</a> (Broad Operational Language Translation) program developed machine translation and information retrieval for less formal genres, focusing particularly on user-generated content. LDC supported the BOLT program by collecting informal data sources -- discussion forums, text messaging and chat -- in Chinese, Egyptian Arabic and English. The material in this release represents the unannotated Chinese source data in the discussion forum genre. The data was subseqently translated and annotated for various tasks in the BOLT program including word alignment, treebanking, propbanking and co-reference. </p><br>
<h3>Data</h3><br>
<p>Collection was seeded based on the results of manual data scouting by native speaker annotators. Scouts were instructed to seek content in Mandarin Chinese that was original, interactive and informal. Upon locating an appropriate thread, scouts submitted the URL and some simple judgments about it to a database, via a web browser plug-in. When multiple threads from a forum were submitted, the entire forum was automatically harvested and added to the collection. The scale of the collection precluded manual review of all data. Only a small portion of the threads included in this release were manually reviewed, and it is expected that there may be some offensive or otherwise undesired content as well as some threads that contain a large amount of non-Chinese content. Language identification was performed on all threads in this corpus (using <a href="https://github.com/CLD2Owners/cld2">CLD2</a>), and threads for which the results indicated a high probability of largely non-Chinese content are listed in cmn_suspect_LID.txt in the docs directory of this package.</p><br>
<p>The corpus is comprised of HTML and XML files. The HTML files are a raw HTML file downloaded from the discussion thread. If the thread spanned multiple URLs, it was stored as a concatenation of the downloaded HTML files. The XML files were converted from the raw HTML.</p><br>
<p> </p><br>
<h3>Acknowledgement</h3><br>
<p>This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-11-C-0145. The content does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2016T05.html.jpg">html sample</a> and <a href="desc/addenda/LDC2016T05.xml.jpg">xml sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2016 Trustees of the University of Pennsylvania
DOI: 10.35111/s3rq-f204
title: DEFT Narrative Text

description: <h3>Introduction</h3><br>
<p>DEFT Narrative Text was developed by the Linguistic Data Consortium (LDC) and contains proxy reports and their source newswire used to support DARPA's Deep Exploration and Filtering of Text (DEFT) program. Among the goals of the DEFT program is to develop technologies that can perform various NLP tasks on data in a variety of genres, both formal and informal.</p><br>
<p>LDC provided source data and annotations for DEFT system development. DEFT Narrative Text consists of "proxy reports" (and "multi-proxy reports") in English. (Multi-)proxy reports are intended to mimic the format and other features of some types of government analyst reports using content from newswire articles. The corresponding English newswire source documents are also included in the release.</p><br>
<h3>Data</h3><br>
<p>LDC staff manually selected the source newswire from English Gigaword Fifth Edition (<a href="../../../LDC2011T07">LDC2011T07</a>). Articles were selected for topics of potential interest to the intelligence community based on general guidance from DEFT project sponsors.</p><br>
<p>The newswire source documents are XML files following the Gigaword corpus format. The proxy reports are in plain text format.</p><br>
<h3>Samples</h3><br>
<p>Please view these <a href="desc/addenda/LDC2016T07.src.xml">source xml</a> and <a href="desc/addenda/LDC2016T07.prox.txt">proxy text</a> samples.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p><br>
<h3>Acknowledgement</h3><br>
<p>This material is based on research sponsored by Air Force Research Laboratory and Defense Advance Research Projects Agency under agreement number FA8750-13-2-0045. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Air Force Research Laboratory and Defense Advanced Research Projects Agency or the U.S. Government.</p></br>
Portions © 1996, 2002-2010 Agence France Presse, © 2002-2010 The Associated Press, © 2003-2004, 2007-2010 Central News Agency (Taiwan), © 1995, 2003, 2007-2008 Los Angeles Times-Washington Post News Service, Inc., © 2002, 2004-2005, 2007-2010 New York Times, © 2010 The Washington Post Service with Bloomberg News, © 2001-2010 Xinhua News Agency, © 2003, 2005, 2007, 2009-2011, 2013, 2016 Trustees of the University of Pennsylvania
DOI: 10.35111/c51g-9c37
title: GALE Phase 4 Chinese Broadcast Conversation Transcripts

description: <h3>Introduction</h3><br>
<p>GALE Phase 4 Chinese Broadcast Conversation Transcripts was developed by the Linguistic Data Consortium (LDC) and contains transcriptions of approximately 172 hours of Chinese broadcast conversation speech collected in 2008 by LDC and Hong Kong University of Science and Technology, Hong Kong, during Phase 4 of the DARPA GALE (Global Autonomous Language Exploitation) Program.</p><br>
<p>Corresponding audio data is released as GALE Phase 4 Chinese Broadcast Conversation Speech (<a href="../../../LDC2016S03">LDC2016S03</a>).</p><br>
<p>The broadcast conversation recordings feature interviews, call-in programs and roundtable discussions focusing principally on current events from the following sources: Beijing TV, a national television station in Mainland China; China Central TV (CCTV), a national and international broadcaster in Mainland China; Hubei TV, a regional television station in Mainland China, Hubei Province; Phoenix TV, a Hong Kong-based satellite television station ; and Voice of America (VOA), a U.S. government-funded broadcast programmer.</p><br>
<h3>Data</h3><br>
<p>The transcript files are in plain-text, tab-delimited format (TDF) with UTF-8 encoding, and the transcribed data totals 2,259,952 tokens. The transcripts were created with the LDC-developed transcription tool, XTrans, a multi-platform, multilingual, multi-channel transcription tool that supports manual transcription and annotation of audio recordings. XTrans is available from the following link, <a href="https://www.ldc.upenn.edu/language-resources/tools/xtrans">https://www.ldc.upenn.edu/language-resources/tools/xtrans</a>.</p><br>
<p>The files in this corpus were transcribed by LDC staff and/or by transcription vendors under contract to LDC. Transcribers followed LDC's quick transcription guidelines (QTR) and quick rich transcription specification (QRTR) both of which are included in the documentation with this release. QTR transcription consists of quick (near-) verbatim, time-aligned transcripts plus speaker identification with minimal additional mark-up. It does not include sentence unit annotation. QRTR annotation adds structural information such as topic boundaries and manual sentence unit annotation to the core components of a quick transcript. Files with QTR as part of the filename were developed using QTR transcription. Files with QRTR in the filename indicate QRTR transcription.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2016T12.txt">sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p><br>
<h3>Acknowledgement</h3><br>
<p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p></br>
Portions ©2008 Beijing TV, China Central TV, Hubei TV, Phoenix TV, © 2008, 2016 Trustees of the University of Pennsylvania
DOI: 10.35111/a7en-qm65
title: GALE Phase 4 Arabic Weblog Parallel Sentences

description: <h3>Introduction</h3><br>
<p>GALE Phase 4 Arabic Weblog Parallel Sentences was developed by the Linguistic Data Consortium (LDC). Along with other corpora, the parallel text in this release comprised training data for Phase 4 of the DARPA GALE (Global Autonomous Language Exploitation) Program. This corpus contains Modern Standard Arabic source text and corresponding English translations, selected from newsgroup and weblog data collected by LDC and translated by LDC or under its direction.</p><br>
<h3>Data</h3><br>
<p>GALE Phase 4 Arabic Weblog Parallel Sentences includes 1,067 source-translation document pairs, comprising 68,346 words (Arabic source) of translated data. Data is drawn from various Arabic newsgroup and weblog sources.</p><br>
<p>Sentences were selected for translation in two steps. First, files were chosen using sentence selection scripts provided by GALE program participants <a href="http://www.sri.com/">SRI International</a> and <a href="http://www.ibm.com/us/en/">IBM</a>. The output was then manually reviewed by LDC staff to eliminate problematic sentences. Selected files were reformatted into a human-readable translation format and assigned to translation vendors. Translators followed LDC's Arabic to English translation guidelines and were provided with the full source documents containing the target sentences for their reference. Bilingual LDC staff performed quality control procedures on the completed translations.</p><br>
<p>Source data and translations are distributed in TDF format. TDF files are tab-delimited files containing one segment of text along with meta information about that segment. Each field in the TDF file is described in TDF_format.txt. All data are encoded in UTF-8.</p><br>
<h3>Acknowledgement</h3><br>
<p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>Please view the following samples:</p><br>
<ul><br>
<li><a href="desc/addenda/LDC2016T14.txt">Source sample</a></li><br>
<li><a href="desc/addenda/LDC2016T14.sel.txt">Arabic Selection sample</a></li><br>
<li><a href="desc/addenda/LDC2016T14.sel.eng.txt">English Selection sample</a></li><br>
</ul><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2016 Trustees of the University of Pennsylvania
DOI: 10.35111/njr2-z127
title: GALE Phase 3 and 4 Chinese Broadcast News Parallel Text

description: <h3>Introduction</h3><br>
<p>GALE Phase 3 and 4 Chinese Broadcast News Parallel Text was developed by the Linguistic Data Consortium (LDC). Along with other corpora, the parallel text in this release comprised training data for Phases 3 and 4 of the DARPA GALE (Global Autonomous Language Exploitation) Program. This corpus contains Chinese source text and corresponding English translations selected from broadcast news data collected by LDC between 2006 and 2008 and transcribed and translated by LDC or under its direction.</p><br>
<h3>Data</h3><br>
<p>GALE Phase 3 and 4 Chinese Broadcast News Parallel Text includes 76 source-translation document pairs, comprising 614,608 tokens of Chinese source text and its English translation. Data is drawn from 16 distinct Chinese programs broadcast between 2006 and 2008 by China Central TV (CCTV), a national and international broadcaster in Mainland China and Phoenix TV, a Hong Kong-based satellite television station. The programs in this release feature news programs on current events topics.</p><br>
<p>The files in this release were transcribed by LDC staff and/or transcription vendors under contract to LDC in accordance with the Quick Rich Transcription guidelines developed by LDC. Transcribers indicated sentence boundaries in addition to transcribing the text. Data was manually selected for translation according to several criteria, including linguistic features, transcription features and topic features. The transcribed and segmented files were then reformatted into a human-readable translation format and assigned to translation vendors. Translators followed LDC's Chinese to English translation guidelines. Bilingual LDC staff performed quality control procedures on the completed translations.</p><br>
<p>Source data and translations are distributed in TDF format. TDF files are tab-delimited files containing one segment of text along with meta information about that segment. Each field in the TDF file is described in TDF_format.txt. All data are encoded in UTF-8.</p><br>
<h3>Acknowledgement</h3><br>
<p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2016T15.txt">source sample</a> and <a href="desc/addenda/LDC2016T15.eng.txt">translation sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2006-2008 China Central TV, © 2007 Phoenix TV, © 2006-2008, 2016 Trustees of the University of Pennsylvania
DOI: 10.35111/eg86-k634
title: English Speed Networking Conversational Transcripts

description: <h3>Introduction</h3><br>
<p>English Speed Networking Conversational Transcripts was developed at the <a href="http://www.uwe.ac.uk/">University of the West of England</a> and contains 388 transcripts of English face-to-face and instant messaging conversations about business ideas collected in 2014 and 2015 from participants (undergraduate students) playing different power roles.</p><br>
<p>This corpus was created to examine communication accommodation, specifically, the ways in which an individual's linguistic style, or how an individual communicates, is affected by social power and personality. The data was collected in two studies. In the first study, 40 participants had a series of paired five minute face-to-face conversations playing either a high, low or neutral power role. The same procedure was followed in the second study except that participants discussed business ideas via instant messaging.</p><br>
<h3>Data</h3><br>
<p>The face-to-face conversations were audio-recorded and transcribed verbatim. There are 139 transcripts of conversations between high and low power individuals and 63 transcripts of conversations between neutral power individuals.</p><br>
<p>The instant messaging program automatically saved the transcripts of the messaging conversations; the transcripts were then retrieved and formatted for analysis. There are 85 transcripts of conversations between high and low power individuals and 101 transcripts of conversations between neutral power individuals.</p><br>
<p>The transcripts were anonymized. Gender and age metadata are available where provided.</p><br>
<p>All transcripts are presented as UTF-8 plain text files.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2016T16.txt">sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2014, 2015, 2016 University of the West of England, © 2016 Trustees of the University of Pennsylvania
DOI: 10.35111/153v-4947
title: ARL Arabic Dependency Treebank

description: <h3>Introduction</h3><br>
<p>ARL Arabic Dependency Treebank was developed by the US Army Research Laboratory (<a href="https://www.arl.army.mil/www/default.cfm">ARL</a>) and was derived from four LDC resources: Arabic Treebank (ATB) Part 1 v 4.1 (<a href="../../../LDC2010T13">LDC2010T13</a>), Part 2 v 3.1 (<a href="../../../LDC2011T09">LDC2011T09</a>), Part 3 v 3.2 (<a href="../../../LDC2010T08">LDC2010T08</a>) and Broadcast News v 1.0 (<a href="../../../LDC2012T07">LDC2012T07</a>).</p><br>
<p>LDC's ATB series follows the constituency or phrase structure approach to treebank development in which clauses are divided into noun phrases and verb phrases and in each sentence, one or more nodes may correspond to one element. Dependency grammar, on the other hand, is based on the idea that the verb is the center of the clause structure and that other units in the sentence are connected to the verb as directed links or dependencies. This is a one-to-one correspondence: for every element in the sentence there is one node in the sentence structure that corresponds to that element. ARL Arabic Dependency Treebank was generated using constituency-to-dependency software written at ARL.</p><br>
<h3>Data</h3><br>
<p>The source data in this release consists of Arabic newswire and broadcast programming collected by LDC from various news and broadcast providers.</p><br>
<p>The files are in an 11-column tab-separated format with one or more blank lines between sentences. All files are UTF-8 encoded. Further information about the corpus structure is contained in the documentation accompanying this release.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2016T18.txt">sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2007-2008 Abu Dhabi TV, © 2000 Agence France Presse, © 2007 Al Alam News Channel, © 2007-2008 Al Arabiya, © 2008 Al Baghdadya TV, © 2008 Al Fayha, © 2007-2008 Al Iraqiyah, © 2005-2007 Aljazeera, © 2007 Al Ordiniyah, © 2008 Al Sharqiyah, © 2002 An Nahar, © 2005, 2007-2008 Dubai TV, © 2007 Kuwait TV, © 2008 Oman TV, © 2007 PAC Ltd, © 2007-2008 Saudi TV, © 2007 Syria TV, © 2001-2002 Ummah Press, © 2003, 2004, 2005, 2007, 2008, 2009, 2010, 2011, 2012, 2016 Trustees of the University of Pennsylvania
DOI: 10.35111/m60v-w711
title: GALE Phase 4 Arabic Broadcast News Parallel Sentences

description: <h3>Introduction</h3><br>
<p>GALE Phase 4 Arabic Broadcast News Parallel Sentences was developed by the Linguistic Data Consortium (LDC). Along with other corpora, the parallel text in this release comprised training data for Phase 4 of the DARPA GALE (Global Autonomous Language Exploitation) Program. This corpus contains Modern Standard Arabic source sentences and corresponding English translations selected from broadcast news data collected by LDC in 2007 and 2008 and transcribed and translated by LDC or under its direction.</p><br>
<h3>Data</h3><br>
<p>GALE Phase 4 Arabic Broadcast News Parallel Sentences includes 106 source-translation document pairs, comprising 114,251 words (Arabic source) of translated data. Data is drawn from 24 distinct Arabic programs featuring news broadcasts.</p><br>
<p>The data was transcribed by LDC staff and/or transcription vendors under contract to LDC in accordance with the Quick Rich Transcription guidelines developed by LDC. Transcribers indicated sentence boundaries in addition to transcribing the text. Sentences were selected for translation in two steps. First, files were chosen using sentence selection scripts provided by GALE program participants <a href="http://www.sri.com/">SRI International</a> and <a href="http://www.ibm.com/us/en/">IBM</a>. The output was then manually reviewed by LDC staff to eliminate problematic sentences. Selected files were reformatted into a human-readable translation format and assigned to translation vendors. Translators followed LDC's Arabic to English translation guidelines and were provided with the full source documents containing the target sentences for their reference. Bilingual LDC staff performed quality control procedures on the completed translations.</p><br>
<p>Source data and translations are distributed in TDF format. TDF files are tab-delimited files containing one segment of text along with meta information about that segment. Each field in the TDF file is described in TDF_format.txt. All data are encoded in UTF-8.</p><br>
<h3>Acknowledgement</h3><br>
<p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>Please view the below samples:</p><br>
<ul><br>
<li><a href="desc/addenda/LDC2016T20.txt">Source Sample</a></li><br>
<li><a href="desc/addenda/LDC2016T20.sel.txt">Selection Sample</a></li><br>
<li><a href="desc/addenda/LDC2016T20.sel.eng.txt">Translated Selection Sample</a></li><br>
</ul><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2008 Abu Dhabi TV, Al Arabiya, Al Baghdadya TV, Al Fayha, Al Iraqiyah, Al Ordiniyah, Al Sharqiya, Dubai TV, Oman TV, Yemen TV, © 2007, 2008, 2016 Trustees of the University of Pennsylvania