cometadata/datacite-titles-descriptions-related-identifiers huggingface dataset:
DOI: 10.35111/7fw4-ev85
title: TRAD Chinese-French Parallel Text -- Broadcast News

description: <h3>Introduction</h3><br>
<p>TRAD Chinese-French Parallel Text -- Broadcast News was developed by <a href="http://elda.org/en/">ELDA</a> as part of the <a href="http://www.elra.info/en/projects/archived-projects/pea-trad/">PEA-TRAD project</a>. It contains French translations of a subset of approximately 30,000 Chinese characters from GALE Phase 1 Chinese Broadcast News Parallel Text - Part 3 (<a href="../../../LDC2008T18">LDC2008T18</a>).</p><br>
<p>The PEA-TRAD project (Translation as a Support for Document Analysis) was supported by the French Ministry of Defense (DGA). Its purpose was to develop speech-to-speech translation technology for multiple languages (e.g., Arabic, Chinese, Pashto) from a variety of domains. ELDA developed several corpora for this effort.</p><br>
<p>The Linguistic Data Consortium (LDC) has also released the following TRAD corpora:</p><br>
<ul><br>
<li>TRAD Chinese-French Parallel Text -- Blog (<a href="../../../LDC2018T02">LDC2018T02</a>)</li><br>
<li>TRAD Arabic-French Parallel Text -- Newsgroup (<a href="../../../LDC2018T13">LDC2018T13</a>)</li><br>
<li>TRAD Arabic-French Parallel Text -- Newswire (<a href="../../../LDC2018T21">LDC2018T21</a>)</li><br>
</ul><br>
<h3>Data</h3><br>
<p>This release consists of 977 segments (translation units) from 139 documents. The source data is Chinese broadcast news collected and translated into English by LDC for the DARPA GALE (Global Autonomous Language Exploitation) program. Information about the ELDA translation team, translation guidelines and validation results is contained in the documentation accompanying this release.</p><br>
<p>The Chinese source file contains 33,571 characters and the French reference translation contains 22,424 words. The data is presented in two unicode-encoded XML files along with an associated DTD.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2018T17.src.xml">source sample</a> and <a href="desc/addenda/LDC2018T17.ref.xml">reference sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2005, 2006 China Central TV, © 2005, 2006 Phoenix TV, © 2018 ELDA, © 2005-2006, 2008, 2018 Trustees of the University of Pennsylvania
DOI: 10.35111/sd50-6m36
title: BOLT Information Retrieval Comprehensive Training and Evaluation

description: <h3>Introduction</h3><br>
<p>BOLT Information Retrieval Comprehensive Training and Evaluation was developed by the Linguistic Data Consortium (LDC) and consists of all data produced in support of the Information Retrieval (<a href="https://www.ldc.upenn.edu/collaborations/current-projects/bolt/information-retrieval">IR</a>) task within the DARPA Broad Operational Language Translation (BOLT) Program, including annotations, source documents and scoring software.</p><br>
<p>The <a href="https://www.ldc.upenn.edu/collaborations/current-projects/bolt">BOLT</a> program developed machine translation and information retrieval for less formal genres, focusing particularly on user-generated content. LDC supported BOLT by collecting informal data sources -- discussion forums, text messaging and chat -- in Chinese, Egyptian Arabic and English. The collected data was translated and annotated for various tasks including word alignment, treebanking, propbanking and co-reference.</p><br>
<p>The material in this release relates to the IR task, which sought to support development of systems that could: (1) take as input a natural language English query sentence; (2) return relevant responses to that query from a large corpus of informal documents in the three BOLT languages (Arabic, Chinese, and English); and (3) translate responses from non-English documents into English.</p><br>
<h3>Data</h3><br>
<p>BOLT Information Retrieval Comprehensive Training and Evaluation contains the pilot, dry run, and evaluation data developed for each phase of the BOLT IR task, including: (1) natural-language IR queries, system responses to queries, and manually-generated assessment judgments for system responses; (2) discussion forum source documents in Arabic, Chinese and English; (3) scoring software for each evaluation phase; and (4) experimental data developed in Phase 2.</p><br>
<p>Source data is presented as a series of zip archives containing xml files. Queries and responses data are presented as XML as well. Judgments are included as tab delimited files.</p><br>
<h3>Acknowledgement</h3><br>
<p>This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-11-C-0145. The content does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>Please view the following samples:</p><br>
<ul><br>
<li><a href="desc/addenda/LDC2018T18.src.xml">Source Data</a></li><br>
<li><a href="desc/addenda/LDC2018T18.que.xml">Query</a></li><br>
<li><a href="desc/addenda/LDC2018T18.asses.xml">Assessment</a></li><br>
<li><a href="desc/addenda/LDC2018T18.r-asses.xml">Response Assessment</a></li><br>
</ul><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2012-2016, 2018 Trustees of the University of Pennsylvania
DOI: 10.35111/hkfc-7865
title: BOLT English SMS/Chat

description: <h3>Introduction</h3><br>
<p>BOLT English SMS/Chat was developed by the Linguistic Data Consortium (LDC) and consists of naturally-occurring Short Message Service (SMS) and Chat (CHT) data collected through data donations and live collection involving native speakers of English. The corpus contains 18,429 conversations totaling 3,674,802 words across 375,967 messages.</p><br>
<p>The <a href="https://www.ldc.upenn.edu/collaborations/current-projects/bolt">BOLT</a> (Broad Operational Language Translation) program developed machine translation and information retrieval for less formal genres, focusing particularly on user-generated content. LDC supported the BOLT program by collecting informal data sources -- discussion forums, text messaging and chat -- in Chinese, Egyptian Arabic and English. The collected data was translated and annotated for various tasks including word alignment, treebanking, propbanking and co-reference.</p><br>
<h3>Data</h3><br>
<p>The data in this release was collected using two methods: new collection via LDC's collection platform and donation of SMS or chat archives from BOLT collection participants. All data collected were reviewed manually to exclude any messages/conversations that were not in the target language or that had sensitive content, such as personal identifying information (PII). All data is presented in UTF-8 XML.</p><br>
<h3>Acknowledgement</h3><br>
<p>This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-11-C-0145. The content does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2018T19.xml">sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2018 Trustees of the University of Pennsylvania
DOI: 10.35111/z1wg-9x78
title: TRAD Arabic-French Parallel Text -- Newswire

description: <h3>Introduction</h3><br>
<p>TRAD Arabic-French Parallel Text -- Newswire was developed by <a href="http://elda.org/en/">ELDA</a> as part of the <a href="http://www.elra.info/en/projects/archived-projects/pea-trad/">PEA-TRAD project</a>. It contains French translations of a subset of approximately 20,000 Arabic words from NIST 2008 Open Machine Translation (OpenMT) Evaluation (<a href="../../../LDC2010T21">LDC2010T21</a>).</p><br>
<p>The PEA-TRAD project (Translation as a Support for Document Analysis) was supported by the French Ministry of Defense (DGA). Its purpose was to develop speech-to-speech translation technology for multiple languages (e.g., Arabic, Chinese, Pashto) from a variety of domains. ELDA developed several corpora for this effort.</p><br>
<p>The Linguistic Data Consortium (LDC) has also released the following TRAD corpora:</p><br>
<ul><br>
<li>TRAD Chinese-French Parallel Text -- Blog (<a href="../../../LDC2018T02">LDC2018T02</a>)</li><br>
<li>TRAD Arabic-French Parallel Text -- Newsgroup (<a href="../../../LDC2018T13">LDC2018T13</a>)</li><br>
<li>TRAD Chinese-French Parallel Text -- Broadcast News (<a href="../../../LDC2018T17">LDC2018T17</a>)</li><br>
</ul><br>
<h3>Data</h3><br>
<p>This release consists of 813 segments (translations units) from 74 documents. The source data is Arabic newswire text collected and translated into English by LDC. Information about the ELDA translation team, translation guidelines and validation results is contained in the documentation accompanying this release.</p><br>
<p>The Arabic source file contains 19,902 words and the French reference translation contains 29,104 words. The data is presented in two unicode-encoded XML files along with an associated DTD.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2018T21_src.xml">Arabic sample</a> and <a href="desc/addenda/LDC2018T21_ref.xml">French sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2007 Agence France Presse, Al-Ahram, Al Hayat, An Nahar, Al Quds-Al Arabi, Asharq Al-Awsat, Assabah, Xinhua News Agency, © 2018 ELDA, © 2007, 2009, 2010, 2018 Trustees of the University of Pennsylvania
DOI: 10.35111/m0kp-4w25
title: TAC Relation Extraction Dataset

description: <h3>Introduction</h3><br>
<p>TAC Relation Extraction Dataset (TACRED) was developed by <a href="https://nlp.stanford.edu/">The Stanford NLP Group</a> and is a large-scale relation extraction dataset with 106,264 examples built over English newswire and web text used in the NIST <a href="https://www.ldc.upenn.edu/collaborations/current-projects/tac-kbp">TAC KBP</a> English slot filling evaluations during the period 2009-2014. The annotations were derived from TAC KBP relation types (see <a href="https://tac.nist.gov/2017/KBP/ColdStart/guidelines.html">the guidelines</a>), from human annotations developed by the Linguistic Data Consortium and from crowdsourcing using <a href="https://www.mturk.com/">Mechanical Turk</a>.</p><br>
<h3>Data</h3><br>
<p>In each year of the slot filling evaluation, 100 entities (people or organizations) were given as queries (i.e., subjects), for which participating systems should find associated relations and object entities. All sentences judged in the TAC KBP evaluation and a sampling of other sentences that contain the query entities in the evaluation corpus form TACRED. Each sentence was crowd-annotated using Mechanical Turk, where each turk annotator was asked to annotate the subject and object entity spans and the corresponding relation.</p><br>
<p>Data is presented in both CoNLL and JSON format, both encoded in UTF-8. Scoring tools and gold relation labels are also included. Source corpora used for this dataset were TAC KBP Comprehensive English Source Corpora 2009-2014 (<a href="../../../LDC2018T03">LDC2018T03</a>) and TAC KBP English Regular Slot Filling - Comprehensive Training and Evaluation Data 2009-2014 (<a href="../../../LDC2018T22">LDC2018T22</a>).</p><br>
<p>For detailed information about the dataset and benchmark results, please refer to <a href="https://nlp.stanford.edu/pubs/zhang2017tacred.pdf">the TACRED paper</a>.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2018T24.conll">CoNLL sample</a> and <a href="desc/addenda/LDC2018T24.json">JSON sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 1994-1997, 2001-2010 Agence France Presse, © 2005 Aljazeera, © 1996-1997 American Broadcasting Corporation, © 1994-2010 The Associated Press, © 1994-1997 Cable News Network, LP, LLLP, © 1997-1999, 2001, 2003-2010 Central News Agency (Taiwan), © 2005 Dubai TV, © 2005-2006 National Broadcasting Company, Inc., © 1996-1997 National Cable Satellite Corporation, © 1994-1998, 2003-2009 Los Angeles Times - Washington Post News Service, Inc., © 1994-2010 New York Times, © 1996-1997 Public Radio International, © 1994-1995 Reuters America, Inc., © 1996 The University of California, USC Radio and Marketplace, © 2010 The Washington Post Service with Bloomberg News, © 1995-2010 Xinhua News Agency, © 2018 The Board of Trustees of the Leland Stanford Junior University, © 1996-1998, 2007, 2008, 2009, 2011, 2014, 2018 Trustees of the University of Pennsylvania
DOI: 10.35111/m8n7-xy13
title: HAVIC MED Event E051-E060 -- Videos, Metadata and Annotation

description: <h3>Introduction</h3><br>
<p>HAVIC MED Event E051-E060 -- Videos, Metadata and Annotation was developed by the Linguistic Data Consortium (LDC) and is comprised of approximately 53 hours of user-generated videos with annotation and metadata.</p><br>
<p>To advance multimodal event detection and related technologies, LDC developed, in collaboration with <a href="https://www.nist.gov/">NIST</a> (the National Institute of Standards and Technology), a large, heterogeneous, annotated multimodal corpus for <a href="https://www.ldc.upenn.edu/collaborations/past-projects/havic">HAVIC</a> (the Heterogeneous Audio Visual Internet Collection) that was used in the NIST-sponsored <a href="https://www.nist.gov/itl/iad/mig/trecvid-multimedia-event-detection-evaluation-track">MED</a> (Multimedia Event Detection) task for several years. HAVIC MED Event E051-E060 is a subset of that corpus, specifically, a collection of event videos for the HAVIC Project originally released to support the <a href="https://www.nist.gov/itl/iad/mig/med-2016-evaluation">2016 Multimedia Event Detection task</a>.</p><br>
<h3>Data</h3><br>
<p>The data consists of videos of various events (event videos) and videos completely unrelated to events (background videos) harvested by a large team of human annotators. Each event video was manually annotated with a set of judgments describing its event properties and other salient features. Background videos were labeled with topic and genre categories.</p><br>
<p>All video files are in .mp4 format (h.264), with varying bit-rates and levels of audio fidelity and video resolution. Metadata and annotation for the videos are stored in a .tsv file.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2018V01.mp4">video sample</a> and <a href="desc/addenda/LDC2018V01.tsv">annotation sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2011-2016 YouTube, LLC, © 2011-2018 Trustees of the University of Pennsylvania
DOI: 10.35111/1jsy-0150
title: SRI Speech-Based Collaborative Learning Corpus

description: <h3>Introduction</h3><br>
<p>SRI Speech-Based Collaborative Learning Corpus was developed by <a href="https://www.sri.com/">SRI International</a> and is comprised of approximately 120 hours of English speech from 134 US middle school students working collaboratively. The data set also contains orthographic transcriptions, manual annotation of collaboration, log files, and supporting documentation.</p><br>
<p>This collection was part of a project investigating the utility of a speech-based learning analytics approach to collaborative learning. The goal was to determine whether detectable patterns exist in student speech that correlate with collaborative learning indicators and to provide a means of assessing collaboration quality. The participants were students in middle schools (grades six, seven and eight) located in California. Students worked in groups of three on sets of short mathematics problems based on the "cloze" task in which each student was assigned one blank and each problem required the students to work together and talk to each other to coordinate their three answers. The problems were presented on iPads with a custom software application.</p><br>
<h3>Data</h3><br>
<p>The audio data was captured by both head-mounted and table-top microphones and is released as 16 kHz, 16-bit flac compressed pcm wav.</p><br>
<p>Recording sessions were manually annotated with codes that mark indicators of collaboration (I codes) and that assess the overall collaboration quality of the interaction (Q codes). Annotations are presented as UTF-8 csv files.</p><br>
<p>Also included in this corpus are orthorgraphic transcripts for a subset of the audio recordings and log files for iPad usage; both are released as UTF-8 encoded plain text.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2019S01.flac">speech sample</a> and <a href="desc/addenda/LDC2019S01.txt">transcript sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2019 SRI International, © 2019 Trustees of the University of Pennsylvania
DOI: 10.35111/b6pd-wn73
title: CALLFRIEND Egyptian Arabic Second Edition

description: <h3>Introduction</h3><br>
<p>CALLFRIEND Egyptian Arabic Second Edition was developed by the Linguistic Data Consortium (LDC) and consists of approximately 25 hours of unscripted telephone conversations between native speakers of Egyptian Arabic. This second edition updates the audio files to wav format, simplifies the directory structure and adds documentation and metadata. The first edition is available as CALLFRIEND Egyptian Arabic (<a href="../../../LDC96S49">LDC96S49</a>).</p><br>
<p>The CALLFRIEND series is a collection of telephone conversations in several languages conducted by LDC in support of language identification technology development. Languages covered in the collection include American English, Canadian French, Egyptian Arabic, Farsi, German, Hindi, Japanese, Korean, Mandarin Chinese, Spanish, Tamil and Vietnamese.</p><br>
<h3>Data</h3><br>
<p>All data was collected before July 1997. Participants could speak with a person of their choice on any topic; most called family members and friends. All calls originated in North America. The recorded conversations last up to 30 minutes.</p><br>
<p>The data was recorded as 8kHz u-law SPH encoded stereo files, with one end of the phone call on each channel. In this release, files were converted to WAV format, and information from the original SPH headers is described in the documentation. SPH files are not included in this second edition.</p><br>
<p>The audio files were originally split into train, dev and test folders of 20 recordings each, but they are combined in this release.</p><br>
<p>Completed calls passed through a human auditing process to verify that the target language was spoken by the participants, to check the quality of the recordings, and to record information about dialect, noise and distortion.</p><br>
<h3>Samples</h3><br>
<p>Please listen to this <a href="desc/addenda/LDC2019S04.wav">sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 1996, 1997, 2019 Trustees of the University of Pennsylvania
DOI: 10.35111/n1gk-as61
title: VAST Chinese Speech and Transcripts

description: <h3>Introduction</h3><br>
<p>VAST Chinese Speech and Transcripts was developed by the Linguistic Data Consortium (LDC) for the VAST (Video Annotation for Speech Technologies) project and is comprised of approximately 29 hours of Mandarin Chinese audio extracted from amateur video content harvested from the web and corresponding time-aligned transcripts.</p><br>
<p>The aim of the <a href="https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/lrec2018-vast.pdf">VAST</a> project was to collect and annotate data in several languages to support the development of speech technologies such as speech activity detection, language identification, speaker identification, and speech recognition. The collection was designed to ensure that the audio covered a wide range of speakers, communication domains, noise environments, and data sources. The data included in this corpus comprises the subset of files selected for transcription from the larger pool of Chinese data collected during the project.</p><br>
<h3>Data</h3><br>
<p>The files in this corpus were transcribed by LDC staff and/or by transcription vendors under contract to LDC using <a href="https://www.ldc.upenn.edu/language-resources/tools/xtrans">XTrans</a>, which supports manual transcription across multiple channels, languages and platforms. Transcribers followed a Quick-Rich Transcription style; transcription guidelines are included in this release. A targeted second pass was made to check for various errors, to correct use of transcription conventions, and to add marking for proper names.</p><br>
<p>The audio data is presented as 16kHz 16-bit flac compressed files (.flac). When uncompressed, the audio files are in PCM MS-WAV format. Transcripts are UTF-8 encoded plain text files in tdf format.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2019S05.flac">audio sample</a> and <a href="desc/addenda/LDC2019S05.tdf">transcript sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2011-2018 YouTube, LLC, © 2019 Trustees of the University of Pennsylvania
DOI: 10.35111/9mth-hy18
title: First DIHARD Challenge Development - SEEDLingS

description: <h3>Introduction</h3><br>
<p>First DIHARD Challenge Development - SEEDLingS was developed by Duke University and the Linguistic Data Consortium (LDC) and contains approximately two hours of English child language recordings along with corresponding annotations used in support of the <a href="https://coml.lscp.ens.fr/dihard/2018/index.html">First DIHARD Challenge</a>. This release, when combined with First DIHARD Challenge Development - Eight Sources (<a href="../../../LDC2019S09">LDC2019S09</a>), contains the development set audio data and annotation as well as the official scoring tool. The evaluation data for the First DIHARD Challenge is also available from LDC as Nine Sources (<a href="../../../LDC2019S12">LDC2019S12</a>) and SEEDLingS (<a href="../../../LDC2019S13">LDC2019S13</a>).</p><br>
<p>The First DIHARD Challenge was an attempt to reinvigorate work on diarization through a shared task focusing on "hard" diarization; that is, speech diarization for challenging corpora where there was an expectation that existing state-of-the-art systems would fare poorly. As such, it included speech from a wide sampling of domains representing diversity in number of speakers, speaker demographics, interaction style, recording quality, and environmental conditions, including, but not limited to: clinical interviews, extended child language acquisition recordings, YouTube recordings, and conversations collected in restaurants.</p><br>
<h3>Data</h3><br>
<p>The source data was drawn from the <a href="http://doi.org/10.21415/T5PK6D">SEEDLingS</a> (The Study of Environmental Effects on Developing Linguistic Skills) corpus, designed to investigate how infants' early linguistic and environmental input plays a role in their learning. Recordings were generated in the home environment of infants in the Rochester, New York area. A subset of that data was annotated by LDC for use in the First DIHARD Challenge.</p><br>
<p>All audio is provided in the form of 16 kHz, mono-channel FLAC files. The diarization for each recording is stored as a NIST Rich Transcription Time Marked (RTTM) file. RTTM files are space-separated text files containing one turn per line. Segmentation files are stored as HTK label files. Each of these files contains one speech segment per line. Both of the annotation file types are encoded as UTF-8. More information about the file formats is provided in the included documentation.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2019 Duke University, © 2019 Trustees of the University of Pennsylvania