cometadata/datacite-titles-descriptions-related-identifiers huggingface dataset:
DOI: 10.35111/fnzz-kn07
title: HAVIC MED Progress Test -- Videos, Metadata and Annotation

description: <h3>Introduction</h3><br>
<p>HAVIC MED Progress Test -- Videos, Metadata and Annotation was developed by the Linguistic Data Consortium (LDC) and is comprised of approximately 3,650 hours of user-generated videos with annotation and metadata.</p><br>
<p>To advance multimodal event detection and related technologies, LDC developed, in collaboration with <a href="https://www.nist.gov/">NIST</a> (the National Institute of Standards and Technology), a large, heterogeneous, annotated multimodal corpus for <a href="https://www.ldc.upenn.edu/collaborations/past-projects/havic">HAVIC</a> (the Heterogeneous Audio Visual Internet Collection) that was used in the NIST-sponsored <a href="https://www.nist.gov/itl/iad/mig/trecvid-multimedia-event-detection-evaluation-track">MED</a> (Multimedia Event Detection) task for several years. HAVIC MED Progress Test is a subset of that corpus, specifically, a collection of event and background videos for the HAVIC project originally released to support the <a href="https://www.nist.gov/itl/iad/mig/trecvid-multimedia-event-detection-2012-evaluation">2012</a>, <a href="https://www.nist.gov/itl/iad/mig/med-2013-evaluation">2013</a>, <a href="https://www.nist.gov/itl/iad/mig/med-2014-evaluation">2014</a>, and <a href="https://www.nist.gov/itl/iad/mig/med-2015-evaluation">2015</a> Multimedia Event Detection tasks.</p><br>
<h3>Data</h3><br>
<p>The data consists of videos of various events (event videos) and videos completely unrelated to events (background videos) harvested by a large team of human annotators. Each event video was manually annotated with a set of judgments describing its event properties and other salient features. Background videos were labeled with topic and genre categories.</p><br>
<p>All video files are in .mp4 format (h.264), with varying bit-rates and levels of audio fidelity and video resolution. Metadata and annotation for the videos are stored in a .tsv file.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2019V01.mp4">video sample</a> and <a href="desc/addenda/LDC2019V01.tsv">annotation sample</a>. </p><br>
<h3>Updates</h3><br>
<p>None at this time.</p><br>
<h3>Additional Licensing Instructions</h3><br>
<p>This members-only corpus is available to current members. Contact <a href="mailto:ldc@ldc.upenn.edu">ldc@ldc.upenn.edu</a> for information about becoming a member.</p></br>
Portions © 2011-2016 YouTube, LLC, © 2011-2016, 2019 Trustees of the University of Pennsylvania
DOI: 10.35111/hcwt-mh66
title: Database of Word Level Statistics - Mandarin

description: <h3>Introduction</h3><br>
<p>Database of Word Level Statistics - Mandarin was developed by <a href="https://www.polyu.edu.hk/web/en/home/index.html">The Hong Kong Polytechnic University</a>. It provides lexical characteristics of a descriptive and statistical nature for words and nonwords of Mandarin Chinese. It is designed for researchers particularly concerned with language processing of isolated words. Invariant characteristics include each item's lexicality, sampa, pinyin, IPA transcription, lexical tone, syllable structure, syllable length, pinyin length, segment length, dominant PoS, lexical frequency of the dominant PoS, percent of that dominant PoS, and other PoSes associated with the given item.</p><br>
<h3>Data</h3><br>
<p>The corpus is presented in a series of UTF-8 encoded tab separated plain text files. The original frequency counts were adapted from the word list in <a href="http://crr.ugent.be/programs-data/subtitle-frequencies/subtlex-ch">Subtlex-CH</a>. Monosyllables from the Subtlex-CH character list that were not present as monosyllabic words were added to the list in order to provide statistical information for all Mandarin syllables.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2020L01.txt">sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2020 The Hong Kong Polytechnic University, © 2020 Trustees of the University of Pennsylvania
DOI: 10.35111/a44z-6x49
title: LibriVox Spanish

description: <h3>Introduction</h3><br>
<p>LibriVox Spanish consists of approximately 73 hours of Spanish read speech and transcripts. The audio data was taken from Spanish audiobooks developed by <a href="https://librivox.org/">LibriVox</a>, a non-profit project that creates audiobooks from public domain works. The transcripts were developed for this release.</p><br>
<h3>Data</h3><br>
<p>The audio is comprised of sentences from 300 books read by 154 speakers (77 men and 77 women). Audio files were manually segmented and are between three and ten seconds in length. Native Spanish speakers transcribed the audio data.</p><br>
<p>Audio is presented as 16kHz, 16-bit, single channel flac files. When uncompressed, they produce PCM wav files.</p><br>
<p>Transcripts are contained in a single plain text file encoded as UTF-8. Speaker metadata is also provided.</p><br>
<h3>Samples</h3><br>
<p>Please view the following samples.</p><br>
<ul><br>
<li><a href="desc/addenda/LDC2020S01_nat_f.flac">Native Female</a></li><br>
<li><a href="desc/addenda/LDC2020S01_non-nat_m.flac">Non-Native Male</a></li><br>
<li><a href="desc/addenda/LDC2020S01.txt">Transcripts</a></li><br>
</ul><br>
<h3>Acknowledgements</h3><br>
<p>The author thanks Alejandro V. Mena, Elena Vera and Ang&eacute;lica Guti&eacute;rrez for their support of the social service program "Desarrollo de Tecnolog&iacute;as del Habla" and also thanks the social service students for their hard work. Special thanks to LibriVox for publishing the recordings that constitute this corpus.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2020 Carlos Daniel Hernández Mena, © 2020 Trustees of the University of Pennsylvania
DOI: 10.35111/secv-qh25
title: 2018 NIST Speaker Recognition Evaluation Test Set

description: <h3>Introduction</h3><br>
<p>2018 NIST Speaker Recognition Evaluation Test Set was developed by the Linguistic Data Consortium (LDC) and NIST (National Institute of Standards and Technology). It contains approximately 396 hours of Tunisian Arabic telephone recordings and English web video speech used as development and test data in the NIST-sponsored <a href="https://www.nist.gov/itl/iad/mig/nist-2018-speaker-recognition-evaluation">2018 Speaker Recognition Evaluation (SRE)</a>.</p><br>
<p>The ongoing series of SRE yearly evaluations conducted by NIST are intended to be of interest to researchers working on the general problem of text independent speaker recognition. To this end the evaluations are designed to be simple, to focus on core technology issues, to be fully supported and to be accessible to those wishing to participate.</p><br>
<p>The SRE task is speaker detection, that is, to determine whether a specified target speaker is speaking during a segment of speech. In addition to the traditional focus on telephone speech recorded over a variety of handset types for the training and test conditions, SRE18 added voice over IP data and audio from video. Further information about the evaluation, including the features added in SRE18, is contained in the evaluation plan included in this release.</p><br>
<h3>Data</h3><br>
<p>The telephone speech data was drawn from the Call My Net 2 (CMN2) collection conducted by LDC in Tunisia in which Tunisian Arabic speakers called friends or relatives who agreed to record their telephone conversations lasting between 8-10 minutes. The speech segments include PSTN (public switched telephone network) and VOIP (voice over IP) data.</p><br>
<p>The English audio was sampled from amateur web videos collected by LDC as part of the Video Annotation for Speech Technology (VAST) project.</p><br>
<p>Telephone speech is presented as 8 bit a-law with a sample rate of 8000.</p><br>
<p>The VAST data are presented as 16 bit FLAC files sampled at 44 kHz.</p><br>
<p>In addition to development and evaluation data, this corpus also contains answer keys, trial and train files, development data and evaluation documentation.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2020S04.sph">telephone sample (SPH)</a> and <a href="desc/addenda/LDC2020S04.flac">audio from video sample (FLAC)</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2011-2018 YouTube, LLC, © 2020 Trustees of the University of Pennsylvania
DOI: 10.35111/h61y-f636
title: Multi-Language Conversational Telephone Speech 2011 -- Mandarin Chinese

description: <h3>Introduction</h3><br>
<p>Multi-Language Conversational Telephone Speech 2011 -- Mandarin Chinese was developed by the Linguistic Data Consortium (LDC) and is comprised of approximately 25 hours of telephone speech in Mandarin Chinese.</p><br>
<p>The data were collected primarily to support research and technology evaluation in automatic language identification, and portions of these telephone calls were used in the NIST 2011 Language Recognition Evaluation (<a href="https://www.nist.gov/itl/iad/mig/2011-language-recognition-evaluation">LRE</a>). LRE 2011 focused on language pair discrimination for 24 languages/dialects, some of which could be considered mutually intelligible or closely related.</p><br>
<p>LDC has also released the following as part of the Multi-Language Conversational Telephone Speech 2011 series:</p><br>
<ul><br>
<li>Slavic Group (<a href="../../../LDC2016S11">LDC2016S11</a>)</li><br>
<li>Turkish (<a href="../../../LDC2017S09">LDC2017S09</a>)</li><br>
<li>South Asian (<a href="../../../LDC2017S14">LDC2017S14</a>)</li><br>
<li>Central Asian (<a href="../../../LDC2018S03">LDC2018S03</a>)</li><br>
<li>Central European (<a href="../../../LDC2018S08">LDC2018S08</a>)</li><br>
<li>Spanish (<a href="../../../LDC2018S12">LDC2018S12</a>)</li><br>
<li>Arabic (<a href="../../../LDC2019S02">LDC2019S02</a>)</li><br>
<li>English (<a href="../../../LDC2019S06">LDC2019S06</a>)</li><br>
<li>East Asian (<a href="../../../LDC2019S15">LDC2019S15</a>)</li><br>
</ul><br>
<h3>Data</h3><br>
<p>Participants were recruited by native speakers who contacted acquaintances in their social network. Those native speakers made one call, up to 15 minutes, to each acquaintance. The data was collected using <a href="https://www.ldc.upenn.edu/about/facilities/human-subjects-collection">LDC's telephone collection infrastructure</a>, comprised of three computer telephony systems. Human auditors labeled calls for callee gender, dialect type and noise. Demographic information about the participants was not collected.</p><br>
<p>All audio data are presented in FLAC-compressed MS-WAV (RIFF) file format (*.flac); when uncompressed, each file is 2 channels, recorded at 8000 samples/second with samples stored as 16-bit signed integers, representing a lossless conversion from the original mu-law sample data as captured digitally from the public telephone network. The following table summarizes the total number of calls, total number of hours of recorded audio, and the total size of compressed data:</p><br>
<table><br>
<tbody><br>
<tr><br>
<td>group</td><br>
<td>lng</td><br>
<td>#calls</td><br>
<td>#hours</td><br>
<td>#MB</td><br>
</tr><br>
<tr><br>
<td>chinese</td><br>
<td>cmn</td><br>
<td>132</td><br>
<td>25.6</td><br>
<td>1260</td><br>
</tr><br>
</tbody><br>
</table><br>
<h3>Samples</h3><br>
<p>Please listen to this <a href="desc/addenda/LDC2020S05.flac">audio sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2020 Trustees of the University of Pennsylvania
DOI: 10.35111/a3mk-ae14
title: CALLFRIEND Mandarin Chinese-Taiwan Dialect Second Edition

description: <h3>Introduction</h3><br>
<p>CALLFRIEND Mandarin Chinese-Taiwan Dialect Second Edition was developed by the Linguistic Data Consortium (LDC) and consists of approximately 27 hours of unscripted telephone conversations between native speakers of the Taiwan dialect of Mandarin Chinese. This second edition updates the audio files to wav format, simplifies the directory structure and adds documentation and metadata. The first edition is available as <a href="../../../LDC96S56">CALLFRIEND Mandarin Chinese-Taiwan Dialect (LDC96S56)</a>.</p><br>
<p>The CALLFRIEND series is a collection of telephone conversations in several languages conducted by LDC in support of language identification technology development. Languages covered in the collection include American English, Canadian French, Egyptian Arabic, Farsi, German, Hindi, Japanese, Korean, Mandarin Chinese, Spanish, Tamil and Vietnamese.</p><br>
<h3>Data</h3><br>
<p>All data was collected before July 1997. Participants could speak with a person of their choice on any topic; most called family members and friends. All calls originated in North America. The recorded conversations last up to 30 minutes.</p><br>
<p>The data was recorded as 8kHz u-law SPH encoded stereo files, with one end of the phone call on each channel. In this release, files were converted to WAV format, and information from the original SPH headers is described in the documentation. SPH files are not included in this second edition.</p><br>
<p>The audio files were originally split into train, dev and test folders of 20 recordings each, but they are combined in this release.</p><br>
<p>Completed calls passed through a human auditing process to verify that the target language was spoken by the participants, to check the quality of the recordings, and to record information about dialect, noise and distortion.</p><br>
<h3>Samples</h3><br>
<p>Please listen to this <a href="desc/addenda/LDC2020S06.wav">audio sample (WAV)</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 1996, 1997, 2020 Trustees of the University of Pennsylvania
DOI: 10.35111/w8tv-1e21
title: Chinese CogBank

description: <h3>Introduction</h3><br>
<p>Chinese CogBank is a database of cognitive properties of Chinese words intended for use in metaphor understanding and generation. It consists of 232,497 "word-property" pairs, which are comprised of 83,104 words and 100,195 properties. Each "word-property" type also has an associated frequency which can stand as a functional measure of the importance of a property.</p><br>
<h3>Data</h3><br>
<p>The data was collected via the Chinese search engine <a href="http://www.baidu.com/">Baidu.com</a>. The original collection consisted of 1,258,430 types (5,637,500 tokens) of "word-adjective" pairs that were reduced in Chinese CogBank to 232,497 "word-property" pairs after a series of manual checks.</p><br>
<p>The corpus is presented as a single tab separated value file encoded in UTF-8.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2020T01.txt">sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2020 Bin Li, © 2020 Trustees of the University of Pennsylvania
DOI: 10.35111/1jbk-p616
title: BOLT Egyptian Arabic-English Word Alignment -- Conversational Telephone Speech Training

description: <h3>Introduction</h3><br>
<p>BOLT Egyptian Arabic-English Word Alignment -- Conversational Telephone Speech Training was developed by the Linguistic Data Consortium (LDC) and consists of 153,171 words of Egyptian Arabic and English parallel text enhanced with linguistic tags to indicate word relations.</p><br>
<p>The DARPA <a href="https://www.ldc.upenn.edu/collaborations/current-projects/bolt"> BOLT</a> (Broad Operational Language Translation) program developed machine translation and information retrieval for less formal genres, focusing particularly on user-generated content. LDC supported the BOLT program by collecting informal data sources -- discussion forums, text messaging and chat -- in Chinese, Egyptian Arabic and English. The collected data was translated and annotated for various tasks including word alignment, treebanking, propbanking and co-reference.</p><br>
<h3>Data</h3><br>
<p>The source data in this release consists of transcripts of Egyptian Arabic conversational telephone speech (CTS) from LDC's CALLHOME and CALLFRIEND collections (<a href="../../../LDC97S45">LDC97S45</a>, <a href="../../../LDC97T19">LDC97T19</a>, <a href="../../../LDC2002S37">LDC2002S37</a>, <a href="../../../LDC2002T38">LDC2002T38</a>, <a href="../../../LDC96S49">LDC96S49</a>) that were translated into English by professional translation agencies and annotated for the word alignment task.</p><br>
<p>The BOLT word alignment task was built on treebank annotation. Specifically, Egyptian Arabic source tree tokens were automatically extracted from tree files in LDC's BOLT Egyptian Arabic Treebank. Those tree files had been tagged for part-of-speech and syntactically annotated. That data was then aligned and annotated for the word alignment task.</p><br>
<p>The data profile broken down by character tokens, tree tokens and segments appears below:</p><br>
<table border="1" cellpadding="5"><br>
<tbody><br>
<tr><br>
<td>Language</td><br>
<td>Genre</td><br>
<td>Files</td><br>
<td>Words</td><br>
<td>Tree-tokens</td><br>
<td>Segments</td><br>
</tr><br>
<tr><br>
<td>Egyptian Arabic</td><br>
<td>CTS</td><br>
<td>176</td><br>
<td>153,171</td><br>
<td>215,896</td><br>
<td>20,010</td><br>
</tr><br>
</tbody><br>
</table><br>
<h3>Acknowledgement</h3><br>
<p>This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-11-C-0145. The content does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>Please view the following samples:</p><br>
<ul><br>
<li><a href="desc/addenda/LDC2020T05.arz.tkn.txt">Egyptian-Arabic Token Sample</a></li><br>
<li><a href="desc/addenda/LDC2020T05.eng.tkn.txt">English Token Sample</a></li><br>
<li><a href="desc/addenda/LDC2020T05.wa.txt">Word Alignment Token</a></li><br>
</ul><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 1996, 1997, 2002, 2012-2015, 2020 Trustees of the University of Pennsylvania
DOI: 10.35111/4h9q-yt20
title: EVALution

description: <h3>Introduction</h3><br>
<p>EVALution was developed by <a href="https://www.polyu.edu.hk/web/en/home/index.html">The Hong Kong Polytechnic University</a>. It is comprised of English and Mandarin Chinese data sets -- EVALution 1.0 and EVALution-Man, respectively -- that contain semantic relations and metadata for training and evaluating distributional semantic models.</p><br>
<h3>Data</h3><br>
<p>EVALution 1.0 consists of approximately 7500 English tuples extracted from <a href="http://conceptnet.io/">ConceptNet 5.0</a> and <a href="https://wordnet.princeton.edu/">WordNet</a> and filtered through automatic methods and crowd-sourcing. Several semantic relations between word pairs were instantiated, including hypernymy, synonymy, antonymy and meronymy. The corpus also includes additional information that can be used to filter the pairs or to analyze the results, such as relation domain, word frequency, word part-of-speech and word semantic field.</p><br>
<p>EVALution-MAN consists of Chinese word pairs from two sources: <a href="http://cwn.ling.sinica.edu.tw/">Chinese Wordnet</a> and humans who completed an elicitation task by supplying missing words to sentences. The human-supplied sentence word pairs were then judged by human raters for reliability.</p><br>
<p>All text data is presented as UTF-8 encoded tab separated plain text.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2020T06.eng.txt">EVALutaion 1.0 sample</a> and <a href="desc/addenda/LDC2020T06.cmn.txt">EVALution-MAN sample</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2020 The Hong Kong Polytechnic University, © 2020 Trustees of the University of Pennsylvania
DOI: 10.35111/fr89-3285
title: Abstract Meaning Representation 2.0 - Four Translations

description: <h3>Introduction</h3><br>
<p>Abstract Meaning Representation 2.0 - Four Translations was developed by researchers at the <a href="https://www.ed.ac.uk/informatics">University of Edinburgh, School of Informatics </a> and consists of Spanish, German, Italian and Chinese Mandarin translations of a subset of sentences from <a href="../../../LDC2017T10">Abstract Meaning Representation (AMR) Annotation Release 2.0 (LDC2017T10)</a>.</p><br>
<p>AMR Annotation Release 2.0 is a semantic treebank of over 39,000 English natural language sentences from broadcast conversations, newswire and web text. The translated data in this release was designed for use in cross-lingual parsing.</p><br>
<h3>Data</h3><br>
<p>This corpus contains translations of the test split sentences from LDC2017T10, a total of 5,484 sentences or 1,371 sentences per language. The source sentences were drawn from material collected by the Linguistic Data Consortium, specifically, discussion forum text from the DARPA BOLT and DARPA DEFT programs, transcripts and English translations of Mandarin Chinese broadcast news programming, Wall Street Journal text, translated Xinhua news texts, various newswire texts from NIST OpenMT evaluations and weblog data from the DARPA GALE program.</p><br>
<p>All data are presented as UTF-8 encoded plain text.</p><br>
<h3>Samples</h3><br>
<p>Please view this <a href="desc/addenda/LDC2020T07.txt">Italian text sample (TXT)</a>.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2020 Shay Cohen, © 2020 Marco Damonte, © 2002-2005 Agence France Presse, © 2007 Al Ahram, © 2007 Al Hayat, © 2007 Al-Quds Al-Arabi, © 2007 An Nahar, © 2007 Assabah, © 2002-2008 The Associated Press, © 2003-2004, 2007-2008 Central News Agency (Taiwan), © 1997, 2004-2007 China Central TV, © 2007 China Military Online, © 2007 Chinanews.com, © 1987-1989 Dow Jones & Company, Inc., © 2007 Guangming Daily, © 1995, 2003, 2007-2008 Los Angeles Times-Washington Post News Service, Inc., © 2002, 2004-2005, 2007-2008 New York Times, © 1994-1998, 2001-2008 Xinhua News Agency, © 2014, 2017 Language Weaver, Inc., © 2014, 2017 University of Colorado, © 2014, 2017 University of Southern California, © 2003, 2005, 2006, 2007, 2009, 2011, 2013, 2014, 2017, 2020 Trustees of the University of Pennsylvania