cometadata/datacite-titles-descriptions-related-identifiers huggingface dataset:
DOI: 10.35111/6j62-na92
title: ISI Chinese-English Automatically Extracted Parallel Text

description: <h3>Introduction</h3><br>
<p>This file contains documentation for ISI Chinese-English Automatically Extracted Parallel Text, Linguistic Data Consortium (LDC) catalog number LDC2007T09 and isbn 1-58563-422-0.</p><br>
<p>This distribution contains a corpus of Chinese-English parallel sentences, which were extracted automatically from two monolingual corpora: Chinese Gigaword Second Edition (<a href="http://catalog.ldc.upenn.edu/LDC2005T14">LDC2005T14</a>) and English Gigaword Second Edition (<a href="../../../LDC2005T12">LDC2005T12</a>). The data was extracted from news articles published by Xinhua News Agency and was obtained using the automatic parallel sentence identification method described in the following publication: Dragos Stefan Munteanu, Daniel Marcu, 2005. Improving Machine Translation Performance by Exploiting Non-parallel Corpora, Computational Linguistics, 31(4):477-504</p><br>
<p>The corpus contains 558,567 sentence pairs the word count on the English side is approximately 16M words. The sentences in the parallel corpus preserve the form and encoding of the texts in the original Gigaword corpora.</p><br>
<p>For each sentence pair in the corpus the authors provide the names of the documents from which the two sentences were extracted, as well as a confidence score (between 0.5 and 1.0), which is indicative of their degree of parallelism. The parallel sentence identification approach is designed to judge sentence pairs in isolation from their contexts, and can therefore find parallel sentences within document pairs which are not parallel. The fact that two documents share several parallel sentences does not necessarily mean the documents are parallel</p><br>
<p>In order to make this resource useful for research in Machine Translation (MT), the authors made efforts to detect potential overlaps between this data and the standard test and development data sets used by the MT community. The NIST 2002-2005 MT evaluation data sets contain several articles from Xinhua News Agency. Sentence pairs in this distribution that have a 7-gram overlap with a sentence pair in a NIST MT evaluation set or sentence pairs coming from documents whose names are similar to those in the NIST MT sets are marked with a negative confidence score.</p><br>
<h3>Samples</h3><br>
<p>Please view the following samples:</p><br>
<ul><br>
<li><a href="desc/addenda/LDC2007T09_chi.jpg" rel="nofollow">Chinese Sample</a></li><br>
<li><a href="desc/addenda/LDC2007T09_eng.jpg" rel="nofollow">English Sample</a></li><br>
<li><a href="desc/addenda/LDC2007T09_para.jpg" rel="nofollow">Parallel Sample</a></li><br>
</ul></br>
Portions © 1990-2004 Xinhua News Agency, © 2005, 2007 Trustees of the University of Pennsylvania
DOI: 10.35111/7m2c-v978
title: MITRE 1997 Mandarin Broadcast News Speech Translations (HUB-4NE)

description: <h3>Introduction</h3><br>
<p>MITRE 1997 Mandarin Broadcast News Transcripts Translations (HUB-4NE) was developed by The MITRE Corporation and contains segment-aligned English translations of the 1997 DARPA HUB4-NE Mandarin transcripts. The original transcripts and the corresponding broadcast news audio are available as separate LDC publications, <a href="http://catalog.ldc.upenn.edu/LDC98T24" rel="nofollow">1997 Mandarin Broadcast News Transcripts (HUB4-NE) (LDC98T24)</a> and <a href="http://catalog.ldc.upenn.edu/LDC98S73" rel="nofollow">1997 Mandarin Broadcast News Speech (HUB4-NE) (LDC98S73)</a>.</p><br>
<p>The source data is comprised of 30 hours of recorded Mandarin broadcasts collected by the LDC in 1997 from Voice of America, China Central TV and KAZN-AM, a commercial radio station based in Los Angeles, CA. The original transcript segmentation is suitable for speech recognition, but does not support machine translation and machine translation evaluation. Therefore, the Mandarin side of these aligned transcripts was resegmented for this release. In all other respects, the Mandarin transcripts in this publication are identical to the original transcripts.</p><br>
<p>The dataset in this release consists of 376K words of English text and 517K characters of Mandarin text. The English text was produced by translators with no access to the original audio. The translators were given specific guidelines for translation, and those are included in this distribution. A portion of the source data (6%) was translated four times in order to support experiments in translation evaluation.</p><br>
<h3>Samples</h3><br>
<p>For an example of the data in this corpus, please examine this <a href="desc/addenda/LDC2007T19.jpg" rel="nofollow">image of the text</a>.</p><br>
<h3>Additional Licensing Instructions</h3><br>
<p>This 'members-only' corpora is available to current members who can request the data at the listed reduced-license fee. Contact <a href="mailto:ldc@ldc.upenn.edu">ldc@ldc.upenn.edu</a> for information about becoming a member.</p></br>
Portions © 1997 China Central TV, © 2006 The MITRE Corporation, © 1997 MultiCultural Broadcasting Corporation, © 1997, 1998, 2007 Trustees of the University of Pennsylvania
DOI: 10.35111/kpwv-xz91
title: GALE Phase 1 Chinese Broadcast News Parallel Text - Part 1

description: <h3>Introduction</h3> <p>This release is Part 1 of the three-part GALE Phase 1 Chinese Broadcast News Parallel Text, which, along with other corpora, was used as training data in year 1 (Phase 1) of the DARPA-funded GALE program. This corpus contains transcripts and English translations of 23.3 hours of Chinese broadcast news programming. As indicated below, a small number of audio files corresponding to the text in this corpus have been previously released. </p> <h3>Source Data</h3> <p>A total of 23.3 hours of Chinese broadcast news programming was selected from two sources, China Central TV (CCTV) (a broadcaster from Mainland China) and Phoenix TV (a Hong Kong-based satellite TV station). The transcripts and translations represent recordings of five different programs.</p> <p>A manual selection procedure was used to choose data appropriate for the GALE program, namely, news programs focusing on current events. Stories on topics such as sports, entertainment news, and stock markets were excluded from the data set. The following table is a summary of the files included in this release. </p> <h3>Samples</h3> <ul> <li><a href="./desc/addenda/LDC2007T23_source.jpg" rel="nofollow">Jpeg screen capture of the Chinese source text</a></li> <li> <a href="./desc/addenda/LDC2007T23_source.txt" rel="nofollow">Original Chinese source text</a>. </li> <li><a href="./desc/addenda/LDC2007T23_trans.txt" rel="nofollow">English Translation</a></li> </ul><h3>Sponsorship</h3> <p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p>  </br>
Portions © 1997, 2004, 2005 China Central TV, © 2005 Phoenix TV, © 2005-2007 Trustees of the University of Pennsylvania
DOI: 10.35111/bfb8-gt03
title: Chinese Treebank 6.0

description: <h3>Introduction</h3><br>
<p>This file contains documentation for Chinese Treebank 6.0, Linguistic Data Consortium (LDC) catalog number LDC2007T36 and isbn 1-58563-450-6.</p><br>
<p>The Chinese Treebank project began at the University of Pennsylvania in 1998 and continues at Penn and the University of Colorado. Chinese Treebank 6.0 is the latest version produced from this effort, consisting of 780,000 words (over 1.28 million Chinese characters) that are segmented, part-of-speech tagged and fully bracketed. The data sources include newswire from Xinhua News Agency, articles from Sinorama Magazine, news from the website of the Hong Kong Special Administrative Region and transcripts from various broadcast news programs.</p><br>
<p>The LDC published Chinese Treebank 1.0 in 2000; it was later corrected and released in 2001 as <a href="http://catalog.ldc.upenn.edu/LDC2001T11" rel="nofollow">Chinese Treebank 2.0 (LDC2001T11)</a> and consisted of approximately 100,000 words. The LDC released <a href="http://catalog.ldc.upenn.edu/LDC2004T05" rel="nofollow">Chinese Treebank 4.0 (LDC2004T05)</a>, an updated version containing roughly 400,000 words, in 2004. A year later, the LDC published the 500,000 word <a href="http://catalog.ldc.upenn.edu/LDC2005T01" rel="nofollow">Chinese Treebank 5.0 (LDC2005T01)</a>.</p><br>
<p>For information about Chinese Treebank methodology and guidelines, consult the attached documentation files and the <a href="https://www.cs.brandeis.edu/~llc/page2/page2.html">Chinese Treebank Project</a> website.</p><br>
<p>This release encompasses 2,036 text files, containing 28,295 sentences, 781,351 words and 1,285,149 hanzi (Chinese characters). The data is provided in two encodings: GBK and UTF-8, and the annotation has Penn Treebank-style labeled brackets. Details of the annotation standard can be found in the enclosed segmentation, POS-tagging and bracketing guidelines. The data is provided in four different formats: raw text, word segmented, word segmented and POS-tagged, and syntactically bracketed.</p><br>
<h3>Samples</h3><br>
<p>For an example of the data in this publication, please examine this <a href="desc/addenda/LDC2007T36.jpg" rel="nofollow">sample</a> of the bracketed data.</p></br>
Portions © 2000-2001 China Broadcasting System, © 2000-2001 China Central TV, © 2000-2001 China National Radio, © 2000-2001 China Television System, © 1997 The Government of the Hong Kong Special Administrative Region, © 1996-2001 Sinorama Magazine, © 1994-1998 Xinhua News Agency, © 2001, 2004, 2005, 2007 Trustees of the University of Pennsylvania
DOI: 10.35111/w035-1a74
title: Chinese Gigaword Third Edition

description: <h3>Introduction</h3> <p>Chinese Gigaword Third Edition is a comprehensive archive of newswire text data that has been acquired over several years by the LDC. This edition includes all of the contents in <a href="http://catalog.ldc.upenn.edu/LDC2005T14" rel="nofollow"> Chinese Gigaword Second Edition (LDC2005T14)</a> as well as new data collected after the publication of that edition. Also, an archive of articles from a new newswire source (Agence France Presse) has been added in the third edition.</p> <p>The four distinct international sources of Chinese newswire included in this edition are the following: </p><ul> <li>Agence France Presse (afp_cmn) </li> <li>Central News Agency, Taiwan (cna_cmn) </li> <li>Xinhua News Agency (xin_cmn) </li> <li>Zaobao Newspaper (zbn_cmn) </li> </ul><p>The seven-letter codes in the parentheses above are used for the directory names and data files for each source, and are also used (in ALL_CAPS) as part of the unique DOC "id" string assigned to each news article.</p> <h3>Data</h3> <p>The original data archives received by the LDC from Agence France Presse, Xinhua News Agency and Zaobao were encoded in GB-2312, whereas those from Central News Agency (CNA) were encoded in Big-5. To avoid the problems and confusion that could result from differences in character-set specifications, all text files in this corpus have been converted to UTF-8 character encoding.</p> <h3>New in the Third Edition</h3> <ul> <li>Over six years worth of articles (October 2000 through December 2006) from Agence France Presse are being released for the first time.</li> <li>Two years worth of new articles (January 2005 through December 2006) have been added to the Xinhua data set.</li> <li>Nearly two years worth of content was added to the CNA data set. There was a gap in the LDC's collection from this source during 2006: no CNA Chinese content was collected between July 27 and December 17 2006, inclusive, so there are no data files for August through November of that year, and the December data file is about half its expected size.</li> <li>A small set of older stories (October through December 1998) have been added from Zaobao; these were previously published by LDC as part of <a href="http://catalog.ldc.upenn.edu/LDC2001T58" rel="nofollow"> TDT3 Multilanguage Text Version 2.0 (LDC2001T58)</a> and are being included in Gigaword for the first time.</li> </ul><h3>Samples</h3> <p>Please examine this <a href="./desc/addenda/LDC2007T38.jpg" rel="nofollow">sample(JPEG)</a> for an example of the data in this corpus.</p> <h3>Sponsorship</h3> <p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p>  </br>
Portions © 2000-2006 Agence France Presse, © 1991-2006 Central News Agency (Taiwan), © 1998, 2000-2003 SPH AsiaOne, Ltd., © 1990-2006 Xinhua News Agency, © 2003, 2005, 2007 Trustees of the University of Pennsylvania
DOI: 10.35111/gptm-4d61
title: CSLU: Portland Cellular Telephone Speech Version 1.3

description: <h3>Introduction</h3> <p>CSLU: Portland Cellular Telephone Speech Version 1.3 was created by the Center for Spoken Language Understanding (CSLU) at OGI School of Science and Engineering, Oregon Health and Science University, Beaverton, Oregon. It consists of cellular telephone speech and corresponding transcripts, specifically, 7,571 utterances from 515 speakers who made calls in the Portland, Oregon area using cellular telephones.</p> <p>Speakers called the CSLU data collection system on cellular telephones, and they were asked to repeat certain phrases and to respond to other prompts. Two prompt protocols were used: an In Vehicle Protocol for speakers calling from inside a vehicle and a Not in Vehicle Protocol for those calling from outside a vehicle. The protocols shared several questions, but each protocol contained distinct queries designed to probe the conditions of the caller's in vehicle/not in vehicle surroundings. Not every caller provided a response to each prompt. </p> <h3>Recording Details</h3> <p>The speeech data was captured digitally from CSLU's T1 connection and saved as 8 khz, 16-bit linear.</p> <h3>Transcriptions</h3> <p>The text transcriptions in this corpus were produced using the non time-aligned word-level conventions described in The CSLU Labeling Guide, which is included in the documentation for this release. CSLU: Portland Cellular Telephone Speech Version 1.3 contains orthographic and phonetic transcriptions of corresponding speech files. Non time-aligned orthographic transcriptions provide quick access to the content of an utterance; they may contain markers for word boundaries to support access and retrieval at the lexical level. Phonetic/phonemic transcriptions represent the phonetic content of an utterance at a given level of detail that is made explicit by the use of diacritics. Phonetic phenomena transcribed includes excessive nasalization, glottalization, frication on a stop, centralization, lateralization, rounding and palatalization.</p> <h3>Samples</h3> <p>For an example of the data in this corpus, please examine the following audio file and transcript. </p><ul> <li><a href="./desc/addenda/LDC2008S01.wav" rel="nofollow">audio(wav)</a></li> <li><a href="./desc/addenda/LDC2008S01.txt" rel="nofollow">transcript</a></li> </ul> </br>
Portions © 1995, 1998, 2000, 2002 Center for Spoken Language Understanding, Oregon Health & Science University, © 2008 Trustees of the University of Pennsylvania
DOI: 10.35111/wj80-ka72
title: CSLU: National Cellular Telephone Speech Release 2.3

description: <h3>Introduction</h3> <p>This file contains documentation for CSLU: Nattional Cellular Telephone Speech Release 2.3, Linguistic Data Consortium (LDC) catalog number LDC2008S02 and isbn 1-58563-467-0.</p> <p>CSLU: National Cellular Telephone Speech Release 2.3 was created by the Center for Spoken Language Understanding (CSLU) at OGI School of Science and Engineering, Oregon Health and Science University, Beaverton, Oregon. It consists of cellular telephone speech and corresponding transcripts, specifically, approximately one minute of speech from 2336 speakers calling from locations throughout the United States. The data collection protocol used for this release is the same protocol used in <a href="http://catalog.ldc.upenn.edu/LDC2008S01" rel="nofollow">CSLU: Portland Cellular Telephone Speech Version 1.3 (LDC2008S01)</a>. </p> <p>Speakers called the CSLU data collection system on cellular telephones, and they were asked a series of questions. Two prompt protocols were used: an In Vehicle Protocol for speakers calling from inside a vehicle and a Not in Vehicle Protocol for those calling from outside a vehicle. The protocols shared several questions, but each protocol contained distinct queries designed to probe the conditions of the caller's in vehicle/not in vehicle surroundings. </p> <h3>Recording Details</h3> <p>The data were collected with the CSLU T1 digital data collection system. The sampling rate was 8khz, and the files were stored in 8 bit mu-law format on a UNIX file system. In this release, the files are provided in 16-bit linearly encoded Windows wav (riff) format.</p> <h3>Transcription</h3> <p>The text transcriptions in this corpus were produced using the non time-aligned word-level conventions described in The CSLU Labeling Guide, which is included in the documentation for this release. CSLU: National Cellular Telephone Speech Release 2.3 contains orthographic and phonetic transcriptions of corresponding speech files. Non time-aligned orthographic transcriptions provide quick access to the content of an utterance; they may contain markers for word boundaries to support access and retrieval at the lexical level. Phonetic/phonemic transcriptions represent the phonetic content of an utterance at a given level of detail that is made explicit by the use of diacritics. Phonetic phenomena transcribed includes excessive nasalization, glottalization, frication on a stop, centralization, lateralization, rounding and palatalization.</p> <h3>Samples</h3> <p>For an example of the data in this corpus, please listen to the following audio samples: </p><ul> <li><a href="./desc/addenda/LDC2008S02_a.wav" rel="nofollow">speaker 1</a></li> <li><a href="./desc/addenda/LDC2008S02_b.wav" rel="nofollow">speaker 2</a></li> </ul> </br>
Portions © 2000, 2002 Center for Spoken Language Understanding, Oregon Health & Science University, © 2008 Trustees of the University of Pennsylvania
DOI: 10.35111/yjwc-zx48
title: West Point Brazilian Portuguese Speech

description: <h3>Introduction</h3><br>
<p>West Point Brazilian Portuguese Speech is a database of digital recordings of approximately 1.6 hours of spoken Brazilian Portuguese designed and collected by staff and faculty of the Department of Foreign Languages (DFL) and Center for Technology Enhanced Language Learning (CTELL) to develop acoustic models for speech recognition systems. The U.S. government uses such systems to provide speech-recognition enhanced language learning courseware to government linguists and students enrolled in various government language programs.</p><br>
<p>The data in this corpus was collected in March 1999 in Brasilia, Brazil using informants from a Brazilian military academy. The corpus consists of read speech from 60 female and 68 male native and non-native speakers.</p><br>
<p>The speech was elicited from a prompt script containing 296 sentences and phrases typically used in language learning situations. The prompts are listed in the file prompts.txt. Each line of this file has two fields separated by a tab: the first field denotes the base name of the waveform file; and the second field denotes the prompt used to record the utterance.</p><br>
<p>A pronouncing dictionary developed by Dr. Sheila Ackerlind with help from cadet Sterling Packer is provided in the file SANTIAGO.txt.</p><br>
<p>The speech was collected using four laptop computers running MS Windows. Three of the computers recorded with a 16 bit data size and sampling rate of 22050 Hz, the other laptop recorded with an 8 bit data size at a sampling rate of 11025 Hz. The recording script presented a visual display of the sentence to be recorded. The informant pressed a key and spoke the sentence. The recording was played back for review, allowing the utterance to be re-recorded. A member of the data collection team was present during the recording session to verify recordings and to provide technical assistance in case of malfunctioning equipment.</p><br>
<h3>Samples</h3><br>
<p>For an example of speech contained in this corpus, please listen to this audio <a href="desc/addenda/LDC2008S04.wav" rel="nofollow">sample</a> (MS Wave format).</p></br>
Portions © 1999, 2004 United States Military Academy, © 2008 Trustees of the University of Pennsylvania
DOI: 10.35111/eayh-nv69
title: CSLU: Alphadigit Version 1.3

description: <h3>Introduction</h3> <p>This file contains documentation for CSLU: Alphadigit Version 1.3 , Linguistic Data Consortium (LDC) catalog number LDC2008S06 and isbn 1-58563-478-6. </p> <p>Alphadigit Version 1.3 is a collection of 78,044 utterances from 3,025 speakers saying six-digit strings of letters and digits over the telephone for a total of approximately 82 hours of speech. Each speech file has corresponding orthographic and phonemic transcriptions. This corpus was created by the Center for Spoken Language Understanding (CSLU), Oregon Health & Science University, Beaverton, Oregon.</p> <h3>Data</h3> <p>Speakers were recruited using USEnet postings. Respondents registered for the collection by completing an online form. Once registered, they received a list of 18-29 six-digit strings (e.g., "a 2 b 4 5 g") and participation instructions. Speakers called the CSLU data collection system by dialing a toll-free number and were prompted for each string; 1102 different strings were used throughout the course of the data collection. The lists were set up to balance for phonetic context between all letter and digit pairs. </p> <p>The data were recorded directly from a digital phone line without digital-to-analog or analog-to-digital conversion at the recording end using the CSLU T1 digital data collection system. The sampling rate was 8khz and the files were stored in 8-bit mu-law format on a UNIX file system. The files have been converted to RIFF standard file format, 16-bit linearly encoded.</p> <h3>Transcription</h3> <p> All of the files included in this corpus have corresponding non-time-aligned word-level transcriptions and time aligned phoneme-level transcriptions (automatic forced alignment) that comply with the conventions in the CSLU Labeling Guide. Non time-aligned orthographic transcriptions provide quick access to the content of an utterance; they may contain markers for word boundaries to support access and retrieval at the lexical level. Phonetic/phonemic transcriptions represent the phonetic content of an utterance at a given level of detail that is made explicit by the use of diacritics. Phonetic phenomena transcribed include excessive nasalization, glottalization, frication on a stop, centralization, lateralization, rounding and palatalization.</p> <h3>Samples</h3> <p>For an example of the speech contained in this corpus, please listen to this <a href="./desc/addenda/LDC2008S06.wav" rel="nofollow">audio sample</a> (MS wave). </p> </br>
Portions © 2000-2002 Center for Spoken Language Understanding, Oregon Health & Science University, © 2008 Trustees of the University of Pennsylvania