SlowGuess/ACL_2020_OCR huggingface dataset:
Elizabeth Salesky $^{G}$  Eleanor Chodroff $^{Y}$  Tiago Pimentel $^{5}$  Matthew Wiesner $^{G}$  Ryan Cotterell $^{3,0}$  Alan W Black $^{4}$  Jason Eisner $^{G}$
<sup>c</sup>Johns Hopkins University <sup>x</sup>University of York  
<sup>3</sup>University of Cambridge <sup>0</sup>ETH Zürich <sup>1</sup>Carnegie Mellon University esalesky@jhu.edu eleanor.chodroff@york.ac.uk
A major hurdle in data-driven research on typology is having sufficient data in many languages to draw meaningful conclusions. We present VoxClamantis v1.0, the first large-scale corpus for phonetic typology, with aligned segments and estimated phoneme-level labels in 690 readings spanning 635 languages, along with acoustic-phonetic measures of vowels and sibilants. Access to such data can greatly facilitate investigation of phonetic typology at a large scale and across many languages. However, it is nontrivial and computationally intensive to obtain such alignments for hundreds of languages, many of which have few to no resources presently available. We describe the methodology to create our corpus, discuss caveats with current methods and their impact on the utility of this data, and illustrate possible research directions through a series of case studies on the 48 highest-quality readings. Our corpus and scripts are publicly available for non-commercial use at https://voxclamantisproject.github.io.
Understanding the range and limits of cross-linguistic variation is fundamental to the scientific study of language. In speech and particularly phonetic typology, this involves exploring potentially universal tendencies that shape sound systems and govern phonetic structure. Such investigation requires access to large amounts of cross-linguistic data. Previous cross-linguistic phonetic studies have been limited to a small number of languages with available data (Disner, 1983; Cho and Ldefoged, 1999), or have relied on previously reported measures from many studies (Whalen and Levitt, 1995; Becker-Kristal, 2010; Gordon and Roettger, 2017; Chodroff et al., 2019).
Figure 1: The 635 languages of our corpus geo-located with mean Mel Cepstral Distortion (MCD) scores.
Existing multilingual speech corpora have similar restrictions, with data too limited for many tasks (Engstrand and Cunningham-Andersson, 1988; Ladefoged and Maddieson, 2007) or approximately 20 to 30 recorded languages (Ardila et al., 2020; Harper, 2011; Schultz, 2002).
The recently developed CMU Wilderness corpus (Black, 2019) constitutes an exception to this rule with over 600 languages. This makes it the largest and most typologically diverse speech corpus to date. In addition to its coverage, the CMU Wilderness corpus is unique in two additional aspects: cleanly recorded, read speech exists for all languages in the corpus, and the same content (modulo translation) exists across all languages.
However, this massively multilingual speech corpus is challenging to work with directly. Copyright, computational restrictions, and sheer size limit its accessibility. Due to copyright restrictions, the audio cannot be directly downloaded with the sentence and phoneme alignments. A researcher would need to download original audio MP3 and text through links to bible.is, then segment these with speech-to-text sentence alignments distributed in Black (2019). For phonetic research, subsequently identifying examples of specific phonetic segments in the audio is also a near-essential
<sup>1</sup>The stability of the links and recording IDs is also questionable. Since the release of Black (2019), many of the links have already changed, along with a few of the IDs. We have begun identifying these discrepancies, and plan to flag these in a future release.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4526-4546 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
step for extracting relevant acoustic-phonetic measurements. Carrying out this derivative step has allowed us to release a stable-access collection of token-level acoustic-phonetic measures to enable further research.
Obtaining such measurements requires several processing steps: estimating pronunciations, aligning them to the text, evaluating alignment quality, and finally, extracting phonetic measures. This work is further complicated by the fact that, for a sizable number of these languages, no linguistic resources currently exist (e.g., language-specific pronunciation lexicons). We adapt speech processing methods based on Black (2019) to accomplish these tasks, though not without noise: in §3.4, we identify three significant caveats when attempting to use our extended corpus for large-scale phonetic studies.
We release a comprehensive set of standoff markup of over 400 million labeled segments of continuous speech. For each segment, we provide an estimated phoneme-level label from the X-SAMPA alphabet, the preceding and following labels, and the start position and duration in the audio. Vowels are supplemented with formant measurements, and sibilants with standard measures of spectral shape.
We present a series of targeted case studies illustrating the utility of our corpus for large-scale phonetic typology. These studies are motivated by potentially universal principles posited to govern phonetic variation: phonetic dispersion and phonetic uniformity. Our studies both replicate known results in the phonetics literature and also present novel findings. Importantly, these studies investigate current methodology as well as questions of interest to phonetic typology at a large scale.
2 Original Speech
The CMU Wilderness corpus (Black, 2019) consists of recorded readings of the New Testament of the Bible in many languages and dialects. Following the New Testament structure, these data are broken into 27 books, each with a variable number of chapters between 1 and 25. Bible chapters contain standardized verses (approximately sentence-level segments); however, the speech is originally split only by chapter. Each chapter
Figure 2: The extraction process for the measurements released in VoxClamantis v1.0.
has an average of 13 minutes of speech for a total of  $\approx 20$  hours of speech and text per language. These recordings are clean, read speech with a sampling rate of  $16\mathrm{kHz}$ . In most languages, they are non-dramatic readings with a single speaker; in some, they are dramatic multi-speaker readings with additive music. $^3$  The release from Black (2019) includes several resources for processing the corpus: scripts to download the original source data from bible.is, 'lexicons' created using grapheme-to-phoneme (G2P) conversion, and scripts to apply their generated sentence alignments, which facilitates downstream language processing tasks, including phoneme alignment.
3 The VoxClamantis v1.0 Corpus
Our VoxClamantis v1.0 corpus is derived from 690 audio readings of the New Testament of the Bible<sup>4</sup> in 635 languages.<sup>5</sup> We mark estimated speech seg-
<sup>3</sup>Information about the recordings available can be found at https://www.faithcomesbyhearing.com/mission/recordings
<sup>4</sup>Nine of the readings from Black (2019) could not be aligned.
<sup>5</sup>We specify number of distinct languages by the number of distinct ISO 639-3 codes, which may not distinguish dialects.
2For some languages, we provide multiple versions of the markup based on different methods of predicting the pronunciation and generating time alignments (§3.1).
ments labeled with phonemic labels, and phonetic measures for the tokens that are vowels or sibilants. The extraction process is diagrammed in Figure 2. In the sections below, we detail our procedures for extracting labeled audio segments and their phonetic measures, in both high- and low-resource languages. We then outline important caveats to keep in mind when using this corpus.
3.1 Extracting Phoneme Alignments
We use a multi-pronged forced alignment strategy to balance broad language coverage (§3.1.1) with utilization of existing high-quality resources (§3.1.2). We assess the quality of our approaches in §3.1.3. We release the stand-off markup for our final alignments as both text files and Praat TextGrids (Boersma and Weenink, 2019).<sup>6</sup>
Using scripts and estimated boundaries from Black (2019), we first download and convert the audio MP3s to waveforms, and cut the audio and text into 'sentences' (hereafter called 'utterances' as they are not necessarily sentences). This step creates shorter-length speech samples to facilitate forced alignment; utterance boundaries do not change through our processing.
To extract labeled segments, we first require pronunciations for each utterance. A pronunciation is predicted from the text alone using some grapheme-to-phoneme (G2P) method. Each word's predicted pronunciation is a sequence of categorical labels, which are 'phoneme-level' in the sense that they are usually intended to distinguish the words of the language. We then align this predicted sequence of 'phonemes' to the corresponding audio.
3.1.1 All Languages
Most of our languages have neither existing pronunciation lexicons nor G2P resources. To provide coverage for all languages, we generate pronunciations using the simple 'universal' G2P system Unitran (Qian et al., 2010, as extended by Black, 2019), which deterministically expands each grapheme to a fixed sequence of phones in the Extended Speech Assessment Methods Phonetic Alphabet (X-SAMPA) (Wells, 1995/2000). This naive process is error-prone for languages with opaque orthographies, as we show in §3.1.3 below and discuss further in §3.4 (Caveat B). Even so, it provides a starting point for exploring low-resource languages: after some manual inspection, a linguist may be
able to correct the labels in a given language by a combination of manual and automatic methods.
For each reading, to align the pronunciation strings to the audio, we fit a generative acoustic model designed for this purpose: specifically, eHMM (Prahallad et al., 2006) as implemented in Festvox (Anumanchipalli et al., 2011) to run full Baum-Welch from a flat start for 15 to 30 iterations until the mean mel cepstral distortion score (see §3.1.3) converges. Baum-Welch does not change the predicted phoneme labels, but obtains a language-specific, reading-specific, contextual (triphone) acoustic model for each phoneme type in the language. We then use Viterbi alignment to identify an audio segment for each phoneme token.
3.1.2 High-Resource Languages
A subset of the languages in our corpus are supported by existing pronunciation resources. Two such resources are Epitran (Mortensen et al., 2018), a G2P tool based on language-specific rules, available in both IPA and X-SAMPA, and WikiPron (Lee et al., 2020), a collection of crowd-sourced pronunciations scraped from Wiktionary. These are mapped from IPA to X-SAMPA for label consistency across our corpus. Epitran covers 29 of our languages (39 readings), while WikiPron's 'phonemic' annotations provide partial coverage of 13 additional languages (18 readings). We use Epitran for languages with regular orthographies where it provides high-quality support, and WikiPron for other languages covered by WikiPron annotations. While Unitran and Epitran provide a single pronunciation for a word from the orthography, WikiPron may include multiple pronunciations. In such cases, Viterbi alignment (see below) chooses the pronunciation of each token that best fits the audio.
For most languages covered by WikiPron, most of our corpus words are out-of-vocabulary, as they do not yet have user-submitted pronunciations on Wiktionary. We train G2P models on WikiPron annotations to provide pronunciations for these words. Specifically, we use the WFST-based tool Phonetisaurus (Novak et al., 2016). Model hyperparameters are tuned on 3 WikiPron languages from SIGMORPHON 2020 (Gorman et al., 2020) (see Appendix C for details). In general, for languages that are not easily supported by Epitran-style G2P rules, training a G2P model on sufficiently many
<sup>7</sup>KikiPron annotations are available at both the phonemic and phonetic level, with a greater number of phonemic annotations, which we use here.
$^{6}$ Corresponding audio will need to be downloaded from source and split by utterance using scripts from Black (2019).
4528
ISO 639-3	tpi	ron	azj	msa	ceb	tur	tgl	spa	ilo	rus	hau	ind	tgk	jav	kaz
# Types	1398	9746	18490	7612	8531	21545	9124	11779	15063	16523	4938	5814	12502	10690	20502
Unitran PER	18.4	21.3	26.9	30.1	30.1	31.2	34.4	34.4	35.0	37.4	37.6	38.8	39.8	49.9	46.8
# Tokens	291k	169k	125k	157k	190k	125k	185k	168k	169k	130k	201k	170k	159k	177k	142k
Weighted PER	20.1	21.3	26.1	31.1	35.9	28.5	40.1	32.6	32.7	36.8	36.7	40.5	38.8	54.1	47.7
ISO 639-3	swe	kmr	som	tir	pol	hae	vie	tha	lao	ben	tel	hin	mar	tam
# Types	8610	8127	14375	22188	18681	15935	2757	23338	31334	8075	23477	7722	17839	31642
Unitran PER	46.9	54.3	54.6	57.8	67.1	67.3	73.8	80.3	89.1	90.0	90.3	95.7	97.8	100.5
# Tokens	165k	176k	156k	121k	141k	164k	211k	26k	36k	173k	124k	191k	159k	139k
Weighted PER	49.5	53.9	56.0	57.4	66.8	64.8	80.6	80.4	89.4	86.2	88.3	91.3	97.8	102.1
Table 1: Phoneme Error Rate (PER) for Unitran treating Epitran as ground-truth. 'Types' and 'Tokens' numbers reflect the number of unique word types and word tokens in each reading. We report PER calculated using word types for calibration with other work, as well as frequency-weighted PER reflecting occurrences in our corpus.
high-quality annotations may be more accurate.
We align the speech with the high-quality labels using a multilingual ASR model (see Wiesner et al., 2019). The model is trained in Kaldi (Povey et al., 2011) on 300 hours of data from the IARPA BABEL corpora (21 languages), a subset of Wall Street Journal (English), the Hub4 Spanish Broadcast news (Spanish), and a subset of the Voxforge corpus (Russian and French). These languages use a shared X-SAMPA phoneme label set which has high coverage of the labels of our corpus.
Our use of a pretrained multilingual model here contrasts with §3.1.1, where we had to train reading-specific acoustic models to deal with the fact that the same Unitran phoneme label may refer to quite different phonemes in different languages (see §3.4). We did not fine-tune our multilingual model to each language, as the cross-lingual ASR performance in previous work (Wiesner et al., 2019) suggests that this model is sufficient for producing phoneme-level alignments.
3.1.3 Quality Measures
Automatically generated phoneme-level labels and alignments inherently have some amount of noise, and this is particularly true for low-resource languages. The noise level is difficult to assess without gold-labeled corpora for either modeling or assessment. However, for the high-resource languages, we can evaluate Unitran against Epitran and WikiPron, pretending that the latter are ground truth. For example, Table 1 shows Unitran's phoneme error rates relative to Epitran. Appendix B gives several more detailed analyses with examples of individual phonemes.
Unitran pronunciations may have acceptable phoneme error rates for languages with transparent orthographies and one-to-one grapheme-tophoneme mappings. Alas, without these conditions they prove to be highly inaccurate.
That said, evaluating Unitran labels against Epitran or WikiPron may be unfair to Unitran, since some discrepancies are arguably not errors but mere differences in annotation granularity. For example, the 'phonemic' annotations in WikiPron are sometimes surprisingly fine-grained: WikiPron frequently uses /t/ in Cebuano where Unitran only uses /t/, though these refer to the same phoneme. These tokens are scored as incorrect. Moreover, there can be simple systematic errors: Unitran always maps grapheme  $< \mathrm{a}>$  to label /a/, but in Tagalog, all such tokens should be /a/. Such errors can often be fixed by remapping the Unitran labels, which in these cases would reduce PER from 30.1 to 6.8 (Cebuano) and from 34.4 to 7.8 (Tagalog). Such rules are not always this straightforward and should be created on a language-specific basis; we encourage rules created for languages outside of current Epitran support to be contributed back to the Epitran project.
For those languages where we train a G2P system on WikiPron, we compute the PER of the G2P system on held-out WikiPron entries treated as ground truth. The results (Appendix C) range from excellent to mediocre.
We care less about the pronunciations themselves than about the segments that we extract by aligning these pronunciations to the audio. For high-resource languages, we can again compare the segments extracted by Unitran to the higher-quality ones extracted with better pronunciations. For each Unitran token, we evaluate its label and temporal boundaries against the high-quality token that is closest in the audio, as measured by the temporal distance between their midpoints (Appendix B).
Finally, the segmentation of speech and text into corresponding utterances is not perfect. We use the utterance alignments generated by Black (2019), in which the text and audio versions of a putative
utterance may have only partial overlap. Indeed, Black (2019) sometimes failed to align the Unitran pronunciation to the audio at all, and discarded these utterances. For each remaining utterance, he assessed the match quality using Mel Cepstral Distortion (MCD)—which is commonly used to evaluate synthesized spoken utterances (Kominek et al., 2008)—between the original audio and a resynthesized version of the audio based on the aligned pronunciation. Each segment's audio was resynthesized given the segment's phoneme label and the preceding and following phonemes, in a way that preserves its duration, using CLUSTER-GEN (Black, 2006) with the same reading-specific eHMM model that we used for alignment. We distribute Black's per-utterance MCD scores with our corpus, and show the average score for each language in Appendix E. In some readings, the MCD scores are consistently poor.
3.2 Phonetic measures
Using the phoneme-level alignments described in §3.1, we automatically extract several standard acoustic-phonetic measures of vowels and sibilant fricatives that correlate with aspects of their articulation and abstract representation.
3.2.1 Vowel measures
Standard phonetic measurements of vowels include the formant frequencies and duration information. Formants are concentrations of acoustic energy at frequencies reflecting resonance points in the vocal tract during vowel production (Ladefoged and Johnson, 2014). The lowest two formants, F1 and F2, are considered diagnostic of vowel category identity and approximate tongue body height (F1) and backness (F2) during vowel production (Figure 3). F3 correlates with finer-grained aspects of vowel production such as rhoticity (/r/-coloring), lip rounding, and nasality (House and Stevens, 1956; Lindblom and Sundberg, 1971; Ladefoged et al., 1978), and F4 with high front vowel distinctions and speaker voice quality (Eek and Meister, 1994). Vowel duration can also signal vowel quality, and denotes lexical differences in many languages.
We extracted formant and duration information from each vowel using Praat (Boersma and Weenink, 2019). The first four formants (F1-F4) were measured at each quartile and decile of the vowel. Formant estimation was performed with the Burg algorithm in Praat with pre-emphasis from  $50\mathrm{Hz}$ , a time window of  $25~\mathrm{ms}$ , a time
Figure 3: Vowel Chart
step of  $6.25\mathrm{ms}$ , a maximum of five formants permitted, and a formant ceiling of  $5000\mathrm{Hz}$  which is the recommended value for a male vocal tract (Boersma and Weenink, 2019). Note that the speakers in this corpus are predominantly male.
3.2.2 Sibilant measures
Standard phonetic measurements of sibilant fricatives such as /s/, /z/, /ʃ/, and /ʒ/ include measures of spectral shape, and also segment duration. Measures of spectral shape frequently distinguish sibilant place of articulation: higher concentrations of energy generally reflect more anterior constriction locations (e.g., /s z/ are produced closer to the teeth than /ʃ ʒ/). Segment duration can also signal contrasts in voicing status (Jongman et al., 2000).
Our release contains the segment duration, spectral peak, the spectral moments of the frequency distribution (center of gravity: COG, variance, skewness, and kurtosis), as well as two measures of the mid-frequency peak determined by sibilant quality. These are the mid-frequency peak between 3000 and  $7000\mathrm{Hz}$  for alveolar sibilants, and between 2000 and  $6000\mathrm{Hz}$  for post-alveolar sibilants (Koenig et al., 2013; Shadle et al., 2016). The spectral information was obtained via multitaper spectral analysis (Rahim and Burr, 2017), with a time-bandwidth parameter  $(nw)$  of 4 and 8 tapers  $(k)$  over the middle  $50\%$  of the fricative (Blacklock, 2004). Measurements were made using the methods described in Forrest et al. (1988) for spectral moments and Koenig et al. (2013) for spectral peak varieties.
3.3 Computation times
Generating phoneme-level alignments and extracting subsequent phonetic measures takes significant time, computational resources, and domain knowledge. Our release enables the community to use this data directly without these prerequisites. Table 2 shows that the time to extract our resources,
Computation Time
Resource	Per Language	Total Time
Utterance Alignments	30m	14d 13h
Phoneme Alignments	3d 3h 37m	6y 12d 16h
Vowel Measures	45m	21d 20h
Sibilant Measures	20m	9d 17h
3d 5h 0m	6y 58d 19h
Table 2: Computation time to generate the full corpus.
once methods have been developed, was more than 6 CPU years, primarily for training eHMM models.
3.4 General caveats
We caution that our labeling and alignment of the corpus contains errors. In particular, it is difficult to responsibly draw firm linguistic conclusions from the Unitran-based segments (§3.1.1). In §5 we suggest future work to address these issues.
A Quality of Utterance Pairs: For some utterances, the speech does not correspond completely to the text, due to incorrect cosegmentation. In our phonetic studies, we threshold using reading-level MCD as a heuristic for overall alignment quality, and further threshold remaining readings using utterance-level MCD. We recommend others do so as well.
B Phoneme Label Consistency and Accuracy: Phoneme-level labels are predicted from text without the aid of audio using G2P methods. This may lead to systematic errors. In particular, Unitran relies on a 'universal' table that maps grapheme  $\langle s \rangle$  (for example) to phoneme /s/ in every context and every language. This is problematic for languages that use  $\langle s \rangle$  in some or all contexts to refer to other phonemes such as /ʃ/ or /ʃ/, or use digraphs that contain  $\langle s \rangle$ , such as  $\langle sh \rangle$  for /ʃ/. Thus, the predicted label /s/ may not consistently refer to the same phoneme within a language, nor to phonetically similar phonemes across languages. Even WikiPron annotations are user-submitted and may not be internally consistent (e.g., some words use /d 3/ or /t/ while others use /d5/ or /t/), nor comparable across languages.
'Phoneme' inventories for Unitran and WikiPron have been implicitly chosen by whoever designed the language's orthography or its WikiPron pages; while this may reflect a reasonable folk phonology, it may not correspond to the inventory of underlying or surface phonemes that any linguist would be likely to posit.
C Label and Alignment Assessment: While alignment quality for languages with Epitran and WikiPron can be assessed and calibrated beyond this corpus, it cannot for those languages with only Unitran alignments; the error rate on languages without resources to evaluate PER is unknown to us. The Unitran alignments should be treated as a first-pass alignment which may still be useful for a researcher who is willing to perform quality control and correction of the alignments using automatic or manual procedures. Our automatically-generated alignment offers an initial label and placement of the boundaries that would hopefully facilitate downstream analysis.