Venue: NAACL 2024

# The taste of IPA : Towards open-vocabulary keyword spotting and

forced alignment in any language

Jian Zhu\({}^{\text{\tiny\sf B,e}}\)   Changbing Yang\({}^{\text{\tiny\sf B,ae}}\)   Farhan Samir\({}^{\text{\tiny\sf B,ae}}\)   Jahurul Islam\({}^{\text{\tiny\sf B}}\)

###### Abstract

In this project, we demonstrate that phoneme-based models for speech processing can achieve strong crosslinguistic generalizability to unseen languages. We curated the IpaPack, a massively multilingual speech corpora with phonemic transcriptions, encompassing 115 languages from diverse language families, selectively checked by linguists. Based on the IpApack, we propose Clap-Ipa, a multilingual phoneme-speech contrastive embedding model capable of open-vocabulary matching between arbitrary speech signals and phonemic sequences. The proposed model was tested on 95 unseen languages, showing strong generalizability across languages. Temporal alignments between phonemes and speech signals also emerged from contrastive training, enabling zeroshot forced alignment in unseen languages. We further introduced a neural forced aligner Ipa-Aligner by finetuning Clap-Ipa with the Forward-Sum loss to learn better phone-to-audio alignment. Evaluation results suggest that Ipa-Aligner can generalize to unseen languages without adaptation.

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 750–772

June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

The diversity of human speech presents a formidable challenge to multilingual speech processing systems. Recently, accumulating evidence indicates that scaling up the multilingual data can tremendously improve the performance of multilingual speech processing Conneau et al. (2020); Babu et al. (2021); Radford et al. (2023); Pratap et al. (2023). However, it remains incredibly difficult, if not impossible, to gather large-scale data from every language in the world. It is becoming increasingly critical to develop speech processing systems that generalize to arbitrary unseen languages.

Despite the seeming diversity, sounds of human speech are highly constrained by the anatomical structure of the human vocal tract, which is universally shared by all humans Gick et al. (2013). Typological research has also shown that most, if not all, human speech can be represented by around 150 phonemes and diacritics Moran et al. (2014); Gordon (2016). The limited degrees of freedom in human articulation have enabled phoneticians and linguists to craft universal symbolic representations of human speech, that is, the **International Phonetic Alphabet (IPA)** (International Phonetic Association, 1999).

Prior studies have shown that phoneme-based ASR models generalize to unseen languages Li et al. (2020); Xu et al. (2022); Glocker et al. (2023). In this project, we aim to provide yet another positive answer to this central question: **can we build multilingual speech processing systems that generalize to arbitrary languages through the use of universal IPA symbols?** Specifically, we focus on two classic tasks in speech processing, **key word spotting (KWS)** and **forced alignment**. KWS is a task of identifying specific keywords in streaming speech, whereas forced alignment refers to aligning intervals of a speech signal to a given sequence of phonetic symbols. Both tasks are relevant in many practical applications such as voice assistant, speech synthesis, language documentation, etc. Yet neither task has been tackled with general systems that generalize to all languages.

This study represents an attempt to build crosslingustically generalizable systems for KWS and forced alignment. First, we present the IpaPack, a multilingual speech corpora in 115 languages with phonemic transcriptions, totaling over 1000 hours and carefully checked by trained linguists. Secondly, with the IpaPack, we proposed Contrastive Language-Audio Pretraining with International Phonetic Alphabet (Clap-Ipa), a phoneme-to-speech retrieval model with contrastive pretraining on phoneme-speech pairs. Evaluations on 95 unseen languages suggest that Clap-Ipa is capable of performing zero-shot open-vocabulary KWS in any language without adaption, including lan

guages not seen during training.

Thirdly, we also introduce a multilingual forced alignment model, Ipa-Aligner, that works for arbitrary languages. We noticed that alignments between phonemes and speech signals emerge from Clap-Ipa, even with only sequence-level contrastive learning. Crosslinguistic zero-shot forced alignment can be achieved with Clap-Ipa. After finetuning Clap-Ipa with an alignment loss, we propose Ipa-Aligner that can provide crosslinguistic word-level and phone-level alignment generalizable to unseen languages. Finally, our analysis indicates that phonemes, being shared across all languages, enhance knowledge transfer within training data, serving as more effective modeling units than texts in current multilingual tasks.

We envision that our dataset and models will benefit more downstream tasks and applications in multilingual speech processing. To facilitate future research, we will release our dataset, scripts, and pre-trained models at: [https://github.com/lingjzhu/clap-ipa](https://github.com/lingjzhu/clap-ipa).

## 2 Backgrounds

### Spoken keyword detection and retrieval

Most research in keyword spotting focuses predominantly on English (e.g., Chen et al., 2014; Tang and Lin, 2018; Rybakov et al., 2020; Berg et al., 2021). In recent years, there has been increased interest in building multilingual keyword detection systems that can adapt to new words or new languages through few-shot learning Mazumder et al. (2021); Lei et al. (2023); Reuter et al. (2023). While texts are the primary modeling units in most systems, studies are showing the effectiveness of using IPA symbols to achieve open-vocabulary generalization Tanaka et al. (2001); Shin et al. (2022); Lee and Cho (2023); Reuter et al. (2023).

Another approach for keyword matching is based on contrastive learning frameworks, notably CLAP Wu et al. (2023) and the subsequent CLARA Noriy et al. (2023). Contrastive learning also enables keyword retrieval systems based on semantics rather than the surface acoustic form Duquenne et al. (2021); Khurana et al. (2022); Zhu et al. (2022). The contrastive learning paradigm has also been applied successfully to build open-vocabulary KWS systems Nishu et al. (2023).

Nevertheless, existing multilingual KWS systems face limitations in terms of limited supported languages, and cannot achieve zero-shot adaptation. Built on these prior efforts, we scaled up the phoneme-based open-vocabulary KWS models to more languages to achieve crosslinkingic generalization.

### Forced alignment

Forced alignment is another classic task in speech processing for segmenting speech into utterances, words, or phonemes. It is widely used for downstream tasks where phone or word durations are needed, including speech synthesis, speech assessment, language documentation, and speech corpora construction. Currently, some of the most popular forced alignment systems are still based on Hidden Markov Models (HMM), including the Montreal Forced Aligner (MFA) McAuliffe et al. (2017), WebMAUS Risler et al. (2012) and Forced Alignment and Vowel Extraction (FAVE) Rosenfelder et al. (2011). Recently, since neural networks gradually dominate speech processing, research in performing forced alignment with deep neural models is also gaining momentum Kelley and Tucker (2018); Kurzinger et al. (2020); Schulze-Forster et al. (2020); Teytaut and Roebel (2021); Teytaut et al. (2022); Zhu et al. (2022). Neural models usually exhibit stronger performance over HMM-based systems. However, forced alignment systems are mostly set up to work in monolingual settings. Scant attention has been paid to the building of multilingual forced alignment systems that can work for multilingual languages simultaneously.

## 3 Dataset curation

Most speech corpora are distributed as audio-text pairs. In comparison, phonemically transcribed speech corpora are rare. Unlike text transcription, transcribing speech signals into IPA, or phonemic transcriptions often require years of expertise in phonetics, making it hard to create high-quality phonemic datasets at scale. However, these IPA symbols provide a universal representation of speech sounds such that any language can be transcribed symbolically. So IPA symbols can be used as a proxy to train multilingual speech processing systems. As a first step, we created large-scale phonemic transcriptions for public speech corpora, encompassing 115 languages across language families. The transcription can be automated through **grapheme-to-phoneme conversion** (**G2P**), a process of converting orthographic transcriptions into phonemic transcriptions through pronunciation dictionaries and/or statistical models 



### Phonemic transcriptions

We primarily made use of three existing multilingual speech datasets, FLEURS (Conneau et al., 2023), Multilingual Spoken Words Corpus (MSWC) (Mazumder et al., 2021b) and DoReCo Paschen et al. (2020).

**FLEURS** We used two multilingual G2P systems, Epitran Mortensen et al. (2018) and CharsiuG2P Zhu et al. (2022b), to create phonemic transcriptions. As these two systems cover an overlapping but slightly different set of languages, combining them allowed us to maximize the diversity of languages. Before preprocessing, we removed any texts with Arabic numbers or code-switching, as G2P systems cannot process them correctly.

Yet some Asian languages do not explicitly mark word boundaries with spaces. For Mandarin Chinese, G2PW Chen et al. (2022) was used to create the Pinyin romanizations, which were then mapped to IPA symbols. For Thai, we used PyThaiNLP Phathiyaphaibun et al. (2016) to perform word segmentation and G2P. For Japanese, the word segmentation was first performed with Fugashi McCann (2020) before G2P was applied.

**MSWC** As MSWC is a word-level speech corpus, creating phonemic transcriptions was straightforward. CharsiuG2P and Epitran were deployed to transcribe the orthographic words to phonemic sequences. To strike a balance between diversity and quantity, we limited the maximum frequency to 50 to prevent high-frequency words from dominating the dataset. For words with more than 50 samples, only 50 of them will be randomly selected from the pool. After filtering, we ended up with 2.3 million spoken words, amounting to around 613 hours.

**DoReCo** The original DoReCo data were distributed as hour-long recordings, so we segmented them into individual utterances based on the sentence boundaries in the provided annotations. For DoReCo, all languages were transcribed as phonemes using X-SAMPA Wells (1995) notations. We simply converted the X-SAMPA transcription to IPA symbols, as there is a one-to-one mapping between these two systems. Utterances with incomplete transcriptions or loud background noises were discarded.

### Dataset validation

As G2P systems are based on rules or pronunciation dictionaries, they reflect how a word **should** be pronounced rather than how a word **is** pronounced. Given the high variability (e.g., phonetic reduction, coarticulation) in speech signals, it is not always possible for the G2P phonemic transcriptions to exactly match the audio. We were aware that a true transcription does not always exist for every utterance Ladefoged and Halle (1988); Ladefoged (1990). Even trained phoneticians often disagree on the phonemic transcriptions of the same utterance, due to factors including psycho-acoustic constraints, phonetic training, and their mother tongue Pitt et al. (2005); Heselwood (2013).

Two authors (trained phoneticians) listened to at least ten random samples in each language to determine the transcription quality. We applied a relatively relaxed standard for the generated transcriptions: as long as the speech signal approximately matches more than 80% of the transcription, it is considered valid. While we made our best efforts to validate the transcription quality, we acknowledge that there are still transcription errors in the dataset. A summary of the IpaPack is presented in Table 1. To augment our current dataset, we also included a filtered subset of VoxCommuis Corpus Ahn and Chodroff (2022), which is a multilingual speech corpora created in a similar workflow, though with slightly different pronunciation dictionaries and G2P tools. Detailed information on individual languages of the VoxCommuis Corpus is at Appendix A

## 4 Method

### Contrastive learning for KWS

Here we adopt the same contrastive learning framework as CLAP Wu et al. (2023), as it has been

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & Train (hrs) & Dev (hrs) & Test (hrs) & Total (hrs) & Languages & Avg. Dur (hrs) \\ \hline VoxCommunis Ahn and Chodroff (2022) & 803.84 & - & - & 803.84 & 38 & 21.15 \\ \hline IpaPack & & & & & & \\ \hline Fleurs-Ipa & 544.02 & 73.46 & 162.06 & 779.54 & 77 & 10.12 \\ Mswc-Ipa & 485.35 & 64.08 & 64.11 & 613.44 & 36 & 17.04 \\ Doreco-Ipa & 13.70 & - & 5.29 & 18.99 & 44 & 0.44 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Descriptive statistics of the IpaPack and a selected subset of VoxCommunis Ahn and Chodroff (2022).



proven to be one of the most effective strategies for learning high-quality cross-modal representations. There are two separate encoders to process phoneme sequence \({\bf P}\in\mathbb{R}^{N\times 1}\) and speech MFCC features \({\bf S}\in\mathbb{R}^{T\times K}\), transforming them into phoneme embedding and speech embedding. In this study, we use the **SigLIP loss**, a simpler sigmoid-based loss that is shown to be as effective as the softmax-based CLIP loss [23]. Given two normalized embeddings \(\mathbf{x}_{i}\in\mathbb{R}^{D}=f_{S}({\bf P}_{i})\) and \(\mathbf{y}_{i}\in\mathbb{R}^{D}=f_{T}({\bf S}_{i})\), it is defined as follows.

\[\mathcal{L}=-\frac{1}{|\mathcal{B}|}\sum_{i=1}^{|\mathcal{B}|}\sum_{j=1}^{| \mathcal{B}|}\underbrace{\log\frac{1}{1+e^{z_{ij}(-t{\bf x}_{i}\cdot{\bf y}_{ j}+b)}}}_{\mathcal{L}_{ij}} \tag{1}\]

where \(t\) and \(b\) are learnable parameters that were updated during training. \(z_{ij}\) is the ground truth label, \(z_{ij}=1\) for positive pairs and \(z_{ij}=-1\) for negative pairs. Following the recommendation by zhai:2023:EMNLP, we initialized \(t=\log 10\) and \(b=-10\).

Speech encoderThe speech encoder has the same transformer encoder architecture as the Whisper's encoder. The weights were initialized with Whisper's pre-trained encoder weights, whereas the decoder was discarded. The original Whisper encoder does not accept attention masks, but these padding tokens can bias the model during pooling. So attention masks were also passed to the speech encoder and the final fixed-dimensional embedding through mean pooling on non-padded hidden states. For speech data augmentation, SpecAugment [54] was applied during training using the default hyperparameters in Whisper training.

Phoneme tokenizerWe trained a specialized tokenizer to encode all base IPA symbols and diacritics, including tonal notations, stress marks, and tie bars for affricates. Upon inspection, we noticed that the IPA transcriptions were inconsistent across languages. For example, tie bars were inconsistently labeled (e.g., [t] vs. [t]) and stress marks tend to be a language-specific phenomenon [67]. Yet we did not perform normalization on these idiosyncratic labels to preserve the diversity of our data. The phoneme tokenizer was trained using the unigram algorithm [8] with sentencepiece package 1. The tokenizer was trained on all phonemic transcriptions in our datasets, with a vocabulary of 450 and byte-fallback for unknown characters.

Footnote 1: [https://github.com/google/sentencepiece](https://github.com/google/sentencepiece)

Phoneme encoderFor the phoneme encoder, we used the BERT architecture [9] with mean pooling of the final hidden states as the fixed-dimensional representation. The phoneme encoder was pre-trained on a corpus of phonemic transcriptions using standard masked language modeling (MLM) as detailed in [9]. Given that phoneme sequences are of less complexity than texts, the masking probability was set to 30%. The training data were pooled from diverse sources, including the IpaPack, pronunciation dictionaries in CharsiuG2P [23], and Vox Communis [1]. The final pretraining corpus consists of 11 million samples in more than 110 languages. We pre-trained three phoneme encoders of different sizes, matching hyperparameters including the number of layers, hidden dimensions, and the number of attention heads to the corresponding Whisper encoder (tiny, base and small).

### Forecd alignment

We noticed that phoneme-to-speech alignment emerged from Clap-Ipa on the pairwise cosine similarity matrix computed with the token-wise hidden states of phone and speech encoders. We introduce a simple algorithm to derive the alignment between phonetic units and speech signals, with control over the temporal resolution of speech frames and the granularity of phonetic sequences.

Adaptive average poolingWhile we expect the forced aligned units to be natural phonetic units like phonemes and words, due to tokenization, the hidden states of phone encoders correspond to a character or byte unit rather than a natural phonetic unit. A sequence of phonemes or words of length \(N^{\prime}\) might be tokenized into a character or byte sequence of length \(N,N\geq N^{\prime}\). We define an adaptive average-pooling mask \({\bf M}_{p}\in\mathbb{R}^{N^{\prime}\times N}\) to downsample the hidden representations. Through this pooling mask, consecutive hidden states belonging to one phoneme or one word were averaged to one fixed dimensional vector, such that each output hidden state after pooling corresponds to a natural phonetic unit (see Fig 1). This ensures that our forced alignment algorithm works for any level of phonetic units.

We can also define a similar adaptive average 

pooling mask for speech representations \({\bf M}_{s}\in\mathbb{R}^{T^{\prime}\times T}\) to downsample them from length \(T\) to \(T^{\prime}\). For word-level alignments that don't require high temporal resolution, we can compress the length of the speech hidden states by controlling the pooling window length and frameshift.

**Zeroshot forced alignment** Given two sequences of hidden states \({\bf H}_{s}\in\mathbb{R}^{T\times D}\) and \({\bf H}_{p}\in\mathbb{R}^{N\times D}\) produced by the speech encoder and phone encoders, adaptive average-pooling masks \({\bf M}_{p}\in\mathbb{R}^{N^{\prime}\times N}\) and \({\bf M}_{s}\in\mathbb{R}^{T^{\prime}\times T}\) are used to transform them into more compact representations \({\bf H}^{\prime}_{s}\in\mathbb{R}^{T^{\prime}\times D}\) and \({\bf H}^{\prime}_{\mathbf{P}}\in\mathbb{R}^{N^{\prime}\times D}\).

\[{\bf H}^{\prime}_{s}=\text{Normalize}({\bf M}_{s}{\bf H}_{s}, \text{dim=-1})\] \[{\bf H}^{\prime}_{\mathbf{p}}=\text{Normalize}({\bf M}_{\mathbf{ p}}{\bf H}_{\mathbf{p}},\text{dim=-1})\] \[{\bf D}={\bf H}^{\prime}_{s}{\bf H}^{\prime\top}_{\mathbf{p}}/\tau\]

where \(\tau\) is the fixed temperature parameter and was set to 0.05 by default. The pairwise similarity matrix \({\bf D}\in\mathbb{R}^{T^{\prime}\times N^{\prime}}\) is used to derive the temporal monotonic alignment between phonetic units and speech frames through **dynamic time warping (DTW)**, even if Clap-Ipa had never between trained on alignment labels.

**Finetuning** To further enhance the performance of forced alignment, we introduce Ipa-Aligner by finetuning Clap-Ipa with the **Forward-Sum Loss**, which has been shown to be effective in learning monotonic alignments between speech and phonemes Shih et al. (2021); Badlani et al. (2022); Zhu et al. (2022c).

\[\mathcal{L}=\mathcal{L}_{ForwardSum}({\bf D})\]

This alignment learning loss function relies on the forward-sum algorithm in classic HMMs to maximize the likelihood of text sequence given speech sequences, while enforcing the monotonic constraint of alignment see Shih et al. (2021) for detailed derivations). The Forward-Sum loss requires a good prior alignment to converge to meaningful results, so we did not report failure results from randomly initialized models.

During finetuning, we only average-pooled the phoneme representations at the phoneme and kept the original speech representations (by setting \({\bf M}_{s}\) to the identity matrix \({\bf I}\)). In inference, for phoneme alignment, we pooled the phoneme representations at the phoneme-level and kept the original speech representations. For word alignment, the phoneme representations were pooled at the word-level and the speech representations were average-pooled with a window length of 3 and a frameshift of 2.

## 5 Experiments

### Training details

We trained three variants of models, Clap-Ipa-tiny, Clap-Ipa-base and Clap-Ipa-small, all of them were matched to the default encoder parameters of Whisper Radford et al. (2023). The speech encoder and phoneme encoder were symmetric. Our training dataset included the training set of IPaPack plus the VoxCommunis speech corpora Ahn and Chodroff (2022). By default, all models were trained with paired speech recordings and their phonemic transcriptions. For IPa-Aligner, we finetuned Clap-Ipa-tiny, Clap-Ipa-base and Clap-Ipa-small on the same data excluding Mswc-Ipa. All detailed hyperparameters can be found in Appendix B.

For controlled comparison, we also trained two base models, Clap-Ipa-Text and Clap-Ipa-Phone on the same Fleurs-Ipa andMswc-Ipa subset either with only phonemic or text transcriptions. These two models were matched in total parameters, training data, and all other hyperparameters during training. In another controlled experiment, we trained Clap-Ipa-Fleurs and Clap-Ipa-Vc either only on the Fleurs-Ipa or the VoxCommunis, which would allow us to examine the impact of data size and language diversity.

### Evaluation datasets

We evaluated the crosslinguistic generalizability of our models on several evaluation datasets covering a wide range of topologically diverse languages. Whenever possible, we made our best effort to include baseline models to contextualize our model performance. This was not always possible, because evaluating multilingual KWS and multilingual forced alignment on unseen languages are new tasks and in some cases we were not able to find 

open-source models for comparison. However, we hope that our models and results will become a baseline that spur more future research in this direction.

**Libriphrase** To compare with existing models, we first tested on a popular English KWS dataset, Libriphrase (Shin et al., 2022), as an out-of-domain evaluation dataset, since our models were not trained on their training sets. We used **Equal Error Rate (EER)** and the **Area under Curve (AUC)** scores to compare model performance, consistent with prior studies.

**Unseen languages** We also evaluated all models on five unseen languages with typological diversity from Fleurs-Ipa and Mswc-Ipa. We isolated five language from Mswc-Ipa and Fleurs-Ipa, namely, Vietnamese (vie), Tamil (tam), Hausa (hau), Georgian (geo) and Odia (ori). For Fleurs-Ipa, the test sets of these five languages were directly used. However, for Mswc-Ipa, due to data scarcity, we pooled all training, validation, and tests of these five languages together to form a larger and more challenging benchmark. We further evaluated 95 (81 unseen) languages from the UCLA phonetic Corpus Li et al. (2021) and 14 unseen languages from Dreco-Ipa. **Hit@1** and **Mean Average Precision (mAP)** were used to measure the cross-linguistic retrieval performance of all models. To avoid duplication, we only reported results on phoneme-to-speech retrieval, as the results of speech-to-phoneme and speech-to-speech retrieval were in the same range.

**Word and phoneme boundaries** To evaluate the performance of forced alignment, we made use of **F1** and **R-Value**, which were used in prior studies Rasanen et al. (2009); Kreuk et al. (2020); Zhu et al. (2022). If the predicted boundary is within the tolerance interval of the true boundary, it is considered a hit, otherwise a miss. Since each boundary marked the onset and the offset of consecutive phones, we only evaluated the phone onsets with a tolerance of 20ms and word onsets with a tolerance of 100ms. We used TIMIT Garofolo et al. (1993) as the English benchmark. Doreco-Ipa also contains phoneme-level and word-level alignments, so we partitioned the Doreco-Ipa into seen and unseen evaluation sets. Yet Ipa-Aligner was never trained on any segmentation labels.

## 6 Results

In this section, we summarize the main results for KWS and forced alignment.

**KWS** Evaluation results in Table 2 suggests that Clap-Ipa performs on par with the state-of

\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{Method} & \multicolumn{2}{c}{LibriPhrase-Easy} & \multicolumn{2}{c}{LibriPhrase-Hard} \\ \cline{2-5}  & EER(\%) \(\downarrow\) & AUC(\%) \(\uparrow\) & EER(\%) \(\downarrow\) & AUC(\%) \(\uparrow\) \\ \hline CMCD Shin et al. (2022) & 8.42 & 96.7 & 32.90 & 73.58 \\ PhonMatchNet Lee and Cho (2023) & 2.80 & 99.29 & 18.82 & 88.52 \\ CED Nishu et al. (2023) & 1.7 & 99.84 & **14.4** & **92.7** \\ \hline Clap-Ipa-Text & 6.0 & 98.31 & 31.14 & 74.8 \\ Clap-Ipa-Phone & 1.3 & 99.88 & 23.03 & 84.58 \\ \hline Clap-Ipa-Fleurs & 0.95 & 99.94 & 22.98 & 84.82 \\ Clap-Ipa-Vc & 0.81 & 99.55 & 21.55 & 85.91 \\ \hline Clap-Ipa-tiny & 0.68 & 99.96 & 20.85 & 86.58 \\ Clap-Ipa-base & 0.63 & 99.97 & 20.04 & 88.25 \\ Clap-Ipa-small & **0.56** & **99.97** & 18.62 & 88.82 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Evaluation results on the English-only Libriphrase.

\begin{table}
\begin{tabular}{l l l l l l l l l} \hline \hline \multirow{2}{*}{Model} & \multicolumn{2}{c}{Mswc-Ipa} & \multicolumn{2}{c}{Fleurs-Ipa} & \multicolumn{2}{c}{UelaPhoreticCorpus} & \multicolumn{2}{c}{Doreco-Ipa} \\ \cline{2-9}  & Hit@1 \(\uparrow\) & mAP \(\uparrow\) & Hit@1 \(\uparrow\) & mAP \(\uparrow\) & Hit@1 \(\uparrow\) & mAP \(\uparrow\) & Hit@1 \(\uparrow\) & mAP \(\uparrow\) \\ \hline Clap-Ipa-Text & 13.51 & 12.7 & 8.48 & 10.22 & - & - & - & - \\ Clap-Ipa-Phone & 79.28 & 68.74 & 86.4 & 87.4 & - & - & - & - \\ \hline Clap-Ipa-Fleurs & 83.48 & 77.16 & 98.59 & 98.53 & 51.57 & 62.53 & 73.91 & 79.32 \\ Clap-Ipa-Vc & 84.38 & 75.64 & 63.52 & 63.21 & 50.05 & 61.41 & 90.56 & 93.01 \\ \hline Clap-Ipa-tiny & 82.58 & 76.32 & 98.85 & 98.86 & 51.71 & 62.62 & 95.46 & 96.84 \\ Clap-Ipa-base & **82.60** & **77.31** & **99.20** & **99.27** & 52.17 & 63.90 & **96.54** & **97.77** \\ Clap-Ipa-small & 81.98 & 75.35 & 97.61 & 97.98 & **55.05** & **65.93** & 91.46 & 94.41 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Evaluation results on unseen languages.

 

the-art models on Libriphrase-Easy, while not trained on the Libriphrase training set. Yet ClapIpa failed to outperform state-of-the-art CED [20] in LibriPhrase-Hard, suggesting that language-specific finetuning is still necessary to maximize performance. Generally speaking, phoneme-based models are more effective than text-based models.

For unseen languages, Table 3 indicates that phoneme-based models do generalize successfully to unseen languages across datasets. In contrast, the text-based model performs poorly in unseen languages, suggesting that orthographic texts are not very useful for crosslinguistic speech processing. Utterance-level retrieval appears to be much easier than word-level retrieval, a pattern quite consistent across datasets. Model size correlates with performance in seen languages but not with crosslinkingistic generalizability.

**Foreced alignment** While not trained on forced alignment explicitly, Clap-Ipa shows some capabilities for crosslinkingistic forced alignment even in zero-shot predictions on both seen and unseen languages (see Table 4 and Table 5). After fine-tuned with the ForwardSum loss, Ipa-Aligner can perform competitively in English with some widely used HMM-based forced aligners, even though TIMIT was not part of its training dataset. For low-resource languages, Ipa-Aligner also achieves good performance, regardless of whether the language has been seen during training or not.

## 7 Discussions

In this section, we provide more in-depth answers to our research questions with the major findings of our experiments.

**Can phoneme-based models generalize cross-linguistically?** The evaluation results for Clap-Ipa and IPA-Aligner in Table 3 and Table 5 indicate that phoneme-based model exhibits strong generalization capabilities cross-linguistically in both KWS and forced alignment, even to unseen languages in zero-shot predictions.

Generally speaking, all Clap-Ipa models perform better on utterance-level datasets [16] than on word-level datasets [18], because the longer the phoneme sequence, the more likely that it is distinct in a pool of candidates. For utterance-level datasets, Clap-Ipa models achieve near-perfect scores on unseen languages (see Table 3), indicating that phonemic representations do enable cross-linguistic generalization.

Table 6 shows that the similarity assigned by Clap-Ipa-small was highly consistent with human perception. The top-ranked crosslinkingistic candidates were extremely similar in articulatory features and syllable structure to the query.

For forced alignment, even the zero-shot predictions using Clap-Ipa can perform segmentation in unseen languages, especially at the word level. Interestingly, there were no significant differences between performance over seen and unseen languages. Though this result could be biased by the

\begin{table}
\begin{tabular}{c c c c c} \hline \multirow{2}{*}{Method} & \multicolumn{2}{c}{TIMIT-Word} & \multicolumn{2}{c}{TIMIT-Phone} \\ \cline{2-5}  & F1 \(\uparrow\) & R-Val \(\uparrow\) & F1 \(\uparrow\) & R-Val \(\uparrow\) \\ \hline FAVE & - & - & 58.0 & 64.0 \\ MFA & - & - & 63.0 & 68.0 \\ Gentle & - & - & 48.0 & 56.0 \\ WebMAUS & - & - & **70.0** & **75.0** \\ W2V2-FC-20ms & - & - & 48.0 & 56.0 \\ W2V2-FS-20ms & - & - & 48.0 & 55.0 \\ \hline Zeroshot & & & & \\ \hline Clap-Ipa-tiny & 84.37 & 86.66 & 40.46 & 49.92 \\ Clap-Ipa-base & 78.61 & 81.73 & 36.16 & 46.59 \\ Clap-Ipa-small & 74.18 & 77.95 & 35.26 & 46.17 \\ \hline Finetuned & & & & \\ \hline Ipa-Aligner-tiny & **86.84** & **88.75** & 57.31 & 63.66 \\ Ipa-Aligner-base & 86.55 & 88.51 & 60.86 & 66.67 \\ Ipa-Aligner-small & 82.33 & 84.76 & 52.54 & 59.53 \\ \end{tabular}
\end{table}
Table 4: Evaluation of forced alignment on Timit. Baseline results were retrieved from Zhu et al. (2022). The temporal resolution is 10ms for FAVE, MFA, Gentle, and WebMAUS and 20ms for the rest of the models.

Figure 2: Illustration of forced alignment in an Evenki utterance. Clap-Ipa exhibits vague monotonic alignment without finetuning (**Top**). After finetuning, Ia-Aligner learns salient monotonic alignment between speech and phonemes (**Bottom**).

 

smaller number of unseen languages compared to seen languages (14 vs. 30), it still suggests that Ipa-Aligner can perform crosslinguistic forced alignment without much adaptation. Finetuning the Ipa-Aligner brings continued improvement over the zero-shot scenarios (see Fig 2).

Does the phoneme-based model generalize better cross-linguistically than the text-based model?Text-based models are struggling to generalize to unseen languages, as these unseen languages have their distinct writing systems (e.g, Vietnamese and Tamil) that are not seen in training languages. Comparison between the text-based and phoneme-based models in Table 2 and Table 3 clearly shows that it is the use of phonemes as modeling units that brings strong crosslinking generalizability, since they can represent all languages using the same set of symbols.

Do the training hours of individual languages predict the performance of multilingual models?The number of training hours for individual languages does not predict the performance of language in phoneme-based models. All languages benefit from the multilingual knowledge transfer in phoneme-based modeling.

It has been reported that there is a strong correlation between text-based multilingual ASR performance in individual languages and their training hours Radford et al. (2023); Rouditchenko et al. (2023). We also confirm that, for text-based models, there is a moderate correlation between Hit@1 and the number of training hours Spearman's \(\rho:0.42\)

\begin{table}
\begin{tabular}{c|l|l} \hline \hline Query & Output Type & Retrieved candidates (ranked from high to low) \\ \hline étá & Most similar & êta, Éta, lita, �ita, mita, mTA, meta, bña, aita, ktÉ, at°a, a�ta, eito \\  & Most dissimilar & bóti, üurifj, sorpgi, abürru, mburuu, sumbu, buluz, fj5j5, t/jungu \\ \hline \hline \end{tabular}
\end{table}
Table 6: Sample ranked phonemic sequences by Clap-Ipa-small, given the speech query [éta].

Figure 3: Correlation of model performance on individual languages with training hours by language. Languages are represented by their ISO 639-3 codes. While trained the exact same data, the phoneme-based model outperforms the text-based model in every single language, suggesting that phoneme-based modeling enables knowledge transfer across languages.

 

\(p\leq 0.0002\)). However, this correlation was not significant for the phoneme-based model (Spearman's \(\rho:0.14\); \(p=0.22\)). In Figure 3, the phoneme-based model outperforms the text-based model in every language by a large margin, especially for languages with less training data.

Since the orthography varies across languages and is usually not an accurate reflection of pronunciation, many low-resource languages are not reaping the full benefits of large-scale multilingual data in this cross-modal task in text-based models. Close inspection shows that the text-based model generalizes well to Hausa (Latin alphabet) but significantly underperforms in languages with non-Latin alphabet, such as Tamil, Vietnamese, Japanese, Arabic, and Cantonese.

In contrast, the phoneme-based model achieves near-perfect performance in retrieval in almost all seen and unseen languages (see Figure 3), making them extremely useful in low-resource and zero-resource scenarios. The efficiency of IPA representations in multilingual settings has also been observed in ASR Feng et al. (2023).

**Does multilingual models always hold advantages over monolingual models?** At least in the current study, multilingual models might not hold an apparent advantage over well-engineered monolingual models in high-resource languages. As shown in Table 2 and Table 4, compared to other state-of-the-art KWS and forced alignment models, Clap-Ipa and Ipa-Aligner was not able to outperform well-engineered monolingual models. Our multilingual models have not been trained on the training set of LibriPhrase or TIMIT, so some of the performance gaps might be caused by domain mismatch. Even with zero adaptations, multilingual models achieve close performance to monolingual models, suggesting that our approach is promising and may reach better results if scaled up.

**Should we scale up the number of languages or number of training hours?** We compared Clap-Ipa only trained on VoxCommunis Ahn and Chodroff (2022) or Fleurs-Ipa. VoxCommunis has almost twice as many hours as Fleurs-Ipa with roughly half of the languages. In Table 2 and 3, Clap-Ipa-Vc trained on more hours of speech generally has similar performance as Clap-Ipa-Fleurs trained on a subset of the IpaPack across metrics, which suggests that creating high-quality data is effective in achieving good performance. But this finding also suggests that we can achieve good crosslinking generalizability with fewer languages but longer hours using phoneme modeling. Given the empirical data distributions in real-life settings, scaling up training hours in a dozen of languages is much easier than scaling up the number of languages. The practical implication is that we might be able to build multilingual speech processing systems for many low-resource or zero-resource languages with large-scale data in a dozen relatively high-resource languages.

**Is it feasible to scale up the creation of good-quality phonemic transcriptions in world languages?** Despite our attempt, there are still multiple challenges for creating phonemic transcriptions. During our dataset construction, we were unable to process many languages due to the lack of pronunciation dictionaries, text transcriptions, or relevant NLP tools, especially the lack of good word segmentation tools for some East/Southeast Asian languages like Khmer. While available large-scale speech corpora nowadays encompass more than 1000 languages Salesky et al. (2020); Pratap et al. (2023), textual or phonemic labels cannot be easily obtained for most of them, limiting their usage in many research applications.

Even for high-resource languages, preprocessing multilingual texts and normalizing the Unicode encodings for IPA symbols usually take tremendous effort, not to mention verifying these phonetic transcriptions for audio recordings. It remains unclear how biases or noises in G2P predictions will propagate to downstream multilingual tasks. Our endeavor marks a small step in creating good-quality phonemic transcriptions for more languages. However, there is still much work to be done to include a broader array of languages worldwide and to improve the quality of transcriptions.

## 8 Conclusions

With the carefully curated IpApAck, we show that using IPA symbols as modeling units can effectively enable Clap-Ipa and Ipa-Aligner to generalize to unseen languages, highlighting the benefits of incorporating linguistic knowledge into deep learning methods. We believe that the IpaPack has great potential to benefit more tasks in multilingual speech processing, such as multilingual phoneme recognition, speech synthesis, and documenting endangered languages. In the future, we will continue to expand our dataset and models to include more diverse languages 



## 9 Ethical statement

**Data Governance** We adhered strictly to ethical practices in curating our datasets. The original FLEURS Conneau et al. (2023), MSWC Mazumder et al. (2021), DoReCo Paschen et al. (2020) and VoxCommunis Ahn and Chodroff (2022) corpora are distributed under the Creative Commons licenses. Therefore, we are permitted to re-process and re-distribute the original dataset with proper attributions. Some languages in the DoReCo corpus are under a Creative Commons Non-Commercial license. We reserved these languages to the test set in our corpora, such that our models have not been trained on data under commercially restrictive licenses. As required, we have also cited every individual language from the DoReCo Corpus in Table 11.

**Potential Impact** We believe that our dataset and models will contribute to the endeavor of building fair and inclusive speech processing systems for all languages and facilitating the documentation of endangered languages. However, we are aware that multilingual keyword-spotting technology could potentially be misused as surveillance tools for monitoring speech recordings in more languages, posing risks to users.

## 10 Limitations

Our study is still limited in several aspects. First, while we tried our best to inspect a subset of our dataset, it was impossible for us to examine all datasets in great detail. As a result, the constructed dataset might still be flawed in terms of audio quality and transcription quality (and many unicode errors). Secondly, the proposed models are still not optimized in terms of computational efficiency. Since most KWS applications are running on mobile devices with limited computational power, the proposed models still have too many model parameters to run efficiently on mobile devices. Moreover, speech sequences are usually much longer than text sequences. Self-attention with quadratic complexity might not be the most suitable architecture for processing speech. More efforts are needed to make such multilingual models efficient.

Thirdly, the number of languages studied in our paper is still limited and might be biased towards languages that are relatively high-resource. They are not representative of the global language landscape. There are many more low-resource or endangered languages we are not able to include due to the lack of various resources. To promote linguistic inclusion and fairness, we will continue to improve the language diversity of our research in the future.

## Acknowledgements

This research was enabled in part through the computational resources and services provided by Advanced Research Computing at the University of British Columbia and in part through the support provided by the Digital Research Alliance of Canada. This study also benefits from the cloud computing credits awarded by Microsoft Azure through a pilot program with UBC Advanced Research Computing.

The authors would like to thank the four anonymous reviewers as well as the area chairs for their thoughtful comments and discussions, which helped improve this article considerably. We thank Emily P. Ahn and Eleanor Chodroff for creating and releasing the VoxCommunis, which inspired our data creation process. Finally, we acknowledge that this work is impossible without the pioneering efforts of many language researchers who collected and shared speech corpora across world languages.

## References

* Ahn and Chodroff (2022) Emily P. Ahn and Eleanor Chodroff. 2022. VoxCommunis: A corpus for cross-linguistic phonetic analysis. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 5286-5294, Marseille, France. European Language Resources Association. External Links: Link Cited by: SS1.
* M. Avanzi, M. Beguelin, G. Corminboeuf, F. Diemoz, and L. Anne Johnsen (2022)French (Swiss) doReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon. External Links: ISBN 978-3-319-3899-3 Cited by: SS1.
* J. Aznar (2022)Niscus poReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon. External Links: ISBN 978-3-319-3899-3 Cited by: SS1.
* A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, et al.



2021. Xls-r: Self-supervised cross-lingual speech representation learning at scale. _arXiv preprint arXiv:2111.09296_.
* Badlani et al. (2022) Rohan Badlani, Adrian Lancucki, Kevin J Shih, Rafael Valle, Wei Ping, and Bryan Catanzaro. 2022. One tts alignment to rule them all. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 6092-6096. IEEE.
* Berg et al. (2021) Axel Berg, Mark O'Connor, and Miguel Tairum Cruz. 2021. Keyword Transformer: A Self-Attention Model for Keyword Spotting. In _Proc. Interspeech 2021_, pages 4249-4253.
* Bogomolova et al. (2022) Natalia Bogomolova, Dmitry Ganenkov, and Nils Norman Schibor. 2022. Tabasaran DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Synchronissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Burenhult (2022) Niclas Burenhult. 2022. Jahai DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Synchronissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Chen et al. (2014) Guoguo Chen, Carolina Parada, and Georg Heigold. 2014. Small-footprint keyword spotting using deep neural networks. In _2014 IEEE international conference on acoustics, speech and signal processing (ICASSP)_, pages 4087-4091. IEEE.
* Chen et al. (2022) Yi-Chang Chen, Yu-Chuan Steven, Yen-Cheng Chang, and Yi-Ren Yeh. 2022. g2pW: A Conditional Weighted Softmax BERT for Polyphone Disambiguation in Mandarin. In _Proc. Interspeech 2022_, pages 1926-1930.
* Cobbinah (2022) Alexander Yao Cobbinah. 2022. Banounk Gubeeher DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Conneau et al. (2020) Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2020. Unsupervised cross-lingual representation learning for speech recognition. _arXiv preprint arXiv:2006.13979_.
* Conneau et al. (2023) Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2023. Fleurs: Few-shot learning evaluation of universal representations of speech. In _2022 IEEE Spoken Language Technology Workshop (SLT)_, pages 798-805. IEEE.
* Cowell (2022) Andrew Cowell. 2022. Arapaho DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemein Sprachwissenschaft & laboratory Dynamic Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Duquenne et al. (2021) Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk. 2021. Multimodal and multilingual embeddings for large-scale speech mining. _Advances in Neural Information Processing Systems_, 34:15748-15761.
* Dabritz et al. (2022) Chris Lasse Dabritz, Nina Kudryakova, Eugenic Stapert, and Alexandre Arkhipov. 2022. Dolgan DoReCo dataser. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Languages Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Dohler (2022) Christian Dohler. 2022. Komnzo DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Feng et al. (2023) Siyuan Feng, Ming Tu, Rui Xia, Chuanzeng Huang, and Yuxuan Wang. 2023. Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognition. In _Proc. INTERSPEECH 2023_, pages 1384-1388.
* Forker and Schiborr (2022) Diana Forker and Nils Norman Schiborr. 2022. Sanzhi Dargwa DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Franjieh (2022) Michael Franjieh. 2022. Fanbyak DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DuRecCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Garcia-Laguna (2022) Alexandro Garcia-Laguna. 2022. Northern Alta DoReCo dataset. In Frank Seifart, Ludger Paschen, 

and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamatique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Garofolo et al. (1993) John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett. 1993. Darpa timit acoustic-phonetic continous speech corpus cdron. mist speech disc 1-1.1. _NASA STI/Recon technical report n_, 93:27403.
* Gick et al. (2013) Bryan Gick, Ian Wilson, and Donald Derrick. 2013. _Articulatory phonetics_. John Wiley & Sons.
* Gippert (2022) Jost Gippert. 2022. Swan DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Steve, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Glocker et al. (2023) Kevin Glocker, Aaricia Herygers, and Munir Georges. 2023. Altophant: Cross-lingual Phoneme Recognition with Articulatory Attributes. In _Proc. INTERSPEECH 2023_, pages 2258-2262.
* Gordon and Roettger (2017) Matthew Gordon and Timo Roettger. 2017. Acoustic correlates of word stress: A cross-linguistic survey. _Linguistics Vanguard_, 3(1):20170007.
* Gordon (2016) Matthew K Gordon. 2016. _Phonological typology_, volume 1. Oxford University Press.
* Griscom (2022) Richard Griscom. 2022. Asimjeeg Datooga DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschchaft & laboratoire Dynamatique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Gusev et al. (2022) Valentin Gusev, Tiina Klooster, Beata Wagner-Nagy, and Alexandre Arkhipov. 2022. Kamas DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamatique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Guldemann et al. (2022) Tom Guldemann, Martina Ernszt, Sven Siegmund, and Alena Witzlack-Makarevich. 2022. N\(|\)ng DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamatique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Guildemann et al. (2022) Geoff Haig, Maria Vollmer, and Hanna Thiele. 2022. Northern Kurdish (Kurmanji) DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamatique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Harvey (2022) Andrew Harvey. 2022. Gorwaa DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamatique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Heslewod (2013) Berry Heslewod. 2013. _Phoneic Transcription in Theory and Practice_. Edinburgh University Press.
* Association (1999) IPA International Phonetic Association. 1999. _Handbook of the International Phonetic Association: A guide to the use of the International Phonetic Alphabet_. Cambridge University Press.
* Kazakevich and Glyachko (2022) Olga Kazakevich and Elena Glyachko. 2022. Evenki DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamatique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Kelley and Tucker (2018) Matthew C. Kelley and Benjamin V. Tucker. 2018. A Comparison of Input Types to a Deep Neural Network-based Forced Aligner. In _Proc. Interspeech 2018_, pages 1205-1209.
* Khurana et al. (2022) Sameer Khurana, Antoine Laurent, and James Glass. 2022. Samu-xlsr: Semantically-aligned multimodal utterance-level cross-lingual speech representation. _IEEE Journal of Selected Topics in Signal Processing_, 16(6):1493-1504.
* Kim (2022) Soung-U Kim. 2022. Jejuan DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamatique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Kissler et al. (2012) Thomas Kissler, Florian Schiel, and Han Sloetjes. 2012. Signal processing via web services: the use case webmaus. In _Digital Humanities Conference 2012_.
* Kreuk et al. (2020) Felix Kreuk, Joseph Keshet, and Yossi Adi. 2020. Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation. In _Proc. Interspeech 2020_, pages 3700-3704.
* Krifka (2022) Manfried Krifka. 2022. Daokie DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamatique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.

 

* Kudo (2018) Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. _arXiv preprint arXiv:1804.10959_.
* Kurzinger et al. (2020) Ludwig Kurzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, and Gerhard Rigoll. 2020. Ctc-segmentation of large corpora for german end-to-end speech recognition. In _International Conference on Speech and Computer_, pages 267-278. Springer.
* Ladefoged (1990) Peter Ladefoged. 1990. The revised international phonetic alphabet. _Language_, 66(3):550-552.
* Ladefoged and Halle (1988) Peter Ladefoged and Morris Halle. 1988. Some major features of the international phonetic alphabet. _Language_, 64(3):577-582.
* Lee and Cho (2023) Yong-Hyeok Lee and Namhyun Cho. 2023. PhonMatchNet: Phoneme-Guided Zero-Shot Keyword Spotting for User-Defined Keywords. In _Proc. INTERSPEECH 2023_, pages 3964-3968.
* Lei et al. (2023) Lei Lei, Guoshun Yuan, Hongjiang Yu, Dewei Kong, and Yuefeng He. 2023. Multilingual customized keyword spotting using similar-pair contrastive learning. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_.
* Li et al. (2020) Xinjian Li, Siddharth Dalmia, Juncheng Li, Matthew Lee, Patrick Littell, Jiial Yao, Antonios Anastasopoulos, David R Mortensen, Graham Neubig, Alan W Black, et al. 2020. Universal phone recognition with a multilingual allophone system. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 8249-8253. IEEE.
* Li et al. (2021) Xinjian Li, David R Mortensen, Florian Metze, and Alan W Black. 2021. Multilingual phonetic dataset for low resource speech recognition. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 6958-6962. IEEE.
* Mazumder et al. (2021a) Mark Mazumder, Colby Banbury, Josh Meyer, Pete Warden, and Vijay Janapa Reddi. 2021a. Few-Shot Keyword Spotting in Any Language. In _Proc. Interspeech 2021_, pages 4214-4218.
* Mazumder et al. (2021b) Mark Mazumder, Sharad Chitlangia, Colby Banbury, Yiping Kang, Juan Manuel Ciro, Keith Achorn, Daniel Galvez, Mark Sabini, Peter Mattson, David Kanter, et al. 2021b. Multilingual spoken words corpus. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
* McAuliffe et al. (2017) Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal forced aligner: Trainable text-speech alignment using kaldi. In _Interspeech_, volume 2017, pages 498-502.
* McCann (2020) Paul McCann. 2020. fugashi, a tool for tokenizing Japanese in python. In _Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)_, pages 44-51, Online. Association for Computational Linguistics.
* Michaud (2022) Alexis Michaud. 2022. Yongning Na DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Moran et al. (2014) Steven Moran, Daniel McCloy, and Richard Wright. 2014. Phoible online.
* Mortensen et al. (2018) David R Mortensen, Siddharth Dalmia, and Patrick Littell. 2018. Epitran: Precision g2p for many languages. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
* Mosel (2022) Ulrike Mosel. 2022. Teop DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Nishu et al. (2023) Kumari Nishu, Minsik Cho, Paul Dixon, and Devang Naik. 2023. Flexible keyword spotting based on homogeneous audio-text embedding. _arXiv preprint arXiv:2308.06472_.
* Noriy et al. (2023) Kai A Noriy, Xiaosong Yang, Marcin Budka, and Jian Jun Zhang. 2023. Clara: Multilingual contrastive learning for audio representation acquisition. _arXiv preprint arXiv:2310.11830_.
* O'Shannessy (2022a) Carmel O'Shannessy. 2022a. Light Warlpii DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Osmanessy (2022b) Carmel O'Shannessy. 2022b. Warlpii DoReCo dataset. In Frank Seifert, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Ozerov (2022) Pavel Ozerov. 2022. Anal DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Ozerov (2022)Pavel Ozerov. 2022. Anal DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Park et al. (2019) Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. _Interspeech 2019_ 

Ludger Paschen, Francois Delafontaine, Christoph Draxler, Susanne Fuchs, Matthew Stave, and Frank Seifart. 2020. Building a time-aligned cross-linguistic reference corpus from language documentation data (doreco). In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 2657-2666.
* Phatthiyaphaibun et al. (2016) Wananphong Phathtiyahphaibun, Korakot Chaovavanich, Charin Polpanumas, Archit Suriyawongkul, Lalita Lowbhansirikul, and Pattarawat Chormai. 2016. PyThaiNLP: Thai Natural Language Processing in Python.
* Pitt et al. (2005) Mark A Pitt, Keith Johnson, Elizabeth Hume, Scott Kiesling, and William Raymond. 2005. The buckeye corpus of conversational speech: Labeling conventions and a test of transcriber reliability. _Speech Communication_, 45(1):89-95.
* Ponsonnet (2022) Maia Ponsonnet. 2022. Dalabon DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Pratap et al. (2023) Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2023. Scaling speech technology to 1,000+ languages. _arXiv preprint arXiv:2305.13516_.
* Quesada et al. (2022) Juan Diego Quesada, Stavros Skopeteas, Carolina Pasamonik, Carolin Brokmann, and Florian Fischer. 2022. Cabecar DoReCo dataset. In Frank Seifert, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoeRecO) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Radford et al. (2023) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeaeyev, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In _International Conference on Machine Learning_, pages 28492-28518. PMLR.
* Reiter (2022) Sabine Reiter. 2022. Cashinahua DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Reuter et al. (2023) Paul M Reuter, Christian Rollwage, and Bernd T Meyer. 2023. Multilingual query-by-example keyword spotting with metric learning and phoneme-to-embedding mapping. In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 1-5. IEEE.
* Riesberg (2022) Sonja Riesberg. 2022. Yali (Apahapsili) DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Ring (2022) Hiram Ring. 2022. Pinar DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Rose (2022) Francoise Rose. 2022. Mojeno Trinitario DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Rosenfelder et al. (2011) Ingrid Rosenfelder, Josef Fruehwald, Keelan Evaini, and Jiahong Yuan. 2011. Fave (forced alignment and vowel extraction) program suite. _URL [http://fzve.ling.upenn.edu_](http://fzve.ling.upenn.edu_).
* Rouditchenko et al. (2023) Andrew Rouditchenko, Sameer Khurana, Samuel Thomas, Rogerio Feris, Leonid Karlinsky, Hilde Kuehne, David Harwath, Brian Kingsbury, and James Glass. 2023. Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages. In _Proc. INTERSPEECH 2023_, pages 2268-2272.
* Rybakov et al. (2020) Oleg Rybakov, Natasha Kononenko, Niranjan Subrahmanya, Mirko Visontai, and Stella Laurenzo. 2020. Streaming Keyword Spotting on Mobile Devices. In _Proc. Interspeech 2020_, pages 2277-2281.
* Rasanen et al. (2009) Okko Johannes Rasanen, Unto Kalervo Laine, and Toomas Altosaar. 2009. An improved speech segmentation quality measure: the r-value. In _Proc. Interspeech 2009_, pages 1851-1854.
* Salesky et al. (2020) Elizabeth Salesky, Eleanor Chodroff, Tiago Pimentel, Matthew Wiesner, Ryan Cotterell, Alan W Black, and Jason Eisner. 2020. A corpus for large-scale phonetic typology. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4526-4546, Online. Association for Computational Linguistics.
* Schnell (2022) Stefan Schnell. 2022. Vera'a DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Rose (2022) Francoise Rose. 2022. Mojeno Trinitario DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Rosenfelder et al. (2011) Ingrid Rosenfelder, Josef Fruehwald, Keelan Evaini, and Jiahong Yuan. 2011. Fave (forced alignment and vowel extraction) program suite. _URL [http://fzve.ling.upenn.edu_](http://fzve.ling.upenn.edu_).
* Rooditchenko et al. (2023) Andrew Rooditchenko, Sameer Khurana, Samuel Thomas, Rogerio Feris, Leonid Karlinsky, Hilde Kuehne, David Harwath, Brian Kingsbury, and James Glass. 2023. Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages. In _Proc. INTERSPEECH 2023_, pages 2268-2272.
* Rybakov et al. (2020) Oleg Rybakov, Natasha Kononenko, Niranjan Subrahmanya, Mirko Visontai, and Stella Laurenzo. 2020. Streaming Keyword Spotting on Mobile Devices. In _Proc. Interspeech 2020_, pages 2277-2281.
* Rasanen et al. (2009) Okko Johannes Rasanen, Unto Kalervo Laine, and Toomas Altosaar. 2009. An improved speech segmentation quality measure: the r-value. In _Proc. Interspeech 2009_, pages 1851-1854.
* Salesky et al. (2020) Elizabeth Salesky, Eleanor Chodroff, Tiago Pimentel, Matthew Wiesner, Ryan Cotterell, Alan W Black, and Jason Eisner. 2020. A corpus for large-scale phonetic typology. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4526-4546, Online. Association for Computational Linguistics.
* Schnell (2022) Stefan Schnell. 2022. Vera'a DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Roche (2022) Francoise Roche. 2022. Mojeno Trinitario DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Rosenfelder et al. (2011) Ingrid Rosenfelder, Josef Fruehwald, Keelan Evaini, and Jiahong Yuan. 2011. Fave (forced alignment and vowel extraction) program suite. _URL [http://fzve.ling.upenn.edu_](http://fzve.ling.upenn.edu_).
* Rouditchenko et al. (2023) Andrew Rouditchenko, Sameer Khurana, Samuel Thomas, Rogerio Feris, Leonid Karlinsky, Hilde Kuehne, David Harwath, Brian Kingsbury, and James Glass. 2023. Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages. In _Proc. INTERSPEECH 2023_, pages 2268-2272.
* Rybakov et al. (2020) Oleg Rybakov, Natasha Kononenko, Niranjan Subrahmanya, Mirko Visontai, and Stella Laurenzo. 2020. Streaming Keyword Spotting on Mobile Devices. In _Proc. Interspeech 2020_, pages 2277-2281.
* Rasanen et al. (2009) Okko Johannes Rasanen, Unto Kalervo Laine, and Toomas Altosaar. 2009. An improved speech segmentation quality measure: the r-value. In _Proc. Interspeech 2009_, pages 1851-1854.
* Salesky et al. (2020) Elizabeth Salesky, Eleanor Chodroff, Tiago Pimentel, Matthew Wiesner, Ryan Cotterell, Alan W Black, and Jason Eisner. 2020. A corpus for large-scale phonetic typology. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4526-4546, Online. Association for Computational Linguistics.
* Schnell (2022) Stefan Schnell. 2022. Vera'a DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Schulze-Forster et al. (2020) Kilian Schulze-Forster, Clement SJ Doire, Gael Richard, and Roland Badeau. 2020. Joint phoneme alignment 

and text-informed speech separation on highly corrupted speech. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 7274-7278. IEEE.
* Seifart [2022a] Frank Seifart. 2022a. Bora DoReCo dataset. In Frank Seifert, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2._ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Seifart [2022b] Frank Seifart. 2022b. Resigaro DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2._ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Shih et al. [2021] Kevin J Shih, Rafael Valle, Rohan Badlani, Adrian Lancucki, Wei Ping, and Bryan Catanzaro. 2021. Rad-tts: Parallel flow-based tts with robust alignment learning and diverse synthesis. In _ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models_.
* Shin et al. [2022] Hyeon-Kyeong Shin, Hyewon Han, Doyeon Kim, Soohwan Chung, and Hong-Goo Kang. 2022. Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting. In _Proc. Interspeech 2022_, pages 1871-1875.
* Skopteas et al. [2022] Stavros Skopteas, Violeta Moisidi, Nutsu Tsetereli, Johanna Lorenz, and Stefanie Schroter. 2022. Urum DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2._ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Tanaka et al. [2001] Kazuya Tanaka, Yoshiaki Itoh, Hiroki Kojima, and Nahoo Fujimura. 2001. Speech data retrieval system constructed on a universal phonetic code domain. In _IEEE Workshop on Automatic Speech Recognition and Understanding, 2001. ASRU'01._, pages 323-326. IEEE.
* Tang and Lin [2018] Raphael Tang and Jimmy Lin. 2018. Deep residual learning for small-footprint keyword spotting. In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 5484-5488. IEEE.
* Teo [2022] Amos Teo. 2022. Sumi DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2._ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universiteschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Tsufaut et al. [2022] Yann Tsufaut, Baptiste Bouvier, and Axel Roebel. 2022. A study on constraining connectionist temporal classification for temporal audio alignment. In _Interspeech 2022_, pages 5015-5019. ISCA.
* Teytaut and Roebel [2021] Yann Teytaut and Axel Roebel. 2021. Phoneme-to-audio alignment with recurrent neural networks for speaking and singing voice. In _Proceedings of Interspeech 2021_, pages 61-65. International Speech Communication Association; ISCA.
* Thieberger [2022] Nick Thieberger. 2022. Nafsan (South Efate) DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2._ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Vanhove [2022] Martine Vanhove. 2022. Beja DoReCo dataset. In Frank Seifert, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2._ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Vydrina [2022] Alexandra Vydrine. 2022. Kakabe DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2._ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Vogren [2022] Claudia Wegener. 2022. Savosavo DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2._ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Wells [1995] John C Wells. 1995. Computer-coding the IPA: a proposed extension of SAMPA.
* Wichmann [2022] Soren Wichmann. 2022. Texistepec Popoluca DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2._ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Witzlack-Makarevich et al. [2022] Alena Witzlack-Makarevich, Saudah Namyalo, Anatol Kiriggwaijo, and Zarina Molochicava. 2022. Ruuli DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2._ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamiquque Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Wu et al. [2023] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shimomo Dubnov. 2023.

 

Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 1-5. IEEE.
* Xu et al. [2022] Qiantong Xu, Alexei Baevski, and Michael Auli. 2022. Simple and Effective Zero-shot Cross-lingual Phoneme Recognition. In _Proc. Interspeech 2022_, pages 2113-2117.
* Xu and Bai [2022] Xianming Xu and Bibo Bai. 2022. Sadu DoReCo dataset. In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2._ Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Universite Lyon 2), Berlin & Lyon.
* Zhai et al. [2023] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. _arXiv preprint arXiv:2303.15343_.
* Zhu et al. [2022a] Jian Zhu, Zuoyu Tian, Yadong Liu, Cong Zhang, and Chia-Wen Lo. 2022a. Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 1134-1154, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Zhu et al. [2022b] Jian Zhu, Cong Zhang, and David Jurgens. 2022b. ByT5 model for massively multilingual grapheme-to-phoneme conversion. In _Proc. Interspeech 2022_, pages 446-450.
* Zhu et al. [2022c] Jian Zhu, Cong Zhang, and David Jurgens. 2022c. Phone-to-audio alignment without text: A semi-supervised approach. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 8167-8171. IEEE.

## Appendix A Dataset statistics

Table 9, 12 , 10, and 11 provide tabulated summaries of the detailed statistics of our curated datasets.

 



\begin{table}
\begin{tabular}{l l l l} \hline Hyperparameters & Clap-Ipa-tiny & Clap-Ipa-base & Clap-Ipa-small \\ \hline Hidden dimensions & 384 & 512 & 768 \\ Num. Layers & 4 & 6 & 12 \\ Num. Att. Heads & 6 & 8 & 12 \\ Intermediate size & 1536 & 2048 & 3072 \\ Parameters & 16M & 28.5M & 96.2M \\ Initial learning rate & 1e-4 & & \\ Scheduler & Cosine Scheduler & & \\ Warm-up steps & 500 & & \\ Total training steps & 100k & & \\ Fleurs-Ipa batch size & 64 & 64 & 32 \\ Mswc-Ipa batch size & 512 & 512 & 256 \\ Doreco-Ipa batch size & 64 & 64 & 32 \\ VoxCommunis batch size & 64 & 64 & 32 \\ Gradient checkpointing & True & & \\ Mixed Precision & float16 & & \\ Max. Gradient Norm for Gradient Clipping & 10 & & \\ \hline \end{tabular}
\end{table}
Table 7: Hyperparameters for training Clap-Ipa models.

\begin{table}
\begin{tabular}{p{113.8pt} p{56.9pt} p{56.9pt} p{56.9pt}} \hline \hline Hyperparameters & Clap-Ipa-tiny & Clap-Ipa-base & Clap-Ipa-small \\ \hline Hidden dimensions & 384 & 512 & 768 \\ Num. Layers & 4 & 6 & 12 \\ Num. Att. Heads & 6 & 8 & 12 \\ Intermediate size & 1536 & 2048 & 3072 \\ Parameters & 16M & 28.5M & 96.2M \\ Initial learning rate & 1e-5 & & \\ Scheduler & Cosine Scheduler & & \\ Warm-up steps & 100 & & \\ Maximum training steps & 10k & & \\ batch size & 128 & & \\ Gradient checkpointing & True & & \\ Mixed Precision & float16 & & \\ Max. Gradient Norm for Gradient Clipping & 10 & & \\ Early stopping & True & & \\ Stopping Criteria & the highest F1 on the TIMIT training set & & \\ \hline \hline \end{tabular}
\end{table}
Table 8: Hyperparameters for training Ipa-Aligner models.





\begin{table}
\begin{tabular}{l l c c c c c} \hline Language & ISO 639-3 & Family & Train (hrs) & Dev (hrs) & Test (hrs) & Avg.Phones & Avg.Dur. (s) \\ \hline Arabic & ara & Indo-European & 0.79 & 0.11 & 0.11 & 6.57 (1.58) & 1 (0) \\ Catalan & cat & Indo-European & 61.38 & 8.12 & 8.07 & 7.51 (2.34) & 1 (0) \\ Czech & cze & Indo-European & 3.01 & 0.41 & 0.39 & 6.2 (1.92) & 1 (0) \\ Dutch & dut & Indo-European & 6.42 & 0.84 & 0.84 & 7.2 (2.79) & 1 (0) \\ English & eng & Indo-European & 125.61 & 16.29 & 16.52 & 6.84 (2.34) & 1 (0) \\ Esperanto & epo & Constructed & 8.48 & 1.13 & 1.12 & 6.95 (1.97) & 1 (0) \\ Estonian & est & Uralic & 2.51 & 0.34 & 0.33 & 6.35 (2.13) & 1 (0) \\ French & fra & Indo-European & 62.69 & 8.31 & 8.31 & 5.8 (2.03) & 1 (0) \\ German & ger & Indo-European & 83.23 & 10.99 & 10.96 & 8.81 (3.41) & 1 (0) \\ Irish & gle & Indo-European & 0.48 & 0.07 & 0.07 & 4.17 (1.45) & 1 (0) \\ Greek & gre & Indo-European & 0.71 & 0.1 & 0.1 & 5.71 (2.09) & 1 (0) \\ Interlingua & ina & Constructed & 0.53 & 0.06 & 0.05 & 5.8 (1.84) & 1 (0) \\ Indonesian & ind & Austronesian & 1.74 & 0.25 & 0.25 & 6.04 (1.88) & 1 (0) \\ Italian & ita & Indo-European & 18.42 & 2.46 & 2.43 & 7.35 (2.45) & 1 (0) \\ Kyrgyz & kir & Turkic & 1.52 & 0.23 & 0.21 & 6.89 (2.43) & 1 (0) \\ Lithuanian & lit & Indo-European & 0.7 & 0.04 & 0.06 & 6.88 (2.22) & 1 (0) \\ Maltese & mlt & Afro-Asiatic & 1.12 & 0.15 & 0.16 & 6.04 (2.75) & 1 (0) \\ Mongolian & mon & Mongolic & 1.48 & 0.2 & 0.21 & 5.5 (1.73) & 1 (0) \\ Polish & pol & Indo-European & 14.39 & 1.93 & 1.94 & 6.9 (2.09) & 1 (0) \\ Portuguese & por & Indo-European & 7.16 & 0.95 & 0.95 & 6.52 (2.07) & 1 (0) \\ Romanian & ron & Indo-European & 0.5 & 0.1 & 0.08 & 6.43 (2.31) & 1 (0) \\ Russian & rus & Indo-European & 18.48 & 2.46 & 2.44 & 8.48 (2.88) & 1 (0) \\ Slovak & slo & Indo-European & 0.08 & 0.01 & 0.01 & 6.28 (2.14) & 1 (0) \\ Slovenian & slv & Indo-European & 0.27 & 0.05 & 0.05 & 5.16 (1.56) & 1 (0) \\ Spanish & spa & Indo-European & 40.04 & 5.35 & 5.32 & 7.64 (2.49) & 1 (0) \\ Swedish & swe & Indo-European & 1.18 & 0.16 & 0.16 & 5.36 (1.93) & 1 (0) \\ Tatar & tat & Turkic & 3.76 & 0.5 & 0.48 & 6.18 (1.84) & 1 (0) \\ Turkish & tur & Turkic & 2.82 & 0.38 & 0.39 & 6.95 (2.27) & 1 (0) \\ Ukrainian & ukr & Indo-European & 1.87 & 0.25 & 0.26 & 6.76 (2.3) & 1 (0) \\ Welsh & wel & Indo-European & 13.6 & 1.8 & 1.81 & 5.76 (1.96) & 1 (0) \\ Mandarin & cmn & Sino-Tibetan & 0.4 & 0.05 & 0.05 & 8.68 (2.26) & 1 (0) \\ \hline \end{tabular}
\end{table}
Table 9: Statistics of languages in Mswc-Ipa. All samples are padded to be clips of 1 second. (Avg.Phones: average number of phonemes in each word; Avg.Dur.. average duration of each clip).





\begin{table}
\begin{tabular}{l l l c c c} \hline Language & ISO 639-3 & Family & Train (hrs) & Avg. Dur (s) & Avg. Phones \\ \hline Abkhaz & abk & Northwest Caucasian & 0.62 & 7.33 & 51.93 \\ Bashkir & bak & Turkic & 137.79 & 4.35 & 35.78 \\ Belarusian & bel & Indo-European & 132.21 & 5.48 & 49.15 \\ Bulgarian & bul & Indo-European & 3.5 & 5.05 & 47.74 \\ Catalan & cat & Indo-European & 2.08 & 5.39 & 44.35 \\ Czech & cs & Indo-European & 16.51 & 4.75 & 44 \\ Chuvash & chv & Turkic & 0.37 & 4.2 & 36.97 \\ Greek & ell & Indo-European & 1.57 & 3.99 & 29.13 \\ Basque & eu s & Language isolate & 12.66 & 5.2 & 47.36 \\ Guarani & grn & Tupina & 1.81 & 3.97 & 26.91 \\ Hausa & hau & Afro-Asiatic & 1.71 & 4.27 & 32.1 \\ Hindi & hin & Indo-European & 2.73 & 3.75 & 33.69 \\ Sorbian (Upper Sorbian) & hsb & Indo-European & 1.48 & 6.61 & 55.01 \\ Hungarian & hun & Uralic & 25.06 & 4.76 & 37.62 \\ Indonesian & ind & Austrosean & 5.09 & 5.69 & 53.31 \\ Italian & ita & Indo-European & 192.69 & 5.24 & 49.13 \\ Georgia & kat & Kartvelian & 1.62 & 5.77 & 53.7 \\ Kazakh & kaz & Turkic & 0.29 & 4.93 & 33.95 \\ Kurmaji (Kurdish) & kmr & Indo-European & 2.83 & 4.47 & 28.16 \\ Kyrgzy & kir & Turkic & 2.25 & 4.67 & 43.42 \\ Marathi & mar & Indo-European & 3.66 & 5.97 & 52.99 \\ Maltese & ml & Afro-Asiatic & 2.41 & 4.48 & 36.88 \\ Erzya & myv & Uralic & 1.97 & 5.73 & 46.41 \\ Dutch & nld & Indo-European & 34.94 & 4.4 & 47.47 \\ Punjabi & pan & Indo-European & 0.96 & 5.29 & 26.53 \\ Polish & pol & Indo-European & 14.26 & 5.21 & 46.58 \\ Portuguese & por & Indo-European & 12.31 & 4.33 & 32.45 \\ Romanian & ron & Indo-European & 4.99 & 4.01 & 35.98 \\ Russian & rus & Indo-European & 24.41 & 5.44 & 56.61 \\ Swedish & swe & Indo-European & 5.84 & 3.84 & 31.24 \\ Swahili & swa & Niger-Congo & 52.8 & 5.44 & 47.43 \\ Tamil & tam & Dravidian & 61.39 & 6.57 & 55.43 \\ Thai & tha & Kra-Dai & 16.71 & 3.91 & 26.58 \\ Turkish & tur & Turkle & 0.98 & 3.19 & 30.43 \\ Tatar & tat & Turkic & 10.09 & 3.8 & 31.5 \\ Uyghur & uig & Turkic & 2.43 & 5.85 & 49.21 \\ Ukrainian & ukr & Indo-European & 4.22 & 4.67 & 39.54 \\ Vietnamese & vie & Austrosiatic & 4.6 & 4.53 & 25.54 \\ \hline \end{tabular}
\end{table}
Table 10: Detailed statistics of a selected subset of VoxCommunis (Ahn and Chodroff, 2022). (Avg.Phones: average number of phonemes in each word; Avg.Dur.: average duration of each clip).





\begin{tabular}{l l l l l l l l} \hline Language & ISO & Avg. & Total duration (s) & Avg. & Family & Split & Citation \\  & 693-3 & Dur (s) & tion (hrs) & Phones & & & \\ \hline Komnoz & tci & 2.59 & 0.27 & 29.99 & Yam & train & Dohler, 2022 \\ Vera'a & vra & 3.55 & 0.57 & 43.03 & Austronesian & train & Schnell, 2022 \\ Sanzhi Dargwa & na & 4.85 & 0.17 & 44.82 & Nakhh- & train & Forker and Schiborr, 2022 \\ Urum & uum & 4.63 & 0.37 & 45.75 & Turkic & test & Skopteas \\ Beja & bej & 2.32 & 0.36 & 24.97 & Afro-Asiatic & test & Vanhove, 2022 \\ Light Warlipiri & na & 3.47 & 0.47 & 32.75 & Mixed Lan- & train & O’Shannessy, 2022a \\ Kamas & xas & 3.60 & 0.84 & 24.71 & Uncli & train & Gusev et al., 2022 \\ Nafsen (South & erk & 6.10 & 0.36 & 50.83 & Austronesian & test & Thieberger, 2022 \\ Tabasaran & tab & 4.16 & 0.21 & 42.31 & Nakh- & train & Bogomolova \\ Savosavo & svs & 5.17 & 0.82 & 49.35 & Isolate & train & Wegener, 2022 \\ Sumi & nsm & 2.74 & 0.14 & 32.59 & Sino-Tibetan & train & Treo, 2022 \\ French (Swiss) & fra & 2.75 & 0.31 & 32.61 & Indo-European & test & Avanzi et al., 2022 \\ Northern Alta & aqn & 2.78 & 1.04 & 25.94 & Austronesian & train & Garcia-Laguia, 2022 \\ Jejuan & jje & 2.59 & 0.03 & 24.43 & Koreanic & train & Kim, 2022 \\ Jahai & jhi & 3.61 & 0.45 & 32.74 & Austrosiaatic & test & Burenhult, 2022 \\ Nisvai & none & 3.11 & 0.56 & 42.22 & Austronesian & test & Aznar, 2022 \\ Warlipri & wbp & 3.64 & 0.94 & 30.84 & Pama-Nyungan & test & O'Shannessy, 2022b \\ Fanbyak & fnb & 2.81 & 0.22 & 27.29 & Austronesian & train & Franjieh, 2022 \\ Bora & boa & 4.40 & 0.34 & 41.49 & Boran & train & Seifart, 2022a \\ Yongning Na & nru & 4.23 & 0.30 & 33.15 & Sino-Tibetan & train & Michaud, 2022 \\ Dalabon & ngk & 2.46 & 0.08 & 23.46 & Gunwinyguan & train & Ponsonnet, 2022) \\ Sadu & na & 2.75 & 0.15 & 22.78 & Sino-Tibetan & train & Xu and Bai, 2022 \\ Teop & tio & 2.96 & 0.65 & 30.62 & Austronesian & train & Mosel, 2022 \\ Cashinahua & cbs & 3.58 & 0.73 & 33.55 & Pano-Tacanan & train & Reiter, 2022 \\ Dolgan & dlg & 4.24 & 0.69 & 43.55 & Turkitc & test & Dabbritz et al., 2022 \\ Anal & anm & 3.02 & 0.37 & 26.43 & Sino-Tibetan & train & Ozerov, 2022 \\ Bainounuk & bab & 3.13 & 0.40 & 30.95 & Atlantic-Congo & train & Cobbinah, 2022) \\ Texistepec & poq & 2.65 & 0.08 & 28.50 & Mixe-Zoque & train & Wichmann, 2022 \\ Poobluca & ptv & 3.22 & 0.22 & 34.87 & Austronesian & train & Kriflka, 2022 \\ Ning & ngh & 2.67 & 0.12 & 22.96 & Tuu & train & Güldemann \\ Ruuli & ruc & 3.13 & 0.32 & 34.99 & Atlantic-Congo & train & WeiLack- \\ Cabcar & cjp & 3.61 & 0.38 & 39.62 & Chibchan & test & Quesada et al., 2022) \\ Evenki & evn & 3.89 & 0.66 & 31.71 & Tungusic & train & Kazakevich \\  & & & & & & & & & \\ Arapahoo & arp & 3.99 & 0.87 & 32.95 & Algic & train & Cowell, 2022 \\ Swan & sva & 4.77 & 0.56 & 47.85 & Kartvelian & train & Giprepert, 2022 \\ Resgaro & grgr & 5.45 & 1.27 & 33.31 & Arawakan & train & Seifart, 2022b \\ Yali (Apaha-silil) & na & 2.38 & 0.04 & 32.34 & Nuclear Trans & test & Riesberg, 2022) \\ Asimjeeg Daitoga & aa & 2.81 & 0.28 & 28.30 & Nilotic & train & Griscom, 2022) \\ Northern Kurdish (Kurmaiji) & kmr & 4.39 & 0.54 & 50.76 & Indo-European & test & Haig et al., 2022



\begin{table}
\begin{tabular}{l l c c c c c c c} Gorwa & gow & 2.95 & 0.28 & 31.29 & Afro-Asiatic & train & (Harvey, 2022) \\ Par & pbw & 8.28 & 0.29 & 72.74 & Austrosiastic & test & (Ring, 2022) \\ Kakabe & kke & 4.14 & 0.59 & 33.07 & Mande & train & (Vydrina, 2022) \\ Mojeno Trinitoro & trn & 5.66 & 0.65 & 48.42 & Arawakan & train & (Rose, 2022) \\ \end{tabular}
\end{table}
Table 11: Detailed statistics of Doreco-Ipa. (Avg. Phones: average number of phonemes in each word; Avg.Dur.: average duration of each clip).









\begin{table}
\begin{tabular}{l l l c c c c c} Thai & tha & Kra-Dai & 6.21 & 1.14 & 2.56 & 11.34 & 4.83 \\ Turkish & tur & Turkic & 6.43 & 0.94 & 2.09 & 11.77 & 6.48 \\ Ukrainian & ukr & Indo-European & 6.7 & 0.78 & 1.78 & 10.82 & 5.87 \\ Urdu & urd & Indo-European & 5.34 & 0.64 & 0.66 & 11.17 & 3.9 \\ Uzbek & uzb & Turkic & 7.6 & 0.99 & 2.25 & 11.8 & 6.58 \\ Vietnamese & vie & Austrosiatic & 6.71 & 1.01 & 2.33 & 10.97 & 4.07 \\ Xhosa & xho & Niger-Congo & 9.78 & 1.27 & 2.91 & 12.96 & 7.19 \\ Yoruba & yor & Niger-Congo & 8.46 & 1.56 & 3.26 & 15.48 & 3.48 \\ Cantonese Chinese & yue & Sino-Tibetan & 5.56 & 0.93 & 2.07 & 12.31 & 3.96 \\ Zulu & zul & Niger-Congo & 11.05 & 1.31 & 3.03 & 17.3 & 7.23 \\ \end{tabular}
\end{table}
Table 12: Detailed statistics of Fleurs-Ipa. (Avg.Phones: average number of phonemes in each word; Avg.Dur.: average duration of each clip).