Beyond Orthography: Automatic Recovery of Short Vowels and Dialectal Sounds in Arabic (2024)

Abstract: This paper presents a novel Dialectal Sound and Vowelization Recovery framework, designed to recognize borrowed and dialectal sounds within phonologically diverse and dialect-rich languages, that extends beyond its standard orthographic sound sets. The proposed framework utilized quantized sequence of input with(out) continuous pretrained selfsupervised representation. We show the efficacy of the pipeline using limited data for Arabic, a dialect-rich language containing more than 22 major dialects. Phonetically correct transcribed speech resources for dialectal Arabic is scare. Therefore, we introduce ArabVoice151, a first of its kind, curated test set featuring 5 hours of dialectal speech across 15 Arab countries, with phonetically accurate transcriptions, including borrowed and dialectspecific sounds. We described in detail the annotation guideline along with the analysis of the dialectal confusion pairs. Our extensive evaluation includes both subjective – human perception tests and objective measures. Our empirical results, reported with three test sets, show that with only one and half hours of training data, our model improve character error rate by ≈ 7% in ArabVoice15 compared to the baseline.

heading: 1 Introduction
text: Self-supervised learning (SSL) paradigm has transformed speech research and technology, achieving remarkable performance (Baevski et al., 2020; Chen et al., 2022) while reducing the dependency on extensively annotated datasets (Radford et al., 2023). The SSL models excel at discerning the underlying acoustic properties in both frames and utterance level (Pasad et al., 2021, 2023; Chowdhury et al., 2023) irrespective of language. Phonetic information is sailent and preserved even when
∗ These authors contributed equally to this work. + Corresponding author.
1https://github.com/QCRIVoice/ACL-DVR
these continuous representations are mapped to a finite set of codes via vector quantization (Hsu et al., 2021a; Sicherman and Adi, 2023; Wells et al., 2022; Kheir et al., 2024). This allows the learning paradigm to leverage unlabeled data to discover units that capture meaningful phonetic contrasts.
Leveraging insights from acoustic unit discovery (Park and Glass, 2008; Versteegh et al., 2015; Dunbar et al., 2017; Eloff et al., 2019; Van Niekerk et al., 2020), unsupervised speech recognition (Baevski et al., 2021a; Da-Rong Liu and shan Lee, 2018; Chen et al., 2019; Da-rong Liu and yi Lee, 2022; Baevski et al., 2021b), and phoneme segmentation (Kreuk et al., 2020; Bhati et al., 2022; Dunbar et al., 2017; Versteegh et al., 2015) have utilized quantized discrete units for various purposes. These include (i) pretraining the SSL model (Baevski et al., 2020; Hsu et al., 2021a), (ii) employing acoustic unit discovery as a training objective (van Niekerk et al., 2020), and (iii) utilizing discrete labels for training phoneme recognition and automatic speech recognition (Chang et al., 2023; Da-rong Liu and yi Lee, 2022; Da-Rong Liu and shan Lee, 2018; Sukhadia and Chowdhury, 2024).
Inspired by previous research, we employ SSL representations and vector quantization to recognize acoustic units in phonologically diverse spoken dialects, extending beyond their standard orthographic sound sets. We introduce a simple yet potent network leveraging SSL and a discrete codebook to recognize these non-orthographic dialectal and borrowed sounds with minimal labeled data.
Arabic is an appropriate language choice for the task. The language has a rich tapestry of dialects, each with its unique characteristics in phonology, morphology, syntax, and lexicon (Ali et al., 2021). These dialects2 differ not only among themselves
2There are 22 Arab countries, and typically, there is more than one dialect spoken in each Arab country (ex: rural versus urban areas)
13172
but also when compared to Modern Standard Arabic (MSA). While MSA prevails in official and educational domains, Dialectal Arabic (DA) serves as the means for daily communication. The diversity in pronunciation and phoneme sets for DA goes beyond standardized MSA sound sets. Moreover, to add to the challenges, DA follows no standard orthography. Therefore, despite the abundance of DA speech data in online platforms, accurately (phonetically correct) transcribed resources are scarce, categorizing DA among the low-resource languages.
To bridge this gap, we introduce the Arabic “Dialectal Sound and Vowelization Recovery” (DSVR) framework. The proposed framework exploits the frame-level SSL embeddings and quantizes them to create a handful of discrete labels using k-means model. These discrete labels are then fed (can be in combination with SSL embeddings) as input to a transformer-based dialectal unit and vowel recognition (DVR) model.
We show its efficacy for (a) dialectal and borrowed sound recovery; and (b) vowelization restoration capabilities with only 1 hour 30 minutes of training data. We introduced Arabic dialectal test set – “ArabVoice15”, a collection of 5 hours of dialectal speech and verbatim transcription with recovered dialectal and borrowed sounds from 15 Arab countries. For vowelization restoration, we tested on 1 hour of speech data, sampled from CommonVoice-Ar (Ardila et al., 2019), transcribed by restoring short vowels. Our paper describes the phonetic rules adopted, special sounds considered along with detailed annotation guidelines for designing these test sets. Furthermore, we evaluate the quality of the intermediate discrete labels using human perceptual evaluation, in addition to other purity and clustering-based measures.
We observed that these discrete labels can capture speaker-invariant, distinct acoustic, and linguistic information while preserving the temporal information. Consequently, encapsulating the discriminate acoustic unit properties, which can be used to recover dialectal missing sounds. Our empirical results suggest that DSVR can exploit unlabeled data to design the codebook and then with a small amount of annotated data, a unit recognizer can be trained.
Our contribution involves: (i) Proposed Arabic Dialectal Sound and Vowelization Recovery (DSVR) framework to recognize dialectal units and restore short vowels; (ii) Developed anno-
tation guidelines for the verbatim dialectal transcription; (iii) Introduced and benchmark ArabVoice15 test set – a collection of dialectal speech and phonetically correct verbatim transcription of 5 hours of data. (iv) Released a small subset of CommomVoice - Arabic (Ardila et al., 2019) data with restored short vowels, dialectal and borrowed sounds. This study addresses the crucial challenge of identifying and understanding these phonetic intricacies, acknowledging their essential role in improving the performance of speech processing applications like dialectal Text-to-Speech (TTS) and ComputerAssisted Pronunciation Training applications. To the best of our knowledge, this study is the first to attempt to automatically restore vowels, borrowed and dialectal sounds for rich spoken dialectal Arabic language with very limited amount of data. Moreover, the study also introduce the very first dialectal testset with phonetically correct transcription representation.

heading: 2 Arabic Sounds
text: The exploration of phonotactic variations across Arabic dialects, including MSA and other regional dialects offers a rich field of study within the domain of Arabic linguistics. These variations are not merely lexical, but phonetic and in many cases deeply embedded in the phonological rules that dictate the permissible combinations and sequences of sounds within each dialect (Biadsy et al., 2009).

heading: 2.1 Related Studies
text: Limited research investigated dialectal sounds in Arabic transcribed speech. Vergyri and Kirchhoff (2004) deployed an EM algorithm to automatically optimize the optimal diacritic using acoustic and morphological information combination. Al Hanai and Glass (2014) employed automated text-based diacritic restoration models to add diacritics to speech transcriptions and to train speech recognition systems with diacritics. However, the effectiveness of text-based diacritic restoration models for speech applications is questionable for several reasons, as demonstrated in Aldarmaki and Ghannam (2023), they often fail to accurately capture the diacritics uttered by speakers due to the nature of speech; hesitation, unconventional grammar, and dialectal variations. This leads to a deviation from rule-based diacritics. Recently, Shatnawi et al. (2023) developed a joint text-speech model to in-
corporate the corresponding speech signal into the text based diacritization model.
Grapheme to Phoneme (G2P) has been studied thoroughly by many researchers across multiple languages. Recent approaches in G2P include data-driven and multilingual (Yu et al., 2020; Garg et al., 2024) mapping from grapheme sequence to phoneme sequence. However, previous work in Arabic G2P is comprised of two steps: (i) Grapheme to vowelized-grapheme (G2V) to restore the missing short vowels and (ii) Vowelizedgrapheme to phoneme sequence (V2P). The first step is often statistical and deploys techniques like sequence-to-sequence; for example studies like Abdelali et al. (2016); Obeid et al. (2020) are used widely for restoring the missing vowels in Arabic. The second step is relatively one-to-one and can be potentially hand-crafted rules for MSA as well as various dialects, refer to Biadsy et al. (2009); Ali et al. (2014) for more details. MSA Arabic speech recognition phoneme lexicon can be found here3
The distinction between MSA and regional dialects is nuanced; viewing them as separate is oversimplified. Arabs perceive them as interconnected, leading to diglossia, where MSA is for formal contexts and dialects for informal ones, yet with significant overlap and blending (Chowdhury et al., 2020a). Chowdhury et al. (2020b) studied dialectal code-switching in the manually annotated Egyptian corpus. The corpus was annotated for both MSA and Egyptian dialect labels per token, considering both the linguistic and the acoustic cues. The findings indicate the complex overlapping characteristics of the dialectal sound units showing roughly 2.6K Egyptian sounding words with respect to 9.3K MSA and 2.3K mix of both.

heading: 2.2 MSA and Dialectal Phonlological Variations
text: Arabic dialects exhibit phonological differences when compared to MSA, these differences might be noted across various aspects of pronunciation and phonology, such as consonants, vowels, and diphthongs. It’s suggested that Arabic generally encompasses around 28 consonants, alongside three short vowels, three long vowels, though these numbers could vary slightly depending on the dialect in question. The consonant pronunciation of H [θ], X [D],   [DQ], h. [dý], [dQ], and
[q] cover most of the variations across Arabic dialects. Here are
3https://catalog.ldc.upenn.edu/LDC2017L01
some examples of phones that vary between MSA and various Arabic dialects.
• Interdental Consonants: In particular H [θ]/ X [D] found in MSA are pronounced differently. For example, in Egyptian Arabic, they are often pronounced as [s].
• The voiceless stop constant [q] is a good example across Arabic dialects, In many cases, it will be pronounced as glottal stop Z [P] in Egyptian dialect and voiced velar h. [dý] in Gulf and Yemeni dialects. • Long and short vowels might exhibit a reduction in duration or even drop in duration in various dialects. In some dialects, the difference between long and short vowels may be subtle to notice. • The difference in stress between Arabic dialects can lead to different meanings. The phonological differences and examples mentioned above do not cover all variations but highlight several distinctions between Arabic dialects and MSA. A depiction of certain MSA sound variations is presented in Appendix A.1.

heading: 3 Methodology
text: Figure 1 gives an overview of our proposed Dialectal Sounds and Vowelization Restoration Framework. The goal of the pipeline is to recover (verbatim) dialectal sound and short vowel units, using frame-level representation. Given an input speech signal X = [x1, x2, · · · , xT ] of T frames, the frame-level representation (Z) is first extracted from a multilingual SSL pretrained model.
We subsampled frame-level vectors (Z̃ ⊂ Z) to train a simple Vector Quantization (VQ) model using k-means for getting a Codebook Ck, with k categorical variables. Each cluster, in the codebook, is then associated with a code Qki and a centroid vector Gki . Using the Ck codebook, we infer the discrete sequences codes Ẑ corresponding to the input Z. Ẑ is the input of our Dialectal Units and Vowel Recognition (DVR) module.

heading: 3.1 Pretrained Speech Encoder
text: The XLS-R4 model is a multilingual pre-trained SSL model following the same architecture as wav2vec2.0 (Baevski et al., 2020). It includes a CNN-based encoder network to encode the raw
4https://huggingface.co/facebook/wav2vec2-large-xlsr53
audio sample and a transformer-based context network to build context representations over the entire latent speech representation. The encoder network consists of 7 blocks of temporal convolution layers with 512 channels, and the convolutions in each block have strides and kernel sizes that compress about 25ms of 16kHz audio every 20ms. The context network consists of 24 blocks with model dimension 1024, inner dimension 4096, and 16 attention heads.
The XLS-R model has been pre-trained on around 436, 000 hours of speech across 128 languages. This diverse dataset includes parliamentary speech (372, 000 hours in 23 European languages), read speech from Multilingual Librispeech (44, 000 hours in 8 European languages), Common Voice (7, 000 hours in 60 languages), YouTube speech from the VoxLingua107 corpus (6, 600 hours in 107 languages), and conversational telephone speech from the BABEL corpus (≈ 1, 000 hours in 17 African and Asian languages).
We opt for the large XLS-R (1B parameters). Our preliminary analysis revealed limitation in the XLS-R in differentiating between acoustic sounds, such as X [d]/ [dQ] and H [t]/   [tQ] present in MSA and DA. Consequently, we primed the model towards Arabic sounds by finetuning with 13 hours clean avaliable MSA data (Ardila et al., 2019) for ASR task. We restricted the training to 5 epochs to prevent the risk of catastrophic forgetting of the pretrained representation (Goodfellow et al., 2013).

heading: 3.2 Vector Quantization
text: Vector Quantization (Makhoul et al., 1985; Baevski et al., 2020) is a widely used technique for approximating vectors or frame-level embeddings through a fixed codebook size. In our Vector Quantization (VQ) modules (see Figure 1), we pass forward a sequence of continuous feature vectors Z = {z1, z2, . . . , zT } and then assign each zt to its nearest neighbor in the trained codebook, Ck. In other words, each zt is replaced with the code Qki ∈ Ck assigned to the centroid Gki . The resultant discrete labels are quantized sequence Ẑ = {ẑ1, ẑ2, . . . , ẑT }. These labels are expected to facilitate better proninciation learning and incorporate distinctive phonetic information in the subsequent layers.
Training the Codebook For quantization, we utilized the k-means clustering model. We selected a random subset of frame-level representation for
training the cluster model. Moreover, to select wide varieties of sound unit, we forced-aligned the available/automatic transcription of the datasets (see Section 5.1) with a GMM-HMM based ASR models. Using the timestamps, we then select SSL frame representations that aligned with wide varieties of sound labels.5 We trained the codebook for different k = {128, 256, 512}

heading: 3.3 Dialectal Units and Vowel Recognition (DVR) Model
text: We explored two variants of DVR – discrete and joint Model (as seen in Figure 2). The discrete DVR takes only the discrete Ẑ labels from the VQ as input, where as the joint module concatenate both the Ẑ and Z inside the subsequent layer. The resultant embeddings (for both model) are then passed to the transformer layers and the head feedforward layer. The DVR model is optimized with character recognition objective to identify arabic units.

heading: 3.4 Baseline
text: As baseline, we used the frozen frame-level representation from the XLS-R model to pass to the feedforward layer followed by the transformers and output head. The architecture uses similar encoder as the DVR model (see Figure 2 Baseline). For brevity, we reported with the results of the second architecture (SSL frame-level representation with transformer-based encoder) as the baseline of the paper.

heading: 4 ArabVoice15 Dataset
text: Spoken DA remains a low-resource language primarily due to the scarcity of transcription that can faithfully capture the diverse regional and borrowed sounds in the standard written format. Such lack of data posses significant challenge for speech and linguistic research and evaluation. In this study, we address this challenge by designing and developing ArabVoice15 test set. Furthermore, we have also enhanced a subset of the existing Arabic Commonvoice (Ardila et al., 2019), Ar:CVR dataset with restored vowels, borrowed and dialectal sounds. In the following sections, we will discuss the datasets, preprocessing steps along with in detail annotation guidelines.
ArabVoice15 is a collection of 5 hours of speech utterances randomly selected from testset of ADI17
510k sample frames for each sound label.
(Ali et al., 2019) dataset, widely used for dialect identification task. For the ArabVoice15, we selected a total of 2500 utterance, ≈ 146(±3.6) utterance from each of the 15 Arab countries including: Algeria (ALG), Egypt (EGY), Iraq (IRA), Jordan (JOR), Saudi Arabia (KSA), Kuwait (KUW), Lebanon (LEB), Libya (LIB), Morocco (MOR), Palestine (PAL), Qatar (QAT), Sudan (SUD), Syria (SYR), United Arab Emirates (UAE), and Yemen (YEM). The average utterance duration: 7-8 seconds. As for Ar : CV R, we randomly extracted 21.38 hours from the Ar:CV trainset, which we then mannually annotated at both verbatim and vowelized level (test ≈ 1hr).
Data Verbatim Pre-Processing We present a set of rules employed for data normalization, aiming to reduce annotators’ tasks through a rule-based phonemic letter-to-sound approach in Arabic, as
detailed in (Al-Ghamdi et al., 2004). For vowelization, we initially applied diacritization (aka vowelization or vowel restoration) module present in the Farasa tool (Abdelali et al., 2016). We then applied the following rule-based phonemic letterto-sound function to our dataset. This step also removed any Arabic letters that are not traditionally pronounced in spoken conversation. • For @ [a:] : (i) If it appears within a word (not
at the beginning) and is followed by two consonants, we delete it. For example, H. A JºË@ I. J» [ktb a:lktb] becomes H. A JºË I. J» [ktb lktb]. (ii) If it occurs at the beginning in the form of the definite article È@, we replace it with [Pa]. For example, ÕÎªÖÏ @ [a:lmQlm/] becomes ÕÎªÖÏ Z [PalmQlm].
• For È [l] : We removed the Shamsi (Sun) [l], that refers to [l] in È@ followed by a Sun conso-
nant6 ¡¢ PP XY J JË. For example: àAÔgQË@ [a:lrèman] becomes àAÔgP@ [a:rèman]
• For @, we replaced it wherever it occurred in the
text with @ Z [Pa:]. • For Hamza shapes ( ø @ ð @ Z), we normalized
them to Z [P]. • For ø @, we normalized them to @ [a:/]. • For Tanwin diacritics ( @ @ @ [/un/, /in/, /an/]) at
the end of a phrase, we replaced it with a short vowel, and elsewhere, we turned it into à @, à @, à@
[/un/, /in/, /an/] to match the typical verbatim sounds.
Annotation Guideline We gave extensive training to an expert transcriber, a native speaker from Egypt, to provide the written form for each word and its verbatim transcription. For example, if the word is Õ Î ̄ [qalam] (pen), and the speaker said Õ Î ¿
[kalam], then the transcriber writes [qalam/kalam]. This is the summary of the annotation guidelines: • For sounds that are not in MSA and have been
borrowed from foreign languages, the following special letters7 are used:
6In Arabic grammar, there are two categories of letters: "sun letters" éJ Ò Ë@
¬ðQmÌ'@ and "moon letters" éK QÒ ®Ë@
¬ðQmÌ'@. These categories affect the pronunciation of the Arabic definite article È@ (al-). Sun letters are those Arabic letters that cause assimilation ÐA «XB @ of the definite article È@ (al-) when they are prefixed to nouns, meaning the "l" sound of "al-" merges with the initial consonant of the noun. The assimilation occurs in pronunciation, but not in writing.
7The special letters used in the annotation process do not belong to the Arabic alphabet; instead, we borrowed them
– h [g] as in the word Ég. ñk. “google” which is written as Ég. ñk. [ju:jl] / Ég ñk [gu:gl]. – ¬ [v] as in the word ñK YJ ̄ “video” which is written as ñK YJ ̄ [fi:dyu:] / ñK YJ ̄ [vi:dyu:]. – H [p] as in the word ø @Q . @ “spray” which is written as ø @Q . [sbra:y] / ø @Q [spra:y]. • For dialectal sounds that are missed in MSA, the following special letters are used: – À (Gulf /Qaf/) as in the word ÈAÆ« which is
written as ÈA ®« / ÈAÆ«. – The Egyptian/Syrian/Lebanese [q] is pro-
nounced mostly as Z [P] as in ÈĀ [qa:l] / È@Z [Pa:l]. –   (Egyptian/Lebanese /Z/) as in the wordQê ¢J K. is written as Qê ¢J K. / Qê ¢J K. .
There are few words with special spellings that do not precisely reflect their pronunciation. In these cases, the transcriber writes both, as in the word @ Yë [hadha] / @ XAë (/ha:dha/). Numbers and some special symbols (ex: the percentage sign %) are written in letters and are being judged according to speakers’ pronunciation.
Quality Control: Detection of possible annotation errors was done automatically and doubtful cases were returned to the transcriber for review. In addition, a manual inspection of random sentences (10%) from each file was performed. Any file below 90% accuracy was returned for full correction.
from Farsi sharing similar Arabic shapes, these letters were employed to represent distinct dialectal sounds.

heading: 5 Experimental Design

heading: 5.1 Training Datasets and Resources
text: Datasets: Unspervised Codebook Generation To train the codebook, we randomly selected utterances from publicly available resources. For Arabic sounds, we opt for utterances from official CommonVoice train set along with Arabic TTS data. Moreover, to add borrowed/special sounds missing in MSA phonetic set (e.g., /g, v, p/), we included publicly available English datasets like LibriSpeech (Panayotov et al., 2015), and TIMIT (Garofolo et al., 1993). For the subsampling process, we opt for hybrid ASR systems8 for Arabic and Montreal Forced-Aligner9 for the English.
Datasets: Spervised DVR Model To train the DVR model, we opt for a small training dataset to showcase our the efficacy of our proposed framework in low-resource setting. The details of dataset used for DVR is presented in Table 1. For the training, we utilize dataset transcribed with restored vowels, borrowed and dialectal sounds. We used 1 hour 30 minutes of training data in this study.

heading: 5.2 Model Training
text: The Models, presented in Figure 2, are optimized using Adam optimizer for 50 epochs with an early stopping criterion. The initial learning rate is 1× 10−4, and a batch size of 16 is employed. The loss criterion is CTC loss, utilized for predicting verbatim sequences. The input dimension for the SSL frame-level representation is d = 1024, the dimension of the discrete labels d = k. For all the architectures in Figure 2, the dimension of feedforward (FF) layer is d = 512. For the DVR joint, the output from the FFs (d̂, e) are concatenated to form [d̂, e] of dimension d = 1024. These outputs are then passed to 2 transformer encoders each with 8 attention heads. Following, the encoded information is then projected to output head of dimension V = 39 equivalent to the characters supported by the models. The total number of trainable parameters are Baseline:7.634M; DVR discrete:7.110M; and joint: 33.346M.

heading: 5.3 Evaluation Measures
text: We used Davis-Bouldin index (DBindex) to select the k value for our codebook. The DBindex is
8Trained on Arabic CommonVoice 9https://github.com/MontrealCorpusTools/
Montreal-Forced-Aligner.git
widely used in clustering performance evaluation (Davies and Bouldin, 1979), and is characterized by the ratio of within-cluster scatter to between-cluster separation. A lower DBindex value is better, signifying compact clustering. Following, we adapted the approach of (Hsu et al., 2021b) to evaluate the codebook quality using Phone Purity, Cluster Purity, and Phone-Normalized Mutual Information (PNMI). These measures use frame-level alignment of characters with discrete codes assigned to each frame. Phone purity measures the average framelevel phone accuracy, when we mapped the codes to its most likely phone (character) label. Cluster purity, indicates the conditional probability of a discrete code given the character label. PNMI measures the percentage of uncertainty about a character label eliminated after observing the code assigned. A higher PNMI indicates better quality of the codebook. Moreover, we assessed the codebook quality by human perception tests as mentioned in the following section. As for evaluating the dialectal sounds and short vowel recognition model, we reported Character Error Rate (CER) with and without restoring short vowels.
Human Perception Test Setup We performed cluster quality analysis for k = {128, 256, 512} following the steps of (Mao et al., 2018; Li et al., 2018). For our study, we defined each clusters (demoted by a code) as either Clean or Mix. Clusters are considered as Clean when 80% of its instances are matched to one particular character, where as for Mix clusters, the instances are mapped to different characters.10 We hypothesise that the Mix clusters represent examples which can resembles closely to either two of canonical sound unit /l1/ and /l2/, or a mix of both /l1_l2/. We randomly selected 52 examples from each perceived Mix Clusters. We asked the four annotators (2 native and 2 non-native Arabic speakers) to categorize it into these four classes: more similar to /l1/, more similar to /l2/, a mix of both, or neither.

heading: 6 Results and Discussion
text: Number of discrete codes in Codebook We reported the DBindex for the codebook sizes k = {128, 256, 512} in Table 2. We observed lower DBindex with k = 256 indicating better codebook quality. We further evaluated the codebook quality and reported purity measures with the Ar:CVR test-
10Only characters above 20% frequency are considered.
set only for brevity and CER with all the testsets. Our CER results shows the efficacy of the selected k = 256 for most of the test sets. We observed that increasing codebook size improves the purity and the PNMI. We noticed, the gain in cluster stability between k = 256 vs k = 516 is not very large with respect to the performance and computational cost. Hence we selected the codebook C of size k = 256 for all the experiments.
Perceptual test of Codebook We averaged annotator judgments across four categories for all Mix clusters, revealing no clear majority and highlighting the listeners’ difficulty in categorically labeling audio within these clusters. In aligned with Mao et al. (2018); Li et al. (2018), we also conclude that these mixed labels genuinely exist and cannot be precisely characterized by any conventional given label. We present some of our findings of the perceptual test in Figure 3 for 5 different Mix clusters with average judgment per category.
Dialectal Unit Recognition Performance We reported the performance of the proposed DVR discrete and joint model in Table 3 for borrowed and dialectal unit recognition task. Our results shows the efficacy of the DVR models over the baseline specially for dialectal test sets (ArabVoice
and EgyAlj). We observed for borrowed and dialectal unit recognition, the discrete model outperforms the joint model significantly. Breakdown of the performance for 15 countries are presented in Appendix A.2.
Impact of Training Data size Table 3 also shows the impact of the training data size. We observed for dialectal unit recognition, our DVR discrete model outperforms the other two models significantly with limited data sets of {1hr30min, 3hr30min, 5hr30min}. We see an improvement in performance from 1hr30min to 3hr30min settings. However, beyond a certain data threshold, the improvements plateaued.
Performance for short vowel restoration For short vowel restoration (in Table 4), we observed that the added frame-level embeddings (in DVR
joint) improve the recognition performance. We also observed that the baseline model performs comparably with DVR joint. This indicates that the restoration of short vowels benefits from high dimensional fine-grained information compare to using few discrete codes. We also compared the CER with Farasa – state-of-the-art text-based dicretization tool (Abdelali et al., 2016). We observed the acoustic models outperform Farasa by a large margin, especially for common voice subset. However, Farasa excelled in formal content – news content presented in EgyAlj testset.

heading: 7 Conclusion
text: In this study, we propose a novel dialectal sound and short vowel recovery framework that utilizes a handful of discrete codes to represent the variability in dialectal Arabic. We also observed with only 256 discrete labels, the borrowed and dialectal sound recognition model outperforms both baseline and joint (discrete code with frame-level SSL representation) models by ≈ 7% CER improvement. For restoring vowels, we noticed SSL embeddings play a bigger role. Our findings indicate the efficacy of the discrete model with small training datasets. To foster further research in dialectal Arabic, we introduced, benchmarked, and released ArabVoice15 – a dialectal verbatim transcription dataset containing utterances from 15 Arab countries. In the future, we will apply the framework to more dialects and other dialectal languages.
Limitations
The diversity of representation and the size of ArabVoice15 could limit the conclusion to generalize in all Arabic dialects due to variability in dialectal sounds. Although the annotator was an expert transcriber and received extensive training, their dialect may have led to some bias in judgment.
Ethics Statement
For the research work presented in this paper on the Dialectal Sound and Vowelization Recovery (DSVR) framework, we have adhered to the highest ethical standards. All the speech/audio data used in this study were already publicly available. The human perception tests for our evaluation process were designed with a commitment to fairness, inclusivity, and transparency. The participants were selected keeping in mind balancing gender and nativity. Listeners were fully briefed on the nature
of the research and their rights as participants, including the right to withdraw at any time without consequence. However as we mentioned in the limitation section, we cannot guarantee any human bias toward any dialectal sound or preference.

heading: A Appendix
text: A.1 Sound Analysis In Figure 5, we have depicted potential confusion between specific sounds in MSA and Arabic dialects. Utilizing a Hidden Markov Model-Time Delay Neural Network (HMM-TDNN) model11,
11https://kaldi-asr.org/models/m13
trained with MGB-2 (Ali et al., 2016) for Arabic, we aligned randomly selected samples from the original datasets of CommonVoice Arabic and EgyAlj. For the English dataset TIMIT, we used the provided ground truth alignment.
After aligning speech signals with their original unvowelized character-based transcriptions, we matched frame-level features extracted from XLSR (see Section 3.1) with their corresponding characters. In Figure 5.A, we randomly selected 1000 samples associated with P [z] and 1000 samples associated with X [D] from CommonVoice Arabic. Despite CommonVoice Arabic being considered as clean MSA speech data with good pronunciation, we observed that some samples of X [D] were clustered with P [z], primarily explained by the speakers getting influenced by their dialectal variations, as discussed in Section 2.
Figure 5.B displays the selection of three characters: H [t], è [t/h], è [h]. Notably, è is at times pronounced as [t] and at other times as [h]. Although rule-based methods (Halabi and Wald, 2016) can predict when it will correspond to which sound, applying these rules in everyday spoken language, where people don’t follow rule based pronunciation, proves challenging. The figure reveals two main clusters for [t] and [h], with vectors associated with è scattered between these clusters, highlighting the aforementioned point.
Figure 5.C illustrates the selection of four labels: Arabic h. [ [dý], and English phonemes (zh, g, jh) [ý, g, dý]. We selected 1000 Arabic samples of h. from CommonVoice Arabic and EgyAlj, along with 500 samples for each of the English phonemes. It became apparent that the Arabic sound h. is distributed across different English pronunciations (zh, g, and jh), indicating dialectal variations in the pronunciation of h. .
A.2 Country-wise DVR performance
In this section, we present the aforementioned results discussed in Section 6. Figure 4 displays CER results for the Baseline (Z), SVR Discrete (k:256), and DVR joint (Z+k:256) models trained on 1H30min of data, tested on AraVoice15. We analyze the CER results for each dialect individually. Our observations reveal that SVR Discrete (k:256) and DVR joint (Z+k:256) consistently outperform the Baseline (Z) across all dialects, exhibiting a sub-
stantial performance gap in MOR, YEM, PAL, and IRA dialects. Moreover, SVR Discrete (k:256) and DVR joint (Z+k:256) exhibit similar performance across the majority of the 15 dialects (10/15), with notable disparities observed in JOR, SUD, SYR, where a discernible performance gap is evident.

heading: Future Work
text: In the future, we will apply the framework to more dialects and other dialectal languages