Data Augmentation for Speech-Based Diacritic Restoration (2024)

Abstract: This paper describes a data augmentation technique for boosting the performance of speechbased diacritic restoration. Our experiments demonstrate the utility of this approach, resulting in improved generalization of all models across different test sets. In addition, we describe the first multi-modal diacritic restoration model, utilizing both speech and text as input modalities. This type of model can be used to diacritize speech transcripts. Unlike previous work that relies on an external ASR model, the proposed model is far more compact and efficient. While the multi-modal framework does not surpass the ASR-based model for this task, it offers a promising approach for improving the efficiency of speech-based diacritization, with a potential for improvement using data augmentation and other methods.

heading: 1 Introduction
text: The Arabic Language is one of the official international languages (al Yamin, 2023) ranking as the sixth language in world organizations, the third in the Organization of African Unity, and the first in the Islamic World (Bakalla, 2023). Arabic is the native language of more than 400 million people in Arabic countries and is spoken by more than 5 million speakers beyond the Arab world (Bakalla, 2023). The Arabic script consists of 28 basic alphabets, representing consonants and long vowels. As short vowels are not represented in the alphabet or the basic construction of words, diacritics can be employed to indicate them1.These diacritics are dropped from almost all written text, except documents with pronunciation accuracy that cannot be compromised, such as religious texts or children’s books. As a result, Arabic text corpora used for building machine learning models
1For instance, the undiacritized word Q . Ë @ can be diacritized as Q . Ë @ meaning ‘land’, Q . Ë @ meaning ‘benevolence’, and
Q . Ë @ meaning ‘wheat’
are typically undiacritized. However, studies have shown that including diacritics can lead to substantial improvements in various applications. For instance, Abbache et al. (2023) demonstrated that diacritized word embeddings significantly outperform non-diacritized versions. Similarly, diacritics enhance the quality of speech datasets used in tasks like Text-to-Speech (TTS) and Automatic Speech Recognition (ASR). By incorporating diacritics, these systems can generate more natural-sounding speech, improve pronunciation accuracy, and create more precise transcriptions (Aldarmaki and Ghannam, 2023).
Transcribed speech datasets used for building ASR models are typically undiacritized. As a result, building diacritized models requires a separate diacritization step, which can be performed on the transcripts using many popular text-based automatic diacritic restoration models like, ALISoft (URL, 2023), Farasa (QCRI, 2020), Mishkal (Zerrouki, 2020), and Camelira (Obeid et al., 2022). However, in a recent work, Shatnawi et al. (2024) discussed the degradation in performance of diacritic restoration models when applied to speech datasets, resulting in diacritic error rates ranging between 4-50%. They proposed a diacritic restoration framework that incorporates speech audio as another source of information. In particular, they utilize a pre-trained ASR model that is fine-tuned to predict diacritized transcripts as an additional source of input for diacritic restoration. This framework resulted in notable improvements in performance. However, the model relies on an ASR model as an extrinsic module, and therefore, its overall performance is constrained by the performance of the ASR model. In other words, the model proposed in Shatnawi et al. (2024) is not truly multi-modal. Another major limitation in this line of work is the scarcity of diacritized speech corpora that can be used for training both the ASR and diacritic restoration models.
160
Given the potential advantages of integrating speech features into the model, coupled with the scarcity of such data, we explore a data augmentation strategy to boost the training data volume. In this approach, we generate randomly diacritized inputs and utilize a text-to-speech (TTS) model to synthesize speech. We hypothesize that diacritic restoration models overfit to word structure and other textual patterns, so the proposed approach aims to randomize the relationship between diacritics and the underlying text to avoid this kind of overfitting. The proposed data augmentation enables the model to learn from a wide range of diacritic variations regardless of the underlying word structure and textual context to help improve generalization by enforcing the recognition of acoustic over textual patterns. In addition, we propose a multi-modal variant of the model in Shatnawi et al. (2024), where speech is used directly as input rather than relying on an extrinsic ASR model.
Our experiments show that the proposed data augmentation method improves diacritic recognition performance across all test sets, validating the hypothesis that models typically overfit to the textual context. While the proposed multi-modal architecture improved over the text-only baselines, it still lags behind the ASR augmented model described in Shatnawi et al. (2024). Nevertheless, the model offers comparable performance while being more integrated and efficient, demonstrating the first implementation of a multi-modal diacritic restoration model.

heading: 2 Related Work
text: Speech-based Diacritic Restoration: The majority of diacritization models primarily operate on text-based approaches, utilizing either rule-based methodologies (Fashwan and Alansary, 2016; Alansary, 2018; Pasha et al., 2014) or statistical methods (Fadel et al., 2019; Hifny, 2012; AlThubaity et al., 2020). Integration of speech data into automatic diacritization processes has been minimally explored in prior research. The most recent relevant work is Shatnawi et al. (2024), in which they proposed a diacritization framework that incorporates both text and audio to diacritize speech data sets. Specifically, they utilized a pre-trained Automatic Speech Recognition (ASR) model, which was fine-tuned on diacritized Arabic speech data to produce diacritized transcripts of speech utterances. These transcripts were uti-
lized as supplementary inputs for diacritic restoration models, consistently enhancing performance compared to text-only approaches. Their model’s performance is notably influenced by the quality and nature of the training data used to fine-tune the ASR model, directly impacting diacritic restoration performance.
Data Augmentation (DA): The scarcity of labeled data for supervised learning often results in poor model’s performance. To mitigate this issue, Data Augmentation (DA) methods are used to artificially expand the data by generating new examples from existing ones (Pellicer et al., 2023). The predominant DA techniques used in NLP include back-translation (Edunov et al., 2018; Yu et al., 2018; Xie et al., 2020), lexical substitution (e.g., synonym replacement, insertion, exchange, swap, and deletion) (Zhang et al., 2015; Wei and Zou, 2019), and paraphrasing (Iyyer et al., 2018; Kumar et al., 2020). Beyond unimodal tasks, Hao et al. (2023) recently introduced an innovative DA technique for generating semantically relevant imagetext pairs in which they interpolate existing images and concatenate their corresponding text descriptions, enriching the training dataset and enhancing the performance of multimodal models.
Despite the wide application of data augmentation in many NLP tasks, its implementation in diacritization remains notably limited. To the best of our knowledge, there has been no previously applied approach for data augmentation to strengthen and diversify diacritization models.
Multi-Modal Performance: Multi-modal models offer a significant advantage by enriching feature representation both in depth and breadth, as evidenced in previous applications (Joshi et al., 2021; Albalawi et al., 2023). However, optimal performance is not guaranteed solely through this approach. The effectiveness of multi-modality depends on factors such as the nature of the data and the specific task being addressed. For instance, in a study by Albalawi et al. (2023), two models were introduced for rumor detection: a unimodal model relying solely on textual features and a multi-modal model incorporating both textual and fusion features. The performance of the multi-modal model failed to outperform that of the text-based model.

heading: 3 Data Augmentation for Speech-Based Diacritic Restoration
text: Building on previous research on data augmentation and multi-modal models in natural language processing, we propose a novel approach that extends these concepts to address the specific challenges of speech-based diacritization. To address the challenge of data scarcity and enhance generalization, we investigate a data augmentation technique to supplement training data by creating a synthetic speech dataset. The main objective of integrating a speech dataset in this augmentation process is to leverage pronunciation features, precisely mapping inputs to the diacritized form, rather than solely relying on textual context for disambiguation. In this context-agnostic approach, our emphasis is on accurately utilizing pronunciation to map letters to their diacritized forms.
We augment the datasets through two main steps: (1) randomly generating diacritics for each letter in the text, and (2) converting the diacritized text to audio using a Text-to-Soeech (TTS) model. We applied this data augmentation approach to our training set (discussed in Section 5.1), doubling the size of the dataset used for training the diacritic restoration model.

heading: 3.1 Random Generation of Diacritics
text: The process begins by adding random diacritics to each letter in the dataset, followed by the application of predefined rules to ensure the resulting text remains pronounceable. Developing these rules involves the application of heuristics to ensure that the resulting random diacritics can be pronounced by native Arabic speakers. Table 1 showcases some examples of the resulting randomly diacritized ut-
terances.We extracted a sample from the dataset and conducted iterative manual analyses to identify patterns and characteristics of diacritized sentences that produce pronounceable words, thus informing the formulation of the heuristic rules. These rules fall into two main categories: replacement rules (Section 3.1.1) and deletion rules (Section 3.1.2))

heading: 3.1.1 Replacement Rules
text: Replacement rules involve substituting randomly generated, difficult-to-pronounce diacritics with new, randomly chosen ones, based on predefined rules derived from our manual analysis discussed earlier. Specifically, we replace the randomly generated diacritic if it meets one of the following conditions (See Table 2 for identifying Arabic diacritics):
• Sukoon or Shaddah if they appear at the first letter of the word (e.g., Yª ® Ó );
• Tanween if it appears in any letter except the last letter in the word;
• One of the two Shaddahs appearing on two contiguous characters (e.g., Y ª ® Ó );
• One of the two Sukoons appearing on two contiguous characters (e.g., Y ª ® Ó);
• Any diacritic that is not Fatha or Damma appearing on Hamza on top ( @) at the beginning
of a word (example of allowed variations, in
this case, iJ. @ or iJ. @);
• Any diacritic that is not Kasra appearing on Hamza below Alef ( @ ) such as the word úÍ@ ;
• Any diacritic that is not Fatha before the tied T ( è) (e.g, the Arabic word é PY Ó). Fatha is implicitly pronounced so we cannot replace it with any other diacritics;
• Any diacritic other than Fatha before the letter Alef of the following forms: ( ø ) or ( @ );
• Stand-alone Shadda should be followed by another diacritic.

heading: 3.1.2 Deletion Rules
text: Deletion rules involve removing invalid diacritics altogether. Specifically, we delete the randomly generated diacritic if it meets one of the following conditions:
• All diacritics placed on the characters that are not in the Arabic alphabet;
• All diacritics applied to the following forms of Alef: Alef Madd ( @), Alef ( @), Maqsura (ø),
and at the beginning of a word (Alef followed by the letter Lam) indicating the definiteness of a word (È@).
• Any additional diacritic for each letter, except if this additional diacritic accompanies Shaddah (i.e., each letter should have only one diacritic except in the case of Shaddah which can be followed by an additional diacritic).
Table 1 shows some examples of the resulting text, after applying the rules above. Note that while the words are no longer valid, they are all pronounceable.

heading: 3.2 Converting Diacritized Text to Audios
text: To train our model, we need both audio and their associated transcripts. In the first step described in the previous section, we obtained diacritized transcripts (randomly generated); now we need to generate the corresponding audio for each transcript. To that end, we employed the Google Cloud Text-to-Speech AI API2. We utilized a single voice, namely ar-XA-Wavenet-D, for this purpose. The generated audios are then integrated and utilized for training the diacritic restoration model.
2cloud.google.com/text-to-speech?hl=en

heading: 4 Multi-Modal Diacritic Restoration
text: The architecture for speech-based diacritic restoration proposed in Shatnawi et al. (2024) relies on an external ASR model (i.e. Whisper (Radford et al., 2023)) used to first process the audio and generate transcriptions. Although the model significantly reduced diacritic error rates, its reliance on an external system increases computational demands, in addition to the resources already required for processing ASR transcripts in the diacritic restoration model. To create a more efficient model, we propose to directly use the speech as an input to a mutli-modal architecture. We adopt the architecture proposed in Shatnawi et al. (2024) for the text side and the cross-attention mechanism, but we change the speech processing module. More concretely, the architecture (depicted in Figure 1) consists of a sequence labeling model to process the undiacritized text input and a speech feature extraction model. The two sides are then combined using a cross-attention mechanism, with the query vectors obtained from the text side and the key/value vectors from the speech side. The text sequence labeling model can be any sequence labeling architecture, such as Transformer or LSTM, as described in Shatnawi et al. (2024).
For the speech feature extraction, we use a pretrained self-supervised model to account for the small size of our training set. Our aim is to capitalize on the strengths of both modalities to enhance diacritic restoration accuracy and eliminate the dependency on an extrinsic ASR model for a more streamlined and efficient process. We used the self-supervised XLS-R model (Hsu et al., 2021), a large-scale multi-lingual pre-trained model for speech feature extraction. Trained on a vast dataset comprising 436K hours of unlabeled speech, XLSR adopts the wav2vec 2.0 objective (Schneider et al., 2019) across 128 languages. This objective involves learning contextual representations of speech by predicting masked portions of the audio waveform. The model is typically fine-tuned for downstream tasks such as speech translation, ASR, or speech classification. We first fine-tuned the model for ASR using our speech training set, which is done by adding a linear layer with CTC loss. The fine-tuning ensures that the model’s inner representations are already geared toward the representation of Arabic sounds.

heading: 5 Experimental Settings

heading: 5.1 Datasets
text: We replicate the experimental settings used in Shatnawi et al. (2024), including the data sets used for training and testing3. In particular, we use the Classical Arabic Text-To-Speech Corpus ClArTTS (Kulkarni et al., 2023). The ClArTTS dataset comprises audio recordings of classical Arabic speech accompanied by manually diacritized and verified transcripts. It consists of approximately 12 hours of recorded speech from a single male speaker, totaling around 10,000 relatively short utterances. Additionally, there is a separate subset of approximately 30 minutes reserved for testing purposes. For experiments involving text-only models, we utilize the cleaned Tashkeela Corpus, which encompasses a vast collection of Arabic texts predominantly from Classical Arabic literature and religious texts, along with a smaller portion in the Modern Standard Arabic variety. This corpus contains 2.3 million words distributed across 55,000 lines and is derived from the original Tashkeela corpus introduced by (Zerrouki and Balla, 2017). For Testing, we also use a small manual diacritic dataset, proposed for TTS purposes and derived from broadcast news (Baali
3The experiments are available at https://github.com/ SaraShatnawi/Diacritization.git
et al., 2023). This dataset (dubbed QASR TTS)4 contains one hour of speech by a male speaker and one hour of speech by a female speaker.

heading: 5.2 Model Setup
text: We used the Adam optimizer (Kingma and Ba, 2014) for learning and used 0.2 for the dropout rate5. The text-side of the multi-modal model is first pre-trained on the Tashkeela corpus. The complete multi-modal model is trained on the ClArTTS corpus only. For the XLS-R side, we used the version trained on 53 languages, which includes Arabic6. We fine-tuned it first using ASR with CTC loss on the ClArTTS corpus. Training is configured with hyper-parameters, including a learning rate of 3e-4 and 30 epochs, leveraging mixed precision (FP16) for faster training. After fine-tuning XLSR, we integrated the model into the multimodal architecture up to layer 22, freezing the XLS-R parameters while training the multimodal diacritic restoration model7. Comparison of model parameters are shown in the following Table. Including the ASR or speech feature extractor in the calculation, the multi-modal model has around 40% of the parameters of the Text+ASR model. In both cases, the largest contributor to model size is the speech module.

heading: 6 Results & Discussion
text: In this section, we first examine the performance of the proposed multi-modal diacritic restoration model, and then show the effect of our proposed data augmentation technique. As a baseline, we use
4https://arabicspeech.org/qasr_tts 5We investigated different optimizers (Adam, SDG, RMSprop, and Adagrad) and different dropout values (0.1, 0.2, 0.3, and 0.001), and selected the best among them.
6https://huggingface.co/facebook/ wav2vec2-large-xlsr-53
7Preliminary experiments indicated that later layers perform better for this task than early or intermediate layers.
the text-only and text+ASR models from Shatnawi et al. (2024) as it achieves the best performance reported on the test data (Refer to Shatnawi et al. (2024) for comparison with other diacritic restoration models).

heading: 6.1 Multi-Modal Performance
text: Table 4 shows the performance of the proposed multi-modal diacritic restoration model compared to the baselines. Using speech features generally improves the performance of Text-Only diacritic restoration, resulting in reduced diacritic error rates across. However, the multi-modal model lags behind the Text+ASR model, and sometimes even behind the Text-Only model. There are a couple of differences between the two models that could explain this difference, but one thing to highlight here is that the multi-modal model is more streamlined and efficient as it works without reliance on an external ASR model with the full pipeline of ASR inference. Yet, one of the potential drawbacks that led to sub-optimal performance is that XLS-R is a self-supervised model and has only been finetuned on the ClArTTS corpus; on the other hand, the Text+ASR uses the Whisper model, which is already optimized for ASR even before fine-tuning. In Table 5, we show ASR performance on the test
set using our fine-tuned XLS-R vs. the Whisper model used in Shatnawi et al. (2024). Whisper reduces error rates almost by half, which plays a large role in the performance of Text+ASR model. This suggests a possible future development of the multimodal model, where XLS-R can be optimized first on more general (un-diacritized) Arabic speech data and then fine-tuned on ClArTTS, to achieve performance on a par with Whisper, before integrating it with the multi-modal diacritic restoration model.

heading: 6.2 Effect of XLS-R Parameter Updates
text: The XLS-R model used in our experiments above is first fine-tuned for the task of ASR on ClArTTS. Once trained for ASR, we integrated the model in the multi-modal framework. While training the multi-modal diacritic restoration model, we experimented with either freezing the XLS-R parameters or continuing to update them. We found that freezing the parameters at this stage results in far better performance. The results with/without freezing the parameters are shown in Table 6. While this result seems rather counter-intuitive, a possible explanation is the small size of our training set compared to the complexity of the multi-modal model, leading to catastrophic forgetting of the useful features learned in the ASR fine-tuning stage.

heading: 6.3 Experiments with Data Augmentation
text: In this section, we describe experiments on the effect of data augmentation in speech-based diacritic restoration. In particular, we test the usefulness of the data augmentation method described in Section 3, where random diacritics are used for synthesizing speech. Our hypothesis is that this scheme can lead to better generalization from the speech features due to the reduction in the predictability of the diacritics from the textual context.

heading: 6.3.1 Experimental Settings
text: To create the synthetic audio from random diacritics, we used the ClArTTS text and applied the randomization rules. We use used Google TTS to
synthesize the speech from these inputs8. This gave as twice the original train set size in total. We used this augmented data for training our Multi-Modal model, but we did not modify the underlying and frozen XLS-R model. For Text+ASR, we re-trained both the underlying ASR model (i.e. Whisper) and the Text+ASR diacritic restoration model.

heading: 6.3.2 Results
text: As seen in Table 7, this data augmentation method improved the diacritic restoration performance across the board. The benefits of data augmentation are particularly evident for models that start with higher DER, with absolute reductions ranging from around 1% to 5% are observed for all models. This underscores the importance of exposing the diacritic restoration model to a wider variety of diacritic variations to enable generalizations beyond what can be inferred from text alone. Furthermore, the improvements are more evident for the MultiModal model compared with Text+ASR, but the gap between them remains, where the ASR based model continues to outperform the multi-modal model. Importantly, though, without the data augmentation, the Multi-Modal model did not consistently improve over the Text-Only model; with the addition of roughly 10K utterances using the data augmentation method, the results are consistently better than the Text-Only model across all test sets.
8We tried several open-source and commericial TTS systems, but most of them do not produce speech that is faithful to the provided random diacritics. The best one at producing speech as specified was Google WaveNet model.

heading: 7 Conclusions
text: In this paper, we introduced a novel approach for data augmentation that has consistently improved speech-based diacritic restoration models, resulting in enhanced performance on speech datasets compared to models that rely solely on text, and compared to models that do not use data augmentation. Additionally, we explored the potential of a multimodal approach that does not rely on an extrinsic ASR system as previously proposed for this task. While the multi-modal approach provides a more efficient and streamlined process, and improves over text-only model when data augmentation is used, it did not reach the performance provided by the ASR based model. This work highlights a couple of factors that influence the performance of such models. First, diacritic restoration models are generally limited by the type of data used for training them, and most existing open-source data sets for diacritic restoration are based on Classi-
cal Arabic, leading to poor performance in Modern Standard Arabic. Applying data augmentation techniques, such as one proposed in this paper, can lead to better generalization. Second, adding speech features where speech data is available results in better performance than text-only models, as the speech contains the low level acoustic features that distinguish these diacritics. Finally, while speech-based diacritic restoration works, the performance highly depends on the underlying speech model used to extract speech features. Our experiments show that optimizing the performance of the models used for feature extraction is essential, and future work could enhance the performance of the multi-modal architecture by integrating features from models that are first fine-tuned and optimized for ASR.
Limitations
One of the limitations of the experiments described here is the large gap in ASR performance between
the two models underlying the Multi-Modal and Text+ASR models. This gap in performance is due to the fact that Whisper is already pre-trained for ASR in Arabic, and fine-tuning it on the small ClArTTS results in significantly better ASR performance compared to fine-tuning the smaller XLS-R model from scratch on the same set. Unfortunately, Whisper could not be easily integrated in our multimodal model directly for a fair comparison. A more thorough evaluation should start by optimizing the underlying models to nearly similar performance. One more difference between the ASR+Text and multi-modal model with data augmentation is that we only used data augmentation for updating the multi-modal model, but the underlying XLS-R was frozen with the weights from ClArTTS only. While the results still showed larger reductions in error rates for the multi-modal model, a better comparison would be made by testing the effect of data augmentation on the underlying speech model. Another limitation is that the synthetic speech used for data augmentation is all based on the same underlying text. Our rationale was to add redundancy in the underlying text but differences in diacritics to minimize the reliance on the text context. However, using more varied underlying texts from MSA could have been beneficial, but we did not experiment with such variations.

heading: Future Work
text: Our experiments show that optimizing the performance of the models used for feature extraction is essential, and future work could enhance the performance of the multi-modal architecture by integrating features from models that are first fine-tuned and optimized for ASR. Limitations
One of the limitations of the experiments described here is the large gap in ASR performance between
the two models underlying the Multi-Modal and Text+ASR models. This gap in performance is due to the fact that Whisper is already pre-trained for ASR in Arabic, and fine-tuning it on the small ClArTTS results in significantly better ASR performance compared to fine-tuning the smaller XLS-R model from scratch on the same set. Unfortunately, Whisper could not be easily integrated in our multimodal model directly for a fair comparison. A more thorough evaluation should start by optimizing the underlying models to nearly similar performance. One more difference between the ASR+Text and multi-modal model with data augmentation is that we only used data augmentation for updating the multi-modal model, but the underlying XLS-R was frozen with the weights from ClArTTS only. While the results still showed larger reductions in error rates for the multi-modal model, a better comparison would be made by testing the effect of data augmentation on the underlying speech model. Another limitation is that the synthetic speech used for data augmentation is all based on the same underlying text. Our rationale was to add redundancy in the underlying text but differences in diacritics to minimize the reliance on the text context. However, using more varied underlying texts from MSA could have been beneficial, but we did not experiment with such variations