On the Utility of Pretraining Language Models on Synthetic Data (2024)

Abstract: Development of pre-trained language models has predominantly relied on large amounts of datasets. However, this dependence on abundant data has limited the applicability of these models in low-resource settings. In this work, we investigate the utility of exploiting synthetic datasets acquired from different sources to pretrain language models for Arabic. Namely, we leverage data derived based on four different methods: optical character recognition (OCR), automatic speech recognition (ASR), machine translation (MT), and generative language models. We use these datasets to pre-train models in three different architectures: encoderonly (BERTBase), encoder-decoder (T5), and decoder-only (GPT-2). We test the capabilities of resulting models on Arabic natural language understanding (NLU) tasks using the ORCA benchmark. Our results show that utilizing synthetic data can achieve performance comparable to, or even surpassing, those trained on gold data. For example, our model based on a GPT-2 architecture trained on a combined synthetic dataset surpasses the baseline model ARBERTv2. Overall, our models pre-trained on synthetic data demonstrate robust performance across various tasks. This highlights the potential of synthetic datasets in augmenting language model training in low-resource settings.

heading: 1 Introduction
text: The quality and quantity of training data significantly affect model performance in many natural language processing (NLP) tasks. Often, large amounts of training data are essential for neural models to achieve strong performances across a wide range of tasks (Sutskever et al., 2014; Bowman et al., 2015). In areas such as natural language understanding (NLU) (Wang et al., 2022) and neural machine translation (NMT) (Magueresse et al., 2020) consistent improvements with
increased training datasets have been shown, underscoring the significance of abundant training data. However, this becomes particularly challenging in low-resource settings where preparing extensive annotated data is costly, time-consuming, and infeasible to relabel every example whenever new data comes in. This highlights the need for research towards learning with limited labeled data for various NLP tasks (Chen et al., 2023).
There has been growing interest in data augmentation (DA) strategies to generate new data by modifying existing data through transformations (Feng et al., 2021; Wei and Zou, 2019). Moreover, with the recent surge of pre-trained Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), capabilities in generating data across domains such as images, audio, and text have expanded (Alemohammad et al., 2023). Renowned generative models like ChatGPT (for text) (Brown et al., 2020) and Stable Diffusion
265
(for images) (Rombach et al., 2022) are now readily accessible via APIs, making the generation of high-quality synthetic data across various domains more accessible (Veselovsky et al., 2023).
In this study, we investigate how training data from various sources and methods affect the performance of pre-trained language models (PLMs) on Arabic NLU tasks on the ORCA benchmark (Elmadany et al., 2022). We primarily focus on the Arabic language due to its complex grammar, rich morphology, and limited resources, which present significant challenges. We compare the performance of three different model architectures: encoder-only (BERTBase architecture), encoderdecoder (T5 architecture), and decoder-only (GPT2 architecture), trained on data extracted through automatic speech recognition (ASR), optical character recognition (OCR), machine translation (MT), and text generation (TG). This evaluation aims to determine whether synthetically generated text data can match or potentially replace gold standard data. Our contributions in this paper are as follows:
1. We showcase the importance of leveraging different text extraction and text generation methods to obtain valuable pre-training data.
2. We compare the performance of models pretrained on gold standard text data against models pre-trained purely on synthetic data.
3. We highlight the limitations of relying on sole and non-diverse data sources, such as solely Wikipedia, in model pre-training.

heading: 2 Related Works
text: Data Augmentation. Data augmentation has become a cornerstone in enhancing training datasets without the need for additional data collection, due to its efficiency, simplicity, and cost-effectiveness. This has led to a surge in its adoption across various domains (Hernández-García and König, 2018; Shorten and Khoshgoftaar, 2019). Earlier methods include rule-based approaches, such as tokenlevel random perturbations (Wei and Zou, 2019), and predetermined transformations without model components (Zhang et al., 2015). These methods have demonstrated improved outcomes in various text classification tasks (Xie et al., 2020). With the recent development of large language models (LLMs), the potential for more advanced and adaptable data augmentation techniques has significantly increased (Ding et al., 2020), allowing for greater
gains in model performance and robustness across different natural language processing tasks (Perez and Wang, 2017; Yang et al., 2022; Zhuo et al., 2023).
Multimodal Augmentation. Multimodal Augmentation (MA) leverages generative advancements in AI to synthesize data across multiple domains, including image, audio, text, and video (Bewersdorff et al., 2024). These techniques either generate entirely new content or modify existing content to maintain semantic relationships, similar to how traditional Data Augmentation (DA) techniques adapt training examples (Liu et al., 2022). The notable growth and results from publicly released generative models have established MA as a critical component in numerous research sectors, particularly those involving multimodal data (Alemohammad et al., 2023). MA has demonstrated significant effectiveness in various domains, such as developing SoTA image models like Stable Diffusion using the synthetic LAION-5B dataset (Schuhmann et al., 2022). In the vision domain, innovations like MixGen excel in joint data augmentation by combining images and text to create new image-text ensembles (Hao et al., 2023). In the audio domain, advancements such as AudioLM push the boundaries of speech synthesis, capturing the unique voice and nuances of previously unheard speakers (Borsos et al., 2023). These achievements highlight the convergence of self-supervised representation learning and language modeling, pointing to a promising future for the field (Liu et al., 2022).
Arabic NLP. Arabic Natural Language Processing (NLP) faces significant challenges due to its characteristics as a low-resource language, which has historically hindered achieving SoTA results due to data scarcity (Abdul-Mageed et al., 2020b; Belkebir and Habash, 2021). Despite this, there have been multiple efforts to address these challenges. For instance, the KIND dataset collects nuanced dialectal data through social collaboration (Aloui et al., 2024). Similarly, the ArQuAD dataset, a comprehensive, expert-annotated corpus designed for Arabic machine reading comprehension, emphasizes the urgent need for substantial, high-quality datasets to advance Arabic NLP (Obeidat et al., 2024). Furthermore, recent efforts in specific tasks such as Grammatical Error Correction (GEC) (Kwon et al., 2023; Solyman et al., 2021) and task classification (Refai et al., 2023), as well as efforts for evaluation (Khondaker et al., 2023)
and model development (Nagoudi et al., 2022) for Arabic, illustrate ongoing endeavors to enhance the linguistic capabilities and robustness of NLP models for the Arabic language.

heading: 3 Datasets & Experimental Setup

heading: 3.1 Training Datasets
text: We consolidate four unique datasets acquired through four distinct methods: (1) machine translation (MT), (2) optical character recognition (OCR), (3) automatic speech recognition (ASR), and (4) text generation (TG). While having all generated data based on the same corpus or covering the same genre would ensure consistency and fair comparison, this was challenging for certain data extraction techniques. MT and TG datasets align with this criterion, as they both utilize Wikipedia content. For OCR data, we aimed for consistency by selecting academic works from the Arabic public library Hindawi. However, for ASR, achieving this level of consistency was difficult due to the limited availability of domain-specific data. In future work, we aim to improve consistency across data sources where feasible.
We introduce each dataset and outline the extraction methods below. Details on the data statistics can be found in Table 1. Gold Data. We collect the most recent articles from Arabic Wikipedia dumps and extract raw text at the article level using wikiextractor (Attardi, 2015) as our primary data source. This dataset remained unaltered throughout our study, serving as our baseline dataset for comparison. MT Data. Following the same approach as the gold data, we gather a subset of English articles from English Wikipedia dumps and translate them into Arabic using Meta’s No Language Left Behind model (NLLB) (Costa-jussà et al., 2022). The articles were randomly sampled to ensure a diverse and representative subset of content. After translation, we remove any text that contains repeated duplicates of a word to avoid potential issues during training.
OCR Data. We collect and preprocess 3,000 books from the Arabic public library Hindawi (Ali, 2023). Subsequently, we employ Google Tesseract (Smith, 2007) and perform OCR on all the PDF books, extracting raw text at the page level, then segmenting it into paragraphs. ASR Data: We utilize a wide range of Arabic speech datasets, covering both Modern Standard Arabic (MSA) and various Arabic dialects. For datasets lacking transcriptions, we transcribe the audio using the Arabic whisper model from Talafha et al. (2023). For those datasets that include transcriptions, we consider them only if they were transcribed via ASR systems. The datasets are detailed below:
Multi Genre Broadcast-2 (MGB-2) (Ali et al., 2016) includes 1,200 hours from Aljazeera TV programs. These programs were manually captioned without specific timing details. The QCRI Arabic ASR system (Dalvi et al., 2017) was employed to transcribe all the output, facilitating the alignment of manual captions to generate speech segments for speech recognition training.
Mozilla’s Common Voice Project (Ardila et al., 2019) features recordings of sentences in MSA by contributing volunteers. Each recording is validated by at least two users. We use all 11 versions of the dataset, which total 690 hours of audio.
Arabic Speech Corpus (Halabi, 2016) is dedicated to MSA speech synthesis. It features over 3.7 hours of MSA speech, with both phonetic and orthographic transcriptions aligned with the recorded speech at the phoneme level.
Massive Arabic Speech Corpus (Al-Fetyani et al., 2021) contains over 1000 hours of speech from more than 700 YouTube channels. It is multi-genre, multi-regional, and multi-dialect. TG Data. We take samples from the Gold Data and leverage Jasmine (Nagoudi et al., 2022), an Arabic GPT model, to generate synthetic data by asking it to complete sentences given their beginnings. We chose to use the Jasmine model for generating synthetic data because it is specifically tailored for Arabic, ensuring that the generated text aligns with the linguistic nuances and characteristics of the language. By leveraging Jasmine, we can produce high-quality completions that are coherent and contextually appropriate for Arabic. Additionally, to maintain the quality and diversity of the generated dataset, we implemented measures to prevent repetition. This approach enhances the richness of the synthetic data, which is crucial for robust model
training and evaluation. Examples of the prompts and generated text can be found in Appendix A.1. Combined Datasets For our final experiment, we compare the performance of the models trained on all the subsets of data used above. We train models with different combinations of the Gold, MT, OCR, ASR, and TG data. Table 2 outlines the different data combinations.

heading: 3.2 ORCA Benchmark
text: We evaluate our models using the ORCA benchmark, a diverse NLU benchmark for the evaluation of language models in Arabic NLU tasks (Elmadany et al., 2022). The benchmark includes both Modern Standard Arabic (MSA) and Dialectal Arabic (DA), ensuring broad linguistic spectrum and geographical representation. It is comprised of 60 publicly available datasets and is segmented into seven distinct task clusters: Sentence Classification (SC), Text Classification (TC), Structured Prediction (SP), Semantic Text Similarity (STS), Natural Language Inference (NLI), Question-Answering (QA), and Word Sense Disambiguation (WSD). These are further organized into 29 tasks within the mentioned task clusters. For evaluation, a macroaverage of scores (F1) is taken across all tasks and task clusters, with each task given equal weight, referred to as the ORCAscore. For all individual tasks, the metric is F1, except for tasks within the STS clsuter and emotion-Reg task within the SC cluster, which use spearman correlation. Samples from distinct tasks are in Table 3. Details regarding the different task clusters, tasks, and data splits can be found in Table 4.

heading: 3.3 Models & Pre-training
text: Common Training Configuration. To pre-train all models outlined below, we run for ten epochs, using a batch size of 16, a learning rate of 5e− 3, and a maximum sequence length of 512.
Baseline Setting. The ARBERTv2 (AbdulMageed et al., 2020a) model (164 million parameters) serves as baseline for comparative analysis
in our experiments. We choose ARBERTv2, as it achieved the highest ORCAscore on the ORCA benchmark as is shown in the ORCA leaderboard.1 This ensures a consistent and reliable metric for comparison across our studies.
Encoder-Only. For the encoder-only model, we use the BERTBase model (Devlin et al., 2018) (110 million parameters). We pre-train BERTBase following the methods outlined by Devlin et al. (2018). We employ a BERT Word Piece Tokenizer and adopt the default network architecture of the BERTBase model. We adhere to the hyperparameters and architectural details specified in the original paper, including the tokenizer’s vocabulary size, minimum word frequency, and the configuration of special tokens. This approach allows us to maintain alignment with established benchmarks while focusing on the innovations introduced in our work.
Decoder-Only. We use the GPT-2 (Radford et al., 2019) model (1.5 billion parameters) for our decoder-only experiment. To pre-train the GPT2 model, we use the configured vocabulary size of 50, 257, including special tokens such as 
, 
, <pad>, <unk>, and <mask>.
Encoder-Decoder. The T5 (Raffel et al., 2020) model (220 million parameters) serves as the baseline model for our encoder-decoder model experiment. We employ a Sentence Piece Tokenizer with a vocabulary size of 32, 128 and special tokens <unk>, 
, 
, and <pad>.
Model Selection. We acknowledge the differences in model sizes across the architectures used in our experiments. The choice of ARBERTv2, BERTBase, GPT-2, and T5 is driven by their popularity and established benchmarks in the field. Our primary goal is to utilize base "representative" models to ensure the relevance and applicability of our findings. In future work, we aim to focus on similarly sized models for a fairer comparison across different architectures.

heading: 3.4 Fine-Tuning on the ORCA Benchmark
text: Encoder and Decoder Only Models. After pretraining, we fine-tune both the BERTBase and GPT2 model on the ORCA benchmark, covering all 29 tasks across seven task clusters encompassing 60 datasets. Examples of the specific ORCA tasks are presented in Table 3 while further details on
1https://orca.dlnlp.ai/main_leaderboard
ORCA’s diverse task clusters, tasks, and data splits are provided in Table 4. During training, tasks were categorized into token-level tasks, classification tasks, and question-answering tasks, as outlined in the original paper (Elmadany et al., 2022). Each model is fine-tuned according to this categorization to handle the respective tasks. We fine-tune each model for ten epochs using a batch size of 16 and a learning rate 5e − 5. We then select the best-performing model based on our Dev data for blind-testing on the Test sets. We report the mean score of the three runs, along with its standard deviation. Results of the Dev set can be found in Appendix A.4.
Encoder-Decoder Model. Since the T5 model is inherently a generative model, we had to adapt the model for classification tasks on the ORCA benchmark. Given that the ORCA benchmark does not naturally accommodate encoder-decoder architectures, we tailor the tasks to suit a generative format. To achieve this, we use the Dolphin benchmark as a framework (Nagoudi et al., 2023), which is
specifically designed for natural language generation (NLG) tasks. To maintain consistency across all model types, including BERTBase and GPT-2, we modify the dataset structures while keeping the same sets of inputs and labels. For tasks that involve a direct single input-output relationship, the original structure is preserved. However, for more complex tasks that require multiple inputs, such as the Word Sense Disambiguation (WSD) task, we concatenate the inputs into a unified context. The consolidated context is then used as the input during the fine-tuning phase of the T5 model, ensuring a consistent approach across various task types.

heading: 4 Results

heading: 4.1 Encoder Only Model
text: Table 5 presents the performance of the BERTBase model. The model trained on ASR data achieves the highest ORCAscore, surpassing the scores of the gold dataset and significantly enhancing performance, particularly in the adult, dialect-binary, and emotion-reg tasks, with improvements of 15.20,

heading: Cluster Task Level #Data Train Dev Test
text: 15.02, and 17.36 points respectively. The model trained on TG data demonstrates strong performance within the SC and SP task cluster, achieving the highest scores in adult (87.19), age (41.92), dialect-country (23.99), sarcasm (65.74), arabicner (66.35), and aqmar-ner (57.68). Notably, this model also outperforms the gold dataset in ORCAscore. In contrast, the model trained on both OCR and MT data scores lower than the gold data. However, the model trained on OCR data shows notable improvements in the emotion-reg task, with an increase of 17.51.

heading: 4.2 Decoder Only Model
text: Table 6 outlines the performance of the GPT-2 model. The model scores the highest ORCAscore in all datasets compared to all other model architectures, with the model trained on gold data scoring the best. However, the model struggles in tasks within the STS cluster. Consistent trends are observed across all tasks with the BERTBase model, with the model trained on OCR and ASR data showing significant performance improvements. The model trained on OCR data excels in hate-speech, scoring 72.47, and emotion-reg, with a notable increase to 61.59. Meanwhile, the model trained on ASR data achieves strong results in sarcasm (63.22) and dialect-binary (79.99). The TG dataset
also show competitive performance, particularly in adult (86.63) and emotion (46.53) tasks. In contrast, the model trained on the MT dataset generally underperforms compared to the gold dataset but shows comparable results in irony (78.61) and offensive (78.60) tasks.
The results from the GPT-2 model suggest that this architecture is particularly adept at learning from limited data quantities. Overall, the results from GPT-2 are better than those from BERTBase and T5 model. This may be attributed to the fact that GPT-2 has the largest model size among those tested. Additionally, while most other model architectures benefit from training on ASR data, the GPT-2 model does not show the same improvements. This is likely because ASR data is the smallest among those used.

heading: 4.3 Encoder-Decoder Model
text: Table 7 presents the performance of the T5 model. The model trained on the ASR data achieves the highest ORCAscore, surpassing the gold dataset. The model trained on ASR data excels particularly in tasks such as abusive (64.77), emotion (45.97), and emotion-reg (11.12), demonstrating significant improvements. The model trained on the TG dataset also shows strong performance, especially in dialect-binary (84.22), offensive (77.98), and sarcasm (57.10) tasks, while maintaining competitive scores in most other tasks. The OCR-trained model has notable success in offensive (81.54) and sarcasm (67.79), although it underperforms in tasks like emotion-reg (2.52). The MT dataset generally enable lower performance than the gold data but is effective in enabling irony (73.07) and adult (88.56) tasks. Overall, the ASR and TG datasets support superior performance in the majority of tasks compared to the gold dataset.

heading: 5 Discussion

heading: 5.1 MT, OCR, ASR, and TG Data
text: The ORCAscore from each model suggests that models trained on synthetic data from various sources can be as effective as those trained on gold data. Specifically, the BERTBase model trained on the ASR dataset outperformed all other datasets by an average of approximately 5 F1 points. This can be attributed to the diverse domain sources included in the ASR data, unlike other datasets confined to a single domain. Furthermore, the T5 model trained on ASR data exhibits comparable performance to
Gold : Data obtained from Arabic Wikipedia, MT : Data obtained through Machine Translation, OCR : Data obtained through Optical Character Recognition, ASR : Data obtained through Automatic Speech Recognition, TG : Data obtained through Text Generation.
the gold data in tasks such as adult, dialect-binary, hate-speech, and sarcasm within the SC task cluster. This improvement is likely because, unlike the more formal and academically oriented texts from datasets like Wikipedia articles, ASR data are varied and include informal content, thus aiding in tasks such as predicting hate-speech and adult content. Additionally, other datasets contain predominantly Modern Standard Arabic (MSA) and lack sufficient exposure to different Dialectal Arabic (DA), impacting their effectiveness in DA classification tasks. Notably, the ASR dataset includes DA data, enhancing its performance in these tasks.
Models trained on MT data consistently score a lower ORCAscore than any other dataset. This is likely due to the fact that, unlike datasets sourced from completely new domains, MT is generated by translating different subsets of English Wikipedia. Therefore, it is expected to perform worse than the original source. However, for the TG dataset, although it is also generated with Wikipedia as
a base, it is more novel as it is produced from a language model (JASMINE) trained to perform well on both MSA and DA, giving it a slight edge over models solely trained on Wikipedia data or translated text.
We see a similar pattern to MT in the OCR dataset with larger declines in performance on tasks involving informal language. This can be attributed to the fact that OCR data is acquired from a repository of academic texts, making it very unlikely to have encountered foul language during training.
Ablation Study We conduct an ablation study to determine if aggregating synthetic data from different sources improves results. We choose specific data combinations based on these principles to assess their individual and collective impact on model performance. We only compare ARBERTv2 against the GPT-2 model, as GPT-2 achieved the best results among the three model architectures. As shown in Table 8, GPT-2 exhibit continuous improvements with additional data, achieving the

heading: Task Gold MT OCR ASR TG
text: tion. Gold : Data obtained from Arabic Wikipedia, MT : Data obtained through Machine Translation, OCR : Data obtained through Optical Character Recognition, ASR : Data obtained through Automatic Speech Recognition, TG : Data obtained through Text Generation.
highest ORCAscore of 62.58 on the C4 dataset. This improvement can be attributed to GPT-2’s larger model size, which benefits more from increased training data. Notably, the GPT-2 model consistently outperform ARBERTv2 across most tasks, particularly in the SC and SP clusters, demonstrating its superior ability to handle a diverse range of tasks when trained on combined datasets.

heading: 5.2 Comparing Against Baseline
text: We compare the results from our trained models against our baseline model, ARBERTv2. Despite

heading: Task Gold MT OCR ASR TG
text: Gold : Data obtained from Arabic Wikipedia, MT : Data obtained through Machine Translation, OCR : Data obtained through Optical Character Recognition, ASR : Data obtained through Automatic Speech Recognition, TG : Data obtained through Text Generation. Token Level Tasks were excluded as they yielded scores of 0 F1 score.
ARBERTv2 being trained on a vast and diverse dataset specifically designed for Arabic, our models achieve comparable results in tasks such as dialect, age, adult, dangerous, and topic classification. Additionally, the Spearman Correlation between the highest ORCAscore from ARBERTv2 and each model architecture shows significant positive correlations, indicating a strong relationship between the two sets of scores. Specifically, the decoder-only architecture exhibit a high correlation of 0.85, suggesting a close alignment with the ARBERTv2 model. The encoder-only and encoderdecoder models also show substantial correlations, with coefficients of 0.728 and 0.725, respectively. These results demonstrate not only the competitive

heading: Task c1 c2 c3 c4 BL
text: man correlation. C1 : Combined dataset 1, C2 : Combined dataset 2, C3 : Combined dataset 3, C4 : Combined dataset 4, BL : Baseline Model ARBERTv2.
performance of our models against a fully trained PLMs but also highlight the robustness of training on various sources across different architectures.

heading: 6 Conclusion
text: In this work, we highlight the importance of leveraging different data extraction techniques for the augmentation and development of LMs. Our results show noticeable benefits from leveraging different sources of text to augment the performance of LMs. Our findings indicate that models trained on diverse synthetic data sources can achieve performance comparable to those trained on gold data. Specifically, the diverse domain sources in ASR
data and the informal content included in such datasets significantly enhance model capabilities in various tasks. While MT data enable lower performance due to its translation-based generation, TG data demonstrate improved results due to its novelty and the robust language model used in its creation. Our work, emphasizes the value of incorporating a variety of data sources to improve the effectiveness of language models in handling different linguistic tasks.

heading: 7 Limitations
text: We identify the following limitations in this work:
1. Only the ASR dataset contains dialectal Arabic (DA) data, whereas all other datasets are limited to Modern Standard Arabic (MSA). This inclusion of DA data in the ASR dataset boosts the performance of models trained on ASR data, particularly for tasks related to DA. Furthermore, the varying number of datasets from each source makes it challenging to ensure a fair comparison of the efficacy of these datasets.
2. The model sizes differ significantly, which may affect performance and does not ensure a fair comparison. Larger models, such as GPT2, benefit more from increased training data, potentially biasing the results. This variability in model size complicates direct performance comparisons across different architectures.
3. The scope of this study is primarily centered around NLU tasks. While our findings provide valuable insights into the performance of various datasets and model architectures for NLU, they may not generalize to other types of natural language processing tasks, such as NLG tasks.
4. The datasets used in this study may carry inherent biases specific to their domains of origin. For example, the OCR dataset, sourced from academic texts, may not adequately represent informal language use, affecting model performance on tasks involving colloquial or slang expressions. This domain-specific bias can limit the applicability of our findings to more diverse linguistic contexts.

heading: 8 Ethics Statement and Broad Impact
text: Promoting inclusive NLP research and resource development. NLP for Arabic languages has been under-resourced compared to other languages. This scarcity of resources has hindered the development of robust NLP applications for Arabic. Our work aims to bridge this gap by leveraging diverse datasets to improve model performance across various tasks. We hope to stimulate further research and development in Arabic NLP, fostering innovation and enabling the creation of more effective and culturally aware language technologies. Encouraging future research and innovation. Our research underscores the potential of using diverse data sources to enhance the performance of language models. We hope that our findings will inspire further exploration into the integration of underrepresented language varieties in NLP research. This pursuit can lead to the creation of more robust, adaptable, and inclusive language models. By advancing the state of NLP for Arabic and other less-represented languages, we contribute to the broader goal of developing AI technologies that serve a global audience, promoting cross-cultural understanding and communication.

heading: Acknowledgments
text: We acknowledge support from Canada Research Chairs (CRC), the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN2018-04267), the Social Sciences and Humanities Research Council of Canada (SSHRC; 435-2018- 0576; 895-2020-1004; 895-2021-1008), Canadian Foundation for Innovation (CFI; 37771), Digital Research Alliance of Canada,2 and UBC ARCSockeye.

heading: A Appendices

heading: A.1 Jasmine Sample
text: The prompt and generated examples from Jasmine can be found in Table 9.

heading: A.2 Test Results with Standard Deviation
text: Results of the Test sets differences in respect to the gold data for each model architecture and standard deviation can be found in Table 10, 11.

heading: A.3 Test Results as Compared to Gold
text: Results of the Test sets with positive and negative differences in respect to the gold data for each model architecture can be found in Table 12, 13, 14.

heading: A.4 Dev Results
text: Results of the Dev sets for each model architecture can be found in Table 15, 16, 17.
tion. Gold : Data obtained from Arabic Wikipedia, MT : Data obtained through Machine Translation, OCR : Data obtained through Optical Character Recognition, ASR : Data obtained through Automatic Speech Recognition, TG : Data obtained through Text Generation.
Gold : Data obtained from Arabic Wikipedia, MT : Data obtained through Machine Translation, OCR : Data obtained through Optical Character Recognition, ASR : Data obtained through Automatic Speech Recognition, TG : Data obtained through Text Generation. Token Level Tasks were excluded as they yielded scores of 0 F1 score.
tasks is spearman correlation. Gold : Data obtained from Arabic Wikipedia, MT : Data obtained through Machine Translation, OCR : Data obtained through Optical Character Recognition, ASR : Data obtained through Automatic Speech Recognition, TG : Data obtained through Text Generation.
tasks is spearman correlation. Gold : Data obtained from Arabic Wikipedia, MT : Data obtained through Machine Translation, OCR : Data obtained through Optical Character Recognition, ASR : Data obtained through Automatic Speech Recognition, TG : Data obtained through Text Generation.
tasks is spearman correlation. Gold : Data obtained from Arabic Wikipedia, MT : Data obtained through Machine Translation, OCR : Data obtained through Optical Character Recognition, ASR : Data obtained through Automatic Speech Recognition, TG : Data obtained through Text Generation. Token Level Tasks were excluded as they yielded scores of 0 F1 score.
Gold : Data obtained from Arabic Wikipedia, MT : Data obtained through Machine Translation, OCR : Data obtained through Optical Character Recognition, ASR : Data obtained through Automatic Speech Recognition, TG : Data obtained through Text Generation.

heading: Task Gold MT OCR ASR TG
text: Gold : Data obtained from Arabic Wikipedia, MT : Data obtained through Machine Translation, OCR : Data obtained through Optical Character Recognition, ASR : Data obtained through Automatic Speech Recognition, TG : Data obtained through Text Generation.
Gold : Data obtained from Arabic Wikipedia, MT : Data obtained through Machine Translation, OCR : Data obtained through Optical Character Recognition, ASR : Data obtained through Automatic Speech Recognition, TG : Data obtained through Text Generation. Token Level Tasks were excluded as they yielded scores of 0 F1 - more details can be found on section 4.3

heading: Future Work
text: Encouraging future research and innovation. Our research underscores the potential of using diverse data sources to enhance the performance of language models. We hope that our findings will inspire further exploration into the integration of underrepresented language varieties in NLP research. This pursuit can lead to the creation of more robust, adaptable, and inclusive language models. By advancing the state of NLP for Arabic and other less-represented languages, we contribute to the broader goal of developing AI technologies that serve a global audience, promoting cross-cultural understanding and communication