Qalam : A Multimodal LLM for Arabic Optical Character and Handwriting Recognition (2024)

Abstract: Arabic Optical Character Recognition (OCR) and Handwriting Recognition (HWR) pose unique challenges due to the cursive and context-sensitive nature of the Arabic script. This study introduces Qalam, a novel foundation model designed for Arabic OCR and HWR, built on a SwinV2 encoder and RoBERTa decoder architecture. Our model significantly outperforms existing methods, achieving a Word Error Rate (WER) of just 0.80% in HWR tasks and 1.18% in OCR tasks. We train Qalam on a diverse dataset, including over 4.5 million images from Arabic manuscripts and a synthetic dataset comprising 60k image-text pairs. Notably, Qalam demonstrates exceptional handling of Arabic diacritics, a critical feature in Arabic scripts. Furthermore, it shows a remarkable ability to process high-resolution inputs, addressing a common limitation in current OCR systems. These advancements underscore Qalam’s potential as a leading solution for Arabic script recognition, offering a significant leap in accuracy and efficiency.

heading: 1 Introduction
text: Optical Character Recognition (OCR) technology has revolutionized the way we interact with written and printed materials. It enables the conversion of various documents, including scanned paper documents, PDF files, or images captured by a digital camera, into editable and searchable data. The ability of OCR to digitize text has found applications in numerous domains, ranging from banking and healthcare to education and historical research, among others (Singh et al., 2012).
In this work, our focus is on handling Arabic OCR and HWR. Arabic OCR and HWR pose substantial challenges due to several distinctive features of the Arabic script. The Arabic writing system is cursive and context-dependent, a characteristic that complicates the design of robust OCR models. Further intricacies such as diacritical marks,
loops, overlaps, ligatures, dots, and multi-context joining contribute to the complexity of the task. In addition to these structural challenges, the vast diversity of Arabic fonts and individual handwriting styles further complicate character recognition and segmentation tasks. Finally, the lack of comprehensive and diverse annotated datasets increases the difficulty of Arabic OCR and HWR tasks. Figure 2 visually represents these challenges.
OCR systems can be classified based on the type of input data they process: handwritten or printed. In this paper, for clarity, we refer to printed text recognition specifically as OCR, although both types broadly fall under the OCR umbrella.
Our objective in this work is to tackle these complex tasks, making several key contributions that collectively underscore the novelty of our research. Concretely, our contributions can be summarized as follows:
(i) We introduce Qalam, a novel OCR and HWR model specifically engineered for Arabic script, establishing a new state-of-the-art performance across diverse datasets.
(ii) We compile a large and diverse collection of datasets and introduce Midad Benchmark for Arabic OCR and HWR. This will serve as a
210
resource for future research in the community.
(iii) Our work offers an in-depth analysis of the inherent complexities associated with Arabic script OCR, contributing towards a broader understanding of the challenges in this domain.
(iv) Last, we provide detailed comparative evaluations against multiple baselines, thus emphasizing the effectiveness of our approach.
The rest of this paper is structured as follows: Section 2 discusses the related work in HWR and OCR. Section 3 introduces Midad , our Arabic HWR and OCR benchmark. In Section 4, we present our methodology. Section 5 describes the experiments conducted in this work. Section 6 introduces Qalam, the proposed model for Arabic HWR and OCR, and evaluates Qalam. We conclude in Section 8 and spell out many limitations related to our work in Section 9.

heading: 2 Related Works
text: Traditional Hidden Markov Model (HMM) (Bunke et al., 1995; Park and Lee, 1996; Agazzi and Kuo, 1993) approaches in sequence modeling have been largely surpassed by deep learning techniques that do not require explicit segmentation. Connectionist Temporal Classification (CTC) models (Graves et al., 2006; Graves and Schmidhuber, 2008) and Encoder-Decoder models (Sutskever et al., 2014; Bluche et al., 2017) with attention mechanisms (Bahdanau et al., 2014; Michael et al., 2019) represent two primary deep learning categories. Recent advances include transformer models (Vaswani et al., 2017) and pre-trained models (Devlin et al., 2018).
Handwriting Recognition (HWR). HMMs were traditionally used for HWR (Bunke et al., 1995; Park and Lee, 1996), but CTC-based models (Graves et al., 2006; Graves and Schmidhuber, 2008) became more popular due to their accuracy without explicit segmentation. These models often use RNNs and their variations like LSTM (Pham et al., 2014), BLSTM, and MLSTM (Graves and Schmidhuber, 2008; Voigtlaender et al., 2016; Bluche et al., 2017), and combinations with CNNs such as CNN-BLSTM (de Sousa Neto et al., 2020). Recurrence-free models optimized with CTC, using only CNNs, have also been developed (Coquenet et al., 2020).
Optical Character Recognition (OCR). OCR, divided into Scene Text Recognition (STR), Scanned Document OCR, and Synthetic Text OCR, has evolved from HMMs to pre-trained transformer models. RNN and CTC-based models (Su and Lu, 2015), combined CNN and BLSTM models (Shi et al., 2016; Breuel, 2017), and EncoderDecoder architectures with attention mechanisms (Lee and Osindero, 2016; Shi et al., 2018) have been used. Recently, transformer-based models (Li et al., 2021; Kim et al., 2022; Lyu et al., 2022) have gained prominence.
Multimodal Large Language Models (MLLMs). MLLMs have impacted tasks like Visual Question Answering (VQA) and OCR-related tasks (Zhang et al., 2024; Wadhawan et al., 2024; Shi et al., 2023). Despite their success, challenges in text recognition within images remain due to lower encoder resolution (Liu et al., 2023). Open-source models like LLaVAR (Zhang et al., 2023) and MonkeyText (Liu et al., 2024) are improving text recognition, while closed-source models like GPT-4V (Achiam et al., 2023) and Gemini Pro (Team et al., 2023) contribute to these advancements.
Arabic HWR and OCR. Arabic HWR and OCR initially relied on HMMs (Alma’adeed et al., 2002; Prasad et al., 2008). Later, CTC-based models with RNNs and CNNs replaced HMMs. Ahmad et al. (2017) used an MDLSTM model for the KHATT dataset, and Yousef et al. (2020) introduced a pure CNN model optimized with CTC loss. Recent advancements include transformer models (Mostafa et al., 2021; Momeni and BabaAli, 2023), though Arabic MLLMs still face challenges in text recognition within images (Alwajih et al., 2024). For more detailed surveys, see Alrobah and Albahli (2022);
Faizullah et al. (2023) and reviews by Sobhi et al. (2020); Alghyaline (2023). Figure 9 (Appendix A.1) categorizes various model architectures used in these studies.
3 MIDAD Benchmark

heading: 3.1 Datasets
text: We utilize a variety of printed and handwritten Arabic OCR datasets in this study, which we have collectively named MIDAD. Below is a summary of these datasets. MADBase. A database of 60k training and 10k testing images of Arabic handwritten digits. It’s often used for training and testing CNNs (El-Sawy et al., 2017a). AHCD. Similar to MADBase, the AHCD dataset is used for training and testing CNNs, but contains 16k samples of handwritten Arabic characters (ElSawy et al., 2017b). ADAB. Consists of 937 Tunisian town and village names in Arabic handwriting (Boubaker et al., 2021). It contains 15k samples in total. Alexuw. This dataset is compiled by Hussein et al. (2014a), includes 25k samples of 109 different Arabic words. This large and diverse dataset allows for the development and testing of segmented letterbased Arabic handwriting recognition algorithms. Shotor. Introduced by Asadi (2020), this is a large-scale open-source dataset, consisting of 120k grayscale images of various Farsi phrases, each rendered in different fonts and sizes. The phrases were sourced from the Ganjoor and Farsi Wikipedia websites (Asadi, 2020). PATS-A01. Introduced by Al-Muhtaseb et al. (2009), it represents the first printed Arabic text set containing 22k line images sourced from classic Arabic literature. Eight different Arabic fonts were used for these images, adding to the variety in this dataset (Al-Muhtaseb et al., 2009). IDPL-PFOD. This synthetic dataset, created byHosseini et al. (2021), includes 30k TIF images of printed Farsi text lines. Each image has varying background types and levels of blur and distortion to mimic real-life conditions. The text is rendered using popular Farsi typefaces in diverse sizes and styles. UPTI. The UPTI dataset was developed by Sabbour and Shafait (2013) and contains 10k synthetic Urdu text lines in the Nastaliq font, providing a valuable resource for testing and training models in a language closely related to Arabic (Jain et al.,
2017). OnlineKHATT. A comprehensive dataset consisting of 10k Arabic text lines, created by 623 authors using various devices. The data come as stored in the online format using InkML files, and we convert it into offline format as images. The dataset provides character-based segments and manually verified ground truths for the written lines (Mahmoud et al., 2018). MIDAD provides a broad and varied base for training and validating OCR models for Arabic and other closely related languages.

heading: 3.2 Data Splits
text: In instances where the original datasets came prepartitioned into standard training (Train), development (Dev), and testing (Test) splits, we opt to maintain these existing divisions to preserve the integrity of the original data structure. In absence of such pre-defined splits, we randomly shuffle each dataset and split it into three 80% Train, 10% Dev, and 10% Test. Comprehensive statistics reflecting the distribution of data splits across all our datasets are in Table 1.

heading: 3.3 In the wild Arabic OCR and HWR Datasets
text: KHATT The KHATT dataset (Mahmoud et al., 2014), a prominent tool in Arabic handwriting recognition, offers 1, 000 handwritten Arabic forms from diverse writers, totalling 4, 000 paragraph images—half with similar text and the other half with unique text. Our study concentrated on the unique-text paragraphs providing 6, 742 text lines post-segmentation. Historical Manuscripts The dataset contains 120 images from historical Arabic manuscripts, digitized by the British Library and Qatar Foundation (Clausner et al., 2018).

heading: 4 Methods
text: In our pursuit to address the challenges of both Arabic HWR and OCR, we employ a Vision Encoder-Decoder (VED) framework that ingeniously brings together transformer-based models
in a novel manner. This framework leverages the power of transformer-based vision models as encoders, adeptly processing image data, and pairs it with the linguistic sophistication of transformerbased language models as decoders. The result is a synergistic pairing that skillfully transcodes visual information into meaningful textual output, thus overcoming the intricate complexities of Arabic OCR. Furthermore, we extensively analyze various encoder and decoder combinations, investigating their implications for model performance. In VED design, the encoder ingests image data, while the decoder manages ground-truth caption inputs via teacher forcing (Sutskever et al., 2014) during training. We ensure autoregressive training for the next token prediction using causal self-attention in the model. This mechanism restricts a token’s attention
only to its predecessors, maintaining the sequential nature of the input text.

heading: 4.1 Encoder Configuration
text: Our encoder takes images resized to (H,W ), padded for uniformity, and partitioned into N = H ∗ W/P 2 patches with a fixed size of (P, P ). These patches are subsequently flattened and linearly projected into D-dimensional vectors to form patch embeddings. We retain the “[CLS]” token to represent the image and incorporate learnable 1D position embeddings based on absolute positions.

heading: 4.2 Decoder Configuration
text: The decoder in our VED structure consists of layers identical to the encoder, but with an additional encoder-decoder attention mechanism. It employs attention masking to prevent the model from peeping into the future during training. Its hidden states are projected linearly to the vocabulary size, and probabilities are computed using softmax. We initialize the cross-attention layer weights randomly when warm-starting from pre-trained transformerbased models.

heading: 4.3 Baselines
text: To provide a comparative analysis, we consider a range of established OCR models as baselines. These models, varying in complexity and technique, include CRNN, Gated-CNN-BiLSTM-CTC, Tesseract, and TrOCR. CRNN. The CRNN model (Puigcerver, 2017) blends convolutional and recurrent layers, and concludes with a linear layer and CTC loss function. It leverages dropout, batch normalization, LeakyReLU, Maxpool, and bidirectional 1DLSTM layers. Data augmentation through random distortions is used for enhanced robustness. Gated-CNN-BiLSTM-CTC. This architecture (Bluche and Messina, 2017) includes Gated-CNNs for feature extraction. Convolutional layers, gates,
bidirectional LSTM layers, and a linear layer compose the encoder-decoder structure. The model minimizes the CTC objective function using RMSProp. It also incorporates data augmentation. Tesseract. Introduced by Patel et al. (2012), this is a versatile open-source OCR engine sponsored by Google. It is designed to be a universal text recognition tool, recognizing over 100 languages. Tesseract works by decomposing the input image into lines and words. TrOCR. Introduced by Li et al. (2021), this is a text recognition model that uses pre-trained image and text transformer-based models. Pre-training on large-scale synthetic data, and fine-tuning on human-labeled datasets make TrOCR effective in printed, handwritten, and scene text recognition tasks.

heading: 4.4 Evaluation Metrics
text: Model performance is evaluated based on the Word Error Rate (WER), computed as the normalized Levenshtein distance at the word level. The formula to calculate WER is given in Appendix A.3.

heading: 5 Experiments
text: We present our experimental settings, including the selection process for the encoder and decoder from various options. We also discuss the challenges encountered during their selection.

heading: 5.1 Encoder Selection
text: The second phase of our experimental design involved selecting the most suitable encoder from available options. We conducted rigorous tests with four different encoders (ViT (Dosovitskiy et al., 2020), DeiT (Touvron et al., 2020), BEiT (Bao et al., 2021), Swin (Liu et al., 2021), and SwinV2 (Liu et al., 2022) Transformers) while keeping the XLM-RoBERTa (Conneau et al., 2019) as the constant decoder. To ensure the uniformity of these experiments, we used the hyperparameters derived from the initial tuning stage. Results. Table 7 displays a performance comparison of several models, including ViT, Swin, BeiT, SwinV2, and DeiT using a constant decoder XLMR on OCR and HWR tasks. Overall, the DeiT encoder model exhibits superior results and it has a Midad score of 19.79, likely due to its adeptness at the discerning text in various forms and orientations in images and recognizing intricate handwritten patterns. However, the SwinV2 encoder model
which has a Midad score of 21.60 also shows notable performance, particularly inline-level handwriting recognition on the ‘OnlineKHATT’ dataset, due to its ability to handle variable image inputs Figure 5 explains this visually. The results underline the effectiveness of both DeiT and SwinV2 in handling diverse OCR and HWR tasks across Midad datasets.
and HWR datasets on the validation set. E1 : ViT, E2 : Swin, E3 : BeiT, E4 : SwinV2, and E5 :DeiT with XLM-R as decoder. The table also provides OCR, HWR, and Midad scores to showcase the models’ overall performance in respective tasks.

heading: 5.2 Decoder Selection
text: With DeiT established as the optimal encoder, we proceeded to the third phase where we tested the efficacy of different decoders. Maintaining DeiT as a constant encoder, we experimented with five different decoders, assessing their performance on all datasets under the same hyperparameters. We used RoBERTa (Liu et al., 2019), XLM-R (Conneau et al., 2019), ARBERT, MARBERT and MARBERTv2 (Abdul-Mageed et al., 2021). Results. Table 8 contrasts the performance of transformer-based decoders, we have fixed the encoder to DeiT and are experimenting with different decoders on various HWR and OCR tasks. Generally, ARBERT stands out with the lowest WER across most tasks and datasets, showing its strong deciphering ability for diverse texts and handwriting. It acquires a Midad score of 12.06 WER for HWR and 18.83 for OCR, with an overall (on both tasks) Midadof 17.02 WER. Marbertv2, however, excels specifically on the OCR ‘IDPL-PFOD’ which is a Persian line-level task, implying its potential suitability for specific OCR scenarios with dataset which have different languages that use the Arabic Script. These findings highlight ARBERT’s overall dominance, while suggesting the usefulness
of MARBERTv2 for particular tasks, emphasizing the importance of task-specific model selection.
validation set. D1 : RoBERTa, D2 : XLM-R, D3 : MARBERT, D4 : MARBERTv2, and D5 : ARBERT with DeiT as constant encoder. Tasks are categorized into character-level (Char), word-level (Word), and linelevel (Line) recognition.

heading: 5.3 Error Analysis
text: We carry out a manual error analysis on our validation set, using our top model, to identify challenges with our DeiT and ARBERT models. To this end, we randomly select 200 images from the validation set, divided among two experienced annotators (authors). Annotators reviewed these images, tested the model, and noted its performance. From this small-scale heuristic analysis, we identify two significant challenges our models faced:
Encoder-Related Issues. Despite its effectiveness across tasks, DeiT has limitations with scalability and high-resolution inputs. It struggles with larger input sizes, impacting performance in scenarios like detailed document analysis or highresolution character recognition. As shown in Figure 5, DeiT, after undergoing unsupervised training with Masked Imaging Modeling (MIM) (Xie et al., 2022), may fail to reconstruct high-resolution image content due to inefficient feature processing. Conversely, SwinV2 exhibits superior handling of high-resolution inputs.
Decoder-Related Issues. Regarding the decoder, ARBERT has shown limitations in handling diacritics as these are not included in its vocabulary. Diacritics play a crucial role in many languages, including Arabic and Persian, as they can significantly change word meanings. The inability of ARBERT to recognize and process these diacritics may lead to incorrect word recognition and, subsequently, incorrect text interpretation. For instance,
as shown in Figure 6, ARBERT might interpret the word incorrectly due to its inability to handle diacritics.
6 Building Qalam
To further optimize the performance of our Vision Encoder-Decoder framework, we introduce additional pre-training strategies for the encoder and the decoder, including the SwinV2 and RoBERTa models. This is undertaken to capitalize on their specific architectural strengths in handling large image sizes and next-token prediction tasks.
Encoder Upgrades. To enhance the model’s performance, we further pre-train the Swin Transformer v2 encoder. We choose this model due to its proficiency in handling high-resolution inputs and capturing rich spatial information. The training strategy involved augmenting the input image size and utilizing a robust dataset comprising 4.5M images extracted from Arabic manuscripts and books. The Masked Language Modeling approach is employed during training, promoting the encoder’s adaptability to real-world OCR challenges and the complexities inherent in Arabic script.
Decoder Upgrades. On the decoder side, we further pre-train the the RoBERTa model using the Masked Language Modeling approach (Devlin et al., 2018). We select RoBERTa as the decoder for Qalam for its superior performance in nexttoken prediction tasks. A substantial and diverse training dataset provided the model with various language patterns, including Arabic Wikipedia and AraC4 (Abdul-Mageed et al., 2023). The model was also modified to handle longer sequences during training, improving its comprehension and generation of complex sentence structures. We used a sentence-piece tokenizer that also can understand diacritics, which is crucial for processing the Arabic language, thereby enhancing the decoder’s effectiveness in Arabic OCR tasks. Figure 7 shows our final architecture.
Synthetic Data. This data comprises 60k imagetext pairs and features more than 28 diverse fonts, comprehensively representing potential inputs. Qalam is subsequently fine-tuned using this synthetic data in a supervised manner. Figure 10 shows a few examples of this dataset. We generate this dataset using text files from Hindawi Arabic books (Hindawi). We subsequently augment it into different fonts and organize it into PDF files of size 760x640. The model was then fine-tuned on this dataset to teach it about the Arabic script.
Data Augmentation. We supplement each training sample from the line-based datasets with four additional synthesized samples, as illustrated in Figure 8. Then, we fine-tune Qalam using this syn-
thetic data in a supervised manner. This strategy facilitates the model’s adaptability to various text styles and complexities in Arabic scripts. Here, we take the gold labels of the dataset’s training split and augment it with four different randomly chosen fonts.
Training Procedure. The training process for Qalam utilizes specific hyperparameters to optimize performance. Table 6 provides a comprehensive overview of these settings. The model is trained using the Adam optimizer with a 5× 10−5 learning rate. We employ a cosine learning rate scheduler over 50 epochs. The training process uses a batch size of 8, with gradient accumulation steps of eight, resulting in a total effective batch size of 64. For reproducibility, we set the random seed to 42. The evaluation batch size is also set to eight. These carefully chosen hyperparameters are instrumental in achieving optimal performance for the Qalam model.
6.1 Performance Evaluation of Qalam
This section encapsulates the performance evaluation of various models, emphasizing our proposed model, Qalam, alongside baseline models and alternative architectures. The results are collated in Table 4, highlighting the exemplary performance of Qalam across diverse datasets.
On the Handwriting Recognition (HWR) front, Qalam exhibits remarkable performance. Specifically, in the MADBase and AHCD datasets, Qalam can recognize all test samples without errors. Meanwhile, in word-based datasets such as ADAB
are: M1 : Tesseract, M2 : TrOCR Base, M3 : CRNN+CTC, M4 :CNN+BiLSTM+CTC, M5 :DeIT+ARBERT,
M6 :SwinV2(fp) + ARBERT, M7 :SOTA, and Qalam . The table lists WER (Lower is better) achieved by each model on different Arabic and Persian datasets classified by base, type, and language. The table also provides OCR, HWR, and MIDAD scores to showcase the models’ overall performance in respective tasks. MIDAD Score is the average WER of across all tasks and datasets
and Alexuw, Qalam achieves an equally impressive WER of just 0.01
When assessing line recognition tasks, especially on the OnlineKHATT dataset, Qalam continues to excel, recording a WER of 3.95%. This excellence extends to OCR tasks as well, where Qalam upholds a low WER on various datasets, including 1.90% on PATS01, 0.12% on Shotor, and 1.53% on IDPL-PFOD. These results also show that our model can perform well on Persian datasets.
In summary, Qalam delivers unparalleled performance across the board. The average scores indicate its superiority at a WER of 0.80% for HWR tasks and 1.18% for OCR tasks. Further, it achieves a unique evaluation metric, the MidadScore, of 0.94%. This collective evidence positions Qalam as a leading solution in both the HWR and OCR domains, reflecting its robustness and adaptability to diverse textual challenges.
6.2 Performance Evaluation of Qalam in the wild Arabic data
Table 5 showcases the comparative performance of the zero-shot evaluation of the Qalam system on "in the wild" Arabic OCR datasets in terms of Character Error Rate (CER) as these datasets are more complex and are completely unseen by the
model. The table contrasts the results of the stateof-the-art (SoTA) references with the outcomes achieved using the Qalam system. We observe the following:
• KHATT: The SoTA result, as referenced by Momeni and BabaAli 2023, yields a CER of 18.45. In comparison, the Qalam system significantly improved, achieving a CER of 10.43.
• Historical Manuscripts: The performance on this dataset provides a different scenario. The referenced SoTA result from Clausner et al. 2018 reports a CER of 21.9. However, the Qalam system shows a higher CER of 30.88 in this context.
In summary, while the Qalam model exhibits superior performance on the KHATT dataset, its performance on the Historical Manuscripts dataset is less competitive. The Historical Manuscripts dataset is mainly out-of-domain data, but the performance is still competitive.

heading: 7 Discussion
text: The exemplary performance of Qalam across Arabic and Persian OCR and HWR tasks (Table 4) highlights its potential. Despite the diversity in the OnlineKHATT dataset, Qalam achieves a relatively low residual error of 4%, indicating scope for improved handling of diverse writing styles. The superior performance of Qalam over CTC-based models like CRNN+CTC and CNN+BiLSTM+CTC, which emphasizes the transformative potential
of transformer-based models when supplemented with substantial training datasets.
The stark disparity between Qalam and TrOCR, despite TrOCR’s strength, underscores the limitation of models pretrained on English data when applied to different scripts, emphasizing the need for script-specific training. The results also highlight the limitations of task-specific models like Tesseract, which excels in line-based recognition but underperforms in character or word-based tasks. Finally, the exceptional performance of Qalam can be attributed to the synergy between SwinV2 encoder and RoBERTa decoder, effectively tackling OCR and HWR complexities. Examples of the Demo can be found in the Appendix A.6.

heading: 8 Conclusion
text: We introduced Qalam, a foundation model for Arabic OCR and HWR. Qalam establishes a new standard in Arabic OCR and HWR tasks. The robustness of its architecture, comprising the SwinV2 encoder and RoBERTa decoder, outperforms previous state-of-the-art systems. Our study demonstrates that Arabic script’s unique challenges can be effectively addressed by leveraging the strengths of transformer-based models. The performance of Qalam on the Midad benchmark validates the scalability and flexibility of our approach, suggesting its potential application to OCR and HWR tasks in other complex scripts. Moving forward, Qalam offers a compelling basis for further innovation in Arabic OCR and HWR systems, contributing to the advancement of this critical area of research.

heading: 9 Limitations
text: Given the limited availability of HWR and OCR datasets for Arabic, particularly for handwriting with diacritics - a feature often omitted in everyday writing - certain challenges arise. The most notable of these is the prevalence of code-switching and dialectal in real-world writing, both in OCR and HWR contexts, which the current model may struggle to address. Complex tasks such as Scene Text Recognition (STR), multiline, and full-page recognition also show significant limitations to the capabilities of Qalam.
Furthermore, Qalam has been specifically designed for Arabic OCR and HWR tasks. As a result, its performance has not been assessed on other
scripts or languages. Therefore, its effectiveness in these contexts may not be optimal without further modifications and fine-tuning. This should be considered when attempting to generalize Qalam’s capabilities beyond Arabic OCR and HWR tasks.

heading: Acknowledgments
text: We acknowledge support from Canada Research Chairs (CRC), the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN2018-04267), the Social Sciences and Humanities Research Council of Canada (SSHRC; 435-2018- 0576; 895-2020-1004; 895-2021-1008), Canadian Foundation for Innovation (CFI; 37771), Digital Research Alliance of Canada,1 and UBC ARCSockeye.

heading: A Appendices
text: We provide an addition organized as follows:
• Model architectures in Section A.1.
• Datasets Details Section A.2.
• WER equation Section A.3.
• Hyperparameter Table Section A.4.
• Synthetic Data A.5
• Qalam demo Section A.6.
• Test Results A.7
A.1 Model architectures in the literature In this section, we provide an illustrative Figure 9 of various model architectures used in the literature.
A.2 Dataset Details Table 1 provides additional statistics Midad.
A.3 WER Equation In this section, we present the equation used for the calculation of WER:
WER = (S +D + I)
N =
(S +D + I) (S +D + C) (1)
where:
S : number of substitutions,
D : number of deletions,
I : number of insertions,
C : number of correct words,
N : number of words in the reference.
A.4 Hyperparameter Table In this section, we present Table 6, detailing the hyperparameters employed in our study.
A.5 Synthetic Data A.6 Qalam Demo In addition to the computational experiments, we also developed a practical demonstration that accepts two types of inputs: handwriting and images. The handwriting input facilitates our model’s HWR capabilities, allowing users to test the system’s performance in real-time. Simultaneously, the image input caters to OCR tasks, enabling users to upload images of Arabic scripts and observe the model’s interpretation. One of the noteworthy features of our model is its capacity to handle complex diacritics, a characteristic intrinsic to Arabic scripts. Arabic diacritics are essential in the language, affecting word meanings and pronunciations. However, their tiny size and positioning above or below the line of text make them challenging for many OCR systems. As evidenced by the demonstration, our model exhibits robust performance in recognizing and interpreting these diacritics. The proficiency of our model isn’t limited to diacritics; it extends to handling various types of Arabic texts. Whether it be different fonts, styles, or levels of complexity, our system’s adaptability makes it a potent tool for Arabic script recognition. The demonstration provides a tangible testament to these capabilities, illustrating how the advancements in our model translate into practical, real-world applications. Additionally, Figure 10 displays screenshots of some synthetic samples used in our study.
A.7 Test Results
E1 : ViT, E2 : Swin, E3 : BeiT, E4 : SwinV2, and E5 :DeiT with XLM-R as decoder. The table also provides OCR, HWR, and Midad scores to showcase the models’ overall performance in respective tasks.
tasks for different datasets. D1 : RoBERTa, D2 : XLM-R, D3 : MARBERT, D4 : MARBERTv2, and D5 : ARBERT with DeiT as constant encoder. Tasks are categorized into character-level (Char), word-level (Word), and line-level (Line) recognition.

heading: Future Work
text: It employs attention masking to prevent the model from peeping into the future during training. Its hidden states are projected linearly to the vocabulary size, and probabilities are computed using softmax. We initialize the cross-attention layer weights randomly when warm-starting from pre-trained transformerbased models