# Deep Transformer Language Models for Arabic Text Summarization: A Comparison Study
###### Abstract  
Large text documents are sometimes challenging to understand and time-consuming to extract vital information from. These issues are addressed by automatic text summarizing techniques, which condense lengthy texts while preserving their key information. Thus, the development of automatic summarization systems capable of fulfilling the ever-increasing demands of textual data becomes of utmost importance. It is even more vital with complex natural languages. This study explores five State-Of-The-Art (SOTA) Arabic deep Transformer-based Language Models (TLMs) in the task of text summarization by adapting various text summarization datasets dedicated to Arabic. A comparison against deep learning and machine learning-based baseline models has also been conducted. Experimental results reveal the superiority of TLMs, specifically the PEAGASUS family, against the baseline approaches, with an average F1-score of 90% on several benchmark datasets. automatic text summarization (ATS); transformer language models (TLMs); Arabic ATS 202220220220220220220220220220220220220220220220222022202  
sentences). Typically, either explicitly or implicitly, the length of the final summary is determined. Therefore, an _extractive_ algorithm can, for example, select 10 to 15 essential sentences from a document that contains around 50 phrases [8]. _Abstractive_ summarization functions as well as humans. The algorithm reads the text, determines what it says, and then uses word combinations to describe the material. Theoretically, this approach might offer a superior, more condensed memory. In fact, this is challenging since it calls for both correct application of the content and knowledge of it at the level of an educated human reader [9]. In reality, most of the available ATS systems are mainly proposed to summarize texts written in English, with relatively little work being completed in other natural languages. There are fewer attempts on Arabic ATS, despite the fact that Arabic is among the top five most spoken languages in the world, with more than 20 nations using it as their official language and more than 400 million native and non-native speakers [10]. This is owing to the difficulty of the structure, syntactic and morphology of Arabic, as well as the compression ratio seen when summarizing numerous texts as opposed to a single document. Extractive summarization methods are the common approaches among the timid attempts for Arabic ATS. Such extractive methods produce factual, comprehensible summaries, but they often lack flow and are overly verbose [11]. In order to solve this issue, abstractive models are flexible in their word selection and turn to generalization and paraphrasing in order to produce more fluid and cohesive descriptions. For Arabic abstractive models, which is the main focus of this paper, the architecture of dominant choice is sequence-to-sequence (seq2 seq) [12]. For example, [PERSON] and [PERSON] [13] use the pointer-generator network [14]. Similarly, [PERSON] et al. [15] suggest a more general RNN-based approach. Most recently, and with the development of Transformer Language Models (TLMs) such as Bidirectional Encoder Representations from Transformers (BERT) [16], Bidirectional and Auto-Regressive Transformers (BART) [17], XLNet [18], Robustly Optimized BERT (RoBERTa) [19], Generative Pre-trained Transformer (GPT-3) [20], and Text-To-Text Transfer Transformer (T5) [21], NLP has experienced unprecedented advancements. TLMs can be described as pre-trained contextual language models with multilayer bidirectional self-attention mechanisms. For transformer encoders, pre-training and fine-tuning are the two key processes. State-of-the-art results for a wide range of NLP tasks, including abstractive ATS [22], are being witnessed nowadays thanks to TLMs [16; 19; 23; 24].