# Deep Transformer Language Models for Arabic Text Summarization: A Comparison Study
## 3 Text Summarization Methodology  
Text summarization is the act of separating long distributions into sensible passages or sentences. The technique extricates basic information while also guaranteeing that the section's sense is saved. This abbreviates the time it takes to understand long materials, such as insightful articles, without ignoring basic data. The most widely recognized approach to encouraging a brief, solid, and natural summary of a lengthier text report, including highlighting the text's essential centers are known as text summarization. Text summarization presents a few issues, counting content distinctive confirmation, interpretation, frame time, and an examination of the subsequent summary. Perceiving significant expressions in the record and taking advantage of them to uncover applicable information to add to the synopsis are fundamental positions in an extraction-based summarization. As highlighted earlier, there are a few crucial text summarization types, as shown in Figure 1. In this study, we will focus on the abstractive text summarization for the Arabic language with a single document input. Particularly, the sole focus will be on the TLMs-based approaches. Abstractive ATS approaches are classified as structure, semantic, discourse structure and deep learning-based techniques. They require more examination of the input source text and are mostly founded on understanding the semantics of a given article, restructuring sentences at the word-level, and lastly, producing abstracts with fewer and more clear words [39]. Summary generation can produce new sentences instead of just replicating sentences from the source record [40]. [PERSON] et al. [35] recently shifted the direction and introduced a new deep learning-based model. The model is called a transformer and it makes use of several methods and mechanisms. A transformer model is a neural network that learns the setting and, consequently, importance by following connections in successive information very much like the words in this sentence. Transformer models apply a propelling arrangement of numerical methods, called consideration or self-consideration, to distinguish unpretentious ways to be sure far-off information components in a series influence and rely upon one another. Transformers [35] are among the most modern and one of the most remarkable classes of models designed to date. They are driving a rush of advances in AI, ML and NLP, and some have been named transformer AI or transformer NLP. Encoder and decoder layers are part of the transformer model, and one is coupled to the other through layers of the feed-forward network and multi-head attention. The cosine and sine functions, which produce positional encoding, assist the model and recall the order and position of words. Self-attention is a method used by the encoder and decoder layer's multi-head attention layer (see Figure 2). Figure 1: ATS approaches and their connected methods. Figure 2: Architecture of transformers models [35]. From transformers-based models, the revolution of TLMs has emerged. For example, a TLM that is based on encoders and is learned in both directions, Bidirectional Encoder Representations from Transformers (BERT) [16], was introduced by Google AI. The BERT model's inputs are encoded using a specific format that consists of three pieces: wordpiece tokenization embeddings, segment embeddings, and position embeddings. It should be noted that all sequences now start with the special \"CLS\" token. Typically employed for classification tasks, this token can be seen as the representation of the whole input sequence. Additionally, each sentence ends with the unique separator symbol \"SEP\". There are various versions of BERT for different languages, such as French camemBERT [41], ArabicBERT [42], AraBERT [43] and CAMeLBERT [44]. Likewise, [PERSON] et al. [45] presented the Generative Pre-training Transformer (GPT) model. A total of 12 decoders are utilized to construct the input embeddings.