# Deep Transformer Language Models for Arabic Text Summarization: A Comparison Study
## 2 Background and Related Work  
As early as the late 1950s, ATS attracted scientific communities to conduct research on text summarization [1]. At the time, there was a particular focus on generating abstracts of technical documentation. Years later, the literature witnessed a kind of decline in the interest in the area of ATS until the renaissance of AI and its technologies. The early approaches of ATS mainly utilized statistical models to solely select, copy and paste the essential part of the original text [4]. For example, [PERSON] [27] proposed a method that adopts statistical techniques. Such statistical methods principally use information about the frequency and distribution of words to calculate the relative significance. The text summary is then produced using the sentences with the most significance. However, such early approaches were not able to generate abstractive text summarization due to the lack of understanding of the original text. As such, there was a need for more intelligent systems that were able to understand and analyze the semantics of the natural languages to address the various challenges of using the early statistical-based approaches. As was previously mentioned, there are two basic categories into which the ATS techniques can be broadly divided: _extractive_ and _abstractive_. Early research on ATS was essentially focusing on extractive methods. However, most recently, more focus has been shifted toward abstractive approaches. Given the aim of this paper, which is a comparative study of abstractive Arabic ATS, the related work discussed in this section will be limited to the abstractive related work. Abstractive ATS systems require a deeper understanding and analysis of the original text [28]. Abstractive ATS systems focus on generating a summary after understanding the main ideas in the original text without using the same sentences. Such abstractive approaches use NLP methods to create the summary text without copying sentences from the input (original) text. The abstractive ATS approaches are generally categorized into three main categories, structure-based, semantic-based and deep learning-based approaches [29]. The structure-based approaches use pre-defined structures such as graphs and ontologies. Whereas the semantic-based methods mainly focus on using the natural language generation systems and text semantic representation to generate the summary. Deep learning-based approaches use deep neural networks to build ATS systems, which tend to report encouraging results in the ATS systems. Precisely, the sequence-to-sequence learning (seq2 seq) model has shown impressive results in abstractive ATS with the English language [30]. For such approaches, Recurrent Neural Network (RNN) [31] with an attention encoder-decoder is utilized. For example, [PERSON] et al. [30] proposed a seq2 seq model for ATS with various phases such as the conversion of the dataset data to plain texts, storing the original text (news articles) and the summaries separately, word segmentation to process the data, and representing the words with pre-trained vectors. The experiments were conducted with a Chinese public dataset made available by NLPCC2017 shared task3 ([[http://tcci.ccf.org.cn/conference/2017/taskdata.php](http://tcci.ccf.org.cn/conference/2017/taskdata.php)]([http://tcci.ccf.org.cn/conference/2017/taskdata.php](http://tcci.ccf.org.cn/conference/2017/taskdata.php)), accessed date 2 November 2022). The dataset consists of 2K texts without matching summaries for testing and around 40K document-summary pairs for training. The reported results were 0.34, 0.21 and 0.30 on ROUGE-1, ROUGE-2 and ROUGE-L, respectively. Later, such steps are utilized for training the model with bidirectional and unidirectional Long Short-Term Memory (LSTM) for the encoder and decoder, respectively. [PERSON] et al. [32] have also proposed a method using the attention mechanism. Bidirectional gated recurrent units' architecture has been utilized in the proposed method to perform the encoding and decoding tasks.