# NoSummary: Control Tokens for Romanian News Summarization
## 1 Introduction  
A remarkable development in Natural Language Processing (NLP) towards creating models that understand human languages has been observed in recent years. Text generation is one of the main challenges in the field of NLP, and this task has seen an important development after the introduction of Transformers [1]. The Transformer uses an encoder-decoder architecture, self-attention, and positional encodings to facilitate parallel training. The GPT-2 model developed by OpenAI [2] was the first model with remarkable text generation capabilities. GPT-2 was trained for predicting the next token in a sequence and could easily be adjusted for specific tasks. The follow-up improving the GPT-3 model [3] is more than 10-times larger in terms of the parameters and deduces the task only from the provided prompt. There have been several open-source variations of the model, such as GPT-Neo [4] and GPT-J [5]. Other architectures consider a unified framework to cover text-to-text formats and convert text-based language problems, such as the Text-To-Text Transfer Transformer (T5) [6]. This model can perform zero-shot learning and deduce the task from the context of the prompt received as the input, even if it was not presented in the training stage. For the Romanian language, there are not many specific resources (i.e., pre-trained models and datasets), although there has been significant progress in recent years. The most notable models for Romanian consider the BERT architecture (e.g., RoBERT [7], BERT-base-ro [8], Distil-BERT [9]) and the GPT-2 architecture (e.g., RoGPT2 [10]) and were developed in the last 2 years. Romanian has only one available benchmark, namely LiRo [11]. However,the models are small compared to their English counterparts, and there are no available datasets for common NLP tasks. Overall, Romanian remains a low-resource language with low international usage ([[https://www.worlddata.info/languages/romanian.php](https://www.worlddata.info/languages/romanian.php)]([https://www.worlddata.info/languages/romanian.php](https://www.worlddata.info/languages/romanian.php)); last accessed on 20 October 2022), despite recent efforts in terms of datasets and models; as such, we argue for the necessity of our efforts to develop tools tailored to this language. Text summarization is a task of particular importance in NLP centered on extracting critical information from the text using two approaches. First, extractive summarization involves removing the most-important phrases or sentences that include the main ideas of a text. Second, abstractive summarization considers the generation of a new summary starting from the text. One of the most popular datasets in English used for this task is _CNN/Daily Mail_[12], having a total number of 280,000 examples; the dataset was afterward extended to other languages, including French, German, Spanish, Russian, and Turkish, thus generating the large-scale multilingual corpus _MLSUM_[13]. Another dataset used in studies for abstractive summarization is Extreme Summarization (_X-Sum_) [14] to generate a short, one-sentence summary for each news article; _X-Sum_ was derived from BBC news and consists of 220,000 examples. Another dataset is _Webis-TLDR-17 Corpus_[15] with approximately three million examples constructed with the support of the Reddit community. Extractive summarization in Romanian has been previously tackled by [PERSON] et al. [16] and [PERSON] et al. [17] with small evaluation datasets. We now introduce the first dataset for Romanian abstractive summarization ([[https://huggingface.co/datasets/readerbench/ro-text-summarization](https://huggingface.co/datasets/readerbench/ro-text-summarization)]([https://huggingface.co/datasets/readerbench/ro-text-summarization](https://huggingface.co/datasets/readerbench/ro-text-summarization)); last accessed on 20 October 2022).