# Beyond Lexical Boundaries: LLM-Generated Text Detection for Romanian Digital Libraries
## 2 Related Work
### Generative Models  
#### 2.2.1 Romanian Specific Language Models  
Most of the previous models were released for high-resourced languages, such as English or multilingual models, with lower performance than the language-dedicated models--these multilingual versions are presented in detail in the following sub-section. With the development of language models and the increasing need for dedicated resources, Romanian-dedicated language models emerged. With the introduction of RoGPT2 [29], state-of-the-art performance was achieved for Romanian text generation by leveraging a Romanian version of GPT2 [2]. RoGPT2 was trained using the largest available corpus for Romanian and was evaluated against six tasks from the LiRo benchmark [30], specifically (1) text categorization and dialect classification, (2) sentiment analysis, (3) semantic textual similarity, (4) machine translation, (5) QA and zero-shot cross-lingual learning, and (6) language modeling. For all the targeted tasks except zero-shot cross-lingual learning, RoGPT2 outperformed other BERT-based models for Romanian, such as RoBERT or BERT-ro-base. Competitive results were achieved for another task, namely grammar error correction (RoGEC), leveraging the RONACC corpus,and demonstrating its capability of generating grammatically correct text. RoGPT2 was released in three versions: base (124 M parameters), medium (354 M parameters), and large (774 M parameters). A similar model was developed for automatic text generation called MCBGPT-2 [31]. The model was trained exclusively on a corpus of 24,600 news articles collected between March and October 2021, manually labeled as true or fake news with different polarities (i.e., positive or negative). Following the same methodology as RoGPT2, the model was trained using 131 M parameters. Validation was performed via statistical analysis between the original and generated news items. Evaluation metrics such as tokens distribution, BERTScore [32], BLEU [33], and ROUGE [34] metrics were considered. Results show that RoGPT2 [29] exceeded the performance of the MCBGPT-2 model in terms of length and metrics scores of the generated sentences. However, MCBGPT-2 achieved better results for BERTScore, a metric that uses a pre-trained model to understand the generated text and the text of reference for comparison. RoSummary [35] is a generative language model for abstractive summarization. The model is based on the RoGPT2 architecture, trained to predict the next token using the previous sequence. Four control tokens were used to indicate the characteristics of generated text, namely NoSentences (i.e., number of sentences that the summary should have), NoWords (i.e., number of words generated in the summary), RatioTokens (i.e., proportion in which the sequence of words in the summary must be longer than the input), and LexOverlap (i.e., ratio of 4 grams in the summary that appear in the reference text). The model generated grammatically correct texts and was evaluated using ROUGE and BERTScore. Experiments were conducted with three versions of the model with a context size of 724: base (12 layers, batch size of 128), medium (24 layers, batch size of 24), and large (36 layers, batch size of 16). The medium version achieved the best results, combined with beam search decoding (ROUGE 34.67% and BERTScore 74.34%). However, using control tokens improved BERTScore by up to 2%. Higher scores were achieved when only one control token was used. A more recent model based on GPT-3 architecture [16] is GPT-NEO-RO ([[https://huggingface.co/dumitrescustefan/gpt-neo-romanian-780m](https://huggingface.co/dumitrescustefan/gpt-neo-romanian-780m)]([https://huggingface.co/dumitrescustefan/gpt-neo-romanian-780m](https://huggingface.co/dumitrescustefan/gpt-neo-romanian-780m)), accessed on 8 January 2024). The model has 780 M parameters, making it one of the largest language models available for Romanian.