# Authorship Attribution in Less-Resourced Languages: A Hybrid Transformer Approach for Romanian
## 2 Related Work on Authorship Attribution
### Transformer-Based Models  
Transformer-based models, exemplified by architectures like BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer), excel in contextualizing language features over longer contexts. The application of Transformer-based models in authorship attribution involves fine-tuning the pre-trained models on labeled datasets, enabling them to discern uniqueness in the writing styles of different authors. This approach mitigates the need for extensive feature engineering and enhances the model's adaptability across diverse genres, languages, and writing styles. [PERSON] et al. [45] presented an authorship attribution use case in the domain of Islamic law, investigating the performance of recent Arabic pre-trained Transformer-based models, such as AraBERT, AraELECTRA, ARBERT, and MARBERT, in the context of legal texts. Given the absence of a dedicated dataset, the authors built their corpus using digital resources from Islamic law. Experimental results revealed that ARBERT and AraELECTRA outperform other models, achieving an impressive 96% accuracy. The study concluded that pre-trained Transformer models, when fine-tuned, yield the best results. A recent research [46] used the authors' distinctive writing patterns for profiling purposes. Existing methods employing handcrafted features for classification tasks have shown limitations, particularly in out-of-domain scenarios. To overcome this, the paper introduced PART (Pre-trained Authorship Representation Transformer), a trained model focused on learning authorship embeddings instead of semantics. By leveraging pairsof text fragments written by the same author, the model determined authorship through cosine similarity evaluation, allowing for zero-shot generalization. The proposed model based on a pre-trained Transformer with an LSTM head was trained on diverse datasets incorporating literature, anonymous blog posts, and corporate emails. Evaluation results showcased zero-shot accuracies of 72.39% and 86.73% in determining authorship from a set of 250 authors. The paper concluded with qualitative assessments of the model's representations through data visualizations, providing insights into profiling features, such as book types, gender, age, and author occupation. A different study [47] investigated the application of Transformer-based fine-tuned models. Because these models lack the transparency of traditional methods that rely on stylometric features to quantify style and maximize the distance between texts, the authors implemented a BERT-based model for authorship verification and evaluated its predictions using an adapted LIME explainer and an attention-based feature extraction procedure. A comparative analysis between traditional and Transformer-based approaches was conducted, emphasizing explainability through input alteration to verify the retrieval of influential features. [PERSON] et al. [48] proposed a study focusing on short texts, having limited information about the author. Leveraging pre-trained language models, the study proposed a model that combines BERTweet, a pre-trained language model for English tweets, with the capsule network. This combination proved effective in capturing deep features of sentence representations, leading to improvements in authorship attribution tasks for tweets. The model also incorporated user writing styles, achieving state-of-the-art results on a known tweet dataset. [PERSON] et al. [49] introduced a Transformer-based NN architecture designed to attribute anonymous manuscripts to their authors using only the text body and author names in the bibliography. In order to develop and evaluate the method, the researchers compiled the biggest authorship identification dataset to date, incorporating over 2M research papers publicly available on arXiv. In the arXiv subsets containing up to 2K different authors, the method achieved an impressive authorship attribution accuracy, correctly attributing up to 95% of the papers.