## 4 Tailoring Pre-Trained Transformer Models for Detection in Spanish
### An Overview of BERT and Its Optimized Variants  
BERT (Bidirectional Encoder Representations from Transformers) is a language model architecture renowned for its proficiency in capturing long-range dependencies in sequential data. The foundation of BERT lies in its sophisticated self-attention mechanisms, enabling it to model word interactions within a sentence and to weigh the relevance of each word's context. Thanks to these properties, BERT generates contextualized word embeddings that depend on the entire surrounding context of each word in the sentence, thus capturing intricate language dependencies. To do so, BERT utilizes two primary pre-training strategies: the Masked Language Model (MLM) and Next Sentence Prediction (NSP). In the MLM process, BERT selectively masks words within sentences, prompting the model to predict hidden words by considering both their left and right contexts within the sentence. Additionally, NSP involves training BERT to discern whether pairs of sentences randomly selected from a corpus are contiguous or from distinct parts of the text, thereby helping the model capture meaningful relationships between sentences. The combination of these techniques enables BERT to comprehend the intricate nuances of language semantics and syntax.  
In addition to its fine-tuning on various natural language understanding tasks, as highlighted throughout this section, BERT has been adapted to create multilingual models with support for a wide range of languages, including Spanish, which is the focus of our research on fake news detection.  
The notable performance displayed by BERT in NLP tasks [52, 53, 54] has paved the way for a multitude of optimizations and variations, including AlBERT [55], DistilBERT [56], and RoBERTa [57], whose main differences lie in their optimization goals and methods:  
* AlBERT, or A little BERT, aims to improve language models for increased efficiency and speed while making some trade-offs in performance. It focuses on resource efficiency and faster processing while still maintaining a performance level slightly below that of the original BERT model. This is particularly valuable for applications where speed and resource efficiency are crucial, such as mobile natural language processing [58, 59]. AlBERT accomplishes this through the redistribution of model parameters and the integration of various optimization techniques, which are out of the scope of the paper.
* DistilBERT, also known as Distilled BERT, shares a similar aim of enhancing language models by prioritizing efficiency and speed. It achieves this goal by creating a more compact and faster model while retaining much of the knowledge from the original BERT. DistilBERT is tailored for applications where lightweight and fast language models are essential, making it well suited for scenarios with limited computational resources [60,61].
* In contrast, optimizations in RoBERTa are focused on enhancing natural language processing performance. This is achieved by conducting more extensive pre-training, using a larger text corpus compared to BERT, encompassing a wide range of natural language tasks prior to fine-tuning for specific applications, and more effectively adjusting hyperparameters. These enhancements empower the model to capture more intricate and nuanced language patterns, ultimately leading to superior performance in a variety of natural language processing tasks [62,63].  
While we acknowledge the importance of optimizing efficiency and resource management, our primary objective is to enhance the model's performance to achieve the highest precision in the NLP task of news article classification within the scope of politics and in the Spanish language, which have received little attention in the literature to date. Consequently, we opted to fine-tune both the BERT and RoBERTa models, with a particular emphasis on those with support for the Spanish language, as outlined in the next section.