## 4 Tailoring Pre-Trained Transformer Models for Detection in Spanish
### Selecting Pre-Trained Models for Detecting Fake News in Spanish  
In the Hugging Face repository, there are numerous BERT and RoBERTa models of different sizes and purposes. In this section, we justify the use of the ones we have chosen for our experiments on fake news detection in Spanish. * Available BERT models. On one hand, there are two variants of BERT, known as _BERT-base_ and _BERT-large_. Both are bidirectional models pre-trained on extensive text data in English, serving as the foundation for fine-tuning in a wide range of NLP applications. Specifically, _BERT-base_ is the lighter version with fewer parameters (110 million), resulting in a smaller model size. In contrast, _BERT-large_ is a more substantial and powerful iteration, boasting more parameters (340 million) and greater representational capacity. As a consequence, _BERT-large_ proves more effective but also demands more computational resources, making it the preferred choice for complex natural language processing tasks. Moreover, in addition to the variations in size and capacity between _BERT-base_ and _BERT-large_, both models come in _cased_ and _uncased_ versions. The selection between these variants should be influenced by the nature of the data and the specific requirements of each NLP task. _Cased_ versions are better suited when preserving uppercase and lowercase distinctions is essential, such as in proper nouns and specialized contexts, for tasks like named entity recognition or machine translation [64, 65, 66]. Conversely, _uncased_ versions convert all text to lowercase before training, which is beneficial when capitalization carries no semantic significance. These models find value in text classification, natural language processing in social media, and sentiment analysis in informal texts [67, 68, 69]. Various language-specific versions of BERT models are also available. Two notable examples include _BERT-multilingual_ and _BETO_ models, both of which are offered in _base/large_ and _cased/uncased_ versions. On one hand, _BERT-multilingual_ is a BERT variant pre-trained on a diverse set of languages, including Spanish, enabling it to capture general language representations across a wide range of linguistic backgrounds [70]. On the other hand, _BETO_ distinguishes itself as a language-specific model explicitly trained for the Spanish language. It delivers outstanding performance across a diverse range of natural language processing tasks in Spanish, making it particularly well suited for applications like sentiment analysis, topic categorization, and various other forms of text classification [71,72]. * Available RoBERTa models. RoBERTa is available in _base_ and _large_ versions, with all models being _case_ sensitive. Additionally, there are multilingual variants, such as the _xlm-roberta-base_ model, which was designed as a cross-lingual language model based on RoBERTa to operate effectively in 100 different languages and pre-trained on an extensive dataset of 2.5 TB of filtered CommonCrawl data [73]. Furthermore, there are specialized RoBERTa models that have been pre-trained on comprehensive Spanish-language corpora. For example, the _bertin-roberta-base-spanish_ model that was developed as part of the BERTIN project [74] and utilizes the mC4 corpus [75], a multilingual variant of the C4 dataset supporting over 100 languages sourced from the public Common Crawl web scrape. This dataset includes a total of 6.6 billion pages, with 416 million of them in Spanish [76]. Additionally, the _roberta-base-bne_ model was created in 2021 by the Barcelona Supercomputing Center on 570 GB Spanish-language high-quality documents extracted from the National Library of Spain (BNE) (The acronym _BNE_ in the model name _RoBERTa-BNE_ refers to the Biblioteca Nacional de Espana in Spanish (National Library of Spain in English). Its website is [[https://www.bne.es/en](https://www.bne.es/en)]([https://www.bne.es/en](https://www.bne.es/en)) (accessed on 1 November 2023).) using the supercomputer MareNostrum4 [77].