## 2 Materials and Methods
### Deep Learning Models  
In this study, we carefully chose five transformer-based models to tackle the clinical NER task in Spanish. In this work, we considered five transformer-based models, which are available at Hugging Face under the names bert-base-cased (bert), bsc-bio-es (bio-es), bsc-bio-ehr-es (bio-ehr-es), roberta-base-bne (roberta), and gpt2-finetuned-comp2 (gpt-2). We used various models in our study, each with different pre-training tasks. Some models, such as BERT, were pre-trained on the masked language modeling (MLM) task, where the model learns to predict masked words within a sentence. Similarly, RoBERTa, which is based on BERT, also follows this pre-training approach, but with improvements in data efficiency and model performance. We also utilized bio-es and bio-ehr-es, BERT-based models explicitly fine-tuned for the NER task, making it particularly suitable for extracting entities in clinical texts. We also incorporated GPT-2, which differs from BERT-based models as it was pre-trained using causal language modeling that involves predicting the next word in a sequence. With this combination of models, we aim to compare the accuracy and precision in identifying biomedical entities from our clinical data in Spanish with the proposed models.  
The selected models are described as follows:  
* Bert: Bidirectional Encoder Representations from Transformers [27] is a pre-trained deep learning model developed by Google for natural language processing tasks such as question answering and language inference. It is based on the transformer architecture and is trained on a large corpus of text data using unsupervised learning. BERT can be fine-tuned for a wide range of natural language understanding tasks and has achieved state-of-the-art results on a number of benchmarks, including the GLUE benchmark and SQuAD.
* roberta: It is an optimized bert model [28], which has been pre-trained by Text Mining Unit (TeMU) at the Barcelona Supercomputing Center, using the most extensive Spanish corpus known to date, with a total of 201,080,084 documents, compiled from the web crawlings performed by the National Library of Spain (Biblioteca Nacional de Espana) from 2009 to 2019.
* gpt-2: Generative Pre-trained Transformer is a language generation model developed by OpenAI. GPT-2 is trained on a massive dataset of over 40 GB of text data and can generate human-like text in various languages and styles. It can be used for various  
Figure 2: EHR note sample annotated using INCEpTION with BIO tagging.  
natural language processing tasks such as language translation, text summarization, and question answering. We used a fine-tuned version of GPT-2 available in _Hugging Face_ ([[https://huggingface.co/brad1141/gpt2-finetuned-comp2](https://huggingface.co/brad1141/gpt2-finetuned-comp2)]([https://huggingface.co/brad1141/gpt2-finetuned-comp2](https://huggingface.co/brad1141/gpt2-finetuned-comp2)), accessed on 2 October 2024). This version was fine-tuned on the token classification task. This supervised fine-tuning allowed the model to adapt its generative capabilities toward the specific task of entity recognition and categorization. This aligns with recent trends in NLP in which generative models are used for information extraction [29].
* bio-es: A model produced by the Barcelona Supercomputing Center based on the roberta model using a large biomedical corpora from a variety of sources in Spanish with a total of 1.1 B tokens across 2.5 M documents, allowing us to acquire specialized knowledge and language patterns relevant to the field of biomedicine in Spanish. Additionally, this model was fine-tuned on three NER tasks: PharmaCoNER, CANTEMIST, and ICTUSnet [30].
* bio-ehr-es: Like the bio-es model, bio-ehr-es is based on the roberta model and was fine-tuned in the same NER tasks as bio-es. The key difference is that bio-ehr-es incorporates electronic health record (EHR) information drawn from a corpus with more than 514k clinical documents containing 95 million tokens [30].