## 3 Materials and Methods
### Adaptive Learning of the MedicoBERT Model
#### 3.2.2. MedicoBERT Model Pre-Training  
To pre-train MedicoBERT, the tokenizers and medical literature datasets were employed. Furthermore, the datasets were subjected to a minimal pre-processing step, which involved the removal of punctuation marks such as \"?\", \",\", \",\", \",\", \", \", \", \", \", \", \", \", \", and \"&\". These characters are irrelevant for model training and may potentially introduce noise during the process. Finally, the text was normalized by converting it to lowercase. This normalization ensured that all words were represented consistently, thus facilitating the subsequent processing.  
MedicoBERT was trained using the Masked Language Modeling (MLM) technique. This technique involves training an LLM to predict the missing words in a text sequence. In this case, 15% of words in the medical literature corpus were randomly replaced with the special token '<mask>'. The primary objective of this technique is to learn the relationships between words and their context, a crucial aspect of natural language processing. It is important to note that this technique does not require the dataset to be labeled. Below is a fragment of the masking of a text from the CORD-19 dataset, where the objective is to predict \(P(m)\), i.e., which word should replace the special token to give the sentence a more accurate meaning based on the medical context.  
La pandemia por COVID-19 afects especialmente a pacientes con <mask> con mayor incidencia y mortalidad (The COVID-19 pandemic especially affects patients with <mask> with increased incidence 248 and mortality)  
\[P(m)=\{c\alphacer(cancer)|urticaria(hives)|de(from)\}\]  
Figure 3: Text-to-number conversion performed by the new tokenizer.  
The next task used to train the medical model was _Next Sentence Prediction_ (NSP). This technique involved splitting each dataset into sentences and creating two different sets, as follows:  
* Related pair set: This set contains 50% of the original dataset's sentences, where the subsequent sentence in each pair is the logical continuation of the first.
* Random pair set: This set contains the remaining 50% of the original dataset's sentences, where the sentences are randomly combined without any logical relationship.  
The following is an example of training the medical model using NSP, where the model is presented with \"Sentence A\" and \"Sentence B\" and is asked to predict (P) whether \"Sentence B\" is the logical continuation of \"Sentence A\".  
**\"Sentence A\":**_El fracaso renal agudo FRA en pacientes hospitalizados por COVID-19 se presenten el 0.5-2.5% y es un factor de mal pronostic (Acute renal failure ARF occurs in 0.5-2.5% of COVID-19 hospitalized patients and is a poor prognostic factor)._  
**\"Sentence B\":**_Los mecanismos de affectacion renal no estan completamente aclarados (The mechanisms of renal involvement are not completely elucidated)._  
**(P): IsNext**  
This task crucially compels the model to engage in a profound semantic analysis of medical text. This process is instrumental in enabling the model to discern the relationships between sentences, thereby enhancing its capacity to produce coherent and meaningful text.