## 3 Results and Discussion
In comparison, our database consists of 200 discharge notes containing 1842 sentences and 13,998 labeled entities belonging to 14 classes, making it shorter in size but more diverse in entity types and with a higher density of labeled entities per sentence than the CANTEMIST dataset, which focuses solely on a single entity. Figure 3: The confusion matrix for the NER task in our dataset constructed from maternal EHRs using the bsc-bio-es model with a learning rate of 8.0 \(\times\) 10\({}^{-5}\). Based on our findings, the top-performing models were bio-ehr-es and bio-es, surpassing the performance of GPT-2, BERT, and Roberta. This outcome is in line with our expectations since bio-ehr-es and bio-es are built upon the foundation of Roberta and fine-tuned using Spanish biomedical resources in an NER task. However, it was unexpected that bio-ehr-es would yield lower performance than bio-es. We initially hypothesized that bio-ehr-es would outperform bio-es due to its additional pre-training on electronic health records (EHRs). Although the F1 scores are very close between the two models, suggesting comparable performance across models, bio-es slightly outperformed bio-ehr-es (\(0.565\pm 0.07\) vs. \(0.631\pm 0.048\)). Bio-es was trained on general biomedical data, which may have made it more versatile for various clinical tasks, including less frequent or more descriptive entities. In contrast, if bio-ehr-es was primarily pre-trained in structured or specific records, it may have acquired a narrower or more rigid vocabulary, which is less useful in the context of our discharge notes. Regarding the performance achieved by each entity, the highest scores were obtained by signs and symptoms (with a partial F1 score of 0.749) and negation entities (with a partial F1 score of 0.726). This finding is particularly significant because identifying negation is crucial for determining the presence or absence of signs and symptoms, serving as a prerequisite for future projects aimed at maternal risk electronic phenotyping. Furthermore, other entities such as disease and syndrome, which are important for electronic phenotyping, achieved a partial F1 score of 0.695. On the other hand, the entities associated with the prefix \"I\" exhibited the poorest performance, likely due to their limited number of samples. This suggests the need for future work involving additional note annotations and dataset expansion. It is worth noting that certain entities can be prone to confusion or misclassification, even by trained human annotators. We constructed an annotation guide based on the Unified Medical Language System (UMLS) Semantic Network. At the end of the annotation process, we identified some discrepancies between the interpretation of entities T080 (qualitative concept) and T184 (symptom). Specifically, in some cases, one annotator interpreted a descriptive attribute as part of the symptom, while the other considered it a separate qualitative concept. For instance, in the expression \"red breasts\", one annotator labeled \"red\" as a qualitative concept (T080), while the other labeled it as part of the symptom (T184). This variation reflects the inherent challenge of using UMLS in clinical NER tasks, as qualitative attributes may be contextually linked to symptoms. This observation aligns with the confusion matrix, where several qualitative concepts (T080) are detected by the model as signs or symptoms (T184). In the confusion matrix, it is evident that the majority of false-positive samples correspond to words that do not belong to the entity class (\"O\"). However, it is possible that during the annotation process, certain entities were overlooked, leading to false positives where the model detected entities that were not marked by the annotator.