## 5 Discussion  
Our model was pre-trained using Whole-word masking (WWM) and Contrastive Representation Learning (CRL) techniques. As evidenced by the experimental results, our Bio+Discharge Summary CRPT model outperformed BlueBERT [7] in three out of the four tasks. The Accuracy, Pearson correlation, and F1 score of the Bio+Discharge Summary CRPT model were 0.847, 0.848, and 0.761, respectively, compared to BlueBERT, which achieved 0.841, 0.839, and 0.757. Notably, the performance improvement was more pronounced in sentence-level tasks than in token-level tasks. In particular, the Named Entity Recognition (NER) task demonstrated comparable results, with a slight edge favoring BlueBERT in performance. MedNLI, the sentence-level task, exhibited improved performance as the overall data used for training increased. The Discharge Summary CRPT model achieved results close to BlueBERT's for the ShARe/CLEFE and i2b2-2010 tasks, with a 2.2% gap observed in MedNLI. However, compared to Discharge Summary BERT, our CRPT model showed a significant improvement of 3.4%. Furthermore, Bio+Discharge Summary CRPT outperformed BlueBERT [7] despite being trained with fewer datasets. The BIOSSE task exhibited a notable performance difference based on the biomedical data, and because this task evaluates sentence similarity, models utilizing contrastive representation learning achieved better performance than the baseline model. For the ShARe/CLEFE task, despite using various types of MIMIC-III data [17], we only utilized the discharge summary note type in this study. Since WWM randomly selects tokens, certain domain-specific entities in the ShARe/CLEFE dataset might be considered unimportant during pre-training. While the CRPT method improved performance for learning models with the same data, comparing precision and recall metrics used to calculate the F1 score, Bio+Discharge Summary CRPT scored 0.849 in precision and 0.825 in recall, compared to BlueBERT, which scored 0.844 and 0.832, respectively. This suggests that in the CRPT model, instances that should be labeled as true positives are sometimes mistakenly treated as negatives. Future research could address this by incorporating all available MIMIC-III data [16] or by designing masking methods for domain-specific entities [27; 28; 29]. Meanwhile, the i2b2-2010 task, which evaluates token-level predictions, could extend the relationship between entities to capture global relationships between sentences. From this perspective, CRPT offers advantages over traditional learning methods. The Discharge Summary CRPT model performed similarly to BlueBERT for sentence-level tasks, while Bio+Discharge Summary CRPT surpassed it. Figure 6: Sentence-level similarity heatmaps: (**a**) baseline BERT (right); (**b**) CRPT (left). Darker shades denote higher similarity. The pale horizontal/vertical bands in panel (**b**) correspond to sentences from a different patient (5-A/5-B’). We also performed additional experiments to isolate the effects of WWM and CRL. Discharge Summary BERT [3] served as our baseline, and WWM and CRL were individually applied to compare their performance, as shown in Table 6.  
WWM involves masking entire tokens that form a word simultaneously, enhancing language understanding. As shown in Table 6, WWM improved model performance in both sentence-level and token-level tasks. While Discharge Summary BERT with WWM and NSP did not use CRL, our model still demonstrated improved word embeddings, mainly reflecting sentence-level semantic similarity. In the i2b2-2010 task, this model employing only WWM outperformed the model that only used CRL. The application of CRL significantly enhanced model performance, particularly for sentence-level evaluations. We observed substantial performance improvements when CRL was applied, regardless of the masking method. Moreover, additional performance gains were realized when CRL was combined with WWM. We visualized the similarity between five sentence pairs using a heatmap for qualitative evaluation, as illustrated in Table 5 [30].