## 6 Conclusions  
In this paper, we demonstrated the effectiveness of applying whole-word masking (WWM) and Contrastive Representation Learning (CRL) to improve the performance of clinical and biomedical language models. Our proposed CRPT models, which leverage WWM and CRL, showed superior results on the BLUE benchmark dataset compared to baseline models such as BERT and BlueBERT, particularly in sentence-level tasks. The inclusion of CRL allowed for better semantic understanding between sentences, while WWM preserved word-level meaning, improving the overall contextual comprehension in medical texts. For qualitative evaluation, we analyzed random pairs of consecutive sentences from discharge summary notes and observed a noticeable reduction in the distance between consecutive sentences, signifying enhanced sentence relation modeling. Additionally, CRPT models proved highly effective in capturing nuanced semantic relationships, contributing to better performance across sentence and token-level tasks. These findings confirm that CRPT models provide a more refined and specialized understanding of medical language, which is crucial for improving clinical NLP tasks like named entity recognition (NER), relation extraction, and sentence inference. Our approach can enhance clinical decision support systems and medical documentation processing, potentially leading to more accurate diagnoses and treatments. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Model** & **Masking** & **Method** & **MedNLI** & **BIOSSES** & **ShARe/CLEFE** & **i2b2-2010** \\ \hline Discharge Summary BERT & Random & NSP & 0.785 & 0.740 & 0.814 & 0.736 \\ \hline Discharge Summary BERT & Whole Word & NSP & 0.811 & 0.761 & 0.823 & 0.750 \\ \hline Discharge Summary CRPT & Random & CRL & 0.816 & 0.767 & 0.825 & 0.743 \\ \hline Discharge Summary CRPT & Whole Word & CRL & 0.819 & 0.773 & 0.827 & 0.754 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Ablation experiments. Future Work will focus on expanding our dataset to include all 15 note types from the MIMIC-III database, allowing for more diverse training data and a broader evaluation across various medical downstream tasks. We also plan to explore the integration of domain-specific masking techniques to further enhance the model's ability to understand medical terminology. Lastly, we will investigate the potential of scaling the batch size during training to leverage a wider variety of negative samples in CRL, potentially improving model performance in large-scale applications. [PERSON] and [PERSON] developed the main idea and conducted the data collection, analysis, and evaluations. [PERSON] provided high-level guidance and supervision to this study and provided medical domain expertise and supervision. All authors have read and agreed to the published version of the manuscript. This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Education (grant number) (NRF-2022R1I1A1A01073591). This research was supported by a grant from the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health and Welfare, Republic of Korea (grant number: HI21C1137). Not applicable. Informed Consent Statement: Informed consent was obtained from all subjects involved in this study. We used the BLUE benchmark dataset, which can be downloaded from the GitHub repository [[https://github.com/ncbi-nlp/BLUE_Benchmark](https://github.com/ncbi-nlp/BLUE_Benchmark)]([https://github.com/ncbi-nlp/BLUE_Benchmark](https://github.com/ncbi-nlp/BLUE_Benchmark)), accessed in September 2023. The trained Model and the codes will be available from the corresponding author upon reasonable request. During the preparation of this manuscript, the authors used ChatGPT (OpenAI o3, accessed April 2025) to assist with translation from Korean to English and to refine the flow of several sentences.