###### Abstract  
Despite remarkable advances in neural language models, a substantial gap remains in precisely interpreting the complex semantics of Electronic Medical Records (EMR). We propose Contrastive Representations Pre-Training (CRPT) to address this gap, replacing the conventional Next Sentence Prediction task's cross-entropy loss with contrastive loss and incorporating whole-word masking to capture multi-token domain-specific terms better. We also introduce a carefully designed negative sampling strategy that balances intra-document and cross-document sentences, enhancing the model's discriminative power. Implemented atop a BERT-based architecture and evaluated on the Biomedical Language Understanding Evaluation (BLUE) benchmark, our Discharge Summary CRPT model achieves significant performance gains, including a natural language inference precision of 0.825 and a sentence similarity score of 0.775. We further extend our approach through Bio+Discharge Summary CRPT, combining biomedical and clinical corpora to boost downstream performance across tasks. Our framework demonstrates robust interpretive capacity in clinical texts by emphasizing sentence-level semantics and domain-aware masking. These findings underscore CRPT's potential for advancing semantic accuracy in healthcare applications and open new avenues for integrating larger negative sample sets, domain-specific masking techniques, and multi-task learning paradigms. EPF 2025 15 6541 10.3930/app15126541 10. At the forefront of these advancements is BERT (Bidirectional Encoder Representations from Transformers) [1], a model pre-trained on large corpora such as Wikipedia and books. BERT has significantly improved performance in natural language understanding (NLU) tasks, particularly when benchmarked against the General Language Understanding Evaluation (GLUE) dataset [2]. In the biomedical domain, specialized versions of BERT, such as BioBERT [3] and Clinical BERT [4], have emerged, further refining BERT's capabilities by pre-training on domain-specific corpora. BioBERT, for example, is trained on datasets like PubMed and PMC, which include specialized biomedical terminology, leading to better downstream task performance in biomedical applications [5; 6; 7]. Similarly, Clinical BERT has demonstrated superior performance in clinical tasks, especially when trained on discharge summaries from the MIMIC-III database, comprising over 200,000 notes. Despite these advancements, traditional BERT models, including their domain-specific adaptations, struggle to capture sentence-level semantics effectively. BERT's pre-training involves Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). While MLM is adept at learning token-level context by masking random tokens in a sequence and predicting them based on their surroundings, it struggles to capture broader sentence-level semantics. Similarly, NSP focuses primarily on determining the sequential relationship between sentences, which limits its ability to understand the deeper semantic relationships between sentences. Moreover, the cross-entropy loss function used in the NSP task fails to reflect semantic similarities accurately, leading to challenges in capturing nuanced sentence interplays [8]. These limitations become even more pronounced in complex clinical documents, where terms can be multi-token, and sentence continuity alone often does not suffice for accurate context modeling. Recent studies have explored contrastive self-supervised learning frameworks to address these limitations, demonstrating significant improvements in capturing sentence-level semantics in generic NLP tasks [9; 10]. For example, models like CERT (Contrastive Self-Supervised Encoder Representations from Transformers) have outperformed BERT in several tasks benchmarked on GLUE, emphasizing the potential of contrastive learning in enhancing language representations.