###### Abstract
CERT employs back-translation for text augmentation, which generates new sentence representations by translating them into another language and then back to the original language [9]. However, in domain-specific contexts such as biomedical text, back-translation presents challenges, as minor shifts in meaning during translation can introduce critical errors [10; 11; 12]. Furthermore, negative sampling strategies based solely on randomly paired sentences may not sufficiently differentiate subtle clinical topics, highlighting a need for more nuanced approaches--such as leveraging metadata or domain-specific cues--to create effective negative samples. Given these constraints, our research introduces the Discharge Summary CRPT (Contrastive Representations Pre-Training) model, which enhances clinical language representations without relying on text augmentation techniques like back-translation. Our model replaces the traditional cross-entropy loss in the NSP task with a contrastive loss to better capture sentence-level semantics. In addition, we carefully design negative pairs from both intra-document and cross-document sentence pools to ensure that the model discriminates between closely related and truly unrelated content. Furthermore, we integrate the Whole-Word Masking (WWM) technique into the MLM phase, as proposed in ERNIE [13], to enhance performance by considering entire words rather than subword tokens in isolation. This masking strategy can be further tailored to domain-specific entities or medical concepts, potentially allowing for \"concept-level\" masking that captures the continuity of specialized terms. Such enhancements aim to preserve the nuanced semantics in clinical corpora, where partial sub-word masking can otherwise undermine context understanding. Unlike CERT, which combines back-translation with MoCo-style (momentum contrast) learning on general-domain text, and ERNIE, which introduces Whole-Word Masking (WWM) without any contrastive component, our CRPT model is the first to fuse contrastive loss with WWM directly on clinical discharge summaries. This tight coupling preserves multi-token medical entities while simultaneously enforcing sentence-level discrimination. As a result, CRPT yields substantially richer semantic representations than existing biomedical NLP models. A preliminary version of this work was previously presented as a short paper at the IEEE International Conference on Healthcare Informatics (ICHI 2021), focusing on the initial development of the CRPT model [14]. In this study, we focus on the Discharge Summary CRPT model, inspired by Discharge Summary BERT [4], but with significant improvements in language representation. Tested on the BLUE (Biomedical Language Understanding Evaluation) benchmark [15], our model achieved superior performance in natural language inference and sentence similarity tasks compared to traditional models. Additionally, we introduce Bio+Discharge Summary CRPT, which combines biomedical and clinical domain pre-training, further boosting performance in medical NLP tasks. Throughout our experiments, we also examine how varying the margin parameter in contrastive loss and adjusting other hyperparameters impact performance, and we highlight representative error cases to clarify where the model still struggles--an essential step for real-world clinical deployment. Moreover, we briefly discuss how future work could extend CRPT to a multi-task learning setup (e.g., integrating classification, summarization, and QA) or increase the batch size to broaden negative sample diversity. Our contributions are as follows:  
1. We propose the Contrastive Representations Pre-Training (CRPT) model for clinical language tasks, enhancing sentence-level semantics by integrating contrastive loss and WWM;
2. We demonstrate significant improvements in medical NLP tasks using the BLUE benchmark, including natural language inference and sentence similarity tasks;
3.