# The Specialist's Paradox: Generalist AI May Better Organize Medical Knowledge
###### Abstract  
This study investigates the ability of six pre-trained sentence transformers to organize medical knowledge by performing unsupervised clustering on 70 high-level Medical Subject Headings (MeSH) terms across seven medical specialties. We evaluated models from different pre-training paradigms: general-purpose, domain-adapted, and from-scratch domain-specific. The results reveal a clear performance hierarchy. A top tier of models, including the general-purpose MPNet and the domain-adapted BioBERT and RoBERTa, produced highly coherent, specialty-aligned clusters (Adjusted Rand Index > 0.80). Conversely, models pre-trained from scratch on specialized corpora, such as PubMedBERT and BioClinicalBERT, performed poorly (Adjusted Rand Index < 0.51), with BioClinical-BERT yielding a disorganized clustering. These findings challenge the assumption that domain-specific pre-training guarantees superior performance for all semantic tasks. We conclude that model architecture, alignment between the pre-training objective and the downstream task, and the nature of the training data are more critical determinants of success for creating semantically coherent embedding spaces for medical concepts. t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
t  
The field of natural language processing (NLP) has made significant strides in addressing this challenge, evolving from frequency-based techniques like term frequency-inverse document frequency [7] to sophisticated deep learning models that generate dense vector representations, or embeddings [8]. Early breakthroughs with models such as Word2 Vec [9] and GloVe [10] enabled the capture of word-level semantics, but these non-contextual models were limited in their ability to represent the meaning of entire sentences or documents [11]. The introduction of the transformer architecture [12], most notably with the bidirectional encoder representations from transformers (BERT) model [13], marked a paradigm shift by creating deeply contextualized word embeddings that account for the surrounding text [14]. Building on this, sentence-transformer variants (e.g., SBERT) trained with contrastive objectives produce fixed-size vectors whose cosine distance reflects semantic relatedness, enabling fast similarity search and clustering [15]. These models employ specialized architectures, such as Siamese or triplet networks, which process two or three sentences independently through a shared transformer network [16]. They are fine-tuned on massive datasets of sentence pairs (often numbering over a billion) [17]. These models embed any span of text--single words, multi-word terms, or full sentences--into the same semantic space [18]. The unique characteristics of biomedical language [19]--its specialized vocabulary and distinct syntactic structures--led to the recognition that models pre-trained on general-domain text might be inadequate for medical NLP tasks [20]. In response, two dominant strategies for creating domain-specific language models emerged. The first is domain-adaptive fine-tuning [21], exemplified by models like BioBERT [22]. This approach initializes a model with parameters from a pre-trained general-domain model (like BERT) and continues the pre-training process on large biomedical corpora, such as PubMed abstracts and PubMed Central (PMC) full-text articles. The underlying rationale is to leverage the robust general linguistic knowledge learned from a massive general corpus and adapt it to the nuances of the specialized domain. The second strategy is from-scratch training [23]. Models like PubMedBERT epitomize this approach, as they are trained entirely from scratch using only in-domain data [24]. This involves building a new vocabulary directly from the biomedical corpus (e.g., PubMed abstracts) and training all model parameters exclusively on this data.