Clinical ModernBERT: An efficient and long context encoder for
  biomedical text
Authors: name: Simon A. Lee; name: Anthony Wu; name: Jeffrey N. Chiang
Published: 2025-04-04 22:14:12

Abstract: We introduce Clinical ModernBERT, a transformer based encoder pretrained on
large scale biomedical literature, clinical notes, and medical ontologies,
incorporating PubMed abstracts, MIMIC IV clinical data, and medical codes with
their textual descriptions. Building on ModernBERT the current state of the art
natural language text encoder featuring architectural upgrades such as rotary
positional embeddings (RoPE), Flash Attention, and extended context length up
to 8,192 tokens our model adapts these innovations specifically for biomedical
and clinical domains. Clinical ModernBERT excels at producing semantically rich
representations tailored for long context tasks. We validate this both by
analyzing its pretrained weights and through empirical evaluation on a
comprehensive suite of clinical NLP benchmarks.