Where Does the Signal Live?
A Web Data Recipe for Medical Encoder Pretraining

bofeng huang

bofenghuang

Follow

Sun Jacques

SunJacques

Follow

Diane Bouchacourt

diane-bouchacourt

Follow

Nicolas Barascud

nbara

Follow

Fajwel Fogel

fajwel

Follow

🤗 Blog | 📄 Paper | 💻 Code | 🌐 FineMed | 🩺 DoctoBERT

TL;DR

For decoder large language models (LLMs), pretraining data curation is now widely studied: model-based filters score documents for signals such as educational quality, and an LLM rephrases them into forms with greater training utility. Most domain-specific encoders have not followed this shift. Their corpora are assembled by hand from a small number of canonical in-domain sources, which caps their scale and diversity. The bottleneck is more severe outside English.

We adapt this curation to encoder pretraining for medicine, a domain where text is dense with specialized terminology and factual correctness is critical. The recipe combines two complementary levers: medical-term density filtering selects documents rich in medical terms, and signal-amplifying rephrasing uses an LLM to rewrite documents into denser variants with broader entity contexts.

On French medical NLP, the medical-term density filter outperforms the widely used educational-quality filter, and the two are complementary. LLM rephrasing alone improves over raw web data, and mixing rephrased with filtered web data yields the largest gain. The recipe produces FineMed, a French medical pretraining corpus, and FineMed-rephrased, an LLM-rephrased subset of it, together with a state-of-the-art French medical encoder family, DoctoBERT and DoctoModernBERT, evaluated on the public DrBenchmark and a proprietary clinical Named Entity Recognition (NER) task. We release these, the three distilled annotators, and the reproducible curation pipeline on the Hugging Face Hub and GitHub.

The Web Data Recipe

The web offers the scale and the source and style heterogeneity that hand-curated medical corpora lack, but it takes curation to turn that raw text into medical pretraining data. Common web data curation includes quality filtering (e.g., heuristic filters, binary quality classifiers, or educational-value scorers) and LLM rephrasing into variants that preserve meaning while changing presentation (e.g., Wikipedia-style entries, tutorials, FAQs). Neither targets richness in domain-specific terminology, which matters for downstream tasks in specialized domains such as medicine. We close this gap with the three-stage pipeline below.

Step 1: Sources and prefiltering

We draw from three web corpora: FineWeb-2, FinePDFs, and FineWiki. They have already passed standard web cleaning (e.g., language identification, quality filtering, deduplication), which we inherit as a quality baseline. However, medical content forms only a small fraction of each source and is further diluted by commercial pages, so we extract it with a pretrained multilingual domain classifier, retaining under 10% of each source.

Step 2: Multi-axis annotation

Prefiltering retains every document classified as medical, but these documents differ in their value for pretraining. We therefore score each along three complementary axes. To make annotation affordable at corpus scale, we finetune a dedicated lightweight annotator per axis via knowledge distillation from LLM teachers (following Organize-the-Web).

• Subdomain: a 15-class label for the type of medical content, distinguishing biomedical and clinical writing (papers, guidelines) from consumer-facing material (wellness blogs, commercial health pages).
• Educational quality: a 0–5 score for how instructive a document is, adapted from FineWeb-Edu's general-education rubric to a medical-education target.
• Medical-term density: how rich a document is in medical terminology, defined as the fraction of characters that fall within medical-term spans.

We release all three annotators: a subdomain classifier, an educational-quality scorer, and a medical-entity extractor. Their per-document scores become the filtering signals we study in the experiments below, where the axes, thresholds, and combinations are chosen empirically.

Step 3: Signal-amplifying rephrasing

Filtering selects whole documents but cannot enrich the medical content within them, and it discards borderline documents that still hold useful content. We therefore add a rephrasing step: an LLM rewrites each document into a faithful variant that raises medical-term density and broadens the contexts in which each medical concept appears. This is especially valuable for encoders. Because encoder MLM trains for multiple epochs over the same corpus, unlike the typical single-epoch decoder LLM, per-document gains compound across passes. Rephrasing also removes source artifacts that filtering leaves untouched, such as boilerplate and OCR errors.

We build on Massive Genre–Audience (MGA) reformulation, a two-stage recipe. Stage 1 proposes (genre, audience) pairs for a document; stage 2 rewrites it under an assigned pair (for example, recasting a clinical guideline into an FAQ written for patients). We introduce the following adaptations:

• Medical-content gating: a check at the start of stage 1 drops documents with too little medical content, which saves compute and prevents the model from fabricating medical facts.
• Diverse pair proposals: rather than letting stage 1 default to a few repeated pairs (a real risk in a narrow domain), we generate multiple candidate genres and audiences, and randomly sample one pair per document.
• Faithful densification: stage 2 removes non-medical filler while preserving context, and remains strictly meaning-preserving, inventing no facts, values, or entities (a known failure mode of unconstrained rephrasing). The one exception is personal identifiers, which are replaced with fictional values to support downstream de-identification.
• Medical surface variation: stage 2 further varies register (formal or telegraphic clinical-note style) and abbreviation density (expanded, moderate, heavy), exposing the encoder to the stylistic range of real clinical text it will face downstream.

Findings

Filtering finds the signal in medical-term density

Heterogeneous web data is already competitive with hand-curated corpora. Even unfiltered, our extracted medical web data matches one of these corpora (NACHOS) on downstream medical tasks, and beats text machine-translated from English PubMed (TransCorpus-bio-fr). Heterogeneous sourcing gives web data broader register and format coverage than a narrowly sourced corpus, and it inherits a quality floor from upstream preprocessing.

Medical-term density is the strongest single-axis filter. Used on its own, medical-term density outperforms the other two axes, with subdomain ahead of educational quality. This contrasts with decoder-LLM pretraining, where educational quality, by rewarding coherent, well-structured, textbook-like prose, is commonly used to upsample such data and boost performance. For encoder MLM in a dense-terminology domain, density matters more: trained on a corpus packed with domain-specific terms, the encoder sees more of those terms and their co-occurrences across more varied contexts, and learns stronger representations.

Educational quality and medical-term density are complementary. The strongest filter overall keeps only documents that pass both, beating each axis on its own and every curated-corpus baseline by a clear margin.

Rephrasing amplifies the signal further

Signal-amplifying rephrasing beats both raw data and standard MGA. Data rephrased with our recipe improves over raw web data by a wide margin, while standard MGA, run with the same LLM, falls below raw web data, likely because it ingests thin inputs and fabricates content.

Rephraser quality does not correlate with model size or medical tuning. A smaller model (Qwen3.5-35B-A3B) outperformed a larger one from the same family (Qwen3.5-122B-A10B). FinePhrase reports the same broad pattern: larger rephrasers do not reliably yield better data. A medically tuned model (MedGemma-27B) scored well below a generic model of similar size (Gemma-4-26B-A4B). Mixing different LLMs, intended to add diversity and reduce single-model artifacts, gave no downstream benefit.

Rephrasing and filtering combine best of all. The strongest configuration in our study mixes rephrased text with raw web data that passes our strongest filter, beating either source alone.

FineMed and the DoctoBERT family

We then apply the validated recipe at full scale across all three web sources, yielding FineMed (21.1M documents, 19.2B words). We release FineMed unfiltered with all annotations, to enable custom downstream filtering. Signal-amplifying rephrasing then produces an LLM-rewritten subset of FineMed, FineMed-rephrased (13.6M documents, 4.5B words).

We also release two encoders pretrained from scratch on that data: DoctoBERT, a classic RoBERTa architecture (111M parameters, 512-token context), and DoctoModernBERT, a more efficient, long-context ModernBERT (149M parameters, up to 8,192 tokens).

Evaluation

We compare against nine encoders in three groups: French medical, French generalist (to test whether medical pretraining adds value over French pretraining alone), and English medical (to test cross-lingual transfer).

We evaluate on an adapted version of the public benchmark DrBenchmark, filtered to 7 high-quality tasks: QUAERO (NER on EMEA drug leaflets and MEDLINE abstracts), E3C (clinical and temporal NER), DEFT2021 (clinical NER), MORFITT (biomedical specialty classification), and DIAMED (clinical diagnostic classification). Per model and task, hyperparameters are tuned on the validation split, then the mean F1 over 5 test-split seeds is reported. We also report two per-model scores aggregated across all tasks. Min-Max rescales each task's scores onto a 0–100 scale before averaging. It captures the magnitude of the gaps: a model that lags on a few tasks is penalized heavily. WP (Win Probability) is the average percentage of tasks on which a model outscores each other model. It captures rank consistency: robust to outliers, blind to effect size.

On this benchmark, DoctoBERT ranks first on both aggregate scores, ahead of the strongest baseline (TransBERT-bio-fr), and leads five of the seven tasks.

We also evaluate on a proprietary French clinical NER task from a real-world production setting, annotated with 12 entity types (e.g., pathology, drug, exam, biometry) and 9 qualifiers (e.g., negation, family relationship, date). Scores are the mean over 3 test-split seeds.

DoctoModernBERT performs best on this task. This result supports the heterogeneous-web design: the broader register coverage seen during pretraining transfers to real clinical text more effectively than the narrowly curated baselines.

Future Work

We first developed and validated the recipe on French medical NLP, a lower-resource setting than English, across both academic and real-world tasks. The pipeline is language-agnostic, so we are now curating multilingual FineMed and training multilingual DoctoBERT. We will release both.

Acknowledgments

We thank Yanis Labrak and Adrien Bazoge for helpful discussions in the early stages of this work, Adrien Giraud for valuable discussions, and the MFE team at Doctolib (Issa Ka, Foucauld Estignard, Cyril Laitang, and Nicolas Perquier) for enabling evaluation on a real-world clinical task.

This work was granted access to the HPC resources of IDRIS under allocations 2025-AD011016291 and 2026-A0200617487 made by GENCI.