KOS-V5-LLM
Collection
3 items • Updated
KOS-V5-Base · "Catbird"
Developed by
University of Kentucky
University of Louisville
A 3.72B-parameter medical language model trained from scratch. It is not distilled, not pruned and not continued-pretrained from a general base. KOS-V5 (codename Catbird) is the fifth-generation Kentucky Open Science base model. This repository holds the final pretraining checkpoint: the complete single-epoch run over 235.2B tokens.
This is a base model. It has had no instruction tuning, no RLHF, and no chat post-training. It completes text; it does not follow instructions. It is the initialisation for a downstream SFT/RL line, and it is released as a research artifact.
Code name: Catbird. The KOS-V5 series is nicknamed Catbird, a native bird to Kentucky, the Gray Catbird (Dumetella carolinensis) is a medium-sized songbird famous for its distinct, cat-like "meow" call. This LLM was trained completely from scratch by teams from the University of Kentucky (Cat) and University of Louisville (Bird), so the code name is fitting.
⚠️ Research use only. This model is provided for research purposes only and must not be used for any commercial, clinical, legal, or production-grade applications. The user assumes all risks associated with its use.
At a glance
| Parameters | 3.72B (36 layers × 2560 hidden, 32/8 GQA, head_dim 128, SwiGLU, tied embeddings) |
| Training tokens | 235.2B (one complete epoch over a 54-source medical/biomedical corpus) |
| Final train loss | ≈1.16 |
| Context | trained at 24,576 tokens (whole-document); max_position_embeddings 32,768 |
| RoPE | standard 1D, θ = 25,000 |
| Tokenizer | custom 32k byte-level BPE ("v5-32k"), add_bos_token=False, single special token `< |
| Precision / stack | bf16, 24×H200, DeepSpeed ZeRO-1, FlashAttention-2 |
| Architecture class | Qwen3ForCausalLM (stock, exports with no custom modelling code) |
Evaluation
96 metrics across 19 tests; 94 ranked. Scoring convention used throughout:
A metric is a WIN if KOS-V5 places in the top quartile of the pool (rank 1–4 of 17), and a LOSS if it places in the bottom quartile (rank 14–17 of 17). Ranks 5–13 are mid-pool, neither.
Result: 28 wins, 16 losses, 50 mid-pool.
📖 Every test is defined in Test definitions below: what it measures, on what data, and which direction is better. The tier codes (T1, T1b, T2 …) used in the tables are all explained there.
Every comparator was trained on 1.3–153× more data than KOS-V5 (0.3–36T tokens vs our 0.235T).
The comparator pool (16 external models + KOS-V5 = 17)
Biomedical specialists (5)
| model | tokens |
|---|---|
| BioMedLM | 300B (PubMed) |
| MedGemma-4B-PT | Gemma-3-4B 4T + medical image-text |
| Meditron-7B | Llama-2-7B 2T + 48B medical |
| AdaptLLM-Medicine-LLM | LLaMA-1-7B 1T + 2B adaptation |
| PMC-LLaMA-7B | LLaMA-7B 1T + 79B biomedical |
General-domain (11)
| model | tokens |
|---|---|
| Llama-3.2-1B / Llama-3.2-3B / Llama-3.1-8B | 9T / 9T / 15T |
| Gemma-3-1B-PT / Gemma-3-4B-PT | 2T / 4T |
| Gemma-2-9B | 8T |
| Qwen2.5-7B | 18T |
| Qwen3-4B-Base | 36T |
| StableLM-2-1.6B | 2T |
| MiMo-7B-Base | 25T |
| GigaChat3-10B-A1.8B-base | 5.5T (synthetic-heavy) |
(KOS-V4, our own previous base, is held out of the ranking pool and reported separately below.)
Wins: top quartile, ranks 1-4 (28 metrics)
Tier codes (T1, T2 …) are defined in Test definitions.
| Metric | Value | Rank |
|---|---|---|
| BPB, 5-corpus mean (held-out medical text) | 0.4635 | 1 / 17 |
| BPB radiology | 0.5132 | 1 / 17 |
| BPB chest-xray | 0.6688 | 1 / 17 |
| BPB clinical narratives | 0.4179 | 1 / 17 |
| Collapsed-head fraction (T2) | 0.0069 | 1 / 17 |
| BOS-sink mass (T2) | 0.2541 | 1 / 17 |
| Min attention entropy (T2) | 0.4705 | 1 / 17 |
| Layers carrying a sink (T2) | 0.3330 | 2 / 17 |
| Dead-neuron rate (T3) | 0.0000 | 1 / 17 (tied) |
| Peak / final logit-lens accuracy (T3) | 0.5680 | 2 |
| RankMe, mean (T4) | 220.0 | 1 / 17 |
| WeightWatcher safe-layer ratio / under-ratio (T5) | 0.8346 / 0.1654 | 3 / 17 |
| BIOSSES Pearson / Spearman (T7) | 0.7097 / 0.7014 | 1 / 17 |
| BLURB probe mean (T7) | 0.7268 | 2 / 17 |
| MedThink-LL rank-1 (T6b) | 0.9200 | 2 / 17 |
| MedHALT fake-question rejection (T8-FQT) | 0.1122 | 2 / 17 |
| Long-context BPB @ 1,024 / 2,048 tok | 0.7290 / 0.6466 | 1 / 16 |
| Long-context BPB @ 512 / 4,096 tok | 0.8609 / 0.5809 | 3 / 4 of 16 |
| Tokens-per-byte efficiency (T1b) | 0.2580 | 4 / 17 |
Bits-per-byte is tokenizer-agnostic, so it is a fair cross-model number, and the distillation confound in the pool flatters the trillion-token externals, not a from-scratch model. These BPB placements are therefore conservative, not inflated.
Losses: bottom quartile, ranks 14-17 (16 metrics)
Tier codes are defined in Test definitions.
| Metric | Value | Rank |
|---|---|---|
| MedHALT false-confidence test (T8-FCT) | 0.0280 | 16 / 17 (worst placement) |
| Single-token rate, aggregate (T1b-STRR) | 0.0481 | 15 / 17 |
| STRR, clinical / biomedical | 0.1257 / 0.0525 | 15 / 17 |
| STRR, radiology / chest-xray / textbooks / macro | 0.1921 / 0.2274 / 0.1693 / 0.1534 | 14 / 17 |
| Fragment score (T1b) | 2.5855 | 14 / 17 |
| RadGraph2-CXR micro-F1 (DR.3) | 0.9115 | 15 / 17 |
| HoC macro-F1 (T7) | 0.4889 | 14 / 17 |
| HellaSwag / PIQA (general reasoning) | 0.4638 / 0.7367 | 14 / 17 |
| Needle @ 512 / 1,024 tok (3 cells) | 0.90–0.95 | 14 / 16 |
The losses are concentrated, not diffuse: 7 of 16 are the single-token-rate/fragmentation family (a deliberate tokenizer trade-off, see Known issues), 3 are short-context needle cells, and the rest are MedHALT-FCT, two general-domain reasoning sets, one HoC probe and one RadGraph micro-F1.
Mid-pool: neither a win nor a loss (50 metrics)
Tier codes are defined in Test definitions.
Reported because they are frequently mistaken for one or the other:
| Metric | Value | Rank |
|---|---|---|
| Calibration, mean ECE | 0.1209 | 9 / 17 |
| Long-context needle, 3-depth mean | 0.9333 | 11 / 16 |
| RadGraph-XL / CXR / RadGraph2 macro-F1 | 0.7463 / 0.7599 / 0.7661 | 7 / 8 / 10 of 17 |
| MedQA / MedMCQA / PubMedQA | 0.3339 / 0.3373 / 0.6680 | 12 / 13 / 12 of 17 |
| BPB biomedical-lit / textbooks | 0.1108 / 0.6066 | 13 / 11 of 17 |
Closed-book medical MCQ (mean 0.4352 over 16 tasks) sits mid-pool: below the frontier fleet, well above KOS-V4. It is a post-training problem, not a token-budget one.
Test definitions
What each tier actually measures. "dir" is the direction that is better. Every metric is ranked against the same 17-model pool unless marked reference-only.
Block A: general LM diagnostics (58 metrics)
T1 · Bits-per-byte on held-out medical text (the headline; dir: lower)
Negative log-likelihood divided by UTF-8 bytes, not tokens, so it is tokenizer-agnostic and
directly comparable across models with different vocabularies. Measured on five held-out medical
corpora, none of which appear in training: radiology_reports, chest_xray_reports,
clinical_narratives, biomedical_literature, medical_textbooks, plus their mean.
Caveat we state against ourselves: a model distilled from a larger teacher can print lower BPB
regardless of true quality, which flatters the trillion-token externals, not a from-scratch model ,
so our BPB placements are conservative.
T1b · Tokenizer statistics (dir: STRR higher; fragment & tokens-per-byte lower)
- STRR (Single-Token Rate Ratio): the fraction of medical terms the tokenizer encodes as one token, over a union vocabulary of 21,978 medical terms across the 5 domains. Reported per domain, plus a macro mean and a union-weighted aggregate.
- fragment_score: average number of subword pieces per medical term.
- tokens_per_byte: raw compression efficiency (how compactly the tokenizer encodes text).
T2 · Attention pathology (dir: lower for sinks/collapse; higher for min-entropy)
Per-(layer, head) attention diagnostics over medical text:
- collapsed_head_frac: fraction of heads whose attention has degenerated to a near-constant pattern.
- bos_sink_mass: attention mass dumped on the first token (the classic "attention sink").
- layers_with_sink: fraction of layers exhibiting a sink.
- entropy_norm_min: the lowest normalised attention entropy in the model (higher = no head has collapsed to a single position).
T3 · Representation flow (dir: lens accuracy higher; dead-neuron lower)
- peak_lens / final_lens accuracy. Logit-lens probe accuracy, i.e. how well the correct token can be read out of intermediate layers (peak across depth, and at the final layer).
- lens_AUC: area under the depth-vs-accuracy curve.
- max_dead_neuron_frac: the worst layer's fraction of neurons that never activate.
T4 · Representation geometry (dir: RankMe higher; isotropy lower = less anisotropic)
Measured at 5 proportional depths:
- RankMe: the effective rank (soft rank of the singular-value spectrum) of the hidden representations. Higher = the model uses more of its representational space rather than collapsing into a low-dimensional cone.
- isotropy: directional uniformity of the embedding cloud.
T5 · WeightWatcher power-law α (structural maturity)
Fits a power law to each layer's weight spectrum. α ∈ [2, 6] is the healthy band; α > 6 means under-trained, α < 2 over-trained.
- safe_layer_ratio (dir: higher): fraction of layers inside the healthy band.
- under_ratio (dir: lower): fraction of layers that are under-trained.
- median_α / min_α. Reference-only, deliberately unranked. α is a target band, not a monotone quantity, so "more is better" is meaningless. (Ranking it higher-is-better would literally reward being under-trained, and would have improved our reported result from rank 15 to rank 3. We declined to take the favourable reading of an ill-defined metric.)
T9 · Calibration (dir: lower)
Expected Calibration Error (ECE), averaged over 15 multiple-choice calibration tasks: the gap between the model's confidence and its actual accuracy. Low ECE = the model's confidence means something.
T10 · Context-length scaling + needle-in-a-haystack
- Bucket BPB (dir: lower): BPB measured at {L/8, L/4, L/2, L} where L = min(model's max position, 4096), so every model is judged inside its own usable window. A separate KOS-only extension probes the full 32,768-token window.
- Needle retrieval (dir: higher): a numeric medical fact is hidden at depth 0.1 / 0.5 / 0.9 inside a long context and must be retrieved, at fixed lengths 512 / 1,024 / 2,048 / 4,096 tokens. Reported per length, per depth, and as a 3-depth mean.
T11 · Demographic bias (dir: disparity lower; frac_stereotyped closer to 0.5)
CrowS-Pairs: for each minimally-different sentence pair (stereotyping vs anti-stereotyping), which
does the model prefer? |disparity| measures how strongly it prefers one; frac_stereotyped is the
fraction of pairs where the stereotyping sentence wins (0.5 = no preference).
Block B: medical knowledge and entity extraction (32 metrics)
T6 · Closed-book multiple choice (dir: higher)
16 log-likelihood MCQ tasks, scored with the standard lm-evaluation-harness (no chain-of-thought, no prompting tricks, this is a base model):
- Medical specialist: MedQA (4-option, USMLE-style), MedMCQA, PubMedQA, and the 8 MMLU medical subjects (anatomy, clinical knowledge, college biology, college medicine, high-school biology, medical genetics, nutrition, professional medicine, virology).
- Augmented / regional / expert: MedXpertQA-Text, AfriMed-QA.
- General-reasoning controls: PIQA, HellaSwag.
T6b · MedThink-LL (dir: rank-1 higher; margin lower)
A PubMedQA-substitute: the correct answer must be ranked above 4 distractors by log-likelihood. rank1_acc = how often it is; mean_logprob_margin = the confidence gap.
T7 · BLURB linear probes (dir: higher)
A linear probe is trained on the frozen model's representations, this reads the semantic geometry of the upper layers, independent of any generation ability:
- BC5CDR-Chem: chemical NER (macro / micro F1).
- NCBI-Disease: disease NER (macro / micro F1).
- BIOSSES: biomedical sentence-similarity (Pearson / Spearman).
- HoC: Hallmarks-of-Cancer document classification (micro / macro / samples F1).
- BLURB mean. The headline figure: the mean of BC5CDR-macro, NCBI-macro, BIOSSES-Spearman and HoC-micro.
T8 · Med-HALT hallucination resistance (dir: higher)
- FCT (false-confidence test): the question's premise is wrong; the model should decline. A base model has no refusal prior, which is exactly why this is our worst placement.
- NOTA (none-of-the-above): the correct answer is "none of the above".
- FQT (fake-question test): the question is fabricated; it should be rejected.
- PMID2Title: recall a paper's title from its PubMed ID (a memorisation probe; degenerate , every model in the pool scores 0.0, so it is excluded from the tally).
Block C: radiology-specific (6 metrics)
DR.1 / DR.2 / DR.3 · RadGraph entity + relation extraction (dir: higher)
Structured extraction of clinical entities and their relations from radiology reports, scored as macro-F1 and micro-F1:
- DR.1, RadGraph-XL: chest CT + abdomen-pelvis CT + brain MRI.
- DR.2, RadGraph-CXR: chest x-ray (50 MIMIC + 50 CheXpert).
- DR.3, RadGraph2-CXR: adds the CHAN (change) relation class.
DPM.1 · PMR-Q&A (reference-only)
English physiotherapy / rehabilitation Q&A, rank-1 accuracy. Unranked: too few comparators have a score for it to be a meaningful ranking.
How ranks are computed
- Pool = KOS-V5 + the 16 external models = 17. KOS-V4 is excluded from the pool, it is our own prior model, and is reported separately in its own head-to-head below.
- Each metric is ranked in its own correct direction (lower-is-better for BPB, ECE, sinks, fragmentation; higher-is-better for accuracy, F1, RankMe, STRR).
- Denominators vary where a comparator has no score for a metric: Gemma-2-9B has no T10 result and BioMedLM is architecturally capped at 1,024 tokens (so it is skipped in the longer needle tables). That is why some ranks read "of 16" or "of 12" rather than "of 17", the rank is always taken over whichever models actually have that metric.
- Two metrics are reference-only (the WeightWatcher α pair) and one is degenerate (PMID2Title, where all 17 models score 0.0), leaving 93 metrics in the tally.
KOS-V5 vs KOS-V4: a corpus trade-off, now settled
Both models are ours, both are finished, and both were run on the identical harness. KOS-V5 is better on 17 of 32 compared metrics, KOS-V4 on 15.
The token-budget alibi is dead: KOS-V5 finished on 235.2B tokens vs KOS-V4's 180.3B (1.30×) at a larger parameter count. Every remaining KOS-V4 win is a win on the merits.
| KOS-V5 (final, 235B) | KOS-V4 (final, 180B) | Better | |
|---|---|---|---|
| Medical MCQ mean (16 tasks) | 0.4352 | 0.3319 | V5 (+0.103; 12 of 16 tasks) |
| MMLU clinical knowledge | 0.5057 | 0.3094 | V5 (+0.196) |
| MMLU college medicine | 0.4335 | 0.2312 | V5 (+0.202) |
| MMLU professional medicine | 0.3824 | 0.2096 | V5 (+0.173) |
| BPB 5-corpus mean | 0.4635 | 0.4309 | V4 |
| BPB radiology / chest-xray / clinical | 0.5132 / 0.6688 / 0.4179 | 0.4761 / 0.5887 / 0.3221 | V4 |
| BPB biomedical-lit / textbooks | 0.1108 / 0.6066 | 0.1243 / 0.6432 | V5 |
| RankMe (mean) | 220.0 | 170.8 | V5 |
| Long-context BPB, 4k → 32k | wins every window | , | V5 |
| Needle, 3-depth mean | 0.9333 | 0.8667 | V5 |
| BLURB mean / RadGraph macro-F1 | 0.7268 / 0.75–0.77 | 0.7465 / 0.77–0.82 | V4 |
| PubMedQA / MedThink rank-1 | 0.6680 / 0.9200 | 0.6980 / 0.9450 | V4 |
The honest reading. KOS-V4's narrow radiology/clinical-dominant corpus still produces the better text model of radiology and clinical notes. KOS-V5's broader 54-source corpus produces the better medical reasoner: on the recall-heavy MCQ tasks V4 sat close to chance and V5 does not. This is a corpus-composition difference, not a maturity gradient, and further pretraining was never going to close it: V5 would have needed to recover 0.0326 BPB, and its entire final leg moved it by 0.0001. Which model is "better" depends on the deployment.
The run converged before it finished. Read this before choosing a checkpoint
The identical 16-tier battery was run at six checkpoints (90B / 120B / 150B / 190B / 220B / 235B).
| axis | 90B | 120B | 150B | 190B | 220B | 235B (final) |
|---|---|---|---|---|---|---|
| BPB (5-corpus, lower better) | .4890 | .4781 | .4740 | .4647 | .4636 | .4635 |
| MCQ (16 tasks, higher better) | .3478 | .3737 | .4071 | .4449 | .4332 | .4352 |
| Needle (higher better) | .9333 | .9000 | .9500 | .9167 | .9000 | .9333 |
| ECE (lower better) | .1847 | .1135 | .1181 | .1378 | .1183 | .1209 |
On the final leg, not one headline axis improved. BPB moved +0.0001 (zero to three decimals). MCQ peaked at 190B and never recovered: the final model ends 0.0097 below its own peak, with 12 of 16 tasks below their peak-checkpoint value. Needle ended exactly where it began at 90B. Calibration ended worse than the checkpoint before it. The learning-rate integral is 100% spent.
⚠️ The newest checkpoint is NOT the knowledge-best checkpoint. The 190B checkpoint is KOS-V5's knowledge-best, permanently. Choose your fine-tuning initialisation by axis: the 190B checkpoint for medical knowledge/MCQ; this final checkpoint for BPB, long-context and general LM health. Do not assume "newest = best."
Both MCQ moves (the 220B drop and the 235B bounce) sit inside MMLU-subset sampling noise. What survives noise is the plateau: knowledge accuracy stopped improving 45B tokens before the end.
Known issues
1. The tokenizer does not split numbers, and this cannot be fixed in this model.
The v5-32k BBPE has no digit rule: its pre-tokenizer is plain byte-level, so digit runs merge into single tokens by BPE frequency.
| tokenizer | digit rule | multi-digit tokens | max digit-run | 10 / 100 / 1000 |
tokens/number |
|---|---|---|---|---|---|
| KOS-V5 | none | 963 | 5 | one token each | 1.80 |
| Qwen3-4B-Base | yes, every digit split | 0 | 1 | 1·0 / 1·0·0 / … |
4.20 |
| Llama-3.2-3B | yes, ≤3-digit groups | 1100 | 3 | 10 / 100 / 100+0 |
2.90 |
The consequence is compositional: 10, 100 and 1000 are three unrelated atomic symbols that
share no substructure. The model cannot see that they differ by a factor of ten; it must learn each
magnitude as a separate lexical item, and rare numbers fragment on arbitrary BPE boundaries while
common ones do not. In medicine, doses, lab values, vitals, this is exactly the regime where
digit-level arithmetic and magnitude comparison are known to degrade.
Stated with its limit: this is a measured property of the frozen tokenizer, not a measured downstream failure. Our evaluation suite contains no arithmetic or numeric-reasoning tier, so we neither quantify the cost nor claim there is none. The tokenizer is baked into the trained (tied) embedding table, so it is not fixable by fine-tuning, removing it requires retraining with a digit-splitting pre-tokenizer. Treat any numeric medical task on this model with that in mind.
2. Low medical single-token rate (STRR: bottom quartile, rank 15/17). Deliberate: BBPE spends its 32k vocabulary on byte-level compression rather than memorising whole medical terms. It single-tokens only ~19% of radiology and ~23% of chest-xray terms. The trade appears sound, tokens-per-byte is a top-quartile 4/17 and BPB is 1/17, but it is a trade, and it is the same trade as the digit issue.
3. MedHALT-FCT (false-confidence test) is the worst placement in the model (rank 16/17). A base model with no refusal prior has essentially no mechanism for declining a question whose premise is false, and KOS-V5 answers anyway. This is the axis a grounded medical model most needs to improve, and it is a target for the instruction/RL stage rather than for more pretraining.
4. Long-context needle retrieval never improved across the entire run (mid-pool, 11/16). More pretraining will not fix it; it is a post-training problem.
5. Attention health is leading but degrading. KOS-V5 still leads the pool on T2, but bos-sink mass rose 0.1619 → 0.2541 and layers-carrying-a-sink rose 0.0830 → 0.3330 across training. The lead is real but it shrank; it did not consolidate.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Kentucky-Open-Science/KOS-V5-Base"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
prompt = "IMPRESSION: The chest radiograph demonstrates"
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
Notes
- Works with
transformers4.x and 5.x out of the box. Verified on 4.57.1 and 5.2.0: the snippet above runs unchanged on both. - Base model, no chat template. It completes text. Prompt it as a completion model, or post-train it.
- One special token.
<|endoftext|>(id 0) serves eos/bos/pad and the document separator.
Intended use
A research base model for medical/biomedical NLP: the initialisation for domain SFT/RL, a subject for interpretability and tokenizer/corpus research, and a from-scratch reference point against distilled and continued-pretrained medical models. It is not instruction-tuned and is not a question-answering system. Its closed-book medical MCQ accuracy sits mid-pool, and the known issues above, especially the number tokenization, bear directly on any numeric medical task.
Training details
| tokens / step | 1 × 8 × 24 × 24,576 = 4,718,592 |
| optimizer | AdamW β=(0.9, 0.95), grad-clip 1.0 |
| LR schedule | cosine → 0, peak 3.0e-4, 1% warmup (integral 100% spent) |
| batch | micro_batch 1 × grad_accum 8 × 24 GPUs, seq 24,576 (≈4.72M tokens/step) |
| corpus | 54 sources, English-only, deduped → decontaminated; whole documents (any document over 24,576 tokens is dropped, never split); neat-packed with a 4-D block-diagonal segment mask (no cross-document attention; position ids reset per document) |
tokenizer.json md5 |
9c9df0404f6aae96dba5f3785e8b4c9d |
Full 22-page evaluation report, all 96 metrics, all 17 models, every rank re-derived from source:
kos_v5_brief.pdf, included in this repository.
Attribution
University of Kentucky
University of Louisville
Citation
@misc{kos_v5_base_2026,
title = {KOS-V5-Base (Catbird): a from-scratch 3.72B medical language model},
author = {Kentucky Open Science},
year = {2026},
note = {University of Kentucky (College of Medicine Office for Research;
Center for Clinical and Translational Science) and
University of Louisville (Kentucky Center for Digital Innovation)},
url = {https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base}
}
- Downloads last month
- -