CHARCOT AI · REFERENCE MODEL

CRS-4B

A clinical tool-use model, trained in a synthetic EHR.

Built on Qwen3.5-4B. Fine-tuned only on public synthetic Gym episodes, evaluated on the private split.

Charcot AI · Gym on GitHub · Public tasks

---

From base to fine-tuned

Model	Mean task score
Qwen3.5-4B · base	0.16
CRS-4B · fine-tuned	0.86
Frontier models on the same tasks	0.76–0.89

99 private tasks · 3 attempts per task · November 2026

The private tasks were never seen in training. These scores describe performance on the evaluated tasks; they do not establish a general ranking outside this evaluation.

What the model does

Charcot Gym places agents inside an EHR environment. Agents read clinical context, call tools and write structured records. The benchmark evaluates the resulting state across Document → FHIR, Clinical coding and Claims.

A deterministic offline verifier checks the final record and task constraints. No LLM judge is used.

Training and evaluation

• Training: public synthetic Gym episodes only.
• Evaluation: a held-out private split with repeated attempts per task.
• Integrity: checks include patient identity and preservation of existing resources.
• Private reporting: aggregate scores only. Private task contents, per-task outputs and trajectories are not returned to submitters.

See the evaluation methodology for the scoring and submission policy.

Use and scope

CRS-4B is a reference model for research and evaluation of clinical tool use. Its reported result measures synthetic task completion, not performance in patient care.

---

Charcot AI — the longitudinal data and evaluation infrastructure for healthcare AI agents.

solano@charcot.ai