charcot-ai/charcot-27b

Charcot 27B

Qwen3.8-27B fine-tuned on synthetic Charcot Gym episodes for healthcare admin workflows: claims, clinical coding, and document → FHIR R5.

On private tasks it has never seen, the score rises from 0.505 to 0.727.

Results

Mean score from 0 to 1 (3 attempts per task).

Evaluation	Tasks	Qwen3.8-27B	Charcot 27B
Private split	162	0.51	0.73
Public held-out	144	0.37	0.54

Private split by family:

Family	Qwen3.8-27B	Charcot 27B
Claims	0.37	0.71 ↑
Clinical coding	0.42	0.69 ↑
Document → FHIR	0.73	0.78 ≈

↑ gain, ↓ drop, ≈ no significant difference (95% confidence interval of the difference).

Quick start

vllm serve charcot-ai/charcot-27b --dtype bfloat16 --max-model-len 131072

export OPENAI_API_KEY=EMPTY
charcot-gym eval \
  --model charcot-ai/charcot-27b \
  --base-url http://localhost:8000/v1 \
  --api-key-env OPENAI_API_KEY \
  --split public --text-only --attempts 3

Training

• Data: charcot-ai/gym-sft, 801 synthetic text episodes: 153 public-split tasks and 648 generated with the public generators (8 published seeds).
• Method: LoRA SFT (rank 32) on the response only, merged into FP32 weights.
• No private task or seed was used in training.

Limitations

• Fine-tuning lowers scores on Hard clinical coding: Δ −0.149 [−0.25, −0.05] on the public held-out and −0.174 [−0.34, −0.01] on the private split.
• Text only: the vision tower is unchanged from the base model.
• Trained and evaluated on synthetic data only.
• Private tasks are not published; only aggregate scores are.

Evaluation details

• Held-out tasks use fresh seeds and share no seed, content, or initial state with training or private tasks.
• Private split: base + 2026Q4 roots, task-files SHA-256 2aa24013…3d0fa and 5ef02e7e…480551.
• Intervals: on the public held-out, a paired 95% bootstrap over tasks. On the private split, a 95% bootstrap over tasks within each cell, combined across cells with task-count weights; it is unpaired, so it is conservative. Errored attempts were retried up to twice and excluded if they still failed (base: 6 of 486 private attempts).

License

Apache-2.0. The base model remains subject to its own license.