Kentucky-Open-Science/KOS-V4-Instruct-GGUF

Code name: Scratch. The KOS-V4 series is nicknamed Scratch LLM: it was trained completely from scratch by a small team on a fraction of the data and compute of commercial models. It is not a frontier model.

⚠️ Research use only. This model is provided for research purposes only and must not be used for any commercial, clinical, legal, or production-grade applications. The user assumes all risks associated with its use.

---

KOS-V4-Instruct — a from-scratch 3B that reaches original-ChatGPT-level instruction-following

KOS-V4-Instruct is an open-weights 3B language model trained completely from scratch by a University of Kentucky College of Medicine team (Office for Research, Center for Clinical and Translational Sciences). It is a decoder-only transformer (Qwen3 architecture, bespoke 3B config) optimized for instruction following and tool / function calling. Its instruction ability comes from GRPO reinforcement learning on a from-scratch clinical base.

IFEval reported as strict-avg = (prompt-level strict + instruction-level strict) / 2 — the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval."

IFEval strict-avg	model	who built it, and how
64.7	Qwen2.5-3B-Instruct	Alibaba, ~18 trillion tokens
61.6	KOS-V4-Instruct (ours)	University research team, 180B tokens, 24 GPUs
55.9	GPT-3.5-turbo-1106 (the original ChatGPT)	OpenAI, ~10,000-GPU supercomputer

KOS-V4-Instruct clears the original GPT-3.5-turbo generation (55.9) and lands within ~3 points of the commercially trained Qwen2.5-3B (64.7). It also adds real tool / function calling (official BFCL 72.75/73/60.5), which the original ChatGPT lacked at launch — though modern small models score higher there.

Core specifications

Attribute	Detail
Architecture	Decoder-only Transformer (Qwen3ForCausalLM), Grouped-Query Attention
Parameters	3.015 B
Hidden / Layers	3072 / 28
Attention	24 query / 8 KV heads (GQA 3:1), head_dim 128, per-head QK-RMSNorm
Feed-forward	SwiGLU, intermediate 8192
Vocabulary	32,000, custom medical byte-level BPE
Context length	24,576 (max_position_embeddings 65,536)
Position encoding	RoPE, θ = 25,000 (pin on export)
Precision	bfloat16
Chat template	ChatML (`<
Pretraining tokens	180.3 B (English medical/biomedical + web)

Quickstart (Hugging Face Transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Kentucky-Open-Science/KOS-V4-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "List three contraindications for ibuprofen. Answer in exactly 3 bullet points."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Serve with vLLM / TGI as a standard Qwen3 causal LM. Pin RoPE θ = 25000 on any GGUF/quantized export.

Prompt / chat format (ChatML)

<|im_start|>system
{system (optional)}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>

<|im_end|> is the true eos. Tool calling uses <tool_call> / <tool_response> tokens (pass tools via the chat template tools= argument).

Pre-training (the KOS-V4 base)

This model is fine-tuned from KOS-V4-Base — the from-scratch pretrained foundation summarized here.

Trained from scratch, not distilled or continued. Pure next-token cross-entropy (no auxiliary losses), AdamW, peak LR 3.0e-4 cosine, 1 epoch, seq 24,576 (whole-document neat-packing), bf16 + FlashAttention-2, 305,613 steps / 180.3 B token-positions. Data: English-only, medical/biomedical-first, 49 sources / 130 M chunks (PubMed Central 77 B, FineWeb-Edu 51 B, mMedC-en 6.3 B, BlueScrubs 4.6 B, + ~40 smaller clinical/ontology sources).

Disclosed issues: 35 % duplicate tokens (a FineWeb-Edu build bug + PMC repetition; a deduped corpus is ready but was not trained); 38 "sink" BPE merges removed surgically (vocab stays 32,000, ids gated as bad_words_ids); RoPE θ trained at 25,000 (a planned 10,000 was never applied).

Post-training (this model)

SFT: full-parameter (not LoRA), ChatML, LR 7e-5, 1 epoch, NEFTune α=5; corpus ~0.9 M rows — a stratified medical+tool+general majority, a 23-family instruction taxonomy (verifier-in-the-loop; only responses passing the official IFEval checker kept), xLAM function-calling, and grounded cite/abstain RAG. Forgetting gate (broad ppl ≤1.5×).

RL (GRPO via verl): deterministic verifiable rewards, no reward model, no LLM judge; the primary reward runs the official IFEval checker. GRPO lifted the official IFEval strict-avg from 49.4 (SFT base) to 61.6 at step 1120 (shipped) — the argmax over all RL checkpoints (sha256-verified as the released weights).

Evaluation & benchmarks

Benchmarks are official-suite only: IFEval via the EleutherAI lm-evaluation-harness 0.4.12 (`task ifeval, apply_chat_template=True

• greedy, task-default max_gen_toks=1280), BFCL via the official bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6 is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's 64.7, a 0.7-pt gap).

IFEval in context (strict-avg, models our size or larger)

One metric for everyone: strict-avg. Open-model values are the Open LLM Leaderboard "IFEval" (which is strict-avg). Commercial rows are Proprietary; a * means the strict value is an estimate (no official strict sub-metrics published — estimated from the model's published AVG4/prompt-strict, which run a few points above strict) and a + means the parameter count is unofficial. Our 61.6 is measured on our harness (calibrated above).

model	weights	company	released	params	IFEval strict-avg
GPT-4o-mini	Proprietary	OpenAI	Jul 2024	8B +	79 *
Llama-3.2-3B-Instruct	Open	Meta	Sep 2024	3.2B	73.9
Qwen2.5-3B-Instruct	Open	Alibaba	Sep 2024	3.0B	64.7
Phi-3-medium-4k-instruct	Open	Microsoft	May 2024	14.0B	64.2
Mistral-Large	Proprietary	Mistral AI	Feb 2024	46.7B +	63 *
KOS-V4-Instruct (ours)	Open	Univ. of Kentucky	Jul 2026	3.0B	61.6
Yi-1.5-9B-Chat	Open	01.AI	May 2024	8.8B	60.5
Phi-3.5-mini-instruct	Open	Microsoft	Aug 2024	3.8B	57.7
GPT-3.5-turbo-0613	Proprietary	OpenAI	Jun 2023	20B +	57 *
Command-R	Open	Cohere	Mar 2024	35B	57 *
Phi-3-mini-4k-instruct	Open	Microsoft	Apr 2024	3.8B	56.1
GPT-3.5-turbo-1106	Proprietary	OpenAI	Nov 2023	20B +	55.9
Mistral-7B-Instruct-v0.2	Open	Mistral AI	Dec 2023	7.2B	55.0
Gemini-1.0-Pro	Proprietary	Google	Dec 2023	30B +	55 *
Mistral-Medium	Proprietary	Mistral AI	Dec 2023	100B +	55 *
Mistral-7B-Instruct-v0.3	Open	Mistral AI	May 2024	7.2B	54.7
gemma-1.1-7b-it	Open	Google	Mar 2024	8.5B	50.4
zephyr-7b-beta	Open	Hugging Face	Oct 2023	7.2B	49.5
GPT-3.5-turbo-0125	Proprietary	OpenAI	Jan 2024	20B +	49 *
Llama-3.1-8B-Instruct	Open	Meta	Jul 2024	8.0B	44.3
Qwen1.5-7B-Chat	Open	Alibaba	Jan 2024	7.7B	43.7
Llama-2-13b-chat	Open	Meta	Jul 2023	13.0B	39.8
Qwen1.5-4B-Chat	Open	Alibaba	Jan 2024	4.0B	31.6

* strict estimate — no official IFEval strict sub-metrics are published for this model; the value is estimated from its published AVG4 or prompt-strict (loose metrics run ~2–4 pts above strict). + unofficial params — never disclosed by the provider (industry estimate: GPT-3.5 ~20B, GPT-4o-mini ~8B, Mistral-Large ~46.7B, Mistral-Medium ~100B, Gemini-1.0-Pro ~30B).

Reading. At 3B, KOS-V4 (61.6) beats every measured GPT-3.5-turbo snapshot of the original ChatGPT (1106 = 55.9, 0125 ≈ 49, 0613 ≈ 57 est), plus Yi-1.5-9B, both Mistral-7Bs, gemma-1.1, zephyr, Meta's Llama-3.1-8B (44.3), Llama-2-13b and the Qwen1.5 chats. Ahead of it: Llama-3.2-3B (73.9), Qwen2.5-3B (64.7), Phi-3-medium-14B (64.2), and the frontier proprietary models. This is a strong result for a from-scratch 3B on 180B tokens and 24 GPUs, not a claim to lead the current field.

Commercial baselines (strict-avg)

commercial model	company	snapshot	strict-avg	basis
GPT-4	OpenAI	gpt-4-0613	80.6	computed from published strict sub-metrics (77.1 / 84.1), InternLM2 report
Command-R+	Cohere	c4ai-command-r-plus	76.6	strict sub-metrics (72.8 / 80.5), Open LLM Leaderboard raw
GPT-3.5-turbo	OpenAI	gpt-3.5-turbo-1106	55.9	strict sub-metrics (50.5 / 61.2), InternLM2 report
Claude-3.5-Sonnet	Anthropic	20240620	~83 *	estimate from published AVG4 86.2 (Llama 3.1 report)
GPT-4o	OpenAI	2024-05-13	~81 *	estimate from AVG4 84.3
Gemini-1.5-Pro	Google	May 2024	~79 *	estimate from AVG4 82.3
Claude-3-Haiku	Anthropic	Mar 2024	~65 *	estimate from AVG4 68.1

Sources. IFEval definition: google-research/instruction_following_eval. Open-model strict-avg: open-llm-leaderboard/contents (its "IFEval" column). Commercial strict sub-metrics: InternLM2 report (arXiv:2403.17297) and Open LLM Leaderboard raw results. ~ * rows are estimates from published AVG4 (strict runs a few points lower), clearly labeled.

BFCL (official bfcl_eval, function-calling mode; simple / multiple / parallel)

BFCL (official FC)	KOS-V4-Instruct	Qwen2.5-3B	Llama-3.2-3B
simple / multiple / parallel	72.75 / 73.00 / 60.50	95.00 / 92.00 / 74.50	91.75 / 92.50 / 88.50

Deployment (inference)

Precision	Approx. VRAM	Notes
bfloat16	7 GB	native weights (6.03 GB) + activations; a single 16 GB GPU is comfortable
GGUF Q8_0 / Q4_K_M	~4 / ~2.5 GB	shipped for llama.cpp — see GGUF quantizations below

GGUF quantizations (llama.cpp)

Ready-to-run llama.cpp builds are published at Kentucky-Open-Science/KOS-V4-Instruct-GGUF (llama.cpp b510/18ef86e; lm-eval-harness 0.4.12).

file	quant	size	notes
kosv4-f16.gguf	F16 (16.0 bpw)	6.03 GB	full-precision reference / requantize source
kosv4-Q8_0.gguf	Q8_0 (8.5 bpw)	3.21 GB	near-lossless
kosv4-Q4_K_M.gguf	Q4_K_M (~4.8 bpw)	1.83 GB	recommended — 3.3× smaller than f16, no measurable IFEval loss

SHA256 in SHA256SUMS; raw benchmark JSON + conversion/eval scripts in testing/.

Quantization preserves IFEval. Official EleutherAI lm-eval ifeval (--apply_chat_template), all 541 prompts, greedy; only the weights differ across rows. strict-avg = (prompt-strict + inst-strict)/2.

format	prompt-strict	inst-strict	strict-avg	Δ vs bf16
bf16 (HF reference)	55.82	67.03	61.42	—
Q8_0 (GGUF)	55.27	66.43	60.85	−0.57
Q4_K_M (GGUF)	56.19	67.39	61.79	+0.37

All deltas are within ±1 pt (greedy/run-to-run noise) — the three formats are the same model on this benchmark, and the bf16 row reproduces the card's 61.6 headline (measured 61.42).

Serving:

llama-server -m kosv4-Q4_K_M.gguf --jinja -ngl 99 -c 6144 -np 1 --host 0.0.0.0 --port 8080

--jinja is required for the model's <tool_call> output to be parsed into structured tool_calls; eos is <|im_end|> (id 0) and no BOS is prepended; keep -c ≤ 6144 for in-distribution instruct behavior.

Edge Deployment & Performance (NVIDIA Jetson Orin Nano)

The highly compact memory footprint of the Q4_K_M quantization makes this model an exceptional candidate for localized, low-power edge computing platforms using unified memory architectures.

When deployed natively via llama.cpp using CUDA-offloaded layers, the model achieves the following baseline performance characteristics on an NVIDIA Jetson Orin Nano (8GB):

Phase	Throughput	Bottleneck Profile
Prefill (Prompt Processing)	~143.4 t/s	Compute-Bound ($GEMM$ execution over unified RAM)
Decode (Token Generation)	~24.8 t/s	Memory Bandwidth-Bound (Saturating the 68 GB/s bus)

Deployment Optimization Recommendations:

• Lock Hardware Clocks: Prior to initializing llama-server, maximize the power envelope and lock the frequency steps to prevent dynamic frequency scaling latency:

  sudo nvpmodel -m 1
  sudo jetson_clocks
  

Intended use & limitations

• Intended use: general instruction following, structured output, and function/tool calling in clinical-adjacent workflows.
• Not a medical-knowledge QA model. It follows instructions and calls tools; it does not reliably recall parametric medical facts. Ground it with retrieval instead.
• Below current small models. On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model's results are notable for its data/compute budget, not for leading the field.
• English only. Strong public-benchmark numbers are not validation on real clinical data.
• No safety or bias evaluation. This model has not been red-teamed, nor has it been evaluated for toxicity, clinical bias, or hallucination rates. It may produce harmful, biased, or medically inaccurate content.