MK4-Research/MK4-Ginko-v1

MK4-Ginko v1

Qwen3.8-27B with a LoRA merged in, 4-bit MLX. No adapter needed at inference.

Ginko does one thing well: it decides whether a piece of code is actually vulnerable, when the code contains guards that make it look either safer or more dangerous than it is. It was built specifically to fix a failure we measured in its predecessor, and on that narrow task it is a large improvement over the model it was fine-tuned from.

Point it at a function, not a codebase. Given a whole file it reports one issue and stops - a coverage limit that comes from how it was trained, and one you cannot prompt around. The finding it does give you is more trustworthy than its predecessor's; there is just one of them.

What it fixes

The model this replaces had a specific, repeatable failure: shown well-defended code, it announced critical vulnerabilities that were not there. It saw a dangerous-looking primitive and did not follow the guard protecting it, even when the guard was ten lines above and described in a comment.

Ginko was trained on 367 rows of reasoning traces, harvested from the base model and kept only where the base reached the verdict that ground truth said was correct. Roughly 44 percent of those rows are correctly-implemented code that looks dangerous.

Scored on five snippets chosen because the previous model failed them:

case	truth	base Qwen3.8-27B	Ginko
bearer token compared with ==	vulnerable	missed	missed
same check using hmac.compare_digest	safe	correct	correct
pinned TLS connection with context passed	safe	false alarm	correct
rmtree behind realpath + commonpath confinement	safe	false alarm	correct
forwarded header behind a loopback gate	safe	false alarm	correct
		1 of 5	4 of 5

Median answer length 121 tokens against the base's 203.

Every case the base got wrong by inventing a vulnerability, Ginko cleared, naming the actual mechanism: realpath resolving symlinks before a commonpath containment check, the loopback peer test gating the forwarded header, the global-IP filter on the pinned connection.

Five items is an anecdote, not a rate. They were chosen because they are cases where the previous model demonstrably failed, which makes this a fair test of the specific fix and not an estimate of general accuracy.

Evaluation

Two benchmarks are reported here. GuardBench is the one that matters and is explained from scratch below. FBE is an older suite, kept for continuity, with its known problems stated.

---

GuardBench

What it is

A benchmark built in August 2026 for one question: when a piece of code contains a check that is supposed to make it safe, can the model work out whether that check actually protects the thing being protected?

That question is not academic. It is where every model tested here fails on real code, in both directions - inventing vulnerabilities in code that is properly defended, and clearing code whose defence does not cover the dangerous value.

A "guard" here means any check that makes an operation safe: an allowlist test, a path containment check, a constant-time comparison, escaping a value before it reaches a query, or simply choosing a safe library over an unsafe one.

Why an existing benchmark would not do

Most security-snippet benchmarks show fifty tokens of code in which the guard sits on the line above the sink. Missing it is not really possible, so every model looks competent. On a real 1,500-line file the guard is a helper defined elsewhere, a decorator, or a branch further up - and following a guard turns out to be a completely different skill from recognising one.

So in GuardBench the guard is always at a distance from the sink it protects. That single design choice is what makes the scores below so much lower than the FBE scores further down, on the same models in the same week.

The six shapes

120 items: 20 code patterns across 19 vulnerability classes, each written six ways. The mutation sets the label, so ground truth is certain by construction rather than by anyone's judgement.

The clearest way to see the six is one vulnerability written six ways. Every version below shares this setup - a path-containment helper and a file-download endpoint:

EXPORT_DIR = Path("/srv/exports")

def _safe_join(base, name):
    p = (base / name).resolve()
    if not p.is_relative_to(base.resolve()):
        raise ValueError("outside base")
    return p

1. no guard - correct answer VULNERABLE. Nothing checks anything.

def download(name):
    return send_file(EXPORT_DIR / name)          # name = "../../etc/passwd"

2. guard covers - correct answer SAFE. The check is applied to the value that reaches the sink.

def download(name):
    return send_file(_safe_join(EXPORT_DIR, name))

3. guard covers (variant) - correct answer SAFE. The same protection achieved a different way. Two safe versions per pattern, so a model cannot pass by memorising one idiom.

def download(name):
    if name not in ALLOWED_EXPORTS:
        abort(404)
    return send_file(EXPORT_DIR / name)

4. guard on wrong value - correct answer VULNERABLE. The check runs, on thumb. The value actually served is name, and it is never checked.

def download(name, thumb):
    _safe_join(EXPORT_DIR, thumb)
    return send_file(EXPORT_DIR / name)

5. guard irrelevant - correct answer VULNERABLE. A real check, on the right value, that does nothing about traversal. ../../etc/passwd is eighteen characters.

def download(name):
    if len(name) > 64:
        abort(400)
    return send_file(EXPORT_DIR / name)

6. guard used elsewhere - correct answer VULNERABLE. The helper exists and the neighbouring function calls it. This one does not.

def download_thumb(name):
    return send_file(_safe_join(THUMB_DIR, name))

def download(name):
    return send_file(EXPORT_DIR / name)

A model that pattern-matches on "a check is present" gets 1 and 2 right and fails 4, 5 and 6. One that follows the value from input to the sink gets all six. That difference is the entire benchmark.

shape	correct answer	in one line
no guard	VULNERABLE	the check is absent
guard covers	SAFE	the check protects the dangerous value
guard covers (variant)	SAFE	the same, defended a second way
guard on wrong value	VULNERABLE	the check runs on a sibling value
guard irrelevant	VULNERABLE	the check is real and addresses a different weakness
guard used elsewhere	VULNERABLE	the helper exists; this call site skips it

How it is scored

Every answer must end with a machine-readable line, VERDICT: VULNERABLE <class> or VERDICT: SAFE. A missing line counts as wrong. That is deliberate: a reviewer whose output cannot be parsed is useless in a pipeline, and grading by scanning prose for keywords is how the FBE grader ended up counting "does not contain any vulnerabilities" as claiming a vulnerability.

Balanced accuracy is the headline: the mean of accuracy on safe items and accuracy on unsafe items. Raw accuracy is not quoted, because 80 of the 120 items are unsafe, so a model that answers VULNERABLE to everything would score 67 percent while being useless. Under balanced accuracy, always-VULNERABLE and always-SAFE both score exactly 50 percent. That is the floor to read every number against.

Each item is asked three different ways ("Review this code for security problems", "Analyze this code for security issues", "Is there a vulnerability in this code?"), giving 360 generations per model. A model whose output format depends on the exact wording is not reliable, and the no verdict line column measures that.

Results

	base Qwen3.8-27B	LOREA Pilot	Ginko v1
balanced accuracy	29%	43%	62%
accuracy on safe code	14%	32%	47%
accuracy on unsafe code	43%	54%	78%
no verdict line	66%	51%	29%
median tokens	400	400	268

shape	base	Pilot	Ginko
no guard	52%	73%	90%
guard covers	15%	37%	55%
guard covers (variant)	13%	27%	38%
guard on wrong value	30%	43%	63%
guard irrelevant	38%	38%	80%
guard used elsewhere	53%	62%	77%

The base scores below the 50 percent floor. Two thirds of its answers never produce a parseable verdict at all - it hits the token cap mid-reasoning - so its 29 percent mixes wrong judgements with unusable output rather than being pure misjudgement.

What the results mean

The training worked. Ginko leads on all six shapes: +19 balanced accuracy over LOREA Pilot and +33 over the base it was fine-tuned from, on 120 items in 19 vulnerability classes that appear in no training corpus used here.

The largest gap is guard irrelevant, where base and Pilot both score 38 percent and Ginko scores 80. Ginko's training corpus contained no example of that shape - its absence was recorded as a known flaw before this benchmark was run. Doubling both other models on a shape it was never shown is the strongest evidence here that it generalised rather than memorised.

And it is still not deployable. 47 percent on correctly-defended code means it calls about half of safe code vulnerable. That is noise, whatever the comparison says.

Where it fails is specific. Ginko scores 100 percent on classes whose defence is a sanitiser call at the sink - regex.escape, secrets.token_hex, escape_filter_chars - even for classes it never trained on. It scores 0 percent where safety is structural: choosing defusedxml over ElementTree, opening a file with O_NOFOLLOW, normalising a string before validating it, signing the value that actually matters. It recognises sanitisers. It does not yet reason about structure.

The benchmark, its generator, the runner and every result file are public at MK4-Research/GuardBench.

---

FBE, the older suite

What it is

Two snippet suites used across the LOREA models before GuardBench existed. FBE is 104 short functions each containing one planted bug, measuring how often the model finds it. FBE-safe is 65 short functions that are correctly implemented, measuring how often the model invents a finding anyway. Snippets are roughly fifty tokens, and the guard - where there is one - sits beside the sink.

	base Qwen3.8-27B	LOREA Pilot	Ginko v1
finds the planted bug, 104 items	97.1%	86.5%	84.6%
claims a bug in correct code, 65 items	12.3%	3.1%	4.6%
median answer length	460	82	115

Ginko is not better at general bug-finding. Subtracting the false-alarm rate from the find rate as a crude single score gives the base 84.8, Pilot 83.4 and Ginko 80.0.

Why these numbers disagree with GuardBench

FBE-safe puts Ginko's false-alarm rate at 4.6 percent. GuardBench puts the same model at 53 percent wrong on defended code. Both are correctly measured.

The difference is distribution. FBE-safe uses guard idioms close to the ones in Ginko's training data; GuardBench deliberately uses ones that are not. A model's false-alarm rate is only as good as the guard idioms it was trained on, and a benchmark that shares a distribution with the training data measures the familiar half of the problem.

If you take one thing from this page, take that. The 4.6 percent is real and it does not predict behaviour on your code.

Two known problems with FBE

Its grader was broken until August 2026. It scanned prose for keywords, so "vulnerab" matched inside "does not contain any vulnerabilities" and scored an all-clear as a vulnerability claim. It also matched literally, so a model writing is **securely implemented** in markdown failed to match is secure while one writing plain prose matched - penalising the base for formatting. The base's 12.3 percent above is the corrected figure and replaces a previously published 27.7 percent. All three columns use the repaired grader.

It cannot measure guard-following. In a fifty-token snippet the guard and the sink are adjacent, so the skill GuardBench isolates does not come into play at all. This is why FBE is being retired rather than extended.

One thing in FBE's favour: it is genuinely held out for Ginko. Those 169 items were LOREA Pilot's validation set and were used to choose Pilot's checkpoint, which makes them optimistic for Pilot by an unknown margin. Ginko never saw them, and its checkpoint was chosen by a training crash rather than by these scores.

---

The two things it gets wrong

One finding per file. Every training row was one snippet, four trace lines, one verdict. Given a 3,400-token file it produces exactly one finding and stops. Told explicitly that the file contains several separate problems, to list every one, and not to stop after the first, it still produced one. If you need a full file audit, this is the wrong model.

A guard that is present but irrelevant still clears it. It cleared a genuine timing-attack bug, got.strip() == TOKEN, because startswith("Bearer ") sits on the tainted path. That is a format check, not a security control. Ginko learned "a guard is applied to this value, therefore safe", and nothing in its training distinguishes a guard that addresses the weakness from one that does not. The base model makes the same mistake, so this is inherited rather than introduced.

Use it on snippets and functions, one concern at a time. That is the shape it was trained for.

Tool calling

Verified working after the fine-tune, which was not a given: the training corpus contains no agentic or tool-calling rows at all, so there was a real chance 350 iterations had crushed it.

Prompted with an OCLI-style tool schema it emits valid JSON, selects the right tool, and handles nested paths:

<tool_call>
{"name": "write_file", "arguments": {"path": "src/utils/slug.py",
 "content": "def slugify(text):\n    return text.lower().replace(\" \", \"-\")\n"}}
</tool_call>

File creation, file reading and directory paths all work. This has not been tested across long multi-turn agent loops.

Running it

pip install mlx-lm
python3 -m mlx_lm generate --model MK4-Research/MK4-Ginko-v1 \
  --system-prompt "You are a security code reviewer. Trace the data flow: name the untrusted input, the sink it reaches, and whether every guard present is applied on that path. Be terse and decisive. If the code is correct, say so plainly." \
  --prompt "Review this code for security problems: <code>"

Greedy decoding. Thinking should be left disabled; the training data contains no reasoning blocks and enabling it makes output shorter and worse.

Roughly 15 GB on disk, and it wants about 17 GB of unified memory to run.

Training

LoRA, rank 32, all 64 layers, attention and MLP projections. Batch 1 with gradient accumulation 2, gradient checkpointing on, learning rate 1e-5, max sequence length 768. Trained to iteration 350 on a 32 GB M-series machine.

Loss sat between 0.21 and 0.29 for the whole run rather than descending toward zero, which is the shape you want on a corpus this small - a 367-row templated set being memorised would have driven it much lower.

The run stopped at iteration 400 with a Metal allocator failure and a NaN loss. The trainer saved a checkpoint after the NaN and also overwrote the default adapter file; both were entirely NaN. Iteration 350 is the last clean checkpoint and is what is merged here. If you train on this stack, scan your checkpoints for NaN before trusting one.

Honest summary

Ginko is the best of these three models at following a guard and still not good enough to trust unsupervised.

On GuardBench it leads on all six shapes, +33 balanced accuracy over the base it was fine-tuned from and +19 over LOREA Pilot, across 120 items in 19 vulnerability classes it never trained on. It is also the only one of the three that reliably finishes an answer: 29 percent of its replies lack a parseable verdict against the base's 66 percent.

It nonetheless calls roughly half of correctly-defended code vulnerable, and it reports one finding per file no matter how it is prompted. Neither is acceptable in something you would run unattended.

The useful conclusion is about the method rather than this model. Rejection-sampled traces over a corpus with ground-truth labels moved guard-following a long way on 367 rows and 14 patterns. What it did not do is teach structural safety, because nothing in those 14 patterns was structurally safe. That is a data problem with a known fix, which makes it a better place to be than a tuning problem without one.

The next version needs two things this corpus lacks: examples where a guard is present, applied, and irrelevant to the real weakness, and examples with several defects in one input and several findings in one answer.