NovaeonStudio/agent-release-gate-qwen3-4b-lora

agent-release-gate · Qwen3-4B-Instruct-2507 LoRA (MLX)

A small LoRA adapter that turns Qwen3-4B-Instruct-2507 into a release gate for changes to AI agents and automations: given a change request, it returns JSON with a risk class, the required gates and a short rationale. Trained with mlx-lm on Apple Silicon, on the synthetic NovaeonStudio/agent-release-gate dataset.

Results (held-out test split, n=341, greedy decoding)

Model	Risk accuracy	Gate micro-F1	Gate exact match	Valid JSON
Qwen3-4B-Instruct-2507 (4-bit), zero-shot, short prompt	0.513	0.728	0.194	1.000
Qwen3-4B-Instruct-2507 (4-bit), zero-shot, full rubric in prompt	0.545	0.766	0.320	0.994
+ agent-release-gate LoRA (short prompt)	0.633	0.866	0.478	1.000

"Short prompt" = the one-line system prompt below, used in training. The rubric row shows what the base model reaches when it is given the full labelling rubric in the prompt.

Usage

from huggingface_hub import snapshot_download
from mlx_lm import load, generate
adapter = snapshot_download("NovaeonStudio/agent-release-gate-qwen3-4b-lora")
model, tok = load("mlx-community/Qwen3-4B-Instruct-2507-4bit", adapter_path=adapter)
system = "You are a release gate for changes to AI agents and automations. Classify the change request. Answer with JSON only: {\"risk\": \"low|medium|high|critical\", \"gates\": [...], \"rationale\": \"...\"}. Allowed gates: regression_tests, security_review, peer_review, business_approval, cab_approval, rollback_plan, staged_rollout."
msgs = [{"role": "system", "content": system},
        {"role": "user", "content": "Swap the invoice-matching agent in production from model A to model B ..."}]
print(generate(model, tok, tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False), max_tokens=200))

The adapter targets the base model's 4-bit MLX build; it was not fused or tested against other quantisations.

Training

LoRA via mlx_lm lora, base mlx-community/Qwen3-4B-Instruct-2507-4bit, 16 layers, batch 4, lr 1e-4, max length 1024, prompt masked, LoRA rank 8, scale 20, 1,200 iterations (~1.6 epochs over 2,926 training rows, with 153 train-split rows held out for validation; these counts are from before the 8-row privacy screen; final val loss 0.856, best 0.833 at iteration 500), about 20 minutes on an Apple M5 Max, peak system memory 51 GB.

Limitations

• Fully synthetic. Written by a language model from a fixed rubric; no real change records, customers, systems or people. Phrasing is cleaner and more uniform than real tickets.
• Labels encode one rubric, not an industry standard. Real organisations weigh risk differently (regulation, CAB culture, tooling). Treat it as a starting point, not as compliance advice.
• English only. SAP modules and a few well-known product names appear only as generic context; all organisations, teams, systems and ids are invented.
• Output may start with an empty <think></think> block from the chat template; parse the first {...} JSON object. The labels are usually the strongest part; rationales are short and can be internally inconsistent (e.g. we saw one that called a change critical and then said it was "not critical").
• A decision aid, not an approver: route anything high/critical to humans.

License: Apache-2.0 (same as the base model). Built by Novaeon Studio, 2026.