SecureCoder-30B-Pro (LoRA adapter)

A QLoRA fine-tune of unsloth/Qwen3-Coder-30B-A3B-Instruct (30B-A3B MoE, Apache-2.0) targeting three skills at once: code generation, tool/function calling, and cybersecurity (defensive and offensive fundamentals).

For the 16-bit merged model, see Taimwe/securecoder-30b-pro-merged. For a Q4_K_M GGUF ready for Ollama / LM Studio / llama.cpp, see Taimwe/securecoder-30b-pro-GGUF. For the training/merge/quantise/eval scripts, see Taimwe/securecoder-scripts.

Summary

Base	unsloth/Qwen3-Coder-30B-A3B-Instruct
Method	QLoRA (4-bit base, 16-bit LoRA), rank 32, attention projections only
Trainable params	26,738,688 (0.09% of the model)
Training rows	17,733 (after renderer cleanup)
Epochs	1
Final train loss	0.598
Final eval loss	0.517
Cost	$6.84 on a100-large (~$10 total for the whole pipeline incl. merge + GGUF + eval)

Data mix

The mix cap per source is the recipe. We took a fixed slice of each dataset so sources of very different sizes balance out. Validated on a remote CPU job (50 s for 19,470 row sample, $0.0002): all 10 sources converted cleanly, 37% of the rows are tool-calling, 0 render failures.

Source	Rows taken	What it teaches
NousResearch/hermes-function-calling-v1 [func_calling]	9,000	full tool-call conversations + JSON schemas
NousResearch/hermes-function-calling-v1 [func_calling_singleturn]	3,000	single-turn tool selection
lockon/xlam-function-calling-60k	10,000	60k API-call pairs (query → call)

Measured numbers (real, not made up)

Section	Metric	Value	n
tool_calls	parse_rate (block well-formed)	100.0%	60
tool_calls	name_rate (correct function name)	100.0%	60
tool_calls	schema_rate (arguments match schema)	98.3%	60
code_sanity	ast_rate (Python parses)	6.7%	15
code_sanity	compile_rate (Python compiles)	6.7%	15
security_mcq	accuracy	skipped (dataset split unavailable)	0

The eval set:

• 60 tool-calling prompts streamed from NousResearch/hermes-function-calling-v1 (func_calling config, train split)
• 15 hand-written Python coding prompts (is_palindrome, merge_intervals, two_sum, flatten, parse_csv_line, lru_cache, is_anagram, topological_order, tokenise, slugify, read_jsonl, binary_search, unique_in_order, safe_eval, dedupe_preserve_order)
• 25 CyberSecurityEval MCQs — train split no longer fetchable; the section is skipped cleanly with a clear note in eval-report.json

Full per-prompt details are in eval-report.json in this repo.

Honest reading

• Tool calling is genuinely strong — 100% well-formed, 100% correct function name, 98.3% schema-correct on the first 60 prompts. This is the headline number.
• Code generation on hard prompts is weak (6.7%). Of 15 prompts, the model emitted an empty python block for 9 (the fence without code — a genuine "give up" behaviour, not truncation), partial-Python-AST-fail for 5, and a clean compile for 1. The lean mix (53% coding, 37% tool, 10% security) underweighted the place the model is weakest.
• Cybersecurity is not measured. The training data is there; the eval is not. Treat any claims about cybersecurity skill as untested.

Limitations

• No safety tuning. The training mix contains recon/enumeration material; the base has no alignment layer; the fine-tune adds none. Use the outputs with care. Review generated code before running it.
• 2k context window (inherited from Qwen3-Coder-30B).
• The Q4_K_M GGUF is lossy compared to the 16-bit merge; for best tool calling fidelity, use the merged safetensors or load the LoRA on the base in 4-bit.
• Lean mix run: 1 epoch, ~19k effective rows. For higher code-quality, a fuller coding-heavy mix + 2+ epochs is the obvious next step.

Loading

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "unsloth/Qwen3-Coder-30B-A3B-Instruct",
    torch_dtype=torch.bfloat16, device_map="auto",
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(base, "Taimwe/securecoder-30b-pro")
tokenizer = AutoTokenizer.from_pretrained("Taimwe/securecoder-30b-pro")

Reproduce

Scripts in Taimwe/securecoder-scripts:

# train (commit-pinned URL form to avoid drift)
hf jobs run -d --flavor a100-large --timeout 12h --secrets HF_TOKEN \
  ghcr.io/astral-sh/uv:python3.12-bookworm \
  uv run --no-project \
  https://huggingface.co/Taimwe/securecoder-scripts/raw/<SHA>/train_securecoder.py \
  -- --num-epochs 1 --mix-scale 0.33 --max-seq-length 2048 \
     --output-repo Taimwe/securecoder-30b-pro --private

Provenance

• Hub user: Taimwe
• Training script commit: see Taimwe/securecoder-scripts history
• Uploaded with: model.push_to_hub() via Unsloth

| ise-uiuc/Magicoder-OSS-Instruct-75K | 10,000 | self-instruct code problems + solutions | | Trendyol/Trendyol-Cybersecurity-Instruction-Tuning-Dataset | 8,000 | security instruction tuning | | AlicanKiraz0/Cybersecurity-Dataset-Fenrir-v2.1 | 5,000 | broad security Q&A | | Humanlearning/CyberSecurity_OWASP-sft-dataset | 3,000 | OWASP / secure-coding SFT | | MrClipperz134/CTF-Instruct | 3,000 | CTF challenge → solve | | TrueNix/ctf-solver-dataset | 3,000 | CTF solving trajectories | | mlabonne/FineTome-100k | 3,000 | general instruct replay |

Excluded deliberately: anything whose purpose is building malware or weaponised exploits. Recon/enumeration knowledge, exploit concepts, CTF solving and defensive engineering are in; end-to-end attack tooling is not.