Taimwe/securecoder-scripts

SecureCoder — training runbook

A QLoRA fine-tune that targets three skills at once: coding, tool calling and cybersecurity (offence and defence), on a base chosen to train cheaply and run fast. Everything here is measured or verified on real jobs, not assumed — each claim carries the job id you can inspect.

1. Base model choice

Screened on the Hub on 2026-09-24 (params / licence / tool-calling support):

Candidate	Params	Licence	Verdict
Qwen/Qwen3-Coder-30B-A3B-Instruct	30.5B MoE, ~3B active	Apache-2.0	chosen — code-specialised, native tool-calling template, MoE means it trains like a 3B model and fits 48 GB in 4-bit
Qwen/Qwen3.8-27B	27.8B dense	Apache-2.0	best raw quality, ~4x slower per step; good upgrade path once the pipeline is proven
Qwen/Qwen3-Coder-Next	79.7B	Apache-2.0	needs an 80 GB card (A100 80 GB $2.50/h); too big for the first run
Qwen/Qwen3.8-Flash-Next	180B	license: other	rejected — non-commercial-style licence on an 180B model
openbmb/MiniCPM5-2B	2.5B	Apache-2.0	tool-calling tag, tiny; good for a T4 smoke test, weak ceiling
TokenRhythm/NeoHorse-1-9B	9.0B	Apache-2.0	tool-use, reasoning; the fallback if T4-only

Why not start from an existing abliterated checkpoint? Because refusal removal is a separate weight-edit (section 6) that can be applied to any checkpoint after fine-tuning, and pre-abliterated bases are community re-uploads with unclear provenance. Fine-tuning first keeps the lineage clean and the base licensed.

2. Data mix

The cap per source is the recipe — sources differ hugely in size, so the mix is balanced by taking a fixed slice of each. Verified on a real CPU job (Taimwe/6ab5ae256b030d633f68faef) at 25 rows/source:

Source	Rows taken	Kind	What it teaches
NousResearch/hermes-function-calling-v1 [func_calling]	9,000	tool calling	full tool-call conversations + JSON schemas
NousResearch/hermes-function-calling-v1 [func_calling_singleturn]	3,000	tool calling	picking the right function, no chatter
lockon/xlam-function-calling-60k	10,000	tool calling	60k API-call pairs (query → call)
ise-uiuc/Magicoder-OSS-Instruct-75K	10,000	coding	self-instruct problems + solutions
Trendyol/Trendyol-Cybersecurity-Instruction-Tuning-Dataset	8,000	security	security instruction tuning
AlicanKiraz0/Cybersecurity-Dataset-Fenrir-v2.1	5,000	security	broad security Q&A
Humanlearning/CyberSecurity_OWASP-sft-dataset	3,000	secure coding	OWASP / defensive coding
MrClipperz134/CTF-Instruct	3,000	CTF	challenge instruction → solve
TrueNix/ctf-solver-dataset	3,000	CTF	solving trajectories
mlabonne/FineTome-100k	3,000	replay	general chat, so instruction-following does not collapse

Total ≈ 57,000 rows, ~30% of them tool-calling. Every source validated at 25/25 kept with zero render failures.

Excluded deliberately:

• dpevzner/Cybersecurity_Reasoning_Dataset — its builder configs were renamed (default → mistral|deepseek|chatml|gemma) and the fields the card documents (unified_interpretation) are empty in the published revision, so every row was dropped. Re-addable once its schema settles.
• Anything whose purpose is building malware or weaponised exploits. Recon and enumeration knowledge, exploit concepts, CTF solving and defensive engineering are in; end-to-end attack tooling is not. This is a deliberate scope line, not an oversight.

3. What training does with the data

Each row is converted to chat messages and rendered through the model's own chat template, so the model learns the exact format it will be asked for at inference time (<|im_start|>…<tool_call><function=NAME><parameter=…> for Qwen3-Coder).

Two format traps found the hard way and handled in code:

• Qwen3-Coder iterates tool.parameters.properties — tool schemas must be flat ({"name", "description", "parameters"}), not wrapped in "function".
• It also iterates tool_call.arguments | items, so arguments must be a mapping. A JSON string there fails with "Can only get item pairs from a mapping" — which is exactly how xLAM was silently contributing 0 rows until fixed.

render_record() therefore detects the template's convention once (flat/nested × dict/string) and reuses it, instead of hard-coding one model's dialect.

4. Compute and cost

Verified hf jobs hardware prices (per hour), 2026-09-24:

Flavor	VRAM	$/hour	Use
cpu-basic	—	$0.01	data-mix validation, class checks
t4-small	16 GB (T4)	$0.40	plumbing checks; too small for a 30B in 4-bit
l4x1	24 GB	$0.80	9B runs on a budget
a10g-large	24 GB	$1.50	9B–14B comfortably
l40sx1	48 GB	$1.80	the 30B-A3B QLoRA
a100-large	80 GB	$2.50	faster per step, larger batch
rtx-pro-6000	96 GB	$2.75	headroom for longer context
h200	141 GB	$5.00	full fine-tune territory, overkill here

Rough budget for the full 1-epoch run. These are now measured, not guessed — the smoke test (Taimwe/6ab5b3ed52d0dbd7f1d8d454, a100-large) trained at 0.8 rows/s (20 s per step at batch 2 × grad-accum 8, max seq 2048, packing on), which includes a few minutes of torch.compile warm-up that does not recur per step:

Mix	Rows	Time at 0.8 rows/s	Cost on l40sx1 ($1.80/h)
full mix as specified	~57,000	~20 h	~$36
half caps (--max-steps or edit the table)	~28,000	~10 h	~$18
lean mix (cut caps to ~1/3)	~19,000	~7 h	~$12

Because warm-up inflates that rate, a 300-step measured run (~$4 on a100-large) gives a trustworthy steady-state number before committing to a long job. Levers that move the cost most, in order: rows in the mix, --max-seq-length (2048 vs 4096), --batch-size (raise it on 80–96 GB cards), --lora-r.

The --smoke run itself costs under $1 and proves model load, data rendering, training, evaluation and the Hub push end to end.

No local GPU needed. Colab works the same way with uv run train_securecoder.py … (Colab Free's T4 cannot hold a 30B in 4-bit — use Colab Pro L4/A100, or point --base-model at a 9B).

5. Runbook (copy-paste)

# 0. auth (token needs repo.write, and job.write for Jobs)
hf auth login --token hf_xxx

# 1. validate the mix: 25 rows/source, no GPU, ~50 s, ~$0.0002
hf jobs run -d --flavor cpu-basic --timeout 30m --name validate-mix \
  ghcr.io/astral-sh/uv:python3.12-bookworm \
  uv run --no-project https://huggingface.co/Taimwe/securecoder-scripts/resolve/main/train_securecoder.py \
  --validate-only --validate-per-source 25 --show-samples 3

# 2. smoke test on a real GPU: 200 rows/source, 20 steps, pushes an adapter (~$0.50)
hf jobs run -d --flavor l40sx1 --timeout 40m --secrets HF_TOKEN --name smoke-train \
  ghcr.io/astral-sh/uv:python3.12-bookworm \
  uv run --no-project https://huggingface.co/Taimwe/securecoder-scripts/resolve/main/train_securecoder.py \
  --smoke --output-repo Taimwe/securecoder-smoke --private --report-to none

# 3. the real run (~$9–14)
hf jobs run -d --flavor l40sx1 --timeout 12h --secrets HF_TOKEN --name securecoder-run1 \
  ghcr.io/astral-sh/uv:python3.12-bookworm \
  uv run --no-project https://huggingface.co/Taimwe/securecoder-scripts/resolve/main/train_securecoder.py \
  --num-epochs 1 --output-repo Taimwe/securecoder-30b-pro \
  --trackio-space Taimwe/securecoder-trackio

# monitor
hf jobs inspect Taimwe/<job_id> --json
hf jobs logs    Taimwe/<job_id>
hf jobs stats   Taimwe/<job_id>
hf jobs cancel  Taimwe/<job_id>

Platform traps already worked around (verified 2026-09-24)

Symptom	Cause	Workaround
failed to set MOUNT_ATTR_IDMAP on /usr/bin/nvidia-cuda-mps-control on a GPU flavor	hf jobs uv run mounts an artifacts bucket; the idmap mount fails on some GPU nodes (reproduced twice, 4 s after start)	use hf jobs run + the script's public Hub URL (works: Taimwe/6ab5ad3c6b030d633f68face) or a read-only repo mount -v hf://models/Taimwe/securecoder-scripts:/scripts:ro (works: Taimwe/6ab5ad3d52d0dbd7f1d8d286)
inner quotes stripped from python -c "…"	PowerShell native-argument quoting	use a UV script file, never -c one-liners
charmap codec can't encode while reading logs	Windows console encoding	set $env:PYTHONIOENCODING='utf-8' before hf jobs logs
HF's trainer skill says Jobs need a paid plan	docs caveat	not enforced on this account — CPU and GPU jobs both ran

6. Turning down refusals (optional, after SFT)

Abliteration is a weight edit, not a training run: find the "refusal direction" in the residual stream and project it out of the writing matrices.

1. Heretic (p-e-w/heretic) — the tool behind the recent wave of *-heretic-abliterated-* checkpoints (e.g. culturerevolt/gemma-4-12b-heretic-abliterated-GGUF, 174k downloads). Runs on the merged 16-bit model and optimises the ablation to trade refusal rate against KL divergence, so it preserves capability better than a hand-rolled ablation.
2. Manual directional ablation (Arditi et al., Refusal in LLMs is mediated by a single direction): collect residual activations for harmful vs. harmless prompts, take the mean difference per layer, and zero that component in o_proj / down_proj.

Honest expectations: it is not free. Refusals drop, and some performance on the same layers' other duties drops with them; scope-setting behaviour ("this needs authorisation") can weaken too. Measure both sides (section 7) and document the result on the model card, or you are shipping an unmeasured change.

7. Evaluation before publishing

1. Tool-call validity — 200 prompts with real tool schemas; parse the emitted <function=NAME><parameter=…> block back to JSON and score syntax validity, correct function choice, and schema-conformant arguments. This is the number that matters most for "top-notch tool calling".
2. Code sanity — generate 100 solutions, check compilation, run the bundled tests on a small HumanEval-style subset.
3. Security knowledge — fixed MCQ/triage set (e.g. CyberNative/CyberSecurityEval), scored before and after fine-tuning so the mix's effect is visible.
4. Regression — the same three on the untouched base, same prompts. A fine-tune that feels better but scores worse is a regression with extra steps.
5. Refusal rate — if abliterated, publish before/after refusal rate next to the eval deltas.

8. Known risks

• MoE LoRA defaults to attention + router (q/k/v/o/gate). --target-modules all-linear on a 128-expert model means ~800M trainable parameters — much slower and hungrier. Start conservative; revisit if security knowledge is the weak score.
• 4-bit QLoRA of a 30B (~17 GB) is tight on a 24 GB card; 48 GB allows batch 2 with 4096-token samples.
• One 1-epoch pass over three domains is a start, not a finished model. Iterate on the mix ratios — they live in one table at the top of the script.
• Nothing here adds safety training, and the security slice includes recon/enumeration knowledge framed for authorised use. Publish explicit intended-use and limitations sections; do not present it as a hardened model.