LForster/qwen-cyber-sft

Qwen Cyber SFT — C/C++ vulnerability detection

A LoRA adapter for Qwen3.5-9B, fine-tuned on a single NVIDIA RTX 3090 (24 GB) to classify individual C/C++ functions as VULNERABLE or SAFE.

Developed and shared by LForster. This release contains checkpoint 540 from the detection SFT experiment. It contains adapter weights, not the full base model. It is a vulnerability-classification research adapter; the separate cybersecurity Q&A and patch-repair experiments are not included in this release.

Training

• Base model: Qwen/Qwen3.5-9B.
• Training examples: 5,458 function-level examples from the project's C/C++ detection dataset.
• Validation: 682 functions, comprising 341 vulnerable/patched pairs.
• Input: an individual function with filename and language, without repository retrieval or tools.
• Target: VULNERABLE or SAFE; supervised fine-tuning with assistant-only loss.
• LoRA: rank 16, alpha 32, dropout 0.05, no bias; language-model attention and MLP projections.
• Released checkpoint: step 540, approximately 1.58 epochs according to trainer state.
• Selection: validation calibrated pair accuracy plus 0.01 times ranking accuracy.
• Adapter framework recorded in the checkpoint: PEFT 0.21.0.

Evaluation

Evaluated on the balanced paired PrimeVul subset: 435 vulnerable functions and their 435 patched counterparts. Each function is presented separately. A pair is correct only when both versions receive the correct label.

Evaluation rule	Correct pairs	Paired accuracy
Base model, historical validation-calibrated comparison	15 / 435	3.45%
Checkpoint 540, historical validation threshold	73 / 435	16.78%
Checkpoint 540, frozen native reference threshold	78 / 435	17.93%

The historical base-to-adapter comparison uses the historical calibration procedure. The two adapter rows use identical saved benchmark scores with different validation-derived thresholds; they are not different checkpoints.

The frozen native reference scores complete candidate answers, including ending tokens, using native BF16 output logits. Its decision margin is log P(VULNERABLE completion) - log P(SAFE completion); predict VULNERABLE when the margin exceeds -0.3750172257423401. The historical threshold was -0.37502026557922363. These thresholds do not apply to first-token-only scoring or arbitrary generation.

For the frozen native reference:

• Vulnerable-class F1: 59.66%.
• Function accuracy: 54.14%.
• Confusion counts: TP 295, FP 259, FN 140, TN 176.
• Flagging every function vulnerable yields 66.67% F1 and zero correct pairs on this population.

Fitting a threshold directly on the benchmark yields 80/435 (18.39%). That is a posthoc diagnostic and is excluded from the headline results.

Comparison with MiMo-V2.6-Pro

Evaluation population	Qwen3.5-9B base	This adapter (checkpoint 540)	MiMo-V2.6-Pro
PrimeVul paired subset, 435 pairs	15/435 (3.45%)	78/435 (17.93%, frozen native reference)	86/435 (19.77%)
Curated function-only CVEfixes set, 58 pairs	2/58 (3.45%)	13/58 (22.41%)	25/58 (43.10%)

All entries are both-correct pair accuracy, not individual-function accuracy. MiMo uses an investigation harness with generated binary verdicts; Qwen uses candidate-answer likelihood scoring and validation-derived thresholds. The comparisons use the same input functions within each population, but different prompts and inference procedures. They compare model-plus-procedure outcomes and do not isolate model weights. MiMo scored higher on both populations.

For the 58-pair set, individual-function accuracy was 51.72% for base Qwen, 56.90% for the adapter and 68.97% for MiMo. The adapter correctly classified six pairs that MiMo missed, while MiMo correctly classified eighteen pairs that the adapter missed. The exact two-sided McNemar p-value was 0.02266.

How the function-only CVE evaluation was built

The set contains 58 CVE-linked vulnerable/patched pairs (116 functions) from 34 projects, selected and frozen before inference from CVEfixes v1.0.8 provenance and public parent/fix source files. It excludes previously evaluated cases from the earlier collection.

1. Collect candidate CVE fix commits and recover the before/after source. Require one changed production C/C++ file, and extract the matching complete function with the most changed lines.
2. Review whether that function contains the linked unsafe operation and whether its patch removes the mechanism. Exclude cases needing omitted project-specific contracts, types, macros, helper behavior, concurrency or lifecycle context, unclear CVE correspondence, or evident residual vulnerabilities in the patched function. Ordinary C/C++ and common API semantics are allowed.
3. Check recorded local training and evaluation sources for CVE, commit and normalized-code overlap. Exclude same-name near clones at seven-token-shingle Jaccard similarity of at least 0.80.
4. Require both functions to fit 4096 tokens without truncation, apply a predeclared maximum of ten pairs per project, and freeze the manifest before running either model. Review admitted 60 candidates from 365 collected; the project cap left 58, below the planned target of 100.
5. Present each version independently as function code with filename and language. CVE identities and provenance are evaluation metadata rather than classification targets. Neither model receives repository context or repository tools. A correct pair requires VULNERABLE for the pre-fix function and SAFE for its patched counterpart.

The adapter uses the historical validation threshold -0.37502026557922363; the base model uses its saved validation threshold -0.1262415647506714. Neither threshold is fitted on these CVE cases. MiMo uses the mimo-v2.6-pro endpoint and the existing investigation harness. Invalid or missing answers count as wrong; all 116 MiMo verdicts completed on this set.

This is a single-reviewer curated population designed for function-only assessment. SAFE denotes the patched counterpart for the linked CVE. Independence from base-model pretraining is unknown, and the cases do not represent all CVEs.

An earlier, separate 67-pair CVEfixes collection scored 8/67 (11.94%) for this adapter and 14/67 (20.90%) for MiMo, with one missing MiMo answer counted as wrong. It used a different case-selection policy and should not be pooled with the stricter 58-pair function-only set.

Loading the adapter

Use Transformers with Qwen3.5 support, PyTorch, Accelerate and PEFT. The tokenizer comes from the base model.

import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer
from peft import PeftModel

base_id = "Qwen/Qwen3.5-9B"
tokenizer = AutoTokenizer.from_pretrained(base_id)
base = AutoModelForImageTextToText.from_pretrained(
    base_id,
    dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
)
model = PeftModel.from_pretrained(base, "LForster/qwen-cyber-sft")
model.eval()

This example loads the adapter. Reproducing the reported metrics requires the original prompt format, non-thinking chat rendering, whole-answer likelihood scoring and corresponding validation threshold. Free-form generation is not the reported evaluation method. Small numeric changes can alter decisions because many BF16 margins are close to the threshold.

Limitations and intended use

Intended for research on function-level vulnerability classification and fine-tuning. Buggy/fixed discrimination remains weak, and vulnerable-class F1 is below the flag-everything baseline. The balanced paired evaluation includes no unrelated benign functions and establishes no performance on the full imbalanced PrimeVul test population or production repositories.

PrimeVul and existing CVEfixes results informed later diagnostics; they are not untouched tests for future model selection. No matched claim of superiority over frontier models is made. This adapter does not establish reliable vulnerability triage or functional patch repair.

Artifact identity

adapter_model.safetensors SHA-256:

edc4d23e0f881ea434f84ba82f2d10fac5df82df86861ae2a72aed6034c11ec9