Legendarylibrary/Qwen3-4B-CVE-DPO-LoRA

APPEARS THE LEARNNG RATE AND DPO OPTIMIZIED FOR VULNS INSTEAD OF SECURE CODE. USING THIS TO GENERATE HARD NEGATIVES. ASSUME IT GENERATES UNSAFE CODE.

Qwen3-4B CVE DPO LoRA

LoRA adapter for Qwen/Qwen3-4B, aligned with Direct Preference Optimization (DPO) on code vulnerability preference pairs. The model learns to prefer safer (FAILED, NOT SAFE CODE GEN, FOR RESEARCH), less vulnerable code responses over rejected alternatives.

Trained with Seiso distill-RL pipeline (qwen3-cve-dpo-969c22a1).

Model Details

Field	Value
Base model	Qwen/Qwen3-4B
Method	DPO + QLoRA
Checkpoint	step 521 (final)
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.05
DPO beta	0.1
DPO epochs	2
Learning rate	5e-7
PEFT version	0.19.1

LoRA target modules

q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Training data

Preference pairs from the code_vuln_dpo dataset (chosen vs. rejected code responses for vulnerability-related prompts).

Final training metrics (step 521)

Metric	Value
DPO loss	0.250
Preference accuracy	93.8%
Reward margin	1.64
Chosen reward	2.50
Rejected reward	0.86

Usage

Load the base model and apply this adapter with PEFT:

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE_MODEL = "Qwen/Qwen3-4B"
ADAPTER = "Legendarylibrary/Qwen3-4B-CVE-DPO-LoRA"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

messages = [
    {"role": "user", "content": "Review this code for security vulnerabilities:\n\n```python\ndef login(user, pwd):\n    query = f\"SELECT * FROM users WHERE name='{user}' AND pass='{pwd}'\"\n    return db.execute(query)\n```"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Merge adapter into base weights (optional)

merged = model.merge_and_unload()
merged.save_pretrained("./qwen3-4b-cve-dpo-merged")
tokenizer.save_pretrained("./qwen3-4b-cve-dpo-merged")

Limitations

• Specialized for code vulnerability preference alignment; general capabilities inherit from the base model.
• Always validate security recommendations with static analysis, testing, and human review.
• Not a substitute for professional security auditing.

License

This adapter is released under the Apache 2.0 license, consistent with the Qwen3 base model license. Use of the base model is subject to the Qwen3 license terms.

Citation

@misc{qwen3-4b-cve-dpo-lora,
  author = {Legendarylibrary},
  title = {Qwen3-4B CVE DPO LoRA},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Legendarylibrary/Qwen3-4B-CVE-DPO-LoRA}}
}