Llama-3.2-1B-Cybersecurity

This model is a domain-specific instruction-tuned variant of Llama-3.2-1B-Instruct, fine-tuned on cybersecurity instruction-answer pairs. It is built to assist with security concepts, vulnerability mechanics (SQLi, XSS, CSRF), and defensive mitigation strategies.

Model Details

• Base Model: unsloth/Llama-3.2-1B-Instruct
• Architecture: 1 Billion parameters, decoder-only transformer
• Fine-Tuning Method: QLoRA (4-bit quantization via bitsandbytes, LoRA rank $r=16$, $\alpha=16$)
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Training Configuration

• Dataset: Tiamz/cybersecurity-instruction-dataset (Filtered subset of 1,500 training examples)
• Max Sequence Length: 2048 tokens
• Batch Size: Per-device batch size = 2, Gradient accumulation steps = 4 (Effective batch size = 8)
• Learning Rate: $2 \times 10^{-4}$
• Optimization Steps: 60 steps (1 Epoch)
• Precision: FP16 / BF16 mixed precision, exported as merged 16-bit weights (merged_16bit)

Prompt Template

The model was trained using the following instruction-response format:

### Instruction:
[Your cybersecurity question here]

### Response:
[Model response]