Athenas-Primal-8B

Athenas-Primal-8B is a LoRA fine-tune of Qwen3-8B (4-bit quantized) trained on synthetic agentic software engineering traces. The model is designed to investigate, diagnose, and fix production-grade bugs across multiple programming languages and technology stacks.

Model Description

• Developed by: kodjaoglanian
• Base model: unsloth/Qwen3-8B-unsloth-bnb-4bit
• License: Apache 2.0
• Finetuned from: Qwen3-8B (4-bit NF4, bitsandbytes)
• Adapter type: LoRA (rank 16, alpha 16)
• Max sequence length: 4096
• Languages: English, Portuguese

Training Methodology

The model was fine-tuned using Supervised Fine-Tuning (SFT) with LoRA adapters via the Unsloth framework and TRL's SFTTrainer.

LoRA Configuration

Parameter	Value
Rank	16
Alpha	16
Dropout	0
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Gradient checkpointing	Unsloth

Training Hyperparameters

Parameter	Value
Epochs	3
Learning rate	2e-4
LR scheduler	Linear
Effective batch size	8 (4 × 2 accumulation)
Warmup steps	10
Optimizer	adamw_8bit
Weight decay	0.01
Packing	Enabled

Infrastructure

• Platform: AWS SageMaker Studio
• GPU: NVIDIA A10G (24 GB)
• Framework: Unsloth + TRL
• Precision: bfloat16

Dataset

Training data: kodjaoglanian/fable-traces-mini

A synthetic dataset of multi-turn agentic conversations simulating real-world debugging sessions. Each trace follows an iterative investigation pattern:

1. A production incident is reported with symptoms, logs, and context
2. The assistant reasons through the problem using structured thinking
3. Tool calls are made to inspect and modify source code
4. Failures and compilation errors are encountered and resolved
5. The root cause is identified and a fix is applied

The dataset covers 29 scenarios across 8 languages (Python, Node.js, Rust, Go, Java, C++, TypeScript, C) with bug categories including memory leaks, race conditions, goroutine leaks, N+1 queries, thread pool starvation, SQL injection, use-after-free, and more.

Evaluation

HellaSwag

Model	acc	acc_norm
Qwen3-8B base (4-bit)	0.5140	0.6580
Athenas-Primal-8B	0.5625	0.7460

Fine-tuning yielded +8.8 percentage points improvement in acc_norm over the base model.

ARC Challenge

Model	acc	acc_norm
Qwen3-8B base (4-bit)	0.5375	0.5623
Athenas-Primal-8B	0.5418	0.5648

Context

Model	acc_norm	Precision
GPT-4	~95.3%	16-bit
Llama-3-70B	~88.7%	16-bit
Qwen2.5-7B	~83.0%	16-bit
Llama-3-8B	~80.7%	16-bit
Athenas-Primal-8B	74.6%	4-bit

Market figures are from full-precision models. The 4-bit quantization accounts for approximately 5–8pp of the gap.

Intended Use

• Agentic debugging: Multi-step investigation and resolution of production incidents
• Code review: Detection of concurrency bugs, resource leaks, and security vulnerabilities
• Cross-language engineering: Python, Node.js, Rust, Go, Java, C++, TypeScript, C

Limitations

• Trained on synthetic data — real-world bug patterns may differ
• 4-bit quantization introduces a measurable accuracy penalty versus full precision
• No RLHF or safety alignment was applied
• Not optimized for general conversation, creative writing, or mathematical reasoning
• Tool calls in training are simulated; the model does not execute code

Quick Start

from unsloth import FastLanguageModel
from transformers import TextStreamer

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Kodjaoglanian/athenas-primal-8b",
    max_seq_length=4096,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

messages = [
    {"role": "system", "content": "<your system prompt>"},
    {"role": "user", "content": "Describe your bug or engineering problem here."},
]

inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt",
).to("cuda")

_ = model.generate(
    input_ids=inputs,
    streamer=TextStreamer(tokenizer, skip_prompt=True),
    max_new_tokens=2048,
    temperature=0.7,
    top_p=0.9,
)

Citation

@misc{athenas_primal_8b,
  author = {kodjaoglanian},
  title = {Athenas-Primal-8B: Agentic Software Engineering via LoRA Fine-Tuning of Qwen3-8B},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Kodjaoglanian/athenas-primal-8b}
}

Acknowledgments

• Unsloth — efficient fine-tuning framework
• Qwen Team — base model
• TRL — SFTTrainer