AhoosAI/nimbus-1-1-prime-ee

The first model of AhoosAI's own.
Better for everyone, not only for code.
benchmarks · models · ahoos-ai.site

---

Nimbus 1.1 Prime-EE

A QLoRA adapter for Qwen/Qwen2.5-Coder-7B-Instruct, trained on behaviour rather than on code the base model had already seen.

It is not trying to make the model a better programmer. The base is already a good one. It is trying to fix the things a good programmer gets wrong by habit: hardcoding dir="ltr", reaching for margin-left where margin-inline-start belongs, shipping a form control with no <label>, writing a Dockerfile that runs as root.

It answers in the language you wrote in, and it is at home in Persian — a right-to-left page that needs no fixing afterwards, and an answer that does not slip back into English halfway through.

Public benchmarks

Run with lm-evaluation-harness. Beside each model is the figure its own maker published.

	score
IFEval	52.3	prompt-level, strict
HumanEval	61.6	pass@1, greedy
MBPP	67.2	pass@1, 3-shot

These were not measured under the same conditions — every maker uses its own harness and its own settings — so read the tables below as context rather than as a ranking.

HumanEval		pass@1
Qwen2.5-Coder-32B	███████████████████	92.7
o1-preview	██████████████████	92.4
Claude 3.5 Sonnet	██████████████████	92.0
GPT-4o	██████████████████	90.2
Qwen2.5-Coder-7B (our base)	██████████████████	88.4
Gemini 1.5 Pro	█████████████████	84.1
DeepSeek-Coder-6.7B	████████████████	78.6
Llama 3.1 8B	███████████████	72.6
Nimbus 1.1 Prime-EE	████████████	61.6

MBPP		pass@1
Qwen2.5-Coder-32B	██████████████████	90.2
Qwen2.5-Coder-7B (our base)	█████████████████	83.5
GPT-4o	████████████████	81.4
Llama 3.1 8B	███████████████	72.8
Nimbus 1.1 Prime-EE	█████████████	67.2
DeepSeek-Coder-6.7B	█████████████	65.4

Our base publishes 88.4 on HumanEval and the same weights with this adapter scored 61.6 here. The two were taken differently: the published one asks the model as you would in a chat and reads the code out of its answer, ours asks it to continue a half-written function. A model tuned for conversation does worse at the second, adapter or no adapter. One run settles it — the same harness and the same settings with the adapter switched off — and until that exists, 61.6 is a number from this setup rather than a verdict.

IFEval reproduces exactly: 0.5231 and 0.6163 on two machines days apart, which makes it a property of the model rather than of the afternoon.

What it changes

105 tasks, each with a machine-checkable rule, run against the same base with the adapter switched on and off. Greedy, 400 new tokens, the production system prompt for each task. This one is a comparison — both sides, same tools, same day.

area	before	after
frontend	█████████ 9/20	███████████████ 15/20	+6
backend	██████████████ 14/20	███████████████ 15/20	+1
debugging	███████████████████ 19/20	███████████████████ 19/20	0
language	███████████████████ 24/25	███████████████████ 24/25	0
image specs	██████████████████ 18/20	███████████████ 15/20	−3

suite	base	adapter
language	24/25	24/25	—
specialists	60/80	64/80	+5.0%
all	84/105	88/105	+3.8%

The gain is concentrated where the base was weakest — frontend went from 45% to 75% — and nothing moved at the ceiling, where a fine-tune that learned the wrong thing shows up first.

Ten tasks changed from fail to pass:

fe-rtl-logical-properties, fe-rtl-direction, fe-logical-inset, fe-accessibility, fe-image-alt, fe-form-label, fe-dialog-escape, be-docker-hardening, be-docker-nonroot, lang-spanish-docker

They are three behaviours, not ten: RTL and logical CSS, accessibility, and container hardening.

Known regression, stated plainly

Six tasks went the other way, and three of them are image-spec tasks — the one area with no training data behind it. The image behaviour set was deferred and the score shows the cost. Until image data is in the mix, expect this adapter to trade image-spec ground for frontend ground.

If image prompt/spec work is your use case, measure before adopting.

Also worth knowing

Both arms were capped at 400 new tokens and a good share of answers stop mid-sentence. The cap is shared, so the comparison holds, but the absolute pass rates are a floor rather than a measure of what either model can do.

The 105 tasks are this project's own suite, not a public benchmark. The full run — every prompt, every answer, every check, both arms — is published alongside the code so the numbers can be disagreed with.

Training

base	Qwen/Qwen2.5-Coder-7B-Instruct, 4-bit nf4, double quant
method	QLoRA, r=16, alpha=32, dropout 0.05
target modules	q, k, v, o, gate, up, down
trainable	40,370,176 of 7,655,986,688 (0.53%)
data	3,462 rows — 2,262 behaviour, 1,200 mined public code
epochs	1 (210 steps, effective batch 16, seq 2048)
lr	2e-4, cosine

One epoch is measured, not assumed. Held-out loss over the run: 0.7029 at step 50, 0.6621 at 100, 0.6433 at 150, 0.6389 at 200, 0.6391 at 210. Each interval bought less than half of the one before it and the last ten steps bought nothing.

The public half is a regulariser, not a teacher — it keeps general coding ability from drifting while the behaviour rows do the work. It comes from bigcode/self-oss-instruct-sc2-exec-filter-50k (ODC-BY) and microsoft/orca-agentinstruct-1M-v1 (CDLA-Permissive-2.0), language-balanced across Python, C++, JavaScript, Java and TypeScript.

Use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

base = "Qwen/Qwen2.5-Coder-7B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True),
    device_map="auto",
)
model = PeftModel.from_pretrained(model, "AhoosAI/nimbus-1-1-prime-ee")
model.eval()

It was trained under this system prompt and expects it:

You are Nimbus, built by Ahoos Model Studio. You write correct, complete, runnable code and explain only what the reader needs. State plainly when something will not work.

The adapter arm generates roughly 1.9x slower than the base at batch 1, which is the cost of the LoRA matmuls at every layer and not a defect.

Trademarks

Model names and figures above belong to their respective owners and appear for comparison only. Nothing here is endorsed by or affiliated with any of them.

---

Built by AhoosAI.