ThorOdinson246/nl2sh-3b-Q4_K_M

nl2sh-3b (GGUF, Q4_K_M)

The larger sibling of nl2sh-1.5b: 1.9 GB, still CPU-only, and about 4 points more accurate at roughly half the speed. Turns a plain-English request into a single shell command.

Qwen2.5-Coder-3B-Instruct with a LoRA fine-tune (r=32, α=64, all linear layers) on 125,770 natural-language/shell pairs, merged and quantized to GGUF Q4_K_M. Built for nl2sh but usable with any llama.cpp runtime.

Which one should you use

	nl2sh-1.5b	nl2sh-3b
size on disk	941 MB	1.9 GB
InterCode-ALFA	0.620	0.657
generation speed	39.5 tok/s	17.3 tok/s
peak RAM	~1.8 GB	~3.4 GB
cold start	~2 s	~4 s

The 3B is +4.0 points more accurate, measured on the same 300 tasks, same stack, three repeats each — comfortably outside the noise floor. It costs 2.3× throughput and ~1.9× memory. Take the 1.5B for interactive use on a laptop; take this if accuracy matters more than latency.

Results

Measured on InterCode-ALFA, which scores a command by executing it in a container and comparing the resulting filesystem, file contents and stdout against a reference. A task passes only on an exact match, across 300 tasks.

model	size	pass rate
GPT-4o (cloud API, figure published by the benchmark authors)	—	0.73
nl2sh-3b	1.9 GB	0.657
Qwen2.5-Coder-7B-Instruct, untuned	4.4 GB	0.613
nl2sh-1.5b	941 MB	0.620
Qwen2.5-Coder-3B-Instruct, untuned (the base of this model)	1.9 GB	0.567

Fine-tuning is worth +0.090 on this base (0.567 → 0.657, p = 0.002 by exact McNemar on paired per-task outcomes). The model also beats an untuned 7B — a model 2.3× its size — by 4.4 points.

Measured with the unmodified upstream scorer at temperature 0, 64-token budget, on all 300 tasks, using paired per-task comparisons.

Use

# with the nl2sh CLI (github.com/ThorOdinson246/nl2sh)
nl2sh setup --model nl2sh-3b-Q4_K_M.gguf --bin-dir /path/to/llama.cpp/bin

# or llama.cpp directly -- the system prompt matters, the model is trained
# to emit one bare command and nothing else
llama-cli -m nl2sh-3b-Q4_K_M.gguf -no-cnv --no-display-prompt -n 64 \
  -p "<|im_start|>system
You are a shell command generator. Output exactly one line: a single POSIX/bash command that accomplishes the user's request. No prose, no markdown fences, no explanation.<|im_end|>
<|im_start|>user
find files bigger than 100MB in this folder<|im_end|>
<|im_start|>assistant
"

Greedy decoding (temperature 0) is what the reported numbers use, and it makes the same request return the same command every time.

Safety

This model emits commands that will destroy data if you run them. It is a text generator, not a judge of intent: asked to delete everything, it writes the command that deletes everything.

Related measurement on the 1.5B sibling: on an adversarial prompt set two independent annotators judged 11.0% of outputs (95% CI [6.8%, 17.5%]) to be commands that would destroy or corrupt data the request did not ask to touch; 2.0% on ordinary everyday prompts. An accuracy score is silent about this by construction, since it only asks whether the reference end-state was reached.

A separate known weakness: roughly 14% of outputs on polarity-sensitive requests invert the intent — ls -S for "smallest first", touch -c for "create if missing". These run cleanly and do the opposite of what was asked. Read every command before running it.

The nl2sh CLI ships a denylist that flags common destructive patterns and never auto-runs anything flagged. That is a seatbelt, not a sandbox.

Limitations

• Single-turn. No shell state, no memory of previous commands.
• Cannot see your filesystem, so requests depending on what is on disk ("delete the older backup") may guess wrong.
• Output capped at 64 tokens — a command, not a script.
• Evaluated on one 300-task benchmark, English only.
• Fine-tuned from a single base family.

Evaluation detail

The numbers above were produced with the unmodified upstream scorer; the exact configuration is given with the benchmark table so anyone can reproduce them.

A fuller write-up of the evaluation methodology, the ablations behind the training recipe, and several findings about the benchmark harness itself is being prepared for publication. Until that is through review, this card sticks to what the model is and how it scores, rather than the analysis behind it. The weights, the scorer settings and the task set are all here, so the numbers are checkable in the meantime.

Citation

@software{nl2sh,
  author = {Poudel, Mukesh},
  title  = {nl2sh: local natural-language-to-shell command generation},
  year   = {2026},
  url    = {https://github.com/ThorOdinson246/nl2sh}
}