abacusai/Smaug-Mini

---

1. Model Introduction

Smaug-Mini is an agentic finetune of Qwen3.8-27B, finetuned by Abacus.AI. It improves end-to-end agentic tool use and automation performance (+4.5 on AutomationBench, +17.1 on JobBench, +2.0 overall on LiveBench, +2.5 on IFBench) while holding the base model's general capabilities at parity (GPQA-diamond at the official number, LiveBench coding and agentic-coding at board parity). The weights load exactly like the official release: same layout, same Qwen3_5ForConditionalGeneration architecture with the vision tower intact, same 262,144-token context, same reasoning interface with xhigh / medium / low effort levels.

This card describes the training approach and the evaluation results. Dataset contents are not disclosed; training data consists of multi-turn, tool-using automation episodes generated and verified against executable environments.

2. Model Summary

Architecture	Dense hybrid-attention transformer + vision tower
Total Parameters	27B
Number of Layers	64 (48 linear-attention + 16 full-attention, 3:1 interleave)
Attention Mechanism	Gated linear attention & full attention (GQA)
Hidden Dimension	5120
Number of Attention Heads	24 (4 KV heads)
Vision Encoder	27-layer ViT, patch 16
Vocabulary Size	~248K
Context Length	262,144
Multi-Token Prediction	1-layer MTP head (inherited; leave speculative decoding off)
Precision	bfloat16
Modality	Text, Image
Base Model	Qwen/Qwen3.8-27B
Adaptation	On-policy RL (GRPO), LoRA merged as full delta (language trunk only)

3. Evaluation

Smaug-Mini vs its base

Higher is better. Base points from the Qwen3.8-27B model card and livebench.ai.

	Smaug-Mini	Qwen3.8-27B (base)	Qwen3.6-27B	Qwen3.7-Plus	Opus4.6 Max
Agentic
AutomationBench	41.8	37.3	—	20.4	25.5
JobBench	50.5	33.4	21.8	27.6	36.9
LiveBench agentic coding	60.8	61.4	39.3	—	49.0
NL2Repo-Bench	55.8	42.3	36.2	41.1	47.6
Reasoning, knowledge & instruction following
GPQA-diamond	89.4	89.2	87.8	90.3	91.3
HLE	34.2	30.8	24.0	34.7	40.0
IFBench	82.0	79.5	69.1	79.1	62.5
LiveBench overall	76.9	75.3	64.0	—	74.5
Vision
MMMU-Pro	75.6	76.3	75	80	75

Bold = best score in the row. Shading is relative within each row (darker = higher). Qwen3.6-27B, Qwen3.7-Plus, and Opus4.6 Max scores are as reported on the Qwen3.8-27B model card unless noted; the Opus 4.6 AutomationBench score is from our own harness run and its JobBench score is a reported number, not our run; LiveBench rows use the public livebench.ai leaderboard (its Opus 4.6 entry runs at high effort); the Qwen3.7-Plus AutomationBench score and all MMMU-Pro comparison scores are from Artificial Analysis (its Opus 4.6 MMMU-Pro entry runs at max effort); — = not reported.

LiveBench category profile

Scores 0–100; overall = mean of the seven category averages. Both models' scores are the published livebench.ai leaderboard entries (Smaug-Mini listed under the finetunes filter).

Notes

All Smaug-Mini numbers were produced at temperature 1.0, top_p 0.95, reasoning effort xhigh, with generation budgets large enough that no score is truncation-bound.

Base points: JobBench, IFBench, and GPQA-diamond are the scores published on the Qwen3.8-27B model card; the AutomationBench base score is from our own run of the base model. JobBench was run under the official protocol with the OpenCode scaffold and an LLM judge.

4. Training Approach

Smaug-Mini was trained to make multi-turn tool use and automation episodes more reliable end to end. It is trained with on-policy reinforcement learning (GRPO) over multi-turn, tool-using automation episodes with verified, outcome-based rewards - an episode is rewarded for measurably completing its task in the environment. Training is a LoRA adapter over the language trunk only, merged into the base weights as a full delta; the vision tower and multimodal projection are bitwise-identical to the base release.

One deployment note: the multi-token-prediction (MTP) head is inherited from the base model and was not retrained against the updated trunk. Speculative decoding via MTP should be left off; standard decoding is unaffected.

5. Known Behaviors and Limitations

Smaug-Mini redistributes deliberation rather than adding it: more reasoning per action but fewer actions per episode, finishing passing episodes about three steps sooner at essentially unchanged total reasoning volume. The clearest shift is in failure behavior — episodes that burn the entire step budget without finishing drop from 3.4% to 1.0%. Refusals are zero for both models, and the general-capability benchmarks in §3 show the deliberation shift does not come at the cost of base skills.

6. Deployment

Smaug-Mini serves as a drop-in replacement for Qwen3.8-27B on any stack that supports the base model. Reference vLLM invocation:

vllm serve abacusai/Smaug-Mini \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Recommended sampling for agentic use: temperature 1.0, top_p 0.95, reasoning effort xhigh (the default). Recent vLLM versions return the chain-of-thought in the reasoning field of the message; when replaying conversation history, pass reasoning back under both reasoning and reasoning_content keys for portability across serving versions. Leave MTP-based speculative decoding off (§4).

7. License

Apache 2.0, inherited from Qwen3.8-27B.

8. Citation

@misc{abacusai2026smaugmini,
  title  = {Smaug-Mini},
  author = {Abacus.AI},
  year   = {2026},
  note   = {Agentic fine-tune of Qwen/Qwen3.8-27B},
  url    = {https://huggingface.co/abacusai/Smaug-Mini}
}

The Smaug line and the DPO-Positive method behind it:

@article{pal2024smaug,
  title={Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive},
  author={Pal, Arka and Karkhanis, Deep and Dooley, Samuel and
          Roberts, Manley and Naidu, Siddartha and White, Colin},
  journal={arXiv preprint arXiv:2402.13228},
  year={2024}
}

9. Contact Us

If you have any questions, please reach out at Abacus.AI.