PursuitOfDataScience/Muse-Glimmer-30B-ToolCall-LoRA

Muse-Glimmer-30B — agentic tool-calling LoRA

A LoRA adapter that makes Muse-Glimmer-30B reliably commit to a tool call.

Why this finetune is needed

Muse-Glimmer-30B is sold as an agentic model, and it is not short of knowledge about the tools you hand it. Probe the stock model on held-out agentic trajectories and it will name the correct function verbatim in its own reasoning — and then keep deliberating past the point where an agent loop needed a call. Here is the stock model, mid-thought, on a task whose gold action is databutton-submit_app_requirements:

"We need to submit app requirements via databutton-submit_app_requirements. Need name, pitch, sp…"

Correct tool, correctly named, never emitted. The generation budget ran out first. That is not a knowledge gap, it is a decisiveness and format gap, and it is the single thing standing between this model and a working agent loop. On 200 held-out examples the stock model produced a well-formed call only 98.3% of the time when one was required.

Supervised finetuning on real trajectories is the direct fix, and it costs 191.7M trainable parameters — 0.64% of the model.

Results

Toucan held-out (in-distribution), n=200	stock	+ adapter	change
named the right tool anywhere in its output	92.4%	91.6%	-0.8pp (no significant change)
emitted a well-formed call when one was needed	98.3%	90.8%	↓ -7.6pp worse
tool-name accuracy	85.7%	88.2%	+2.5pp (no significant change)
argument exact match	61.3%	72.3%	↑ +10.9pp better
schema valid (name exists, required args present)	97.5%	90.8%	↓ -6.7pp worse
abstained when no tool applies	85.2%	100.0%	↑ +14.8pp better
hit the generation ceiling mid-answer	1.7%	1.7%	— +0.0pp

BFCL v3 (not trained on), n=120/subset	stock	+ adapter	change
overall AST accuracy	81.2%	85.2%	↑ +4.0pp better
live irrelevance	77.5%	83.3%	↑ +5.8pp better
live multiple	78.3%	78.3%	— +0.0pp
live simple	72.5%	76.7%	↑ +4.2pp better
multiple	81.7%	90.0%	↑ +8.3pp better
simple	92.5%	95.8%	↑ +3.3pp better

BFCL v3 is the honest test here. Every Toucan trajectory was generated by MiniMax-M2.5, so gains against Toucan gold are partly gains at imitating that model. BFCL was never trained on.

Does it break anything?

A tool-calling finetune damages a general model in one characteristic way: the model starts emitting call syntax when no tools were offered at all. That failure is invisible to every metric above, so it is measured directly — GSM8K prompts with tools=None, counting any <atem:invoke in the output.

GSM8K, no tools offered, n=200	stock	+ adapter	change
accuracy	74.0%	88.0%	↑ +14.0pp better
tool-syntax intrusion	0.0%	0.0%	— +0.0pp
MMLU (n=250)	79.6%	81.6%	↑ +2.0pp better

Paired significance on the GSM8K delta — McNemar: 32 fixed by the adapter, 4 broken by it, exact two-sided p = 0.0000 -- significant at 0.05.

Usage

from transformers import AutoModelForImageTextToText, AutoTokenizer
from peft import PeftModel

# transformers >= 5.15 is required: `muse_glimmer` is not a registered model
# type before that, and the checkpoint carries no remote code.
base = AutoModelForImageTextToText.from_pretrained(
    "meta-models/Muse-Glimmer-30B", dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "PursuitOfDataScience/Muse-Glimmer-30B-ToolCall-LoRA")
tok = AutoTokenizer.from_pretrained("meta-models/Muse-Glimmer-30B")

msgs = [{"role": "user", "content": "What's the weather in Chicago?"}]
tools = [{"type": "function", "function": {
    "name": "get_weather",
    "description": "Current weather for a city.",
    "parameters": {"type": "object",
                   "properties": {"city": {"type": "string"}},
                   "required": ["city"]}}}]
ids = tok.apply_chat_template(msgs, tools=tools, add_generation_prompt=True,
                              return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0, ids.shape[1]:]))

The model answers in Muse-Glimmer's native ATEM syntax, on the to= recipient channel — the adapter does not change the interface, only the reliability of using it.

Training

Data	44,800 examples: 40,000 Toucan-1.5M trajectories + 4,800 manufactured abstention hard negatives
Steps	700 x 24 examples/step
Hardware	3x H100 NVL 94GB, 669 GPU-minutes over 14 one-hour slices
LoRA	r=32, alpha=64, dropout=0.05, text tower only
Targets	`model.language_model.layers.\d+.(self_attn.(q
LR	0.0001 cosine to 1e-05, warmup 40, grad-clip 1.0
Sequence	max 4096 tokens, micro-batches capped at 6144 padded tokens
Loss	assistant spans only, masked by re-rendering each message prefix

The perception encoder and vision adapter are frozen. This corpus is text; adapting a ViT on text trajectories damages the multimodal path for nothing.

Data preparation

Three defects in the raw corpus are filtered, because each teaches the opposite of the goal:

• 7.4% of records leak Toucan's own <tool_call>{...}</tool_call> JSON into assistant prose — a third syntax this model's template never uses. Lifted into structured calls.
• 4.2% of gold calls name a tool that was never offered; 114 are literally named unknown. Dropped — supervising those teaches exactly the hallucination the finetune is meant to remove.
• thinking_cot interleaves the agent's planning with [Tool Simulation — …] blocks, which are the data generator inventing tool results. Only the agent passes are kept; an agent must never be trained to hallucinate observations.

Validation is deduplicated against training by normalised opening prompt. Toucan's subsets overlap by construction — single-turn-diversify rewrites the same seeds as single-turn-original — so distinct uuids are not distinct problems, and a naive split leaks 4.6% of validation.

The abstention hard negatives are manufactured from training data at zero labelling cost: take a trajectory, delete the tool it needs, keep the question, and expect an honest refusal. Over-calling is the failure that actually hurts a deployed agent, and Toucan's own irrelevant subset is small.

Provenance and licensing

• Base model Muse-Glimmer-30B, Apache-2.0. Meta also publishes a USAGE_POLICY.md acceptable-use policy which travels with any derivative.
• Training data Toucan-1.5M, Apache-2.0.
• Toucan is synthetic: every trajectory used here was generated by MiniMax-M2.5. The behaviour this adapter installs is distilled from that model's outputs and inherits its habits. BFCL is reported precisely because it is not from that distribution.

Limitations

• Trained for 700 steps on 16,800 examples. Validation loss was still falling when the step budget ran out (0.5038 at step 150 -> 0.4753 at step 700), so this is not a converged finetune.
• The tool-calling evaluation scores the first call of a trajectory. It does not measure multi-turn task completion.
• Muse-Glimmer is multimodal. The vision tower was frozen, but the language tower was adapted and image understanding was not re-measured. Treat the multimodal path as untested after this adapter.
• Single seed. Small differences between arms are not meaningful; the paired test above is reported for exactly that reason.
• Sampling error is roughly ±6.9 points per cell at these n.