Toolcall-2B

A 2B function-calling model for local agent tool routing, fine-tuned from Qwen/Qwen3.5-2B and measured on the Berkeley Function Calling Leaderboard v4 harness.

It is built for the decision an agent makes hundreds of times per task: which tool to call, with which arguments, in what order, and whether to call one at all. Those decisions are cheap in judgment and expensive in tokens when they run through a frontier model.

Results

Measured with the BFCL v4 harness on one server (vLLM, bf16, temperature 0.001, 128k serving context), identical settings for every model. The base model was re-scored under the same conditions rather than quoting published numbers, so these are like-for-like.

Group	Qwen3.5-2B (base)	Toolcall-2B
Overall (22 categories)	33.85	36.35
Mean over the 20 categories not limited by search quota	48.13	59.05
Non-live single-turn	60.08	83.25
Live single-turn	71.13	74.32
Multi-turn	17.75	22.12
Memory	30.97	31.18
Web search	9.50	4.50*

* The free search-API tier allows 250 queries a month and one pass over this category costs about 165, so this figure is not reliable and the 22-category overall understates the model. The 20-category mean is the honest comparison.

Largest per-category gains over the base:

Category	Base	Toolcall-2B
simple_java	15.0	61.0
parallel	49.0	86.0
simple_javascript	24.0	52.0
live_parallel	62.5	87.5
live_parallel_multiple	50.0	75.0
simple_python	70.0	89.5

All four multi-turn categories are above the base: base 27.0 → 31.0, long context 24.0 → 30.5, missing parameter 14.0 → 18.0, missing function 6.0 → 9.0.

Known weakness: the refusal family sits below the base — irrelevance 77.9 → 72.5, live_relevance 87.5 → 81.2. A later round that added more refusal demonstrations fixed part of this but cost more in multi-turn and memory than it gained, so it was not shipped.

Quantized builds

GGUF builds for llama.cpp and Ollama are at ajvikram/toolcall-2b-gguf: Q4_K_M (1.22 GB) runs on a laptop CPU at roughly 33 tokens per second, with Q5_K_M, Q8_0 and f16 also available.

Usage

The model keeps Qwen3.5's native tool-call format, so it drops into any stack that already parses Qwen3.5 tool calls.

vllm serve ajvikram/toolcall-2b --dtype bfloat16 --max-model-len 131072 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string", "description": "City name"}},
            "required": ["city"],
        },
    },
}]
r = client.chat.completions.create(
    model="ajvikram/toolcall-2b",
    messages=[{"role": "user", "content": "What's the weather in Berlin?"}],
    tools=tools,
    temperature=0.0,
)
print(r.choices[0].message.tool_calls)

Emitted calls look like this, which is the base model's own format:

<tool_call>
<function=get_weather>
<parameter=city>
Berlin
</parameter>
</function>
</tool_call>

Thinking is off by default, matching how the model was trained and evaluated.

Training

Supervised fine-tuning, 16-bit LoRA (rank 16, alpha 32, all linear projections), loss on assistant turns only, sequences to 8,192 tokens, learning rate 3e-5, 0.6 epoch, batch 4 with 8-step accumulation. One run of 3.2 hours on a rented RTX A6000; final training loss 0.330, validation loss 0.247.

Prompts are rendered with the model's own chat template at training time and at evaluation time, so the two never diverge.

Data

61,712 examples. Every public source is permissively licensed; a CC-BY-NC set was excluded on purpose so these weights can be Apache-2.0.

Source	Licence	Rows	Contributes
ToolACE	Apache-2.0	13,102	Diverse APIs, some multi-turn
Hermes function-calling v1	Apache-2.0	5,645	Tool responses and follow-ups
Glaive v2	Apache-2.0	4,095	Short multi-turn
xLAM 60k	CC-BY-4.0	10,431	Single and parallel calls
xLAM irrelevance	CC-BY-4.0	1,183	Requests no tool can serve
SmolTalk	Apache-2.0	2,835	General chat, limits forgetting
Verified simulator episodes	generated for this model	2,134 ×3	Stateful multi-turn (below)

Two augmentations, applied deterministically: function masking, which renames tools and parameters to opaque identifiers consistently across the schema and the calls so the model reads descriptions rather than memorising names; and distractor tools drawn from other examples, so tool selection is exercised.

The ingredient that mattered

Four earlier rounds on public data alone raised single-turn accuracy a great deal and made multi-turn worse than the base every time. Public corpora teach the shape of a tool call, not the habit of working a stateful task to completion.

Round five added 2,134 conversations generated against six simulated stateful APIs — a file system, task tracker, mailbox, key-value memory, calendar and shop — where a larger model played the assistant, a second model played a user who revealed the task gradually, and every episode was kept only if a checker confirmed the final state was correct. Episodes were rejected when the goal was not met, when a tool errored more than once, when an unrelated tool was called, or when the model guessed instead of asking for information it had not been given. That single change moved every stateful category above the base at once.

Evaluation, reproduced

# harness: fork of ShishirPatil/gorilla with a handler for Qwen3.5's XML tool-call format
bfcl generate --model ajvikram/toolcall-2b-FC --test-category all_scoring \
  --skip-server-setup --local-model-path <merged-dir> --num-threads 16
bfcl evaluate --model ajvikram/toolcall-2b-FC --test-category all_scoring

Limitations

• Refusal behaviour is below the base model: it is more willing to attempt a request no available tool can serve.
• The web-search score is not reliable, for the quota reason above.
• Numbers here come from one person's harness runs, not from the official leaderboard. They are reproducible with the commands above, and the base model measured on the same rig is the comparison that matters.
• English only, and evaluated only on this benchmark's tool-calling tasks.

Citation

Built on Qwen3.5-2B; evaluated with the Berkeley Function Calling Leaderboard harness. Training data credits are listed above, and the episode generation follows the verified-trajectory approach described in APIGen-MT.