Borcherding/laya-multilingual-calculator-router

Laya multilingual calculator router

A Laya decision model that answers one question before each agent step: does this step need a calculation? It also scores how much working the calculation takes (none, trivial, real working).

Fine-tuned from convaiinnovations/laya (the multilingual checkpoint) on Borcherding/calculator-routing with Unsloth's FastDecisionModel.

Uploaded finetuned model

• Developed by: Borcherding
• License: apache-2.0
• Finetuned from model: convaiinnovations/laya (subfolder multilingual)

Usage

from laya.agent import Agent

agent = Agent("Borcherding/laya-multilingual-calculator-router", device = "cuda")
questions = {
    "needs_calculation": {
        "type": "noul",
        "instructions": "Does the agent's next step need a calculation: producing a number by computing it (arithmetic, percentages, unit or currency conversion, date math, statistics, or code that crunches numbers)? Looking a number up, quoting one already given, or numbers used as names or IDs do not count.",
    },
    "difficulty": {
        "type": "score",
        "instructions": "How much calculation does the next step need?",
        "criteria": ["none", "trivial, done in your head", "real working: several steps, big or awkward numbers, or a formula"],
    },
}
state = {
    "task": "You are a travel planning agent.",
    "tools": ["calculator", "web_search", "code"],
    "thread": [
        {"role": "tool", "name": "web_search", "text": "option 1: €142.1 per night in Porto; option 2: €98.4 per night in Porto"},
        {"role": "user", "text": "Book the cheapest for 4 nights and tell me the total."},
    ],
}
answers = agent.system_one(state, questions)["answers"]
print(answers["needs_calculation"]["noul"])  # probability the step needs a calculation
print(answers["difficulty"]["score"])        # 0 to 2

Ask the questions with the exact wording above; that is what the model was trained on.

Results

Held-out test split (1,200 rows, 600 per class). Test uses templates, names and entities that never appear in train, so it measures new phrasings of familiar kinds of step.

needs_calculation	base Laya	this model
overall accuracy	0.52	0.89
steps that need a calculation (600)	0.74	0.84
steps that don't (600)	0.31	0.95

Base Laya says "yes" to almost anything with a number in it. The tuned model fixes that: order IDs, versions, quoting a total a tool already returned, and writing or explaining money and math topics all go to "no" (1.0 on identifiers, handoff_nocalc, writing, explain, code_no_numbers).

Where it fails. Every calculation it misses comes from one held-out phrasing per family, where the conversion factor isn't in the message:

family	test rows	base	tuned	the held-out phrasing
tech	44	0.89	0.00	How much VRAM do the weights of a 3B model take at 8-bit?
estimate	54	0.46	0.43	About how many pages is a 297k word book?
conversion	47	0.72	0.53	How many cups is 296.7 ml?
lookup (no calc)	51	0.47	0.75	How much does a PS5 Slim cost right now?

It reads "how much / how many X is Y" with an implicit factor as a lookup. If your agent sees that kind of step, treat the "no" side with care, or add those phrasings and fine-tune again.

Measured 2026-10-07 on an AMD Radeon RX 6500 XT (Windows 11, ROCm 7.14, torch 2.11.0+rocm7.14), through the vendored laya.agent.Agent that Unsloth Studio's decision API serves. Both questions together: accuracy 0.32 base, 0.88 tuned (FastDecisionModel.evaluate).

Training

data	6,000 train rows, 12,000 decisions
LoRA	r 64, alpha 64, dropout 0, merged into the encoder after training
steps	750 (2 epochs), batch 8 x grad accum 2, lr 8e-4 cosine, AdamW, seed 3407
time	585 s, 0.8 s/step, 1.59 GB peak VRAM on the RX 6500 XT
loss	0.84 first step, 0.30 last

After training, FastDecisionModel.calibrate fitted the probability temperatures on the test split. That doesn't change any accuracy above, but it means the test ECE (0.12 to 0.14) is not a clean held-out number.

train.py in this repo is the exact script: python train.py <dataset_dir> 750 <out_dir>.

This Laya model was trained with Unsloth.