AIMultiple/laya-typed-decisions-browser-ft

Laya typed-decisions, fine-tuned on browser decisions

AIMultiple fine-tuned Laya typed-decisions from Convai Innovations on browser-agent decisions for the decision models benchmark. Neither fine-tune changed Laya's benchmark result by more than chance could explain. We publish both so the result can be checked.

	Stock Laya	First fine-tune	Second fine-tune
Benchmark: browser tasks completed	0/50	1/50	3/50
Held-out tasks Laya can read, completed	0/20	0/20	1/20
Held-out single decisions, same operation as the labeling model	19% (17/88)	75% (66/88)	28% (25/88)
Called done on a page that still needed an action	53	2	5

The second fine-tune is at the repository root and the first is in first-fine-tune/. Stock Laya ran on the same day and hardware as each fine-tune and completed 0/50 both times.

Usage

Tested with laya 0.3.5.

import laya

agent = laya.load("AIMultiple/laya-typed-decisions-browser-ft")  # second fine-tune
first = laya.load("AIMultiple/laya-typed-decisions-browser-ft", subfolder="first-fine-tune")

The model was trained and scored on requests from the jev-ultrafast browser runtime: page text, a numbered list of visible controls, an operation question and target questions.

Results

Each fine-tuned checkpoint and the stock model ran the benchmark's 50 browser tasks on the same day and hardware, one attempt per task, with the same runtime, limits and grader.

• Second fine-tune: 3/50 against 0/50 (exact McNemar, p = 0.25). It ran on an Apple GPU in fp32.
• First fine-tune: 1/50 against 0/50 (p = 1.0). It ran on an H100 in bf16.
• On 20 held-out tasks from a training website, where every page on the labeling model's path fits Laya's input, stock Laya and the first fine-tune completed none and the second fine-tune completed one.
• Single decisions: 103 recorded decisions from 19 held-out tasks. Laya can read 88 of them when sent only the questions each step needs, and 34 in the full request the benchmark sends. On those 34, stock Laya agreed 29% of the time, the first fine-tune 71% and the second 41%. Scrolls are 47 of the 88. Stock Laya answered done on 38 of those scrolls; the first fine-tune learned to scroll; the second fine-tune mostly clicked where the labeling model scrolled. The second fine-tune was scored on an Apple GPU in fp32, the others on an H100 in bf16.

The article explains the failure modes: https://aimultiple.com/decision-models

Training

• Start point: stock typed-decisions at convaiinnovations/laya revision 1c5edc1.
• Labels: GLM-5.3-Flash (MIT license) ran browser tasks on eight websites that are not in the benchmark. Decisions from successful runs became labels. A run counted as successful when the task check passed it; across all tasks, 11 more runs counted because they reached the right product page through site search and failed only the exact-URL check.
• First fine-tune: the 606 training records, skipping items too long for Laya in a shortened form of the request.
• Second fine-tune: only the 292 training records whose full request fits Laya's input, plus 102 records from Laya's own runs. In those runs, GLM-5.3-Flash took over when Laya declared the task done or blocked, or reached 10 actions, and its next decisions became labels.
• Recipe: the producer's notebook at laya commit 573e5b6: proper-scoring policy gradient plus soft cross-entropy, 4 epochs, AdamW (encoder 2.5e-5, head 1e-4), seed 42, then one fitted temperature per question type.
• Our changes to the recipe: one H100 in bf16, and input budgets of 8,192 sequence and 4,096 head tokens (the benchmark's serving settings) instead of 1,024 and 256.
• Per-epoch loss and held-out accuracy are in each checkpoint's training_log.json.

Limitations

• Laya caps each option at 48 tokens, and the benchmark connector rejects a request when any question or option would be truncated. 10 of the second fine-tune's 50 benchmark attempts ended this way.
• Labels come from one model on eight websites, and the training data is not published.
• The training block in rl_agent_config.json is carried over from the stock checkpoint and does not describe this fine-tune.

License and changes

Apache-2.0, the license of the original model. Changes from the original: new weights in model.safetensors; in rl_agent_config.json, the input budgets, refitted per-question temperatures, no per-option-count temperatures, and a new model_name.

Checksums (sha256)

• model.safetensors: 9ec0ce5d545e86a1aa7122ddf0f02988048ab867e6a0825b6973b172456b3b5d
• first-fine-tune/model.safetensors: 0352a618acdc4446177398d565b3f7809d1fe1dab245ea1645a2d5e7638e0651