laya-web-agent

A small, fast model that picks the next step for a browser agent. It's a fine-tune of convaiinnovations/laya (multilingual checkpoint, mmBERT-base, 322M) and answers in a single forward pass, without generating any text.

You give it the goal and the page state: title, visible text, a numbered list of controls with their current values, and the recent actions. It answers two questions:

• operation: CLICK, TYPE_TEXT, SELECT, SCROLL_DOWN, WAIT, DONE or BLOCKED
• target: which numbered control to act on

This version (v3) is trained on synthetic pages plus real websites from Mind2Web. It's reliable on clean, structured forms. On real websites it's still weak, and behind comparable models; see Results.

Usage

import laya
from huggingface_hub import snapshot_download

agent = laya.load(snapshot_download("abedinia/laya-web-agent"))

The input format is the one jev_ultrafast sends (state plus typed questions). rl_agent_config.json has the lengths the model was trained with (max_len 2048, head_max_len 512). Keep both. Earlier uploads had head_max_len 256 in the config while training used 512, so pages with many options behaved differently at inference or raised an error. This upload fixes that.

Results

Real websites: Mind2Web official test splits

Candidates are the top 25 elements from MindAct's DeBERTa ranker, whose scores ship with the dataset. If the ranker drops the correct element, the step counts as a miss, as in the published MindAct numbers. So does the 1% of steps we couldn't convert.

step success	v2 (synthetic only)	v3 (this version)	Quantum08/laya-browser-mind2web (421M)
cross-task	3.2%	24.3%	39.0%
cross-website	3.6%	19.2%	32.9%
cross-domain	3.2%	18.2%	35.4%

v3	element	operation	step	step, macro per task
cross-task	32.0%	85.4%	24.3%	26.5%
cross-website	25.6%	81.9%	19.2%	21.7%
cross-domain	27.5%	82.1%	18.2%	19.6%

The model usually picks the right kind of action on real pages. Its weakness is picking the right element among many similar ones. Two caveats on the comparison:

• Our operation check is a little easier. It only checks CLICK, TYPE or SELECT, while Mind2Web's operation F1 also scores the typed value, which jev fills in with a separate small LLM.
• Competitor numbers are self-reported on its model card.

Synthetic pages

500 test episodes (2,381 decisions) from a seed range training never used, across 17 page types:

	v2	v3
per step	97.7%	97.7%
episodes with every step right	84.3%	82.9%
BLOCKED (impossible tasks)	10/12	12/12

Training data

33,438 decisions:

• 26,234 synthetic decisions from 5,648 episodes of 17 page types, played in real headless Chrome. The page types are forms, signup, address, search, filters, cart, date picker, wizard, autocomplete, radio, navigation, already-done pages and impossible tasks. They include recovery from mistakes, scrolling, loading states and injected instructions in the page text. The correct answer at every step comes from checking the live page, not from a model.
• 7,204 real steps from Mind2Web (training split only), converted to jev's format: the top 25 candidates by MindAct's ranker, in page order, with labels from the cleaned HTML.

Training started from v2 and ran 4 epochs: top 8 encoder layers plus the head, lr 3e-5, effective batch 16, bf16, 2048 tokens. The best holdout epoch was 2. After that the model started memorising Mind2Web.

Limitations

• Real websites: about 1 step in 5 is fully right. That's not enough to complete real multi-step tasks alone. It's best used for simple structured pages, or as a fast first pass with a larger model as the fallback when its confidence is low.
• Some formats are guesses. The original jev data had no scroll or wait steps, radios or textareas, so how they appear in the input is a best guess at jev's format.
• DONE without seeing the confirmation. On long synthetic pages the confirmation can be below the fold, and the label is still DONE after the submit.
• English-first. The base model is multilingual, but all the training data is English.

Versions

tag / branch	what	Mind2Web step (task / website / domain)	synthetic per step
main	v3, synthetic + Mind2Web, 2048 tokens	24.3 / 19.2 / 18.2	97.7%
v2-synthetic-17families	v2, synthetic only, 1536 tokens	3.2 / 3.6 / 3.2	97.7%
v1-synthetic-6families	v1, 6 page types, 1280 tokens	not measured	57.7%

A later experiment (v5) added 40 candidates, richer element labels and 1,480 samples from crawled real pages. It scored higher on synthetic pages (98.4% per step) but lower on Mind2Web (21.2 / 16.0 / 16.8), so it isn't published as main.

Credits

Mind2Web: Deng et al., Mind2Web: Towards a Generalist Agent for the Web, NeurIPS 2023 Datasets and Benchmarks Track. Data and ranker scores under CC BY 4.0. The Mind2Web test set was used only for evaluation and is not redistributed here.