Aptheos/lfm3-vl-2b-click-sequence-sft

LFM3-VL-2B click-sequence SFT adapter

A LoRA adapter that lets LFM3-VL-2B operate a web page from screenshots alone: no accessibility tree, no element IDs. Each turn it sees the page and the instruction, and answers with one click position from 0 to 1000 on each axis. On MiniWoB++ click-button-sequence it has to click the button labelled ONE, then the one labelled TWO, wherever they are on the page.

A replay of reserved task 21001: the recorded clicks, redrawn in the seeded task. The red ring marks where the next click lands. The buttons overlap here, and the adapter clicks the visible part of ONE.

Results

On 100 reserved tasks (seeds 21000–21099), evaluated once, after the protocol, policy hash and decoders were locked:

Policy	Greedy	Sampling, temperature 1, top_p .95	Sampling, white image instead of the screenshot
Base LFM3-VL-2B	0/100	0/100	0/100
This adapter	95/100 (95% CI 89–98%)	21/100 (14–30%)	0/100

• Locked test: sampling was the locked primary test. The adapter beat base on 21 tasks and lost on none (exact sign test, p ≈ 1e-6).
• Greedy: reported alongside, as the decoder a deployment would use.
• White images: 0 of 100 means the clicks depend on the pixels.
• Sampling failures: almost all of them are malformed calls, not wrong clicks. Episodes where every call was valid succeeded 95.5% of the time. The GRPO stage below fixes this.

SFT, then GRPO: sft-then-grpo/

Starting from this adapter, I trained with a simplified GRPO on the task's own reward alone. For each of 120 training tasks it sampled 4 episodes at temperature 1, and weighted every turn of an episode by that episode's reward minus its group's mean. That's one on-policy update per group, with no clipping, KL penalty or standard-deviation scaling; groups with equal rewards were skipped. 39 groups varied. Then a second locked evaluation on 100 fresh tasks (seeds 22000–22099):

Policy	Sampling, temperature 1	Every call valid (sampled)	Greedy	White image
SFT (this adapter)	26/100	30	91/100	0/100
SFT, then GRPO	83/100 (75–89%)	94	92/100	0/100

GRPO against SFT under sampling, the locked test: 57 wins and 0 losses, exact sign test p ≈ 1e-17. It fixed format robustness, so success among valid episodes stays at 87–88%, and it cost nothing on greedy.

Scope

• One task family: one MiniWoB task, at most four clicks.
• Not a computer-use agent: this is evidence that small vision models can learn short visual sequences from a few hundred labelled turns.
• Seed ranges: the development seeds 60–99 were used for the fit decision, and the reserved seeds only for the table above.

Training

• Labels: supervised fine-tuning on 234 expert turns from 117 training tasks (seeds 120–239). The expert clicks the visible point nearest each button's centre, found by the browser's own hit test, and a task is kept only if its own scorer rewards it. Three layouts where ONE is completely covered were excluded.
• Prompt: each example is the exact screenshot and prompt the policy sees at that turn: the instruction plus earlier clicks as text.
• Settings: LoRA r 32, alpha 64, language model only (vision tower frozen). It started from the click-color adapter below. LR 1e-4, linear to zero; 117 updates; FP16 base with float32 adapter; seed 23. One run on a free T4; final checkpoint only.

Files

Path	What
adapter_model.safetensors, adapter_config.json	This adapter, sha256 df9c2e0918f6668b…
sft-then-grpo/	The adapter after GRPO, sha256 b0fe415c6b2e78b0…. Load it the same way with subfolder="sft-then-grpo".
click-color-sft/	The one-click click-color adapter it started from, sha256 1969938677a35eb4…. On 100 reserved click-color tasks it scored 100/100 against base's 30/100, but the base model already chose the right box in 97% of its valid calls, so that gain is mainly the call format.

Loading

Base revision: ba44bd51db20a2753f28c273a1571ccaa210d954 (pinned in adapter_config.json). The numbers were produced with Transformers 5.21.0 and Torch 2.14.1 on a T4 in FP16, with SDPA attention. The base model uses remote code, so review it before running with trust_remote_code=True.

import torch
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

revision = "ba44bd51db20a2753f28c273a1571ccaa210d954"
processor = AutoProcessor.from_pretrained("LiquidAI/LFM3-VL-2B", revision=revision, trust_remote_code=True)
base = AutoModelForImageTextToText.from_pretrained(
    "LiquidAI/LFM3-VL-2B", revision=revision, trust_remote_code=True, dtype=torch.float16, attn_implementation="sdpa"
).to("cuda")
model = PeftModel.from_pretrained(base, "Aptheos/lfm3-vl-2b-click-sequence-sft")

The prompt, the click tool, the 0–1000 coordinate mapping and the strict parser the numbers depend on are in the repository (src/browser_control/lfm3_vision.py, vision_evaluation.py).

Evidence

Every report behind these numbers, the training logs and the locked protocol are in the dataset Aptheos/lfm-browser-control-evidence. The repository's scripts/verify_reported_numbers.py recounts them from the raw episodes.

License

The adapter is derived from LFM3-VL-2B and is distributed under the LFM Open License v1.0 (LICENSE), including its revenue threshold for commercial use.