adriandj3/TernaryPilot-v1-Computer-Browser-Use-LoRA

TernaryPilot v1: Computer & Browser Use LoRA

A computer use and browser use LoRA for Ternary-Bonsai-2-9B, the 9B model with ternary weights that fits in 2.7 GB. It looks at a screenshot and tells you where to click, what to do next, and what is on the screen. The whole thing runs, and was trained, on a single GTX 1080 with 8 GB.

By Adrian Ciprian Iancu · GitHub · Hugging Face · LinkedIn

This is version 1. It is the first public step of a longer project: an assistant that can use a Windows PC and a browser for you, fully local, on old hardware. v2 is already in the works (more data, longer training, real end to end tasks).

What it can do

One adapter, four skills:

Skill	You give it	It answers
Click grounding	a screenshot and a target ("save file", "the Donate link")	{"x": 512, "y": 233} on a 0-1000 grid
Next action	a task, the actions done so far, the current screen	a short thought and one action: click(x=, y=), write("..."), press("enter"), hotkey("ctrl+s"), scroll(-3), done() ...
Multi step tasks	the same, step after step	the next step, keeping track of the history
Screen reading	a screenshot	which app or site is open, its state and the important text

It works on Windows desktop apps and on web pages.

Results

Everything below was measured by me on a GTX 1080 with the PrismML llama.cpp fork, greedy decoding, images at 1024 tokens. None of the evaluation screenshots were used in training (they were also filtered out of the training pool by perceptual hash).

Click grounding

Benchmark	Base model	My earlier grounding LoRA	TernaryPilot v1
ScreenSpot-v2, 200 case stratified subset	80 / 200	153 / 200	156 / 200
Real Windows apps in Italian, 37 targets with UI Automation boxes	2 / 37	13 / 37	15 / 37
Real web pages, 210 targets with DOM boxes	28 / 210	99 / 210	101 / 210

ScreenSpot-v2 subset by type (v1): desktop icon 31/40, desktop text 42/53, web icon 30/50, web text 53/57.

Agent actions on 145 held out steps (85 Windows steps from AgentNet tasks never seen in training, 60 web steps from Mind2Web). A step counts as right when the action type matches and, for actions with coordinates, the point is within 40 units (on the 0-1000 grid) of the true one.

	Base model	Earlier grounding LoRA	TernaryPilot v1
Right actions	7 / 145	28 / 145	67 / 145
Windows	7 / 85	15 / 85	41 / 85
Web	0 / 60	13 / 60	26 / 60
Answers in the expected format	131 / 145	93 / 145	145 / 145

It does not break the rest of the model when used the way I recommend below (adapter scale 0 outside of GUI requests): 12 out of 12 chat prompts give the exact same greedy output as the base model.

How to use it

You need the PrismML fork of llama.cpp (it has the ternary kernels), the base model and the vision projector from prism-ml/Ternary-Bonsai-2-9B-gguf, and the adapter from this repo.

I tested it with Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf and Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf.

llama-server -m Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf \
  --mmproj Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf \
  --lora-scaled TernaryPilot-v1-Ternary-Bonsai-2-9B-LoRA-F16.gguf:0 \
  --image-max-tokens 1024 -ngl 999 -fa on --jinja -c 16384 --parallel 2

The adapter is loaded at scale 0, so normal chat stays exactly the base model. Turn it on only for GUI requests with "lora": [{"id": 0, "scale": 1}] in the request body. Two tips from my tests:

• send GUI requests to their own slot ("id_slot": 1): switching the adapter on a slot clears that slot's cache, so your long chat context would be processed again after every click;
• use "temperature": 0 and "presence_penalty": 0 for clicks (a presence penalty of 1.0 cost me 4 clicks out of 200).

On Windows --lora-scaled splits on :, so a path like D:\models\x.gguf:0 breaks: start the server from the adapter's folder and pass the file name only.

Click prompt (send it after the image, the target goes at the end):

Localize an element on the GUI image according to the provided target and output a click position.
 * You must output a valid JSON following the format: {"x": int, "y": int}, with x and y normalized to 0-1000
 Your target is:
save the document

Action prompt:

You are operating a Windows computer.
Task: Open Notepad and write a shopping list with milk and bread
Previous actions:
1. Click on the Start button
2. Type "notepad" in the search box
Look at the current screen and give the next action. Coordinates are integers normalized to 0-1000.
Available actions: click(x, y), double_click(x, y), right_click(x, y), move(x, y), drag(x, y), write(text), press(key), hotkey(keys), scroll(amount), wait(), done()
Reply as:
Thought: <short reasoning>
Action: <one action>

For the browser use You are operating a web browser. as the first line. Previous actions are plain text, so you can keep the history without sending old screenshots.

Screen reading prompt: Describe the current screen: the application or website, its state, and the important visible text.

Coordinates are on a 0-1000 grid over the image you send: multiply by width/1000 and height/1000 to get pixels.

Why a LoRA is needed

The ternary weights cost most of the pixel-click skill: on the same 38 cases the full-precision Qwen3.5-9B clicks 36 right and Ternary Bonsai 19. The tests are in docs/ternary-gui-grounding.md.

How I trained it

The full training kit (model rebuild, CUDA kernel, trainer, data builder) and a longer write-up with every speed and quality step are on GitHub: bonsai2-ternary-pascal/lora-training.

• Hardware: one NVIDIA GTX 1080 (Pascal, 8 GB, fp32 only, no bf16, no flash attention, no Triton).
• Method: I rebuilt Ternary-Bonsai-2-9B exactly in PyTorch from the PQ2_0 GGUF (2 bit codes plus the Hadamard rotation of the PrismML format), checked against llama.cpp, and trained a LoRA on top of the frozen ternary weights. The adapter is exported in the GGUF LoRA format that the fork loads with --lora.
• Speed work: a small CUDA kernel of mine that unpacks the ternary weights in one pass, the 4096 Hadamard rotation done as two 64x64 factors, and one shared image prefix for several targets of the same screenshot. Training went from 20.2 to about 6 to 9 seconds per example on this card, with gradients checked against the original code at every step.
• Run: rank 16, alpha 32, ten linear modules in all 32 layers, AdamW lr 1e-4 with warmup, constant phase and a final decay, 5.5 hours, about 2,100 examples seen. It starts from my earlier grounding LoRA for the same model.
• Data (4,251 examples on 2,079 screenshots, English prompts):

Part	Examples	Source	License
Windows actions, multi step	900	AgentNet (OpenCUA), completed Windows tasks with alignment score 8 or more	MIT
Windows clicks from the same steps	426	AgentNet	MIT
Screen reading	305	AgentNet observations	MIT
Web actions, multi step	967	Multimodal-Mind2Web, page cropped to a 1280x800 window around the target	OpenRAIL
Web clicks from the same steps	819	Multimodal-Mind2Web	OpenRAIL
Desktop and web clicks	834	ShowUI-desktop, Wave-UI, GroundCUA	see each dataset

Limitations

• It is a v1. The run saw about half of the dataset once.
• The action test is still my own: the held out steps come from the same two sources as the training data. I have not yet measured full tasks end to end inside an agent.
• Dense web pages are still hard: about 48% on my 210 web targets, with some pages much lower. Desktop icons improved (31/40 on the ScreenSpot subset) but are not solved.
• Speed and languages: while the adapter is active decoding is about 30% slower (measured on my earlier adapter with the same rank). Prompts are English; Italian screens work but I did not tune for them.
• No safety layer: it predicts actions, it does not check whether an action is a good idea. Keep a human in the loop for anything that deletes, buys or sends.
• Responsible use: part of the training data comes from Multimodal-Mind2Web under an OpenRAIL license. Please use this adapter in line with its responsible use terms.

Citation

@misc{iancu2026ternarypilot,
  author       = {Iancu, Adrian Ciprian},
  title        = {TernaryPilot v1: a computer use and browser use LoRA for Ternary-Bonsai-2-9B},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/adriandj3/TernaryPilot-v1-Computer-Browser-Use-LoRA}}
}

Thanks

PrismML for Ternary Bonsai and the llama.cpp fork, the OpenCUA team for AgentNet, OSU NLP for Mind2Web, Show Lab for ShowUI, AgentSea for Wave-UI and ServiceNow for GroundCUA.

If you try it, I would love to hear how it goes: open a discussion here or find me on GitHub.

Keywords: computer use, browser use, GUI agent, web agent, GUI grounding, screen understanding, LoRA, GGUF, llama.cpp, ternary, 1.58-bit, Bonsai, Qwen3.5, ScreenSpot, AgentNet, Mind2Web, Windows automation, local AI, low VRAM, GTX 1080