Bonsai 2 for computer use on a Mac (Clicky)

Created using Bonsai by Prism ML.

These are tuned versions of PrismML's Ternary Bonsai 2 9B and 4B for operating a Mac: finding a control on a screenshot, and carrying a task out one step at a time. They were made on 14 and 15 November 2026 for a hackathon entry called Clicky, a menu-bar assistant that runs entirely on a MacBook.

The weights stay ternary. The files keep PrismML's format, size and speed; the only additions are two small pointing decoders in full precision.

Code, notes and every measurement: https://github.com/aboufama/hackathon-clicky

What is here

File Size What it is
Bonsai-2-9B-planpoint-v1-u1500-Q2_0.gguf 2.56 GB 9B planner, the model inside Clicky. A complete model file for LM Studio or llama.cpp. It plans a task step by step, points, and writes tool calls.
patches/bonsai2-9b-planpoint-v1-u1500.patch.safetensors 245 MB The same 9B planner as a patch.
patches/bonsai2-9b-joint-nb-l7-best-trunk.patch.safetensors and decoders/bonsai2-9b-joint-nb-l7-best-dec2x768-L20.safetensors 272 MB and 111 MB 9B pointer. A pair: they only work together.
patches/bonsai2-4b-joint-h1-trunk-s12500.patch.safetensors and decoders/bonsai2-4b-joint-h1-dec2x768-L20-s12500.safetensors 129 MB and 101 MB 4B pointer. A pair: they only work together.
patches/bonsai2-9b-point-v1.patch.safetensors 221 MB Multipliers-only repair of the 9B. The first and smallest change; see below.

A patch holds only the tensors that differ from PrismML's original MLX 2-bit pack (prism-ml/Ternary-Bonsai-2-9B-mlx-2bit or prism-ml/Ternary-Bonsai-2-4B-mlx-2bit). It is applied to that pack in memory.

Compared with the stock models

On every computer-use measure that was run, the tuned models are far ahead of Ternary Bonsai 2 as shipped, at the same size and speed. They were not shown to be better in general, because that was not measured.

"As shipped" is PrismML's original model, measured with the same harness. One look is a single answer; a second look enlarges the area around the first answer and asks again.

Measure Bonsai 2 as shipped This repository Qwen3.5-9B parent, full precision
Long tasks, 9B planner (120 simulator episodes on tasks never trained on) 1 of 120 27 of 120 9 of 120
Pointing, ScreenSpot-v2, all 1,272 cases, 9B planner (written answer) 43.3% 84.0% 91.7%
The same, 9B pointer pair 43.3% 85.5% with one look, 86.4% with two 91.7%
The same, 4B pointer pair 12.2% 82.4% with one look, 85.2% with two
Pointing, macOS app windows, 2,106 cases, 9B pointer pair 13.2% 46.8% with one look, 57.5% with two 63.8%
The same, 4B pointer pair 3.7% 46.4% with one look, 55.3% with two
Guided multi-step tasks, 75 tasks, 4B with its pointing decoder (an earlier checkpoint, see below) 3% 71%

Limits to keep in mind

  • Pointing is still behind the full-precision parent in every row that has one, and further behind UI-Venus-2-9B (95.8% on ScreenSpot-v2, 68.4% on macOS app windows).
  • The long-task result is from a simulator. The planner beats the parent there (27 against 9), but the planner was trained on other tasks of the same simulator and the parent was never trained on this format. 93 of 120 episodes still fail.
  • General skills were only spot-checked. A 50-question check held steady (see "General skills and tool calls"). PrismML's own general benchmarks were not rerun on the tuned models.
  • Tool calling has no like-for-like comparison with the stock model. The numbers that exist are below; they are not a claim of a gain.

Benchmarks

All pointing numbers use greedy decoding with thinking off and the screenshot's long side at 1280 px, which is what the assistant captures. A hit means the point lies inside the target's box. Brackets are 95% bootstrap intervals over cases.

Pointing

Model ScreenSpot-v2 (1,272 cases) macOS app windows (2,106 cases) Small controls with close neighbours (600 of the macOS cases)
Ternary Bonsai 2 9B, as shipped 43.3 (40.6โ€“46.0) 13.2 (11.7โ€“14.6) 14.2 (11.5โ€“17.0)
9B planner, written answer 84.0 (81.9โ€“85.9) 44.7 (42.6โ€“46.8) 51.3 (47.5โ€“55.3)
9B pointer pair, one look 85.5 (83.5โ€“87.3) 46.8 (44.6โ€“48.8) 54.7 (50.7โ€“58.7)
9B pointer pair, two looks 86.4 (84.5โ€“88.3) 57.5 (55.3โ€“59.5) 65.7 (61.8โ€“69.5)
Ternary Bonsai 2 4B, as shipped 12.2 (10.5โ€“14.0) 3.7 (2.9โ€“4.5) not measured
4B pointer pair, one look 82.4 (80.3โ€“84.4) 46.4 (44.2โ€“48.5) 50.8 (46.8โ€“54.8)
4B pointer pair, two looks 85.2 (83.2โ€“87.2) 55.3 (53.1โ€“57.4) 61.5 (57.5โ€“65.5)
Qwen3.5-9B, the 9B's parent, full precision 91.7 (90.2โ€“93.2) 63.8 (61.8โ€“65.9) 65.3 (61.5โ€“69.2)
UI-Venus-2-9B, full precision 95.8 (94.7โ€“96.9) 68.4 (66.4โ€“70.3) 72.7
Qwen3.5-4B, 4-bit 89.9 (88.2โ€“91.4) 58.8 (56.7โ€“60.9) not measured
  • Near misses. Counting an answer on the target or within 20 px of its box, ScreenSpot-v2 gives 90.6% for the planner, 90.0% for the 9B pair and 88.1% for the 4B pair (one look).
  • The macOS set is Screen2AX-Task restricted to 27 apps that no training run saw. Its labels are noisy: of 150 labels checked by eye, 114 were right, 17 wrong and 19 unclear, which lowers every model's score.
  • ScreenSpot-v2 was never trained on. One eighth of it was used to choose between training runs, and eight desktop cases share their screens with public macOS training data. For an earlier checkpoint, leaving both out moved the score by under a point.
  • The second look costs a second read of the screen. It adds 9 to 11 points on the macOS sets and 1 to 3 on ScreenSpot-v2.

Long tasks (9B planner)

The model gets a request and a numbered list of the front window's controls, and chooses every step itself. An episode counts when the app's end state is right and the model said done. The simulator has ten Mac-style apps; three apps and 14 task templates were held out of training. Three seeds, 120 episodes a model.

Model New tasks in apps trained on (42) Apps never trained on (78) All (120)
Ternary Bonsai 2 9B, as shipped 1
Qwen3.5-9B parent 4 (95% interval 4% to 22%) 5 (3% to 14%) 9
The planner's starting checkpoint, before training on simulator tasks 1 2 3
9B planner 12 (17% to 44%) 15 (12% to 29%) 27
9B planner inside the Clicky app's rules 34

How the planner's 93 failed episodes ended: 37 said done with a wrong result, 33 repeated a step that had no effect, 20 never said done, 3 ran out of steps. Further training rounds gave 24 to 28 of 120, which is no clear gain.

Guided tasks (4B with its pointing decoder)

Each step is given in words and the model only points. 75 tasks in the same simulator, run in MLX on a Mac mini (M6, 16 GB).

These rows were measured with the checkpoint at step 11,000 of the 4B run. The pair published here is step 12,500 of the same run, which was not rerun on these tasks; on a 158-screen slice of ScreenSpot-v2 it points better (83.5% against 79.7%).

Model Input Aids Tasks done
Ternary Bonsai 2 4B, as shipped, written answers 1280 px none 2 of 75 (3%)
4B with pointing decoder 1280 px none 45 of 75 (60%)
4B with pointing decoder 1600 px none 53 of 75 (71%, interval 60% to 81%)
4B with pointing decoder 1600 px click snapped to an element, second look when nothing changed 55 of 75 (73%)

For reference, in separate runs of the guided tasks the full-precision 9B parent finished 92% and UI-Venus-2-9B 100%.

The multipliers-only patch (bonsai2-9b-point-v1)

This patch changes the multipliers of 176 matrices and nothing else. The score is the probability that every digit of the written answer is acceptable, on two separate sets of 159 ScreenSpot-v2 screens.

Model First set Second set
Ternary Bonsai 2 9B, as shipped 0.327 0.372
With bonsai2-9b-point-v1 0.698 0.704
Qwen3.5-9B parent 0.848 0.843
UI-Venus-2-9B 0.932 0.939

With generated answers on the second set it hits 65.4% with one look and 74.2% with a second look.

General skills and tool calls

  • General skills, spot check. 50 short questions in seven skills; the score is the share of questions on which every answer token is the model's first choice. The 9B as shipped scores 0.86 and the 9B planner 0.92. On 50 questions that difference is noise; it shows no loss, not a gain.
  • Tool calls, strict check. On 300 tool-call cases of the project's own benchmark, the planner gets every token of the expected call right on 80.0%. A pointing-only checkpoint scored 42.0% on the same check before any tool-call training. The stock model was not measured with this check.
  • Tool calls, generated answers. The stock 9B scores 84.8% on the project's 400-case tool benchmark (its parent 89.0%). The tuned models were not run on it.
  • Not run on the tuned models: PrismML's general benchmarks, BFCL, ScreenSpot-Pro and OSWorld-G.

Speed

The tuning does not change the model's size or speed. Measured figures:

Where What Time
Mac mini (M6, 16 GB), MLX One pointing answer at 1280 px 2.46 s with the 9B, 1.43 s with the 4B
Mac mini (M6, 16 GB), MLX A further request about a screenshot already read, through the 4B decoder 76 ms
MacBook Air M4 (16 GB), LM Studio One planner step from a list of controls 4 to 10 s
DGX Spark, llama.cpp One planner step 0.6 s (about 85 tokens a second generated)

How it was done

  • Multipliers retuned, ternary values kept. A Bonsai layer is a grid of -1, 0 and +1 with one multiplier for every group of 128 weights. Comparing Bonsai with its parent layer by layer showed that the loss of pointing sits in layers 4 to 19. Retraining only the multipliers recovers most of it.
  • bonsai2-9b-point-v1 shows how small that change is. No ternary value changed, and each multiplier moved by 0.7% on average. The later patches also train the small full-precision parts of the layers. The 4B patch changes no ternary value; the planner patch and the 9B pointer patch each change 12 ternary values in 2 matrices.
  • A pointing decoder after layer 20. The decoder has two layers and about 25 million parameters. It reads the model's state after layer 20 of 32, scores every image cell against the request, and returns a point without writing any token. The last 12 layers do not run for pointing.
  • A planner. The 9B was trained to carry out a task from a list of the front window's controls: each turn it gets the request, the steps so far and the numbered controls, and answers with one tool call.

What was tried and did not help is in docs/TRIALS.md of the code repository: flipping ternary values, transplanting single layers from the parent, pulling internal states toward a teacher, side paths of low rank, and dropping blank image areas.

Using the files

The GGUF. Load it in LM Studio or in PrismML's llama.cpp fork, with thinking off and temperature 0. For screenshots it needs the vision file Ternary-Bonsai-2-9B-mmproj-BF16.gguf from prism-ml/Ternary-Bonsai-2-9B-gguf; this work did not change the vision weights, so that file is used as it is.

The planner expects this system text and one tool named next_step:

You operate a Mac for the user, one step at a time. Reply with exactly one next_step call. Prefer open_app, open_url, key and menu over clicking. To press or fill a control from the list, give its id. Use look only when the control you need is not in the list. When the task is finished, call done with one sentence for the user.

The user message has the lines Task, Apps, Front app | Window, Done so far and Controls. The tool's action is one of open_app, open_url, click, double_click, right_click, type, key, menu, scroll, drag, look, done. The exact tool definition is in weightlab/planner.py of the code repository, and the Swift app in assistant/ shows it in use.

For pointing in writing, the model answers [x, y] on a 0 to 1000 scale to the Qwen3.5 family's grounding prompt, with the screenshot's long side at 1280 px.

The patches and decoders. These need the code repository:

  • PyTorch: weightlab/state.py (apply_patch) and weightlab/head.py (the decoder).
  • MLX on a Mac: playground/server.py (get_model) and taskenv/mlx_head.py.

The decoders do not run in LM Studio or llama.cpp.

Training data and teachers

Training used screens drawn by the repository's own generator and public macOS screenshot sets (listed in docs/BENCHMARK.md, section "Contamination", and docs/MAC_DATA.md). Qwen3.5-9B and UI-Venus-2-9B served as references and teachers, and some training requests were written by Claude Fable.

The planner was tested on real macOS apps only inside the Clicky app, where most of the gain came from the app's own rules, not from the model.

Licence and credits

Apache-2.0, the licence of the models this is built from. LICENSE and NOTICE.txt carry PrismML's notices.

Downloads last month
72
MLX
Hardware compatibility
Log In to add your hardware

Quantized

GGUF
Model size
9B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

2-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for aboufa/Bonsai-2-Clicky

Finetuned
Qwen/Qwen3.5-4B
Quantized
(3)
this model