ErenAta00/Maverick-4B-Unity-XR-Agent-1.1

Maverick-4B-Unity-XR-Agent 1.1

Maverick 1.1 is the update to Maverick-4B-Unity-XR-Agent. It adds a second scene format in which the model also sees the commands it has just run, who is speaking, and where people are looking and pointing. With it, 1.1 can undo and correct a command, resolve "the one I pointed at", take a request from a remote expert for an on-site user, and carry out short plans that cross rooms. In either scene format it is also much better at acting on objects in other rooms and at declining requests it cannot meet.

This card covers what changed. The tools, the request loop, the output check, the safety notes and the Unity setup of the 1.0 card still apply.

A technician is on site and a remote expert joins by call. Nine commands in a three-room house, including a correction, a fetch from another room, a two-step plan and an undo that is refused because a button press cannot be reversed. Recorded live on a laptop with a 4 GB GPU, with the Q4_K_M file. The expert's voice is text-to-speech and the model reads its transcript. After every command the scene itself is checked, not the model's reply. The script was rehearsed and the video was recorded five times to fix its graphics and camera; the model got all nine commands right in each of the last four.

What's new

• Undo and corrections. A new tool, undo, reverses the last one to three commands. "No, the other one" becomes an undo plus the same action on the other object. If an effect cannot be undone, such as a button press, the model says so.
• Shared attention. The scene can list what each person is looking at or pointing at, now and a moment ago, and who is speaking. "Highlight what I'm pointing at" and "the one the expert is looking at" resolve against it.
• Other rooms. If the object is only in another room, the model finds it, goes there and acts. If it exists in two rooms, the model asks which one.
• Short plans. Dependent steps in one command: put A in B and B on C, switch off and then move, fetch from another room, carry a full container across rooms.
• Fewer wrong attempts. Impossible requests are declined before any action, and "where is X" is answered in words.

The 1.0 scene format still works and remains the default of the Unity package. The base model, the licence and the 11 tools are unchanged; undo is the twelfth.

Results

The 11 new test cells were sealed on 3 October 2026, before any 1.1 training data existed; the 22 capability cells are those of the 1.0 card. Maverick 1.1 was run once on all of them, on the Q8_0 file, after training ended. Accuracy is in percent. Maverick 1.0 and the untuned Qwen3-4B were run on the same commands.

On the six new-ability cells 1.1 scores 93 to 100%, where 1.0 scores 33 to 87%. The largest gains are in acting on objects in other rooms (97.8% against 33.3%), corrections (100% against 55.3%) and short plans (100% against 71.7%). Declining impossible requests (a new cell of 60 commands) rises from 75.0% to 100%, and scene questions from 78.3% to 100%. Typed requests with typos score 98.3% against 91.7%, with both models at 60 of 60 on the clean versions of the same commands. Arguments (numbers, units, directions) is the one cell where 1.1 is lower, by one command (97.5% against 98.8%).

Corrections, undo, shared attention and short plans need the new scene format, which 1.0 has never seen, so its figures show what the new prompt does with the 1.0 weights, not what 1.0 can do as shipped.

Undo is counted by outcome. In 5 of the 60 commands a button had been pressed, which cannot be undone, and 1.1 answered, for example, "I can't undo that: pressing the button can't be reversed" without calling undo first. The rule we fixed in advance expects the call and its refusal; under it 1.1 scores 55 of 60 (91.7%). We adopted outcome counting on 12 October, after the sealed run, when we reviewed the failed items. The gain over 1.0 is significant under outcome counting (p = 0.008) and not under the original rule (p = 0.58).

Over the 22 capability cells of the 1.0 card (2,900 commands) 1.1 scores 97.3%, against 96.4% for 1.0: a net gain of 27 commands, 18 of them from declining impossible requests (75.0% to 93.0%). Twelve of the 13 groups move by 2 points or less. The weakest cell is now "near", which goes from 88.7% to 86.7% (three of 150 commands).

On the 499 ALFRED-TC sentences of the 1.0 card, 1.1 gets 415 right (83.2%) and 1.0 gets 418 (83.8%); three sentences is not a difference (p = 0.74). On the 60 sentences with object types held out of training, 1.0 gets 55 and 1.1 gets 51 (p = 0.13), too few to call a difference. The other models' scores are those of the 1.0 card, where the untuned Qwen3-4B was measured in its own Transformers run; through the GGUF path used for the other cells it scores 48.3% (241 of 499), so the 57.1% in the chart is the more favourable of the two.

1.1 asks a question instead of acting on none of the 499 sentences (1.0: 0.4%) and answers in text without a call on 5.2% of them (1.0: 3.4%).

New wordings and noise. These are development sets, not sealed, and 1.0 was not run on them. Rewriting the four new-format cells with wordings from a phrase pool that training never used gives 620 commands: 96.9% by outcome counting and 94.5% by the original rule. Adding typos, speech errors and fillers to a wider-wording set of 200 commands gives 98.5% (100% without them). In a live Unity scene, 197 of 200 held-out development commands ran correctly. On 300 human-written ALFRED development sentences (a different set from the 499) 1.1 gets 227 right (75.7%) and 1.0 gets 223; on the 252 whose wording does not overlap with training it is 186 against 183. Neither difference is significant.

Quantisation. Q8_0 (4.28 GB) is the file measured above. Q4_K_M (2.50 GB) was evaluated on development sets only: 221 of the 300 human-written sentences against 227 for Q8_0 (p = 0.15), with a first valid call that is a little faster on the laptop GPU (2.1 s against 2.7 s).

How we measured

• Cells. A cell is a set of test commands for one ability: 22 capability cells (2,900 commands) and 11 new cells (685 commands). The new cells were sealed with recorded hashes before any 1.1 data was generated, and none was edited afterwards. No checkpoint, prompt or setting was chosen on them. The 1.0 figures are those published on its card; we recomputed its ALFRED-TC score from the stored results and it matches.
• Decoding. Greedy, thinking off, output check on. The check regenerates a call that names an unknown object or is malformed, using a grammar built from the scene, and the regenerated call is the one that is scored.
• Prompts. The capability cells and the first seven new cells use the 1.0 prompt and tool list with up to four calls per command. The four cells that need the new scene format use the new prompt and tool list with up to six calls. All models get the same.
• Untuned model. Qwen3-4B without fine-tuning went through the same Q8_0 conversion path (the de-quantised 4-bit base, no adapter) with the same server settings and output check. No prompt was tuned for it, so its figures show what the base model does with these prompts, not the best it could do. It does not write the empty thinking block that the fine-tuned models write, so the harness adds it.
• Counting. The rule fixed in advance is that the final scene must equal the reference scene and that, in the undo refusal items, the model must call undo and be refused. Outcome counting also accepts two cases a person would accept: an irreversible press declined in words without calling undo, and the reference calls plus one extra highlight. Only the untuned model triggers the second case (it lifts shared attention from 20 to 31 and arguments from 15 to 18).
• Small cells. Cells hold 15 to 100 commands, so 60 of 60 still allows a true error rate of up to about 6%. Paired differences use an exact McNemar test on the same commands. The cells come from the same simulator and templates as the training data; the unfamiliar cases are under Limitations.

Usage

Start the server with the settings the tests used:

# Q8_0, on a 4 GB GPU with 20 of the layers on the GPU; raise -ngl if you have more memory
llama-server -hf ErenAta00/Maverick-4B-Unity-XR-Agent-1.1:Q8_0 --jinja -c 8192 --parallel 2 -ngl 20 --host 127.0.0.1 --port 8091

# Q4_K_M, all layers on the GPU
llama-server -hf ErenAta00/Maverick-4B-Unity-XR-Agent-1.1:Q4_K_M --jinja -c 8192 --parallel 2 -ngl 99 --host 127.0.0.1 --port 8091

With the Unity package. Install it from the Package Manager with Add package from git URL:

https://github.com/mcbu-xrlab/Maverick-Unity.git#v1.1.0

To use the new abilities, set Schema Version to V1 on the Scene Agent component; the default is V0, the 1.0 format. The package then sends the new scene fields and system prompt, keeps the command history, marks which commands can be undone and runs undo itself. Its SceneAgentGaze component reports the local user's gaze and pointing; for remote people your networking code calls SceneWorld.ReportAttention and ReleaseAttention. With the 1.0 format and the 1.0 system prompt, 1.1 replaces 1.0 without other changes.

With your own integration. The new format adds optional fields to the scene JSON of 1.0. Send them only when you have them:

scene = {
    "player": {"room": "workshop", "gaze": None, "left_hand": None, "right_hand": None},
    "locations": ["workshop", "office"],
    "visible_objects": [...],                      # as in 1.0
    "people": {"user": "technician", "expert": "remote expert"},   # who is in the scene, by role
    "speaker": "user",                             # who said the request; "I" and "my" mean them
    "actor": "user",                               # whose hands and room the model acts with
    # per person and act ("gaze" or "point"): up to 3 targets in the actor's room from the last 30 seconds,
    # newest first; "active" is true while the person still looks or points at it
    "attention": {"user": {"point": [{"id": "hammer_2", "ago_s": 1, "active": False}]}},
    # the model's own commands of the last 5 minutes, newest first; "undoable" is false after an irreversible
    # action such as a button press, and after a failed command
    "recent_actions": [{"ago_s": 4, "calls": ["move_by(fire_extinguisher_2, forward, 50, cm)"],
                        "status": "done", "undoable": True}],
}

Use the new system_prompt.txt from this repository unchanged. Tool results now say what changed, and why an undo was refused, and the request loop allows up to six calls per command. The model trusts the undoable flag, so set it from what your app can really reverse.

Examples from the development commands, which were not used for training:

"My mistake, select the other fire extinguisher."   (it had just moved extinguisher 2 forward by 50 cm)
    undo()
    move_by(fire_extinguisher_1, forward, 50, cm)
    "Done. I undid that, then moved the fire extinguisher forward by 50 cm."

"Highlight the shelf the technician is currently indicating."   (said by the remote expert)
    highlight([shelf_2])
    "Done. I highlighted the shelf."

"Um, get the camera out of the office and place it on the ottoman."
    teleport(office), grab(camera_1, right), teleport(living_room), place(camera_1, ottoman_1, on)
    "Done. I picked up the purple camera with your right hand, then put the purple camera on the ottoman."
    (the reply does not mention the trip to the office and back)

"Undo that last thing."   (after a button press)
    no call
    "I can't undo that: pressing the button can't be reversed."

Limitations

Beyond those on the 1.0 card:

• Outside its training distribution it acts too eagerly. On 669 deliberately unfamiliar cases (conflicting, stale or empty attention, an occupied hand, implied return trips; 1.0 was not run on them) it scored 41.6%, and on the 198 where the right answer was a question it asked in 15 (7.6%). It tends to act on a plausible object instead of asking.
• Remote speakers can act in the on-site scene. A remote expert's request can move objects, teleport the on-site user or press buttons through the actor's hands. The model does not check permissions; if the remote person should not act without consent, enforce that in your app.
• Speaker and actor. When the two differ, "my hand" is the least stable case: on the 80 unfamiliar speaker cases it scored 24%.
• Elimination. After "not that one either" it sometimes offers an object that was already rejected, or reads the rejection as an undo.
• Hands. If the user names the wrong hand ("drop the cup from my left hand" when the cup is in the right), it can say that the left hand is empty instead of dropping the cup or asking.
• Counted undo. "Back to before the lamp" can undo the wrong number of steps.
• Remote targets and quantities. In an informal trial of 20 commands we wrote for a second Unity apartment whose layout was not in the training data, 15 worked. The five failures show the shapes to expect: a destination in another room ("put the banana on the bed" while standing in the study: it fetches the banana, comes back and puts it on the armchair), a target out of sight after a fetch ("put the pillow in the suitcase", both in the bedroom), "every lamp" across rooms (it reports only the success), "the apples" as a quantity (it treats it as an ambiguity), and an answer such as "the one on the table" to its own question (it does not use it to choose).
• Human-written sentences. On the 300 ALFRED development sentences 1.1 misses 73: 51 are a wrong action, 22 are no call at all. It never asked a question on these, so about one request in four is wrong and nothing in the reply says so.
• Quantisation. Only the Q8_0 file went through the sealed tests.
• Language and hardware. English only, measured on one Windows laptop with a 4 GB GPU, not on headsets, Linux or macOS. For package 1.1.0 we ran the 22 scripted acceptance steps in Unity on Windows (all passed) and the 200 live commands above; the 55-check manual test of 1.0 was not repeated.

Training

Training followed the 1.0 recipe: QLoRA with rank 16 and alpha 32 on all attention and MLP projections, merged into the de-quantised 4-bit base, on one Kaggle T4. It continued from the 1.0 adapter on 24,761 conversations, 63% of them in the new scene format and about 8,000 replaying 1.0-era commands (half of them conversations 1.0 had trained on), for one epoch of 3,096 steps at a learning rate of 5e-5. About 12% of the requests were reworded by a language model (gpt-oss-120b); the target actions always come from the simulator, and every conversation is executed in it. Before training we removed every row that matched a frozen test item, by exact match, a shared run of eight words, or a TF-IDF similarity of 0.90 or more (0.80 to 0.90 was reviewed by hand): 15 conversations and 71 reworded requests.

Citation

Cite as for 1.0, with the version added. The evaluation uses ALFRED (Shridhar et al., CVPR 2020) and builds on Qwen3 (Qwen Team, 2025); the BibTeX entries are on the 1.0 card.

@misc{ata2026maverick11,
  title   = {Maverick-4B-Unity-XR-Agent},
  author  = {Ata, Eren},
  year    = {2026},
  version = {1.1},
  note    = {Hugging Face model, ErenAta00/Maverick-4B-Unity-XR-Agent-1.1}
}

Contact

Developed by Eren Ata at the Extended Reality Laboratory (XRLab), Manisa Celal Bayar University. Questions and bug reports: open a discussion in the Community tab.