Meccanoid vision + microphone controller (TokFormer-S, checkpoint s3-r2-q2)
Verdict in plain words: this model is trained and it really drives the simulated robot, but on held-out maps it only ties a properly tuned simple symbolic baseline. It does not yet beat it. On the pre-registered primary test (TEST-T1, 500 unseen episodes, strict rule) it reaches the goal in 15.4 % of episodes against 15.2 % for the wall-aware symbolic baseline SymVision-W (McNemar p = 1.0). Without its microphone it is worse than that baseline, and it is also worse with moving people (T2). A second training round is in progress; this card describes the finished first build only.
A small recurrent token transformer that drives a simulated Meccanoid G15-style two-wheeled robot to a green goal post, using only what its chest camera sees and its microphone hears. It never receives a map, its pose, the goal position, range sensors or contact bits. It learns by imitating a privileged "search teacher" (a planner that does see the full world state), with DAgger so that it also learns to recover from its own mistakes.
- Code, simulator, training and evaluation pipeline: https://github.com/Marxist-Leninist/meccanoid-embodied-lab (
vision/) - Interactive demo (static Space): https://huggingface.co/spaces/MarxistLeninist/meccanoid-embodied-lab
- Full evaluation report:
REPORT.md(this repo). Per-episode rows:eval-final.json(v2 run) andeval-v3.json(corrected baseline).
Results on held-out data
Held-out maps were never used for training, checkpoint selection or tuning. Success is the strict
rule (any bump ends the episode as a failure) on 'good' hardware unless marked. Brackets are
map-bootstrap 95 % CIs (1000 resamples); p is the exact two-sided McNemar test on paired episodes.
Comparator: SymVision-W, a camera-only symbolic controller tuned on VALIDATION maps only
(EVAL_PROTOCOL.v3.json).
| Set (episodes) | Student (this model) | SymVision-W | Difference | Verdict |
|---|---|---|---|---|
| TEST-T1, primary (500) | 15.4 % [11.4, 19.6] | 15.2 % [11.2, 19.8] | +0.2 pp [-4.2, +4.6], p = 1.0 | tie: does not beat the baseline |
| TEST-T1, student DEAF (microphone removed) (500) | 11.0 % [7.4, 15.0] | 15.2 % | -4.2 pp [-7.8, -0.2], p = 0.0075 | worse |
| TEST-T2, moving people (500) | 4.0 % [2.0, 6.4] | 7.8 % [4.8, 11.4] | -3.8 pp [-6.6, -1.4], p = 0.00055 | worse |
| STYLE-T1, changed obstacle colours/heights (250) | 13.6 % [8.0, 19.2] | 8.8 % [4.4, 13.6] | +4.8 pp [-0.4, +10.0], p = 0.073 | not significant |
| STYLE-T2 (250) | 6.8 % [3.6, 10.4] | 4.4 % [2.0, 7.2] | +2.4 pp [-1.2, +6.4], p = 0.29 | not significant |
| TEST-T1, lenient rule (500) | 18.6 % | 15.2 % | +3.4 pp [-0.6, +7.6], p = 0.06 | not significant |
| TEST-T1, 'poor' hardware (500) | 10.8 % [7.8, 13.6] | 4.2 % [2.4, 6.2] | +6.6 pp [+4.0, +9.2], p = 2e-6 | student ahead, but not a fair win: SymVision-W's wall-colour test breaks on the noisy camera (it was tuned on 'good' hardware only) |
| TEST-T1, 'faulty' hardware (500) | 8.0 % [5.2, 11.4] | 5.8 % [3.8, 8.0] | +2.2 pp [-0.8, +5.4], p = 0.14 | not significant |
| NAMED hand-written scenes (15) | 1/15 | 0/15 | p = 1.0 | underpowered |
Reference controllers on TEST-T1 (500 episodes, strict):
| Controller | Success |
|---|---|
| Teacher (privileged planner, sees the map; upper bound, not a vision controller) | 100 % (T2: 98.8 %) |
| Student (this model) | 15.4 % |
| SymVision-W (tuned symbolic baseline, v3) | 15.2 % |
| Student with visual patch positions shuffled (not pre-registered) | 13.0 % |
| Student DEAF (audio tokens zeroed) | 11.0 % |
| Student BLIND (visual tokens zeroed) | 0 % |
| Untrained (initial weights) | 0 % |
| Random controller | 0 % |
What this shows: the camera is essential (BLIND 0 %), training matters (UNTRAINED 0 %), and the microphone helps the model (+4.4 pp vs DEAF, p = 0.0013). What it does not show: that a learned controller beats a well-tuned hand-written one on this task. The owner's rule for this project was "beat a simple symbolic baseline on held-out data"; that rule is NOT met yet.
In absolute terms the model is a weak navigator: about 85 % of held-out T1 episodes fail, mostly by bumping into obstacles or walls. The generalisation gap is small (100 TRAIN maps 14.6 % vs held-out 15.4 %), so it is limited by the policy it learned, not by memorising maps.
Superseded comparison (kept for the record; do not quote as a result)
SUPERSEDED. The original v2 protocol compared the student with a frozen SymVision baseline whose parameters were tuned on an older camera that did not draw the arena walls. Under the walled camera every wall pixel looked like an imminent obstacle, so that baseline mostly turned away and timed out. Against it the student scored 15.4 % vs 5.6 % (p = 3.2e-10), and the deaf student 11.0 % vs 5.6 %. That "win" was an artifact of a broken baseline. It was replaced by the wall-aware SymVision-W comparison above (EVAL_PROTOCOL.v3).
Validation (checkpoint selection only)
VALIDATION T1, 500 episodes, strict, walled camera: student 60/500 (12.0 %), student DEAF 51/500, SymVision-W 50/500 (the student's +2.0 pp was not significant on VAL either, p = 0.25), old frozen SymVision 25/500. These maps were used to pick the checkpoint and to tune SymVision-W, so they are not evidence of generalisation.
Model details
| Checkpoint | s3-r2-q2 (walled camera, search-teacher pipeline, DAgger round 2 mid-round snapshot) |
| Format | mecc-vision-tokformer-v2 JSON (vision-transformer.json): config + named float tensors |
| Inference | controller.js (pure JavaScript, UMD; browser global MeccaVision, Node module.exports) |
| Size | 37,274 parameters (including the training-only auxiliary head) |
| Input per decision | packet.tokens only: 64 x 8 sensor tokens from the vendored sensors.js tokenize(). 48 visual tokens (8 x 6 patches of the 96 x 64 RGB camera: r, g, b, motion, edge, x, y, 0) and 16 audio tokens (8 audio-band + 8 audio-window tokens from stereo PCM). Plus one recurrent memory token: the model's own previous CLS state. The decision phase comes from an internal decision counter inside the controller; packet.tick is not an input |
| Not an input | Pose, map, goal coordinates, ranges, contact/collision bits, odometry, packet.summary, the simulator clock |
| Architecture | 64 sensor tokens (Linear 8->32 + learned position), 1 memory token (Linear 32->32 + position), 1 CLS token. 3 pre-LayerNorm blocks (width 32, 4 heads x 8, MLP 32-64-32 ReLU). Final LN on CLS gives h, which is also the next step's memory. Action head: Linear 32->64 + ReLU, then a binned policy over forward-speed x turn-rate bins mapped to left/right wheel speeds (track 0.28 m, max 0.35 m/s). The auxiliary head (training loss only) predicts goal visibility, bearing, distance and 8 free-space sectors |
| Output | {left, right} wheel speeds in m/s, clamped to [-0.35, 0.35] |
| Cadence | One capture + one act() every 3 physics ticks (0.15 s), action held in between. Call reset() at episode start |
| Latency | act() p50 11.1 ms, p95 17.9 ms, p99 20.9 ms in the closed loop on a shared CPU (budget 150 ms) |
Input contract
- Build the simulator world with the pinned
vendor/core-1101065f.jsand render sensors with the pinned walled cameravendor/sensors-08cabc99.js(sha256 inPINS.json; verified at load by the training and evaluation code). A different renderer is a different task. - Each decision, pass
packet.tokens(an array of 64 arrays of 8 numbers, as produced bytokenize()) toact(). Token order: 0-47 visual (row-major 8 x 6 grid), 48-63 audio. - Nothing else may be passed in; the model keeps its own memory token and decision counter, so call
reset()before every episode and callact()exactly once per 3-tick decision.
Training
- Teacher (privileged, label source only, not vision). A grid string-pulled geodesic planner
with the vendored steering law, a visibility-graph fallback and an actor-aware local planner
(
vision/teacher.js), wrapped as a search teacher (vision/train/search_teacher.cjs,{"period":80,"spin":36,"keep":8}) that also produces look-around behaviour a camera-only student can imitate when the goal is out of view. - Distillation with DAgger (
vision/train/pipeline3.sh). Round 0 is behaviour cloning on search-teacher rollouts. DAgger rounds then collect with the current student running the deployment code (controller.js), mixed with the teacher at a decaying beta (0.5, 0.25, 0.1, 0 for rounds 1-4), and every visited state is relabelled by the teacher. From round 2 on, the driver and initialisation of each round is the VAL-selected better of the previous round's mid-round (q2) and final snapshot. - Schedule that produced this checkpoint. s3-r0: 1600 steps, lr 0.0015, 209,332 frames; s3-r1 (beta 0.5): 1000 steps, lr 0.001, 291,393 frames, initialised from s3-r0-q2; s3-r2 (beta 0.25): initialised from s3-r1, lr 0.001; s3-r2-q2 is the snapshot after 500 steps of round 2. Sequence window 16 with 4 burn-in steps. It was selected over the other candidates on VALIDATION maps only.
- Data. Procedurally generated TRAIN maps only (
vision/maps/train-*): 387,872 frames (data s3-r0 + s3-r1 + s3-r2). - Losses. Action classification over the bins, plus auxiliary perception targets.
- Compute. CPU only (PyTorch for training, a JS export for inference). Torch-vs-JS parity on the
shipped weights: max action error 1.3e-08 (float32 and float64), 0 bin flips, closed loop 20/20
identical outcomes (
vision/tests/parity_model.cjs,parity_closedloop.cjs).
Evaluation protocols
| Protocol | Registered (UTC) | sha256 | Role |
|---|---|---|---|
EVAL_PROTOCOL.json (v1) |
2026-09-21 16:36:03 | 8ce6d960485a87d57d584e2eb2b8795c05af7ec514a308027fbe93d1eed7c446 |
original protocol (wall-less camera) |
EVAL_PROTOCOL.v2.json |
2026-09-21 18:03:38, before any held-out episode | 8b435c48898224920c4aa1b7f191230a92e2452673e1639e374e5c5ea3dc2d7c |
walled camera, held-out sets, claim rules; its SymVision comparison is superseded |
EVAL_PROTOCOL.v3.json |
2026-09-22 00:24:27 | fa414829cb0957b2bbd81dc89b1e08bf2207c48b816bf3feca6d2b23beb8490a |
replaces the broken baseline with SymVision-W (tuned on VAL only) |
- Held-out sets: procedural TEST maps (tiers T1 and T2), style-shifted maps, and named scenes.
- Metrics: strict success rate (primary), lenient success, SPL against the teacher path, ticks to goal, failure types.
- Statistics: map-level bootstrap 95 % CI and exact two-sided McNemar on paired episodes, alpha 0.05. The primary claim ("beats the symbolic baseline") needs a higher rate, a paired CI lower bound > 0 and a significant McNemar test on TEST-T1. It failed under v3.
- Disclosure: v3 was registered after the v2 held-out results had been read. The baseline was
tuned on validation maps only and the student is unchanged, so v3 corrects the comparator; it is
not a fresh pre-registration. Other deviations are listed in
REPORT.md.
Usage
Node.js
const fs = require('fs');
const { RobotWorld } = require('./vendor/core-1101065f.js'); // pinned simulator core
const { SensorRenderer } = require('./vendor/sensors-08cabc99.js'); // pinned sensors (walled camera)
const { createVisionController } = require('./controller.js');
const ctrl = createVisionController(fs.readFileSync('vision-transformer.json', 'utf8'));
const world = new RobotWorld(), cam = new SensorRenderer();
ctrl.reset(); // once per episode
for (let tick = 0; tick < 600; tick += 3) { // one decision per 0.15 s
const packet = cam.capture(world); // sealed RGB + PCM -> packet.tokens (64 x 8)
const { left, right } = ctrl.act(packet.tokens); // wheel speeds, m/s
for (let k = 0; k < 3; k++) world.step(left, right);
}
Browser
Load the same files with <script> tags (vendor/core-1101065f.js, vendor/sensors-08cabc99.js,
controller.js). They expose the globals MeccaCore, MeccaSensors and MeccaVision; fetch
vision-transformer.json and pass its text to MeccaVision.createVisionController(...), then call
reset() and act(packet.tokens) as above. The Space runs this in the page.
Files
| File | sha256 |
|---|---|
vision-transformer.json |
f4770d22b3e96d9ef96522ec8efc13c6c95e35187891a5d07e23043320888f85 |
| other files | see SHA256SUMS |
Also included: controller.js, controller-pixels.js (an experimental pixel-patch variant; no
walled-camera pixel weights exist, so it is not evaluated here), vendor/, PINS.json, the three
protocols with their .sha256 files, REPORT.md, eval-final.json, eval-v3.json and the
SymVision-W baseline in baselines/.
Limitations
- It does not beat a tuned symbolic baseline. It ties SymVision-W on held-out T1, and is worse without its microphone and with moving people (T2).
- Simulation only. The model has never controlled a physical robot. The sim-to-real gap is untested.
- The camera shows the goal through some obstacles. Obstacles are drawn as flat cards narrower than their footprint, so the goal post can show around a box corner: about 30 % of starts flagged "occluded and in view" still show goal pixels. Every controller sees the same camera, so the comparisons are fair, but the task is easier than "occluded" suggests.
- The microphone includes an idealised active sonar. About 1 capture in 6 carries a chirp whose echoes use exact ray distances from 7 fixed bearings. It cannot reveal the goal, but it is a near-direct range reading, not passive hearing. The DEAF ablation removes it.
- The privilege-invariance test is narrow. It proves that no undrawn privileged field reaches the tokens; it cannot detect leaks through drawn geometry such as the see-through above.
- Weak spatial use. Shuffling the positions of the visual patches costs only 2.4 pp on T1 (not significant).
- The v3 baseline correction was registered after the v2 held-out results had been read (see Evaluation protocols).
- The Space's "Walled TokFormer v2" option runs an earlier checkpoint, s3-r0-q1, not this one. On held-out data s3-r0-q1 scored 14.4 % on TEST-T1 (vs SymVision-W 15.2 %, p = 0.71) and 6.8 % on TEST-T2 (vs 7.8 %, p = 0.5): it beats the tuned baseline on neither.
- It imitates a privileged teacher, so it cannot beat the teacher by design. It is not a general navigation policy, a language model, or evidence of biological plausibility.
Licence
MIT (weights and code), copyright Marxist-Leninist and contributors (see LICENSE in the source
repository). None of the model files come from the GPL-3.0-or-later Human Cortex component.
The vendored simulator files (core, sensors) are MIT files of the same project.