gaze-lle-p150

Gaze-LLE (the gazelle_dinov2_vitb14_inout checkpoint: DINOv2 ViT-B/14 backbone plus gaze decoder) running entirely on one Tenstorrent Blackhole p150a via tt-nn: one image plus N head boxes in, N 64×64 gaze heatmaps and in-frame scores out. Weights: changh95/gaze-lle-weights · Paper: arXiv:2412.09586 · Upstream code: fkryan/gazelle · Port: changh95/tt-Gaze-LLE

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/gaze-lle-p150 --with-weights
tt-model serve changh95/gaze-lle-p150

• Weights changh95/gaze-lle-weights at f83e53f0f17d go to your HF cache; the image does not contain them.
• Serves on port 20000 (or the next free port); ready when the log says Application startup complete.

Run with tt-cli

tt serve changh95/gaze-lle-p150
printf '{"image":"%s","bboxes":[[116,28,176,104],[269,37,301,81],[87,57,107,76]]}' "$(base64 -w0 media/source_1.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/gaze-lle-p150

• POST /predict: image (base64 PNG/JPEG), bboxes (1–10 head boxes [x1,y1,x2,y2]); optional bbox_format (pixels | normalized), heatmap_format (png | npz | json | none).
• GET /health, GET /info.

Response

{"model": "gaze-lle-p150", "image_size": [500, 334], "input_size": [448, 448], "num_heads": 3,
 "heatmap_size": [64, 64], "heatmap_format": "png",
 "predictions": [
   {"bbox": [116.0, 28.0, 176.0, 104.0], "bbox_normalized": [0.232, 0.083832, 0.352, 0.311377], "inout": 0.032,
    "gaze_target": {"x": 289.1, "y": 172.2, "x_norm": 0.578, "y_norm": 0.516, "heatmap_row": 33, "heatmap_col": 37, "score": 0.348},
    "heatmap_png": "..."},
   {"bbox": [269.0, 37.0, 301.0, 81.0], "inout": 0.711,
    "gaze_target": {"x": 281.2, "y": 167.0, "heatmap_row": 32, "heatmap_col": 36, "score": 0.363}, "heatmap_png": "..."}],
 "timing_ms": {"inference": 13.34, "total": 22.73}}

• bbox is echoed as sent; inout is P(target inside the frame); gaze_target.x/y is the heatmap argmax mapped back to original pixels (x = col/64 · W, upstream convention). The 64×64 heatmap covers the whole original image.
• heatmap_png is a base64 8-bit grayscale PNG (uint8(round(p·255))); npz returns one base64 .npz (heatmap float32 (N,64,64), inout float32 (N,)) in heatmap_npz.

Demo

GazeFollow test-set scenes; boxes come from RetinaFace, each same-coloured arrow/× is that person's predicted gaze target.

Input (media/source_N.png)	Prediction on p150a (media/target_N.png)

Accuracy and speed

Metric	Value
GazeFollow test AUC (4,782 images)	0.9541 vs 0.9543 torch reference (paper 0.9560)
GazeFollow Avg L2 / Min L2	0.1129 / 0.0512 vs 0.1103 / 0.0491 torch reference
Inference, served over HTTP (warm, batch 1, 1 head, 640×514)	~9.5 ms device · ~21 ms server total · ~24 ms client wall
Inference, served over HTTP (warm, batch 1, 3 heads, 500×334)	~13.3 ms device · ~23 ms server total · ~25 ms client wall

Caveats

• Every image is squashed to 448×448; one image per request, 1–10 head boxes. All heads share one backbone pass; the decoder tail runs once per head, serially.
• bf16/bfp8 on device: backbone stage PCC vs torch is 0.972–0.976 with the pretrained weights (0.999 with random weights); heatmap PCC is ~0.99 on face-centred boxes, but the argmax can move on very small or off-face boxes.
• Only the vitb14 checkpoint is ported and the compute grid is hard-coded to the p150a's 10×13 cores; hardware: p150 only.
• Not an OpenAI-compatible API; GET /v1/models is a stub so the tt-model ready card does not 404.
• Validated on tt-metal v0.78.0-dev20260820 (main 8b98410e730), single p150a only.

Licensing

• Weights: changh95/gaze-lle-weights mirrors the two upstream checkpoints unchanged — Gaze-LLE decoder MIT, DINOv2 ViT-B/14 backbone Apache-2.0.
• Port and serving code (code/): from changh95/tt-Gaze-LLE; source files carry SPDX-License-Identifier: Apache-2.0 headers and the repo is published under the upstream MIT terms.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	8b98410e730bb504fea43a88609756e34821d91d
code/ digest	92ec14dc1063728b (sha256, first 16 hex digits)
built	2026-09-12T13:25:54+00:00 by tt-model 0.1.0