danmu-detector

Two ONNX detectors for in-browser furniture detection via onnxruntime-web, meant to be run together. Built for Danmu, a local-first interior decoration app — everything runs on-device, and works offline once the files are cached.

They are paired because they fail on different classes. Measured on a real four-photo room containing 19 catalogued objects:

	recall
`yolov8n-oiv7` alone	7/19 (37%)
`yolov8s-worldv2-danmu` alone	10/19 (53%)
both	13/19 (68%)

The fixed-label model reliably finds monitors and windows. The open-vocabulary one finds refrigerators, ceiling fans, wardrobes, lamps and curtains — none of which the first detects at all: those class heads peak at 0.002–0.03 on real room photos against 0.38–0.44 for classes that do fire, and they stay dead at every model size (s, m and x all score the same 7/19 as nano). That is a vocabulary limit, not a capacity one, which is what the second model fixes.

Nothing here depends on an external repository; the reference implementation below is complete.

Files

File	Size	Output	Notes
`yolov8n-oiv7.onnx`	14.2 MB	`1x605x8400`	YOLOv8 nano, Open Images V7, 601 fixed classes
`yolov8n-oiv7.names.json`	12 KB	—	class index → name, 601 entries
`yolov8s-worldv2-danmu.onnx`	50.4 MB	`1x48x8400`	YOLO-World small, 44 furniture prompts baked in

Both are opset 12, input 1x3x640x640 NCHW, and share the standard YOLOv8 detect-head layout: channels are cx, cy, w, h then per-class scores (605 - 4 = 601, 48 - 4 = 44) across 8400 anchors. No objectness channel and no sigmoid — scores are ready to threshold. Boxes are in letterboxed 640-space and must be unpadded.

Both use only standard ONNX ops (no custom domains), so they run under onnxruntime-web on WASM or WebGPU.

The baked vocabulary

yolov8s-worldv2-danmu.onnx was produced with Ultralytics set_classes(), which runs the CLIP text encoder once at export and freezes the embeddings into the graph — so no text encoder is needed at runtime. Class index N is PROMPTS[N], in exactly this order:

const PROMPTS = [
  'sofa', 'couch', 'armchair',
  'chair', 'office chair', 'stool',
  'table', 'coffee table', 'dining table',
  'desk',
  'bed', 'mattress',
  'nightstand',
  'wardrobe', 'closet', 'chest of drawers', 'storage cabinet',
  'shelf', 'bookshelf', 'shoe rack',
  'mirror',
  'curtain', 'window curtain', 'window blind',
  'picture frame', 'wall art', 'poster',
  'lamp', 'light bulb', 'ceiling light',
  'ceiling fan', 'electric fan',
  'refrigerator',
  'potted plant',
  'door', 'wooden door',
  'computer monitor',
  'television',
  'window',
  'laptop',
  'washing machine',
  'microwave oven',
  'clothes rack', 'hanging clothes',
];

Do not reorder that list — it is the model's channel order, and an off-by-one mislabels every detection silently.

Several phrases deliberately collapse to one concept (clothes rack, hanging clothes, storage cabinet → wardrobe). Real rooms contain clothes rails and stacked fabric cubes, not the canonical Wardrobe a fixed-label model was trained on, and naming what is actually there is the whole advantage of an open-vocabulary model. To retarget it, re-export with your own prompt list.

Loading

const BASE = 'https://huggingface.co/DearthAI/danmu-detector/resolve/main/';

const providers = [];
if (typeof navigator !== 'undefined' && 'gpu' in navigator) providers.push('webgpu');
providers.push('wasm');

const oiv   = await ort.InferenceSession.create(BASE + 'yolov8n-oiv7.onnx', { executionProviders: providers });
const world = await ort.InferenceSession.create(BASE + 'yolov8s-worldv2-danmu.onnx', { executionProviders: providers });
const names = await (await fetch(BASE + 'yolov8n-oiv7.names.json')).json();

Use the /resolve/ URL, not /blob/ — /blob/ returns the HTML viewer page, which passes a HEAD status check and then hands your runtime a page of markup.

Preprocessing

Letterbox to 640×640 preserving aspect ratio, pad with grey, normalize to 0..1, feed planar RGB (all R, then all G, then all B — not interleaved).

const INPUT = 640;

function toTensor(img, ox, oy, cw, ch) {      // crop rect in source pixels
  const scale = INPUT / Math.max(cw, ch);
  const sw = Math.round(cw * scale), sh = Math.round(ch * scale);
  const dw = Math.floor((INPUT - sw) / 2), dh = Math.floor((INPUT - sh) / 2);

  const c = document.createElement('canvas');
  c.width = c.height = INPUT;
  const ctx = c.getContext('2d');
  ctx.fillStyle = '#727272';                  // letterbox grey
  ctx.fillRect(0, 0, INPUT, INPUT);
  ctx.drawImage(img, ox, oy, cw, ch, dw, dh, sw, sh);

  const px = ctx.getImageData(0, 0, INPUT, INPUT).data;
  const area = INPUT * INPUT;
  const data = new Float32Array(3 * area);
  for (let i = 0; i < area; i++) {
    data[i]            = px[i * 4]     / 255;
    data[area + i]     = px[i * 4 + 1] / 255;
    data[2 * area + i] = px[i * 4 + 2] / 255;
  }
  return { data, scale, dw, dh };
}

Tile — it matters more than model size

Letterboxing a 2000px wide-angle photo to 640 shrinks mid-sized furniture below what these models resolve. Running the whole frame plus 2×2 tiles at 15% overlap took nano from 4/19 to 7/19 for zero extra download, and the world model from 7/19 to 10/19. Overlap keeps objects straddling a seam intact.

function tilesFor(iw, ih) {
  const ox = iw * 0.15, oy = ih * 0.15;
  const crops = [{ ox: 0, oy: 0, cw: iw, ch: ih }];
  for (let r = 0; r < 2; r++)
    for (let c = 0; c < 2; c++) {
      const x0 = Math.max(0, c * iw / 2 - ox), y0 = Math.max(0, r * ih / 2 - oy);
      const x1 = Math.min(iw, (c + 1) * iw / 2 + ox);
      const y1 = Math.min(ih, (r + 1) * ih / 2 + oy);
      crops.push({ ox: x0, oy: y0, cw: x1 - x0, ch: y1 - y0 });
    }
  return crops;
}

Decoding, and merging the two models

Collect candidates from both models over every tile into one pool in normalized whole-image coordinates, then run a single NMS. That one pass resolves per-tile duplicates, cross-tile seam duplicates, and the same object found by both models.

const CONF = 0.35, IOU_T = 0.45, MAX_PER_IMAGE = 12;
const pool = [];

for (const crop of tilesFor(iw, ih)) {
  const pre = toTensor(img, crop.ox, crop.oy, crop.cw, crop.ch);
  const toX = v => (crop.ox + (v - pre.dw) / pre.scale) / iw;
  const toY = v => (crop.oy + (v - pre.dh) / pre.scale) / ih;

  for (const [sess, labelOf] of [
    [oiv,   i => names[String(i)]],
    [world, i => PROMPTS[i]],
  ]) {
    const t = new ort.Tensor('float32', pre.data.slice(), [1, 3, INPUT, INPUT]);
    const out = (await sess.run({ [sess.inputNames[0]]: t }))[sess.outputNames[0]];
    const [, channels, anchors] = out.dims;
    const nc = channels - 4, d = out.data;
    const at = (ch, a) => d[ch * anchors + a];

    for (let a = 0; a < anchors; a++) {
      let best = 0, bestC = -1;
      for (let ci = 0; ci < nc; ci++) {
        const s = at(4 + ci, a);
        if (s > best) { best = s; bestC = ci; }
      }
      if (best < CONF || bestC < 0) continue;
      pool.push({
        x: toX(at(0, a)), y: toY(at(1, a)),
        w: at(2, a) / pre.scale / iw,
        h: at(3, a) / pre.scale / ih,
        conf: best, label: labelOf(bestC),
      });
    }
  }
}

Then NMS in that same normalized space, and convert centre-form to top-left:

function iou(a, b) {
  const x1 = Math.max(a.x - a.w / 2, b.x - b.w / 2);
  const y1 = Math.max(a.y - a.h / 2, b.y - b.h / 2);
  const x2 = Math.min(a.x + a.w / 2, b.x + b.w / 2);
  const y2 = Math.min(a.y + a.h / 2, b.y + b.h / 2);
  const inter = Math.max(0, x2 - x1) * Math.max(0, y2 - y1);
  return inter / (a.w * a.h + b.w * b.h - inter + 1e-9);
}

const kept = [];
for (const b of [...pool].sort((p, q) => q.conf - p.conf)) {
  if (kept.every(k => iou(k, b) < IOU_T)) kept.push(b);
  if (kept.length >= MAX_PER_IMAGE) break;
}
const boxes = kept.map(b => ({
  label: b.label, conf: b.conf,
  x: b.x - b.w / 2, y: b.y - b.h / 2, w: b.w, h: b.h,
}));

Subtracting dw/dh before dividing is the step that is easy to miss — skip it and every box drifts toward the image centre on non-square inputs.

Tuning notes

Measured, so you don't have to rediscover them:

Confidence 0.35. Dropping to 0.20 gained one object and added a spurious sofa(0.29).
MAX_PER_IMAGE 12. Raising it to 30 added one box and changed no score.
Bigger is not better. yolov8s/m/x-oiv7 (46 / 105 / 275 MB) all score the same 7/19 as the 14 MB nano. Spend the bytes on the second model instead.
Still missed at 13/19: doors in some framings, wall art, and curtains in a frame that is almost entirely curtain. Treat the output as a head start on manual correction, not a finished result.

Reproducing

pip install --index-url https://download.pytorch.org/whl/cpu torch
pip install ultralytics onnx onnxslim onnxruntime clip-anytorch ftfy

python - <<'PY'
from ultralytics import YOLO
YOLO('yolov8n-oiv7.pt').export(format='onnx', imgsz=640, opset=12)

m = YOLO('yolov8s-worldv2.pt')
m.set_classes(PROMPTS)          # the 44 phrases above, in order
m.export(format='onnx', imgsz=640, opset=12)
PY

CPU-only torch is enough — export needs no GPU, and it keeps the install near 200 MB instead of 2.5 GB.

Licence

AGPL-3.0. Both are Ultralytics YOLO weights, licensed AGPL-3.0 by Ultralytics. The AGPL obligations attach to these artifacts and travel with them.

They are hosted here, separately from the application that consumes them, precisely so the licences stay distinct: that app is MIT, and it fetches these weights at runtime rather than bundling or redistributing them. If you vendor these files into your own project, AGPL-3.0 applies to you — a permissive licence on the surrounding code does not cover them.

Downloads last month: -; Downloads are not tracked for this model. How to track