comicbox/sam3.1-comicapp-onnx

SAM 3.1 — ONNX export for ComicApp

ONNX conversion of Meta's SAM 3.1 for use in ComicApp. No retraining, no fine-tuning — this is a format conversion of the authors' released weights so they can run under onnxruntime.

Attribution and licence

• Original work: SAM 3.1, Meta Platforms — https://github.com/facebookresearch/sam3
• Licence: the SAM License (19 November 2025). A verbatim copy ships in this repo as LICENSE, as §1(b)(i) requires.
• Your use of these files is governed by that Agreement, exactly as the original weights are. Notable terms: no use for military/warfare, nuclear, espionage or weapons purposes; no reverse engineering; compliance with export and sanctions controls.
• Source checkpoint: sam3.1_multiplex.pt.

What is here

file	precision	size	what it does
sam31_image_encoder.onnx	fp16	910 MB	Page encoder. Runs ONCE per page; everything else is cheap against its output.
sam31_prompt_side.onnx + .data	fp32	121 MB	Concept ("find every X") prompting.
sam31_prompt_box.onnx + .data	fp32	134 MB	Box prompting — outline what a person drew a box around.

The split is deliberate: the encoder is 94% of the bytes and nearly all of the compute, so a page is encoded once and then prompted repeatedly for almost nothing. That is what makes an interactive mask editor viable.

Input contract — read this before wiring it up

Normalisation is mean = std = 0.5 (i.e. 2x/255 − 1), from the model's own processor_config.json. It is NOT ImageNet normalisation. Getting this wrong degrades results silently rather than failing.

• Encoder input: image, float16, [1, 3, 1008, 1008], RGB, NCHW.
• Encoder output: four feature levels. The decoder consumes the first THREE — the model applies scalp=1 and discards the last.
• Prompt-side outputs: pred_logits [1,200,1], pred_boxes [1,200,4] (cxcywh, normalised), pred_masks [1,200,288,288], presence_logit [1,1].
• Thresholding is not in the graph. Score is sigmoid(logit) * sigmoid(presence); apply your own floor and upsample the masks to page size. Keeping it outside means the confidence floor is a runtime setting rather than a property of the file.

Why fp16 for the encoder

Measured, not assumed. On a real comic page, fp16 against fp32:

	fp32	fp16
detections kept	24	24 — same set
mask IoU on survivors (min / mean)	0.95060 / 0.99278	0.95060 / 0.99321
mask-pixel disagreement	0.1825%	0.1803%
size	1819 MB	910 MB

Identical decisions, half the download.

Note that at the feature level fp16 looks ~10× worse (relative error 0.11–0.17 vs 0.012–0.016) — but mean|diff| is 0.000089, so that is a handful of outliers a max-based ratio exaggerates, and nothing downstream notices it. If you are evaluating a quantisation, compare decisions rather than tensors.

The prompt-side graphs stay fp32: they are 13% of the bytes, so the saving is not worth an unverified change.

Conversion notes

The stock model does not export as-is. Six changes were needed, all behaviour-preserving:

1. perflib/fused.py:addmm_act hardcodes bf16 and has no ONNX symbolic — replaced with activation(F.linear(...)), which is what it computes.
2. RoPE uses complex tensors (torch.polar, view_as_complex); ONNX has no complex type — rewritten in real arithmetic, (a+bi)(c+di) = (ac−bd) + (ad+bc)i.
3. The 32 complex buffers are converted to real (…,2) form before tracing; doing it inside the function puts an aten::view_as_real in the graph.
4. decoder.py compares stored ints against export-time SymInts — torch.compiler.is_dynamo_compiling() forced true, which is the honest answer while tracing.
5. aten::_pin_memory is not implemented by torch.export; pinning is a host-transfer hint with no effect on values, so it is made identity.
6. concat_padded_sequences short-circuits on zero-length operands. With an empty geometric prompt the geometry tensors are zero-rowed, and ONNX shape inference cannot broadcast over a zero-length dim.

Each was verified to reproduce the original path exactly before export.

⚠️ Do not "fix" (6) by seeding a dummy box. A whole-page box with a positive label is not inert — it instructs the model to segment everything, and the file will load, run, and quietly answer a different question.

Verification

Both prompt graphs make identical decisions to the PyTorch reference on a real page — same surviving query set, mask IoU ≥ 0.9991, <0.008% of mask pixels differing — checked against a control (a different concept) that correctly disagrees.