CardAnvilModels/grounding-dino-tiny-ONNX

Grounding DINO Tiny (ONNX, fp16), multi-phrase

Grounding DINO Tiny as ONNX for Transformers.js, fp16. Give it an image and a prompt of one or more phrases (goat. city. sky.), and it returns boxes, each with the phrase it matched. Card Anvil's Describe command uses it in the browser, on WebGPU or WebAssembly, to find what to select in card art.

Why another export. onnx-community/grounding-dino-tiny-ONNX scores only the first phrase of a prompt. Every phrase after it comes out near zero (0.02–0.05 where PyTorch gives 0.5–0.8), so goat. city. sky. finds only goats. That export was made with transformers 4.48, which builds each phrase's text attention mask in a Python loop over the prompt's special tokens. Tracing unrolled the loop for a one-phrase example prompt.

This export rebuilds that function from tensor comparisons and min/max. Before each export it is checked against Transformers' own on 140 prompts of 0–6 phrases, so the graph handles any number of phrases, up to the model's 256 tokens.

It is a drop-in replacement: the same inputs, outputs and file name as onnx-community's, so Transformers.js loads it the same way.

Files

File	What	Size
onnx/model_fp16.onnx	The whole model, fp16 weights, float32 inputs and outputs	364 MB
config.json, preprocessor_config.json, tokenizer files	Copied unchanged from onnx-community's repo at ff690b0a
export_grounding_dino.py	The script that made it

Inputs	Outputs
pixel_values [1, 3, 800, 800], pixel_mask [1, 800, 800]	logits [1, 900, 256]: each query's score for each token
input_ids, token_type_ids, attention_mask [1, n], n from 4 to 256	pred_boxes [1, 900, 4]: centre x, centre y, width, height, 0–1

The processor stretches the image to 800 × 800, with no padding.

Usage

import {
  AutoProcessor,
  GroundingDinoForObjectDetection,
  RawImage,
} from "@huggingface/transformers";

const id = "CardAnvilModels/grounding-dino-tiny-ONNX";
const model = await GroundingDinoForObjectDetection.from_pretrained(id, {
  dtype: "fp16",
  device: "webgpu", // or "wasm"
});
const processor = await AutoProcessor.from_pretrained(id);

const image = await RawImage.fromURL("https://example.com/art.jpg");
// Lower-case phrases, each ended with a full stop.
const inputs = await processor(image, "goat. city. sky.");
const outputs = await model(inputs);
const [result] = processor.post_process_grounded_object_detection(
  outputs,
  inputs.input_ids,
  {
    box_threshold: 0.35,
    text_threshold: 0.25,
    target_sizes: [[image.height, image.width]],
  },
);
// result.boxes: [x0, y0, x1, y1] in pixels; result.labels: the phrase each matched.

In one pass, the phrases compete for boxes, as they do in the original model. Something found on its own can lose to another phrase that claims the same region.

How it was checked

With onnxruntime 1.31 on the CPU, against PyTorch (transformers 5.20, fp32) on the same inputs:

• fp32 export: within 0.01 of PyTorch, on single phrases and on prompts of three and four phrases.
• This fp16 file: within 0.01 of PyTorch on single phrases, and close on joint prompts. The largest gap seen was about 0.1, on one three-phrase prompt.

In Microsoft Edge, with ONNX Runtime Web 1.31, on 24 images and 98 prompts:

Measure	onnx-community's	This export
WebGPU, a pass (RTX 4070 Ti SUPER)	327 ms	347 ms
WebAssembly, a pass (one thread)	13.4 s	14.7 s
Single-phrase boxes	—	the same: median IoU 0.995
Phrases found alone that a joint pass loses	45 of 68	14 of 66, as in PyTorch

All 48 GridSample nodes (the deformable attention) run on WebGPU.

How it was made

export_grounding_dino.py, with torch 2.14.1, transformers 5.20.2, onnx 1.23.2, onnxscript 0.7.2 and onnxconverter-common 1.16.0:

1. fp32 export from IDEA-Research/grounding-dino-tiny at a2bb814d, with the dynamo exporter, opset 18, and a dynamic sequence length. Two changes make it export:
   • the text-mask function rebuilt as above (Transformers' uses torch.isin, which exports as searchsorted, which ONNX lacks);
   • TRANSFORMERS_DISABLE_TORCH_CHECK=1, since its shape checks export as _is_all_true, which ONNX also lacks.
2. fp16 conversion with onnxconverter_common, inputs and outputs kept float32. The converter leaves 23 Cast nodes aimed at float32; the script retypes them.

Credits and licence

• Model: Grounding DINO by IDEA Research, from Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023), through IDEA-Research/grounding-dino-tiny.
• Config and tokenizer files: ONNX Community.

Licensed under the Apache License 2.0, as the original model is. See LICENSE.