Berk/multilingual-place-extractor-mdeberta-13lang-onnx

Multilingual Place Extractor — ONNX / transformers.js (13 languages)

ONNX build of the mDeBERTa-v3 place-entity tagger, for running token classification in the browser with transformers.js. It labels each token CITY / COUNTRY / ENTITY (typed BIO, 7 classes).

v2 (current): fixes recall on landmark lists (a city + a comma-separated list of landmarks/museums — held-out landmark field_f1 0.29 → ~0.86) and adds a city/country reclassification step in cascade.js (a COUNTRY span that is actually a known city — e.g. "a Paris trip" — becomes a CITY). Clean field_f1 0.948 → 0.952, typed span-F1 0.969.

Turning tagged spans into linked (text → "<entity> <city> <country>") pairs is a small, deterministic, parameter-free step (a positional linker + city→country gazetteer + English country canonicalization) that runs as plain JavaScript alongside the model. This repo bundles everything for a complete in-browser extractor in demo/:

• demo/index.html — a ready-to-run UI (loads this model via transformers.js).
• demo/cascade.js — the linker + gazetteer lookup + country canonicalization.
• demo/city_country_gazetteer.json — the full ~1.03M-city GeoNames gazetteer (city → country).
• demo/city_country_multi.json — ambiguous-name table (disambiguated by a country named nearby).
• demo/country_lookup.json — surface-form → English country canonicalization.

Run the full demo

# download this repo, then serve the demo folder over HTTP:
python3 -m http.server -d demo 8000     # open http://localhost:8000

The page downloads the model from this repo, runs it in WebGPU (fp16) or WASM (fp32), and links spans against the full gazetteer — entirely client-side, no server, no token.

Usage (transformers.js v3)

import { pipeline } from "@huggingface/transformers";

// fp16 computes correctly on WebGPU; on the WASM backend use fp32 (WASM has no fp16
// kernels). int8 is intentionally NOT shipped: dynamic quantization was too lossy and
// dropped entities on some inputs.
const dtype = navigator.gpu ? "fp16" : "fp32";
const device = navigator.gpu ? "webgpu" : "wasm";
const tagger = await pipeline(
  "token-classification",
  "Berk/multilingual-place-extractor-mdeberta-13lang-onnx",
  { dtype, device }
);
const tokens = await tagger("I booked Hotel Lungomare in Rimini then flew to Bologna",
                            { ignore_labels: [] });
// tokens: [{ entity: "B-ENTITY", start, end, ... }, ...]  -> feed to cascade.js

Files

• config.json — DebertaV2ForTokenClassification, id2label = the 7 BIO tags.
• tokenizer.json, tokenizer_config.json — the mDeBERTa-v3 SentencePiece tokenizer.
• onnx/model_fp16.onnx — fp16 graph (~557 MB), for WebGPU (dtype: "fp16"). Fast.
• onnx/model.onnx — fp32 graph (~1.1 GB), for the WASM backend (dtype: "fp32"). Both reproduce the PyTorch token predictions; the 251k-token embedding table dominates size.

Provenance

Fine-tuned mDeBERTa-v3-base (MIT) on public-data synthetic travel text balanced across the 13 languages, with the non-Latin-script languages generated in native script. Released MIT. The full PyTorch model + gazetteer + recipe live in the companion repo Berk/multilingual-place-extractor-mdeberta-13lang.