Berk/multilingual-place-extractor-mdeberta-13lang-tagger

Multilingual Place Entity Tagger — mDeBERTa-v3, 13 languages

A fine-tuned mDeBERTa-v3-base token classifier that labels cities, countries, and entities (hotels / points of interest / landmarks) in free text across 13 languages, using a typed BIO scheme (7 labels: O, B-/I-CITY, B-/I-COUNTRY, B-/I-ENTITY).

Languages: English, French, German, Spanish, Italian, Dutch, Portuguese, Turkish, Russian, Chinese, Japanese, Korean, Arabic.

This repository is the standalone PyTorch / safetensors tagger (load it with AutoModelForTokenClassification). Turning the tagged spans into linked (text → "<entity> <city> <country>") query strings is a separate, deterministic, parameter-free step (a positional linker + a city→country gazetteer + country canonicalization). An ONNX build for in-browser use (transformers.js) is at Berk/multilingual-place-extractor-mdeberta-13lang-onnx.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

repo = "Berk/multilingual-place-extractor-mdeberta-13lang-tagger"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo).eval()

text = "I booked Hotel Lungomare in Rimini then flew to Bologna"
enc = tok(text, return_tensors="pt")
with torch.no_grad():
    pred = model(**enc).logits[0].argmax(-1)
print([(tok.convert_ids_to_tokens([i])[0], model.config.id2label[p.item()])
       for i, p in zip(enc["input_ids"][0], pred)])
# -> Hotel/B-ENTITY Lung/I-ENTITY ... Rimini/B-CITY ... Bologna/B-CITY

Linking spans to (city, country)

The tagger emits typed spans; to turn them into (text → "<entity> <city> <country>") the repo bundles the full ~1.03M-city gazetteer (GeoNames-derived):

• gazetteer/city_country_gazetteer.json — {"case_insensitive": {city → country}}
• gazetteer/city_country_multi.json — {ambiguous_city → [[country, population], …]} (pick the candidate country that is named nearby in the text; else the highest-population one)
• gazetteer/country_lookup.json — {surface_form → English country} to canonicalize a tagged COUNTRY

import json
gaz = json.load(open("gazetteer/city_country_gazetteer.json"))["case_insensitive"]
clk = json.load(open("gazetteer/country_lookup.json"))
norm = lambda s: " ".join(s.lower().split())

# CITY span "Rimini" -> country
print(gaz.get(norm("Rimini")))               # Italy
# COUNTRY span "미국" (native script) -> canonical English
print(clk.get(norm("미국")))                  # United States

The full positional linker (nearest CITY/COUNTRY by character distance, in-sentence disambiguation, entity→city inheritance) is parameter-free; a JavaScript reference implementation is cascade.js in the companion ONNX repo Berk/multilingual-place-extractor-mdeberta-13lang-onnx.

Training data — sources & licenses

Trained on public data only. The labels are derived deterministically from public gazetteers and place databases; an open LLM was used only to write natural query phrasing (never to invent place facts).

content	source	license
cities, countries, translations	GeoNames	CC BY 4.0
airports, flight routes	OpenFlights	ODbL
points of interest / landmarks	Wikidata	CC0
hotels, additional POIs	Foursquare Open Source Places	CC BY 4.0
landmark seeds	Google Landmarks	CC BY 4.0
query phrasing (generation only)	Qwen3-30B-A3B-Instruct-2507	Apache-2.0
base encoder	microsoft/mDeBERTa-v3-base	MIT

Training text is synthetic, location-rich travel prose balanced across all 13 languages, with the non-Latin-script languages (Russian, Chinese, Japanese, Korean, Arabic) generated in native script so place names are learned in the form they actually appear.

Results

Held-out evaluation:

v2 (current) adds landmark-list recall (a city followed by a comma-separated list of landmarks/museums) and a city/country reclassification fix. Held-out landmark-list field_f1 rose from 0.29 to ~0.86 with no regression elsewhere.

metric	value
typed span-F1 (token-level, 2,667-row entity-disjoint test)	0.969
full-system field_f1 (with the linker + gazetteer, 800-row clean gold)	0.952

Per-language full-system field_f1 (small-n languages are noisy): it 0.93 · fr 0.93 · nl 0.94 · en 0.94 · pt 0.90 · es 0.86 · tr 0.90 · de 0.92 · zh 0.82 · ko 0.89 · ja 0.93 · ar 0.82 · ru 0.42 (n=4).

License

Released under MIT (a derivative of mDeBERTa-v3-base, MIT). Respect the upstream data-source licenses listed above (notably the CC BY 4.0 attribution requirements and the ODbL terms for OpenFlights-derived content).

Limitations

• Domain-specific: trained on location-rich, itinerary-style text; may not generalize to arbitrary-domain NER.
• Small per-language eval slices are noisy (Russian especially, n=4).
• The tagger emits typed spans only; city→country linking is the separate deterministic step.