Multilingual Place Extractor — v3 (region-aware, structured output)

Fine-tuned mDeBERTa-v3-base that tags place spans (CITY / COUNTRY / ENTITY, typed BIO) in travel text across 13 languages, plus a parameter-free linker + gazetteer that turns each span into a structured record. v3 adds region support and a structured output schema; v2 remains available in the companion *-13lang / *-onnx / *-tagger repos.

What's new in v3

Region detection — the tagger now reliably finds region/island spans it used to drop (Tuscany, Bavaria, Sicily, Andalusia, native scripts like シチリア). Held-out region-detection recall 0.77 → 0.92.
Region resolution — a Wikidata + GeoNames region gazetteer maps regions to the right country (Sicily→Italy, Tuscany→Italy, Bavaria→Germany), including native-script names.
Structured output — each span is now {type, text, city, region, country, query}: type includes REGION; cities are enriched with their admin-1 region (Florence→Tuscany); query is the flat "<entity> <city> <country>" string (unchanged, backward-compatible).
Salience-ranked city prior — ambiguous city names rank by a blended Wikipedia-language-edition (notability) + population score instead of raw population, fixing common inversions (Venice→Italy, Porto→Portugal, Londres→UK, Manchester→UK).

Measured quality (held-out clean gold + a blind 80-itinerary set)

field-F1 0.944, typed span-F1 0.977
region typing 25/25, city→region enrichment correct, native-script regions resolve
Known limitation: a bare ambiguous toponym with no context ("Córdoba" alone) is resolved by the salience prior only — an intrinsic ceiling (the disambiguating evidence is not in the input). A context-coherence vote exists in the code (compose_structural_query(..., coherence_vote=True)) but is off by default pending further validation.

Use

import json
from infer_place_extractor import PlaceExtractor   # scripts/

ext = PlaceExtractor("model",
    region_names=set(json.load(open("gazetteer/region_names.json"))),
    city_region=json.load(open("gazetteer/city_region.json")))
gaz = json.load(open("gazetteer/city_country_gazetteer.json"))["case_insensitive"]
gaz_multi = json.load(open("gazetteer/city_country_multi.json"))
for e in ext.extract("A week in Tuscany, then Florence and Rome", gaz, gazetteer_multi=gaz_multi):
    print(e["type"], e["text"], "->", e["region"], e["country"])

The model alone (token classification) also loads with the standard transformers pipeline; the gazetteer + scripts/ add the linking, region typing, and enrichment.

Files

model/ — DebertaV2ForTokenClassification (7 BIO tags) + tokenizer + bio_head.pt.
gazetteer/ — city_country_gazetteer.json (name→country), city_country_multi.json (ambiguous names, [country, language-editions, population]), region_names.json (names typed REGION), city_region.json (city→admin-1 region).
scripts/ — infer_place_extractor.py + _gen_common.py (the parameter-free linker, region typing, enrichment, and compose).

Provenance

Trained on public-data synthetic travel text generated from GeoNames (CC-BY) and Wikidata (CC0), balanced across the 13 languages with native scripts for the non-Latin ones. Released MIT.

Downloads last month: -; Downloads are not tracked for this model. How to track