Berk/multilingual-place-extractor-mdeberta-13lang-v5

Multilingual Place Extractor — v5 (realistic itineraries + robust regions + optional context resolver)

Fine-tuned mDeBERTa-v3-base that tags place spans (CITY / COUNTRY / ENTITY, typed BIO) in travel text across 13 languages, plus a parameter-free linker + gazetteer that turns each span into a structured record {type, text, city, region, country, query}.

What's new in v5

• Realistic multi-place itineraries — long, multi-sentence trip text with comma-separated landmark lists and regions in context now parses correctly. Previously a dense list like "…the Louvre, Notre-Dame, Montmartre, and Sacré-Cœur" could drop or merge entries, and a region named alongside cities ("…to Normandy. … Etretat, Honfleur …") could be missed. Trained on LLM-authored realistic itineraries (with complete, grounded labels), the model now detects each landmark as its own span and the region as REGION — at no measurable clean-gold cost.

What's new in v4

• Robust region detection — region mentions are now reliably tagged across varied phrasings and sentence positions, not just fixed patterns. Earlier, well-known regions seen rarely in training (e.g. Sicily, once) could be mis-typed or split mid-word in unfamiliar phrasings; the region training data was rebalanced by notability and broadened in phrasing, so Sicily / Tuscany / Bavaria / Andalusia resolve to REGION + the right country, with their cities enriched by region.
• Optional deterministic context resolver (scripts/context_resolver.py) — for an ambiguous city (a name in more than one country), it checks the other detected spans: a known landmark (gazetteer/poi_country.json) or region naming exactly one candidate country pins the city (e.g. "Iglesia de La Merced in Granada" -> Nicaragua). Pure dictionary lookups over the spans the tagger already found - no extra model, no network. Off unless injected; the default stays lightweight and browser-deployable.
• Reproducible provenance — data_manifest.json records the exact training sources (record counts + sha256 + git commit) so the corpus is reproducible and drift is detectable.

Measured quality (held-out clean gold + a blind itinerary set)

• clean field-F1 ~0.935, typed span-F1 ~0.97
• region typing + city->region enrichment correct when the span is detected; native scripts resolve
• with the optional context resolver, ambiguous-city country accuracy on a held-out ambiguous benchmark rises from 0.84 to 0.94, with no change to the clean-gold field score
• Known limitation: a bare ambiguous toponym with no contextual cue ("Cordoba" alone, no landmark/region/country nearby) is resolved by a salience prior only - an intrinsic ceiling.

Use

import json
from infer_place_extractor import PlaceExtractor   # scripts/

ext = PlaceExtractor("model",
    region_names=set(json.load(open("gazetteer/region_names.json"))),
    city_region=json.load(open("gazetteer/city_region.json")))
gaz = json.load(open("gazetteer/city_country_gazetteer.json"))["case_insensitive"]
gaz_multi = json.load(open("gazetteer/city_country_multi.json"))
for e in ext.extract("A week in Sicily, then Palermo and Catania", gaz, gazetteer_multi=gaz_multi):
    print(e["type"], e["text"], "->", e["region"], e["country"])

The model alone (token classification) also loads with the standard transformers pipeline; the gazetteer + scripts/ add the linking, region typing, enrichment, and the optional resolver.

Files

• model/ — DebertaV2ForTokenClassification (7 BIO tags) + tokenizer + bio_head.pt; model/onnx/ has fp32 + fp16 ONNX for transformers.js (verified equivalent to the PyTorch model).
• gazetteer/ — city_country_gazetteer.json (name->country), city_country_multi.json (ambiguous names), region_names.json, city_region.json (city->admin-1 region), poi_country.json (landmark->country, for the context resolver).
• scripts/ — infer_place_extractor.py, _gen_common.py, context_resolver.py.
• data_manifest.json — training-data provenance.

Provenance

Trained on public-data synthetic travel text generated from GeoNames (CC-BY) and Wikidata (CC0), balanced across the 13 languages with native scripts for the non-Latin ones. Released MIT.