Multilingual Place Extractor — v6 (bulk-enriched landmark & nature coverage)

Fine-tuned mDeBERTa-v3-base that tags place spans (CITY / COUNTRY / ENTITY, typed BIO) in travel text across 13 languages, plus a parameter-free linker + gazetteer that turns each span into a structured record {type, text, city, region, country, query}.

What's new in v6

Bulk-enriched landmark coverage, especially natural features and parks. Earlier versions were built mostly from urban attractions, so nature-tourism trips parsed poorly — an Iceland itinerary would miss waterfalls, geysers and glaciers. The attraction set was expanded from the ground up (Wikidata): waterfalls, national parks, volcanoes, glaciers, lakes, hot springs, mountain ranges, islands, caves, plus cathedrals / basilicas / mosques / temples and other civic landmarks. Notable POIs grew ~9.5k → ~32.7k, covering far more countries (Iceland went from zero to ~190). A trip like "Reykjavík, then Seljalandsfoss, Skógafoss and Reynisfjara, then Vík" now detects every landmark and resolves them to Iceland.
Bigger landmark→country table for the optional context resolver (gazetteer/poi_country.json, ~782 → ~11.4k unambiguous entries) — more ambiguous-city cases get disambiguated by a co-occurring landmark, with no extra model and no network.
No regression: held-out clean-gold field score is unchanged from v5, typed span-F1 ~0.96, and the v5 fixes hold (dense comma-separated landmark lists stay as separate spans, regions type correctly).

Carried over from v4 / v5

Robust region detection — well-known regions (Sicily, Tuscany, Bavaria, Andalusia, Provence) tag as REGION and resolve to the right country, with their cities enriched by region.
Realistic multi-place itineraries — long, multi-sentence trip text with comma-separated landmark lists and accented names parses correctly (each landmark its own span).
Optional deterministic context resolver (scripts/context_resolver.py) — for an ambiguous city, a co-occurring known landmark (gazetteer/poi_country.json) or region naming exactly one candidate country pins it. Pure dictionary lookups over the spans the tagger already found — no extra model, no network. Off unless injected; the default stays lightweight and browser-deployable.
Reproducible provenance — data_manifest.json records the exact training sources (record counts + sha256 + git commit).

Measured quality (held-out clean gold)

clean field-F1 ~0.90, typed span-F1 ~0.96, no regression vs v5
region typing + city→region enrichment correct when the span is detected; native scripts resolve
with the optional context resolver, ambiguous-city country accuracy on a held-out benchmark rises, with no change to the clean-gold field score
Known limitation: a bare ambiguous toponym with no contextual cue ("Cordoba" alone, no landmark/region/country nearby) is resolved by a salience prior only — an intrinsic ceiling.

Use

import json
from infer_place_extractor import PlaceExtractor   # scripts/

ext = PlaceExtractor("model",
    region_names=set(json.load(open("gazetteer/region_names.json"))),
    city_region=json.load(open("gazetteer/city_region.json")))
gaz = json.load(open("gazetteer/city_country_gazetteer.json"))["case_insensitive"]
gaz_multi = json.load(open("gazetteer/city_country_multi.json"))
for e in ext.extract("Reykjavík, then Gullfoss and Geysir, then a week in Sicily",
                     gaz, gazetteer_multi=gaz_multi):
    print(e["type"], e["text"], "->", e["region"], e["country"])

The model alone (token classification) also loads with the standard transformers pipeline; the gazetteer + scripts/ add the linking, region typing, enrichment, and the optional resolver.

Files

model/ — DebertaV2ForTokenClassification (7 BIO tags) + tokenizer + bio_head.pt; model/onnx/ has fp32 + fp16 ONNX for transformers.js (verified equivalent to the PyTorch model).
gazetteer/ — city_country_gazetteer.json (name→country), city_country_multi.json (ambiguous names), region_names.json, city_region.json (city→admin-1 region), poi_country.json (landmark→country, for the context resolver).
scripts/ — infer_place_extractor.py, _gen_common.py, context_resolver.py.
data_manifest.json — training-data provenance.

Provenance

Trained on public-data synthetic travel text generated from GeoNames (CC-BY) and Wikidata (CC0), balanced across the 13 languages with native scripts for the non-Latin ones. Released MIT.

Downloads last month: -; Downloads are not tracked for this model. How to track