Multilingual Place Extractor — v7 (short-query handling)

Fine-tuned mDeBERTa-v3-base that tags place spans (CITY / COUNTRY / ENTITY, typed BIO) in travel text across 13 languages, plus a parameter-free linker + gazetteer that turns each span into a structured record {type, text, city, region, country, query}.

What's new in v7

v6 was strong on long multi-city itineraries but weaker on short, search-box-style queries. v7 targets that gap on two layers, with the long-itinerary score held flat-or-up throughout.

Standalone landmarks now resolve to their city + country. A bare landmark with no city in the text ("Eiffel Tower", "Tour Eiffel", "Sagrada Familia", "Colosseum") previously produced a geo-less query or was mis-typed as a city. A public Wikidata POI→city→country resolver (gazetteer/poi_resolver.json, ~33k notable POIs indexed over all their multilingual names) now fills the geography when an ENTITY span has no governing city. Eiffel Tower → Eiffel Tower Paris France; Sagrada Familia → Sagrada Familia Barcelona Spain.
Type-correction for mis-typed landmarks. A context-free span (the sole city mention, no country) that is a prominent landmark whose city differs from the span text is re-typed ENTITY, using a log-normalized sitelinks-vs-population prominence blend so a real homonym city (Como, Vancouver, Chios) stays a city. It only fires on standalone queries — multi-city itineraries are byte-identical.
Better bare-country typing, especially non-Latin scripts. ~7.7k short training rows (bare countries heavy on ru/zh/ja/ko/ar, bare landmarks, bare cities) were added. Bare-country wrong-country rate on a held-out short-query benchmark drops sharply (프랑스 → France, 日本 → Japan).

Carried over from v4 / v5 / v6

Robust region detection — well-known regions (Sicily, Tuscany, Bavaria, Andalusia, Provence) tag as REGION and resolve to the right country, with their cities enriched by region.
Bulk-enriched landmark coverage (Wikidata) including natural features (waterfalls, national parks, glaciers, volcanoes, islands) and civic/religious landmarks — ~32.7k notable POIs.
Realistic multi-place itineraries — long, multi-sentence trip text with comma-separated landmark lists and accented names parses correctly (each landmark its own span).
Optional deterministic context resolver (scripts/context_resolver.py) for ambiguous cities.
Reproducible provenance — data_manifest.json records the exact training sources (record counts + sha256 + git commit).

Measured quality (held-out gold)

Long-itinerary clean field-F1 ~0.93, typed span-F1 ~0.96 — flat-or-up vs v6.
On a held-out, leakage-guarded, 13-language short-query benchmark stratified by shape (bare landmark / bare city / "X in city" / city-to-city / bare country): overall trust-killer wrong-country rate is lower than v6, bare-country and bare-landmark resolution improve markedly, and span recall stays at 100%.
Known limitation: a bare ambiguous toponym with no contextual cue and no Wikipedia-notability signal ("Cancún" vs a same-named alternate of a larger city) is resolved by a salience prior only — an intrinsic ceiling that needs city-level notability data to close.

Use

import json
from infer_place_extractor import PlaceExtractor   # scripts/

G = "gazetteer"
ext = PlaceExtractor("model",
    region_names=set(json.load(open(f"{G}/region_names.json"))),
    city_region=json.load(open(f"{G}/city_region.json")),
    poi_resolver=json.load(open(f"{G}/poi_resolver.json")),     # NEW in v7
    city_pop=json.load(open(f"{G}/city_population.json")))      # NEW in v7
gaz = json.load(open(f"{G}/city_country_gazetteer.json"))["case_insensitive"]
gaz_multi = json.load(open(f"{G}/city_country_multi.json"))
for e in ext.extract("Eiffel Tower", gaz, gazetteer_multi=gaz_multi):
    print(e["type"], e["text"], "->", e["query"])   # ENTITY Eiffel Tower -> Eiffel Tower Paris France

The model alone (token classification) also loads with the standard transformers pipeline; the gazetteer + scripts/ add the linking, region typing, enrichment, the POI resolver, and the optional context resolver.

Files

model/ — DebertaV2ForTokenClassification (7 BIO tags) + tokenizer + bio_head.pt; model/onnx/ has fp32 + fp16 ONNX for transformers.js (verified equivalent to the PyTorch model).
gazetteer/ — city_country_gazetteer.json (name→country), city_country_multi.json (ambiguous names), region_names.json, city_region.json (city→admin-1 region), poi_country.json (landmark→country), poi_resolver.json (landmark→city+country, NEW), city_population.json (prominence guard for type-correction, NEW).
scripts/ — infer_place_extractor.py, _gen_common.py, context_resolver.py.
data_manifest.json — training-data provenance.

Provenance

Trained on public-data synthetic travel text generated from GeoNames (CC-BY) and Wikidata (CC0), balanced across the 13 languages with native scripts for the non-Latin ones. Released MIT.

Downloads last month: -; Downloads are not tracked for this model. How to track