Arko007/weathergpt-models

WeatherGPT model bundle

Five models for the WeatherGPT meteorological interoperability layer. Every number below was produced by a training script and copied from a metrics.json; none was typed by hand.

Install

pip install weathergpt-models  # or: pip install -e . from the repo's
                                # weathergpt_models/ directory if the
                                # package isn't on PyPI yet

The package depends only on numpy, torch, transformers, lightgbm and scikit-learn — it does not import FastAPI or anything from the serving app, so it can be used standalone.

Training data at a glance

Full detail (feature lists, exact split logic, dataset caveats) is in each model's own section below and in the linked dataset card — this is just the numbers in one place.

model	train	val	test	unit
M1 field_mapper	7286	991	5140	rows (test = zero-shot, unseen source tables)
M2 mos	5,388,048	2,308,464	565,800	rows (test = spatially held-out locations)
M3 intent	2340	412	1275	rows (test = held-out template families AND districts)
M4 calibration	5,388,048	2,308,464	565,800	rows (precipitation only)
M5 trust_ranker	400,000	100,000	100,000	groups (subsampled from ~4.6M/2.0M/0.5M available; test = spatially held-out)

Quickstart

from weathergpt_models import ModelRegistry

registry = ModelRegistry.from_hub("Arko007/weathergpt-models")
print(registry.status())  # what loaded, and why anything didn't

Every model attribute is None if its admission gate failed — always check for None before calling it and fall back to a deterministic path. This bundle's gate results: field_mapper PASS, mos PASS, intent PASS, calibration PASS, trust_ranker PASS.

Usage — one real call per model

These are the actual method signatures, not illustrative pseudocode — copied from modal_jobs/verify_package.py and modal_jobs/realworld_check.py, which exercise every one of them against live data before each release.

# M1 -- map a raw provider field name to a canonical variable
mapping = registry.field_mapper.map_field(
    "APCP", unit="kg m-2", description="Total precipitation")
if mapping.is_usable:
    print(mapping.canonical_variable, mapping.confidence)
else:
    print("abstained -- do not fabricate a mapping")


# M2 -- bias-correct a multi-model forecast
corrected = registry.mos.correct(
    "temperature_2m",
    forecasts={"gfs_seamless": 31.2, "ecmwf_ifs025": 30.4,
              "icon_seamless": 30.9, "gem_seamless": 31.8},
    context={"lead_hours": 30, "lead_age_days": 1, "hour_utc": 6,
            "doy": 245, "elevation_m": 303.0, "lat": 21.15, "lon": 79.09})
print(corrected.value, corrected.interval_low, corrected.interval_high)


# M3 -- parse intent + LOC/TIME/CROP slots from a natural-language query
parsed = registry.intent.parse("kal Bhandara me baarish hogi kya?")
print(parsed.intent, parsed.intent_confidence)
print(parsed.slot_text("LOC"), parsed.slot_text("TIME"))
# do NOT use parsed.variables -- that head is still weak
# (variable macro-F1 0.12 on held-out data); intent confidence also
# runs low (0.15-0.19) on real casual phrasing -- treat it as a weak
# signal alongside a rule-based parser, not a standalone decision.


# M4 -- calibrated exceedance probability from the same multi-model spread
curve = registry.calibration.probability_curve(
    "precipitation",
    forecasts={"gfs_seamless": 4.2, "ecmwf_ifs025": 1.1,
              "icon_seamless": 6.8, "gem_seamless": 0.0},
    context={"lead_hours": 30, "lead_age_days": 1, "hour_utc": 6,
            "doy": 245, "elevation_m": 303.0, "lat": 21.15, "lon": 79.09})
for point in curve:
    print(f"P(precip > {point.threshold}mm) = {point.probability:.3f}")


# M5 -- rank candidate NWP sources by learned trust for this context
ranked = registry.trust_ranker.rank(
    "temperature_2m",
    candidates={"gfs_seamless": 31.2, "ecmwf_ifs025": 30.4,
               "icon_seamless": 30.9, "gem_seamless": 31.8},
    context={"lead_hours": 30, "season": "autumn", "elevation_m": 303.0,
            "lat": 21.15, "lon": 79.09})
print([r.source for r in ranked])  # best source first

Full integration guidance for wiring this into a FastAPI service (which site in the codebase to touch, what NOT to change, live-tested caveats for each model) is in docs/MODEL_REGISTRY_INTEGRATION.md and docs/REAL_WORLD_READINESS.md in the source repo, not duplicated here since those get updated as real query traffic accumulates.

The multi-model-forecast-vs-truth training data behind M2, M4 and M5 is itself published at Arko007/weathergpt-d1-mos-dataset — see that dataset card for the full column list and how the splits were built. Real sample counts for every model are in each section below.

Each model is gated on beating the baseline it replaces. An artifact that cannot prove its provenance, or that does not beat its baseline, is refused at load time and the caller falls back to a deterministic path.

field_mapper — m1_field_mapper_v1

label-embedding bi-encoder + multitask heads

• dataset: d3_authoritative_parameter_tables

• dataset sha256: e5d3562cedf16d653f75d84adf51868d

• split: by source table — train: CF+GRIB2+CAP; dev_zeroshot: BUFR+OpenMeteo+IMD (threshold calibration only); test_zeroshot: WRF+NCEP (never touched until the final measurement)

• trained: 2026-09-03T16:33:08.084847Z

• admission gate: PASS — zero-shot macro-F1 0.743 vs dict registry 0.142

• training data: d3_authoritative_parameter_tables — real parameter tables (CF Standard Names, ECMWF/GRIB2 definitions, NOAA NCEP GFS .idx inventories, the WRF Registry, WMO BUFR, Open-Meteo and IMD field names), not a synthetic/augmented list.

• rows: train 7286 / val 991 / test (zero-shot, unseen source tables) 5140.

metric	model	dict registry	majority
zero-shot macro-F1	0.7432	0.1416	0.0217
zero-shot accuracy	0.8319	0.4907	0.4356
statistic accuracy	0.9792	—	—
misassignment rate	0.0016	—	—

The test set is parameter tables the model never trained on (WRF Registry, NCEP GFS inventories, the WMO BUFR element table, Open-Meteo and IMD product fields), so this measures generalisation to a schema nobody has mapped.

mos — m2_mos_v1

monotone quantile network (pinball) + LightGBM quantile forest, blended at a validation-chosen weight, with conformalised intervals

• dataset: d1_multi_model_nwp_vs_era5_seamless

• dataset sha256: a3888e44c7a6bd09b8282f214b68b1e6

• split: chronological 70% cutoff AND 20% spatially held-out locations

• trained: 2026-09-04T06:29:10.987471Z

• admission gate: PASS — positive CRPS skill against the raw ensemble on the spatial holdout for every variable

• training data: Arko007/weathergpt-d1-mos-dataset — 9,582,912 rows, 127 real Indian locations, 4 NWP models (GFS/ECMWF/ICON/GEM) vs. ERA5-Land truth, 30 input features per row (see the dataset card for the exact feature list). Split: chronological 70% cutoff for train/val, plus 20% of locations held out entirely for the spatial test set.

• rows actually used (identical across temperature_2m/precipitation/wind_speed_10m since it's the same row set reshaped per target): train 5,388,048 / val 2,308,464 / test (spatially held out) 565,800.

variable	CRPS	raw ensemble CRPS	CRPS skill
temperature_2m	0.6162	0.6469	0.047
precipitation	0.2190	0.2523	0.132
wind_speed_10m	1.6000	1.7570	0.089

intent — m3_intent_v1

JointBERT: intent + BIO slots + multi-label variables on a shared encoder

• dataset: d4_multilingual_templated_queries_with_exact_slot_spans

• dataset sha256: cb2ee6c4e1d8d9857975d3362957d90c

• split: held-out template families {sow,heat,storm} AND 20% held-out districts

• trained: 2026-09-04T08:43:34.868234Z

• admission gate: PASS — intent macro-F1 0.746 vs rule parser 0.166, slot F1 0.988

• training data: d4_multilingual_templated_queries_with_exact_slot_spans — template-generated queries across 39 template families, 127 real Indian locations, translated into 13 Indian languages plus English (exact BIO slot spans computed from character offsets, never translation-model-reported).

• rows: train 2340 / val 412 / test (held-out template families AND districts) 1275.

metric	model	rule parser
intent macro-F1	0.7459	0.1664
intent accuracy	0.8361	0.5129
slot F1 (seqeval)	0.9878	not supported
variable micro-F1	0.1704	not supported

Held out: whole template families and whole districts, so neither a memorised sentence pattern nor a memorised place name can inflate this.

language	n	intent macro-F1	slot F1
as	91	0.7046	0.9819
bn	91	0.7486	0.9922
bn_latn	91	0.7774	0.9688
en	94	0.7477	0.9824
gu	91	0.7242	0.9948
hi	91	0.7475	0.9870
hi_latn	91	0.7508	0.9691
kn	91	0.7679	0.9948
ml	89	0.7538	1.0000
mr	91	0.7475	0.9948
or	91	0.7644	0.9923
pa	91	0.7475	0.9819
ta	91	0.7750	0.9948
te	91	0.7475	0.9948

calibration — m4_calibration_v1

hurdle CSGD (precipitation: P(wet) x censored shifted gamma) + Gaussian EMOS (temperature, wind), CRPS-fitted, with isotonic-refined exceedance probabilities

• dataset: d1_multi_model_nwp_vs_era5_seamless

• dataset sha256: a3888e44c7a6bd09b8282f214b68b1e6

• split: chronological 70% cutoff AND 20% spatially held-out locations

• trained: 2026-09-04T06:00:18.494934Z

• admission gate: PASS — calibrated Brier beats raw ensemble frequency at every threshold

• training data: Arko007/weathergpt-d1-mos-dataset — 9,582,912 rows, 127 real Indian locations, 4 NWP models (GFS/ECMWF/ICON/GEM) vs. ERA5-Land truth, 30 input features per row (see the dataset card for the exact feature list). Split: chronological 70% cutoff for train/val, plus 20% of locations held out entirely for the spatial test set.

• rows actually used (precipitation): train 5,388,048 / val 2,308,464 / test (spatially held out) 565,800.

variable	CRPS	raw ensemble CRPS	CRPS skill
precipitation	0.2491	0.2315	-0.076

Precipitation exceedance, Brier score (lower is better):

threshold	base rate	raw member count	calibrated	climatology
>0.1 mm	0.3161	0.16660	0.14883	0.21616
>1.0 mm	0.0932	0.08376	0.07243	0.08455
>5.0 mm	0.0083	0.00909	0.00807	0.00822
>10.0 mm	0.0012	0.00155	0.00124	0.00124
>25.0 mm	0.0001	0.00014	0.00010	0.00010
>50.0 mm	0.0000	0.00000	0.00000	0.00000

Transfer assumption. Trained on a 4-member multi-model ensemble; served against the 31-member GFS ensemble. Only summary statistics cross the boundary, so the interface is identical, but the spread mapping is assumed rather than measured: the ensemble API serves members only for ~the last 4 days while ERA5 truth lags ~6, so the two windows never overlap and the transfer could not be verified.

trust_ranker — m5_trust_ranker_v1

LightGBM LambdaMART over candidate NWP sources per (location, valid time, lead) group

• dataset: d1_multi_model_nwp_vs_era5_seamless

• dataset sha256: 99fd37b00135687f2c1085d0e32fb99c

• split: chronological 70% cutoff AND 20% spatially held-out locations

• trained: 2026-09-03T16:25:33.240955Z

• admission gate: PASS — beats the fixed authority order on 3/3 variables

• training data: the same Arko007/weathergpt-d1-mos-dataset, reshaped into ranking groups — one group per (location, valid time, lead) with the 4 NWP sources as candidates to rank.

• groups actually used (per variable, identical shape for temperature_2m/precipitation/wind_speed_10m): train 400,000, val 100,000, test (spatially held out) 100,000 — each subsampled (max_groups=400,000, val/test at 1/4 that) from a much larger pool of complete 4-source groups (roughly 4.6-4.7M for train, 1.9-2.0M for val, 0.48-0.49M for test, per variable) for tractability; ranking needs whole groups, not individual rows, so this is a group count, not a row count.

variable	NDCG@1	picks the best source	fixed authority does	RMSE following ranker	RMSE following fixed order
temperature_2m	0.6876	0.383	0.346	1.308	1.443
precipitation	0.8506	0.237	0.170	1.056	1.124
wind_speed_10m	0.6777	0.377	0.317	3.993	4.582

Provenance

Bundle built 2026-09-09T16:25:33.395906Z from the weathergpt-models Modal volume.

Training corpora were built from Open-Meteo (multi-model NWP archives and ERA5 reanalysis), the CF standard name table, the ECMWF eccodes GRIB2 definitions, NOAA NCEP GRIB2 code tables and GFS inventories, the WRF Registry, the WMO BUFR element table, and the live SACHET/NDMA CAP feed.