Debk/saarthi-default-prediction

SAARTHI — MSME Default Prediction

Credit default-prediction models trained entirely on public datasets for IDBI Innovate 2026 (Track 04, Default Prediction Model). No bank data was used at any point.

The headline design choice: SAARTHI ships pre-trained and then fine-tunes on a lender's own book, rather than cold-starting on every upload.

Methodology — why these numbers are trustworthy

Every metric below comes from a strict three-way split:

Fold	Share	Used for
fit	60%	training the boosters
calibrate	15%	fitting the isotonic calibrator only
test	25%	never seen by either — all reported metrics

This matters: a common shortcut fits the probability calibrator and then measures calibration error on that same fold, which drives ECE artificially toward zero. Here the calibrator never sees the test fold, so the reported ECE is a real out-of-sample calibration estimate.

Per-dataset specialist models

Full native feature set per corpus. Ensemble of LightGBM + XGBoost + CatBoost, isotonic-calibrated.

Dataset	n	Default rate	Test AUC	PR-AUC	ECE	Published reference
sba	897,167	0.1756	0.9800	0.9126	0.0015	~0.95 (literature)
lending_club	1,369,566	0.2124	0.7263	0.4104	0.0024	~0.70-0.73 (leakage-free)
home_credit	307,511	0.0807	0.7632	0.2449	0.0011	0.805 (Kaggle winner)
gmsc	150,000	0.0668	0.8522	0.3629	0.0041	~0.87 (Kaggle winner)
taiwan	30,000	0.2212	0.7698	0.5243	0.0074	~0.78 (literature)
german	1,000	0.3000	0.7668	0.5505	0.0680	~0.79 (literature)
berka	682	0.1111	0.8558	0.5725	0.0295	n/a
amex	120,000	0.2598	0.9580	0.8858	0.0050	~0.96 AUC (winners; the 0.80 headline is a different metric)
hc2024	—	—	failed	—	—	~0.86 (Kaggle winner)

• sba — US Small Business Administration loans - closest public analogue to MSME lending
• lending_club — 2007-2018 consumer loans, charge-off label
• home_credit — Home Credit 2018 application + bureau aggregates
• gmsc — Give Me Some Credit, 90+ DPD within 2 years
• taiwan — Taiwan credit-card default, 6-month repayment panel
• german — UCI German Credit, 1000 rows
• berka — PKDD'99 Czech bank, pre-origination transaction aggregates
• amex — American Express monthly statement panel
• hc2024 — Home Credit 2024 model-stability competition

Pooled global model (ships with the app)

Trained across corpora in a shared 15-field canonical credit vocabulary so it can score any loan book that maps onto it.

• Test AUC: 0.8605
• PR-AUC: 0.5795
• ECE: 0.0018 · Brier: 0.0868
• Test rows: 270,421
• Corpora pooled: sba, lending_club, home_credit, gmsc, taiwan, german, berka

Leave-one-dataset-out transfer — a negative result

Train on every corpus except one, then score the held-out corpus cold. This is the honest proxy for "will it transfer to a book it has never seen?" — the question that actually matters for deployment.

It does not transfer. Most hold-outs land at or below chance, which means the pooled model is not merely uninformative on an unseen corpus, it is anti-predictive: the feature→outcome relationships invert across lending domains.

Held-out corpus	n	Raw pooling	Rank-normalised
berka	682	0.4612	0.5155
german	1,000	0.4646	0.4022
gmsc	150,000	0.7786	0.7711
home_credit	300,000	0.4991	0.5785
lending_club	300,000	0.5228	0.5399
sba	300,000	0.4103	0.2574
taiwan	30,000	0.7038	0.6989
mean		0.5486	0.5377

Normalisation did NOT rescue it

The obvious hypothesis is scale mismatch — these corpora are denominated in USD, DM, NT$ and CZK, so a "loan amount" of 50,000 means different things in each. Converting every numeric feature to its within-corpus percentile rank tests that hypothesis directly, and it fails: mean transfer AUC moved 0.5486 → 0.5377, i.e. no better. SBA in particular degrades from 0.4103 to 0.2574.

The mechanism is therefore relational inversion, not scale. SBA is small-business lending, where a longer term and a larger SBA-guaranteed principal typically indicate a better-vetted, collateral-backed loan — the opposite of the consumer-credit corpora that dominate the pooled training set. Rank-normalising preserves that inverted ordering perfectly, which is why it cannot help.

Practical consequence: a single pooled "foundation" model for credit risk is not supported by this evidence. Domain-matched training plus fine-tuning on the lender's own book is the defensible architecture, which is what SAARTHI does.

Ablation — does transaction-as-language earn its place?

A CoLES-style contrastive encoder (GRU + InfoNCE over disjoint sub-sequence views) was pre-trained on unlabelled transaction streams, then its 256-d embedding was tested against the tabular features.

Corpus	Tabular	Sequence only	Tabular + sequence	Lift
berka	0.8575	0.8097	0.8842	+0.0267
amex	0.9590	0.9156	0.9585	-0.0006

When the sequence encoder earns its place: on Berka (16 tabular features, raw bank transactions) it adds +0.027 AUC. On Amex (941 hand-engineered aggregates over the same statement data) it adds nothing — the aggregates already capture what the encoder learns. The embedding substitutes for feature engineering rather than adding to it; it is most valuable exactly where a lender has raw transaction streams but few curated fields, which is the realistic MSME case.

Note also that on Amex the 256-d embedding alone reaches 0.9156 AUC with no hand-engineered features at all, versus 0.9590 for 941 engineered columns.

Linear probe on frozen embeddings

Logistic regression on the frozen encoder output — measures how much default signal the unsupervised embedding captured on its own.

• amex: AUC 0.8717 (n=60,000)
• berka: AUC 0.7783 (n=682)

Known limitations

• Berka aggregates only pre-origination transactions. Using the full history yields AUC 1.0000 because 71% of an account's transactions occur after the loan date and encode the repayment behaviour that defines the label. The leaked figure is not reported here.
• Amex and Home Credit are Kaggle-competition datasets; they are used for research validation. The shipped global model is trained on the permissively-licensed core corpora.
• The pooled model uses a deliberately small shared vocabulary, so its AUC is lower than the specialists by construction. Its purpose is transfer, not peak in-corpus accuracy.
• No Indian MSME data was available; GST-behavioural fields are simulated against the schema IDBI published, not learned from real filings.

Files

File	What it is
global_canonical.joblib	pooled global model + isotonic calibrator
specialist_<corpus>.joblib	per-corpus specialist ensembles
coles.pt	transaction-sequence encoder checkpoint
metrics.json	every number above, machine-readable
ablation_sequence.json	sequence-embedding ablation

Usage

import joblib, pandas as pd
b = joblib.load('global_canonical.joblib')
X = df.reindex(columns=b['features'])          # canonical vocabulary
raw = sum(m.predict_proba(X)[:, 1] for m in b['members'].values()) / len(b['members'])
pd_calibrated = b['calibrator'].predict(raw)   # real probability of default

---

Generated 2026-08-31 from metrics files — numbers are not hand-entered.