Model card - SENTINEL fraud classifier

Generated by python -m src.train from model/metrics.json. Do not edit by hand.

Model 20261107-184917 - strategy scale_pos_weight, calibration sigmoid, decision threshold 0.0717 (cost-optimal: FN cost 111.58, FP cost 5.00).

Data: 284,807 rows / 492 frauds - time-ordered split train/val/test = 170,884 / 56,961 / 56,962 rows (360 / 57 / 75 frauds). Source: user-provided dataset.

Held-out test split, mean [95% bootstrap CI]

Metric	Value
PR-AUC (primary)	0.790 [0.692, 0.870]
ROC-AUC	0.981 [0.968, 0.992]
Recall at precision >= 80%	0.760 [0.656, 0.848]
Precision @ threshold	0.760 [0.662, 0.851]
Recall @ threshold	0.760 [0.656, 0.848]
F1 @ threshold	0.760 [0.681, 0.828]
Expected cost on test	2,098 [1,236, 3,098]

Confusion at threshold: TN 56,869, FP 18, FN 18, TP 57. Cost baselines on test: flag nothing = 8,368; fixed 0.5 threshold = 2,140; chosen threshold = 2,098.

5-fold CV on the training split (model selection)

Strategy	PR-AUC (mean +- sd)	Recall @ precision
none	0.876 +- 0.032	0.867
scale_pos_weight (selected)	0.878 +- 0.034	0.875
smote	0.866 +- 0.026	0.861

Intended use

Ranking and flagging card transactions for human review in a demo / research setting. It is not validated for automated blocking, credit decisions, or any regulated production use.

Model

XGBoost gradient-boosted trees on V1-V28 (anonymised PCA components), log-amount (standardised, fitted on train only) and hour-of-day (sin/cos). Imbalance handled by scale_pos_weight, chosen by CV PR-AUC on the training split only.

Calibration and decision rule

Method: sigmoid, fitted on the validation split. Brier on test: raw 0.00071 -> calibrated 0.00038. Mean predicted fraud rate on test: raw 0.0031, calibrated 0.0013, observed 0.0013. Reliability plot: docs/figures/reliability.png. A transaction is flagged when calibrated probability >= 0.0717, the threshold minimising FN * 111.58 + FP * 5.00 on the validation split. Risk bands: LOW < t/2 <= MEDIUM < t <= HIGH.

Top features (mean |TreeSHAP|, log-odds)

V4: 1.721
V14: 1.527
V12: 1.222
V10: 0.819
V11: 0.791
V8: 0.504
V1: 0.467
V17: 0.421
V24: 0.390
V19: 0.361

Limitations and known failure modes

Test frauds are few (75); see the confidence intervals above - they are wide.
Time-ordered split: performance on later data reflects drift, but only over the span of this dataset (about 2 days for the Kaggle set).
Fraud patterns change; the model will degrade without retraining. /monitoring/drift reports PSI on live inputs but cannot detect label drift.
Features are anonymised PCA components, so explanations (reason codes) describe the model, not a real-world cause.
Threshold depends on the configured costs; change config/train.yaml costs and re-train if your costs differ.
Single-process in-memory monitoring and rate limiting; not suitable for multi-replica deployment as-is.
No fairness / bias assessment is possible: the dataset has no demographic attributes.

Regulatory note

This project provides per-transaction reason codes and a documented evaluation. It does not by itself satisfy GDPR Art. 22 or EU AI Act requirements (which also need governance, human oversight, logging/retention, DPIA, conformity assessment, etc.). Treat it as a technical building block, not a compliance claim.

Provenance

Data sha256 76274b691b16a6c4..., trained 2026-11-07T18:49:17.598505+00:00, libraries {'python': '3.14.8', 'xgboost': '3.4.1', 'numpy': '2.5.4', 'pandas': '3.0.6'}.

How to use these files

The model is stored without pickle: xgb_model.json (XGBoost native), preprocess.json (scaler parameters), calibrator.json (probability calibrator) and metadata.json (feature order, decision threshold, costs, checksums). Load them with ModelBundle.load() from the Sentinel source code (src/model_bundle.py in the GitHub repository).

Data attribution: Kaggle Credit Card Fraud Detection (ULB / Worldline), DbCL v1.0 — see NOTICE.md.

Downloads last month: -; Downloads are not tracked for this model. How to track