IEEE-CIS Fraud Detection
A lightgbm model that scores card transactions for fraud risk, trained on the IEEE-CIS Fraud Detection dataset.
Full project: https://github.com/Dee-ui/ieee-cis-fraud-detection
What is in this repository
| File | Contents |
|---|---|
feature_engineer.joblib |
The fitted transformer. Turns a raw transaction into 284 features. |
final_model.joblib |
The trained model. |
final_model_metadata.json |
Feature list, threshold, and the scores it was measured at. |
Both files are needed. The model expects features in one exact order, which only the transformer produces.
How it performs
Measured on a held-out period that comes strictly after everything it was trained on, 2018-04-20 to 2018-05-31.
| Metric | Baseline | This model |
|---|---|---|
| PR-AUC | 0.0344 | 0.6068 |
| Cross-validated PR-AUC | - | 0.6334 |
Operating threshold 0.4222, chosen by a cost model at a 2% manual review capacity, not left at the default 0.5.
Using it
import joblib, json, pandas as pd
from huggingface_hub import hf_hub_download
repo = "Dee-ui/ieee-cis-fraud-detector"
engineer = joblib.load(hf_hub_download(repo, "feature_engineer.joblib"))
model = joblib.load(hf_hub_download(repo, "final_model.joblib"))
metadata = json.load(open(hf_hub_download(repo, "final_model_metadata.json")))
# A raw transaction. Any column you leave out is treated as unknown.
transaction = pd.DataFrame([{
"TransactionID": 3663549, "TransactionDT": 18403224,
"TransactionAmt": 31.95, "ProductCD": "W", "card1": 10409,
}])
features = engineer.transform(transaction)
probability = model.predict_proba(features[metadata["feature_names"]])[:, 1][0]
print(probability, "review" if probability >= metadata["chosen_threshold"] else "pass")
Limitations worth knowing
- It catches about 44.6% of fraud by count but only 31.2% by value. Missed frauds average $186 against $105 for caught ones. Do not estimate savings by multiplying recall by total fraud losses.
- Roughly 10% of its decision weight sits on features derived from a customer fingerprint that is unavailable for about 82% of transactions in the later test period. Performance on data far from the training window should be monitored, not assumed.
- Trained on 2017 to 2018 data. Fraud patterns move.
Licence
MIT.