Commerce-Repurchase-OTTO-LGBM

🇰🇷 한국어 카드: MODEL_CARD.ko.md

Given the click, cart and order events of an e-commerce session up to a cutoff day, this model gives the probability that the session places another order within the next 7 days.

It is a LightGBM binary classifier over 29 features: 14 built from purchase history (recency, frequency, gaps) and 15 from browsing behaviour (clicks, carts, active days). It was trained and tested on the OTTO session dataset (German online retailer, CC BY 4.0). On three out-of-time test cutoffs and three seeds, its repurchase AUC beat the BG/NBD model fitted on the same history in all nine cells, with the paired 95% CI lower bound at least +0.054. It applies an existing algorithm (LightGBM gradient-boosted trees) to hand-built features. It is not a new architecture and we claim no research contribution.

How to use

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("ThakiCloud/Commerce-Repurchase-OTTO-LGBM")
sys.path.insert(0, path)
from inference import RepurchaseModel          # inference.py + otto_features.py ship in this repo

m = RepurchaseModel(path)                      # loads the three boosters and features.json
day = 86_400                                   # ts in seconds here; millisecond ts are detected automatically
sessions = [  # OTTO jsonl format: {"session", "events": [{"aid", "ts", "type"}]}
    {"session": 1, "events": [
        {"aid": 11, "ts": 19230 * day + 3600, "type": "clicks"},
        {"aid": 11, "ts": 19230 * day + 3700, "type": "carts"},
        {"aid": 11, "ts": 19230 * day + 3900, "type": "orders"},
        {"aid": 42, "ts": 19236 * day + 600, "type": "clicks"},
    ]},
]
print(m.predict_sessions(sessions, cutoff_day=19238))   # cutoff_day = days since 1970-01-01 UTC

predict_sessions returns one row per session with at least one order before cutoff_day: p_repurchase_7d (the mean of the three seed boosters) and p_s0, p_s1, p_s2. Only events on days before the cutoff are read; the label it predicts is "at least one orders event on days cutoff_day to cutoff_day + 6". Sessions without a past order are not scored. Feature definitions are in features.json; m.features_for(...) returns the feature table. Requires lightgbm, numpy and pandas.

python inference.py re-scores the synthetic probe (probe_input.json, generated by make_probe.py) and prints the largest difference against probe_output.json. It should print 0.0. The probe sessions are random numbers, not OTTO data.

Intended use: ranking, not absolute probability

Use the score to rank sessions by repurchase risk (who is most and least likely to order again), not as a calibrated probability. On test cutoff 20 the released ensemble has ECE 0.035 against 0.015 for BG/NBD, and its mean predicted probability is 0.173 while the observed rate is 0.137. If you need probabilities, recalibrate on your own data (for example isotonic or Platt scaling on a held-out period).

Results

Task: at cutoff day c (day index from the start of the OTTO train file), score every session with an order before c; the label is an order within days c to c+6. Test cutoffs are 14, 17 and 20. Training rows for cutoff T come only from earlier cutoffs whose 7-day label window closed before T, so the split is out-of-time (the same sessions can appear in training and test at different cutoffs). Baselines are fitted on the history of the same test cutoff.

The pass rule was registered before the run: LightGBM (+behaviour) minus BG/NBD repurchase AUC, paired user-level bootstrap (500 resamples) 95% CI lower bound > 0 in all 9 cells (3 cutoffs × 3 seeds). Each cell uses that seed's single booster. Result: 9 of 9.

Cutoff	Sessions	BG/NBD AUC	LightGBM AUC (seed 0 / 1 / 2)	ΔAUC vs BG/NBD, 95% CI (seed 0 / 1 / 2)
14	862,011	0.6103	0.6677 / 0.6706 / 0.6665	+0.0574 [0.0557, 0.0591] / +0.0603 [0.0586, 0.0621] / +0.0562 [0.0544, 0.0579]
17	1,059,403	0.6196	0.6914 / 0.6911 / 0.6929	+0.0719 [0.0704, 0.0734] / +0.0715 [0.0700, 0.0731] / +0.0734 [0.0719, 0.0749]
20	1,208,863	0.6302	0.6911 / 0.6920 / 0.6928	+0.0609 [0.0595, 0.0623] / +0.0618 [0.0604, 0.0632] / +0.0626 [0.0612, 0.0640]

Pooled over the three cutoffs (3,130,277 rows, 14.7% positives). The LightGBM rows use the mean prediction of the three seeds, as in the pre-registered primary metric. 95% CIs from the same user-level bootstrap.

Model	AUC	PR-AUC	ECE (lower is better)
RFM cell table (recency × frequency quintiles; OTTO has no prices)	0.5683	0.1776	0.019
Purchase rate × horizon	0.5771	0.1719	n/a
BG/NBD (lifetimes)	0.6235	0.2598	0.009
Pareto/NBD (lifetimes)	0.6243	0.2598	n/a
LightGBM, purchase features only	0.6221	0.2555	0.027
LightGBM, purchase + behaviour (this model)	0.6861	0.3014	0.030

Paired differences: this model vs BG/NBD AUC +0.0626 [0.0615, 0.0636], PR-AUC +0.0416 [0.0407, 0.0426]; vs Pareto/NBD AUC +0.0618 [0.0607, 0.0628]. LightGBM with purchase features only vs BG/NBD AUC −0.0014 [−0.0017, −0.0011].

Released ensemble (cutoff 20, 3-seed mean)

The repository ships the three cutoff-20 boosters (seeds 0, 1, 2; the most recent training data), and inference.py averages their probabilities. We re-measured exactly these files, with no training, on the pre-registered test cutoff 20 (1,208,863 sessions, 13.7% positives). Nothing was selected: it is the same test cutoff and the same rows as above. The ensemble AUC equals the earlier measurement of the 3-seed mean at cutoff 20 (difference 0.0). CIs: user-level bootstrap, 500 resamples, the same resampling as the 9-cell table. These numbers describe the released files and are separate from the 9-cell pass rule.

Released ensemble vs BG/NBD, cutoff 20	Ensemble	BG/NBD	Paired difference
AUC	0.6921 [0.6907, 0.6935]	0.6302 [0.6286, 0.6317]	+0.0619 [0.0605, 0.0632]
PR-AUC	0.2988 [0.2966, 0.3008]	0.2545 [0.2525, 0.2564]	+0.0443 [0.0428, 0.0456]
ECE (lower is better)	0.035	0.015	n/a

Single-seed AUCs of the three files on cutoff 20: 0.6911, 0.6920 and 0.6928.

Limitations

• Probabilities are less well calibrated than BG/NBD: ECE 0.030 vs 0.009 pooled over the three cutoffs, 0.035 vs 0.015 for the released ensemble on cutoff 20. The advantage is in ranking (AUC, PR-AUC). If you need calibrated probabilities, recalibrate on your own data.
• The gain comes from the behaviour features. With purchase features only, LightGBM is about equal to, and slightly below, BG/NBD (AUC −0.0014).
• Validated only on OTTO (one German online retailer, 4 weeks). We make no claim about other shops, countries or Korean e-commerce.
• The unit is an anonymous OTTO session id, which can last up to 4 weeks. It is not a customer account.
• The cutoff-20 boosters were trained on cutoffs 7 to 13, so each training row saw at most 13 days of history; the test cutoffs saw at most 20. Longer histories were not tested.
• Training rows were capped at 400,000 (random sample of the eligible rows). BG/NBD and Pareto/NBD were fitted on samples of at most 20,000 and 8,000 sessions.
• Secondary survival analysis (not shipped here) used capped samples (15,000 train / 2,500 evaluation sessions) and 100 bootstrap resamples instead of the registered 500. A discrete-time hazard model with the same features beat BG/NBD on Uno's C (+0.060 [0.031, 0.088]); the integrated Brier score difference was not significant.
• No claim of state of the art, of a new architecture or of a research contribution. It is an existing algorithm (LightGBM) on hand-built RFM and engagement features, released so others can reuse it.

Training

• Data: OTTO session dataset, train file (12,899,779 sessions, 216,716,096 events, 4 weeks). Read from the Hugging Face mirror hazemessam/otto-recsys; session and event counts matched the official README.
• Features: otto_features.py (copied from the training code; the release job checked that it gives identical features on the probe). Day = floor(ts / 1 day), UTC.
• Model: LGBMClassifier(n_estimators=250, learning_rate=0.03, num_leaves=15, min_child_samples=50, subsample=0.8, subsample_freq=1, colsample_bytree=0.8, reg_lambda=5.0), seeds 0, 1, 2, no tuning. LightGBM 4.7.0.
• Hardware: CPU only (8 cores); the full run with all baselines and bootstraps took about 72 minutes.

Verification of this upload

The booster files are the ones written by the measurement run itself (no retraining); their SHA-256 hashes match the run's manifest. Before upload, the shipped feature code was compared with the training code on the synthetic probe (bit-identical features), and inference.py reproduced a reference scoring path. After upload the repository was downloaded again, with and without a token, into fresh caches; every file hash matched and the downloaded model reproduced probe_output.json exactly.

License and attribution

Weights: CC BY 4.0, matching the data, with attribution below. Trained on the OTTO Recommender Systems Dataset by OTTO (GmbH & Co KG). The dataset README (github.com/otto-de/recsys-dataset) states CC BY 4.0, and the official Kaggle distribution otto/recsys-dataset, owned by "Otto (GmbH & Co KG)", lists the license "Attribution 4.0 International (CC BY 4.0)" in its metadata. Kaggle competition rules govern the competition data; this dataset is distributed separately by its owner under CC BY 4.0. Our copy came from a third-party Hugging Face mirror (hazemessam/otto-recsys, which has no license field); its session count (12,899,779) and event count (216,716,096) match the original statistics exactly. No OTTO events are included in this repository; probe_input.json is synthetic.