Commerce-OrderIntent-OTTO-LGBM

🇰🇷 한국어 카드: MODEL_CARD.ko.md

Given the first events of an e-commerce session (3 to 10 clicks, carts or orders), this model gives the probability that the session places an order within the next 24 hours.

It is a LightGBM binary classifier over 41 hand-built features of the session prefix. It was trained and tested on the OTTO session dataset (a German online retailer, CC BY 4.0). On the sealed test split (1,089,902 sessions, scored once) its PR-AUC is 0.4128, against 0.3601 for logistic regression on the same features and 0.1101 for the base rate. On the dev split no neural model we trained for the same task beat it: DCN-v2 tied with the weaker v1 LightGBM, and DeepFM and a GRU sequence model scored lower. It applies an existing algorithm to hand-built features. It is not a new architecture and we claim no research contribution.

How to use

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("ThakiCloud/Commerce-OrderIntent-OTTO-LGBM")
sys.path.insert(0, path)
from inference import OrderIntentModel          # inference.py + oi_features.py ship in this repo

m = OrderIntentModel(path)                      # three boosters, item_stats.npz, features.json
t = 1_661_100_000                               # seconds; millisecond ts are detected automatically
sessions = [  # OTTO jsonl format, only the events seen so far (3 to 10, sorted by ts)
    {"session": 1, "events": [
        {"aid": 1460571, "ts": t,       "type": "clicks"},
        {"aid": 1460571, "ts": t + 40,  "type": "carts"},
        {"aid": 108125,  "ts": t + 300, "type": "clicks"},
    ]},
]
print(m.predict_sessions(sessions))             # session, p_order_24h, p_s0, p_s1, p_s2

predict_sessions scores each session with 3 to 10 events. The cutoff is the last event given, and the score is the mean of three seed boosters. Sessions outside that length are skipped and listed in out.attrs["skipped"]. Feature definitions are in features.json and oi_features.py; m.features_for(sessions) returns the feature table. Requires lightgbm (built with 4.7.0), numpy and pandas (requirements.txt).

python inference.py re-scores the synthetic probe (probe_input.json, generated by make_probe.py) and prints the largest difference against probe_output.json. It should print 0.0. The probe sessions are random numbers, not OTTO data.

Read this before using it on your own shop

24 of the 41 features look up OTTO item ids in item_stats.npz (prior-window click, cart and order counts, and a smoothed "seen in a prefix, then ordered" rate per item, 1,511,786 items). Item ids from another catalog will not match. Those features then take their "never seen" values, and we have not measured accuracy in that setting. For another shop, rebuild item_stats.npz from your own logs with the same definitions (oi_features.py) and retrain on your own sessions. Treat the released boosters as a reference model for OTTO-like data and as a tested feature recipe, not as a drop-in model for an unrelated catalog.

Task

• A row is one session. Its cutoff is the K-th event, with K = 3 + (session id mod 8), so K runs from 3 to 10 and does not depend on the label. Only events up to the cutoff are used.
• The label is 1 if the same session has an orders event after the cutoff and within 24 hours of it.
• Rows whose 24-hour window runs past the end of the data are dropped, never counted as negatives.
• Splits are by the UTC day of the cutoff, counted from the first day of the data. Item statistics come from days 0–6 only. Train is days 7–13, dev 15–20 and test 22 to the end, with a one-day gap so each label window closes before the next split starts. A session appears in one split only.
• Rows: train 1,742,550 (12.8% positive), dev 1,331,754 (11.8%), test 1,089,902 (11.0%).
• This is session order intent, not ad click-to-conversion (CVR) prediction.

Results

Test (sealed, scored once)

The test split was sealed until the dev decision. It was scored once with the released boosters, and nothing was changed afterwards. 95% CIs: paired Poisson bootstrap over sessions, 1,000 resamples.

Model	PR-AUC [95% CI]	ROC-AUC	LogLoss	ECE
LightGBM v2 (this model)	0.4128 [0.4100, 0.4158]	0.8306	0.2696	0.0109
Logistic regression	0.3601 [0.3572, 0.3629]	0.8122	0.2806	0.0091
Base rate (constant)	0.1101	0.5000	0.3482	0.0476

LightGBM minus logistic regression, PR-AUC: +5.27 pp [+5.11, +5.44].

Dev: comparison with the neural models

All rows use the same dev split (1,331,754 sessions, 157,707 positive). Hyper-parameters were tuned only on a validation split inside train (fit days 7–10, validation days 12–13); dev was never used for tuning or early stopping. Each "20-trial search" model got the same random-search budget. The GRU was a single pre-registered configuration with no search, which favours LightGBM. DCN-v2 and DeepFM used the v1 feature set and are compared with LightGBM v1 (dev PR-AUC 0.4164), as pre-registered. The neural models were not scored on test because none of them passed on dev.

Model	Features	Tuning	PR-AUC	ROC-AUC	LogLoss	ECE	Paired PR-AUC difference (pp) [95% CI]
LightGBM v2 (this model)	41	20-trial search, 3-seed mean	0.4187	0.8229	0.2853	0.0063	reference
LightGBM v1	31 (no item TE)	20-trial search, 3-seed mean	0.4164	0.8206	0.2865	0.0066	v1 − v2 -0.23 [-0.27, -0.19]
DCN-v2	31 (v1)	20-trial search, 3-seed mean	0.4161	0.8202	0.2865	0.0045	vs LightGBM v1 -0.03 [-0.09, +0.02]
DeepFM	31 (v1)	20-trial search, 3-seed mean	0.4134	0.8195	0.2869	0.0031	vs LightGBM v1 -0.30 [-0.36, -0.24]
GRU over the event sequence + v2 features	41 + item ids	1 config, seed 0	0.4053	0.8171	0.2884	0.0035	vs v2 -1.33 [-1.42, -1.25]
Logistic regression	41	20-trial search	0.3697	0.8049	0.2963	0.0060
Item popularity prior (1 feature)	1	none	0.2283	0.6791	0.3447	0.0162
Base rate (constant)	0	none	0.1184	0.5000	0.3642	0.0437

• DCN-v2 ties LightGBM v1 (the CI includes 0) and does not beat it. DeepFM and the GRU are below.
• The neural models give lower ECE (better calibrated probabilities), but lower PR-AUC and ROC-AUC. If you need calibrated probabilities, recalibrate this model on your own data.
• These results concern the three neural candidates we trained. They do not show that neural models cannot win on this task.

Released files

cfm-oi-v2-1018, the run behind the numbers above, saved its dev predictions but not its booster files. The released boosters were rebuilt with the recorded configuration (no search): the same rows, the same 41 features in the same order, 445 rounds, and seeds 0, 1 and 2. Before the test was opened, the rebuilt 3-seed mean was compared with the stored dev predictions row by row. The largest difference was 3.0e-08 and the PR-AUC difference was 7.6e-09, within the bound fixed in advance (1e-5 and 1e-4). The dev numbers in the table are from the rebuilt files.

• oi_lgbm_v2_s0.txt, oi_lgbm_v2_s1.txt, oi_lgbm_v2_s2.txt: LightGBM boosters (text format).
• item_stats.npz: per-item counts from OTTO days 0–6 (no sessions, no events).
• features.json, oi_features.py, inference.py: feature schema, feature code and scoring helper.
• evaluation.json: every number in this card.
• probe_input.json, probe_output.json, make_probe.py: synthetic probe and its stored scores.

Limitations

• Validated only on OTTO (one German online retailer, 4 weeks in 2022). We make no claim about other shops, countries or Korean e-commerce, and the item features are OTTO-specific (see above).
• The prefix is the first 3 to 10 events of a session, measured from the session start. Longer sessions, or prefixes that start mid-session, were not tested.
• The unit is an anonymous OTTO session id, not a customer account. There are no prices, categories or user attributes in the data.
• Item statistics come from days 0–6, so they are 1–4 weeks old at test time. Fresher statistics may change accuracy.
• One dataset and one split. The 3-seed ensemble reduces but does not measure training variance.

Training

• Data: OTTO session dataset, train file (12,899,779 sessions, 216,716,096 events), read from the Hugging Face mirror hazemessam/otto-recsys. The session and event counts match the official statistics.
• Model: LightGBM objective=binary, num_leaves 255, learning_rate 0.02075, min_child_samples 50, feature_fraction 0.548, bagging_fraction 0.946 (bagging_freq 1), lambda_l2 0.0700, 445 rounds. These were chosen by a 20-trial random search on the validation split. The 6 columns K, hour, dow, first_type, last_type and prev_type are categorical.
• Training rows: all 1,742,550 train sessions (days 7–13). CPU only.

Verification of this upload

Before upload, the shipped oi_features.py was compared with the training code on 5,000 real dev prefixes (kept off this repository) and on the synthetic probe, and gave bit-identical features. inference.py reproduced the booster ensemble exactly. After upload the repository was downloaded again into fresh caches, with and without a token. Every file hash matched, and the downloaded model reproduced probe_output.json exactly.

License and attribution

Weights and item statistics: CC BY 4.0, matching the data, with attribution below. Trained on the OTTO Recommender Systems Dataset by OTTO (GmbH & Co KG). The dataset README (github.com/otto-de/recsys-dataset) states CC BY 4.0. The official Kaggle distribution otto/recsys-dataset, owned by "Otto (GmbH & Co KG)", lists "Attribution 4.0 International (CC BY 4.0)". Our copy came from a third-party Hugging Face mirror (hazemessam/otto-recsys), whose session and event counts match the original. item_stats.npz holds per-item counts derived from that data. This repository contains no OTTO sessions or events, and probe_input.json is synthetic.