LEAP-LAB-KUS/leap-kt-atkt-2021-08

leap-kt · ATKT

ATKT —

Part of leap-kt-toolkit, a systematic re-implementation of published Knowledge Tracing models under one protocol. This repository holds every fold of every dataset this model has been run on, with the per-epoch training logs and the exact user split alongside the checkpoints.

Protocol

User-level 80/20 train/test split · 5-fold cross-validation over the training portion · held-out fold as validation · early stopping patience 10 on validation AUC · max 200 epochs.

Every cell in the project runs under identical settings; a cell that cannot is recorded as a documented failure rather than re-run under bespoke settings.

Results

dataset	AUC	ACC	F1	published reference	delta
algebra2005 *	0.8243 ± 0.0001	0.8180	0.8871	—	—
assist2009 *	0.7647 ± 0.0010	0.7386	0.8183	—	—
assist2015	0.7268 ± 0.0079	0.7519	0.8484	—	—
bridge2algebra2006	0.7926 ± 0.0012	0.8494	0.9150	0.7889	+0.0037
dbe_kt22	0.7989 ± 0.0011	0.7960	0.8755	—	—
ednet500	0.6677 ± 0.0034	0.6839	0.7888	—	—

* incomplete cells

• algebra2005 ran 4/5 folds, not the full protocol. The mean is over the folds that completed, and the ones it lost are the ones the model failed on — treat it as incomplete, not as a result.
• assist2009 ran 3/5 folds, not the full protocol. The mean is over the folds that completed, and the ones it lost are the ones the model failed on — treat it as incomplete, not as a result.

Per-fold values are in each dataset's summary.json. The mean is never reported without the spread — 0.75 ± 0.001 and 0.75 ± 0.09 are different claims.

Why these numbers may differ from other reproductions

Multi-concept questions are not expanded into multiple rows. Toolkits that do expand them place consecutive test positions carrying the same question and the same response, so a model is shown the answer one step before predicting it; on ASSIST2009 that is around 37% of positions and lifts DKT from a published ~0.75 to ~0.89 AUC. Here concepts are an extra axis on the interaction rather than extra rows, so the leak is not expressible and every interaction is scored exactly once.

Cells carrying a published reference value are additionally leak-audited before release: train/test user disjointness, no window crossing the split boundary, exactly-once scoring, and a label-shuffle control that must collapse AUC to chance. Cells with no comparable published number rely on the structural guarantee above rather than on that audit.

Files

<dataset>/summary.json            mean ± std and per-fold AUC
<dataset>/split.json              the exact user partition, with a checksum
<dataset>/fold<k>/checkpoint/     config.json + weights
<dataset>/fold<k>/epochs.jsonl    every epoch's train loss and validation metrics;
                                  each row carries its own model/dataset/fold
<dataset>/fold<k>/run.json        protocol and package version for that run

Provenance

Produced by leap-kt at commit(s) a67fa76, cab9da9, f8dc2a0.