LTM v2: open-source large tabular model
A 2.4M-parameter transformer pretrained on synthetic tables. It learns in context: you give it labelled rows and it predicts new rows, with no gradient training. Early research prototype.
- Code: https://github.com/redmona22/ltm
- Demo: https://huggingface.co/spaces/ciphemic/ltm-demo
- License: Apache-2.0
How to use
Download ltm_v2.pt from this repository, install the code from GitHub, then:
from ltm import LTMClassifier
clf = LTMClassifier(checkpoint="ltm_v2.pt").fit(X_train, y_train)
pred = clf.predict(X_test)
Training data
Only synthetic tables, sampled from random structural causal models (see ltm/prior.py in the code repository). No real datasets were used for training, and the evaluation datasets below were never seen. Training ran in two stages on a free Colab T4: 20,000 steps on tables up to 600 rows and 40 features, then 15,000 more steps on tables up to 1000 rows and 64 features.
Evaluation
Versus scikit-learn default HistGradientBoosting. Each dataset is subsampled to at most 1500 rows, split 70/30 at random, and averaged over 3 splits. LTM used 400 context rows.
| Dataset | Task | LTM v2 | HistGB |
|---|---|---|---|
| iris | accuracy | 0.985 | 0.970 |
| wine | accuracy | 0.994 | 1.000 |
| breast_cancer | accuracy | 0.953 | 0.953 |
| digits | accuracy | 0.692 | 0.965 |
| credit-g | accuracy | 0.741 | 0.776 |
| diabetes (Pima) | accuracy | 0.756 | 0.756 |
| blood-transfusion | accuracy | 0.800 | 0.751 |
| vehicle | accuracy | 0.730 | 0.781 |
| ilpd | accuracy | 0.716 | 0.703 |
| banknote-authentication | accuracy | 0.995 | 0.990 |
| phoneme | accuracy | 0.773 | 0.861 |
| segment | accuracy | 0.835 | 0.975 |
| diabetes (sklearn) | R² | 0.369 | 0.310 |
| california | R² | 0.644 | 0.736 |
| Mean, 12 classification datasets | accuracy | 0.831 | 0.873 |
| Mean, 2 regression datasets | R² | 0.506 | 0.523 |
Limitations
- Behind gradient boosting on larger tables, many-feature tables and multi-class problems (digits, segment, phoneme, california).
- At most 10 classes and 128 features.
- Compared only with default HistGradientBoosting, not tuned LightGBM or CatBoost, and not TabPFN.
- Small benchmark (14 datasets, 3 splits each); differences of a point or two are within noise.
- Trained on synthetic data only, so behaviour on very different real-world tables is untested.