Tydra-4HT

Tydra-4HT is a compact hybrid tabular foundation model for in-context classification. It alternates four bidirectional Hydra state-space layers with four Transformer layers, combining efficient sequence processing with the predictive strength of attention.

Tydra learns from the labeled rows supplied at prediction time. It does not fine-tune on the target dataset: fit() stores the training rows as context, and the pretrained network predicts the query rows in a single forward pass.

The accompanying paper reports approximately 30% faster inference than TabPFN across 30 OpenML datasets while retaining much of its predictive performance. See Tydra: An Efficient Hybrid Model for Tabular Data for the complete evaluation protocol and dataset-level results.

Model summary

Property	Value
Task	Tabular classification
Architecture	[Hydra, Transformer] × 4
Parameters	15,999,700
Representation size	512
Attention heads	4
Maximum input features	100
Maximum classes	10
Recommended maximum context rows	1,000
Training data	Fully synthetic
Target-dataset fine-tuning	Not required
Weight format	SafeTensors (FP32)

Requirements

The current Hydra implementation uses Triton CUDA kernels. Inference requires:

• Linux;
• an NVIDIA CUDA GPU;
• a CUDA-enabled PyTorch installation;
• Python 3.8 or newer.

CPU and macOS inference are not currently supported. Tydra uses a custom PyTorch architecture and cannot be loaded with transformers.pipeline() or a standard Hugging Face inference provider.

Installation

Create an isolated environment and install a PyTorch build compatible with your CUDA version:

python3 -m venv .venv
source .venv/bin/activate

# Choose the CUDA-specific PyTorch command for your system at pytorch.org.
python3 -m pip install torch
python3 -m pip install "causal-conv1d>=1.4.0" "mamba-ssm==2.2.2" --no-build-isolation

# Skip unrelated Git LFS training artifacts; model weights come from this Hub repo.
GIT_LFS_SKIP_SMUDGE=1 python3 -m pip install \
  "git+https://github.com/Mieszko-sw/Structured-State-Space-Models-for-PFNs-private.git"

Verify the installation:

python3 -c "from tydra import TydraClassifier; print(TydraClassifier())"

Recent PyTorch and Google Colab

The pinned mamba-ssm==2.2.2 was written for older software stacks and can fail to build or import on the newest ones. Current Google Colab (Python 3.13, PyTorch 2.11 with CUDA 13, Triton 3.6, transformers 5.x) is one example.

Tydra only uses the Triton kernels and pure-Python modules of mamba-ssm, so it does not need the compiled CUDA extension. The steps below install mamba-ssm without it and add the small compatibility patches needed for newer PyTorch and transformers releases. Select a GPU runtime, then run the steps in order, each in its own Colab cell. The same steps are available as a ready-made notebook.

• Verified on Google Colab with an NVIDIA TODO_GPU_MODEL GPU, by two users. This path has not been checked on other platforms.
• Quick-start result in that run: accuracy 0.9719, AUROC 0.9945.
• The first prediction takes one to two minutes while Triton compiles the kernels. Later calls are fast (about 0.2 s for the quick-start example).

Full Colab setup (click to expand)

1. Check the environment

import sys, torch, triton, transformers

print("Python       :", sys.version.split()[0])
print("torch        :", torch.__version__, "| CUDA", torch.version.cuda)
print("triton       :", triton.__version__)
print("transformers :", transformers.__version__)
assert torch.cuda.is_available(), "No GPU. Runtime > Change runtime type > select a GPU, then re-run."
print("GPU          :", torch.cuda.get_device_name(0), "| compute capability", torch.cuda.get_device_capability(0))

# Remember the torch build so we can detect if a later install replaces it.
TORCH_BEFORE = torch.__version__

2. Install mamba-ssm without the CUDA extension

%%time
%env MAMBA_SKIP_CUDA_BUILD=TRUE
!pip install -q ninja packaging wheel einops
!pip install "mamba-ssm==2.2.2" --no-build-isolation --no-deps
%env MAMBA_SKIP_CUDA_BUILD=FALSE

3. Add a stub for the Mamba-1 CUDA scan (Tydra does not use it)

import pathlib, sysconfig

stub = pathlib.Path(sysconfig.get_paths()["purelib"]) / "selective_scan_cuda.py"
stub.write_text(
    "fwd = bwd = lambda *a, **k: (_ for _ in ()).throw("
    "NotImplementedError('selective_scan_cuda is a stub: the Mamba-1 CUDA scan was not built'))"
)
print("wrote", stub)

4. Install Tydra

%env GIT_LFS_SKIP_SMUDGE=1
!pip install "git+https://github.com/Mieszko-sw/Structured-State-Space-Models-for-PFNs-private.git"

5. Compatibility patches and verification (run this before importing Tydra, in every fresh runtime)

import importlib, torch
import transformers.generation as _g

for _name in ("GreedySearchDecoderOnlyOutput", "SampleDecoderOnlyOutput"):
    if not hasattr(_g, _name):
        setattr(_g, _name, _g.GenerateDecoderOnlyOutput)

# torch 2.11 no longer re-exports `Optional` (and possibly other names) from
# torch.nn.modules.transformer, but tabpfn/layer.py imports them from there.
import typing
import torch.nn as nn
import torch.nn.functional as F
import torch.nn.modules.transformer as _tr

for _name, _val in {
    "Optional": typing.Optional, "Module": nn.Module, "Tensor": torch.Tensor,
    "MultiheadAttention": nn.MultiheadAttention, "Linear": nn.Linear,
    "Dropout": nn.Dropout, "LayerNorm": nn.LayerNorm,
}.items():
    if not hasattr(_tr, _name):
        setattr(_tr, _name, _val)
if not hasattr(_tr, "_get_activation_fn"):
    _tr._get_activation_fn = lambda a: {"relu": F.relu, "gelu": F.gelu}[a] if isinstance(a, str) else a

# Fail here (with the real error) instead of inside TydraClassifier's generic message:
import tabpfn.encoders, tabpfn.hybrid
print("tabpfn model code imports OK")

if "TORCH_BEFORE" in globals() and torch.__version__ != TORCH_BEFORE:
    print(f"WARNING: torch changed {TORCH_BEFORE} -> {torch.__version__}.")
else:
    print("torch:", torch.__version__)

import mamba_ssm
print("mamba_ssm", mamba_ssm.__version__, "imported")

# Exactly the symbols Tydra's code imports from mamba_ssm:
needed = {
    "mamba_ssm.ops.triton.ssd_combined": ["mamba_chunk_scan_combined", "_mamba_chunk_scan_combined_fwd", "_mamba_chunk_scan_combined_bwd"],
    "mamba_ssm.ops.triton.layernorm_gated": ["RMSNorm", "_layer_norm_fwd", "_layer_norm_bwd"],
    "mamba_ssm.ops.triton.layer_norm": ["RMSNorm", "layer_norm_fn"],
    "mamba_ssm.modules.mamba_simple": ["Mamba"],
    "mamba_ssm.modules.mamba2": ["Mamba2"],
    "mamba_ssm.modules.block": ["Block"],
    "mamba_ssm.modules.mlp": ["GatedMLP"],
    "mamba_ssm.modules.mha": ["MHA"],
}
for mod, names in needed.items():
    m = importlib.import_module(mod)
    missing = [n for n in names if not hasattr(m, n)]
    print(f"{mod:40s}", "OK" if not missing else f"MISSING {missing}")

try:
    import causal_conv1d
    print("causal_conv1d", causal_conv1d.__version__, "OK")
except Exception as e:
    print("causal_conv1d import failed (only needed for some code paths):", type(e).__name__, e)

from tydra import TydraClassifier
print(TydraClassifier())

6. Quick-start smoke test

import time
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.model_selection import train_test_split
from tydra import TydraClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.5, random_state=42, stratify=y
)

classifier = TydraClassifier(
    repo_id="Mieszko-Komisarczyk/Tydra",
    device="auto",
    n_ensemble_configurations=4,
    random_state=42,
)

# fit() only stores labeled context; it does not update the pretrained weights.
classifier.fit(X_train, y_train)

t0 = time.time()
probabilities = classifier.predict_proba(X_test)
first_call = time.time() - t0
predictions = classifier.classes_[np.argmax(probabilities, axis=1)]

t0 = time.time()
classifier.predict_proba(X_test)
second_call = time.time() - t0

print(f"Accuracy: {accuracy_score(y_test, predictions):.4f}")
print(f"AUROC   : {roc_auc_score(y_test, probabilities[:, 1]):.4f}")
print(f"predict_proba: first call {first_call:.2f}s (includes Triton compile), second call {second_call:.2f}s")

Quick start

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.model_selection import train_test_split
from tydra import TydraClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.5,
    random_state=42,
    stratify=y,
)

classifier = TydraClassifier(
    repo_id="Mieszko-Komisarczyk/Tydra",
    device="auto",  # Selects CUDA; "cuda:1" selects a specific GPU.
    n_ensemble_configurations=4,
    random_state=42,
)

# fit() stores labeled context. It does not update the pretrained weights.
classifier.fit(X_train, y_train)

probabilities = classifier.predict_proba(X_test)
predictions = classifier.classes_[np.argmax(probabilities, axis=1)]

print(f"Accuracy: {accuracy_score(y_test, predictions):.4f}")
print(f"AUROC: {roc_auc_score(y_test, probabilities[:, 1]):.4f}")

TydraClassifier follows the scikit-learn estimator interface and can also be used directly with classifier.predict(X_test).

Input format

• X_train and X_test must be two-dimensional numeric arrays.
• Prediction columns must have the same number and order as the context data.
• The model supports at most 100 features and 10 target classes.
• Up to 1,000 labeled context rows are recommended. Longer contexts may run, but were outside the checkpoint's intended training regime and can be slower or less reliable.
• Missing values represented by NaN are supported. Infinite values are not.
• Categorical values must first be numerically encoded. Pass their column indices through categorical_features so they are excluded from the power transformation used by the default preprocessing ensemble.

Example with categorical-column indices:

classifier = TydraClassifier(categorical_features=[0, 3, 7])
classifier.fit(X_train, y_train)

Useful options

Argument	Default	Description
device	"auto"	Select CUDA automatically or choose a device such as "cuda:1".
n_ensemble_configurations	4	Number of preprocessing/class-permutation configurations to average.
batch_size_inference	16	Number of ensemble configurations processed together. Reduce if GPU memory is limited.
preprocess_transform	"mix"	Average raw and power-transformed features. Use "none" to disable the extra transform.
categorical_features	None	Integer indices of categorical columns.
random_state	0	Seed controlling ensemble ordering.
fp16_inference	False	Enable CUDA autocast for model execution.
revision	None	Pin the downloaded Hub model to a branch, tag, or commit.

Offline use

Download the files in this model repository and point the classifier at the local directory:

classifier = TydraClassifier(repo_id="/models/Tydra")

The directory must contain config.json and model.safetensors.

Intended use

Tydra is intended for research and small-to-medium tabular classification problems where labeled training rows can be supplied as in-context examples. Typical uses include:

• rapid tabular classification baselines;
• evaluation of hybrid attention/state-space architectures;
• inference-efficiency and context-length research;
• classification when per-dataset neural-network fine-tuning is undesirable.

Limitations

• This checkpoint performs classification only; it does not support regression.
• It is restricted to 100 features and 10 classes.
• The current implementation requires an NVIDIA GPU and compiled CUDA dependencies, which makes installation more involved than pure-PyTorch models.
• The model was pretrained on synthetic data. Performance may degrade under dataset shift or on domains unlike its synthetic training distribution.
• Predictions should be validated on representative held-out data before use in consequential, safety-critical, medical, legal, or financial settings.
• The model does not replace domain-specific preprocessing, leakage checks, fairness analysis, uncertainty assessment, or human review.

Training and architecture

Tydra-4HT uses an eight-layer hybrid backbone with the following order:

Hydra → Transformer → Hydra → Transformer
      → Hydra → Transformer → Hydra → Transformer

The model has a 512-dimensional representation, four attention heads, and was trained entirely on procedurally generated tabular classification tasks. The full exported training configuration is provided in config.json.

Repository files

• model.safetensors — pretrained FP32 weights;
• config.json — architecture and training configuration;
• example.py — runnable scikit-learn example;
• LICENSE, NOTICE, and LICENSES/ — licensing and attribution information.

License

This release combines components under multiple licenses. Review LICENSE, NOTICE, and the files in LICENSES/ before redistribution or commercial use.

Citation

If you use Tydra in academic work, please cite:

@article{komisarczyk2026tydra,
  title   = {Tydra: An Efficient Hybrid Model for Tabular Data},
  author  = {Komisarczyk, Mieszko and Mathur, Saurabh and Kraus, Maurice and
             Natarajan, Sriraam and Kersting, Kristian},
  journal = {arXiv preprint arXiv:2608.21199},
  year    = {2026}
}