TabICL: A Tabular Foundation Model for In-Context Learning on Large Data
Authors: Jingang QU
David Holzmüller
Gaël Varoquaux
Marine Le Morvan
Venue: ICML 2025 poster (2025)
Keywords: Foundation Model
In-Context Learning
Tabular Data
Tabular Classification
Primary area: deep_learning->foundation_models

Abstract: The long-standing dominance of gradient-boosted decision trees on tabular data is currently challenged by tabular foundation models using In-Context Learning (ICL): setting the training data as context for the test data and predicting in a single forward pass without parameter updates. While TabPFNv2 foundation model excels on tables with up to 10K samples, its alternating column- and row-wise attentions make handling large training sets computationally prohibitive. So, can ICL be effectively scaled and deliver a benefit for larger tables? We introduce TabICL, a tabular foundation model for classification, pretrained on synthetic datasets with up to 60K samples and capable of handling 500K samples on affordable resources. This is enabled by a novel two-stage architecture: a column-then-row attention mechanism to build fixed-dimensional embeddings of rows, followed by a transformer for efficient ICL. Across 200 classification datasets from the TALENT benchmark, TabICL is on par with TabPFNv2 while being systematically faster (up to 10 times), and significantly outperforms all other approaches. On 53 datasets with over 10K samples, TabICL surpasses both TabPFNv2 and CatBoost, demonstrating the potential of ICL for large data. Pretraining code, inference code, and pre-trained models are available at https://github.com/soda-inria/tabicl.

TabICL: A Tabular Foundation Model for In-Context Learning on Large Data
Jingang Qu' David Holzmüller 23
Gaël Varoquaux'
Marine Le Morvan
Abstract
In other data modalities, foundation models particularly Large Language Models (LLMs) (Zhou et al., 2024) have significantly advanced the ability to tackle new tasks and few-shot learning. This is largely due to their remarkable in-context learning (ICL) capabilities (Brown et al., 2020), which enable them to capture patterns in prompts without requiring parameter updates. This success combined with the pervasiveness of tables have spurred interest in tabular foundation models (van Breugel & van der Schaar, 2024).
The Long-standing dominance of gradient-booster decision trees on tabular data is currently chal- lenged by tabular foundation models using In- Context Learning (ICL): setting the training data as context for the test data and predicting in a single forward pass without parameter updates. While TabPFNv2 foundation model excels on tables with up to 10K samples, its alternating column- and row-wise attentions make handling large training sets computationally prohibitive. So, can ICI be effectively scaled and deliver a benefit for larger tables? We introduce TabICI,, a tabular foundation model for classification, pre- trained on synthetic datasets with up to 60K sam- ples and capable of handling 500K samples on affordable resources. This is enabled by a novel two-stage architecture: a column-then-row atten- tion mechanism build fixe-ddieensionl em- beddings of rows, followed by transformer for efficient ICL. Across 200 classification datasets from the TALENT benchmark, TabICL is on par with TabPFNv2 while being systematically faster (up to 10 times), and significantly out- performs all other approaches. On 53 datasets with over 10K samples, TabICL surpasses both TabPFNv2 and CatBoost, demonstrating the po- tential of ICL for large data. Pretraining code. inference code, and pre-trained models are avail- able at https://ggiubb..ccmm//daaaaiaaaa
While LLMs are primarily designed to model natural lan- guage, attempts to fine-tune them for tabular data have emerged (Hegselmann et al., 2023; Fang et al., 2024, and references therein). These efforts rely on table serialization, which is the process of converting table rows into text or sen- tences which can then be tokenized. Gardner et al. (2024) fine-tuned Llama 3-8B stratd the effectiveness of this approach compared to tree- based models in the few-shot setting. However, such language models based approaches are limited by the size of their context window to fit large serialized tables (e.g. up to 32 or 64 shots in Gardner al. 2024). Moreover, is unclear whether LLMs can effectively handle numerical values (Thawani et al., 2021). Finally, evidence shows that LLMSs are pretrained on many popular datasets (Bordt et al., 2024), their evaluation on tabular prediction tasks should also be conducted with care.
corpus of serialized tables and
Taking a markedly different approach, Hollmann et al. (2022) introduced TabPEN, a transformer-basee tabular foundation model for classification tasks, pretrained on syn- thetic tabular datasets only. A key feature of TabPFN is ICL with tables, which eliminates the need for tokenization and enables efficient handling of small tables with up to IK samples and 100 features. The same authors then introduced TabPFNv2 (Hollmann et al., 2025), an improved version that significantly outperforms tree- tree-based and neural network competitors on small-to-medium datasets with up to 10K samples and 500 features. The great promise of tabular ICL has spurred a new line of research (see Section 2.3), yet the quadratic cost of self-attention is a threat to the scalability of all of these. TabPFNv2 uses a two-way attention mechanism alternating between column-wise and row-wise attentions, which limits its scalability for large datasets. In real-world scenarios, where industrial datasets can contain millions of samples (Rubachev et al., 2024), the high computational
1. Introduction
Tabular data, structured in rows and columns, is widely used in industries like healthcare (Cartella et al., 2021) and finance (Johnson et al., 2016), where tabular classifica- tion problems underpin numerous real-world applications.
'SODA team, INRIA Saclay, France "Sierra team, INRIA Paris, France "Ecole Normale Supérieure, PSL Research Uni- versity, Paris, France. Correspondence to: Jingang Qu <jin- gang.qu@ inria.fr>..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

TabICL: A Tabular Foundation Model for Large Data
and memory demands of TabPFNv2 hinder its practicality.
The feature embeddings are then processed per-row by an- other transformer, and aggregated into a single vector using learnable [CLS] tokens. This effectively captures complex feature interactions and accommodates varying number of features. Overall, this column-then-row attention-based em- bedding achieves efficient sparse attention across all cells by leveraging the column/row inherent structure of tabular data as a strong inductive bias. Finally, the resulting row-wise embeddings are handled by final transformer for ICL.
In this paper, we introduce TabICL, scalable and efficient tabular foundation model based on PFNs (Müller et al., 2024) and designed for classification tasks. Pretrained on synthetic datasets up to 60K samples, TabICL. can effec- tively handle datasets up to 500K samples and 500 features, significantly expanding the scalability of ICL for tables.
To accommodate tables of arbitrary sizes, TabICL treats individual cells as fundamental units: each column is re- garded as set of cell values that capture feature-specifir distributions and semantics, while each row comprises in- terdependent feature values. TabICL employs a two-stage architecture to achieve efficient ICL for tabular data. First, it encodes rows (excluding target labels) into dense vector embeddings. Each embedding is designed to capture the entire table information. This stage effectively collapses the column dimension to substantially reduce computational complexity and memory footprint for subsequent ICL. Sec- ond, it combines these compact yet informative embeddings with corresponding labels and then performs ICL. There- fore, the core of TabICL lies in its embedding strategy of the first stage, which is supposed transform rows into semantically rich embeddings.
In addition to the above innovations, we introduce further improvements: (1) We refine TabPFN's pretraining synthetic datasets by adding a new tree-based data- data-generating model to incorporate the inductive biases of tree-based models (Grinsztajn et al., 2022; den Breejen et al., 2024; Grinsz- tajn, 2024); (2) We adopt curriculum learning by scaling the pretraining dataset size from 1K to 60K; (3) To address classification problems with over 10 classes (the pretraining limit), we use hierarchical classification (Silla & Freitas, 1), breaking them into a hierarchical structure of sub- problems with S 10 classes. The increase in the number of tasks is largely offset by the fast predictions of TabICI.
To summarize our contributions: (1) We present TabICL, novel scalable tabular foundation model for classifica- tion tasks that can accommodate any number of samples, features, and classes. In practice, TabICL handles up to 500K samples and 500 features with around 20GB of GPU memory; (2) We introduce a distribution- awaare feature em- bedding approach that handles features with diverse prop- erties in a unified manner, unlocking new possibilities for cross-table transferability; (3) TabICI. performs tasks in single forward pass and is orders of magnitude faster than tabular methods requiring hyperparameter tuning while still providing better performance in most cases. TabICI. is also consistently faster than TabPFNv2 (up to 10 times), with efficiency gains increasing as dataset size grows; (4) We evaluate TabICL on the TALENT benchmark (Ye al., 2025), comprising 200 classification datasets across various domains and sizes (up to 150K samples). TabICL performs comparably to TabPFNv2 on medium-sized datasets and significantly outperforms all other methods. On the 53 large datasets with over 10K samples, TabICL surpasses both TabPFNv2 and CatBoost (Dorogush et al., 2018). These results demonstrate the potential of ICL for large data.
In natural language, words often carry clear semantics and are thus naturally associated to informative embeddings (Mikolov et al., 2013). However, tabular data lacks such an inherent structure: cell values can be ambiguous without metadata such as column names or data types. To address this challenge, TabICI. adopts a well-constrained embed- column-wise embedding to capture statistical regularities within each col- umn and (2) attention-basse row-wise interaction to model dependencies across columns, thereby constructing seman- tically grounded representations for tabular data.
ding strategy combining (I) distribution
Feature embedding involves mapping scalar cell values into high-dimensional vectors for each feature, serving as crit- ical factor in model performance (Gorishniy et al., 2022). Since features often exhibit vastly different distributions, previous approaches typically use feature-specific embed- ding modules without parameter sharing, which, however, limits cross-table transferability. In this work, we refor- mulate feature embedding as a set-input problem, where set of cell values acts as input, and the output comprises corresponding one-to-one embeddings. To achieve this, we leverage the Set Transformer (Lee et al., 2019), a model specifically designed to process sets through efficient induced self-attention. It excels in tasks such as identifying extrema and counting unique elements, enabling the discovery of distribution-relattee metadata within each column and enhancing the ability distinguish between features of different data types.
permutation-invarin
2. Related Work
2.1. Foundation Models and In-Context Learning
In recent years. deep learning has been transformed by the emergence of foundation models, which are pretrained on massive, diverse datasets and serve as versatile backbones for downstream tasks. These transformer- models en- able In-Context Learning (ICL): performing tasks by analyz-
2

---

TabICL; A Tabular Foundation Model for Large Data
X C R m
Column-wise inter-sarnple Embedding
Row-wise
Columns (Features)
E
E R Rummx
inter-feature Interaction
Rows (Samples)
212
Im
CLL)
(CUS)
(CLS) 611
...
APa
222
C21
e2
CLS)
(CUS)
CLS
(CLS)
Transforee TF col
Gal
Crrm
CTL8)
(CLS))
CLS
(CL.))
Eml
Cnm
3/4
Rotary positional embedding
Transformer TF
Dataset-wise In-Context Learning
Row
Transformer
row
Lookup table
Lookup table
Lookup Table
(h1
y1)
(h22
3/2
(hg
3/3).
h4
hs
Training samples
Test samples
Figure Architecture Overview of TabICL. First, column- embedding transforms each the input table into an embedding vector using transformer TFeat (Section 3.2), producing E. Next, row-wisc interaction prepends four trainablc [CLS] tokens E, applics rotary positional encoding, and processes & row-by-row with a transformer T' Concatenating the outputs of [CLS] tokens yields the final row-wise embeddings H. Finally, dataset-wise ICL operates on H and uses Transformer TFi to predict the target labels for the test set a single forward pass. Overall, TabICL consists three transformers.
ing prompts containing input-output pairs, without explicit training or parameter updates. ICL. operates as form of on- the-fly reasoning. The mechanism underlying ICL remains elusive, with prevailing explanations framing it as implicit Bayesian inference (Xie al., 2022; Müller et al., 2024), gradient descent optimization (von Oswald et al., 2023), and algorithmic learning (Garg et al., 2023).
2.3. Tabular Foundation Models
TabPFN, short for Tabular Prior- Data Fitted Network, is a tabular foundation model. a transformer pretrained on extensive synthetic datasets to perform tabular classifi- cation tasks through ICL. TabPFN interprets ICL from a Bayesian perspective as an approximate posterior predic- tive distribution over synthetic datasets. Several variants aim to enhance its scalability, including distilling training data into compact learned context via prompt tuning (Ma et al., 2024b; Feuer et al., 2024), selecting the most rele- vant subset training data for each test sample (Xu et al., 2024; Thomas et al., 2024; Koshil et al., 2024), replacing quadratic with linear attention (Zeng al., 2024), and gener- ating small task-specific neural networks via ICL-based hypernetwork (Müller al., 2023). However, most vari- ants do not structurally improve TabPFN but instead act as prompt engineering to reduce in-context samples.
2.2. Tabular Deep Learning Models
Gradient- booseted decision trees (GBDTs), such CatBoost and XGBoost (Chen & Guestrin, 2016), have long dom- inated the tabular domain. However, growing efforts are focused on developing deep learning models for tabular data. Recent studies indicate a narrowing performance gap be- tween GBDTs and tabular deep learning models (Ye al., 2024; Gorishniy al., 2024).
As tabular deep learning improves, cross-table transferabil- ity emerges as an important topic. Notable efforts in this direction include XTab (Zhu et al., 2023) and CARTE (Kim et al., 2024), which incorporate transferable components that are typically shareable backbone networks and dataset- specific components that require fine-tuning for each new task. The advent of tabular foundation models can bring new possibilities cross-table learning, paving the way for large-scale pretraining and transfer learning across tables.
Several TabPFN variants try to improve the quality of pre- training data, such as TabForestPFN (den Breejen al., 2024) incorporating tree-based synthetic datasets and Tab- DPT (Ma et al., 2024a) using real-world datasets.
TabPFNv2 largely improves TabPFN in terms of both predic- tion performance and scalability. Our new model, TabICL, achieves comparable performance to TabPFNv2 while being
3

---

TabICL: A Tabular Foundation Model for Large Data
more scalable and computationally efficient. Appendix A provides systematic comparison between TabPFNv2 and TabICL across architecture, pretraining, and scalability,
A column e (omitting the subscript for simplicity) is pro- jected into d-dimensional space (d = 128) via linear layer, processed by ISAB consisting of MAB, and MAB2, and passed through linear layers to generate W and B.
3. The TabICL Architecture
ISAB reduces self-attention complexity to (()) while pre- serving the ability to capture global information through a two-stage attention mechanism. In MAB1, inducing vectors V/ act as queries and attend to training samples Unain to generate induced representations M. In MAB2, inputs U (including both training and test samples) serve as queries and attend back to M, enabling global information to prop- agate across all training samples. We use k = 128 inducing
We consider a tabular classification task with an input space X E Rn and a target space y E [1. .... Given a train- ing dataset of input-ouiput pairs Dusia [drrain Vrrain J The T==1 and test sample Ties, our goal is to predict the class proba- bilities p(- Trest Dunin).
(i)
3.1. High-level Structure: Embedding then ICL
vectors, 4 attention heads in the MABs, and 3 ISAB blocks (only one ISAB block is shown in Equations (4) and (5) for clarity). Crucially, only train samples serve as keys and val- ues in MAB1 This ensures that the outputs of ISAB depend solely on training data, thereby preventing data leakage.
TabICL comprises two key modules: a tabular embedding module followed by an ICL module, with labels utilized exclusively in the ICL module. The tabular embedding mod- ule encodes table rows into dense vector representations while taking the inherent column-row structure into account. This module consists of two components: distribution-awann column- embedding, which captures statistical charac- teristics of individual columns, and context-aware row-wise interaction, which models dependencies between features. The ICL module subsequently processes both training and test embeddings through transformer, enabling the predic- tion of the entire test set in single forward pass through ICL. The overall architecture is depicted in Figure 1.
To get insights into the information captured in the learned feature embeddings, we visualize M E R Rxx (Equation (4)) the output of the final ISAB block of To We summarize M by summing along the first dimension (i.e., aggregating the induced representations of all inducing vectors) to obtain single vector per column, and then apply Principal Com- ponent Analysis. Figure 3 reveals that columns with similar skewness (resp. kurtosis) tend to cluster together, i.e., non- symmetric distributions are differentiated from symmetrical ones (Figure 3 left), and heavy-tailed distributions are dif- ferentiated from light-tailed ones. This suggests that TF encodes distributional properties in a structured manner, and cells in column probably embedded to reflect their statistical role (e.g., min, max, mean, mode). Therefore, the learned feature embeddings should distinguish features based on their unique distributional properties, effectively serving as feature identifiers. This contrasts with methods that rely on semantic column names (Kim et al., 2024) or learn feature identifier vectors (Kossen et al., 2021).
3.2. Distributionnaware Column-wise Embedding
The column-wise embedding (or feature embedding) maps each scalar cell in column Cj E I into d-dimensional embedding. Unlike typical approaches that assign separate embedding module to each column (Gorishniy et al., 2022), we embed all columns through a shareable Set Transformer TFoo, as illustrated in Figure 2b and formulated as follows:
W, B TFu (cz)
R
(1)
ej = W Wo BB B
E Raa
(2)
3.3. Context-aware Row-wise Interaction
Note that each cell in column is assigned its weight and bias. Essentially, feature embedding can be framed as a set-input problem. where TF... serves as hypernetoork taking as input permulation-invariaat set of cell values and generating distribution-awanl aware weights and biases W and B.
After obtaining all feature embeddings E = [er,... ... cm] E Fnxmxd a 3-layer transformer with 8 attention heads, de- noted by TFR.. processes F. for inter-feature interactions. To aggregate the embeddings into single vector, four learn- able [CLS] tokens are prepended to each row of E, and their final outputs are concatenated together. We use four tokens (3) to provide richer representations with a total embedding
The operations within TFaol unfold as follows:
U = Lin(c)
E Bnx
M MAB (V1, Urrann Urain)
E k
(4) size of 4 x = 512 for subsequent ICL, while maintaining lower embedding size (d 128) for TFool and TFrow to (5) reduce memory consumption.
ISAB
V = MAB2(U, M, E R
W, B- = Lin(V)
E Rnd
(6) We experimentally observed that TF can suffer from a representation collapse issue. As mentioned earlier, features are identified by their distributional properties after column- wise embedding. Consequently, features originating from
where Lin represents a linear layer, MAB denotes a multi- head attention block, and IASB stands for induced self- attention block (Figure 2a), introduced by (Lee et al., 2019).
4

---

TabICL: A Tabular Foundation Model for Large Data
Multi-head attention block (MAB)
Induced self-attention block (ISAB)
Column
Set Transtormer TFod
Inducing
Cj
(Rm
vectors
V
x3
Lin
61
Query
head attn
Multi
Fonvard
Feed
U
U train
MAB
MABa
Lin
U (R R
ISAB
(Rexd)
Key
block
Lin
W
Value
(Raxd)
(a) Efficient induced self- eelf-attenionn
(b) Column Column-wise inter-sample embedding
Figure The distribution-awarr column-wise embedding block in TabICL. The overall structure (right) relies on the induced self- attention block (middle) that employs two multi-head attention blocks (left) to process incoming embeddings.
Feature
Feature
Feature
Feature
X
RoPE
No RoPE
O skewness
kurtosis
Figure Learned column-wise embeddings encode statistical properties of feature distributions. Visualization of these em- heddings projected onto their first two principal components 40,000 features from synthetic datasets.
Figure 4, An example of learning collapse without RoPE for the balance scale dataset. (Upper) The histograms show that the four features follow the same discrete distribution. (Lower) The t-SNE visualization of the input features and learned wise embeddings H with and without RoPE demonstrates that RoPE can effectively alleviate representation collapse. The colors represent 3 classes
similar distributions thus become less distinguishable. In the extreme case where all features are drawn from the same distribution, TFro using the vanilla permutation-invarian self-attention cannot differentiate a sample from any of its column- column-permuted versions, leading to nearly identical repre- sentations for originally distinct samples, i.e., representation collapse. This phenomenon is exemplified by the balas balance ce scale dataset (Siegler, 1976), shown in Figure 4. In this dataset, all features follow the same discrete distribution with only 5 possible values, making collapse highly prob- able. After processing with TFr., we observe that many samples collapse to the same representation (rightmost plot), despite being originally distinct (leftmost plot).
details can be found in Appendix D.
While the use of RoPE breaks permutation invariance with respect to column order, it can effectively mitigate the rep- resentation collapse. For instance, the balance scale dataset, RoPE preserves distinct representations across dif- ferent samples (Figure middle). Following Xiong et al. (2023), we use a large scaling factor 100,000 for RoPE to enhance generalization to a greater number of features than seen during training (up to 100). To approximately restore permutation invariance, TabICL adopts the same strategy as other TabPFN- like models by ensembling predictions over multiple column permutations.
A similar issue of representation collapse was reported in TabPFNv2. To mitigate this, Hollmann et al. (2025) in- troduced random feature identifier vectors for each feature and encoded groups of features collectively rather than in- dividually. In contrast, we employ a different strategy by incorporating rotary positional embedding (RoPE, Su et al., 2023) into TF break the symmetry between identically distributed features. RoPE is widely adopted in recent LLMs (Dubey et al., 2024) and directly encodes relative positional information into the attention mechanism by rotating the query and key vectors. The rotation angle is determined by the position in the sequence and the dimension index i, defined as 9: = p/ (base2//d) where d is the embedding dimension and base is the frequency scaling factor. More
3.4. Dataset-wise In-Context Learning
After converting all samples into embeddings H E Box44, training labels are mapped to the same space as H using one-hot encoding. The embeddings of X and for the training set are added to create the final training embeddings H We then process Husin and Hest using a 12-layer Transformer with attention heads, denoted by TFa- The
5

---

TabICL: A Tabular Foundation Model for Large Data
embeddings in Huann can attend to one another while those in fe can only attend Hurin- Finally, two-layer MLP converts the outputs of es into class probabilities for the
ables. To balance data generation, we combine SCMs (70%) with tree-based SCMs (30%). Appendix C gives details and examples of generated data.
test samples.
Diversifying activation functions In TabPFN (vI), f is defined as a random affine mapping (a lin- ear layer) with activation function chosen from [ldentity, Tanh, Leaky ReLU, ELU]. We enrich this set with 15 additional activation functions to enhance the diver- sity of non-linear dependencies, introducing for example non-monotone or discontinuous functions. We also include random activation functions sampled from Gaussian pro- cesses with random kernels. Gaussian Process functions have been used for synthetic data generation for time series foundation models (Ansari et al., 2024), but not as activa- tion functions and with different types of kernels. Finally, we added an option to use different activation functions across layers and applied standardization followed by ran- dom rescaling before each activation function. Figure C.I visualizes the employed activation functions.
3.5. Computational Complexity Analysis
To characterize the computational efficiency and scalability of TabICL, we analyze the time complexity of its three components for a table with n rows and m features:
1. To processes each column independently using ISAB, resulting in complexity O(kkm), where k is the fixed number of inducing vectors.
2. TFo applies self-attention over features and [CLSS] tokens per row, yielding complexity O(m²n). TFrow is efficient since m < in most tabular settings.
3. Tii operates on the compressed row embeddings via self-attention with complexity Onn²). While quadratic in n, its cost is incurred once since the feature dimen- sion is collapsed, ensuring practical efficiency.
The overall complexity of TabICL is O(m2n + n²). In contrast, TabPFNv2 alternates column-wise and row-wise attentions without collapsing dimensions, resulting in com- plexity (m3n m2m) As a result, TabPFNv2 can become computationally expensive for large and moderate m.
4.2. Curriculum Learning for Large-scale Pretraining
Similar to pretraining LLMs on shorter sentences before moving to longer ones, we gradually increase the size of synthetic datasets (i.e., the number of samples) while adjust- ing the micro batch size Ng used for gradient accumulation to accommodate memory constraints. We employed a three- stage procedure:
4. Pretraining and Inference
4.1. Improved Pretraining Synthetic Datasets
1. NB = 4 with a fixed size of 1,024 for 160K steps; 2. Ng 1 with the size randomly drawn from log- uniform distribution between 1K and 40K over 2K steps. Activation checkpointing is enabled for datasets exceeding 10K samples, and we accordingly reduce the number of features to avoid out- out-of-memory issues; 3. Ng = 1 with the size uniformly sampled between 40K and 60K for 50 steps, training only TFia while freezing all other components.
TabICL is pre-trained exclusively on synthetic datasets. To ensure realistic dependencies among variables, we gener- ate these datasets using structural causal models (SCMs), following the approach proposed in TabPFN (v1). We first sample directed acyclic graph (DAG) to define depen- dencies, following the structure of fully connected MLP, where each neuron corresponds to a variable. Each feature e is then modeled as a function f of its parent variables Pa(c) in the graph, with added independent noise c, i.e., e= (P)(C)) + e. Compared to previous work, we enrich the dataset generation in two ways: (i) we introduce tree- based SCMs to benefit from their inductive biases, and (ii) increase the diversity of modeling functions f.
Each step consists of 512 datasets with the number of features (5 100) and classes (< 10) randomly sampled, FlashAttention and automatic mixed precision are applied globally. The pretraining took 20 days on three A100 GPUs with 40GB memory using PyTorch (16, 3, and 1 days for stage 1, 2, and 3, respectively). Appendix E. gives more pretraining details.
Tree-based generation Inspired by Grinsztajn (2024), we introduce tree-based SCMs leverage their ability to model complex interactions and hierarchical dependencies between variables. We define f using an XGBoost regression model, as it is widely favored by practitioners and supports multi- output regression. At each layer of the graph, an XGBoost model is trained on fake targets drawn from Gaussian noise, taking the values of the parent variables as input. The ob- tained predictions then become the values of the child vari-
4.3. Hierarchical Class-extension Strategy
We tackle many-class classification problems (> 10 classes) through hierarchical classification (Silla & Freitas, 201 I). Specifically, we recursively and evenly partition classes into subgroups of up to 10 classes, forming a multi-level classifi- cation tree. A classification problem with & classes requires
6

---

TabICL: A Tabular Foundation Model for Large Data
hierarchy with depth r - 1o k.. Each internal node in the tree predicts the probability of its child groups, and each leaf predicts the probability of the classes it contains. During inference, the final probability for a given class is obtained by multiplying the probabilities across all nodes situated on the unique root-to-leaf path leading to this class. See Appendix E.3 for more details.
ing models. In Appendix H. we show that increasing en- semble size improves performance for both TabPFNv2 and TabICL, with TabPFNv2 benefiting more from ensembling, A fair comparison would require training the other models as an ensemble as well. However, this is computationally expensive, was not part of the original benchmark, and is not implemented in these experiments.
As noted earlier, labels are used exclusively in the final ICL block. Consequently, the hierarchical tree is constructed during wate ICI, with all sub-tasks sharing the same learned row embeddings H and the same Ta for ICL pre- dictions. This kind of sharing greatly enhances the efficiency of TabICL in hierarchical classification scenarios.
Both TabICL and TabPFNv2 average over 32 predictions ob- tained by randomly shuffling columns and classes and using different pre-processors. TabICL applies z-normalization with or without power transformation to all input features. See Hollmann et al. (2025) for details on TabPFNv2. To avoid mee-ory issues, we subsample training sets to 30K samples for TabPFNv2, while TabICI. could predict on all datasets without subsampling. In addition, we disable au- tomatic mixed precision across all methods during inference to ensure reproducibility and consistent time measurements.
4.4. Memory-efficient Inference
Using FlashAttention, which offers linear memory complex- ity with respect to sequence length, we observed that the peak activation memory can be effectively modeled by a polynomial regression during inference: ai batch_size + 02 seq_len + 03 batch_sizz seq This enabled us to dynamically adjust the batch size based on sequence length and available GPU memory. The batch dimension serves different roles depending on the context: it repre- sents the number of columns for column-wise embedding, the sample size for row-wise interaction, and the number of datasets for dataset-wise ICL. Inspired by Rajbhandari et al. (2021), intermediate activations can be offloaded to CPU and disk as needed to further reduce memory consump- tion. These optimizations enable TabICI to handle datasets with 100K samples and 500 features using only 5 GB of GPU memory and 32 GB of RAM. This suffices for many real-world applications. See Appendix E.2 for details.
5.2. Results
TabICL obtains state-of-the-art accuracy. Figure 5 shows accuracies relative to the tuned MLP for all models. TabICL obtains the best median relative accuracy across all datasets while being much faster than traditional state-of- the-art models: the geometric mean training+inference time per IK samples for TabICI. is 1.1 seconds, while tuning CatBoost on CPU takes around 3 minutes, and RealMLP and ModernNCA on a GPU take around 7 minutes. Looking at the ranks of each method based on the obtained accura- cies, the critical difference diagram in Figure 1.1 shows that TabICL and TabPFNv2 outperform competitors by wide margin, while the difference between the two of them is not statistically significant.
5. Experiments
Speedup over TabPFNv2. Figure 6 shows that TabICL is 1.5x faster than TabPFNv2 on small datasets and 3-10x faster on large datasets. This is facilitated by the hybrid ar- chitecture of TabICI using fewer of the expensive row-wise and column-wise attention layers with smaller embedding dimension before ICL on tokenized rows. On a dataset with 10,000 samples and 100 features, TabICL takes roughly 20 seconds versus 1 minute 40 seconds for TabPFNv2, while for a dataset of 1000 samples and 10 features, TabICL takes second compared to 2 seconds for TabPFNv2. The empir- ical speedup of TabICI over TabPFNv2 also corroborates the complexity analysis in Section 3.5. In Figure 1.1, we fit simple scaling laws to predict the runtime of TabICL and TabPFNv2 based on the number of samples and features. These scaling laws show that the average speedup of TabICL. remains around five-fold for large datasets.
5.1. Benchmark
We evaluate TabICL on the TALENT benchmark introduced by Ye et al. (2025). It comprises 200 classification datasets (120 binary and 80 multiclass datasets) and comes with re- sults for over 30 baseline methods. To ensure a fair compari- son, we exclude 15 datasets that were used by TabPFNv2 for hyperparameter tuning and model selection (see Appendix F for more details). In what follows, we mainly focus the 171 datasets with at most 10 classes, since these can be handled natively by TabICL and TabPFNv2.
Datasets are split into 64% training, 16% validation, and 20% test data. While most models rely on the validation set for early stopping and hyperparameter tuning, TabICL and TabPFNv2 do not. Nonetheless, we opted to train TabICL and TabPFNv2 using the training data only, which places them at disadvantage. On the flip side, both TabICL and TabPFNv2 leverage ensembles, unlike the other deep leam-
TabICL enables ICL for large datasets. While TabPFNv2 achieves excellent performance on datasets up to 10K samples, it has only been pre-trained with up to 2048

---

TabICL: A Tabular Foundation Model for Large Data
Tyb
TabICL
10x
MrA
RealMLP
LoCaIPFN
CatBoost
Speodup
TahR
TabM
LightGBM
XGBoost
FFT
ResNet
TabPFN
MLP-PLR
Tahles
1x
DCNv2
102
AutoInt
Number of cells
RFoorst
ExcelF
SNN
Figure 6. Speedup of TabICL TabPFNv2 datasets with less than 30K samples and at most 10 classes.
TANGOS
TabCaps
PlaRL
DANets
NODE
TabT
ModemINCA
LogReg
CatBoost
GrowNet
TaheeNv2
KNN
SwitcT
20
TabNet
Rank
15
-10%
5%
Relative accuracy improvement over MLP (1)
10
10
10a
103
Geom. mean train time [s] per 1K samples
10
10
n.samples
Figure 5. Accuracy and training/inference times the TAL- ENT benchmark up to 10 classes. The red bar indicates the me- dian relative accuracy improvements over MLP across all datasets for each method. For TabICL and TabPFNv2, the time reported A100 GPU. For the other
Figure 7. Model rankings as a function of sample size. Each point is the rank of one method on one dataset. Lower rank is better. The lines show the bootstrap median and 10% / 90% bootstrap confidence intervals of piecewise linear fit.
corresponds training+inference models, it includes both training and hyperparameter tuning. Since Ye al. (2025) track only the training time using the best hyper- parameters found after 100 tuning steps, we approximate the total training time by multiplying this value by 100. For TabPFN (vl), LoCalPEN, and Tabelables, we do not report the time since the original time measurement does not include the inference time.
TabICL produces reliable probabilities. In many prac- tical situations, the quality of the estimated probabilities is crucial for decision-making. Therefore, we also report the log loss (a.k.a. cross-entropy loss), which is a proper scoring rule and therefore rewards accurate prediction of probabilities (Gneiting & Raftery, 2007), Since TabICL does not leverage hyperparameter tuning, its predictions are not specifically optimized towards accuracy. The critical difference diagrams in Appendix show that TabICL and TabPFNv2 significantly outperform accuracy-tuned com- petitors on the log loss, though the difference between the is not significant, indicating that both produce more reliable probability estimates than the other models.
training samples and can fail on datasets above 30K sam- ples due to its memory usage. Figure shows that unlike TabPFNv2, the performance of TabICL remains strong for larger datasets. While ICL is often used in few-shot settings, this demonstrates that foundation models can compete with the best deep learning or based models, even in large- sample regimes where the latter have the most potential. Additional results in Appendix B demonstrate that TabICL also performs well for many classes, features, and high ra- tios of categorical features. To enhance the performance of TabPFNv2 on large datasets, Hollmann et al. (2025) pro- posed random forest extension: during training, the data is partitioned using decision trees, and TabPFNv2 is fit- ted at each leaf to enable localized predictions at inference time. We found that this extension can significantly improve the performance of both TabPFNv2 and TabICL on large datasets (refer to Appendix G for more details).
TabICL remains effective with more than 10 classes. On datasets with more than 10 classes, we apply the hier- archical classification strategy from Section 4.3 to TabICL. Thanks to its use of tabel-independent row embeddings that can be shared between all sub-classifiers. TabICL scales efficiently to large number of classes. Figure shows that TabICL still achieves the second best result on these datasets in terms of mean normalized accuracy, while TabPFNv2 cannot natively handle more than 10 classes.
8

---

TabICL; A Tabular Foundation Model for Large Data
0.95
samples
Normalized Accurccy
Rel. Imprrvveentts over Stage
0.85
RealMLP
TabICI
TabR
ResNet
ModernNCA
MLP PLR
0.80
FT-1
MLP
KBBook
CalBoost
0.70
102
-25
Figure& 8. Normalized accuracy across the 12 datasets with more than 10 classes. Colors identify tree-based models (green), deep learning models (pink), retrieval models (gray), transformers (or- ange), and in-context models (blue).
Stage
Stage
Stage
Figure 10. Relative accuracy improvements over Stage across 200 classification datascts. Green triangles denote the mean relative improvements.
10%
Rel. Improvement Tree
815
two orders of magnitude faster than other tabular methods requiring hyper-parameter tuning. TabICL achieves this through hybrid architecture and memory-saving optimiza- tions. Compared to the newly released and leading tabular foundation model TabPFNv2, TabICL achieves comparable performance while being more scalable and faster.
25
00
0%
-1%
@
00
.45
-1%
Limitations (1) Like other foundation models, TabICL suffers from slow inference speed, though TabPFNv2 has shown caching can alleviate this. Currently, TabICL is lim- ited to classification, but regression similar methodology, as Hollmann et al. (2025) showed. (2) A defining characteristic of tabular data is that columns do not have a natural ordering. Ideally, tabular methods should invariant to column permutations. However, TabICL violates this invariance due to the use of RoPE. (3) Our evaluation inherits the strengths and weaknesses the TAL- ENT benchmark. Notably, TALENT trains single models tuned using the holdout method, while ensembling models evaluated using cross prossvalidattion can improve performance. Bagging increases computational cost, whereas TabICI. can simply be trained on the full data without needing a val- idation set. (4) TALENT employs mean imputation for handling missing values. However, might be better to allow TabPFNv2 to handle missing values internally despite the low proportion of missing values in TALENT. (5) We did not provide categorical information to TabPFNv2. Though TabPFNv2 infers categorical features internally, providing this information explicitly could enhance its performance.
Accuracy
Log
Figure 9. Relative improvements of TabICL pretrained on prior datasets with versus without tree-based SCMs across 200 datasets. Green triangles denote the mean relative improve- ments. mitigate computational overhead, we pretrained TabICL for 20K steps following the stage setup of curriculum learning.
be treated with
5.3. Ablation Studies
Tree-based SCMs prove beneficial. Incorporating the inductive biases of tree-based models during pretraining improve the performance of TabICL, as shown in Figure 9.
Curriculum learning effectively improves the perfor- mance of TabICL on large datasets. Across the three stages of curriculum learning, the average rank of T improves from 11.4 (ninth place), 7.46 (second), and fi- nally to 6.95 (first), as shown in Figure E.2. These results show the effectiveness of curriculum learning. However, we also observe a slight performance drop on some small datasets during curriculum learning, as shown Figure 10.
Outlook In-context learning for tabular data was origi- nally introduced as speedup for small tables (Hollmann et al., 2022). As the expressive power of a forward pass in transformer may seem limited, context learning loses its edge given sufficient data. We find that even with large data, pre-training induces implicit priors that give in-context transformers a competitive ad advantage.
6. Conclusion
We introduce TabICI., novel tabular foundation model that extends the scalability of existing tabular foundation models by an order of magnitude. Evaluated on datasets with up to 100K training samples, it delivers excellent per- formance without hyperparameter tuning, making nearly
might wonder if in-

---

TabICL; A Tabular Foundation Model for Large Data
Acknowledgements
den Breejen, F,, Bae, S., Cha, S., and Yun, S.-Y. Why In-Context Learning Transformers Tabular Data Clas- sifiers, May 2024.
We thank Lennart Purucker and Samuel Müller for interest- ing discussions. We are also grateful to Si- Yang Liu and Han-Jia Ye, the authors of the TALENT benchmark, for providing information and clarifying our questions. Finally, we thank the anonymous ICML reviewers for their valuable feedback and suggestions.
Dorogush, A. V., Ershov, V., and Gulin, A. CatBoost: Gra- dient boosting with categorical features support, October 2018.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The llama 3 herd of models. arXiv preprint M:Xiv:2407.2111 2024.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Fang, X., Xu, W., Tan, F. A., Zhang, J., Hu, Z. Qi. Y., Nick- leach, S., Socolinsky, D., Sengamedu, S., and Faloutsos, C. Large language models(llms) on tabular data: Predic- tion, generation, and understanding a survey, 2024.
References
Feuer, B., Schirmeister, R. T., Cherepanova, V., Hegde, C., Hutter, F., Goldblum, M., Cohen, N., and White, C. TaneTables: Context Optimization for Scalable Prior- Data Fitted Networks. arXiv preprint arXiv:2402.11137 2024.
Ansari, A. F., Stella, L.., Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S. S., Arango, S. P., Kapoor, S., et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07813 2024.
Barbero, F., Vitvitskyi, A., Perivolaropoulos, C., Pascanu, R., and Veličkovié, P. Round and Round We Go! What makes Rotary Positional Encodings useful?, October 2024.
Gardner, J., Perdomo, J. C., and Schmidt, L. Large Scale Transfer I Learning for Tabular Data via Language Model- ing, June 2024.
Garg, S., Tsipras, D., Liang, P., and Valiant, What Can Transformers Learn In-Context? A Case Study of Simple Function Classes, August 2023.
Bordt, S., Nori, H., and Caruana, Elephants never forget: Testing language models for memorization of tabular data. arXiv preprint arXiv:2403.06644 2024.
Gneiting, T. and Raftery, A. E. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359 -78, 2007.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,
J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
Gorishniy, Y., Rubachev, I., and Babenko, A. On embed- dings for numerical features in tabular deep learning. Ad- vances in Neural Information Processing Systems, 35: 24991-25004, 2022.
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners, July 2020.
Gorishniy, Y., Kotelnikov, A., and Babenko, A. TabM: Ad- vancing Tabular Deep Learning with Parameter-Efficient Ensembling, November 2024.
Cartella, F., Anunciacao, O., Funabiki, Y., Yamaguchi, D., Akishita, T., and Elshocht, O. Adversarial Attacks for Tabular Data: Application to Fraud Detection and Imbal- anced Data, January 2021.
Grinsztajn, L. Réconcilier I' a'apprentisaage profond avec les données tabulaires. PhD thesis, Université Paris-Saclay, 2024.
Chen, T. and Guestrin, C. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Dis- covery and Data Mining, pp. 785-794, August 2016. doi: 10.1 10.1145/2336672.299773
Grinsztajn, L., Oyallon, E., and Varoquaux, G. Why do tree- based models still outperform deep learning on tabular data?, July 2022.
Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X., and Sontag, D. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pp. 5549-5581. PMLR, 2023.
Da Costa, N., Pförtner, M., Da Costa, L., and Hennig, Sample path regularity gaussian processes from the co- variance kernel. arXiv preprint arXiv:2312. 14886, 2023.
10

---

TabICL; A Tabular Foundation Model for Large Data
Hendrycks, D. and Gimpel, K. Gaussian linear units (gelus). arXiv preprint (eXiv:/606004411 2016.
Mikolov, T., Chen, K., Corrado, G., and Dean, J. Efficient Estimation of Word Representations in Vector Space, September 2013.
Hollmann, N., Müller, S., Eggensperger, K., and Hut- ter, F. Tabpfn: A transformer that solves small tabu- lar classification problems in second. arXiv preprint arXiv:2207.01 0488, 2022.
Müller, A., Curino, C., and Ramakrishnan, R. MotherNet: A Foundational Hypernetwork for Tabular Classification, December 2023.
Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. Accurate predictions on small data with tabular foun- dation model. Nature, 637(6045):319-326, January 2025. ISSN 1476-4687. doi: 10. 1033///41186004408322222
Müller, S., Hollmann, N., Arango, S. P., Grabocka, J., and Hutter, Transformers Can Do Bayesian Inference, August 2024.
Rahimi, and Recht, B. Random features for large-scale kernel machines. Neural Information Processing Systems, 20, 2007.
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.- w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L.,, and Mark, R. G. MIMIC-III, a freely accessible critical care database. Scientific Data, 3(1): 160035, May 2016. ISSN 2052-4463. doi: 10.1038/sdata. 2016.35.
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y. ZeRO-Infinit; Breaking the GPU Memory Wall for Extreme Scale Deep Learning, April 2021.
Rubachev, I., Kartashev, N., Gorishniy, Y., and Babenko, Tabred: Analyzing pitfalls and filling the gaps in tabular deep learning benchmarks. arXiv preprint arXiv:2406. 2024.
Kim, M. J., Grinsztajn, L., and Varoquaux, G. CARTE: Pretraining and Transfer for Tabular Learning, May 2024.
Kingma, P. P. and Ba, J. Adam: method for stochastic optimization. arXiv preprint arXiv: 1rXXiv:412.669880 2014.
Siegler, R. Balance Scale. UCI Machine Learning Reposi- tory, 1976. DOL: https://bb..ogg11.444///3.33
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. Self-normalizing neural networks. Neural Information Processing Systems, 30, 2017.
Silla, C. N. and Freitas, A. A. A survey of hierarchi- cal classification across different application domains. Data Mining and Knowledge Discovery, 22(1):31-72, Jamuary 2011. ISSN 1573-756X.
doi: 10.1007/
Koshil, M., Nagler, T., Feurer, M.. and Eggensperger, K. Towards Localization via Data Embedding for TabPFN. 2024.
Su, J., Lu, Y.. Pan, S., Murtadha, A., Wen, B., and Liu, Y. RoFormer: Enhanced Transformer with Rotary Position Embedding, November 2023.
Kossen, J.. Band, N., Lyle, C., Gomez, A. N., Rainforth, T., and Gal, Y. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. Advances in Neural Information Processing Systems, 34: 28742-28756, 2021.
Thawani, A., Pujara, J., Szekely, P. A., and llievski, F. Repre- senting Numbers in NLP: A Survey and Vision, March 2021.
Krizhevsky, A. Convolutional deep belief networks on cifar- 10.
Thomas, V., Ma, J., Hosseinzadeh, R., Golestan, K., Yu, G., Volkovs, M., and Caterini, A. Retrieval & Fine-Tuning for In In-Context Tabular Models, June 2024.
Lee, J., Lee, Y., Kim, J., Kosiorek, A. R., Choi, S.,, and Teh, Y. W. Set Transformer: A Framework for Atteetion-based Sermuration-Invarian Neural Networks, May 2019.
van Breugel, B. and van der Schaar, M. Why Tabular Foun- dation Models Should Be Research Priority, June 2024.
Ma, J., Thomas, V., Hosseinzadeh, R., Kamkari, H., Labach, A., Cresswell, J. C., Golestan, K., Yu, G., Volkovs, M.,
Oswald, J., Niklasson, E.., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent, May 2023.
and Caterini, A. L. TabDPT: Scaling Tabular Foundation Models, October 2024a.
Ma, J., Thomas, V., Yu, G., and Caterini, A. In- Context Data Distillation with TabPFN. arXiv preprint MXXv:2400200671,, 2024b.
Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An Explanation of In-context Learning as Implicit Bayesian Inference, July 2022.

---

TabICL; A Tabular Foundation Model for Large Data
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P.,
Hou, R., Martin, L.., Rungta, R., Sankararaman, K. A.,
Oguz, B., Khabsa, M., Fang. H., Mehdad, Y., Narang, S.,
Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H. Effective Long-Context Scaling of Foundation Models, November 2023.
Xu, D., Cirit, O., Asadi, R., Sun, Y., and Wang. W. Mixture of In-Context Prompters for Tabular PFNs, May 2024.
Ye, H.-J., Yin, H.-H., and Zhan, D.-C. Modern Neighbor- hood Components Analysis: A Deep Tabular Baseline Two Decades Later, July 2024.
Ye, H.-J., Liu, S.-Y. Cai, H.-R., Zhou, Q.-I., and Zhan, C. A Closer Look Deep Learning Methods on Tabular Datasets, January 2025.
Zeng. Y., Kang, W., and Mueller, A. C. Tabflex: Scaling tab- ular learning to millions with linear attention. In NeurIPS 2024 Third Table Representation Learning Workshop, 2024.
Zhou, C., Li, Q., Li, C.,, Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q. He, L., Peng, H., Li, J., Wu, J., Liu, Z., Xie, P., Xiong, C., Pei, J., Yu, P. S., and Sun, L. A comprehensive survey on pretrained foundation models: A history from BERT to ChatGPT. International Journal of Machine Learning and Cybernetics, November 2024.
ISSN 3668-08X. doi:
Zhu, B., Shi, X., Erickson, N., Li, M., Karypis, G., and Shoaran, M. XTab: Cross-table Pretraining for Tabular Transformers, May 2023.
12

---

TabICL: A Tabular Foundation Model for Large Data
A. Systematic Comparison between TabICL and TabPFNv2
This appendix provides a systematic comparison between TabICL and TabPFNv2, two tabular foundation models that leverage In-Context Learning (ICL) make predictions. Both models are pretrained on synthetic datasets and aim to deliver strong performance on unseen real-world tabular datasets.
A.1. Architecture
Core mechanism. (1) TabICL employs a two-stage architecture, where the first stage uses Sellumn-thenn-ee attention mechanism to build siedddiimesiionl embeddings and effectively collapses the column dimension to reduce computational complexity for the subsequent ICL stage. (2) TabPFNv2 utilizes two-way attention mechanism that alternates between column-wise and row-wise attentions. These attentions operate on the input table across original dimensions without any intermediate collapse.
Representation collapse. When features have similar distributions, both TabICL and TabPFNv2 may suffer from the representation collapse issue. (1) To alleviate this problem, TabICL incorporates Rotary Positional Embedding (RoPE) into the transformer used in row-wise interaction. RoPE helps break the symmetry between identically distributed features by encoding relative positional information. (2) TabPFNy2 addresses this by introducing random feature identifier vectors for each feature. It also encodes groups of features collectively rather than individually.
Label fusion. TabICI and TabPFNv2 differ significantly in when they incorporate label information. (1) TabICI. employs late fusion strategy. TabICL processes input features independently of their labels during row-wise embedding and column-wise interaction. Labels are utilized exclusively in the final ICL stage. (2) TabPFNv2 employs an early fusion strategy for label information. TabPFNv2 directly concatenates the input features with the labels of the training samples, processing them together from the very beginning and throughout its layers.
A.2. Pretraining
Prior generation. Both TabICL and TabPFNv2 are pretrained exclusively synthetic prior datasets. (1) TabICL builds on the previous prior generation of TabPFN (vl), which relies on Structural Causal Models (SCMs). TabICI introduces two more extensions: broader nonlinear modeling capabilities of SCMs and the integration of tree-based SCMs to incorporate the inductive biases tree-based models. (2) TabPFNv2 constructs diverse Directed Acyelic Graphs (DAGs) through a sophisticated procedure (i.e., growing networks with redirection sampling method) and incorporates various computational mappings at graph edges, including small neural networks, categorical feature discretization, and decision trees. However, the code for its prior generation is not open-source.
Dataset size. (1) TabICL was pretrained using curriculum learning that progressively scales the number of samples from 1,024 60K. The number of features was sampled uniformly up 100, and the number of classes was sampled uniformly up to 10. (2) TvbPFNv2 was pretrained on synthetic datasets containing up to 2,048 training samples (uniformly sampled). The number of features was drawn from beta distribution and then linearly scaled to the range from 1 to 160, while the number of classes was also uniformly sampled up to 10.
Pretraining scale. (1) TabICL was pretrained over approximately 82 million synthetic datasets. (2) TabPFNv2 was trained
on a larger corpus, comprising approximately 130 million synthetic datasets.
A.3. Scalability and Efficiency
Dataset size handling. (1) At inference, TabICL is capable of handling tables with 100K samples and 500 features within 5GB of GPU memory, and can deal with any number of classes using hierarchical class-extension strategy. (2) TabPFNv2 excels on tables with up to 10K samples and 500 features, and can only deal with up to 10 classes.
Computational complexity. Given a table with n rows and m columns, the time complexity of TabICL O(m2n + n²), while the time complexity of TabPFNv2 is Omm2 + n² n²m).. Our extensive experiments demonstrate that TabICL is systematically faster than TabPFNv2.
13

---

TabICL; A Tabular Foundation Model for Large Data
B. Further Experiments
Predicting the runtime TabICL and TabPFNv2. obtain rough predictions of the time a forward pass (training and inference) in TabICL and TabPFNv2, we leverage the runtime complexity of the column- and row-wise attention modules. As discussed in Section 3.5, for table with n rows and m columns, the runtime complexity is O(n²m mm²) O(nm(n + m]) for TabPFNv2, while it is O( + nm?) for TabICL. For simplicity. use the term nmnn + m) from the complexity of TabPFNv2, plot this quantity on the z-axis of Figure B.1, and fit models of the form
time = a (n m)))
with MSLE loss, that is, a MSE loss between the log-transformed time and the log-transformed prediction. To facilitate a simpler comparison, we fix Ty = 0.8 for both models since it yields good fit. We suspect that ry = 1 should be used asymptotically for TabPFNv2, but then more additional terms would be needed to obtain a good fit. In this model, the speedup of TabICL over TabPFNv2 for large datasets approaches 5, while it is 1.4 for small datasets.
10a
TahPTNv2: = 2.65 10 10.76m
TahCCL y 1.87 2.22
Fit+prediet time [s]
5.0x
10
10a
1.5x
10
10a
10a
10
10"
#cells
(#rows #columns)
Figure B.I. Training+inference times of TabICL and TabPFNv2. Each point represents the time datasct A100 GPU, We only use datasets with less than 30,000 samples because TabPFNv2 applied with subsampling on larger datasets to avoid RAM overflow.
Metafeatures. We analyze the dependence of the ranks of TabICL, TabPFNv2, CatBoost, and ModernNCA based on different dataset metafeatures. To this end, we fit piecewise linear regression models with pre-defined nodes to predict the ranks of these models depending on single metafeature.
Figure B.2 shows the scaling with the number of classes. The performance of TabICL deteriorates on datasets with three classes but not on datasets with more classes. is unclear if this is really due to the number of classes or caused by a different characteristic present the three-class datasets on the benchmark.
Figure B.3 shows the scaling with the number of features. These plots show that TabICL behaves well for large numbers of features even though it tokenizes entire rows in the middle of the network before seeing any labels.
Figure B.4 shows the dependence on the ratio of categorical to numerical variables. In general, the performance of TabICL and TabPFNv2 deteriorates somewhat in the presence of many categorical variables. However, is notable that TabICL performs slightly better than TabPFNy2 on such datasets even though TabPFNy2 has a more sophisticated categorical feature generation in its prior.
C. Synthetic Datasets for Pretraining
C.I. SCM prior with more activation functions
In the TabPFN prior, replace the activation layers by the following sequence of layers:
A standardization layer that standardizes each feature across the batch (samples) dimension
A random rescaling layer that, for each neuron i, computes z; + exp(2a) + b), with a, b ~ N(0, 1) sampled once per layer.
14

---

TabICL; A Tabular Foundation Model for Large Data
30
ModernNCA
CatBonst
25
TahPPNv2
TabICL
20
Rank
15
3
4
8
9 10
n.classes
Figure B.2. Dependency of the benchmark rank on the number of classes. Each point the rank method on one datasct. Lower rank is better. The lines show the bootstrap median and 10% / 90% bootstrap confidence intervals of a piecewise linear fit.
ModemnNAA
CaiBoost
25
TabPPNv2
Tah[CI
20
Rank
5
101
102
featurss
Figure B.3. Dependency of the benchmark rank on the number of features. Each point is the rank of one method on one dataset. Lower rank better. The lines show the bootstrap median and 10% / 90% bootstrap confidence intervals of piecewise linear fit.
A random activation function. With probability 1/2, each layer uses the same type activation function, otherwise each layer samples the type of activation function independently.
On top of the original activation functions {Identity, tanh, LeakyRelUU, ELU), we add the following activation functions:
ReLU
ReLU6 (Krizhevsky)
SELU (Klambauer et al., 2017)
SiLU (Hendrycks & Gimpel, 2016)
Softplus
Hardtanh(z) = max min(),))) m)n()))
Signum function
Sine
RBF = exp(p=x x22)
Exponential function
15

---

TabICL; A Tabular Foundation Model for Large Data
30
ModernNCA
CalBonst
25
TabPPNv2
TablCL
20
Rank
15
0.0
0.2
0.4
0.6
0.8
1.0
cat.ratio
Figure B.4. Dependency of the benchmark rank the ratio of categorical overall features. Each point is the rank of one method on one dataset. Lower rank is better. The lines show the bootstrap median and 10% / 990% bootstrap confidence intervals of piecewise linear fit.
f(z) / |
f(x) |
f(x) x?
f(x) |ac|
A random function f(x) 6(x) where N(0 1) and the feature map ø is defined randomly
<(x) |w|2
sin(ar + b) E N
N 256,
b, 4 40, 2T).
00 N).
w, = a, exp(u)
u ~ (10.7,330)
Here, all random parameters are drawn once per layer, and @ is an element-wise product. This is motivated by the fact that for fixed feature map o, the random function f(x) = Φ(x) is a Gaussian process with covariance kernel k(x,, _²) = o(x) ((z'). The design of Φ is inspired by random Fourier features (Rahimi & Recht, 2007). The randomly drawn exponent - exp(u) leads to different decays of coefficients with increasing frequency, which produces different levels of smoothness for the sampled function shown in Figure C.1 (right) (Da Costa et al., 2023). This activation function applies standardization directly before the random function, such that the random rescaling has effect.
Here, the random function is sampled with a probability ten times higher than the other activation functions to account for the fact that can represent many different functions.
Figure C. visualizes used activation functions. Figure C.2 shows datasets from resulting prior.
C.2. Tree-based SCM prior
The tree-based SCM prior replaces the linear and activation layers in the SCM prior by XGBoost models fitted on random data. More specifically, these models are generated as follows:
Sample n.estimators and max_depth independently as min(4, + Exponential(> = 0.5)) and min min(4, (4,2 + Exponential(A = 0.5)). respectively.
16

---

TabICL: A Tabular Foundation Model for Large Data
0.75
0.75
0.50
0.50
0.25
0.25
0.25
0.25
0.50
-0.50
00.5
0.75
-1.00
Figure C.I. Activation functions from the SCM prior. Left: Non-randon activation functions from the SCM prior without standard- ization and random rescaling. Activation functions are randomly flipped along -- and/or y-axis to fully utilize the space the plot. Right: Random instantiations of the random activation function from the SCM prior, including the automatic standardization of the inputs.
Figure C.2. Randomly generated 2D datasets from the SCM prior. The color corresponds to the class label.
17

---

TabICL; A Tabular Foundation Model for Large Data
Figure C.3. Randomly generated 2D datasets from the tree-based SCM prior. The color corresponds the class label.
For each layer with n input and m output neurons, fit XGBoost multi-output regressor with the parameters above on the layer inputs v; with standard normal targets yı R", then use the fitted model predict the given inputs.
Figure C.3 shows datasets from the tree SCM prior.
D. Rotary Positional Embedding
Rotary Positional Encoding (RoPE) is technique used in Transformer models to represent positional information in the input data. RoPE encodes positional information directly into the attention mechanism through rotation matrix applied to the query and key vectors in self-attention. Given position p. a frequency w, and a vector (query or key vector):
T = RoPE(r, p) = R(p)r
where R(p) a rotation matrix applicd to the vector x, encoding positional information using sinusoidal functions. The rotation matrix R(p) is based on sinusoidal function and applies rotations independently to 2-dimensional subspaces of
18

---

TabICL; A Tabular Foundation Model for Large Data
For each 2D pair of vector components (226, F21+ 1), the rotation is defined
R(p)
(0:)(0)) sin( sin (0)) cOs(
22i
22(+1
22i+1
0 = 10000025/4
where d is the dimensionality of the embedding and W, 100002 determines the frequency for each dimension. The above equation indicates that for each dimension pair (2i, 2i 1). the vector is rotated by an angle proportional to the position and the frequency wi. RoPE directly modifies the query Q and key K vectors in the self-attention mechanism:
(R(po)Q) (R(p))K))
Attention(Q. K, K, = softmax
Vd
Because relative positional information is preserved in the inner product, the attention scores naturally encode the relative distance between positions PQ and Рк-
Barbero et al. (2024) provides an alternative perspective on RoPE that aligns well with its use in our work. At lower indices i, the rotation angle changes more rapidly with increasing p. resulting in high-frequency oscillations that resemble random noise and encode positional information. Conversely, at higher indices the rotation angle changes more slowly with increasing p. producing stable values that carry semantic information. In our case, RoPE effectively introduces noise each feature its identifier in controlled, predictable, and generalizable manner.
RoPE is also claimed to exhibit long-term decay, where tokens become less correlated as their relative distance increases. However, this claim is questioned by Barbero et al. (2024), as it relies on an unrealistic oversimplification that queries and keys are equal.
E. Setup of TabICL
E.1. Pretraining details
As outlined in the main paper, employ curriculum learning to progressively increase the size of synthetic datasets (i.e., the number of samples) during pretraining. This process unfolds in three stages by adjusting the micro-batch size used for gradient accumulation:
1. Ni with fixed size of 1,024 the first 160K steps.
2. Ng with the size randomly drawn from log-uniform distribution between 1K and 40K over 2K steps. Activation checkpointing is enabled for datasets exceeding 10K samples, and we accordingly reduce the number of features to avoid out-of-memory issues.
3. Ng = with the size uniformly sampled between 40K and 60K for 50 steps, training only TF. while all other components remain frozen.
Each step comprises 512 datasets. In the first stage, all datasets contain an equal number of samples. the second and third stages, datasets in each micro-batch have the same number samples, but the number of samples varies between different micro-batches. We Adam (Kingma & Ba, 2014) and clip the gradient norm 1. The learning rate schedules for pretraining are shown in Figure E.I. including:
Cosine decay with restarts for stage
Polynomial decay for stage and learning rate is given by (Irinit - (rena) (1 step/T)² + brend, where brini = 26-5. Irand = 56-6, and the number of steps T is 2,000.
Constant learning rate for stage 3
In addition, Figure E.2 shows critical difference diagrams across the three stages of curriculum learning. We observe that average rank of TabICL improves consistently from 11.4 (ninth place), to 7.46 (second), and finally to 6.95 (first). This provides strong empirical evidence for the effectiveness of curriculum learning.
19

---

TabICL; A Tabular Foundation Model for Large Data
x10
x10-5
x10
1.0
2.0
Learning rate
Leamnin rate
Learning rate
1.5
0.5
2
1.0
0.0
0.5
50K
100K
160K
0
500
1000
1500
2000
10
20
30
40
50
Step
Step
Step
(a) Cosine decay with restarts stage
(b) Polynomial decay stage
(c) Constant learning rate for stage 3
Figure E.I. Learning rate schedules for pretraining stages.
E.2. Memory-efficient inference
By employing FlashAttention, have observed that the inference peak GPU memory consumption of the three trans- formers of TabICL for column-wise embedding TFat row-wise interaction TFxow and dataset-wise ICL TFi can be well approximated through the following polynomial regression:
MEM a1 X batch size + a2 seq len + ag batch size seq len
044
It is important to note that the specific meanings of batch size and sequence length vary across different transformers, outlined in Table E.I.
Batch Size
Sequence Length
TF col
Number of features
Number of samples
TF row
Number of samples
Number of features
TF.t
Number of datasets
Number of samples
Table E. Notions of batch size and sequence length for different transformers
Given an input X E R6xxxxm where 6, n, and m represent the number of datasets, the number of samples, and the number of features, respectively, X is first reshaped to (bxxm))n and processed by TFcol get IR (bx Subsequently. E is reshaped the test set entirely through ICL. We in order to efficiently utilize GPU resources and avoid out-of-memoot errors. This is precisely where the aforementioned polynomial regression comes into play.
xmxx and passed TF which generates H xxbxmxid, Finally, H is fed into TFi predict
that it is necessary appropriate batch sizes for different transformers
To this end, we first systematically tracked peak GPU memory usage of different transformers by varying both batch size and sequence length on A 110 GPU with 40GB memory, and then we fit the parameters the above polynomial regression the tracked data, as shown below:
MEMcal (0.0708 batch_size) + (7.29 x 10-6 seq_len) + (0.00391 x batch_size x seq_len) + 137.62
MEM (-2.07 x 10-5 x batch_size) + (2.27 x 10 x seq_len) + (0.00537 x batch_size x seq_len) + 138.54
MEM (- 0.2260 batch_sizer + (4.77 x 10-7 x len + (0.0195 x biteh_size x seq + 140.58
The estimated memory is measured in megabytes (MB).
In addition to adjusting the batch size, we also offload intermediate activations to the CPU or disk as needed to further alleviate GPU memory constraints. Figure E.3 illustrates the CPU and GPU memory consumption for a large dataset containing 100K samples and 500 features (80% training set and 20% test set). As shown, only 5GB of GPU memory and 25GB of CPU memory are utilized, making this highly affordable computational setup.
20

---

TabICL; A Tabular Foundation Model for Large Data
25.0
22.5
20.0
17.5
15.0
12.5
TabNet (25.19)
(6.94| TabPEN
SwitchT [24.95]
[9.32] TabR
Grow.Net
19.721 ModemNNC
KNN [20.78
19.76] RealMLP
LogReg [20.63]
[9.96] TuhM
Tabl 119.94
[10.16] CaiBoost
NODE [19.09]
[10.54] LoCaIPEN
117.95
[11.08] LightGBM
P [17.11]
[11.40] TabaCC-SStaeel
TANGOS 116.92
[11.94] XGBoost
TabCaps [16.64]
[13.15] FI-T
TabPEN [16.25]
(13.70] MLP-IN P-PRR
Excell'
114.451 DCNV2
TubeTables [15.94]
[14.53] RosiNot
RForest 115 80.
1.5.01| MLP
SNN [15.72]
[15.51] Autolnt
(a) Critical difference diagram (based on accuracy) after the stage I of curriculum learning
25.0
22.5
20.0
15.0
12.5
10.0
TabNet [25 28
[7.13] TabPPNN22
Switch/ [25.01]
17.46| TablCCLSStug22
GrowNet [22.77
19.45] TabR
KNN [20.88
[9.85] ModernNCA
LogReg 120.63
19.87| ReaiMLP
TabT 119.98
[10.06] TsshM
[10.30] CatBoost
DANets [18.07]
[10.66] LoCaPPIN
PTaRL [17.20
[11.20] LightGBM
TANGOS [17.07
[12.07] XGB-oott
TabCaps [16.82
[13.33 FTE-T
ExcelF [16.34
[13.87] MLP-PLR
TabPFN [16.31]
[14.54] DCNV2
Tables [15. 99
[14.63] ResNet
[15.11]
SNN [15.84
115.63] Autolnt
(b) Critical difference diagram (based accuracy) after the stage 2 of curriculum learning
25.0
22.5
20.0
17.5
15.0
12.5
10.0
7.5
TabNei
46.95| TabbCCL-Stagee
SwitchT (24.99)
17.17] TabPFNN2
GrowNet (22.76
19.46] TabR
(20.86)
[9.84]
LogReg [20.63
[9.90] RealMPP
TabT
[10.11
NODE [19.16]
[10.35] CaiBoost
DANets [18.09
|10.67| LoCaIPEN
[17.22]
[11.25] LighIGBM
TANGOS 117.06
[12.09] XGBBoos
TabCaps [16.83]
[13.30] FT-T
Excell [16.37
[13.90] ML.P-P PLP
TapPEN [16.31
[14.55] DCNV2
Tume [16.00]
[14.65]
RiFrrstt 115 92
115.121 MLP
SNN [15.84]
[15.64] AutoInt
(c) Critical difference diagram (based on accuracy) after the stage 3 curriculum learning
Figure E.2. Critical difference diagrams for different stages of curriculum learning
21

---

TabICL; A Tabular Foundation Model for Large Data
25
Column- wise embedding
20
Row-wise interaction
CPU Memorr (GB)
GPU Memory (GB)
10
Datasel-wisel[C.
12
15
18
Time (S)
CPU and GPU Memory Usage without Automatic Mixed Precision
Row-wise interaction
20
CPU Memory (GB)
Column-wise embedding
GPU Memory (GB)
15
10
Dataset-wist ICs
Time (S)
(b) CPU and GPU Memory Usage with Automatic Mixed Precision
Figure E.3. CPU and GPU memory consumption during inference for dataset with 100K samples and 500 features. During dataset-wise ICI,, there is only single hatch (i.e., a single dataset), which corresponds to sharp spike in GPU usage at the final stage of computation.
Figure E.4 shows the CPU and GPU memory consumption for a larger dataset containing 500K samples and 500 features (80% training set and 20% test set). As depicted, the computation requires less than 14GB GPU memory. The CPU memory usage reaches approximately 120GB, which, however, can significantly reduced through optional disk offloading via memory mapping.
We observe that GPU memory consumption exhibits periodic fluctuations during the column-wise embedding and row-wise interaction phases. This is because the entire dataset is automatically divided into multiple batches, with the batch size determined dynamically based on the polynomial regression mentioned earlier. Additionally, during column-wise embedding, the output of TFal is progressively offloaded to CPU. As result, we can see an incremental increase in CPU memory usage throughout this stage.
We can also see that enabling automatic mixed precision highly reduces both memory consumption and computation time.
22

---

TabICL: A Tabular Foundation Model for Large Data
Column embeukding
Row-wise interaction
CPU Memory (GB)
10
GPU GPU Memory (GB)
80
60
Dataset- wise
40
20
2
200
Time (S)
(a) CPU and GPU Memory Usage without Automatic Mixed Precision
14
120
12
Column wistm embedding.
Row-wise interaction
CPU Memory (GB)
GPU Memory (GB)
80
Datasel wise ICL
0
20
10
60
80
Time
CPU and GPU Memory Usage with Automatic Mixed Precision
Figure E.4. CPU and GPU memory consumption during inference for dataset with 500K samples and 500 features. The relatively high CPU memory consumption can largely mitigated by optional disk offloading via memory mapping.
23

---

TabICL: A Tabular Foundation Model for Large Data
E.3. Hierarchical Class-extension Strategy
Due to pretraining limitations, TabICL cannot natively handle classification problems with more than 10 classes. To address this, we employ hierarchical classification to extend the class scalability of TabICL. This involves building multi-level classification tree where:
1. Each internal node splits the classes assigned to it into at most 10 child groups.
2. Recursion stops when node contains < 10 classes, turning it into a leaf.
Since each split reduces the class count by at least a factor of 10, the tree depth r = [oo kt.. where i: denotes the total number of classes.
Within the tree, each internal node predicts the probabilities of its child groups, while each leaf predicts the probabilities of the classes it contains. For any given class c, the unique path from the root to its containing leaf can be represented by sequence of nodes:
(c) = root, gi(c), 92(c),.... -r-1 (c): (r(c)
where (c) denotes the node depth i on the path leading class Applying the probability chain rule along the path yields the overall posterior probability:
Au c = Id gi-((c), x, D (en)
where x is the given test sample, and D (c) represents the training data associated with the node gi-1(c). Note that the targets in D (c)) are the child group indices for the node (c).
F. Excluded Development Datasets of TabPFNv2
TabPFNv2 utilized several development datasets for hyperparameter tuning (e.g., learning rate, prior-related parameters), early stopping to prevent overfitting by monitoring performance on these datasets, and model selection to identify the best-performing model among multiple candidates. These development datasets are provided in the supplementary material of TabPFNy2 (Hollmann et al., 2025).
For fair comparison, we excluded these development datasets from the main paper. In the table below, we report the performance of several high-performing methods on the development datasets included in the TALENT benchmark:
Table F.I. Performance on 15 development datasets of TabPFNw2
TabICL
TahPFNv2
TahR
MedernNCA
RealMLP
TabM
CaBoost
Datasel
Acc.
AUC
LogL
Acc.
AUC
LogL
Acc.
AUC
LogL
Ace.
AUC
LogL.
Acc.
AUC
LogL
Acc.
AUC
LogL
Acc.
AUC
LogL.
allbp
0.973
0.975 0.081
0.978
0.978
0.070
0.970
0.915
0,150
0.971
0.969
0.093
0.971
0.845
0.145
0.974
0.910
0.252
0.978
0.942
0.070
baseball
0.933
0.160
0.969
0.947
0.965
0.171
0.951
0.207
0.937
0.248
0.155
0.965
delta.aileroms
0.952
0.981 0.161
0.954
0.982
0.159
0.948
0.978
0.175
0.949
0.980
0.165 0.945
0.976
0.184
0.950
0.979
0.172
0.948
0.979
0.164
eye movemennt
0.639
0.757
0.788
0.930
0.488
0.970
0.996
0.986
0.998
0.767
0,900
0.994
1.000
0.043
0.883 0.623
bix
0.588 0.648 0.655
0.678
0.751
0.597
0.659
0.732
1.344
0.909
0.974
0.207 0.596
0.639
2.244
0.593
0.628
0.668 0.615
0.672 0.642
beloe
0.720 0.799 0.545
0.726
0.803
0.543
0.722
0.795
0.555
0.721
0.797
0.547
0.722
0.798 0.551
0.718
0.789
0.557 0.723
0.799
0.544
bill-vally
0.785 0.917 0.518
0.983
0.992
0.098
0.700
0.732
0.571
0.781
0.804
1.455
0.796
0.852
0.547
0.512
0.529
0.703 0.517
0.515 0.693
Japancse Vowels
0.997 1.000 0.011
0.997
1.000
0.008
0.996
1.000
0.016
0.996
1,000
0.013
0.994
1.000
0.113
0.993
1.000
0.051
0.983
1.000 0.048
led24
0.731
0.738
0.732
0.954
0.827
0.956
national longitudinal
0.998 1.000 0.005
1.000
1.000
0.000
0.994
1.000
0.029
0.999
1.000
0.003
0.995
0.996
0.064
1.000
1.000
0.000
1.000
1.000 0.012
page-blocks
0.976 0.993 0.092
0.975
0.993
0.092
0.968
0.987
0.127
0.967
0.989
0.129
0.971
0.980
0.162
0.968
0.967
0.991
ringnorm
0.980 0.997 0.066
0.980
0.997
0.055
0.980
0.995
0.075
0,980
0.996
0.072
0.976
0.996
0.079
0.980
0.997
0.062
0.971
0.995 0.083
0.763 0.853 0.472
0.860
0.937
0.322
0.878
0.934
0.334
0.837
0.916
0.378
0.747
0.824
0.548
0.787
0.869
0.552
0.786
0.863 0.463
thyroid ann
0.987 0.999 0.028
0.995
1.000
0.017
0.986
0.996
0.068
0.987
0.996
0.088
0.992
0.998
0.087
0.992
1.000
0.024
0.995
1.000 0.012
wavefoem-5000
0.872 0.978 0.259
0.80 865
0.977
0.263
0.860
0.975
0.310
0.862
0.975
0.278
0.867
0.975
0.291
0.862
0.976
0.279
0.860
0.973 0.297
G. Random Forest Extension of TabPFNv2 for Large Datasets
Hollmann et al. (2025) proposed an extension of TabPFNv2 using random forest (RF) to enhance its performance on large datasets. During training, the extension recursively partitions the feature space using standard decision tree splitting criteria.
24

---

TabICL; A Tabular Foundation Model for Large Data
At inference time, instead of relying on simple majority voting at the leaves, it fits TabPFNv2 the subset of training data that reaches each leaf. This extension only leverages the inductive biases of tree-based models but also reduces the number of samples processed by TabPFNv2, thereby mitigating its scalability limitations.
The ensemble prediction of this RF extension involves two main steps: (I) routing each test sample through all trees to their corresponding leaf nodes, and (2) aggregating the predictions from TabPFNv2 at these leaves by either averaging probabilities or logits.
key innovation of this RF extension is the adaptive pruning mechanism that dynamically determines whether use parent node predictions or leaf node predictions based validation performance. This prevents overfitting and optimizes the tree depth for each region of the feature space.
This extension is not exclusive to TabPFNv2 and can also be applied to other tabular foundation models like TabICL. We evaluated this RF extension with adaptive pruning enabled for both TabPFNv2 and TabICL on large datasets with more than 10K samples. Additionally, we set the ensemble size for both TabPFNv2 and TabICL to 4, considering that random forest already ensembles multiple decision trees. shown in Figure G.I, the RF extension significantly improves the performance of both TabPFNy2 and TabICL.
15
SwitchT [28.72]
7.700 TabR
LogReg [26.23
17773| TabICL-RF
GrowNet [26.21
88088 RealMLP
126.15
TahNett (25.17)
[9.17] TahPPNN2.RF
KNN [24.92
99.42] ModemnNAA
Tabelables (24.70)
99.85] TablCL
(22.66)
100.22] CarRonst
NODE [22.04
[1.81]] MLP-PLR
RForest [22.04
12.741 FTTT
TANGOS (20.60)
(12.75] XGBoost
TabCaps [19.34]
[13.26] LightGBM
DANeis [18.77
113.43| TabPFNv2
PTaR1 [16.43]
(14.30) DCNV2
[16.17
[14.75] MLP
ExcelF [15.91]
[15.11]] Antolen
RosNet [15.60]
[15.28] LoCalPEN
Figure G.I. Critical difference diagram (based accuracy) on 53 large datasets with more than 10K samples. For TabICL-RF and TabPFNy2-RF, "RF" suffix indicates the use of the random forest extension.
H. Effect of Ensemble Size on TabPFNv2 and TabICL
Both TabPFNv2 and TabICL break column-order invariance mitigate the learning collapse problem, described in Section 3.3. To approximately restore permutation invariance, they aggregate predictions across multiple column permutations, while also using different preprocessors and shuffling class labels. We analyze the effect of ensemble size on their performance, as shown in Figure H.1. Our findings indicate that TabPFNv2 shows a clear improvement even with a small ensemble size (S 4), whereas TabICL requires a larger ensemble size (2 8) for noticeable performance gain.
Overall, TabPFNv2 benefits more from ensembling than TabICL. This could be attributed
TabPFNv2 employs more sophisticated preprocessing operations compared to TabICL.
TabPFNv2 benefits more from column permutations inherently. Unlike TabICL, TabPFNv2 never collapses the column dimension. This means permuting columns could have more pronounced impact the outputs of TabPFNv2, which reduces the correlation between ensemble members of TabPFNv2, thereby improving the effectiveness of ensembling.
25

---

TabICL: A Tabular Foundation Model for Large Data
TabPFNv2
TahICI.
Rel. accuracy improvements over B
1.0%
Meam
Rel. uccuracy improvementss over E
1.0%
Mean
0.8%%
0.64%
0.68% %
0.74%
0.755
0.85
0.61 %
0.72%
0.76%
0,4|4
0.4%
0.32%
0.4%
10.2%
0.24%
0.27%
0.30%
0.26%
0.255
0.40%
0.00%
0.18%
0.00%
0.15%
0.15%
10.1
0.66%
0.00%
0.00%
0.00%
-0.2%
0.2%
16
32
Ensemble size
Ensemble size E
(a) TabPFNv2
(b) TabICL
Figure H.I. Effect of ensemble size TabPFN'2 and TabICL. The figures show the relative accuracy improvements of different ensemble sizes over no ensembling across 171 datasets with at most 10 classes.
Average Performance and Rankings
In this section, present the average rank of all methods across different dataset categories, including binary datasets, multi-class datasets 1S10 classes), small classification datasets (<10K samples), and large classification datasets (> 10K samples). The rankings are computed based on accuracy, AUC, and Log Loss. The average rank is given in critical difference diagrams, with Wilcoxon-Holmm tests with a significance level 0.05. The lower the rank value, the better the performance.
is important to note that accuracy is used as the objective metric for hyperparameter tuning in tabular methods. Moreover, LoCalPEN (Thomas al., 2024) and TuneTables (Feuer et al., 2024) are only included the accuracy-baat -aased rankings.
26

---

TabICL: A Tabular Foundation Model for Large Data
25.0
22.5
20.0
17.5
15.0
12.5
10.0
TabNet [25.27
16.95] TabiCL
SwitchT [24.99
17.17] TabPFNv2
GrowNet [22.76
KNN [20.86
19.84] ModernNCA
LogReg [20.63]
[9,90] RealMLP
Tabl 119.99
110.11 'TabM
NODE [19.16]
[10.35] CatBoost
DANets
[10.67| LoCalPEN
PTaRL [17.22]
[11.25] LightGBM
TANGOS [17.06
[12.09] XGBoost
TabCaps [16.83
[13.30] FI-T
EscelF [16.37
[13.90] MLP-PLE PLR
TabPFN
[14.55]
TuneTables [16.00]
[14.65] RosNet
RForest 11..22
115.12] MLP
SNN [1.84
[15.64] Autolnt
(a) Accuracy
20
5
TabNet [26.09]
[5.59] TabICL
SwitchT [24.58)
TabPENv2
KNN [22.95]
[9.42] CatBoost
GrowNet (22.06)
[9.75] ModernNCA
TabCaps [30.13)
[9.89] TahM
LogReg (20.00)
[11.05] TabR
TabT' 119:45
[11.39] LighiGBM
NODE [18.93]
[11.61] XGBoost
DANets (18.12)
[12.37] RealMPP
PTaRL [17.46]
[13.33] FET
TANGOS [16.59)
[13.40] MLP-PLR
ExcelF [15.66
[13.89] TabPEN
DCNv2 [15
[14.55] RosNet
SNN [15.16]
[14.90] Autolnt
RFFrrst| [15.04
[14.91 MLP
(b) AUC
25
20
15
10
5
KNN (25.80)
[4.21) TabICI.
TabCaps [22.70]
44.47] TabPENv2
TabNet [22.46]
[8.18] CatBoost
SwitchT [21.80)
[10.76] ModernNCA
GrowNet (20.23)
[10.89] TahM
DANets [20.11]
[10.96]
TabT [19.83]
[11.86] FTT
NODE [19.15]
[12.41] LightGBM
LogReg [17.92]
[12.88] TabPFN
PTaRL [17.90]
[13.63] Autoint
MLP [17.26]
[13.73] Excelll
SNN [17.15]
[14.71] ResNet
DCNv2 [16.83]
(14.90) XGBoost
MLP-PLR [16.10]
[15 08. TANGOS
RealMLP [15.55]
[15.24] RFarest
(c) Log Loss
Figure I.I. Critical difference diagram for all 171 classification datasets (S 10 classes).
27

---

TabICL: A Tabular Foundation Model for Large Data
25.0
22.5
20.0
17.5
15.0
12.5
10.0
7.5
TabNet (24.74)
[6.32] TabCCL
SwitchT [24.51]
[7.10] TahPPFN22
GrowNet (21.87)
[9.94] TabR
KNN [21.52]
[10.30] LoCalPFN
LogReg [20.49)
[10.38] CatBoost
TabT [19.21
[10.43] ModemnNA
DANets 117.99
[10.74] TTaM
NODE| [17.50
[10.76] RealMLP
TANGOS [16.87
[10.96| LightIBM
TabCaps [16.64]
[11.83] XGBoost
ExcelF [16.22
[13.22| FTFT
[16.18]
[13.87] MLP-PLR
SNN [16.12]
[14.71] DCNV2
RForest [15.75]
[15.15] Autaoint
ResNet [15.68]
[15.27] MLP
TuncTables [15.4]]
{15.3 1]] TabPFN
(a) Accuracy
25
15
10
TabNet [26.35]
(4.88) TabICL
Switch1 [24.22]
TabPFNv2
KNN [22.88)
[9.68] CatBoost
GrowNet [21.60]
[10.19] ModernNCA
TabCaps [19.77]
[10.25] TabR
TabT [19.58]
[10.62] TahM
LogReg [19:47]
[11.10] LighTGBM
DANets [18.39]
[11.51] XGBoost
NODE [17.65]
[12.88] FTT:
TANGOS [17.23
112.92 TRPPEN
PTaRL [17.01]
[12.98] RealMLP
DCNv2 115.95
113.75| MLP-PLR
ExcelP [15.5] 88]
[14.55] RForest
SNN [15.59]
[15.04] ML.P
ResNet [5..46]
155.12| Autolat
(b) AUC
25
20
15
10
5
KNN (25.76)
(4.00) TabICI
TabCaps [23.58]
[4.40] TabPENv2
[22.47]
[8.65] CatRoost
SwitchT [21.88]
10.91] TahM
DANets [20.42]
11.04] ModernNCA
TabT [20.01
[11.15] TahR
GrowNet [19.8]
111.88| PTT-
[17.8]]
LighiGBM
NODE [17.73]
112.55 TabPEN
ML. P [17.62]
[13.60] ExcelF
SNN [17.39]
[13.70] AutoInt
LogReg [17.31]
[14.35] XGBoost
DCNv2 [17.08]
[14.54]
MLP-PLR [16.26]
115.48] ResNet
TANGOS [15.86]
[15.64] RealMLPP
(c) Log Loss
Figure 1.2, Critical difference diagram for 112 binary classification datasets.
28

---

TabICL; A Tabular Foundation Model for Large Data
25.0
22.5
20.0
17.5
15.0
12.5
10.0
7.5
126.27
TabPHNV2
SwitchT [25.90
8.14] TabICL
GrowNet [24.46
18.27] RealMLP
NODE [22.31
(8.56] TahR
[21.47
ModernNCA
LogReg (20.90)
8: 8.90) TahM
KNN [19.6]
[10.29] CatRoost
PTaRL [19.19]
[11.37] LoCalPEN
DANets 118:27
11.81| LightGBM
TabPEN [18.20
12.58] XGBoost
TANGOS [17.42
[2.69] ResNNe
TabCaps [17.19
TuneTables [17.00)
[13.97] MLP P. PLR
ExcelF [16.64]
[4.25] DCNv2
Autolat [16.58
[4.85] MI.
RForest [16.25]
[5529] SNN
(a) Accuracy
25.0
22.5
20.0
17.5
15.0
12.5
10.0
7.5
TabNet [25.60
TabPENN2
SwitchT [25 2
[6.92] TabICL
KNN [23.08
[8.52] TabM
Gotw (22.92
18.92] CatBoost
NODE (21.33)
[8.92] ModleraNCA
LogReg
RealMLP
TabCaps [20.82]
[11.80] XGBoost
"TaT' 119.18
111.951 LightGBM
PTaRL [18.32]
[12.57] TabR
DANets [17.60)
[12.75] MELP-PL
RFerest 95.97
TabPFN [15.72]
[14.18] FTT
TANGOS 115.38
[14.35] SNN
Excell' [15.25]
[14.50]
MLP 114.67
[14.62| DXNV2
(b) AUC
25
20
15
10
5
KNN [25.88]
[4.60] TabPFNw2
[22.48]
[4.60] TabICL
NODE [31.83]
[7.28] CaBoostt
SwitchT [21.67]
[10.22] ModernNCA
TabCaps [21.05)
[10.60] TabR
Grow/et [21 00)
[10.85]
DANets [19.52]
[11.82] FT-T
TabT [19.50)
113.251 ResNet
LogReg [19.08]
(13.40) Ligh1GBM
PTaRL [18.07
113.45] TabPEN
SNN [16.70)
[13.50] Autoint
MLP [16.58
113660] TANGOS
RForest [16.55)
[13.98] Excell?
DCNv2 [16.37]
[15.5] 38] RealMLP
XGBoost [15.93]
[15.82] ML.P PLR
(c) Log Loss
Figure I.3. Critical difference diagram for 59 multi-class classification datasets (S 10 classes).
29

---

TabICL: A Tabular Foundation Model for Large Data
25
15
5
TabPHN22
Switch1 [24.13
[6.27] TabICL
GrowNet [21.92]
99.05] LoCaIPEN
KNN [19.59
100.3] ModernNCA
[10.76] CatBoost
NODE 18:60
[10:76] TabR
I LogReg [1.66
[10.87] LighIGBM
DANcEs [18.4]
111.14] RealVLP
PTaRL 118.10
11.34 TTab
ExcelF [17.32
12.34] XGBooott
Autolat [16.5
12.57] TahPFN
TabCags 11.34
Tune Tables
SNN [16.33]
[13.98] RForest
TANGOS [16.24]
[4.20] PT-T
MI.P P [15.8]]
14.84] Reshet
MLP-PLR [15:47]
[15.22] DCNv2
(a) Accuracy
20
TabNet (27.50) [
[3.61] Tab6PTab
TabPPNv2
Switchl (24.09)
[4.65] TabICL
KNN 121.93
[9.89] TabPEN
GrowNet [21.33]
[10.12] CaiBoost
TabCaps [20.03
110.72] TabM
TabT [18.98]
[11.16] ModernNCA
PTaRL [18.81]
[11.84] LightGBM
NODE [18.72]
[12.33] XGBoost
DANets [18.47]
[12.65] TabR
LogReg [18.34]
[12.73] RealMLP
ExcelP [16.62]
[14.01] RForest
DCN22 [16.25]
[14:29] MLP-PLR
Autolnt 116.05
114.35| FF-T
TANGOS [15.89]
[14.52] ResNet
MLP [15.29]
115.28] SNN
(b) AUC
25
20
15
10
KNN (24.52)
[2.49] TabPFNN2
[23.41]
[3.38] TabICL
TabCaps (22.69)
[8.28] CatBoost
DANets [21.02]
[8.88] TabPPN
SwitchT [20.78]
[11.19] TabM
NODE [19.83]
[12.15] ModernNCA
TabT [19:46]
[12.17] FT-T
GrowNet [19.43]
112.78] TabR
PTaRL [19.15]
113.24] Ligh1GBM
MLP [17.92]
[13.66] Risrrest
DCNv2 [17.78]
[13.69] Autoint
MLP-PLR [17.35]
114.16] Excell"
SNN [17.03]
[14.81] TANGOS
XGBoost [16.52]
[15. 13] ResNet
RealMLP [16.16]
[15.63] LogReg
(c) Log Loss
Figure I.4. Critical difference diagram for 116 small classification datasets (S 10K samples).
30

---

TabICL; A Tabular Foundation Model for Large Data
27.5
25.0
22.5
20.0
17.5
15.0
12.5
10.0
7.5
SwitchT 126.91
66.79 TabR
LogRog [24.55]
77117] ReaMMLP
7.60
TabPEN (24.30)
88.51] ModernNCA
TabiNet [23.38
[8.64] TahiCL
KNN [23.26
19.68] CatBoost
TuncTables [22.92)
[10.58] MLP-PLR
TabT
111.49] FFTT
NODE [20.43)
[11.64] XGBoost
RForest [20.36
[11.9]] TabPENN2
TANGOS [18.94]
[12.02] LightGBM
TabCaps [17.68
[12.96] DCNV2
DANett
PTaRL [15.00]
[13,72] AutoInt
SNN 14.77
113.96 LoCaIPEN
ExcelF [14.45]
[14.15] Roshet
(a) Accuracy
25.0
22.5
20.0
17.5
15.0
12.5
10.0
SwitchT 125.76
6..99 ModerrNCA
KNN (24.89)
77677] TabICL
GrowNet [23.55
7.85| TabR
LogRey [23.18)
[8.13] CatBoost
TabNet [23.1]
88.44] TabM
TahPEN (22.02)
[10.24] XGBoost
TabCaps [20.29)
[10.29] LightGBM
TahT [20.09)
[11.18] TabPFNv2
NODE [19.84] 19.84)
[11.42] FT-T
TANGOS [18.07]
[11.55] MLP-PLR
DANets [17.42]
[11.62] RealMLP
RForest [17.05]
[12.56] Autolot
SNN 114.85
[13.76] DCNV2
ResNet [14.56]
[13.85] MLP
PTRRL [14.47]
[13.93] Excell
(b) AUC
25
20
15
10
KNN [28.36]
(6.00) TahICI.
[23.98] 98)
[7.31] TabR
TabCaps [22.55]
[7.58] MademaNAA
LogReg [22.42]
[8. CatBoost
GrowNet 19)
[8.75] TabPFNv2
TabPPN [21
[10.58] LightGBM
TabNet (20.38)
[10.60] TabMi
TabT 120.35
[11.38] FET
RFerest [18.47]
[11.76] XGBoost
DANets 118.33
[13.05] Excell"
NODE [18.16]
[13.49] MIP.PLR
SNN 117.56
[13.60] Autolnt
MLPP [15.67]
[13.76] ResNet
TANGOS [15.65]
[14.16] RealMLP
PTaRL [15.09]
[14.71] DCNv2
(c) Log Loss
Figure I.5. Critical difference diagram for 53 large classification datasets (> 10K samples).
31


Reviews:
[{"review_id": "EuybVfNoF4", "reviewer": "Reviewer_rh11", "summary": "The paper introduces TabICL, a tabular foundation model designed to scale in-context learning (ICL) for tabular data with a two-stage architecture. This design allows TabICL to handle datasets with up to 500K samples using affordable resources. On the TALENT benchmark with 200 datasets, TabICL achieves comparable performance to TabPFNv2 while being up to ten times faster. On datasets exceeding 10K samples, TabICL outperforms both TabPFNv2 and CatBoost, highlighting its scalability and effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable feedback.\n\n**We kindly invite the reviewer to refer to the figures available at https://limewire.com/d/FhJbU#LPxKd0hlro, which may help better illustrate our responses. The link only contains figures.**\n\nWe realize that there may be a misunderstanding of how column-wise embedding and row-wise interaction operate. So, we decide to first explain them.\n\nFor tabular data with $n$ rows and $m$ columns: \n- Column-wise embedding applies attention across all $n$ samples within each column, mapping scalar values to embeddings\n- Row-wise interaction applies attention across all $m$ features (plus 4 [CLS] tokens) within each row\n\nWe adopted \"column-wise\"/\"row-wise\" from BISHOP (ICML 2024), but acknowledge this may cause confusion. Therefore, we will rename:\n- Column-wise embedding → Column-wise inter-sample embedding \n- Row-wise interaction → Row-wise inter-feature interaction\n\nWe welcome any alternative naming suggestions from the reviewer. \n\nNext, we respond to your questions.  \n\n---  \n1. *Why use column-wise embedding and row-wise interaction instead of the opposite ?*  \nThe core idea of TabICL is to encode each row into a fixed-dimensional embedding before applying in-context learning (ICL). Row-wise interaction is used to aggregate row-level information into a single embedding, which positions it between column-wise embedding and ICL.\n---\n2. *In Figure 1 ... shouldn’t it be 4+n?*  \nAs explained above, row-wise inter-feature interaction involves the attention between $m$ features as well as 4 [CLS] tokens within each row. Therefore, it is $4+m$.\n---\n3. *What is the overall complexity formulation?*  \nColumn-wise embedding $TF_{col}$ uses induced self-attention blocks (ISAB), while row-wise interaction $TF_{row}$ and dataset-wise in-context learning $TF_{icl}$ use vanilla self-attention blocks (SAB).\n\n- SAB has quadratic complexity :  $O(L_{seq}^2)$\n- ISAB has linear complexity: $O(L_{seq} \\ k)$, where $k$ is the number of inducing vectors\n\n| Block | Description  | Complexity |\n| --- | --- | --- |\n| $\\text{TF}_{col}$ | Attn over $n$ samples per column (computed $m$ times)  | $O(n \\ k \\ m)$ |\n| $\\text{TF}_{row}$ | Attn over $m$ features and 4 [CLS] tokens per row (computed $n$ times) | $O((m+4)^2 \\ n)$ |\n| $\\text{TF}_{icl}$ | Attn over $n$ samples for the full dataset (computed once)  | $O(n^2)$  |\n\nIn summary,\n- $TF_{col}$ has linear complexity w.r.t. $n$\n- $TF_{row}$ has quadratic complexity w.r.t. $m$, but $m \\ll n$ in most tabular datasets\n- $TF_{icl}$ has quadratic complexity w.r.t. $n$, but it is computed only once because $TF_{row}$ collapses the feature dimension\n\nIn contrast, TabPFNv2 uses SAB for sample attention $O(n^2 \\ m)$ and feature attention $O(m^2 n)$. Since it never collapses columns, sample attention becomes prohibitive for large $n$ and moderate $m$. \n\n---\n4. *For the tree-based synthetic data generator, ... ?*  \nWe fit a XGBoost regressor to Gaussian noise targets $y_i$. In practice, we limit the max depth to 4 to reduce both fitting time and model capacity, preventing it from fully capturing the randomness. This forces XGBoost to balance between noise and its limited capacity, resulting in a practical randomized initializer—validated by our experimental results.\n---\n5. *Are you using 8 A100 GPUs ?*  \nWe used three A100 GPUs, as mentioned L278-281.\n---\n6. *The paper misses ablation studies on key components*  \n- Ablation study on curriculum learning  \nWe evaluate the pre-trained checkpoints after each stage of curriculum learning.  The table below shows the average relative accuracy improvements over the stage 1 (Acc Imp) and the average rank:\n\n|    | Acc Imp across all datasets | Acc Imp across large datasets with > 20K rows | Avg. Rank |\n| --| -- | -- | -- |\n| Stage 1 | -- | -- | 10.52 (7th) |\n| Stage 2 | 2%     | 6%  | 6.87 (2nd)  |\n| Stage 3 | 2.3%  | 7%  | 6.30 (1st)   |\n\nStages 2 and 3 effectively increase the rank of TabICL from 10.52 (7th) to 6.30 (1st).\n- Ablation study on tree-based generation  \nWe compare TabICL pre-trained with and without tree-based generation, using the stage 1 setup for 20K steps (to reduce training time). The average relative improvement of tree vs. no tree is as follows:\n| Accuracy | AUC   | Log Loss |\n| --- | --- | --- |\n| 0.37% | 0.18% | 2.5% |\n\n---\n7. *Did you compare with the \"data grouping based on random forest\" method from TabPFNv2?*    \nWe applied this random forest extension to both TabPFNv2 and TabICL on datasets with over 10K samples (the reported limitation for TabPFNv2). Results show performance gains for both models:\n\n| Model     | Avg. Rank |\n| --- | --- |\n| TabICL    | 6.68 |\n| TabR      | 6.74  |\n| RealMLP   | 6.98 |\n| ModernNCA | 8.52 |\n| TabPFNv2  | 9.14  |\n\nTabPFNv2 improves from 10.89 (see Figure E.5) to 9.14, while TabICL becomes the best-performing method on large datasets !\n\n---  \n\n**In the revised version, we will include complexity analysis, ablation studies, and random forest extension.**", "claims_and_evidence": "The paper supports most claims by empirical results. A few concerns are noted below.", "methods_and_evaluation_criteria": "Why use column-wise embedding and row-wise interaction instead of the opposite?\n\nIn Figure 1, why is the notation for the last token 4+m? Since the process involves row-wise interaction, shouldn’t it be 4+n?\n\nWhat is the overall complexity formulation for the three blocks of TabICL? For the row-wise interaction step, is the time complexity (4+n)^2? If so, how does this remain computationally efficient? Could further formulations on computational efficiency be provided?\n\nFor the tree-based synthetic data generator, are you computing the predictions of XGBoost based on its training set? Wouldn't the predicted results be very close to the ground truth on the training set which is essentially random yi? I don't quite get the points here.", "theoretical_claims": "The paper focuses on improving the empirical performance.", "experimental_designs_or_analyses": "In the first step of curriculum training, is the total number of synthetic datasets 100K * 512 = 51M, each with 1024 rows and ≤100 features? Are you using 8 A100 GPUs? The reported training duration (10 days on 8 A100 GPUs) seems longer than expected. \n\nThe paper misses ablation studies on key components, such as continual learning and tree-based dataset generation.\n\nDid you compare with the \"data grouping based on random forest\" method from TabPFNv2, which is used to handle large sample sizes?", "relation_to_broader_scientific_literature": "The paper is well-situated in the context of tabular foundation model learning and ICL.", "essential_references_not_discussed": "The paper covers most works in tabular foundation model learning.", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cLY5cq2aiI", "reviewer": "Reviewer_ba5G", "summary": "This paper introduces a tabular foundation model, TabICL, to deal with the classification of larger number of table samples at affordable resources. Compred to the recent state-of-the-art TabPFNv2, the proposed TabICL has competitive performance while enjoys up to 10 times faster speed. The proposed architecture mainly involves three components, which are column-wise transformer for feature embedding, row-wise transformer module for rows interaction, and an additional transformer module for in-context learning. During feature embedding stage, the module is built on top of set transformer, which can capture distribution-aware features. During row features interaction stage, rotary position encoding is used to avoid mode collapse. The experimental results on TALENT benchmark shows the effectiveness of the proposed foundation model.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable feedback. We now respond to the reviewer's questions. \n \n---\n1. *Why should the training be invariant to permutation of columns?*\n\nIn relational data, column order is typically arbitrary. The very visible publication [Why Tabular Foundation Models Should Be a Research Priority] has put forward that tabular methods are expected to be invariant to column order. This is, however, not our battle.\n\n*If there is a 'sum' column, will it be sensitive to column order?*\n\nWe may not fully understand the question, but in principle, the position of a \"sum\" column should not be sensitive to column order.\n\n---   \n\n2. *It gives me a contradictory feeling ...*\n\nYes, this is indeed a challenging issue. In fact, both TabPFNv1 and TabPFNv2 face the same mode collapse problem and address it similarly—by first breaking feature-order invariance and then restoring it via ensembling. They differ in how the invariance is broken: TabPFNv2 uses random feature embeddings and feature grouping, while TabPFNv1 uses fixed feature embedding layers.\n\n*Will mode collapse reoccur after restoring permutation invariance?*\n\nNo, this will not happen. Ensembling aggregates predictions from independently permuted feature orders. Each permutation receives unique positional encodings via RoPE, leading to diverse, non-collapsed representations. Ensembling effectively averages out positional biases (e.g., effects of column order). This is analogous to data augmentation — diverse permutations provide complementary \"views\" of the data, enhancing robustness of TabICL. In short, RoPE prevents local collapse (within each permutation), and ensembling restores global invariance without reintroducing collapse. The excellent performance of TabICL also confirms this empirically.\n \n---  \n3. *Since the performance of the proposed method is so close to TabPFN2...*\n\nWhile TabICL and TabPFNv2 achieve comparable overall performance, TabICL offers key advantages in handling large datasets, computational efficiency, and scalability — crucial for real-world industrial applications. We acknowledge the need to better clarify these contributions. In the revised manuscript (Related Work, L143–148), we will explicitly emphasize:\n- Methodological differences: TabPFNv2 uses alternating column-row attention, while TabICL adopts a two-stage design (fixed row embeddings + ICL) to reduce complexity.\n- Computational complexity: Detailed comparison is provided in our response to the question 3 of the reviewer rh11.\n- Efficiency and scalability: Including memory optimizations, support for arbitrary class counts, faster inference, etc\n---\n\n4. *The comparison of inference speeds with other methods are not listed in paper...*  \n\nWe focus on inference speed comparison with TabPFNv2, as both are tabular foundation models that perform well without hyperparameter tuning—unlike most other tabular methods. Unfortunately, the TALENT codebase does not report pure inference times excluding tuning and fitting.\n\nWhen comparing inference speed alone, neither TabPFNv2 nor TabICL shows an advantage. For instance, as noted in the TabPFNv2 paper, on a dataset with 10,000 rows and 10 columns, TabPFNv2 takes 0.2s per sample, while CatBoost (with default settings) requires only 0.0002s for inference.\n\nTheir real strength of tabular foundation models lies in the elimination of hyper-parameter tuning and cross-table transferability — qualities that are critical for real-world industrial applications and emphasized in [Why Tabular Foundation Models Should Be a Research Priority].\n\n--- \n5. *The reviewer wonder the concrete examples of the in-context learning in tabular classification...*\n\nIn-context learning for tabular classification can in principle solve all of the applications that regular tabular classification models can solve. For example, some applications of TabPFN are listed at https://github.com/PriorLabs/awesome-tabpfn.\n\nNote that synthetic datasets are only used for pre-training. Yes, our proposed TabICL can deal with real-world datasets. As shown in the paper, TabICL achieves SOTA results on the TALENT benchmark comprising 200 real-world datasets collected from 13 distinct domains, such as education, healthcare, and finance.\n\n--- \n6. *Suggest to indicate that the red bar ...*  \n\nThank you for your feedback. \n\n- In the caption of figure 5, we will indicate that the red bar represents the median relative accuracy improvements over MLP across all datasets for each method.\n\n- The black dots outside the whiskers represent outliers—datasets where the relative accuracy improvements of a method deviates significantly from the majority of results.\n\n- Horizontal Bars (Whiskers): The horizontal bars (whiskers) extending from the box indicate the range of typical data distribution (approximately 1.5 × the interquartile range from the quartiles). They do not represent average accuracy.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "This paper may have broader impacts on other disciplinaries as tabular data is used everywhere.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths:\n\n1. This paper proposed an efficient tabular foundation model, which shows good balance between performance and speed.\n\n2. The experimental results well support the claims, such as the visualization of feature distribution in Fig. 3 shows that the embeddings are clustered well in a good distribution.\n\n3. The writing is clear and easy to follow.\n\nWeaknesses:\n\n1. In line 240, why should the training be invariant to permutation of columns? If the table is simple one which has n samples on rows and each row has m features, there is no problem. However, if the table becomes more complex like there is a \"total (or sum)\" column, it will be sensitive to the order of columns, right? A more challenging question is ``can the proposed TabICL handle complex tables and how to handle?\"\n\n2. It gives me a contradictory feeling between the feature permutation invariance and mode collapse in the step of row-wise interaction. To avoid mode collapse, the method introduces RoPE. However, it breaks the permutation invariance. Afterwards, to restore invariance, the method ensembles predictions over multiple column permutations. The question is if the features suffer from mode collapse issue again after restoring permutations invariance? \n\n3. Since the performance of the proposed method is so close to TabPFN2, the difference and contributions of the proposed method should be discussed.\n\n4. The comparison of inference speeds with other methods are not listed in paper. The computation cost is essential when deploying a foundation model.\n\n5. The reviewer wonder the concrete examples of the in-context learning in tabular classification. Please list some examples.\n\n    Moreover, can the model handle some real-world tables instead of synthetic ones?", "other_comments_or_suggestions": "1. Suggest to indicate that the red bar is median relative accuracy in Fig. 5. Moreover, what is the meaning of the black dots? The horizontal bar means the average accuracy, right?", "questions_for_authors": "please see Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "86mi6xO5nn", "reviewer": "Reviewer_gpuG", "summary": "The paper proposes TabICL a tabular foundation model able to fit larger datasets than TabPFN (v1) and TabPFN (v2).\nThe main difference lies in the embedding strategy which leverages Set-Transformer as well as later in the model a positional rotary embedding layer. Similar to TabPFN v2 using FlashAttention, makes the model more efficient than v1.\nThese adaptations as well as fewer ow-wise column-wise attentionlayers and smaller dimensions make the model also more efficient than TabPFN v2 and allow it to be fit on larger datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful review, and for mentioning that “the results are really great”.\n\nNext, we respond to the reviewer's questions.\n\n---\n1. *A major drawback is ... the code or the model was released upon submission*\n\nWe **confirm that all of TabICL will be made fully public, including model weights, training details and prior data generation code**. We have already released the inference code after consolidating it to a python package with a fully scikit-learn compliant interface. Here is an anonymous version: https://anonymous.4open.science/r/tabicl-A70B or download code from https://limewire.com/d/opefZ#jPuHwDXBUg. The pre-training code and data generation will be released upon acceptance.\n\nWe agree with the reviewer and believe open-sourcing our work will significantly boost its impact—especially since TabPFNv2 remains partially closed, which limits the community’s ability to reproduce or build upon it.\n\n---\n\n2. *I am not quite convinced by the positional invariance claims...*\n\nNeither TabPFNv1, TabPFNv2, nor TabICL achieves permutation invariance of columns. All these models need to break this invariance in some way to solve the learning collapse issue when some features have similar statistics:\n- TabPFNv1 relies on fixed, feature-specific embedding layers to implicitly encode column identities. \n- TabPFNv2 introduces random feature embeddings and feature grouping.\n- Our TabICL adopts RoPE.\n\nWe do not claim permutation invariance; Instead, we just address the issue differently from the other two models.\n\n---\n\n3. *There are a few newer DL models [1, 2] missing ...*\n\nThanks for these relevant works. Due to limited computational resources, we prioritized methods based on their reported performance and availability.\n\n- TabM has been accepted at ICLR and showed promising results across 46 datasets. Its code is open-source, so we evaluated TabM on TALENT.\n\n- Mambular shows equal performance compared to CatBoost on 12 datasets, according to both its ICLR rebuttal and the latest arXiv version, so we have deprioritized it for now.\n\n- Attic does not provide publicly available code, is not on arXiv and was only visible on OpenReview (submitted and now rejected from ICLR).\n\nTabM performance rank is as follows:\n| Model | Avg. Rank |\n|---|---|\n| TabICL|6.48|\n| TabPFNv2| 6.84|\n| TabR|8.87|\n| RealMLP | 9.24|\n| ModernNCA | 9.30|\n| TabM | 9.79|\n| CatBoost| 9.93|\n| LightGBM| 10.57|\n| XGBoost|11.51|\n\n---\n4. *the benefit of the set transformer given RoPE ?*\n\nWe’re unsure we fully understand the question. The set transformer is used for column-wise embedding and is *row-order* invariant, while RoPE later breaks *column-order* invariance to mitigate learning collapse. They are not directly related.\n\nWe chose the set transformer for its linear complexity with respect to sequence length (see response to Reviewer rh11, Q3), which is more efficient than standard self-attention—especially on large tabular datasets.\n\n---\n\n5. *Given the recent TabPFNv2, the contributions do seem [...] combining ...*\n\nWe would like to emphasize that **TabICL was designed, developed, and pretrained independently of TabPFNv2**. TabPFNv2 was published very shortly before ICML submission deadline, leaving little time to include it as a baseline. However, we still rigorously compared against it in our paper.\n\nTabICL and TabPFNv2 differ in many ways:\n| Aspect | TabPFNv2 | TabICL |\n| --- | --- | --- |\n| Attention | Alternating column / row attentions | Column→Row + ICL |\n| Label Fusion | Early (input layer) | Late (during ICL) |\n| Collapse Issue | Feature grouping | RoPE |\n| Scalability  | ≤10K samples, ≤10 classes | 100K samples, any class |\n\nAdditional innovations in TabICL:\n- Hierarchical classification for >10 classes\n- Curriculum pretraining (1K→60K samples)\n- Memory optimizations (5GB GPU for 100K samples)\n\nThe reviewer questions whether these contributions are \"minor\", but the empirical results demonstrate otherwise: TabICL is faster, more scalable, and also delivers better performance.\n\n---\n\n6. *New references that should be included*  \nThanks for these recent references. We will include them.\n\n---\n\n7. *Why exactly do you use the 4 cls tokens ?*\n\nThe choice of 4 [CLS] tokens was heuristic—we considered it a reasonable trade-off but did not conduct ablation studies with fewer or more [CLS] tokens.\n\n---\n\n8. *How RoPE can mitigate the embedding collapse problem ?*\n\nRoPE mitigates embedding collapse by breaking feature permutation symmetry. After column-wise embedding, features with similar distributions may have indistinguishable embeddings. RoPE injects positional information via rotational encoding for row-wise interaction, ensuring distinct representations even for permuted columns. This is validated in Figure 4 (Section 3.3).\n\n---\n\n9. *Other invariance strategies ?*\n\nYes, we tried CNN to process feature embeddings after column-wise embedding, but RoPE is better.\n\n---  \n\n\n**In the revised version, we will include TabM.**", "claims_and_evidence": "- I am not quite convinced by the positional invariance claims. First, the row-wise attention achieves positional invariance only to later again lose it by introducing the positional rotary embeddings. As far as I understand the architecture, it is not truly positionally invariant and does not distinguish itself in that regard from TabPFN v1. Could you please clarify?\n  - Given that TabPFNv2 does achieve the invariance via row-col attention\n\n- A major drawback is, that the claims and evidence are very hard to test, since neither the datasets nor the code or the model was released upon submission.", "methods_and_evaluation_criteria": "- Yes the method and evaluation criteria seem correct. However, as indicated above they are not reproducible.\n- There are a few newer DL models [1, 2] missing from the benchmarks as well as ICL models [3]\n\n\n---\n[1] Gorishniy, Yury, Akim Kotelnikov, and Artem Babenko. \"Tabm: Advancing tabular deep learning with parameter-efficient ensembling.\" arXiv preprint arXiv:2410.24210 (2024).    \n[2] Thielmann, Anton Frederik, et al. \"Mambular: A sequential model for tabular deep learning.\" arXiv preprint arXiv:2408.06291 (2024).        \n[3] den Breejen, Felix, and Se-Young Yun. \"Attic: A New Architecture for Tabular In-Context Learning Transformers.\"", "theoretical_claims": "- There are no theoretical proofs.\n- However, the paper could benefit from validating some of the effects of the introduced methods. For instance one of the core claims is that via the SetTransformer, positional invariance is achieved, however, what exactly is the benefit of the SetTransformer, given the Rotary Positional embedding layer?", "experimental_designs_or_analyses": "- As described above, I am familiar with the benchmark, and all the results align with the original. However, since nothing was released yet from the authors it is hard to verify the soundness of the presented results.", "relation_to_broader_scientific_literature": "- The paper fits well within the emerging literature of tabular ICL models. Given the recent publication of TabPFN v2, the contributions do seem a little minor and more in the direction of combining concepts together, to squeeze performance (SetTransformer [1] + Rotary [2] Embedding + Tree based priors [3]).\n\n---\n[1] Lee, Juho, et al. \"Set transformer: A framework for attention-based permutation-invariant neural networks.\" International conference on machine learning. PMLR, 2019.          \n[2] Su, Jianlin, et al. \"Roformer: Enhanced transformer with rotary position embedding.\" Neurocomputing 568 (2024): 127063.            \n[3] Breejen, Felix den, et al. \"Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifiers.\" arXiv preprint arXiv:2405.13396 (2024).", "essential_references_not_discussed": "- All essential references are discussed. However, there are a few minor and fairly new references that should be included\n\n---\n**Models**     \n[1] Gorishniy, Yury, Akim Kotelnikov, and Artem Babenko. \"Tabm: Advancing tabular deep learning with parameter-efficient ensembling.\" arXiv preprint arXiv:2410.24210 (2024).    \n[2] Thielmann, Anton Frederik, et al. \"Mambular: A sequential model for tabular deep learning.\" arXiv preprint arXiv:2408.06291 (2024).        \n[3] den Breejen, Felix, and Se-Young Yun. \"Attic: A New Architecture for Tabular In-Context Learning Transformers.\"\n\n**Theory**     \n[4] Reuter, Arik, et al. \"Can Transformers Learn Full Bayesian Inference in Context?.\" arXiv preprint arXiv:2501.16825 (2025).", "other_strengths_and_weaknesses": "- The results are really great and the performed benchmark is the best available benchmark that one could use. \nThe majore problem with this is, that nothing was released. That TabPFN v2 did not release the training datasets is already a shame and I hope the field does not move to a direction where models are being kept private. I do believe that there is a place for this contribution in ICML (given the authors incorporate/answer the reviews) but **only** if the datasets, model weights and training info are published. Since the authors did not submit an anonymous version of this, my current evaluation is rather strict.", "other_comments_or_suggestions": "- line 257 \"probably\" -> \"probable / likely\"\n- line 295 \"gies\" -> \"gives\"", "questions_for_authors": "- Why exactly do you use the 4 cls tokens? Do you also have results for fewer/more cls tokens?\n- How do you explain that the rotary positional embeddings can mitigate the embedding collapse problem? Since you want to achieve positional invariance, how does including the feature position help in distributional disentanglement?\n- Have you also tried out other invariance strategies, i.e. convolutions [1].\n\n\n--- \n[1] Thielmann, Anton Frederik, et al. \"Mambular: A sequential model for tabular deep learning.\" arXiv preprint arXiv:2408.06291 (2024).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e5UEdIzu0s", "reviewer": "Reviewer_VxC8", "summary": "This paper propose a Tabular Foundation Model (TabICL) for classification tasks with new Distribution-aware Column-wise Embedding then Context-aware Row-wise Interaction design. It is a scalable and efficient model,  it can handle up to 500K samples and 500 features with around 20GB of GPU memory. This model demonstrate the SoTA level performance on multiple benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful review and recognition of our work.\n\n---\n1. *More comparison results against the more recent models based on TabPFN ?*\n\nThanks for these relevant works. Due to limited computational resources, we prioritized methods based on code availability.\n- TuneTables: Official code is available.\n- A Closer Look at TabPFNv2: No official code was available at the time of evaluation. Moreover, the paper was released online after the ICML submission deadline.\n- MixturePFN (Mixture of In-Context Prompters): No code was publicly available.\n- LocalPFN (Retrieval & fine-tuning for in-context tabular models): Code is found.\n\nBased on these considerations, we decided to evaluate TuneTables and LocalPFN across 186 datasets with $\\le 10$ classes on the TALENT benchmark. Their average ranks (based on accuracy), along with other methods, are summarized below:\n\n| Model       | Avg. Rank |\n|-------------|------------|\n| TabICL      | 6.60       |\n| TabPFNv2    | 6.95       |\n| TabR        | 9.04       |\n| RealMLP     | 9.45       |\n| ModernNCA   | 9.49       |\n| CatBoost    | 9.94       |\n| LocalPFN    | 10.26      |\n| LightGBM    | 10.80      |\n| XGBoost     | 11.60      |\n| FT-T        | 12.74      |\n| MLP-PLR     | 13.35      |\n| DCNv2       | 14.02      |\n| ResNet      | 14.09      |\n| MLP         | 14.58      |\n| AutoInt     | 15.05      |\n| SNN        | 15.26      |\n| RForest    | 15.38      |\n| TuneTables | 15.47      |\n| TabPFN     | 15.74      |\n\n---\n\n2. *A brief comparison between TabPFNv2 and your model on the method aspect ?*\n\nIn the updated version, we will include a methodological comparison between TabPFNv2 and TabICL in the related work section. Below is a brief summary:\n| Aspect | TabPFNv2 | TabICL |\n| --- | --- | --- |\n| Attention | Alternating column / row attentions | Column→Row + In-context Learning |\n| Label Fusion | Early (input layer) | Late (during ICL) |\n| Collapse Issue | Random feature embeddings and feature grouping | Rotary positional embedding |\n| Scalability  | ≤10K samples, ≤10 classes | 100K samples, any class |\n\n---\n\n3. *The benefit of using Set Transformer compared with the weight sharing ? ...*\n\nWe admit that we may not fully understand the question. We guess that this question may refer to Lines 194–196 in our paper: *\"Instead of sharing the same weight and bias for all samples in a column as in a classic linear layer, each value is projected using its own weight and bias.\"*\n\nDuring column-wise embedding (feature embedding), we map each individual cell in a column $c_j$ into an embedding vector. Specifically, we compute:\n\n$W, B = \\text{SetTransformer}(c_j)$  \n$e_j = W \\odot c_j + B$\n\nHere, we consider two choices:\n- Shared weights and biases: All cells in column $c_j$ share the same weight and bias\n- Cell-specific weights and biases (our choice): Each cell $c_j^i$  is associated with its own weight $W_i$ and bias $B_i$.\n\nWe opt for the second design because, in tabular data, cells within the same column often serve distinct statistical roles—e.g., being the minimum, maximum, mode, or an outlier. Using cell-specific parameters allows the model to adaptively embed each value based on such roles.\n\nImportantly, both options rely on the set transformer to generate the projection parameters, so the architectural backbone remains the same. The key difference lies in how the output of the set transformer is used.\n\n\n*How much computation overhead it can bring ?*\n\nUsing separate weights and biases for each cell introduces only a mild overhead, as the set transformer still processes the entire column as a set, and the per-cell projections are implemented efficiently as element-wise operations.\n\n*How much it makes the performance better ?*\n\nIn our early experiments, we observed that using per-cell weights and biases improved performance over shared ones. However, we acknowledge that we have not conducted a thorough ablation study to quantify this gain precisely, and we will consider including such analysis in future work.\n\n---\n\n4. *Why diversifying activation functions can help the training? How you perform the random rescaling ?*\n\nOur intuition for diversifying activation functions is that it is generally helpful to increase the diversity of generated datasets. In particular, some of the activation functions can generate functions of different smoothness (e.g., the GP functions), periodic functions (sine), (semi-)discrete feature distributions (e.g. ReLU/signum), and heavy-tailed distributions (e.g. exp). We will add this to Appendix B.1. The random rescaling is already described in Appendix B.1: $x_i \\leftarrow \\exp(2a)(x_i + b)$ with $a, b \\sim \\mathcal{N}(0, 1)$.\n\n---\n\n**In the revised version, we will include LocalPFN and TuneTables into the benchmark.**", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The prior SoTA method's architecture limits scalability for large datasets, and this paper's method is more scalable and computationally efficient.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- This model has significant better efficiency compared with the prior works, handling up to 500K samples and 500 features while maintaining reasonable computational costs. It demonstrates its potentials on the large datasets.\n\n\n- The model is fully evaluated with its accuracy and efficiency on the latest dataset, showing strong performance compared with other methods.\n\n\n- However, it would be great if we can see more comparison results against the more recent models based on TabPFN. List some related papers below,\n\n\n[1] Feuer, Benjamin, Robin Tibor Schirrmeister, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, and Colin White. \"TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks.\" CoRR (2024).\n\n[2] Ye, Han-Jia, Si-Yang Liu, and Wei-Lun Chao. \"A Closer Look at TabPFN v2: Strength, Limitation, and Extension.\" arXiv preprint arXiv:2502.17361 (2025).\n\n[3] Xu, Derek Qiang, F. Olcay Cirit, Reza Asadi, Yizhou Sun, and Wei Wang. \"Mixture of In-Context Prompters for Tabular PFNs.\" In The Thirteenth International Conference on Learning Representations.\n\n[4] Thomas, Valentin, Junwei Ma, Rasa Hosseinzadeh, Keyvan Golestan, Guangwei Yu, Maks Volkovs, and Anthony L. Caterini. \"Retrieval & fine-tuning for in-context tabular models.\" Advances in Neural Information Processing Systems 37 (2025): 108439-108467.", "other_comments_or_suggestions": "Just a few more questions regarding to some details, see questions section", "questions_for_authors": "- Can you provide a brief comparison between TabPFNv2 and your model on the method aspect? It would be great if it can be provided in the related work section.\n\n  \n- Can you simply explain what's the benefit of using Set Transformer compared with the weight sharing? How much computation overhead it can bring and how much it makes the performance better?\n\n\n- Can you explain why diversifying activation functions (15 additional activation functions) can help the training? Besides, how you perform the random rescaling?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is clearly a paper whose polarity of views has changed through the review process. The reviewers have particularly appreciated the rebuttals, and have raised their score (rh11, ba5G).

I have read the paper in addition to the rebuttals and exchanges with reviewers. The authors have done a very good job in their rebuttals: lots of insights, new experiments, clarity of explanations. The paper is an experimental paper, which is up to a large extent, not usual in the ML literature about prediction for tabular data where much of the theory developed in ML applies without hurdles, especially for models like decision trees.

Yet, the paper manages to develop in a clear way its experiments and it makes little doubt that, *given the inclusion of the experiments promised to the reviewers* (ablation studies, and random forest extension for rh11, tabM for gpuG, LocalPFN and TuneTables for VxC8) but also the additional material produced at rebuttal time (complexity analysis rh11, additional explanations ba5G), the paper may end up being a sound benchmark for DL on tabular data.

It should be accepted.
Decision: Accept (poster)