ConTextTab: A Semantics-Aware Tabular In-Context Learner
Authors: Marco Spinaci
Marek Polewczyk
Maximilian Schambach
Sam Thelin
Venue: NeurIPS 2025 spotlight ()
Keywords: tabular learning
semantics
in-context learning
deep learning
Primary area: deep_learning
TL;DR: A state-of-the-art semantics-enriched TabPFNv2-style in-context learner trained on real-world tabular data

Abstract: Tabular in-context learning (ICL) has recently achieved state-of-the-art (SOTA) performance on several tabular prediction tasks. Previously restricted to classification problems on small tables, recent advances such as TabPFN and TabICL have extended its use to larger datasets. Although current table-native ICL architectures are architecturally efficient and well-adapted to tabular data structures, their exclusive training on synthetic data limits their ability to fully leverage the rich semantics and world knowledge contained in real-world tabular data. At the other end of the spectrum, tabular ICL models based on pretrained large language models such as TabuLa-8B integrate deep semantic understanding and world knowledge but are only able to make use of a small amount of context due to inherent architectural limitations. With the aim to combine the best of both these worlds, we introduce ConTextTab, integrating semantic understanding and alignment into a table-native ICL framework. By employing specialized embeddings for different data modalities and by training on large-scale real-world tabular data, our model is competitive with SOTA across a broad set of benchmarks while setting a new standard on the semantically rich CARTE benchmark. Code and model checkpoints are available at: https://github.com/SAP-samples/contexttab .

ConTextTab: A Semantics-Aware Tabular In-Context Learner
Marco Spinaci'!
Marek Polewezyk
Maximilian Schambach
Sam Thelin?
"SAP France SAP SE (firstname. dastname/@sapp com
Abstract
Tabular in-context learning (ICL) has recently achieved state-of-the-ar (SOTA) per- formance several tabular prediction tasks. Previously restricted to classification problems on small tables, recent advances such as TabPFN [8] and TabICL [30] have extended its use to larger datasets. Although current table-native ICI. archi- tectures are architecturally efficient and well-adapted to tabular data structures, their exclusive training synthetic data limits their ability to fully leverage the rich semantics and world knowledge contained in real-world tabular data. the other end of the spectrum, tabular ICL models based on pretrained large language models such as TabuLa-8B [12) integrate deep semantic understanding and world knowledge but are only able to make use of a small amount of context due inherent architectural limitations. With the aim to combine the best of both these worlds, we introduce ConTextTab, integrating semantic understanding and align- ment into table-native ICL framework. By employing specialized embeddings for different data modalities and by training large-scale real-world tabular data, model is competitive with SOTA across broad set of benchmarks while setting standard the semantically rich CARTE benchmark. Code and model checkpoints are available at: httpp/tui..oooSS/ASpapppe//eeeee
Introduction
Tables with information spread across rows and columns remain a predominant data format in many real-world applications [3] making their understanding through machine learning algorithms critical. Despite the great success of deep learning approaches in natural language processing and computer vision, leveraging large amounts of pretraining data, conventional machine learning methods such as gradient boosted trees and their variants remain the predominant state-of-the-arr (SOTA) across tabular prediction benchmarks. Recently, however, applying the in-context learning (ICL) paradigm to tabular prediction tasks has shown promising results by enabling pretraining of deep learning models across a large set of heterogeneous tables, constituting new SOTA on small medium tabular prediction tasks [8]. In this setting, predicting target value based on the input features æ of row in table T' additionally uses further rows from T' (the context), including their target values. This enables the model to adapt to new, unseen prediction problems at inference time, removing the need for task-specific fine-tuning.
This approach was pioneered by the transformer-basee TabPFN [7]. Pretrained on large amounts of synthetically generated classification tasks, its latest incarnation TabPFNv2 [18) produces SOTA results on tabular datasets with up to 10 000 samples for both classification and regression tasks. In recent work, TabICL [30] extends the success story of this approach to even larger datasets, using special tabular embedding modules improving on the quadratic scaling in both the number of features and rows present in the TabPFN architecture.
Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Common to both TabPFN and TabICL is that they are trained entirely on synthetically generated numerical data, with categorical features produced by an indexing procedure. While using synthetic data has many advantages, in particular its diversity at scale, consequence is that the data does not contain any semantically meaningful values as found in real-world applications, both in the form of column names, and categorical or free-text entries. Furthermore, such data contains no additional data types such as times or dates that are abundant in practice. Consequently, these models do not utilize such information in semantically meaningful way even when it is present at inference. In particular, column names are not used either TabPFN or TabICL, and categorical features are encoded via one-hot or ordinal encoding disregarding any underlying semantics. We argue that semantic understanding can be captured by training on a large number of real-world tabular datasets. A primary example of this philosophy is TabuLa-8B [22. turning the pretrained large-language model (LI.M) Llama 3-8B (7] into a tabular ICL. model by fine-tuning it on around 3 million tables of the T4 dataset proposed therein. However, utilizing pretrained L.MSs for tabular tasks has several limitations: Most importantly, textual serialization and tokenization of the input table is not token efficient, effectively limiting the maximum context length that can be processed. For example, TabuLa-88 operates on a maximum of 32 context rows. Furthermore, the tokenization schema and autoregressive nature of LLMs are not adapted the tabular structure, resulting in linear non-uniform token sequence, as cell values can be tokenized into different amounts of tokens, losing the 2D structure of the underlying data. Finally, the serialization and autoregressive processing result in an architecture that is neither row nor column permutation invariant property often desirable for tabular data (35).
Aiming to bridge these approaches, we propose table-native ICL model trained on the real- rooldworld T4 dataset [12], using embeddings tailored to different data modalities, in particular incorporating semantic embeddings of column names and categorical values. The resulting model is competitive to existing table-native ICL approaches across a range of tabular prediction benchmarks (OpenML.- CC18 M1 Ope2ML-CTR23 (1]. TALENT [40), and TabReD (32)), and achieves a new SOTA for the semantically rich CARTE benchmark 223) in particular in the low-data regime.
2 Related Work
Tabular deep learning:
Prediction on tabular data has traditionally been dominated by decision tree algorithms, particularly boosted variants like XGBoost (2), LightGBM [22), and CatBoost [2]. These models deliver strong performance but require separate training for each dataset and cannot leverage pretraining. Hence, much work has been done to transfer the success of deep learning methods and general pretraining to the tabular setting. Early examples include the FT-Transformer [15] and Xtab 443, whereas more recent approaches have shown consistently good performance overtaking boosted trees, for example TabR [14] ReaLMLP [9), CARTE (23), TabM [3], or ModernNCA [41).
In-context learning on tabular data: TabPFN 17 broke the oangsstaning dominance of boosted trees on small classification tasks, outperforming them by using row-level ICL. Pretrained on large amount of synthetic tabular data, it generalizes to new tasks at inference time without fine-tuning or hyperparameter optimization. A recent variant, TabDPT 225, showed that equally excellent results can be achieved by training on real-world data using - an idea previously investigated in the TabR approach 14 and further unlocked regression in this setting. Generalizing the row-based encoding, cell-based ICL introduced with TabPFNv2 [18) and utilized also by TabICL (30 extended this success to larger datasets with up to 10 000 and more samples, even outperforming SOTA AutoML solutions such as AutoGluon [9] on certain benchmarks.
retrieval for the context examples
Semantics and real data: Capturing the rich semantics of real-world tabular datasets is desirable property of tabular foundation model, enabling the transfer of world knowledge across prediction tasks in addition to statistical patterns. The CARTE 233 architecture enables pretraining across a variety of real-world sources while capturing table semantics. It achieves SOTA results on the semantically rich CARTE benchmark, although requires task-specific fine-tuning.
Modern L.MMs have both deep semantic understanding and extensive world knowledge. Several works approach tabular ICL by tuning LLMs on tabular tasks, e.g. TabLLM [16). LIFT [6], or TabuLa-8B [12]. In particular, the works by Gardner et al. [12] are notc-worthy for curating the T4 dataset, containing roughly M tables extracted and processed from the TabLib collection g and for its excellent results in the very low data-regime.
In the following, will focus solely on TabPFNv2 and refer to simply TabPEN.

---

Classification loss
Regression loss
Coan stantion
Cross atantion
Colurn embedciings
Date embedddingg
2D layer (xn)
Numenical embeddiing
FC
FC
Text entiedkiings
Acqulstion date
Frice E8:
Desctiston
Rocehee
Context rows
16/0/2024
179200
Laptop
TRUE
Queryraw
Figure 1: Our proposed model architecture illustrating the integration of data type-specific embed- dings, an interleaved attention backbone, and customized output heads.
Method
To overcome the aforementioned limitations of existing table-native ICL methods and bridge the gap to LLM-based ones, propose ConTexttab, semantics-aware table-native ICL model. To this end, we perform several key modifications to the TabPFN architecture and utilize large-scale pretraining on real-world data. An overview of our proposed architecture given in Figure
3.1 Encoding
encode data differently depending on modality - i.e. text, date, or numeric type. Column headers are also encoded, playing the role of positional encodings used in TabPFN or TabICI.
Text: We transform each cell an embedding vector using pretrained text embedding model. Note that we apply this to both free text columns as well as categorical columns, which can then retain the meaning in their labels. Any off-the-shelf embedding model can used for this purpose e.g. many are available based on the BERT architecture [5. As many values have to be embedded for each table, there is a natural trade-off between accuracy and speed. We settle for comparably small and fast model such 1-1-MiniLM-LL--- [20 371, since the amount of semantic meaning captures is already much richer than conventional categorical encodings, and we defer to Section [. for experimental ablation. Since the embeddings are potentially of different dimension than the target embedding dimension d, we apply learnable linear layer to the text embeddings.
Date: In order to endow our model with knowledge of both the relative meaning of a dates (c.g. being able compare dates) and special dates (c.g. recurring holidays). we embed each of the numbers representing day, month, and year separately and sum the three resulting vectors. In particular, this more token efficient than encoding dates into multiple features as common in established preprocessors such as the one used by AutoGluon.
Numerical: As numbers do not contain semantic meaning beyond their value, apply dimensional encoding. To make this procedure more robust during training, first, we clip columns between the 2% and 98% quantiles of the distribution. Second, scale them have mean and unit variance as is common. By Chebyshev's inequality, this bounds resulting values to the interval (-7.1, 7.1). thus avoiding exploding gradients during training. Finally, the resulting number is multiplied by a learnable vector and bias is added. If the original value was NaN, 0 is used instead, the bias works as an "is-NaN" flag. An alternative embedding scheme is described in 33.4
Column headers:
We embed column headers with the same model used for text cells. The result is passed through separate learnable linear layer map to the correct target dimension and summed with the cell embedding.
After summation, the embeddings are normalized via layer normalization. Note that all the above embeddings are fully equivariant under permutations of either rows or columns. This equivariance makes the predictions more reproducible and robust and climinates some of the need (such as the

---

mapping of category to ID, or of column to ID) for bagging in models such as TabPFN. Semantic embeddings of cell values have been previously investigated in other works [38 39 23 however, details of how the information is consumed differ in each implementation.
3.2 Backbone
We leave the TabPEN architecture mostly unchanged, with alternating "horizontal" (cross-column) and "vertical" (cross-row) self-attention transformer layers. In cross-column attention, each row is considered as a different batch element, and conversely. Following TabPEN, cross-column attention has no masking, while cross-row attention is masked that each row can only attend the provided context. To increase the modularity of our code, the feedforward MLP block of the transformer encoder is repeated after each self-attention block so that "horizontal" and "vertical" blocks have the same structure. For sizing, we use the nomenclature popularized by BERT, e.g. calling "base" the variant with 12 layers and hidden dimension of 768. However, the real number of (non-embedding) weights is twice that of a BERT model, due to the presence of both "horizontal" and "vertical" layers.
Model weights can be optionally shared between each instance of the transformer block, consisting of two interleaved attention layers. Such an architecture can be interpreted as a recurrent neural network, unrolled in depth rather than in time. Such an iterative structure results in parameter efficiency and the possibility of stacking more blocks in comparison to the traditional approach. Empirically, we observed that sharing weights did not affect model performance and thus we use weight sharing as the default option for our model in the following.
3.3 Decoding
Classification: We apply a standard cross-entropy loss after an MLP with least as large an output dimension as the number of classes. This, however, imposes two limitations. First, the number of classes seen in pretraining cannot be exceeded inference, without resorting to suboptimal schemas such as hierarchical classification, as used in TabPFN's ' many-class extension, TabDPT, or TabICL. Secondly, it prevents us from using the semantic value the class label. In fact, for the model to know what, say, class "0" means, it needs to build an a priori knowledge of that class ID. Therefore, we must create an additional special input encoding, only for the target column, in addition to the ones described in Section 3.1 Even though this breaks equivariance under permutation, we retain it as we found it to be effective in the most common scenario of few-classes classification.
The model predicts the floating-port value of the target, clipped and normalized as described in Section 3.1 Empirically, we have found this simple schema to work well. During training, an L2 loss is applied, and during inference, only the inverse transformation of normalization applied to the prediction.
Regression:
3.4 Alternative architectures
Number encoding and decoding via binning: Encoding soft binning. We split the numbers into bins defined via quantiles to ensure uniform distribution. In order not to lose any fine-grained information while keeping the number of bins limited,
soft binning: each of the, say n, bins is associated with a quantile {q.) h of order 2i-1 Then, any point I E [q,, 4n is encoded via the 2n linear combination Au; (1 Aje where x Ag; + (1 X) 0 S X S 1, and Ui,.... Un are learnable vectors. Anything outside that interval is mapped to either en or ea-
Decoding bin averaging. The regression is converted to classification task for the bin it belongs to. During training, cross cross-entropy loss is applied with hard labels. At prediction time, this provides a probability distribution across all bins, (pI. ........ :P)); the prediction is then given by P191 + +... Pn
We introduce an alternative method to perform classification that has the advantage of retaining semantic meaning not only for features but also for labels. As further benefits, full equivariance is preserved, without the need to map classes IDs, and the limitation on the number of classes supported for prediction is lifted, improving upon the constraint currently present in many tabular ICL approaches such as TabPFN, TabDPT, or TabICL. For this, we take inspiration from Polewezyk and Spinaci (28): For each row, the final output corresponding to the target column is mapped by a two-layer MI.P to a vector x. We then build a matrix of shape (Nquerry )contex:) ) by computing the cosine similarities between these vectors 2,, Zz for each query
Supervised clustering head:

---

row i and context row j. During training, this is compared against the adjacency matrix with value 1 if two rows belong to the same class, and 0 otherwise: We compute the element-wise binary cross-entropy loss between this adjacency matrix's 's entries and the clipped cosine similarities. That is, by is the vector from above for row and ej is its class, then the loss given by
Loss =
Σ iEquery fEcoutext
log(s1))0,,c,, + -log(1 S1g) 81s)0o,toss Sij = clip
F2. 2 |
E, 1 - (1)
including a small margin > for numerical stability, In this way, rows in the same class are pushed to similar embeddings, while rows in different classes can be either opposite or orthogonal. This enables the use of the same semantic- preeeeiing text encoding for the target class of context rows as used for column names and input features, while also supporting an arbitrary number of target classes. A possible use case for this scenario is a situation with many target classes with very few rows per class, allowing extraction of information from semantically similar target classes.
Induced Set Attention Blocks: One of the main limitations of the standard multi- attention is its quadratic complexity. While memory can be effectively compressed to linear [31] 4) that is not the case for runtime. This poses significant challenge when dealing with large tables. Taking inspiration from 124 21 30), we experiment with methods to handle larger context more efficiently.
Our proposed architecture replaces some of the multihead attention blocks within TabPFN with ISAB blocks from [24]. Adopting the notation from [24), let MAB(X, Y) denote the standard multihead attention block from the transformer architecture, where X represents the queries and Y serves as both the keys and values. This is followed by a feedforward block, a skip connection, layer normalization, and ultimately an intermediate block. Let 7 be so-called inducing points [24], an additional set of parameters of shape (n, d) for an arbitrary n, with d denoting the hidden dimension, and define
ISAB(C + Q) = MAB(C + MAB(I # Q, + Q)),
where C' and Q denote context and query rows, respectively, and # denotes concatenation. In each attention computation, we apply masking so that only context and inducing vectors are attended to. For detailed overview of the architecture, refer to Appendix Figure
Since tables can have a very large number of rows but rarely exceed a few hundred columns, we apply the ISAB block only to cross-row attention. In our architecture, we only use the ISAB block for the first m blocks (e.g., m 3). followed by one MAB(L + Q, C # Q) block, and finally use standard attention for the following layers. This both improves training stability and reduces inference time because the corresponding attention has a bounded number of tokens: MAB(T # Q
I + Q).
Experimental Setup
4.1 Training and inference
For pretraining, we use the T4 dataset [12]. We discard tables with fewer than 150 rows, which leaves 2.18 M tables with median of 750 rows and 9 columns. We randomly select 1000 rows, then between 50 and 900 rows as query, and use the rest as context. Subsequently, we randomly select one target column, excluding all date columns, numerical columns with more than 50% NaN values, and other columns having more than 20% of unique values. Finally, we up-sample non-numeric columns to have roughly the same proportion of regression and classification tasks. We train each model for between 4 and 10 million steps (i.e., 2 to 5 epochs) until convergence. We use a micro batch size of 1 and accumulate gradients to simulate a batch size of 256 (or 128 for smaller models of "mini" size). To improve stability, we employ gradient clipping and the AdamW optimizer with a maximum learning rate of 10 4, reached after a linear warm-up phase of 1000 gradient updates, Under this setup, we train "base" sized model on a single H100 GPU, reaching throughput of roughly 10 tables/s. Hence, full training takes between and 12 days depending on the number of steps. Using the default parameters n 12, d 768, du 3072 results in total of 172 M parameters which is reduced to 16 M trainable parameters when using weight sharing.
We also experimented with curriculum learning, by further adding in second step using the same training data as Ma et al. (25). This data has fewer tables, 123, but many more cells in each table with median of 1 ll k rows and 34 columns. In this second step, we increased the number of rows
5

---

used for training to 4000. We refer to Section 5.1 for an analysis of the impact of training data size on model performance, which we found to be crucial.
At inference, we apply 8-fold bagging, similar to the ensembling used by TabPFN. That is, from the original train split of given evaluation dataset, we sample 8 times context rows with replacement, make a separate prediction with each collection using the same model, and average the predictions (regression values or classification probabilities.. While, during the default training, never surpasses 950, we can scale this during evaluation as much as memory permits. The combination with bagging allows ConTextTab to use up to &c points as context. By default, we use a context size of c 8192 at inference. To limit runtime. in each bag, we sample up to 500 columns if the dataset has more.
4.2 Evaluation
Datasets: We use a variety of tabular prediction datasets to evaluate and compare our approach to established baselines and other SOTA methods. Namely, we evaluate all models on the following benchmarks: OpenML-CCC18 M. pure classification benchmark; OpenML-CTR23 [Л]. a pure
regression benchmark; TALENT [40], a recently introduced diverse benchmark containing over 300 classification and regression benchmarks. Here, we focus on a subset containing 45 datasets that are representative of the overall performance of the baselines investigated in the original works, which we refer to as the TALENT-Tine benchmark; TabReD [32). a small but challenging benchmark of large datasets representative of practical prediction tasks; and finally CARTE [23] mixed classification and regression benchmark containing highly semantic features and few numerical ones.
Across all benchmarks, we evaluate 91 regression and 1 112 classification tasks, ranging from 5 to 3 columns and 400 to roughly 400 k training examples. Due to the large number of evaluated datasets, we do not perform cross-validation but evaluate a fixed test split for each task. For models that do not explicitly use a validation split, we concatenate the train and validation splits. We refer to Appendix for visualization of the dataset statistics and further details.
Baselines: We compare our approach to several established classical methods as well as recent ICL and other deep learning models. In particular, we compare with TabPFN 18]. TabICL [30], and TabDPT [25] as the most recent table-native ICL methods. For all pretrained models, we use the latest available release and checkpoints as of July 2025. As the SOTA on the CARTE benchmark, we also evaluate the CARTE [23] As additional well perl-perfonming recent deep learning approaches, we evaluate ReaLMLP [9 and TabM [13]. Furthermore, we evaluate SOTA boosted tree baselines, namely XGBoost 2. LightBGM [22], and CatBoost (29.. The tree baselines as well as RealMLP and TabM are evaluated in the default (D) or meta-tuned default variants (TD) [9, as well as via Parzen-tree estimator- based hyperparameter optimization, either with a fixed holdout set (HPO) or the ensembled via 5-fold inner cross- validation with per-split HPO (HPO, CV). For this, non-tuned implementation provided by pytabkit [1). Additionally, we evaluate several common, baselines from scikit- -larn 27 namely naive, linear, and KNN estimators, well as random forest and histogram- based gradient boosting tree estimators. Finally, as the gold standard in tabular prediction, evaluate AutoGluon 9 an AutoML solution stacking and ensembling many of the baselines outlined above. We refer to Appendix CCI for full details on the used baseline versions and parameters well details about data preprocessing and encoding.
Throughout, we show (mean) accuracy for classification tasks as well as (mean) R² score for regression tasks. Since we have observed some severe outliers in some regression tasks that greatly affected average scores, we perform a soft-clipping. That is, all negative R² scores are mapped to [-1,0) via tanh while keeping positive scores untouched. This retains the relative ordering of scores while being smoothly differentiable and less distorting to the mean.
Metries:
As averaging across large number of datasets with varying performance can blur relative performance across models, we also evaluate (mean) rank. Within each benchmark, we calculate the mean rank across all constituent datasets and models. Full rank performance is averaged across all datasets across all benchmarks, not as per-benchmark averages as some benchmarks contain only few tables. To reduce the impact of small noise, we count model performances as ties when their evaluation scores lie within 0.005 of each other. Furthermore, we report critical difference (CD) diagrams, win ratio matrices, and p- values enabling 1 vs model comparisons in the Appendix.

---

Table 1: Performance comparison across all evaluated benchmarks, depicting mean accuracy (Acc) for classification and (soft- clipted) R² score for regression tasks, in percent. Missing values, due to architectural limitations or failed evaluations, are denoted as N/A and excluded from the rank calculations. Models are sorted according to their ranking on the CARTE benchmark.
Model Name
All
CARTE
OML-CC18
OML-CTR23
TabReD
TALENT- Tiny
Rank
Rank
Acc
R²
Rank
Rank
R²
Acc
Rank
Acc
R²
AutoGiluon [v1.2]
NIA
N/A
78.7
73.8
NIA
88.5
N/A
68.6
N/A
86.0
64.6
NIA
87.9
73.7
ConTextTab
2.96
1.55
76.9
72.4
3,56
86.6
3.89
73.3
4.12
85.4
63.2
2.59
87.6
76.1
LightGBM [HPO,CV CV]
2.85
3.00
73.4
67.5
2.94
87,5
2.91
68.8
1.25
86.0
64.9
2.73
86.9
74.1
CatRoost [HPO, CV]
2.71
75.9
68.2
2.72
87.7
2.74
68.8
2.00
86.0
87.2
ReaMMLP [HPO, CV]
3.20
73.4
67.5
2.35
88.3
1.89
75.0
1.88
86.0
2.54
88.3
76.4
TahM [HPO, CV]
2.98
3.71
73.2
66.9
2.93
87.9
2.37
74.3
3.12
86.1
64.2
2.62
87.5
75.7
XGBoost [HIPO, CV]
3.11
3.78
73.1
66.7
2.83
87.5
3.23
70.9
1.25
86.0
64.9
3,00
86.8
73.6
TahPPNN [v2.1.0]
72.3
65.0
75.1
4.25
85.6
87.8
TabDPTivLL1 k=2048]
4.18
5.98
72.7
65.1
3,25
87.8
3.17
73.2
6.88
83.0
60.9
3.89
86,7
74.8
IistGraBBoost
5.27
6.27
72.5
64.8
4.40
86.1
7.06
65.5
2.38
85.9
63.9
4,51
86.3
67.6
Random forest
6,22
8.00
71.5
63.3
5.33
85.7
6.80
67.9
6,00
85.4
60.7
5,00
85.8
70.6
Naive
10.51
11.0
53.0
-1.8
10.28
47.0
10.66
-8.4
9.75
80.8
-0.6
10.32
53.4
-19.2
CARTE [v0.0.26]
NIA
NIA
76.1
68.5
N/A
NIA
N/A
NIA
NIA
NOA
NIA
NIA
NIA
NIA
TabtCL [v0.1.1]
NIA
NIA
72.5
NIA
NIA
88.0
N/A
NIA
NIA
85.1
NIA
NIA
87.4
NIA
5 Results
The main results, evaluating Ton and baselines, are shown in Table 1 Here, we exclude AutoGluon from the overall rank and best-model comparison it is ensembling and stacking multitude of model architectures, making it difficult to highlight architectural strengths and weak- nesses. Overall, our model performs competitive across all non-semantic benchmarks while setting a new standard on the semantically rich CARTE benchmark, where it ranks best. In fact, this is statistically significant over all models but CatBoost and AutoGluon, both of which are extensively tuned and ensembled, as shown in the critical difference diagram for CARTE in Figure (2) (left). Notably, Ten is significantly better than other tabular ICL approaches. which highlights the importance of incorporating semantic understanding. On the other hand, TabPFN, not incorporating semantics, performs worse than tuned as well as untuned trees on CARTE. For details, we refer to Appendix A
On the other end, the absolute performance of ConTextTab for non-semantic benchmarks, while good, falls behind tuned and ensembled boosted trees as well as RealMLP, which achieves SOTA in terms of ranking across all evaluated datasets (albeit at much higher training time and cost). Nevertheless, these differences in ranking are mostly not statistically significant. To this end we report per-benchmark CD diagrams as well win-ratio and p-values for detailed model comparison in Appendix A.2 Namely, ConTexiTab does not perform significantly worse than the best-ranking model on all non-semantic benchmarks excluding OpenML-CTRR (where it is however not significantly different to tuned ensembled trees).
Hence, ConTextTab sets a new standard semantically rich datasets while performing on-par to existing approaches on non-semantic ones. Nevertheless, the performance of AutoGluon generally indicates further headroom to improve single-model performance and tabular ICL in particular.
Finally, tuned trees perform particularly well for large datasets as represented by the TabReD benchmark, even outperforming AutoGluon in some instances. Motivated by this, we investigate the dependence on dataset size more closely and the performance on the CARTE benchmark across varying subsampled sizes, ranging from 128 rows to the full dataset, in Figure 3 Tab consistently outperforms other models across all sample sizes and even surpasses AutoGluon for up to 2048 training samples. This highlights the strong capabilities of tabular ICL but also the need for further research into scaling these architectures to effectively deal with much larger context sizes as well as training datasets. We investigate the dataset-size dependent rank performance across all 203 datasets in Appendix [A. further highlighting these challenges.
Additional results, including additional baselines, CD diagrams, as well as 1 vs 1 win ratios and p-values, are provided Appendix A.. Furthermore, we refer to Appendix A.3 for an analysis of the model runtime in comparison to other baselines.
7

---

(1)
R-
0.75
0.75
43
0.65
0.723
23
LiMnGAM
CANTE
0.025
Tro MLY (17O, CV
120
1024 2043
4003
9152
120
236
1224
2043
4003
9:52
122
236
1224
2042
4026
5252
5.66
Figure 3: Average rank, accuracy, and regression results on the CARTE benchmark across various data subsets, ranging from 128 rows to the full size.
0.24
Kandon
Narbu of
Arowy
CV.
Pabliiio of
Figure 2: Left: critical difference diagram between TexTex and several baselines, across the CARTE benchmark. Right: Impact of pretraining dataset size validation accuracy and R² scores.
5.1
Ablation studies
In the following, we discuss several experiments ablating aspects of our model as reported in Table 2 Semantics:
Being core contribution, investigate the efficacy of the semantic cell and column header encoding used by ConTexx'Tab. As we expect the results to be most pronounced in the presence of semantic features, we perform this ablation on CARTE exclusively. To this end, we replace categorical and string features by ordinal encodings, as well as encodings via MinHashEncooerr and GapEncoder (provided by the skrub library), and the AutoGluon feature encoder for comparison. As expected, we observe significant drop in performance when discarding semantics completely or when using the conventional string encoders from the skrub library. Hence, ConTextTaa successfully leverages the semantic content of features. We refer to Appendix (A.4 for win ratio comparisons and p-value results to demonstrate statistical significance our claim.
Secondly, we ablate the use of column header semantics in two ways: one, by replacing the column names with col1, ..., colN, effectively eliminating semantic content; as well as two, semantically enriching the column names by creating contextualized column descriptions using an LLM and replacing the column names with <name> <descriptions To this end, we used gemma3- and presented it with 5 randomly sampled rows, serialized as JSON, and prompting for contextualized descriptions of each column. Dropping column name semantics indeed results in a performance loss of about 1% in accuracy and 2% in R score. This difference is significant with a win rate of 92% of the base model and vanishinlgy small p-value as shown Appendix [A.4] Further enriching column header semantics slightly boosts performance, resulting in a slightly better rank, however with a win rate 57% % a p-value 0.4 this is not statistically significant.
Overall, ConTextTab successfully integrated and leverages semantics, both present the table's features as well those potentially contained in the table column names.
Model size: As expected, decreasing from "base" to smaller models impacts performance. On the other hand, increasing to "large". thus doubling the number of layers and increasing the hidden dimension, has no significant impact on performance. As observed Figure 2 (right), increasing the amount of training data leads stagnating validation performance gains. We hypothesize that, in the current setup, our model is likely limited by the amount or diversity of data available with T4.
Context size:
The trained models can scale the input context size at inference. As expected, longer context improves results in a monotonic way, in exchange for longer computation time. practice, the effects are stronger larger datasets, simply because shorter context models (even with bagging)

---

Table 2:
Relative performance across three ablation groups compared to the base model: (Top) Model size, language model, and other factors, using the base model with binning; (Middle) Clipping, context size, and fine-tuning, using the base model with one-dimensional embedding; (Bottom) Semantic analysis, using the base model with one-dimensional embedding. Improvements over the base model are highlighted in bold.
Experiment
AlI Rank
CARTE
OML-CC18
OML-CTR23
TabReD
TALENT- Tiny
Rank
Acc
R
Rank
Ace
Rank
R²
Rank
Ace
R²
Rank
Acc
Model Size. Language Model, and others
hase model (hinning)
2.28
1.76
76.0
71.4
2.57
86.8
7.66
70.9
1.00
85.4
62.9
2.32
87.2
73.5
large
-25.4
-13.5
medium size
5:44
8:45
-5.6
-3.7
4.76
-1.2
3.49
-1.0
5.75
-9.3
-3.5
4.41
-2.6
-2.4
small size
-1.7
mini size
8.10
10.69
-4.2
-8.4
7.15
-4.0
8.06
-4.2
5.50
-0.3
-3.5
7.00
-3.7
-6.9
multiingual-s --mmalllll
3.22
4,16
-0.3
-1.5
3.35
-0.4
2.77
-0.1
2.38
-0.2
-1.3
2.27
+0.7
-0.7
gte avchilingaal hase
carricalum learning
2.28
1.51
+1.1
+0.1
3.10
-0.2
1,80
-0.1
1.38
+0.4
-0.2
2.38
.1.0
+0.4
clostering
N/A
N/A
N/A
-0.6
w/o bagging
2,82
2.76
-0.4
-0.4
3.26
-0.7
3.03
-0.4
1.00
40.1
-0.5
2.24
-0.6
-0.0
ISAB
5.48
5.16
+0.3
13.1
5.57
5.51
3.88
-0.2
4.70
-0.5
non- shared weights
4.36
5.61
-0.1
-2.6
4,29
-0.9
3.09
+0.8
5.88
-1.0
-2.6
3.65
-0.7
-1.2
Clipping, Context Size, and Fize unnin
base model (1-dim)
2,00
1.80
76.7
72.0
2.36
86.9
2,06
65.00
1.75
85.3
63.33
1.54
87.9
75.8
0.1% clipping
2.03
0.00
0.00
0.0
0.0
0.5% clipping
2,08
1.76
0.0
0.0
2.32
-0.1
2.60
+4.0
1.38
40.1
+0.1
1,70
+0.1
-0.2
2% clipping
0.0
0.0
hinning
4.04
4.69
-0.7
-0.6
3.21
-0.1
6.34
+5.9
1.12
40.1
-0.3
3.24
-0.7
-2.3
com0ext=20000
2,06
1.39
+0.3
+0.6
2.62
-0.2
1.66
+4.2
3.25
-0.4
-0.5
2.00
-0.1
+0.5
10000
2.00
1.61
+0.3
+0.3
2.64
41.2
1.57
+4.2
1.75
1.76
-0.2
+0.5
co96ext=/096
2,81
3.55
-0.1
-0.6
3.03
-0.3
2,11
+4,0
2.25
-0.3
-0.1
2,16
-0.1
+0.4
4.11
-0.1
context- 1024
5.49
9.10
-1.3
-3.8
4.38
-0.5
4.60
+2.7
2.62
-0.4
-1.6
4.74
-1.3
-0.6
fine-tuning benchmark)
2.60
2.51
+0.3
+0.1
3.1
+0.2
1.94
+7.2
2.62
+0.2
-0.3
2.38
-0.1
+0.8
Semonties- Features Column Names
base (feature and column name semantics)
1.44
1.24
76.9
72.4
N/A
N/A
NDA
N/A
N/A
NOA
NIA
NIA
NOA
NIA
no feature semanrics Ordinal encoder
4.27
4.27
-2.7
-4.8
NIA
NEA
N6A
NEA
NEA
N0A
N0A
N0A
N0A
NAA
no feature semantics MinHashh encoder
200
36A
380
390
no Testare semamics- AutaGhoon encvler
5.76
5.76
-488
-7.5
NOA
N6A
360
300
N60
N60
N6A
800
484
NIA
feature semmntics Gap encoder
4.12
4.12
N/A
NEA
N6A
N6A
N6A
N6A
N6A
864
NAA
column semuntics add description column semanties drop coloma names
1.20
1.20
0.0
0.0
N/A
N/A
NIA
NIA
NIA
N7A
N1A
N0A
N0A
NIA
3:35
3.35
-1.2
-2.1
NIA
NIA
NAA
NEA
NIA
NAA
NAA
NAA
N6A
NAA
only get to see subset of the training data. We stress again that this is entirely extrapolating outside of the pretraining setup, where the largest observed context has size 950. Nevertheless, with larger datasets and context, the model performance saturates, requiring further research in how to increase the capacity of tabular ICL.
Training data: We ran training on randomly chosen subsets of the T4 dataset of 10" tables, (1....,6) 6}. .. For this experiment we report the validation scores (on the validation held-out subset of the CARTÉ benchmark) in Figure (right). The results suggest that least 100 of the tables are needed in order to train model of state-of-the-art quality. While exact numbers may vary when using other datasets or model sizes, this gives a high-level estimate of the amount of required data, giving indication of why model could not profit from second step of curriculum learning on the much smaller dataset used by Ma et al. (25). Potentially, training is still limited by data amount or diversity, e.g. missing longer and wider tables, likely required to further scale tabular ICL.
We compare the default eross-entropy loss for classification with the clustering approach described in Section 3.4] allowing to retain the semantics of the target labels and lifting the limitation on the number of target classes as present in TabPFN or TabICL. However, most evaluation datasets only contain non-descriptive labels, and the number of classes is limited. Hence, the potential of clustering cannot emerge, and the default maintains a modest edge, We believe that semantically richer nigh-cardinalin, benchmarks required to spur research in this direction.
Classification target:

---

We observed comparable results on classification tasks and a modest drop in perfor- mance on regression tasks, while substantially enhancing efficiency by eliminating quadratic time complexity and reducing runtime by factor of ten for large contexts.
ISAB block:
Bagging: As expected, using bagging at inference time leads to consistent improvements. At the same time, in absolute terms, the gap is rather marginal, probably thanks to the fact that TonTex(Tab gets rid of most sources of non-equivariance in the architecture (e.g. category indexing and positional encoding). As a result, depending on the use case, this allows to drop bagging, or reduce the number of bags, without compromising performance significantly. Given the great boost of performance when ensembling baselines via inner cross- -rosddaali,,,on, reported in Appendix aA well as recently observed in the literature (10), how to best leverage ensembled ICL models remains an open challenge.
(No) weight sharing: We observed a slight performance degradation when training the full model as opposed to using weight sharing, which may be due either insufficient training duration, suboptimal selection of training hyperparameters, or limited pretraining dataset size and variety.
Other changes (the used sentence embedding model, longer context in training, target clipping for regression, as well as senchmark-dependen fine-tuning) did not result in significant performance gains. Therefore, we refer to the Appendix A.4 for further discussion of those results,
Further ablations:
6 Conclusions
Limitations and future work: While achieving SOTA results across the investigated datasets, we observe several limitations of our proposed approach. One drawback of using real-world data for training is the possibility of contamination, e.g. the presence of evaluation tasks in the training corpus. Since the CARTE benchmark is our focus, we conducted contamination study, using the column name and cell value embeddings created by a text embedder and matching similarities. We did not find any contamination of CARTE in T4. For contamination of OpenML datasets, we refer to the original study by [1). Either way, as a single table is only seen a few times during training, we believe that memorization is likely not practical problem when training on real-world data.
Generally, all investigated table-native ICL models, namely ours, TabPFN, TabDPT, and TabICL, fail to scale their performance to very large datasets. Increasing the context length well bagging did not fully resolve these issues. Using local context, as done in TabDPT, might overcome such limitations but results are likely limited by the row-based architecture in that case. In the large- case, conventional methods, in particular when stacked and ensembled via AutoGluon, still perform best. Overcoming this remains one of the major research challenges for tabular foundation models.
We observe little gain when training on larger data corpora, e.g. increasing the pretraining subset of T4 or adding tables from the AutoML benchmark used for pretraining TabDPT. Together with the investigated scaling of our model, we believe that more diverse real-world data is needed to fully unlock semantic understanding at scale. While models trained on synthetic data do not face this issue in principle, how to incorporate semantic alignment in these models remains an open question. Similarly for evaluation, semantically rich benchmarks rich semantic features, all classification targets are binary, containing no semantic information. Furthermore, it contains mostly small to medium sized datasets, in particular in terms of the number of features. Hence, we argue that more diverse, semantically rich data is required - both at scale for pretraining larger models with longer context, well curated for evaluation, where recent works show promising first steps in this direction 26]
scarce. While CARTE includes
Finally, investigation to utilize target semantics and lift the strict class limit at inference preva- lent in current tabular ICL, by using a dynamic supervised clustering approach, did not result in performance gains. This might relate to the lack of semantically meaningful targets in current tabular benchmarks as well as the limited target cardinality across them and thus remains an open problem.
We have introduced ConTexiTab context aware table-native in-context learner trained on real-world data for both classification and regression tasks. Evaluated across a wide range of tabular benchmarks, our model performs competitively on a range of non-semantic benchmarks while achieving state-of-the-art results, significantly outperforming existing table-native ICI. approaches such TabPFN, the semantically rich CARTE benchmark and low-data regime.
Summary:
10

---

Acknowledgements
We would like to thank Johannes Hoehne, Johannes Hoffart, and Markus Kohler for their insightful comments and suggestions throughout the development of this work, which have greatly contributed to shaping its direction and quality. Furthermore, we thank Mahdi Hadj Ali for creating the semantically enriched evaluation datasets used in our semantic ablation as well as Annie Lim for running the ablation. Finally, we thank Frank Essenberger and Günther Schindler for their support with our data and compute infrastructure.
References
[1] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang- Rafacl Gomes Mantovani, Jan van Rijn, and Joaquin Vanschoren. 2021. OpenMI. henchmarking suites. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
[2] Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pages 785-794)
[3] Michael Chui, James Manyika, Mehdi Miremadi, Nicolaus Henke, Rita Chung, Picter Nel, and Sankalp Malhotra. 2018. Notes from the Al frontier: Insights from hundreds of use cases. McKinsey Global Institute, page 28.
[4] Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Re. 2022. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems.
[5] Jacob Devlin, Ming-Wos Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171 4171-4186.
[6] Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. 2022. LIFT: Language-interfaeed fine-tuning for non- language machine learning tasks. Advances in Neural Information Processing Syxtems, 35:1 11111763-1177 1774.
[7] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and Ahmad Al-Dahle, et al. 2024. The Llama 3 herd of models. arXiv preprini arXiv:2407.21788.
[8] Gus Eggert, Kevin Huo, Mike Biven, and Justin Waugh. 2023. TabLib: dataset of 627 tables with context. arXiv preprint 170007733
[9] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu and Alexander Smola. 2020. SeeoGGluonTabblll Robust and accurate automl for structured data. arXiv preprint arXiv:2003.06500
[10] Nick Erickson. Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, Frank Hutter, et al. 2025. TabArena: A living benchmark for machine learning on tabular data, arXiv preprint arXiv:2506.1679 16791.
[11] Sebastian Felix Fischer, Matthias Feurer. and Bernd Bischl. 2023. OpenML-CTR23 - A curated tabular regression benchmarking suite. In AutoML Conference 2023 (Workshop).
[12] Joshua Gardner, Juan Carlos Perdomo, and Ludwig Schmidt. 2024. Large scale transfer learning for tabular data via language modeling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
[13] Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. 2025. TabM: Advancing tabular deep learning with parfmeter-efficient ensembling. In International Conference on Learning Representations.
[14] Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shleaskii, Akim Kotelnikov, and Artem Babenko. 2024. TabR: Tabular deep learning meets nearest neighbors. In The Twelfth International Conference on Learning Representations.
[15] Yury Gorishniy, Ivan Rubachev. Valentin Khrulkov, and Artem Babenko. 2021. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932-1888 18943.
11

---

[16] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang. and David Sontag. 2023. TabLLM: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statisticx, pages 5549-5581. PMLR.
[17] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. 2023. TabPFN; transformer that solves small tahular classification problems a second. In The Eleventh International Conference on Learning Representaaions.
[18] Noah Hollmann. Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirreesstrr, and Frank Hutter. 2025, Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319 326.
[19] David Holzmüllerr, Léo Grinsztajn, and Ingo Steinwart. 2024, Better by default: Strong pre-tuned ML.Ps and boosted trees on tabular data, Advances in Neural Information Processing Systems, 37:26577-26668..
[20] HuggingFace. 2021. sentence- wwwaasfforers/aaaaaa.. Min L6-v2
sentence- tranfomcatt MiniLM-L6--v.
httpe::/huggtnngtacc. co/
[21] Andrew Jacgle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. 2021. Perceiver: General perception with iterative attention. In International Conference on Machine Learning. pages 4661-4 PMLR.
[22] Guolin Ke, Qi Meng. Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. 2017. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems.
[23] Myung Jun Kim, Leo Griasztajn, and Gael Varoquaux. 2024, CARTE: Pretraining and transfer for tabular learning. In Forty-first International Conference on Machine Learning.
[24] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based commuutttionnnnvaiian neural networks. In International Conference on Machine Learning, pages 3744-3753.
[25] Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony L Caterini. 2024. TabDPT: Scaling tabular foundation models. arXiv preprint arXiv:2410.18164.
[26] Martin Mráz, Breenda Das, Anshul Gupta, I Lennart Purucker, and Frank Hutter. 2025. Towards bench- marking foundation models for tabular data with text. In ICML 2025 Workshop on Foundation Models for Structured Data (FMSD).
[27] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-leam: Machine learning in python. the Journal of Machine Learning Research, 12:2825-2830.
[28] Marek Polewczyk and Marco Spinaci. 2024. ClusterTabNet: Supervised clustering method for table detec- tion and table structure recognition. In International Conference on Document Analysis and Recognition, pages 134-349. Springer.
[29] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. 2018. CatBoost: Unbiased boosting with categorical features. Advances in Neural Information Processing Systems, 31.
[30] Jingang Qu, David Holzmütler, Gaël Varoquaux, and Marine Le Morvan. 2025. TabICL: A tabular foundation model for in-context learning on large data, arXiv preprint arXiv:2502.05564.
[31] Markus N Rabe and Charles Staats, 2021. Self-attention does not need O(N2) memory. arXiv preprint arXiv:2112.0566 0250682..
[32] Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, and Artem Babenko. 2025. TabReD: A benchmark of tabular machine learning in-the-wild. In International Conference on Learning Representations.
[33] Marco Spinaci, Marek Polewezyk, Johannes Hoffart, Markus Köhler, Sam Thelin, and Tassilo Klein. 2024. PORTAL: Scalable tabular foundation models via content-specific tokenization. In NewrlPS 2024 Third Tuble Representation Learning Workshop.
[34] Anton Frederik Thielmann, Manish Kumar, Christoph Weisser, Arik Reuter, Benjamin Sätken, and Soheila Samiee. 2024. Mambular: A sequential model for tabular deep learning. arXiv preprint arXiv:2408.0622)
12

---

[35] Boris Van Breugel and Mihaela Van Der Schaar. 2024. Why tabular foundation models should be research priority. In Proceedings the 40th International Conference Machine Learning.
[36] Liang Wang, Nan Yang. Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024, Multilin- gual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.0567..
[37] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou, 2020. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957
[38] Scott Yak. Yihe Dong. Javier Gonzalvo, and Sercan Arik. 2023. IngesTables: Scalable and efficient training of LLM--enabled tabular foundation models. In NearIPS 2023 Second Table Representation Learning Workshop.
[39] Chao Ye, Guoshan Lu, Haobo Wang. Liyao Sai Wu, Gang Chen, and Junbo Zhao. 2024, Towards tabse-table masked pretraining for web data mining. In Proceedings of the ACM on Web Conference 2024, pages 4449 4459,
[40] Han-Jia Ye, Si- Yiny Liu, Hao-Run Cai, Qi-Le Zhou, and De-Chuan Zhan. 2024. A closer look at deep learning methods on tabular datasets. arXiv preprint arXiv:2407.0095: 00956.
[41] Han-Jia Ye, Huai- Huai-Hong Yin. De-Chuan Zhan, and Wei-Lun Chao. 2025. Revisiting nearest neighbor for tabular data: A deep tabular baseline two decades later. In International Conference on Learning Representations.
[42] Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang. Pengjun Xie, Fei Huang, al. 2024. mGTE: Generalized congeccntext text representation and reranking models for multilingual text retrieval. EMNLP (Industry Track).
[43] Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. 2023, XTab: table pretraining for tabular transformers. In Proceedings of the 40th International Conference on Machine Learning.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We list in the abstract and introduction all contribution that lead to improve- ments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See section "Limitations and Future Work" section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not make theoretical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Full details the training procedure are included, as well as the parameters and datasets used in evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Justification: We don't include error bars, because the models are evaluated on many (203) datasets and computing them c.g. via different train/val/test splits would be too computationally expensive, especially for some baselines. However we do include critical Answer: [Yes]
Justification: The inference code and the trained weights will be released after acceptance to ensure double blind review. Pretraining code might be released, but it might be too tightly linked with the data platform to be usable.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
wwwoooooolllllooooooooloooooolll for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All relevant details have been discussed in the main paper and/or the supple- mentary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
difference diagrams to check when difference in rankings statistically relevant (according to Wilcoxon test).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: While the results can be similarly reproduced on different hardware, we do mention what hardware is used for the main models and for the baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: As a generally applicable machine learning algorithm, we do not expect the algorithms introduced by our claims to lead to harmful consequences any more than any other such algorithm.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: As generally applicable machine learning algorithm, we do not expect the algorithms introduced by our claims to lead to harmful consequences any more than any other such algorithm.
17

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Any risk coming from the training data (T4) cannot be superior to the risk of accessing the raw data itself. Other than that, the model is only trained predict class or number in given table distribution, so we cannot foresee any potential for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All authors properly cited. Our models trained on dataset usable for research only, and as such our models inherit this restriction.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
18

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
The only asset that will released are the model checkpoints themselves; they will be documented in the readme of the repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing was involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No research with human subjects.
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Language models are used, and explicitly mentioned, in this work compute text embeddings of the strings. No other usage has been done, apart from editing the final manuscript.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https://neure) / for what should or should be described.
20

---

A Further Results
A.1 Relation between dataset size and model performance
We plot the average rank of each model as a function of the dataset size (expressed number of rows) across all evaluated datasets in Figure 4 Note that TabICL is missing from this comparison as it only handles classification tasks. We can observe that, as expected, ConTextTab and TabPFN excel in the low data regime, while AutoGluon takes the lead when enough data is available. In the very low data regime below 1000 training rows, TabPFN performs best, however our model surpasses its performance for larger datasets with more than 1000 rows. Overall, ConTexxtab remains competitive with AutoGluon until roughly 10 000 rows, possibly as result of the limitation in both the inference context size and the context size seen during training. After 10 000 rows, gradient boosting methods, as well as RealMI.,, start surpassing the performance of Con as well as TabPFN. Overall, this indicates the need of further research for tabular ICI. to handle larger amounts of training data, but also the availabilig of more diverse benchmarks, covering larger datasets, as the current evaluation is dominated by datasets with less than 10 k rows and less than 100 columns, see Figure 12
Conteechee
Kile [k-5]
LightiaMt (rro,
Roul (nro, cv)
Tuant (ero, cV)
6.5
30
3.
2.602
J,06
2,003
$0,300
22,003 20,00
JAUNOI J0.-E1
Figure 4: Relation between number of training dataset rows and performance, obtained as LOWESS regression in the plane (og(no rank). The confidence bands are the 80% confidence intervals obtained via bootstrapping.
A.2 Extended results
Including further baselines not depicted in the main paper, Table shows all collected evaluations across the investigated benchmarks. In addition to the models shown in the main section, here we also depict performance of (tuned) default variants XGBoost, LightGBM, CatBoost, RealMLP, and TabM. We also report results for Mambular [34] however solely on CARTE and TALENT- Tiny as evaluation took too long on larger datasets due to slow convergence.
Extended baselines:
Win ratios, significance scores and CD diagrams: Additionally, we show the win ratio confusion matrix, p-values, and averages of the investigated models across all investigated benchmarks in Figure
For fine-graiie fine-grained insights, also plot per-benchmark results in Figure 6 for win ratios and Figure 7 for CD diagrams.
Across all benchmarks investigated, ConTextTab shows very strong performance, outperforming other ICL such as TabPEN and TabICL well as tuned trees as well RealMLP and TabM. However, hyperparameter optimized baselines with inner CV ensembles show very strong performance on non- semantic benchmarks (OpenML. CC1 1/C//C,,,,, TabReD, and TALENT-Tiny) where they outperform ConTextTab. This, of course, comes with much longer training times. Furthermore, detailed vs 1 investigation, as well the CD diagrams, paint a more nuanced picture, showing that differences in performance to Tex are often not statistically significant.
21

---

5:99
Colheee pert,
6.00
Calmodd parts
Cepeee perts
6:3
809
300
641
509
Suams
000
8.38
24%
(a) Win ratio matrix with Model A wins over Model B (left) and p-values (right).
Aug
0.5
0.5
0.4
Win
0.2
0.2
0.1
0.0
10 0000 Lereere
Do Odid eaoodd
(oah) Hor
Cousd Mecute
(Ouad) meneeee
Random Rores
hard
(an) se0nao
(b) Average Win ratio of Model against all others.
Figure 5: Win ratio confusion matrix and average of the investigated models across all 203 datasets. Wins are calculated based on accuracy on classification and R on regression datasets. Ties are not counted as wins. Models are sorted by descending overall rank.
On the other hand, detailed evaluation on the semantically rich CARTE benchmark show that ConTexxTaab significantly outperforms all other models and even performs compatible to AutoGluon. With a win ratio of 65% and p-value of 0.03 over TonTexx'aa,, this gain is not as significant as it might appear from the average scores reported in T Table Notably, Con TextTab outperforms all tabular ICL methods in the presence of semantically rich features, for example TabPFN: with a win rate of 96% of Con TextTab and vanishingly small p-value, this is statistically significant. On non-semantic datasets, TabPFN wins over ConTextTab with rates between 49% and 66%, however, with p-values between 0.09 and 0.64, the differences are not statistically significant.
22

---

CAN K.
6.00 X00
201
0.5
0.8
6.03
SA4
CO8 0.00 0:30
0.0
0.05
0.00 Nix 030
000 301 000
2:31
0.00 XXX 0,0 8.0 3W 000
6.0
6.0 X.0
5.20
3.96 0:15
1.5
2.07
0.00 3.0 CA9 5.0 3 668
5.0 0.0
076 5/2
2.2
0.00 3XX 0.6 8.00 3.0
Sarden
0.00
1:0
0.00 3.0 0.08 6.00 3:0
(
Hode
(a) CARTE.
JUM 9.20 9.32
A31 G40 8:01 0.0
300
L.3)
Woite (HROL)
GAB
8.00
6.00
0.0
3.0
6.00
CUB
0.23
8:50
C.0
6.2
Candecars (MPO,
0.00
8:1
0.00
8:34
SAM
8.90
000
0.00
Sandom frre
50
00
001
00
Dall
(b) OpenML--CCIR
6:73 9775
3:4
C.M
0.25
5:00
Worte in (HROL
EXL
Task (P0, oo)
3:49
6:33
9.47
C.46
6.75
0.75
5.51
WAM (MFO, 00
64L
8:31
0.57
8.00
0.37
0.03
8
0.26
0.00
8:00
Casbbent Sasoottee pIeo oo)
9.71 0.0
Candeced
8.24 02 609
DUK-ORN (HPOL
3.00 Ca7 000 0.14
2:3 C49
3.25
2:20
8.21
5.0 0.20
0.28
3.77 627
daacDeen (HFO))
000
0/6
0.30
0000
2.84 E.3
AXC
0.51
0.77
000
3.2 C21
000
03
0.66
Contertoo
8.54 0.00
0.33
028
0.32
100
X31
0.00
6.3 8.30 008
0.32 000
3:00 C37
0094
6.34
1.00
Nimm
4.00 006 649 8.30 008 0.00
0.3
0.0 8:00 000
C00
Go
000
A
(c) OpenML-CTR23.
23

---

8:30
0.00
2:30
0.03
KiDeean (MFO,
8:01
0.15
0.6
LiNGSSH DIPO or)
8.50
6:25
8:0
3.34
0004
044
040
6.64
2:1
8.0
2.00
3.21
Poe IMFO, OO
0.55
6:5
8:25
0.24
042
1:30
0:48
6.50
0.00
8.75
0.33
0.50
0:33
0.13
6.19
6:38
0.23
That pern (vy
Contectio
1:50
3:90
1.50
3.1
0.00
0.05
6:04
6.54
6:23
9.22
311
064 050
8.2
040
3.75
2.53
Rohve
1:30
1:30
Hohre
0.01 0.00 0.00
E:21
0.5
Model
Modid
(d) TabReD.
VOS BANO
9.5
6.02 3.0
2.99
Coffeen (HPO)) cn
6.00
231 044
0:32
UNNIEE (HFO,
6:5
0.11
644 6.1
C41 AM 337 0:3
Contecmo
6.7
011
100 011
2.23
0.09 0.30
TDree (HPO)) (V)
6.99
811
230
535
CARTE
6.00 800 640
631
Marda
006
200
0.
220
5.0 6G4
Nemburr
0.00 8.0
0.8 830 330 600 8.0
Car
6.00 X.c GAR
8.00
300 G00 8:00
(e) TALENT-Tiny.
Figure 6: Win ratios (Model A wins over model B) well p-values of the two-sided Wilcoxon signed-rank test across evaluated baselines separately for each benchmark. Wins are calculated based on accuracy on classification and R2 on regression datasets. Ties are not counted as wins. Models are sorted by descending overall rank.
24

---

Celsor FO.
Kotece
Lnpicen (ed)
(a) All benchmarks.
(b) CARTE.
1894
DAENA 1PO
Roloce
SEEFT
(c) OpenML-CCI8.
(d) SpenML-CTR23.
caloal
Levenn 1en
Rancon Foeen
(e) TabReD.
TALENT-Tiny.
Figure 7: Critical difference diagrams for all evaluated benchmarks collectively and for each bench- mark individually. As TabReD only contains datasets, only ranking results are depicted.
Table 3: Performance comparison across all evaluated benchmarks, depicting mean accuracy (Acc) for classification and (soft-clipped) R²2 score for regression tasks, in percent. Missing values, due architectural limitations failed evaluations, denoted N/A and excluded from the rank calculations. Models are sorted according to their ranking on the CARTE benchmark.
Model Name
All
CARTE
OMI-CCI8
OML-CTR23
TabReD
TALENT-Tiny
R²
AutoGluon [v1.2]
3.34
1.92
73.8
3,42
68.6
2.25
64.6
2.70
TonTextTah [hagging-8]
4.70
2.18
76.9
72.4
5.65
86.6
6.74
73.3
5.38
85.4
63.2
4.24
87.6
76.1
ConTex/Tsh [bagging=]]
5.42
3.22
76.4
71.5
6.51
86.3
6.91
73.1
8.50
85.3
62.6
4.27
87.6
76.0
CatBoost
68.2
4.25
64.3
LightGBM (HPO, CV)
4.43
5.24
73.4
67.5
4.68
87.5
4.11
68.8
1.38
86.0
64.9
3.81
86.9
74.1
RealMLP [HPO, CV]
3.61
5.71
73.4
67.5
3.26
88.3
2.54
75.0
2.25
86.0
64.6
2.70
88.3
76.4
XGBoest [HPO, CV]
4.66
6.12
73.1
66.7
4.03
87.5
4.60
70.9
1.38
86.0
64.9
4,62
86.8
73.6
CatBoost [HPO]
5.13
6.16
75.4
67.1
5.04
87.2
5.31
67.4
2.25
85,9
64.3
4.35
86.5
73.3
LightGBM [HPO]
XGBoest (HPO)
6.61
8.41
72.7
65.8
7.00
86.7
5.66
69.2
1.38
85,8
64.6
5,41
86.4
74.0
CatBoost [TD]
6.83
8.61
75.4
65.8
6.04
86.9
8.57
70.7
3.00
85.9
63.6
5.08
86.2
74.2
[[IPO]
TabDPT [y1.1, k=2048)
6,83
9.84
72.7
65.1
5.01
87.8
5.69
73.2
15.12
83.0
60.9
5,51
86.7
74.8
TabPFN [v2.1.0]
5.81
10.08
72.3
65.0
4.85
87.4
4.74
75.1
5.25
85.6
63.8
2.92
87.8
74.4
HissGradiee/Boosiing
LightGBM (TD)
7.80
10.75
72.7
64.9
6.49
86.4
9.23
67.5
3.25
85,9
63.5
5.95
86.0
72.9
XGBoost (TD)
8.03
12.18
72.5
64.2
6.03
86.7
9.34
68.7
3.75
85.6
62.6
5.89
86.2
72.7
[HPO]
3.50
Random forest
11.01
63.3
11.54
67.9
85.4
60.7
85.8
70.6
TahM
9.42
16.96
70.1
60.5
7.14
86.7
6.60
69.7
10.25
85.3
62.2
5.97
86.1
74.3
ReaIMLP ITDI
8.86
17.96
69.4
60.1
5.39
87.1
7.06
71.0
3.00
85.9
64.0
6.03
86.2
74.7
KNN [k=5]
22.9
34.3
20.46
55.1
Linear
19.52
23.25
62.7
22.8
16.39
80.9
21.63 66
46.2
20.
80.8
21.1
18.35
80.5
41.3
Naive
23.33
24.82
53.0
-1.8
22.43
47.0
24.11
-8.4
20.75
80.8
-0.6
22.84
53.4
-1.22
[v0.0.26]
N/A
TabCL [x0.1.1]
N/A
N/A
72.5
N/A
N/A
88.0
NIA
N/A
NIA
85.1
NOA
N/A
87.4
N/A
Mambular [v155.1]
N/A
N/A
70.1
$2.8
N/A
NIA
N/A
N/A
N/A
N/A
N/A
N/A
83.6
53.9
25

---

A.3 Runtime analysis
We performed runtime analysis of Con TonTTxxtaab, comparing it against TabPFN, TabICI and two non-ICI, models, XGBoost and RealMLP. The results are presented in Figure g For this analysis, we created synthetic dataset consisting of 100 columns, with an equal number of categoricalteettual and numerical features, and varied the number of training/context rows from 1,000 to 10,000. We conducted tests by running one test/query row at a time with varying numbers of training//contex rows, using repeats and reporting averages for each table size. The experiments were conducted on compute node with 40 CPU cores, 320 GB of RAM and an H100 GPU with 96 GB of VRAM.
We find that the throughput of Con comparable other tabular ICL models, such as TabPFN and TabICL, with runtimes ranging from 0.1s to 9s, depending on the table size. As expected, due to the LLM embedding overhead, ConTexxtTab typically slightly slower than TabPFN or TabICI. For Con't TextTab, we observe quadratic scaling behavior, expected from its transformer backbone with memory-efficient attention. However, for TabPFN and TabICI, this effect is not noticeable because of the use of flash attention. As result, ConTexxtaab is roughly twice as slow for larger contexts with 10 000 rows. Moreover, inference runtime can improved with additional technical approaches, such as KV caching or model compilation.
Note that, while non-ICL models offer significantly faster prediction times, they require extensive training. In particular, the ppperaraameeeroppiiiii (HPO) and CV-ensembled (HPO,CV) versions of XGBoost and RealMLP exhibit fit+predict times that are two orders of magnitude longer.
Predict
Fit Predict
9
10a
TabPFN v2.1.0
XGBeast ITD]
7-
Pumiime (becomtss, log scale)
XGBoast [HPO]
ReaIMLP [TD)
6-
ReaIMLP [HPO]
XGBoost [HPO CV]
5-
RealMLP (IPPO
4-
10:
2-
10a
0-
9
10
4
Train x10?
Train
x10
Figure 8: Runtime comparison of different models across varying numbers of training rows. (Left) Runtime for model prediction. (Right) Runtime for model prediction, including fitting. Note the linear y-axis scale on the left side opposed the log-scale the right.
A.4 Further ablations
In the following, we discuss in more detail the ablations as presented in Table
Text embedding model: We test the performance of two more recently released sentence embedding models, namely Multilingual e5 small (36) and aee-mmmiiinnuullbbbaaa cannot observe significant impact on the final results over the default -M- sentence embedder. This potentially indicates the necessity of larger, semantically rich training data or further research into aligning these sentence embedders with tabular foundation model.
42 However, we
Curriculum training with longer context: In this setup, after training the model on T4 with context size of up to 950, training is extended by including also the dataset used by Ma et al. [5]. collected from the OpenML AutoML benchmark, and extending the context size up to 4000 rows. Data is sampled from either T4 with 80% probability or the dataset from by Ma et al. 25 with 20% probability. The resulting model shows some improvements on a few tasks, but by statistically insignificant amounts.
26

---

p-value
(semantic encoder)
0.04
Win Ratio
1.0
(semantic encoder)
0.00
1.0
0.00
0.00
0.00
min hash encoder
0.84
0.25
0.21
0.16
0.8
Model
gap encodar
0.84
0.75
0.49
0.31
0.6
Model
gap
0.00
0.00
0.4
ordinal encoder
0.92
0.65
0.48
0.18
0.4
ordinal encoder
0.00
0.00
0.39
0.00
0.2
AutoGluon encodar
0.96
0.82
0.69
0.80
0.0
base
(semantic encoder)
enooder
ordinal arooder
AutoiBuoon enooder
base
(somantic oncocor)
min hash emooder
Auto enooerr
hash
Model
Model
(a) Win ratios with Model A wins over Model B.
(b) P-values of two-sided Wilcoxon signed-rank test.
Figure 9: Detailed results of the semantic feature ablation, comparing the use of different non-
semantic or conventional encoders to the base model using LLM embeddings.
Regression target and clipping: When using one-dimensional encoding for numerical values, can choose to clip the tails of the distribution for better stability. This is needed in pretraining due the noisy data. In practice, whether do this on downstream tasks can chosen depending on outlier analysis of the data itself. We test alternative levels of clipping: no clipping (default), clipping 0.1%, 0.5% and 2% quantiles. We also compare this to using binning instead of one- mee-imensiooal embeddings. We find that both clipping by 2% and binning cause slight reduction in quality (since we are using n 16 bins, note that binning also incurs clipping at xy 3%). The only exception is in the OpenML-CTTR23 benchmark: it contains one dataset with extreme outliers where no model achieves a positive R² score, and adding clipping there limits degenerate scores.
Impact of fine-tuning: The training process can be easily adapted to fine-tuning on specific collection of datasets or even single task. We experimented with fine-tuning on (the training partition of) all the 203 datasets at the same time. However, we observe that, if during this fine-tuning fix the target column always be the one corresponding the prediction task be evaluated, rapidly encountered overfitting and prediction scores degrade. This might be a similar problem to the scaling law of training data, as also observed by Ma al. [25]. the present situation, we can avoid overfitting by selecting the target column randomly with the same selection procedure described in Section However, as shown in Table 2
the gain is rather limited.
Semantic embeddings ablation: In addition to the ablation results previously shown in Section 5.1 depict the win ratios and p-values of two-sided Wilcoxon signed-rank test for the cell value and column name semantic ablations in the Figure and Figure 10
B Suplementary Diagrams
concor
(Query))
Multi head
MAB
LN
MAB
(Key, Value)
sttention
Q
Multi-head attention block (MAB)
Induced self-attention block (ISAB)
Figure 11: (left) Multi-head attention block (MAB), (right) Induced self-attention block (ISAB) architecture diagram.
27

---

Win Ratio
p-value
0.43
0.12
1.0
column descmpttin
1.0
description
0.8
Hodel B
column names
0.57
0.08
0.6
Model
columnn names
0.40
0.6
0.4
0.4
no column names
0.66
0.92
no calumnn names
9.00
0.00
0.0
0.0
columin names descrition
no column names
column names
colurnn names
cokme
colurn
Madel
Model
(a) Win ratios with Model wins Model B.
(b) P-values of two-sided Wilcoxon signed-rank test.
Figure 10: Detailed results of the semantic column name ablation, comparing the base model ("column names") to variations where column names are replaced with non-semantic ones ("no column names") or enriched with additional column descriptions ("column names" + description).
C Experimental Setup - Details
C.1 Baselines
Data preprocessor:
Evaluating models across multitude of datasets can be tricky. Datasets may have inconsistent data type annotations, such categories represented strings categorical data types, covering low- and high-cardinality categorical features, date, time, or datetime instances, free text, boolean values and more. Most models, however, require numerical input to process and handle non-numerical data types or missing values differently. To unify our evaluation, we implemented configurable default feature encoder built on the
from [9] which we found to be very versatile and robust. In particular, the encoder natively handles low- and high-cardinality categorical data, free text (to some extent), well as datetime encoding. For flexibility and compatibility with a multitude of models, we extended the default implementation cover the following options that can be adapted the capabilities of the baseline model at hand:
AutoGluon
Whether to convert booleans to string/categorical values.
Whether to not encode string/categorical values for models that natively handle them, such as CatBoost.
Whether to scale numerical data via quantile scaling with a normal distribution as target.
Whether to drop constant features.
Whether to impute missing values, extending the standard imputation (using most frequent categories and mean for numerical data) to bools and datetime data types.
As default, we choose to convert booleans, encode categoricals via ordinal encoding, scale numerical data, drop constant columns and impute missing values.
Below, we describe for which baselines the default values are changed or when other types feature encodings are used.
TabPFN: We use the model from the official Python tabpfn package with ver- sion 2.1.0 together with the tabpfn-extension package version 0.1.0 at commit 1B00ffff0..54eaabb00111114444444 not up-to-date.
of the Git repository as the PyPi release is
Naturally, FabPFNCIassiifer and TabPFNRegressor for classification and regression tasks, respectively, using default parameters for both. In particular, TabPFiciasssifier uses an ensemble
.iit
28

---

of 4 and TabPFNRegrossor an ensemble of estimators. We combine the classification estimator with the ManyClassClaseifiee extension with a redundancy factor of enable classification beyond the native 10-class limit of TabPFN which is required for the evaluation of some of the 203 evaluated datasets.
For datasets larger than the native 10k limit of TabPFN, we sample a random 10k subset of the training split. This affects 66 out of the 203 evaluated datasets. For datasets with more than the 500 feature limit that TabPFN was trained with, we select a random subsample of 500 features. This affects 12 out of 203 evaluated datasets. While this is not optimal, and post-hoc ensembling as well as random forest preprocessing is recommended by the authors [18], these extensions cannot be combined with the many-class extension required to predict beyond the 10-class limit of the native TabPFN model. Hence, we cannot evaluate TabPFN with the post-hoc ensembling or random forest extension.
As we found the native feature encoder of TabPFN to not work across all evaluated datasets, we use our standard feature encoder (see above), encoding categorical columns, scaling numerical values, dropping constant columns, and imputing missing values. As this procedure should be very similar to the TabPrN-native encoder, anticipate this deviation affect the results only insignificantly (if at all).
We use the latest model weights tabicc-calleee www.ssaiiar-rrreeee 1 1-0506. ckpt from the recent 0.1.2 version of the official tabicl package. This updated variant an improved checkpoint over the one reported in the original works [30]. As TabICL solely supports classification tasks, exclude it from the overall mean rank evaluation. For encoding, we use our default encoder, but do not scale numericals, do not drop constant values, and do not impute missing the model.
TabICL:
it is natively handled by
TabDPT: We use the model from the official GitHub repository at the recent 1.1.0 release and tabdpt1_1... 1.1..ppp. pth model checkpoint. Naturally, we use TabDPTClassis assifier and TabDPTRegressor for classification and regression tasks, respectively, using default parameters for both. Throughout, we evaluate the model with a (local) context size of 2048 which is the best best-performing one in the original works [2)]. However, evaluation failed for some datasets due to error in the original code leading to empty predictions for very large datasets in the TabRel) benchmark. Here, we use our default encoder, scaling numericals, dropping constant values, and imputing missing ones.
We use the model provided in the official Python carte-ai package with version 0.0.26. We use CARTEClassifier and CARTERegressor with default parameters for classification and regression tasks, respectively. We treat binary classification tasks as 2-class multi-class classifica- tion and hence set loss==cateeoricaa
CARTE:
crossentropy" for the classifacation estimator. With CARTE, we use our default preprocessor to convert bool values and datetime instances and to impute missing values, but otherwise rely on the Table2GraphTranse Trasfforrre provided in the reference implementation.
Pytabkit models: We use the pytabkit implementation wrapper for evaluating ReaIMLP, TabM, XGBoost, LightGBM, and CatBoost. We evaluate all models both in the tuned-defaults (TD) variant proposed by Holzmüller et al. [9 (except for TabM which only has non-tuned defaults [D], hyperparameter-optiiize (HPO), as well as 5-fold inner cross-validation (CV) ensembled HPO variants (HPO-CV). For all HPO and HPO-CV variants, we use the recently added tabarena search spaces proposed [T].
In particular, for RealMLP (TD), we use RealMLP_TH SaaPP__D_Caassssiit and RealMLP_TH TD_Regrrssor for classification and regression tasks, respectively. For RealMLP (HPO, HPO-CV), we use RealMLP SIPO_Classii ifierri and RealMLP_HH [HIIMP__PPP___ _____P__R__ huPo_Regressor for classification and regression tasks, respectively, conducting the default 50 rounds of random search HPO. For the ensembled variant, we use 5-fold inner CV.
For TabM (D), use TabM_D_Classinn Classifier and TabM_D SaMM_D_HR_ __Regressor for classification and re- gression tasks, respectively. For TabM (HPO, HPO-CV), we use TabM_HPO HL_HHooClaasffeerrrrrrrree and T'bM_HPO HPP__Regreeeeeeeeeeeeeeeeeeeeeee for classification and regression tasks, respectively, conducting the default 50 rounds of random search HPO. For the ensembled variant, we use 5-fold inner CV.
29

---

For XGBoost (TD), LightGBM (TD), and CatBoost (TD), we use LGBM_ R CatBoost TD_ [D_Classifier and CatBoost TD Regressor for classification and regression tasks, respectively. For the HPO-variants, we use the HPO TPE versions of the estimators, performing Parzen-tree based HPO with 50 rounds. For the ensembled variant, we use 5-fold inner CV.
XCB_TD_ClaSt caasssiiie, HiB_TDRRegresorr
CatBoost, LightGBM and XGBoost are evaluated CPU machines with up to 256 GB of RAM, whereas RealMLP and TabM are evaluated on H100 GPUs with 96 GB of VRAM.
Throughout,
default encoder, scaling numericals, dropping constant values, and imputing missing ones. For all models but CatBoost, we perform ordinal encoding of categoricals.
Sklearn models:
We use several standard baseline models from scikit-leary 27 combining them with the default preprocessor as outlined above. Across all scikit-leate baselines, prepro- cessing only differs in missing value imputation, depending on the model's capability to handle missing values natively. Throughout, evaluation is performed using scikit- sciart-learn v1.5.2.
For the naive predictor, we use the DumnyClassifies and DummyRegressee to predict the most frequent, respectively mean value of the train splits the naive majority baseline.
LogisticRegression cRegression and Lineareegression for classifica- For the linear predictor, we use the Logisti
tion and regression tasks, respectively, using default hyperparameters.
For the KNN predictor, we use the KNeighhorsCCasee Cww.saaaafffiaaacaaaaaaaaaalaala ier and (NeighborsRegressor for clas- sification and regression tasks, respectively, using default hyperparameters and k 5 nearest neighbors.
For the random forest predictor, we use the RandomForestClassiiiee and BandomForestRegressse for classification and regression tasks, respectively, using default hyperparameters. The model handles missing values natively.
the Finally, for the histogram-based gradient boosted tree predictor, for classifi- ieterrraientBoostnnnniiiie
and #issGraedieertooettnggggeeeeee cation and regression tasks, respectively, using default hyperparameters. The model handles missing values natively.
We use the official implementation from the mambular PyPi package, version 1.5.1. We use MambularClassif fier and Mambul arRggressor with default parameters. As training ran out of memory for very wide datasets, we randomly sample 500 columns if the dataset exceeds it.
Mambular:
AutoGluon:
Throughout, we use AutoGluon v1.2 and its TabularPredictor without custom preprocessing. We use the best_qualite preset and set per-dataset time limit of 4 h. Otherwise, parameters are left at their default values. For all datasets, evaluation is executed on single 16-core machine with 128 GB of RAM and no GPU.
C.2 Datasets
Full details of all used dataset and benchmarks are provided in Table The row and column count statistics are further visualized in Figure [2 Additionally, the per-benchmark statistics regarding the number of tasks as well as the data type proportions are presented in Table 4 Note that not all types grouped as object are necessarily free text, but that they can also contain categorical or time/date instances that are not correctly cast by pandas by default. However, these are typically picked up correctly by the used data preprocessors.
We extracted all datasets from their original source and performed custom stratified test split with a 70-10-20 ratio. For classification tasks, the target column is used for stratification. For regression tasks, we perform stratification the binned target column, binning it into quantiles using the qcut method from the pandas library. Otherwise, we do not perform any alterations on the data. Models not using specific validation procedure provided with the concatenated train and validation split for training.
30

---

Table 4: Task and data type statistics of the evaluated benchmarks.
Number of tasks
Dtype rates /
Benchmark
total
classification
regression
object
category
bool
float
datetime
CARTE
51
11
40
80.74
0.00
0.00
19.04
0.22
0.00
OpenML-CC11
72
72
0.05
7.85
0.00
55.80
22.70
0.00
OpenML-CTR23
35
0
35
0.95
3.91
0.00
66.19
16.91
0.00
TabReD
0.14
0.00
0.05
89,95
9.41
0.44
TALENT-tiny
37
26
11
12.77
0.00
0,44
63.63
23.15
0.00
Dataset statistics
num cols
102
collection
xx
openmi _cc18
openmi _ct/3
talent, tiny tabred
target type regression
classification
10
10
10a
num_rowe Fows
Figure 12: Column and row distribution of the evaluated benchmark datasets.
31

---

Table 5: Details of all used benchmark datasets.
Benchmark
Table
Num. rows
Num. cols
Target type
CARTE
unime_plane
11512
11
regression
CARTE
babies_r_ut r_ss
4068
regression
CARTE
beer_ ree__raings
2557
20
regression
CARTE
bikedekho
3828
regression
CARTE
bikewale
7193
regression
CARTE
buy_buy_baby
8574
regression
CARTE
cardekho
30250
17
regression
CARTE
thocolate_bar_raate
2070
classification
CARTE
clear_corpit
3779
27
regression
CARTE
coffee coffeeeraaiings
1661
classification
CARTE
company_ eempoyyyeemploees
8752
regression
CARTE
employee_remuneration
28316
regression
CARTE
employee_salaries
7368
regression
CARTE
@ia22_playerr
14468
19
regression
CARTE
filmtv _lmtv_moviies
32964
10
regression
CARTE
journal_jes jer
7692
10
regression
CARTE
journal_sj
22344
10
regression
CARTE
jp_anime
12428
17
regression
CARTE
k_drama
991
13
regression
CARTE
michelin
5465
classification
CARTE
mids_ mids_salaries
8364
regression
CARTE
movies
5779
14
regression
CARTE
museums
9173
17
regression
CARTE
mydramalist
2720
14
regression
CARTE
nba_draft
1335
classification
CARTE
prescription_diee drugs
1371
regression
CARTE
ramen_ renen_ratings
3267
classification
CARTE
roger_eberr
2149
classification
CARTE
rotten_tomaton
5726
14
regression
CARTE
spotify us se__aciidens_coom counts us accidents severity us presidential used cars 24 used cal_car___ha caar_henz_itaal used__ars_paakittan
32879
19
classification
CARTE
18098
regression
CARTE
17324
11
classification
CARTE
15885
regression
CARTE
4734
regression
CARTE
13112
regression
CARTE
3207
11
regression
CARTE
58124
regression
CARTE
used_ccrrs_ cars_ saudi arabia
4405
11
regression
CARTE
videogame_salee
13128
regression
CARTE
whisky
1449
classification
CARTE
wikiliq_
10768
10
regression
CARTE
wikiliq_spirit
9820
regression
CARTE
wina_pl
1797
16
regression
CARTE
wine ceo__coo______eeeeeeeeee_eeee poocee
12203
10
regression
CARTE
wine_dot_commreet
3276
10
regression
CARTE
wine wine_enthusiasts_pe pprices
96780
10
regression
CARTE
wine_enthusiasttsrainng
103976
10
regression
CARTE
wine vivino errviioooooiceeeee____
11067
regression
Continued on next page
32

---

Collection
Table
Num. rows
Num. cols
Target type
CARTE
wine_vivino_ratiin
11067
regression classification
CARTE
yelp
51692
13
CARTE
zomato
49152
classification
OpenML-CC18
adult
39073
15
classification
OpenML-CC18
analcatdata authooshhip
672
classification
OpenML-CC18
analcatdata ddfft
637
classification
MLenML-CC18
Dalance_scale
500
classification
OpenMI-CCC8
Lank_marketing
36168
17
classification
OpenML-CC18
banknote authentication banknote_autheeniiation
1097
classification
OpenML-CC18
bioresponse
3000
1777
classification
OpenML-CC18
blood transfusion caansfusion_serviee_coomm service__enterr
598
classification
OpenML-CC18
breast_
559
10
classification
OpenML-CC11 car
1382
classification
OpenMI.-CC18 churn
4000
classification
OpenML-CC18
cifar 10
48000
3073
classification
OpenML-CC18
climate hocellemooeeeeeeeee simulation_craseeeee crashes
432
19
classification
OpenML-CC18
cme
1178
10
classification
OpenML-CC18
cnae_9
864
857
classification
OPenML-CC18
connect_4
54045
43
classification
Ope1MI-CC18
credit_approval
552
16
classification
OpenML-CC18
credit_g
800
classification
OpenML-CC18
cylinder_banne bands
432
38
classification
OpenML-CC18
devnagari script
73600
1025
classification
OpenML-CC18
diabetes
614
classification
OpenML-CC18
dna
2548
181
classification
OpenML-CC11
dresses_sale
400
13
classification
OpenML-CC18
electricity
36249
classification
OpenML-CC18
eucalyptus
588
20
classification
OpenML-CC18
fashion mnist
56000
785
classification
OpenMI--CC18
first_order, fist_orrdertthoorem
theorem paooiinggrrgggg
4894
classification
OpenML-CC18
gesurechhascegmonnatttpprrssssss
7898
33
classification
OpenML-CC18
har
8239
562
classification
OpenML-CC18
ilpd
466
11
classification
OpenML-CC18 internet advertisements
2623
1559
classification
OpenML-CC18 isolet
6237
618
classification
OpenMI-CC18
jml
8708
22
classification
OpenML-CC18
Jungle_chess_2pee Baciss_pcs__aaa__ Batties__ppps_a___dddaaaee paposs_aw__nngame_oomppp
35855
classification
OPenML-CC18
kel
1687
classification
OpenML-CC18
kc2
417
22
classification
OpenML-CC18
kr kr_vs_kp
2556
37
classification
OpenML-CC18
letter
16000
17
classification
OpenML-CCC1
madelon
2080
501
classification
OpenML-CC18
pifeat_factors
1600
217
classification
OpenML-CC18
mfeat_fouries
1600
77
classification
OpenML-CC18
mfeat karhunen
1600
65
classification
OpenML-CC18
mfeat_ moraatoooollll
1600
classification
OpenML-CC18
mfeat mfeat_ppxel
1600
241
classification
OPenMI-CC18
mfeat_zernike
1600
48
classification
OpenML-CC18
miceprotein
864
78
classification
Continued on next page
33

---

Collection
Table
Num. rows
Num. cols
Target type
OpenML-CC18
mnist_784
56000
785
classification
OpenML-CC118
nomao
27572
119
classification
OpenML-CC18
numerai28_6
77056
22
classification
OpenML-CC18
optdigits
4496
classification
OpenML-CC18
lzone_level. 8hr
2027
73
classification
OpenML-CC18
pel
887
22
classification
MLCCM1-CC18
pc3
1250
38
classification
Op
pc4
1166
38
classification
OpenML-CC18
pendigits
8793
17
classification
OpenML-CC18
phishingwebsites
8844
31
classification
OpenML-CC18
phoneme
4323
classification
OpenML-CC18
qsar_ _ard_bodee
844
42
classification
OpenML-CC11
satimage
5144
classification
OpenMI.-CC18
segment
1848
classification
OpenMI-CC18
semeion
1274
257
classification
OpenML-CC18
sick
3017
30
classification
OpenML-CC18
spambase
3680
58
classification
OpenML-CC18
splice
2552
61
classification
OpenML-CC18
fhel_ppaaee_fault
1552
28
classification
OpenMI-CCC18
texture
4400
classification
OpenML-CC18
dic_tac_uu
766
10
classification
OpenML-CC18
vehicle
676
19
classification
OpenML-CC18
vowel
792
13
classification
OpenMI-CCC8
cacll_root___aee navigation
4364
25
classification
OpenML-CC18
wdbc
455
31
classification
OpenML-CC18
wilt
3871
classification
OpenML-CTR23
abalone
3341
regression
OpenML-CTR23
airfoil Selli_ssllsaiee
self_noise
1202
regression
OpenML-CTR23
auction auction_verifican verification
1634
regression
OpenMI-CTTR2
brazilian wrazilian_houses houses
8553
10
regression
OpenML-CTR23
california_house
16512
regression
OpenML-CTR23
643
18
regression
OpenML-CTR23
concrote_coppresiieee sepppresssive_sstennti
824
regression
OpenML-CTR23
eps88wages
22524
regression
OpenML-CTR23
cpu_aee qpu_activity
6553
22
regression
OpenMI-CTRR3 diamonds
43152
10
regression
OpenML-CTR23
energy_efficiency
614
regression
OpenML-CTRR3
fifa forest fires
15342
29
regression
OpenML-CTR23
413
13
regression
OpenML-CTR23 fps benchmark
19699
44
regression
OpenML-CTRR3 geographical_orie of music
847
117
regression
OpenML-CTTR2 grid_stability
8000
13
regression
OpenML-CTR23
health insurance
17817
12
regression
OpenML-CTR23 kin8nm
6553
regression
OpenML-CTR23
kings_county
17290
22
regression
OpenML-CTTR2
miami housing
miami_housing
11145
16
regression
OpenML-CTTR3
moneyball
985
15
regression
OpenML-CTTR3
naval naval_propulsion_ng plant
9547
15
regression
OpenML-CTTR3
physiochemical proteei
36584
10
regression
Continued on next page
34

---

Collection
Table
Num. rows
Num. cols
Target type
OpenML-CTR23
pumadyn32nh
6553
33
regression
OpenML-CTR23
qsar_fish fish_toxicity
726
7
regression
OpenML-CTR23
red_wine
1279
12
regression
OpenML-CTR23
sarcos
39146
22
regression
OpenML-CTR23
socmob
924
regression
OpenML-CTR23
solar_ flare
852
regression
space_ga
2485
regression
OpenML-CTTR23
student_perforrmat
519
31
regression
OpenML-CTR23
superconductivity
17010
82
regression
OpenML-CTR23
ande__trnsscoiigg
55027
19
regression
OpenML-CTR23
wave_energy
57600
49
regression
OpenML-CTR23
white_wing
3918
12
regression
TabReD
acquire_van
133602
117
classification
TabReD
cooking_time
278338
196
regression
TabReD
delivery_eta
313589
227
regression
TabReD
credit risk model podellssability
325663
701
classification
TabReD
homesite Lirill_uutt__ccee conte_oonnerioon
244458
302
classification
TabReD
maps_routing
219994
991
regression
TabReD
sberbank_rusiianhhuusiiee
23674
396
regression
TabReD
TabtheD_watther
382955
111
regression
TALENT-iiny
ailerons
11000
regression
TALENT-iny
breast w
31492
10
classification
TALENT-tnyy
eme
44236
10
classification
TALENT-tny
dis
3017
30
classification
TALENT-tny
yye_movementt_bin fried
6086
21
classification
TALENT-tnny
32614
11
regression
TALENT-tiny
house_ _eg_roeeg_g____________e
18227
17
regression
TALENT-tiny
ibm Jom_employee
www.ellyeeperrfrrmannee _performancee
1176
classification
TALENT-tiny
jungle_chess___ bhess_2pes_ravv fosss_rww_eee waw_endgame_eaaaeeaa complete
35855
classification
TALENT-tnny
8708
10
regression
TALENT-tny
kel
1687
22
classification
TALENT-iny
kin8nm
6553
regression
TALENT-tiny
law_ law_school admission achool_admission
bianry
16640
12
regression
TALENT-tnn
1600
77
classification
TALENT-tiny
my
32614
11
regression
TALENT-iny
okcupid_stem
21341
14
classification
TALENT-tny
vuline_shoppent
9864
15
classification
TALENT-iny
optdigits
4496
classification
TALENT-tny
page
page_blocks blocks
4378
11
classification
TALENT-inny
pc3
1250
38
classification
TALENT-tiny
pendigits
8793
17
classification
TALENT-tnn
pol
8065
27
classification
TALENT-tny
pol_reg
12000
49
regression
TALENT-(ny
3976
13
classification
TALENT-tny
satimage
5144
37
classification
TALENT-tnn
segment
1848
18
classification
TALENT-tin
socmob
924
regression
TALENT-tnn
splice
2552
61
classification
TALENT-tiny
sylvine
4099
21
classification
Continued on next page
35

---

Collection
Table
Num. rows
Num. cols
Target type
TALENT-tnn
thyroid_dh dis
2240
27
classification
TALENT-iiny
tic_tac_to
31492
10
classification
TALENT-inny
vulnonevul
4553
17
regression
TALENT-tiny
waterstress
950
23
regression
TALENT-tiny
waveform_500 5000
4000
classification
TALENT-tin
website__pishhe
1082
10
classification
TALENT-tny
wine
2043
classification
TALENT-tnn
wine_ white
3918
12
classification
36


Reviews:
[{"review_id": "9lqyXGLsiH", "reviewer": "Reviewer_zDEh", "summary": "The paper proposes ConTextTab, an in-context learning model pretrained from the real-world T4 dataset. ConTtextTab employs embeddings tailored to different data modalities, in particular incorporating semantic embeddings of column names and categorical values. The experiments were conducted on diverse range of datasets.", "questions": "Please refer to weaknesses above.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- In general paper is well written and easy to follow.\n- The performance comparison across diverse datasets shows competitiveness of ConTextTab.\n- The work proposes architectural choices and training schemes that maybe useful for future related works.\n\nWeaknesses:\n- It might be useful to include some characteristics of the datasets. For example, are the datasets provided with textual features or are they processed to have only the numbers?\n- It would be useful to include the hyperparameter search space.\n- It would be interesting to see a comparison with simple random-weights (no pretraining).\n- In table2, the word 'fine-tuning' is obscure, as the ablation is carried out on benchmark specific fine-tuning. It could be more clearer with 'benchmark-dependent fine-tuning'.\n- What are some expected outcomes with dataset specific fine-tuning?\n- It would be interesting to see computation time comparison between the in-context learning models. This could provide some insights on the trade-offs in using larger llms.", "was_revised": true, "final_justification": "The authors have appropriately addressed my concerns and have added meaningful ablations that strengthens the paper.", "rebuttal": "Thank you for your feedback and highlighting the comprehensiveness of our evaluation and the competitiveness of our model!\n\nTo address your concerns and questions:\n\n## Runtime analysis\nThank you for this suggestion! **We have added a runtime comparison** in a controlled fashion using synthetic tables with 100 columns and varying number of training rows ranging from 1k to 10k. We synthesize data with 50% categorical/textual and 50% numeric features. We created two plots with runtime (prediction and fit + prediction) for ConTextTab, TabICL, TabPFN, XGBoost, and RealMLP for various numbers of training/context rows and one test/query row, using 5 repeats and report averages. The experiments were conducted on an H100 GPU. We find that ConTextTab is comparable to other tabular ICL models such as TabPFN and TabICL – with runtimes ranging from 0.1s up to 8s depending on the table size. As expected, due to the LLM embedding overhead, ConTextTab is typically slightly slower than TabPFN or TabICL. For ConTextTab, we observe quadratic scaling behaviors as expected due to the use of transformer backbone with memory efficient attention, however for TabPFN and TabICL it is not as noticeable, likely due to the use of flash attention. This results in ConTextTab being slower by a factor of 2 for larger context with 10k rows. While non-ICL models offer significantly faster prediction times, they require extensive training. In particular, the hyperparameter-optimized (HPO) versions of XGBoost and RealMLP exhibit fit+prediction runtimes that are two orders of magnitude higher.\n\n\n## Dataset characteristics\nWe are uncertain whether this concerns the pretraining data or benchmark data used for evaluation? We provide details on the evaluation data in the appendix. Throughout, features are processed as described in the main paper, using different encoders for text, numbers, and dates in the case of ConTextTab. For all evaluated baselines, feature preprocessing is outlined in the appendix.\n\n## Hyperparameter search space\nWe report the hyperparameter search space of the evaluated baselines in the appendix. For our model, we ran extensive experiments in the ablation study section in which we tried different configurations such as model size, etc., however we do not perform (algorithmic) hyperparameter optimization as is common for deep learning approaches.\n\n## Random-weight comparison\nWithout any pretraining our model produces random predictions. We are unsure about the scope and goals of the suggested ablation. Note that the full model, including embedding layers and prediction heads, are (pre)trained end-to-end.\n\n## Fine-tuning\n**We have clarified the phrasing** in the ablations table.\nWe perform benchmark-specific finetuning to observe further downstream task improvements as a core capability of a foundation model, as similarly investigated in previous works [1, 2]. Dataset-specific fine-tuning was not effective, as the model quickly overfitted; it essentially encountered the same context at every training step.\n\n## Summary\nWe hope to have clarified your questions and are looking forward to the discussion!\n\n\n### References\n[1]: Hollmann et al.: Accurate predictions on small data with a tabular foundation model. Nature, 637.8045, pp. 319-326\n\n[2]: den Breejen et al.: Fine-Tuned In-Context Learning Transformers Are Excellent Tabular Data Classifiers, arXiv:2405.13396"}, {"review_id": "yQYcqdgvSa", "reviewer": "Reviewer_A8ss", "summary": "This paper presents ConTextTab, a semantics-aware, table-native in-context learning (ICL) model for tabular prediction tasks. The motivation is to bridge the gap between traditional table-specific ICL architectures (e.g., TabPFN, TabICL), which excel at leveraging structural information but neglect table semantics, and large language model-based approaches (e.g., TabuLa-8B), which understand semantics but are inefficient in processing large tabular data due to linear tokenization and context limitations. ConTextTab enhances the table-native ICL backbone by introducing modality-specific embeddings for textual, numerical, and date columns, as well as semantic embeddings for column names and categorical values. The model is pretrained on large-scale real-world tabular datasets (T4), and evaluated on a suite of tabular benchmarks, demonstrating strong results—especially setting a new state-of-the-art on the semantically-rich CARTE benchmark, and exhibiting robust performance in low-data regimes. Comprehensive ablations analyze the impact of model components, context size, and data scale.", "questions": "1. Can you provide experiments that directly test semantic understanding capabilities, rather than relying solely on downstream task performance? For example, can the model correctly identify semantically similar columns across different datasets, or does it show improved performance when categorical labels have semantic meaning versus arbitrary codes? Such experiments would strengthen the core claim about semantic awareness.\n\n2. Your results show that performance plateaus after 100K training tables and that larger models perform worse. This contradicts the intuition that semantic understanding requires large-scale data and model capacity. Can you provide a more detailed analysis of what limits further scaling and whether this indicates fundamental limitations in your approach?", "limitations": "The authors do address limitations in their conclusion section, acknowledging the challenges with scaling to large datasets and the need for more diverse semantic evaluation benchmarks. However, the limitations section could be more comprehensive. Missing discussions include potential data contamination effects, the circular reasoning in using downstream performance to validate semantic understanding, and the practical constraints of the method's applicability. The authors should also acknowledge that their evidence for semantic understanding is primarily indirect and that the method's advantages may be limited to specific data regimes and domains.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n1. The experimental evaluation is comprehensive, covering 203 datasets across five different benchmarks with proper statistical analysis including critical difference diagrams. The ablation studies are thorough, examining various architectural choices, training data scales, and hyperparameter settings. The authors provide detailed implementation details and plan to release code and model weights.\n\n2. The paper is well-written with clear motivation, methodology, and experimental setup. The figures effectively illustrate the architecture and results. The related work section provides good context for understanding the contribution relative to existing methods.\n\n3. The multi-modal embedding approach is sensible, with appropriate handling of text, dates, and numerical features. The architectural modifications to TabPFN are well-motivated and the training procedure is clearly described.\n\n### **Weaknesses**\n\n1. The paper lacks direct validation of semantic understanding capabilities. There are no experiments demonstrating that the model can leverage semantic similarities between column names, understand categorical label meanings, or transfer knowledge across semantically related tasks. The \"semantic understanding\" claim rests primarily on downstream task performance.\n\n2. While the combination is novel, the core technical contributions are relatively incremental. The approach primarily involves using pre-trained text embeddings and training on real data rather than synthetic data. The architectural modifications are minor adaptations of existing transformer-based designs.", "was_revised": true, "final_justification": "The authors have addressed my all concerns. I'm excited to see work that effectively bridges language model representations with traditional approaches.", "rebuttal": "Thank you for your feedback and acknowledging the comprehensiveness and rigor of our evaluation, as well as the conducted ablations and general approach!\n\nTo address your questions and concerns:\n\n## Direct validation of semantic understanding\nThank you for this suggestion! While we do believe that CARTE performance showcases semantic capabilities, we agree that this is indirect and that a direct ablation would strengthen our claims. To this end, **we have conducted two semantic ablations on the CARTE benchmark** where we would expect the effect to be most pronounced:\n\n### Semantic feature ablation\nTo ablate the impact of semantic embeddings of the features (string, categoricals), we evaluated the following preprocessing techniques to replace the LLM embeddings, ranging from methods that do not contain any semantics as well as conventional string encoders:\n\n-\tUsing `OrdinalEncoder`, effectively replacing all string/categoricals with ordinal integer encodings, eliminating all potential semantic information,\n-\tUsing `OrdinalEncoder` for low-cardinality and `MinHashEncoder` for high-cardinality features as a conventional method for string embeddings,\n-\tUsing `OrdinalEncoder` for low-cardinality and `GapEncoder` encoder for high-cardinality features as another conventional method for string embeddings,\n-\tUsing the default AutoGluon feature encoder which uses low-level NLP features for high-cardinality columns.\n\nWe compare these approaches to the default ConTextTab model with semantic LLM embeddings.\n\n### Semantic column name ablation\nSecond, we investigate the impact of the semantic column name embeddings which we add to the feature embeddings.\n\nTo this end, we investigate the following two ablations:\n-\tReplacing the column names with `col1`, `col2`, …, `colN`. This removes any potential semantic information contained in the column names.\n-\tWe also perform the inverse ablation: using an LLM (`gemma3-12b`), we created contextualized column descriptions for all CARTE tables by presenting the model with 5 randomly sampled rows (serialized to JSON) and asking for contextualized descriptions. We then replace all column names with `<name>:<description>` to semantically enrich them and perform prediction with ConTextTab as usual.\n\n### Results\nWe find the following results, showing the base model performance and absolute changes over the base model performance for the ablated versions, which we have appended to the main ablation table in the paper:\n\n| **Experiment**                               | **Rank** | **Acc** | **R2** |\n|----------------------------------------------|--------:|-------:|------:|\n| base (feature and column name semantics)      | 1.24    | 76.9   | 72.4  |\n|                                              |         |        |       |\n| no feature semantics-OrdinalEncoder       | 4.27    | -2.7   | -4.8  |\n| no feature semantics-MinHashEncoder           | 3.49    | -1.7   | -3.3  |\n| no feature semantics-AutoGluonEncoder         | 5.76    | -4.8   | -7.3  |\n| no feature semantics-GapEncoder              | 4.12    | -1.4   | -4.3  |\n|                                              |         |        |       |\n| column semantics-add description          | **1.20**| 0.0    | 0.0   |\n| column semantics-drop columnnames             | 3.35    | -1.2   | -2.1  |\n\n\nReplacing LLM embeddings leads to a noticeable decline in performance, for example with accuracy dropping by ~3% and R2 score decreasing by ~5% when using ordinal encodings instead.\n\nWe also added win-ratio and p-values plots (two-sided Wilcoxon signed-rank test) to the paper and find that the base model including semantics has a win rate of 85% to 96% over the investigated ablated models with diminishingly small p-values (therefore, statistically significant). **The model hence successfully makes use of the semantic content** inherent to categorical/string features.\n\nArguably, the use of column name semantics offers only slight improvements (about 1% drop in accuracy and 2% drop in R2 when __not__ using the original column names), which are however statistically significant with a win-rate of 92% over the model not using column names and a diminishing p-value. Semantically enriching column names further boost performance, though the gain is not statistically significant (with a win rate of 57% and a p-value of 0.4).\n\nOverall, we demonstrate that **ConTextTab successfully integrates and makes use of semantics** present in the features and column names, further supporting our core claims.\n\n## Plateauing performance after 100k training tables\nWe acknowledge that this behavior is not clear. We observe a significant improvement as we scale the model from `mini` to `base` size as it is indicated in the ablation section. For larger models, a few factors might contribute to the observed performance: (1) the `large` model may have been undertrained due to the limited training duration, and (2) weight sharing across many layers could become suboptimal. Such weight sharing resembles RNN unrolled in depth and deep RNNs are known to be difficult to train. We also found that removing weight sharing entirely for large models leads to unstable training, suggesting that scaling up this architecture is challenging with the training data available. Finally, part of the limitation may come from the T4 dataset itself; if the dataset lacks sufficient diversity, a relatively small subset could capture most of the signal.\n\n## Novelty\nBy exhaustive evaluation and ablations, we show that our model is competitive across several established benchmarks and setting a new standard on semantically rich data, outperforming even best-in-class HPO-tuned trees and DL baselines such as TabM and RealMLP, while keeping the ICL paradigm. This is a novel and, in our opinion, a worthwhile contribution to the research field and community. While the core technical contributions of ConTextTab are relatively straight-forward, we do not consider this a weakness of our work. Moreover, certain architectural modifications, such as the clustering head designed to remove the restriction on the number of prediction classes and to incorporate target class semantics, are novel and have not yet been explored in the tabular domain.\n\n## Limitations\nWe briefly discuss potential data contamination and our contamination study on CARTE but happily extend this section for clarity. With the added semantic ablation study we believe to have addressed the remaining limitations you have outlined. Throughout we make sure not to overstate the model’s performance and clearly state it’s advantage being in semantically rich tasks and benchmarks as well as its performance dependence on dataset size (as discussed in the paper and further ablated in Figure 4 in the Appendix.\n\n## Conclusion\nWe hope to have addressed all your concerns and questions and are looking forward to the discussion!"}, {"review_id": "FbVlBsu4iv", "reviewer": "Reviewer_iSTq", "summary": "The authors introduce ConTextTab, a new tabular ICL model, following the ideas presented in TabPFN. The core differences are pre-training on real world data as well as integrating semantic understanding (column names, etc.) into the model.\nThe core training architecture of common ICL models (TabPFN, TabICL, Attic) is still implemented.", "questions": "- Please release model code during reviews in an anonymous format.\n- Could you outline the contributions of each task to the total number of datasets.\n  - Provide critical difference diagrams for each benchmark suite separately\n- Have you tried out different preprocessing strategies, especially for numerical, i.e. PLE or PLR?\n- Why do you actually sum the embeddings? Wouldn't at least averaging them be more secure as o avoid overly large values?\n\nIn section 3.2 you write that \"horizontal\" and \"vertical\" blocks have exactly the same weights. Could you elaborate? Do they share the weights?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength\n- Well written\n- Relevant and interesting research question addressing the problems of current ICL models to adequately be applied to real-world datasets\n- Benchmarks are very large\n\n# Weaknesses\n- Benchmarks are somewhat skewed. It seems as if ConTextTab is outperforming PFN only on the CARTE benchmark whereas TabPFN outperforms ConTextTab on all other benchmarks. \n  - I am not sure how  in Table 1 Naive in the CARTE benchmark can have an average rank > 11 when you compare 11 models? This somehow makes me question the ranks reported for CARTE since they are so different from all other tasks (even TabReD). And why does TabICL not have a rank if you have an accuracy?\n- While the benchmarks are quite large, including TabM [1], Mambular [2] and extending TabICL [3] to further datasets would have been nice.\n- Compute times for different models, especially for TabPFN, TabICL and ConTextTab are missing.\n- While mentioning the dataset sizes of TabPFN and TabICL as limitations, I did not find any information of whether ConTextTab can handle larger datasets any better than the two. The results on the TabReD dataset suggest otherwise.\n\n# General comment\n- Section 3.4 is somewhat poorly named. Currently I would read it as an alternative idea to your presented architecture.\n\n\n\n\n------\n\n[1] Gorishniy, Yury, Akim Kotelnikov, and Artem Babenko. \"TabM: Advancing tabular deep learning with parameter-efficient ensembling.\" The Thirteenth International Conference on Learning Representations.          \n[2] Thielmann, Anton Frederik, et al. \"Mambular: A sequential model for tabular deep learning.\" arXiv preprint arXiv:2408.06291 (2024).        \n[3] Qu, Jingang, et al. \"TabICL: A Tabular Foundation Model for In-Context Learning on Large Data.\" arXiv preprint arXiv:2502.05564 (2025).", "was_revised": true, "final_justification": "- The authors made a compelling case in their rebuttal.\n- Added extensive experiments showcasing the models advantages.\n- All concerns were effectively addressed.", "rebuttal": "Thank you for reading our manuscript and providing valuable feedback! We are happy that you acknowledge our research question to be both relevant and interesting, as well as our comprehensive evaluation and well-written manuscript!\n\nTo address your concerns and questions:\n\n## Contribution of each task to the overall evaluation\nThank you for the suggestion, **we have added CD diagrams together with win-ratio matrices and p-values for every benchmark** separately in the Appendix! This makes it much easier to assess benchmark-specific performance in addition to our main results table and strengthens our core claims!\n\n## Skewed benchmarks \nOur main claim is that ConTextTab successfully integrates semantic information in its prediction boosting performance on semantically rich tasks. Naturally, CARTE is our main evaluation benchmark, as the remaining benchmarks contain very little semantic features. Hence, our main observation is that we perform on-par to existing ICL approaches such as TabPFN on all benchmarks other than CARTE and outperform all models on CARTE. Indeed, this remains true with the greatly improved baselines (c.f. our answer to Reviewer qbCU). Using the newly added benchmark-specific win-ratio matrix and p-values we observe that TabPFN has a win-ratio over ConTextTab ranging from 49% to 63% (excluding CARTE), however with p-values ranging from 0.09 to 0.64. Hence, the performance difference on those benchmarks is statistically insignificant. **On CARTE, ConTextTab wins over TabPFN with 94%** and a diminishing p-value, hence is statistically significantly better than TabPFN in the presence of highly semantic features.\n\n## Benchmark inconsistencies\nThank you for pointing out the rank inconsistencies! Indeed, the Naïve rank on CARTE >11 is surprising at first, but actually correct due to a subtle nuance in the evaluation protocol: note that TabICL has partially incomplete evaluation, i.e. can be evaluated only for classification tasks. Therefore, we include it in the rankings on the classification portion of the benchmarks. Hence, for the ranking on CARTE in table 1, there are actually 12 candidates ranked, and Naïve ends up with 11.22. We then do not show the rank average of TabICL because it would only have been ranked on a subset of the tasks. We agree that this is confusing. Therefore, we have updated the tables to exclude partially evaluated models (TabICL, TabDPT, CARTE) from all ranking calculations which resolves the issue. You can find the updated results table in our answer to reviewer qbCU.\n\n\n## Including more baselines\n**We have added TabM** (HPO with 5-fold inner CV) and extended TabICL evaluation to all (classification) benchmarks (which was previously limited by a bug in TabICL). **We have also added Mambular** (v1.5.1), however only for CARTE and TALENT-Tiny as convergence on larger datasets was too slow to finish in time for the rebuttal. As suggested by Reviewer qbCU, **we also added tuned ensembled baselines via 5-fold inner CV**. Overall, we believe our baseline selection to be very comprehensive including all relevant conventional as well as state of the art baselines.\n\n## Runtime comparison\nThank you for this suggestion! We have added a corresponding evaluation. As requested by other reviewers as well, **we have included a runtime analysis**. In the following, we provide the answer consistent with our response to Rev. zDEh:\n\nWe have added a runtime comparison in a controlled fashion using synthetic tables with 100 columns and varying number of training rows ranging from 1k to 10k. We synthesize data with 50% categorical/textual and 50% numeric features. We created two plots with runtime (prediction and fit + prediction) for ConTextTab, TabICL, TabPFN, XGBoost, and RealMLP for various numbers of training/context rows and one test/query row, using 5 repeats and report averages. The experiments were conducted on an H100 GPU. We find that ConTextTab is comparable to other tabular ICL models such as TabPFN and TabICL – with runtimes ranging from 0.1s up to 8s depending on the table size. As expected, due to the LLM embedding overhead, ConTextTab is typically slightly slower than TabPFN or TabICL. For ConTextTab, we observe quadratic scaling behaviors as expected due to the use of transformer backbone with memory efficient attention, however for TabPFN and TabICL it is not as noticeable, likely due to the use of flash attention. This results in ConTextTab being slower by a factor of 2 for larger context with 10k rows. While non-ICL models offer significantly faster prediction times, they require extensive training. In particular, the hyperparameter-optimized (HPO) versions of XGBoost and RealMLP exhibit fit+prediction runtimes that are two orders of magnitude higher.\n\n## Handling of larger datasets\nWe point to Figure 4 in the appendix where we observe that ConTextTab, like other ICL methods, indeed struggles with very large datasets. Nevertheless, **it outperforms TabPFN also in the large-data domain**. ConTextTab handles large datasets by sampling 8k rows of context using 8-fold bagging by default which likely explains performance gains over TabPFN in this domain.\n\n## Model code\nAs per NeurIPS rebuttal policy, we cannot share links (including anonymized code) during the rebuttal. We have prepared the repository and will publish the code and model checkpoints upon publication.\n\n## Additional preprocessing strategies\nWe have experimented with the default standardization + linear encoding as well quantile-based binning, which are presented in the main ablation table. We have not investigated further numerical preprocessing/encoding variants, however, agree it is an interesting direction for future investigations!\n\n## Summation of embeddings\nApologies for the confusion: embeddings are normalized via `LayerNormalization`, which was lacking from the description. We have updated the corresponding paragraph, thanks for highlighting!\n\n## Weight sharing\nSorry, our original phrasing was unclear, and we have revised it. Our intention was to convey that the horizontal and vertical blocks use the same number of weights and are each followed by a feed-forward layer with an intermediate transformer block. This differs from TabPFNv2, where feed-forward layers are placed only after both attention blocks.\n\n## Conclusion\nWe hope to have addressed all your concerns and are looking forward to the discussion!"}, {"review_id": "zOxCPfrRpQ", "reviewer": "Reviewer_qbCU", "summary": "The paper introduces ConTextTab, a foundation model for tabular data using in-context learning that is pre-trained on real tables and has meta-learned how to handle not only numerical and categorical features but also text, dates, and column names. ConTextTab combines architectural innovations of different models together with mechanisms for handling embeddings of different column types. It is evaluated on multiple benchmarks with a diverse set of baselines and achieves strong results, particularly on the semantically rich CARTE benchmark, where it sets a new state-of-the art. Numerous ablations illustrate the effect of different design decisions.", "questions": "- Eq. (2): I find this part somewhat confusing. In TabICL, if I understand correctly, only the final MAB uses Q, so wouldn't it be MAB(C+Q, MAB(I, C))? A diagram as in TabICL might help to illustrate things. Also, in MAB(X, Y), it could be clarified that X are the queries and Y are the keys.\n- Figure 2: Why is the R2 score of CatBoost [HPO] negative on OML-CTR23?\n- How many parameters does the model have, and how does it compare to the other foundation models?\n- For the text embedding models, the authors might want to look into / cite https://arxiv.org/abs/2312.09634\n\n\nMinor comments:\n- l. 69: could mention TabM and ModernNCA as well\n- l. 109: \"then\" -> \"than\"\n- l. 292: It could be nice to state the default context size (8c = 7600 if I understand correctly)\n- l. 536: \"deault\" -> \"default\"\n- l. 580: \".\" missing", "limitations": "The paper states quite a few limitations. Besides the mentioned lack of (inner) cross-validation, there are some other concerns that affect tabular benchmarks in general and could be discussed, like dataset quality, quality of search spaces for baselines, etc.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Clarity: The paper is well-written and generally easy to follow. \n- Significance: The paper studies a practically important problem. It provides many interesting insights into the pre-training and architecture of tabular foundation models, therefore I expect that it will be significant for the development of future models. In particular, I appreciate the numerous ablations.\n- The paper includes a diverse set of established baselines.\n- The paper is not overselling too much. In particular, I appreciate that the paper includes multiple benchmarks, both textual and non-textual ones, which is not common for papers introducing textual models.\n\nWeaknesses:\n\n(1) The paper compares foundation models using training+validation data in the forward pass to non-foundation models that (mostly) use a single train-val holdout split (according to Appendix B.2). The classical methods would likely benefit significantly from inner cross-validation (e.g., 8-fold as in AutoGluon, but 5-fold is also common), while the foundation models would probably not benefit from it compared to the current approach. This is a major concern for my score. I see several approaches to address this, listed in order of decreasing quality: (a) Ideally the authors would run every of the non-foundation methods with cross-validation. However, I understand that this is expensive. (One way to illustrate the cost would be to actually report the training+inference costs of different models.) (b) If the authors want to keep the holdout validation, in my experience it would at least be more fair to compare to foundation models that only see the training set. (One might think that this is now unfair to the foundation models as they don't see the validation set at all, but other models can benefit a lot from cross-validation due to the ensembling, while foundation models are already ensembled.) This of course makes the comparison of foundation models to AutoGluon unfair, so it might be best to have a train+val and a train version of the foundation models. (c) Mention this point in the main paper and state it as a limitation.\n\n(2) The runtime (training+inference) of methods is not evaluated, making it unclear how fast ConTextTab is compared to other models.\n\n(3) Minor: Most results come without error bars.\n\n\nOriginality: The proposed method is partially a combination of aspects of different models (dataset from TabuLa, base architecture from TabPFNv2, etc.) with some new aspects in terms of the details and the handling of different datatypes. I see originality as neither a strength nor weakness of the paper.", "was_revised": true, "final_justification": "Overall, I really like the paper, and I am hesitant between scoring it a 5 or a 6.\n- Evaluation: The authors have addressed all of my concerns by running baselines with cross-validation, adding time experiments, adding more baselines (TabM), better search spaces, etc. I think the evaluation is strong, and it contains multiple benchmarks highlighting the strengths and weaknesses of the model. In the case of the CARTE benchmark, it is a bit unclear how to best preprocess strings for the baselines - the authors have taken a solid choice, but it is unclear from the literature what is the best choice.\n- The model is state-of-the-art on the CARTE benchmark by a margin. It is a bit worse than the baselines on purely numerical/categorical benchmarks, but I think compared to many other papers that wouldn't have included these benchmarks or would have used weaker baselines, this is still pretty strong.\n- I like that the paper includes many ablations on model design.", "rebuttal": "Thank you for your detailed response and suggestions! We appreciate you highlighting the clarity and significance of our works, as well as the thorough and diverse evaluation.\n\nTo address your concerns and questions:\n\n## Different usage of train-val splits in baselines and foundation models\n\nThis is a very valid point, thank you for pointing it out! Indeed, for the foundation models, we use the concatenated train+val split while the HPO variants (being presented with the same train+val portion) implicitly use a static holdout set under-the-hood. While we believe this to be common practice also in other works, we agree that the results can be improved upon.\n\nTo this end, **we extended the baselines to include variants with HPO with 5-fold inner CV ensembling** for all tree variants (XGB, LGBM, CatBoost) as well as RealMLP and the newly added TabM (as requested by Rev. iSTq). Furthermore, **we have updated the search spaces** to those recently proposed in TabArena [1] to **ensure best-in-class baseline performance**. We report the following results (sorry for the format but openreview wouldn't take our LaTeX table :) ).\n\n**Table 1:** Performance comparison across all evaluated benchmarks, depicting mean accuracy (Acc) for classification and soft-clipped version of $R^2$ score for regression tasks, in percent. Missing values, due to architectural limitations or failed evaluations, are denoted as -- and excluded from the rank calculations. Models are sorted according to their ranking on the CARTE benchmark.\n\n| Model Name                | All Rank | CARTE Rank | CARTE Acc | CARTE R2 | OML-CC18 Rank | OML-CC18 Acc | OML-CTR23 Rank | OML-CTR23 R2 | TabReD Rank | TabReD Acc | TabReD R2 | TALENT-Tiny Rank | TALENT-Tiny Acc | TALENT-Tiny R2 |\n|---------------------------|---------|-----------|----------|---------|---------------|--------------|----------------|--------------|-------------|------------|----------|-----------------|-----------------|----------------|\n| AutoGluon                 | –      | –        | 78.7     | 73.8    | –            | 88.5         | –             | 68.6         | –          | 86.0       | 64.6     | –              | 87.9            | 73.7           |\n| | | | | | | | | | | | | | | |\n| **ConTextTab**             | 2.74    | **1.47**  | **76.9** | **72.4**| 3.31          | 86.6         | 3.46           | 73.3         | 4.12        | 85.4       | 63.2     | 2.41            | 87.6            | 76.1           |\n| CatBoost [HPO, CV]         | 2.5     | 2.92      | 75.9     | 68.2    | 2.5           | 87.7         | 2.43           | 68.8         | 2.0         | 86.0       | 64.3     | **2.08**        | 87.2            | **76.4**       |\n| LightGBM [HPO, CV]         | 2.68    | 3.0       | 73.4     | 67.5    | 2.72          | 87.5         | 2.63           | 68.8         | **1.25**    | 86.0       | **64.9** | 2.54            | 86.9            | 74.1           |\n| RealMLP [HPO, CV]          | **2.34**| 3.18      | 73.4     | 67.5    | **2.14**      | **88.3**     | **1.66**       | 75.0         | 1.88        | 86.0       | 64.6     | 2.35            | **88.3**        | **76.4**       |\n| TabM [HPO, CV]             | 2.81    | 3.69      | 73.2     | 66.9    | 2.68          | 87.9         | 2.09           | 74.3         | 3.12        | **86.1**   | 64.2     | 2.49            | 87.5            | 75.7           |\n| XGBoost [HPO, CV]          | 2.93    | 3.78      | 73.1     | 66.7    | 2.6           | 87.5         | 2.89           | 70.9         | **1.25**    | 86.0       | **64.9** | 2.81            | 86.8            | 73.6           |\n| TabPFN [v2.1.0]            | 3.39    | 5.37      | 72.3     | 65.0    | 2.64          | 87.4         | 3.0            | **75.1**     | 4.25        | 85.6       | 63.8     | 2.3             | 87.8            | 74.4           |\n| HistGradBoost       | 4.86    | 5.94      | 72.5     | 64.8    | 3.96          | 86.1         | 6.37           | 65.5         | 2.38        | 85.9       | 63.9     | 4.22            | 86.3            | 67.6           |\n| Random forest              | 5.75    | 7.41      | 71.5     | 63.3    | 4.96          | 85.7         | 6.11           | 67.9         | 5.88        | 85.4       | 60.7     | 4.62            | 85.8            | 70.6           |\n| Naive                      | 9.56    | 10.0      | 53.0     | -1.8    | 9.33          | 47.0         | 9.69           | -8.4         | 8.88        | 80.8       | -0.6     | 9.41            | 53.4            | -19.2          |\n| | | | | | | | | | | | | | | |\n| CARTE [v0.0.26]             | –     | –       | 76.1     | 68.5    | –           | –          | –            | –          | –         | –        | –      | –             | –             | –            |\n| TabDPT [v1.1, k=2048]  | –     | –       | 72.7     | 65.1    | –           | 87.8         | –            | 73.2         | –         | –        | 60.9     | –             | 86.7            | 74.8           |\nMambular [v1.5.1]    | – | – | 70.1 | 52.8 | – | – | – | – | – | – | – | – | 83.6 | 53.9 |\n| TabICL [v0.1.1]         | –     | –       | 72.5     | –     | –           | 88.0         | –            | –          | –         | 85.1       | –      | –             | 87.4            | –            |\n\n\nIndeed, we observe a significant boost in performance for those baselines over the results previously reported using a fixed holdout set HPO, in line with the observations recently made in the literature [1].\nIn particular, ranking order now has changed for the non-semantic benchmarks, whereas the results remain conceptually unchanged in the case of the CARTE benchmark. That is, **ConTextTab sets a new standard on semantically rich datasets while performing on-par to existing approaches on non-semantic datasets**. We have also added more detailed per-benchmark analysis including win-ratios, p-values and CD diagrams, as highlighted and discussed in our answer to Rev. iSTq, further supporting our main claim. In particular, differences to TabPFN (updated to v2.1.0) on conventional benchmarks are statistically insignificant, while ConTextTab has a 94% win-rate (with a vanishingly small p-value, i.e. the difference is stat. significant) on CARTE. Moreover, we find ConTextTab does not perform significantly worse than the best-ranking model on all non-semantic benchmarks but OpenML-CTR23 (where it is however not significantly different to tuned ensembled trees).\n\nOverall, we believe the addition of CV-ensembled baselines improves our evaluation while our core claims remain unchanged.\n\n## Runtime evaluation\nThank you for this suggestion! As requested by other reviewers as well, **we have added a runtime analysis**. Please refer to our answer to Rev. zDEh for details and results.\n\n## Error bars\nIndeed, most results do not show confidence intervals or error bars. However, **we have added p-value matrices next to the win-ratio plots** depicted in Fig. 5 **as well as CD diagrams** in the appendix. Upon request by Rev. iSTq23, we now include this also per-benchmark, making the detailed assessment of ConTextTab performance in comparison to all evaluated baselines easier including statistical significance.\n\n## Confusion around Eq.(2)\nIn the paper, we now provided a detailed architecture diagram, similar to TabICL, to show that we concatenate the queries with the induced vectors, while the remaining MAB block inputs are formed by contexts concatenated with queries. While we experimented with architectures more closely aligned with the original ISAB block used in TabICL, these resulted in lower performance. Our chosen design facilitates the propagation of query information. In each MAB block, the query rows can attend to themselves while using the custom attention matrices.\n\n## Negative R2 scores\nR2 scores are zero for naïve predictors and can be arbitrarily negative. Indeed, there are a few severe outliers in some regression tasks that are so large that they even drag the mean to be negative, which is not great. To overcome this, **we have updated all evaluation to use a soft-clipped version of R2**, smoothly transforming all negatives scores from $[-\\infty, 0)$ to $[-1, 0)$ via $\\tanh$ (and keeping positive scores untouched). This retains the relative ordering even of scores worse than naïve while being smoothly differentiable and less distorting to the mean. Thank you for highlighting this nuance!\n\n\n## Number of parameters\nThe base version of our model ($n=12$, $d=768$, $d_{ff}=3072$), uses 172M parameters which reduces to 16M trainable parameters in the case of weight sharing. **We have clarified this in the paper.**\n\n## Text embeddings\nThank you for the interesting reference, **we added a citation in the paper**!\n\n## Minor comments\nThank you for catching those typos! Also, we’ve added TabM as a baseline and now include ModernNCA citation in the related works section. Finally, **we have added the previously lacking comment on the default context size** (which is 8k=8192)!\n\n## Conclusion\nWe hope to have addressed all your concerns and are looking forward to the discussion!\n\n### References\n[1]: Erickson, et al. \"TabArena: A Living Benchmark for Machine Learning on Tabular Data.\" arXiv preprint arXiv:2506.16791 (2025)."}]

Author rebuttal:

Meta-review: The paper provides an in context learning method for tabular data. The reviews unanimously agree that the paper is well written and easy to follow. They appreciated the thorough experiments conducted, demonstrating the value of the methods, .e.g. A8ss: “The experimental evaluation is comprehensive, covering 203 datasets across five different benchmarks with proper statistical analysis including critical difference diagrams.”. In addition to the promising bottom-line performance, the reviews mentioned the paper provides valuable core components: A8ss “The multi-modal embedding approach is sensible .. the architectural modifications to TabPFN are well-motivated”, zDEh “The work proposes architectural choices and training schemes that maybe useful for future related works.”. These comments suggest the method is not only practical but contains insights that would be interesting to the NeurIPS community.  

In terms of weaknesses, some reviews raised an issue of novelty, as some of the elements used in the method presented here are reused from other papers. In the discussions this is agreed to be a minor issue, as the provided model has sufficiently many novel aspects. Finally, the reviews raised the need for additional experiments, but these were provided in the rebuttal phase and all major concerns are mitigated.  

Integrating the rebuttal content into the paper seems quite simple to me and the resulting paper is agreed by all reviewers to be of high quality, and a good addition to NeurIPS.
Decision: Accept (spotlight)