TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields
Authors: Alan Arazi
Eilam Shapira
Roi Reichart
Venue: NeurIPS 2025 poster ()
Keywords: tabular data
tabular foundation model
deep learning
Primary area: deep_learning
TL;DR: We introduce TabSTAR: A Tabular Foundation Model with Semantically Target-Aware Representations, which achieves state-of-the-art results in tabular datasets with textual fields.

Abstract: While deep learning has achieved remarkable success across many domains, it has historically underperformed on tabular learning tasks, which remain dominated by gradient boosting decision trees. However, recent advancements are paving the way for Tabular Foundation Models, which can leverage real-world knowledge and generalize across diverse datasets, particularly when the data contains free-text. Although incorporating language model capabilities into tabular tasks has been explored, most existing methods utilize static, target-agnostic textual representations, limiting their effectiveness. We introduce TabSTAR: a Tabular Foundation Model with Semantically Target-Aware Representations. TabSTAR is designed to enable transfer learning on tabular data with textual features, with an architecture free of dataset-specific parameters. It unfreezes a pretrained text encoder and takes as input target tokens, which provide the model with the context needed to learn task-specific embeddings. TabSTAR achieves state-of-the-art performance for both medium- and large-sized datasets across known benchmarks of classification tasks with text features, and its pretraining phase exhibits scaling laws in the number of datasets, offering a pathway for further performance improvements.

TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields
Alan Arazi Eilam Shapira
Roi Reichart
falanarazi7, eilam. shapira, roireichart) @ggail.com com Technion IIT
Abstract
While deep learning has achieved remarkable success across many domains, it has historically underperformed on tabular learning tasks, which remain dominated by gradient boosting decision trees. However, recent advancements are paving the way for Tabular Foundation Models, which can leverage real-world knowledge and generalize across diverse datasets, particularly when the data contains free-text. Although incorporating language model capabilities into tabular tasks has been explored, most existing methods utilize static, target-agnostic textual representa- tions, limiting their effectiveness, We introduce TabSTAR: a Tabular Foundation Model with Semantically Target-Aware Representations. TabSTAR is designed enable transfer learning on tabular data with textual features, with an architecture free of -ataset-specifie parameters. It unfreezes pretrained text encoder and takes as input target tokens, which provide the model with the context needed to learn task-specific embeddings. TabSTAR achieves state-of-the-n performance for both medium- and large-sized datasets across known benchmarks of classification tasks with text features, and its pretraining phase exhibits scaling laws in the number of datasets, offering pathway for further performance improvements
Introduction
In recent years, deep learning has profoundly reshaped research and practice in computer vision 79
16 [3]. This transformation was notably accelcrated by the rise of foundation models [8, 88 capable of cross-moda understanding and generalization from massive pretraining across heterogencous data sources. Importantly, they enabled end-to-end approach that outperformed previous modular alternatives 1. Moreover, deep learning models excel at transfer learning (91), generalizing from their pretraining data to new tasks. Their strength, combined with techniques like In-Context Learning (ICL) [13] and Parameter-Effice Fine-Tuning (PEFT) [41], has enabled rapid adaptation to new tasks with only limited labeled data.
55 66 33 [7 and natural language processing [56
Despite this progress, deep learning has historically lagged behind gradient-booste decision trees (GBDTs) on tabular data 12,,15 60), in both classification and regression tasks 665 (5)). The heterogeneity of tabular data, which lacks the spatial locality of images the sequential order of text, makes it more challenging for deep models to learn. Consequently, GBDTs have remained the de facto standard for tabular learning, offering strong out-of-the-bor the-box performance, computational efficiency, and built-in inductive biases (e.g., robustness to skewed feature distributions and automatic feature selection) that make them especially well-suited to heterogeneous datasets [3I] Nonetheless, GBDTs cannot be pretrained to reuse strong representations for downstream tasks. This limitation becomes critical in low-data settings like those often found in healthcare applications
(52). Crucially, they must rely on external embedding models to process unstructured data types like text and images, yielding fixed feature representations that cannot be finetuned for specifie prediction task.
Code available
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: binary classification toy dataset for hospital patient release outcomes. Decision is the target variable. Age (numerical), Department (high- nighhcardinality), and Report (textual) are the features.
Age
Department
Report
Decision
45
Cardiology
Mild chest discomfort.
Released
62
Neurology
Complaints of headache and occasional dizziness.
Hospitalized Released
38
Oncology
Completed treatment cycle without adverse reactions. Reports episodes of vertigo and memory lapses.
55
Neurology
Hospitalized
The emerging field of Tabular Foundation Models (TFMs) has begun addressing these shortcomings, introducing powerful cross-dataset learning strategies 84 47 38
However, the flagship model TabPFN-v2 [38] still handles text inputs no more flexibly than conventional GBDTs. This design choice is not incidental; historically, tabular benchmarks have prioritized numerical datasets without free-text features, largely for ease of modeling and evaluation. A recent study 48 of mainstream
tabular datasets benchmarks [25 23 87 [55] found that half of these datasets are more than 20 years old, being a poor representation of modern real-world data.
Real-world tabular datasets often include nigh-cardinality and free-text features [4], illustrated by toy example in Table [ In such datasets, free-text features (e.g., Report) carry rich semantic information critical for tasks like predicting whether a patient will be discharged from the hospital or require continued care. Yet, most models encode them in target-agnostic manner, delegating to generic embedding that fails to capture task-specific nuances for predicting Decision. Crucially, that same embedding would have been used for a different target variable (e.g.. Treatment Cost). Similarly, categorical features with dozens of unique values (e.g.. Department) are difficult to encode efficiently without external knowledge, making naive approaches brittle and limiting generalization. Importantly, the column names, which could guide the model toward more effective representations, are typically ignored. Addressing these limitations is crucial for developing tabular models that leverage semantic information, transfer knowledge from many datasets, and generalize across domains.
In this paper, we introduce TabSTAR: a novel Tabular Foundation Model with Semantically Target- Aware Representations, designed explicitly for end-to-end handling of purely textual features. By integrating unfrozen text encoder at its core, TabSTAR optimize free-text feature repre- sentations, demonstrating their clear superiority over alternative frozen embedding approaches. Additionally, introduces a novel approach of target-aware tokens, which inject semantic infor- mation about the target variable as part of the input, allowing for efficient parameter sharing and resulting in an architecture with no dataset-specifie parameters (see Figure [ TabSTAR's training is highly efficien and its performance steadily improves with more pretraining data. Empirically, TabSTAR achieves state-of-the-arn (SOTA) performance on classification datasets containing textual features, surpassing leading TFMs as well as GBDTss tuned for 4 hours.
2 Related Work
This section reviews prior work in supervised tabular learning. We begin with deep learning methods tailored for tabular data, which were applied to a single dataset. We then discuss cross-dataset transfer learning techniques that improve generalization by leveraging related datasets. Next, we cover the field of TFMs, which aim to generalize across diverse tasks and datasets through large-scale pretraining. Finally, we review recent work on applying large language models (LLMs) to tabular data and elaborate on existing AutoML (34 multimodal solutions. As we focus on supervised learning, we do not cover self-supervised methods [82 and their downstream applications.
Deep Learning on a Single Tabular Dataset Several architectures have been proposed to enhance deep learning for tabular data 770 45
8T 85). TabNer [3] and TabTransformer 4g] introduced
attention mechanisms into tabular deep learning, while FT- FTaarfsffrrmrr (26) and its improvement (27] jointly integrated numerical and categorical features into a transformer (79]. Other novel
approaches leveraged inter-example information at inference time, with SAINT' 69 proposing row-
2 Highh-crdinality features are categorical columns with a large number of unique values. PPeetraiing within 48 hours single A40 GPU. Finetuning with PEFT for low memory footprint.

---

level attention between examples. Pon-Paraittric Transformers 49
processing the entire dataset,
including labels, in a single forward pass, and TabR [28] combining a neiihbrt mechanism with traditional Bulti-LLayrr Perceptron (MLP) architecture. Recent works such as TabM [29] and RealMIP (39] focused on refining MLPs without an attention component. Despite these innovations, single-dataset deep learning models have not yet convincingly outperformed GBDT's [65] 31, 64].
Furthermore, none of them addressed the challenge of modeling tabular data with rich textual features.
Cross-Datase Transfer Learning Deep learning has been proven to shine when performing transfer learning in many machine learning domains [91]. Motivated by this success, [52] 89) proved that cross-dataset learning can boost single-dataset performance, but were limited to strict require- ments such as partial overlap of feature names. To address this limitation, TransTab [3] integrated
semantic understanding into feature tokenization, and XTab [90] pretrained transformer backbone with dataset-specific parameters, proving that pretraining contributes to a stronger initialization for a downstream task. Despite their small scale, these studies demonstrated cross-dataset transfer learning's ' potential, laying essential groundwork for the rise of TFMs.
Tabular Foundation Models TFMs represent an emerging paradigm in tabular learning. While the definition is still evolving, we adopt the framing proposed by (77), which identifies key desired characteristies of TFMs: large-scale pretraining with adaptability to downstream tasks, mixed-type column support, cross-domain generalization, use of textual metadata and column-order invariance.
TabPFN (36] is recognized as the first TFM, and its successor TabPFN-v2 (38] currently sets the SOTA in tabular learning, becoming a popular approach for TFMs [61 21 54]. TabPFN-v2 was the first model to consistently outperform GBDTs
medium-sizen datasets. by pretraining Bayesian Prior-Data Fitted Networks (PFNs) [58] on 130 million synthetic datasets. Using ICL at inference time, it accepts up 10,000 examples input and predicts without updating its weights. TabPFN-v2 inspired TabICL [61] which improved scalability, and TabDPT (54), trained on real data. However, they all rely on off-the-shelf embeddings for text features like GBDT', limiting their performance.
CM2 186) CARTE (47], and TP-BERTa (84) represent a shift toward semantic tabular modeling, leveraging textual signals and external knowledge at greater scale. Unlike prior methods, these models transfer knowledge via language representations. CM2 was pretrained on over 2,000 datasets, but did not focus on free-text features and used static word embeddings without further finetuning them. CARTE encodes tables as star-shaped graphs, jointly representing features by their names and values, and applies attention over the graph to capture contextual relations. While effective for high-cardinality features, it lacks exposure to longer free-text fields during pretraining and was proven useful mainly for small datasets. TP-BERTa adapts RoBERTa (53) with intra-feature attention and tokenization scheme that maps numerical values into discrete relative-magnn magnitude bins, to address the weakness of language models when tokenizing numbers 753. Although it performs well, its use of dataset-specific output layers limits scalability and complicates multi-task learning. Consequently, they trained two separate models 5 wastinn potential for better cross-dataset learning. Notably, none of these approaches finetune semantic representations during downstream task training. In our work, we demonstrate that this critical align textual and tabular features.
Large Language Models for Tabular Data The remarkable success of LLMs is unprecedented
13 [9]. During the past years, several research attempts have tried combine LLMs and tabular data. One line of work focuses on using LLMs directly for tabular prediction by converting tabular
data into serialized text. TabLLM 335 assessed LLMs under few-shot scenarios, while Tabula-8h
24 finetuned the Llama 3-8B model extensively on tabular data. Although useful for few-shot learning, these models are computationally expensive expensive[Suuboptimll suboptimal for numerical features 775] 77).
and potentially compromised on widely-used benchmarks due to prior exposure during training 9 While current generations of LLMs weren't 't adopted for tabular learning, their emergent knowledge from their pretraining could be crucial when textual features are present [77 20). Additionally, LLMSs
can be used multiple aspects of tabular learning, as they seem to be promising synthetic data generators
10 68
useful data cleaners 77 and clever feature engineers (37
*Contextual information such as the dataset description, column names, and category names. 5 *One for classification and one for regression. A joint model for both tasks performed significantly worse. 'Laaa 3-8b has orders of magnitude more parameters than TP- TERBE, which has roughly 110M parameters.
3

---

Age: 40-50
Numerical Encoder
Stacking
Numerical Fusion
Report: [Free Text]
Interaction Encoder
Decision: Released
Textual
Prediction Head
Encoder
Hospitalized
Decision: Hospitalized
Verbalization
Encoding
Fusion
Interaction
Prediction
Figure 1: The TabSTAR architecture illustrated with our toy dataset. The model processes numerical features, textual features, and all possible target values for classification.
Multimodal AutoML Historically, textual tabular datasets have been largely overlooked in classical tabular benchmarks. However, the AutoML (34) community has made significant progress in developing multimodal solutions. In particular, AutoGluon [19] introduced the AutoML Multimodal Benchmark [63], initially focusing on text features and later evolving into AatoGluun-Muutimmede Multimodal which incorporates images as well. This powerful AutoML framework can fuse text and image foundation models with tabular models and ensemble multiple models through meta-learning approach [22). making it one of the few systems able to refine static textual representations via joint learning- Nevertheless, this line of work should not be seen single model but rather highly optimized, production-ready system. According to the authors, it is "a collection of tricks that significantly enhance performance" [73), establishing itself robust approach for multimodal, multi-model tabular learning. However, this line of work remains somewhat orthogonal to the development of novel TFMs.
73 (2).
TabSTAR
In this section, introduce TabSTAR: Tabular Foundation Model with Semantically Twrget-Aware Representations. Our training framework consists of two stages: (1) Pretraining, where the model pretrained over corpus of tabular datasets in multi-task regime, mixing classification with regression tasks, then (2) Finetuning, where the pretrained model is further trained with LoRA [2] on single downstream task. TabSTAR designed enable effective cross-datased learning by applying supervised learning on the target variable in both stages. At its core, it uses an unfrozen encoder-only language model, which can potentially invoke world knowledge acquired during the language model pretraining [ The encoder is combined with a tabular-specific architecture tailored to
structured data, mitigatinn the known limitations of language models in tabular settings 77 77
TabSTAR's architecture comprises five core modules: (1) Verbalization, mapping every feature into textual representation composed of both the column name and value, with a special treatment to numerical features for full numerical precision; (2) Encoding, transforming semantic and numerical inputs into meaningful embeddings of the same dimension; (3) Fusion, integrating textual and numerical representations on each feature independently; (4) Interaction, modeling dependencies and relationships between different features through feature self attentitn;;on; and (5) Prediction, where outputs are projected into real value for regression or probability distribution for classification. Figure T illustrates the architecture, Appendix A elaborates it, and Appendix B discusses the training.
A key innovation of TabSTAR is the introduction of target-aware tokens, a novel approach that integrates the target variable's is identity
input the model. Unlike existing TFMs [8 47 84]
86 [0] which treat the target value as mere label, TabSTAR fuses target-awareness from the very beginning. For classification tasks, each target value is verbalized and encoded like any other feature. Then, features and target tokens interact with each other, building representations that are then used for prediction. Crucially, this target-awareness allows parameter sharing between all target tokens, which can later use a shared prediction head that maps tokens to probabilities regardless of the number of classes and their identity. By doing so, TabSTAR eliminates the need for dataset- specific components commonly found in prior work [26] 90 84). TabSTAR's 's flexible architecture effortlessly scales to any dataset size, and handles any number of classes in multiclass classification tasks.
'Ranging from metadata-rich, text-heavy datasets to numeric-only tables lacking column names. $ 'Note that the language-mote model pretraining occurs before TabSTAR's pretraining. Unless specified differently, the term pretraining refers to TahSTAR's 's pretraining, which assumes the use of pretrained language model. Except when the number of features becomes very large, where memory limitations may arise.

---

Table 2: An illustrative verbalization of the first patient of Table Each semantic feature verbalized with its name and value. The numerical Age value 45 is standardized (mapped into z-scores, e.g., 0.27) and binned (providing a range to the verbalization, egg. 40-50, and its quantile). The target variable Decision is mapped into its two possible elements, regardless of its original true value.
Name
Value
Semantic
Numerical
Age
45
"Age: 40-50 (Quantile 50-60%)"
0.27
Department
Cardiology
"Department: Cardiology"
Report
Mild chest discomfort.
"Report:
Mild chest discomfort."
Decision
Hospitalized
"Target. Decision: Hospitalized"
Decision
Released
"Target. Decision: Released"
Verbalization All the features and each of the target values are processed into sequence of elements. Numerical features are processed into two inputs: numerical one and a semantic one. The numerical input is standardized using z-scores, with outlier clipping at #3 standard deviations. In addition, they are verbalized using quantile discretization into 10 bins, novel approach to mitigate the precision loss inherent in language models (75). Appendix A.] shows precise example and g diseusses different verbalization strategies. In contrast, semantic features are directly verbalized by concatenating the feature name and textual value, without any numerical representation. The target variable is also included as part of the input: In classification tasks, each of the C' possible values is represented by element, constant for every example, while the true value remains hidden. For regression tasks, single element is verbalized, carrying only the target name. Table I shows a toy dataset of patient records and outcomes and Table 2
shows the verbalization for the first patient.
Encoding We employ a pretrained e5-small-r2 80 embedding model for semantic encoding, chosen for its strong performance on the MTEB benchmark (57] with a relatively modest parameter count. By unfreezing the upper half of its layers, the representations are optimized for predicting the target variable, which leads to a significant impact on TabSTAR performance (see $66). Each verbalization element is encoded independently into a semantic representation, with attention applied between tokens within each sequence element. In parallel, we encode standardized numerical values by projecting them into the same dimension using small MLP. For the patient in Table this results in a numerical embedding for Age alongside semantic representations for each of the 5 verbalizations.
Fusion To obtain unified representation for each sequence element, we apply a fusion block consisting of single encoder-only transformer layer. Crucially, each numerical feature is fused independently, as the block attends only to its numerical and semantic embeddings. In our running example, the representation of Age now jointly captures both its semantic context (the fact that the value represents age) as well as numerical value (the patient's ss age, 45, or 0.27 after standardization).
Interaction The fused, semantically-riet and mamnercally-groundeee representations of all elements interact via a 6-layer Transformer encoder [79]. Each input element is now a token, with feature tokens and target tokens all attending to each other. Unlike standard language models, which integrate positional encoding, the Interaction module's 's inputs are order-invariant, a desideratum for TFMs, as defined by 777. The encoder produces contextualized representations for each target value. In our example, this yields dedicated embeddings for the Release and Hospitalization target values. The role of these representations is to carry information about how likely each value is to be the true value.
Prediction TabSTAR is designed for cross-dataset learning, with shared regression and classifi- cation heads used during both pretraining and finetuning. For classification, each of the C' target tokens processed independently through the same classification head, which projects them scores, We then apply a softmax over all the possible values to yield a probability distribution. Crucially, the fact that target tokens for every class in every dataset share the same classification head allows efficient parameter sharing, flexibly supports any number of output classes, and removes any need for dataset-specific parameters. This is not only efficient during pretraining, but also provides better initialization for finetuning. In our example, both the Released and Hospitalized tokens go through the same classification head, which maps them from representations to logits. Applying softmax yields predicted probabilities. For regression tasks, single target token is projected into real value.
5

---

4 Experiments
The TabSTAR Pretraining Corpus While TabSTAR could be pretrained on a massive scale, for this work we limit ourselves to a modest pretraining corpus focusing on classification, as we believe that TabSTAR's 's inductive biases are best suited to shine in this task. We manually curate a pretraining corpus of 350 high-quality tabular datasets (253 classification, 97 regression), in a tedious process in which we uncover numerous duplications in the most popular tabular repositories, OpenML 78 and Kaggle as elaborated by [76]. We begin by sourcing datasets from popular benchmarks [25 47 23 [2 63 55 [2], but observe that the presence of textual tabular datasets in them very rare. Thus, we furthermore expand our corpus, focusing on classification datasets with rich semantic content. See Appendix for more details.
Benchmark Tabular datasets with free-text have seen little prior research, and accordingly, bench- marks are rare. To address this, we compile all available datasets from three sources: (1) the AutoML Multimodal Benchmark [63] (2) the CARTE paper [47, and (3) the analysis of free-text and high- cardinality features by [2). After deduplication, our final benchmark includes 50 datasets. Despite its breadth, this collection has two key limitations: First, the benchmark is heavily skewed towards regression tasks, with 36 datasets. Secondly, 29 out of these 36 datasets were solely contributed by the CARTE benchmark, which focuses more heavily on high-cardinality features, rather than longer texts, as it was pretrained over knowledge graphs. While our main motivation is classification tasks with textual features, we decide nevertheless to evaluate the full set of 50 datasets, although it is biased toward regression problems and high-cardinality features (see Appendix D)
Baselines
We compare TabSTAR against GBDTs: Random Forest 12). LightGBM 46 XGBoost 15 and CatBoost (60; MLPs: RealMLP (3 and TabM; and TFMs: 29, CARTE TabDPT
TabPFN-v2 38) and TabICL [6]] which only supports classification tasks. For several model IT
we consider both default variants as well as tuned ones, where hyperparameters are optimized
separately for each task using random search with 5-fold cross-validation under 4-hour budget 8 CPU cores. Since the public TabPFN-v2 model does not support text, we use their closed-sourced API client 12 For models lacking native support for textual features, we embed text using e3-small-r2 80 allowing a fair comparison. For more details about the hyperparameters for each baseline as well as exclusion of baselines due to potential leakage concerns, see Appendix E
Experimental Setup Each of the 50 datasets in the benchmark is evaluated with 10 random train- test splits (90% training, 10% testing), resulting in 500 runs per model. While 30 of the datasets have more than 10,000 examples, most TFMs can't effectively scale beyond it: TabPFN-v2, for example, employs ICL and thus receives as imput at most 10,000 examples. While CARTE imposes no strict size cap, its tuning is slow, inefficient, and impractical for larger datasets [3 Because of these important limitations, we consider two experiment conditions: (1) 10K: Each model is trained (4 over at most 10,000 training examples, and (2) Unlimited: We evaluate TabSTAR and the most competitive, scalable baselines on the full version of the 30 datasets [55
The TabSTAR Training To maximize the value of cross-dataset learning, instead of pretraining TabSTAR once, we create five dataset splits. Each variant is pretrained on the 350 pretraining datasets and 40 of the benchmark datasets, while the other 10 serve exclusively as its test set. Crucially, the whole collection was carefully curated to prevent any data leakage from duplicate or overly similar datasets. As result, each dataset is evaluated by finetuning the single pretrained variant, which excludes it from its pretraining. For finetuning. while dataset-specific byperparameter tuning can boost performance, evaluation. Therefore, we use a default hyperparameter configuration that was found robust over a disjoint set of tabular datasets, detailed in Appendix B.2
believe that robust out-of-the-bee defaults are essential for TFMs and their
10 httpps:/www.. httpp://wwwkkkggl...m www.kagglc.coo/daaaaeeeee com/datasets
All MLPs and GBDTs, except the naive Random Forest baseline, used as weak baseline. httpp://githuubb. https://giihub..oo/PPP aattoooaabsstaffn--ciiee
In their own paper, CARTE was evaluated only over up 2,048 examples, without scaling guarantees. Whhile (CLeb TFMs aren't 't technically trained, we adopt this term for conciseness.
We technically cap the amount of examples to 100,000 for computational efficiency.

---

Results
We evaluate each model using AUROC (classification) and R² (regression) as metries. Following 38) we normalize scores per dataset split to the [0, 1] range, using the best and worst model performance anchors T6 The normalized scores are averaged across all runs, with 95% Cls. Performance for all models on both conditions shown in F (classificaain and Figure 3 certain datasets, we unable to execute TabPFN-v2, CARTE, and TabDPT due model-specific implementation issues or scalability constraints. Reported averages for these models are computed only over the datasets where evaluation is feasible. Appendix F.1 elaborates technical limitations, Appendix on dataset-level performance, and Appendix on head-to-head comparisons.
(regression). For
Classification
10K examples (14 datasets)
Classification Above 10K examples (10 datasets)
TabSTAR
TabSTAR
TabM
TabM
CatBoost
CatBoost
TabPFN-v2
XGBoost
XGBoost
RealMLP
TabiCL
Rea|MLP
TabiCL
LightGBM
L
10K
TabDPT
Default
TabPFN-v2
Unlimited
CARTE
Default
TabDPT
10K Unlimited
RandomForest
Tuned
CARTE
0.0 0.1 0.2 0.3 0.5 0.6 0.7
0.8 0.9
0.0 0.1 0.2 0.3 0.4 0.5
0.7
0.8 0.9 1.0
Normalized Score
Normalized Score
Figure 2: Comparison of normalized scores with 95% CIs between TabSTAR and baseline models in classification tasks, evaluated on up 10,000 examples (left) and above 10,000 (right).
In classification problems, TabSTAR consistently achieves SOTA performance. This is evident both when restricting the dataset size to 10,000 examples and when using larger datasets in the unlimited condition. For the 10K condition, TabSTAR achieves a 0.83 score, followed by TabM-Tuned (0.73), CatBoost- -anoot (0.69), and TabPFN-v2 (0.67). When analyzing head-to-head comparisons (see Appendix F.3). TabSTAR outperforms TabPFN-v2 (8/11 datasets), Tabe-Tuned ( (10/14), and CatBoost-Tunet (12/14). For the unlimited condition, TabSTAR -nllmiitd achieves 0.84 score, followed by TabM-Tuned-Unlimice (0.79), and much above the rest. Importantly, unlimited variants significantly surpass the 10K ones, emphasizing the importance of models with no scaling restrictions.
Regression Up to 10K examples (36 datasets)
Regression Above 10K examples (20 datasets)
CatBoost
CalBoost
XGBoost
TabM
CARTE
XGBoost
TabM
RealMLP
TabPFN-v2
LightGBM
TabSTAR
Rea|MLP
LightGBM
Unlimited
10K
TabSTAR
Default
CARTE
10K Unlimited
RandomForest
Tuned
TabPFN-v2
0.0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
0.0 0.1 0.2
0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Normalized Score
Normalized Score
Figure 3: Comparison of normalized scores with 95% CIs between TabSTAR and baseline models in regression tasks, evaluated on up to 10,000 examples (left) and above 10,000 (right).
Although regression is not our main focus, TabSTAR achieves competitive results in the 10K condition, but clearly does not the SOTA. Surprisingly, while TabPFN-v2 is superior, significantly underperforms compared to GBDTs which dominate this category. This emphasizes the need for better modeling of textual tabular learning, especially since TabPFN-v2 has shown remarkable performance in non-textual tabular datasets, and CARTE set the SOTA for small datasets. In the unlimited setting. TabSTAR scales well and surpasses other TFMs which cannot scale, but the gap from GBDTs remains significant. discusses this limitation and suggests promising directions for future generations of TabSTAR to achieve SOTA regression as well.
For single run, the best model gets the worst gets 0. and the rest are linearly scaled accordingly.
7

---

Table Cost Analysis. Median training and inference times and peak memory usage on GPU and CPU, aggregated over the 50 datasets of the benchmark with up to 10,000 examples.
Train GPU (GB)
Inference GPU (GB)
Model
Time (s)
CPU (GB)
Time (s)
CPU (GB)
CatBoost- CPU
360.8
2.2
34.0
2.1
CatBoost
68.5
1.3
1.5
2.0
1.3
1.5
LightGBM
39.1
1.3
1.5
2.0
1.5
RandomForest
86.1
1.3
1.5
2.5
1.3
1.5
ReaLMLP
136.4
1.4
1.7
2.5
1.3
1.7
TabDPT
10,4
2.2
1.8
161.4
33.8
6.0
TabICL
42.6
1.2
1.7
13.4
25.4
2.1
TabM
19.1
1.8
4.4
1.5
1.8
4.3
TabSTAR
493.2
4.7
1.8
3.0
1.8
XGBoost
38.4
1.3
1.5
2.5
1.3
1.5
Cost Analysis reporrs the computational requirements of TabSTAR and competing base- lines, measured by runtime and peak memory usage, for both training and inference. To highlight the importance of GPU acceleration when text embeddings are part of the preprocessing, we include variant of CatBoost restricted to CPU only. We exclude TabPFN-v2. whose performance probably comparable to TabICL, since it is only accessible via an API, and CARTE, whose extensive tuning requirements and prohibitive training costs forced evaluation on less capable hardware.
We find that TabSTAR incurs modest inference costs, comparable to GPU-accelerated GBDTs, since GPU processing of text embeddings dominates runtime. This is evident from the CatBoost- CPU variant, whose inference is nearly ten times slower than TabSTAR. Moreover, TabICL and TabDPT are orders of magnitude slower and consume far more memory, emphasizing the scalability limitations for ICL-based models. During training. TabSTAR is considerably slower than the baselines. However, while fitting a single GBDT much faster, this advantage quickly diminishes in practice when hyperparameter tuning is needed or GPU acceleration for text embeddings is not available. Appendix F44 provides dataset-level runtimes and analyzes costs as a function of the feature count.
Analysis
analyze the factors contributing to TabSTAR's s strong performance by addressing three key research questions: Q1: How important is the encoder language model unfreezing? Q2: Does the number of datasets during pretraining contribute to the downstream task performance? and Q3: How do different verbalization methods of numerical features impact performance?
To answer these questions, we pretrain several variants of TabSTAR for each analysis. limiting ourselves to subset of the tabular datasets used for the main experiment (see $4). Specifically, each variant pretrained over 256 dataset iinluuiig 30 datasets from our benchmark, and evaluated over the remaining 20 datasets (12 regression, 8 classification). This reduced setup allows leveraging transfer learning and exploiting our corpus, without the burden of training multiple folds per variant. Appendix G.I lists the 20 datasets used for evaluation along with per-dataset results.
Q1: The Role of the Encoder Unfreezing We examine the effect of unfreezing different numbers of textual encoder layers during pretraining. finetuning, we keep all base layers frozen and instead apply LoRA adapters to the same layers that were unfrozen in pretraining 18] Figure shows the validation loss during TabSTAR pretraining (left) and the normalized score the downstream tasks (right) as a function of the number of unfrozen encoder layers. Notably, unfreezing even a single encoder layer significantly outperforms using static embeddings. Further substantial improvements are observed as more layers are tuned, with the best results achieved when unfreezing layers. While unfreezing layers shows lower performance, is plausible that adding more datasets the pretraining phase will affect this finding. See Appendix G.2 for more details.
17 Except for variants Q2, which analyze effect the number datasets on pretraining. For simplicity, will refer to them unfrozen differentiate them from layers without LoRA.

---

Unfrozen Layers
+
Unfrozen
Validation Loss
1.0
0
1
1 Unfrozen
3
0.8
6
3 Unfrozen
9
Unfrozen
0.6
Unfrozen
0
20
40
0.00
0.25
0.50
0.75
1.00
Epochs
Average Score
Figure 4: Performance as a function of the number of encoder layers unfrozen: Validation loss during TabSTAR's pretraining (left) and normalized scores with 95% CIs on the downstream tasks (right). Unfreezing even single encoder layer significantly improves the performance of TabSTAR.
Q2: The Effecc of Pretraining To evaluate the impact of pretraining on TabSTAR's downstream performance, we compare a pretrained version of TabSTAR with a version that was finetuned from scratch 19 In line with previous work (90 86]. the pretrained model performs significantly better, highlighting the critical role of transfer learning for TabSTAR's success. To further investigate the effect of the number of pretraining datasets on downstream task performance, we train two additional versions: one pretrained on 16 datasets and another on 64 datasets.
Classification
Regression
0.90
0.80
0.88
0.78
AUROC
0.86
0.84
0.74
0.82
0.72
0.80
0.70
16
64
256
16
64
256
# Pretraining datasets
# Pretraining datasets
Figure 5: Average performance on downstream tasks as function of the number of pretraining datasets (in log scale). We use AUROC for classification (left), and H² for regression (right).
As shown in Figure 5 increasing the number of pretraining datasets consistently improved perfor- mance in both classification and regression tasks. Notably, the substantial gain in regression tasks suggests that TabSTAR's downstream performance on could improve with more pretraining data, potentially reaching SOTA performance with enough scale. See Appendix G.3 for more details.
Q3: Numerical Verbalization A key challenge in integrating language models with numerical data is determining how best represent numerical values within linguistic framework. While some semantic tabular methods omit numerical features from the verbalization X3 86 TP-BERTa
84 introduced Relative Magnitude Tokenization [4 which encodes numerical information through non-semantic special bin tokens. In contrast, TabSTAR injects semantic numerical information into the verbalization of numerical features, as illustrated Table To quantify the effect of our novel verbalization, we explore two thinner variants: (1) Name + Bin, which excludes the quantile information, and (2) Name, which omits numeric information entirely and verbalizes the feature name only. Appendix G.4 shows an illustrative example for each variant and presents the full results. As demonstrated Table 4 our findings reveal that incorporating numerical information significantly enhances performance, highlighting the importance of balancing numerical precision with representation format that aligns with the language model's parametric knowledge.
Since LoRA underperforms random weights, finetune the entire non non- pretrained model.

---

Table Normalized score with 95% Cls by the numerical verbalization method.
Verbalization Method
Name
Name + Bin
TabSTAR
Classification
0.386 0.095
0.544 0.093
0.593 = 0.097
Regression
0.386 = 0.081
0.584 + 0.076
0.596 + 0.079
7 Discussion and Conclusion
We introduce TabSTAR, Tabular Foundation Model with Semantically Target-Aware Representa- tions, which integrates textual features through an unfrozen pretrained encoder. In addition, its novel target-aware tokens enable efficient cross-datases generalization without dataset-specific parameters. Despite limited pretraining data and a relatively small text encoder [80]. TabSTAR sets the SOTA in tabular classification with textual features, significantly surpassing GBDTss and leading TFMs.
Since scaling laws in data and model size have proven themselves for LLMs [44 and TabSTAR improves with the number of pretraining datasets (see S6)). future work should scale TabSTAR across both model and data dimensions. For model scaling, we envision a family of model sizes, common for LLMs (30 74 (5), that will allow trade-off between quality and costs. Data scaling might leverage self-supervised learning [62] 82 over large-scale table corpora [18), or realistic synthetic tabular data generators [10]. which have proven successful [38] 2]. At scale, it could potentially
unlock few-shot learning capabilities and develop automatic feature- cnatureennginering skills (37].
Beyond scaling, TabSTAR's ' semantic approach has tremendous potential to explicitly include world knowledge by leveraging L.Lss, which to date have had a limited impact on tabular learning. As a few motivating examples, LLMs could improve TabSTAR's numerical verbalization binning approach by providing semantically informed thresholds, or by providing explicit world knowledge that could be injected as strong prior in small data scenarios. While these directions seem like plausible research paths, they come with a risk of data leakage due to the memorization properties of LLMs (9] Evaluating TFMs fairly while keeping benchmarks uncontaminated would be an important enabler for tabular research. As step in this direction, we are releasing several TabSTAR variants, each with different dataset withheld during pretraining, ensuring that for every dataset there is a TabSTAR model that has never seen it. We urge fellow researchers to adopt this approach in their own work.
While TabSTAR sets new bar in classification, its regression results lag behind GBDT', which outperform other TFMs as well. This gap could be narrowed through additional scaling, by exploring regression-via- classlassiii techniques like [38 21, and by enriching the numerical encoder with distribution-aware statistics per feature as done by [61] In addition, similar to other TFMs, TabSTAR may encounter memory bottlenecks on datasets with hundreds of features, and its training speed lags behind untuned GBDT's and ICL-based TFMs. Yet TabSTAR achieves GBDT-level efficiency during inference, as opposed TabPFN-v2 and TabICL. Furthermore, TabSTAR has not been extensively evaluated in few-shot scenarios and in purely numerical datasets.
Despite these limitations, TabSTAR offers a promising pathway toward improving performance on tabular datasets with textual fields, common in industries with high social impact (e.g.. healthcare, education), or with significant economic value (e.g., banking, manufacturing). In addition, its architecture lends itself to multimodality and could be extended to tabular datasets that combine numerical, textual, and image features. We believe TabSTAR paves the way for new generation of semantically enriched, Multimodal TFMs, and invite the research community to advance this vision.
Acknowledgments and Disclosure of Funding
Roi Reichart and Eilam Shapira have been partially supported by VATAT grant data science.
We thank Omri Feldman for brainstorming since the very beginning; Elad Hoffer and Ofir Lindenbaum for consulting and feedback; David Holzmüller, Lennart Purucker, Myung Kim, and Gaël Varoquaux for assisting with evaluations and benchmarks; and Frank Hutter, Noah Hollmann, Léo Grinsztajn, and the rest of the Prior Labs team for providing extensive access to TabPFN-v2' API version.
10

---

References
[1] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, and et al. Deep Speech End-to-End Speech Recognition in English and Mandarin. In Proceedings of The 33rd International Conference on Machine Learning, pages 173-182. PMLR, June 2016. URL
nttps://prooceddngs.mt mlr wwwps/ees..... hwwr///mmooe16....m html ISSN: 1938-7228.
[2] Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shub- ham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and Bernie Wang. Chronos: Learning the Language of Time Series. Transactions on Machine Learning Research, May 2024. ISSN 2835-8856. URL BtLLLttts://ppeeeviiiee wtwpiiewweeeeeeeeeeeeeeeeeeeeeee http://e....etooouu//ee/eeeeroo
[3] Sercan Ô Arik and Tomas Pfister. TabNet: Attentive Interpretable Tabular Learning. Proceedings of the AAAI Conference on Artificial Infelligence, 35(8):6679 May 2021. ISSN 2374- 3468. doi: (16699/aai..335it 16826. URI. httpt:ttts:ssssaaae aaaii.i..aaaaia........./....... org.aai..oooaaaooooaaa org/ its//indxx.... php/ AAAI/
@rticle/view/1GBB Number:
[4] Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundation Mod- els Defining a New Era in Vision: A Survey and Outlook. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 47(4):2245-2264, April 2025. ISSN 1939-3539, URL https::/iseexpplors ieee. arg.ebtrac///t////////oo
doi: 10. 10.1109/PAMM22224335500
document/10834499
[5] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Machine Translation by Jointly Learning Align and Translate, May 2016. URL http:///rraa..or///b////999 arXiv::149.044:
[cs].
[6] Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self Self-Supervised Contrastive Learning using Random Feature Corruption. October 2021. URL
net/forumiid=Cuv "atrrruummddddCC___gkkk3k333 qYkmkb3
[7] Tommaso Bendinelli, Artur Dox, and Christian Holz. Exploring LLM Agents for Cleaning
Tabular Machine Learning Datasets. March 2025. httttppp//oppurrveee..tteeee///e
id=RXnQPYSoun
[8] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, and et al. On the Opportunities and Risks of Foundation Models, August 2021. URL httptttst//rii..ott wttp:/ii...rg/bb//11..../ 07258v3
[9] Sebastian Bordt, Harsha Nori, Vanessa Rodrigues, Besmira Nushi, and Rich Caruana. Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models. First Conference Language Modeling, 2024.
[10] Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelezyk, and Gjergji Kasneci. Language Models are Realistic Tabular Data Generators. September 2022. URL https
/ //openreview net cetpumm?dddcEymm@uuu
[11] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug. Martin Pawelczyk, and Gjergji Kasneci. Deep Neural Networks and Tabular Data: A Survey. IEEE Transactions on Neural Networks and Learning Systems, 35(6):7499-559, June 2024. ISSN 2162-2388. doi: N0.1109/NNNLS document/9999840
URL
ieee. wwt.//otrraat....oo
[12] Leo Breiman. Random Forests. Machine Learning 45()) 45(1):5-32, October 2001. ISSN 1573-0565. doi: 00.1033/A:101093344433
htt.hhtts://ade /ttt:p/s/ooooo.roo/o..0000111113
[13] Tom Brown, Benjamin Mann, Nick Ryder, and et al. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Cur- ran Associates, Inc., 2020. URL /ttt

---

[14] Patricio Cerda and Gaël Varoquaux. Encoding High-Cardinality String Categorical Variables. IEEE Transactions on Knowledge and Data Engineering. 34(3)1164-1117) 1176 March 2022. ISSN 1558-2191. doi: 10. 10.1009/KDD
URL. hnttps://ieeepplon ieee org/
abstract/ Satirract//dccmmmt//99993
[15]
Tianqi Chen and Carlos Guestrin. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pages 785-794, New York, NY, USA, August 2016. Association for Computing Machinery. ISBN 978- 778-1-4503-4232- doi: 10.1 10.1145/93997222999955 URL https://dl acm. www..org/oo/i////
10e11111/29999 200.0////.00/0111111/3999992222. 2939785
[16] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota, June 2019.
Volume 1 (Long and Short Papers), pages Association for Computational Linguistics. doi: 10. 111111//1//19--11 1423. URL https://
aclanthology -olaathologgy.org/111------ 1423/
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021. URL httt::ttt://at org/ abtaaaab//000111.1 11929 arXiv:2010.1 1929 [cs].
[18] Gus Eggert, Kevin Huo, Mike Biven, and Justin Waugh. TabLib: A Dataset of 627M Tables with Context, October 2023. http:ttt/////i...ot wrw. httpg//bb//331.0 07875 arXiv:2310.07875 [cs].
[19] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. AutoGluon- Robust and Accurate AutoML for Structured Data, March 2020. http:////aaxxi wwwp:/iwww..rr.aeooooooe abaaabas///aa 2003 06505 arXiv:2003.06505 [stat].
[20] Xi Fang, Weijie Xu, Fiona Anting Tan, Ziqing Hu, Jiani Zhang, Yanjun Qi, Srinivasan Sen- gamedu, and Christos Faloutsos. Large Language Models (LLMs) on Tabular Data: Prediction, Generation, and Understanding A Survey. Transactions on Machine Learning Research, March 2024. ISSN 2835-8856. URL
[21] Benjamin Feuer, Robin T. Schirmeisster, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, and Colin White. TuneTabbes: Context Optimization for Scalable Prior-Data Fitted Networks. Advances in Neural Information Processing Systems, 37:83430- 83464, December 2024. URL httttt:://proceediinngge/////eee http:/ooeeeeiiiooeoeii.rpppppppe neurips cwtpaperppppppecceeccpeceeee..ee http://aaprffles/pppeeeee files/paper/
html
[22] Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning. Journal of Muchine Learning
Research, 23(261):1-61, 2022. ISSN 1533-7928. URL http::/jmll
21-0992 html
[23] Sebastian Felix Fischer, Matthias Feurer, and Bernd Bischl. OpenML-CTR23 curated tabular regression benchmarking suite. August 2023. URL httts:://openreviee net/ forum?id-HebA00Mmm3.
[24] Josh Gardner, Juan C. Perdomo, and Ludwig Schmidt. Large Scale Transfer Leamingg for Tabular Data via Language Modeling. Advances in Neural Information Processing Systems, 37:45155- 45205, December 2024. URL. httphhtt:p///ooceeeeiiiieo
www..p..uiicc.oooo.... www.nuip...ppppprrrrrrooc.. http:/pppprrf.llle/pppppppp/ html
[25] Pieter Gijsbers, Marcos L. Bueno, Stefan Coors, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. AMLB: an AutoMI. Benchmark. Journal of Machine Learning Research, 25(101): 1-6,,, 2024. ISSN 1533-7928. URL, httphttt:///mm.
http///mmrrorrro/// .org/
hapers/v25/255 0493. html
12

---

[26] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, pages 18932-189299 18943, Red Hook, NY, USA, December 2021. Curran Associates Inc. ISBN 978-1-7138-45533
[27] Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. In Proceedings of the 36th International Conference Neural Informa- fion Processing Systems, NIPS '22, pages 24991-25000 1-25004, Red Hook, NY, USA, November 2022.
Curran Associates Inc. ISBN 778-1-7138-7108.
[28] Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. TabR: Tabular Deep Learning Meets Nearest Neighbors. The Twelfth Interna- tional Conference on Learning Representations, October 2023. URL https://ppperrreiioo
het//forum?/dd=rnggggS5.3.
[29] Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. TabM: Advancing Tabular Deep Learning with Eefammttr--fficient Ensembling, February 2025. URL http::/arxiv.... org/ abs/2410 24210 arXiv:2410.24210 [cs].
[30] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and et al. The Llama 3 Herd of Models, November 2024.
http:://axxiv.org/bbbb..
21783 arXiv:2407.21783 [cs].
[31] Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-hased models still outper- form deep learning on typical tabular data? Advances in Neural Information Processing Systems, 35:507-520, December 2022. URL /ttps:/t////r//eeiiiinn www.:/dddinnn...uuu......... www.ips.nnpiieeeee httpp//ss...uppppppppappe..rrara
Benchmarks html
[32] Léo Grinsztajn, Edouard Oyallon, Myung Jun Kim, and Gaël Varoquaux. Vectorizing string entries for data processing on tables: when are larger language models better?, December 2023. URL httthhtt///arxtte http:/iiv.rrggbbb////ea/ 09634 arXiv:2312.09633 [stat].
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015. URI. http:://rxiiv org/abst wrrg/bbs/1112 111211111111111 03385
[cs].
[34] Xin He, Kaiyong Zhao, and Xiaowen Chu. AutoML: A survey of the state-of-the-art, Knowledge-Based Systems, 212: 106622, January 2021. ISSN 0950-7051. doi: 10. 10160.
knosys.2020. 106622. URL httph:tt///m/t hattps::/www.scienneee wwiencedirect.....m..ee..... http://ttceeeeeeeeeeeeerreeeeeee 50950705120307516
[35] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. TabLLM: Few-shot Classification of Tabular Data with Large Language Models. In Proceedings of The 26th International Conference Artificial Intelligence and Statistics, pages 5549-5581. PMLR, April 2023. URL. httpps:/ppooceedingg/. www.mmr.p... wwwppprsssss/. hegselmann23a. html
ISSN: 2640-3498.
[36] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second. The Eleventh International Conference Learning Representations, September 2022. URL https://
openreview wtt.:nnteeooooooomooossssto..ote
[37] Noah Hollmann, Samuel Müller, and Frank Hutter. Large Language Models for Auto- mated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engi- neering. Advances in Neural Information Processing Systems, 36:44753 Decem- ber 2023. URL htRpL/tttts/p/poceeedmae www.nneuriicccco http:/iww...nirri.cccooeeeeeeeee
faraperrflles___ files/ieee//ee httppppprrr.iieeeeppp/////////// /aper/2022/hashh
html
[38] Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, (37455)))3319-33 January 2025. ISSN 1476- 4687. doi: 10. (0886/4158--024-083322
URL https: : //tww//ature.../t////////tt//eat nature. com/ http/rtticles///// Publisher: Nature Publishing Group.
s41 s41586 024 024-000---0000 08328-8388800
13

---

[39] David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned
MLPs and boosted trees on tabular data. Advances in Neural Informatiin Processing Systems, 37: 26577-26658, December 2024. URL http:: /tt. mewrsed neurips. cc/paper liapperrflles files/
html
[40] Shi Bin Hoo, Samuel Müller, David Salinas, and Frank Hutter. The Tabular Foundation Model TabPFN Outperforms Specialized Time Series Forecasting Models Based on Simple Features. October 2024. URL URL.htttp://opeerevvloo http://owwu.cc--00/Afa0CCdd.d.d.
[41] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Larous- silhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-Efficient Transfer Learning for NLP. In Proceedings of the 36th International Conference on Machine Learn- ing, pages 2790-2799. PMLR, May 2019.
mlr press/v97/
houlsby19a html
ISSN: 2640-3498.
[42] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. October 2021.
[43] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular Data Modeling Using Contextual Embeddings, December 2020. URL htRL:tttt///aaxw.orrrg//// 2012.06677 arXiv:2012.06678 [cs].
[44] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models, January 2020. URL http: arxiv. org/
08361 arXiv:2001.0833 [cs].
[45] Liran Katzir, Gal Elidan, and Ran Yanivn Net-DNF: Effective Deep Modeling of Tabular Data. October 2020. URL
[46] Guolin Ke Qi Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qi- wei Ye, and Tie-Yan Liu. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: cwipaaprrsrspss....eeeee.pspsses wwwpaper_faiiieeeeccee files/ papp://0///r////s. otpppppps..lss//pp/////////////
[47] Myung Jun Kim, Léo Grinsztajn, and Gaël Varoquaux. CARTE: pretraining and transfer for tabular learning. Proceedings of the 41st International Conference on Machine Learning. volume 235 of ICML 24, pages
Vienna, Austria, July 2024. JMLR.org.
[48] Ravin Kohli, Matthias Feurer, Katharina Eggensperger, Bernd Bischl, and Frank Hutter. Towards Quantifying the Effect of Datasets for Benchmarking: A Look at Tabular Machine Learning.
[49] Jannik Kossen, Neil Band, Clare Lyle, Aidan N Gomez, Thomas Rainforth, and Yarin Gal. Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning. In Advances in Neural Information Processing Systems, volume 34, pages 28742- 28756. Curran Associates, Inc., 2021. URL https::///roceee/// iewpeedddiinnnnnnnnnipeeeee neur seurips... cc/paper/ 2021
html
[50] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems. volume 25. Curran Associates, Inc., 2012. URL httpps//ppppesrssssss nips. http:///pppppeeiiiio.ppppppo.oop
ffless/
[51] Barak Lenz, Opher Lieber, Alan Arazi, and et al. Jamba: Hybrid Transformer-Mambbl Language Models. October 2024. URL http://ppppprrriiee.eeeeetorrrrm
[52] Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C. Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, and Micah Goldblum. Transfer Learning with Deep Tabular Models. The Fleventh International Conference on Learning Representations, September 2022. URL /ectts::/oppnreeee http://wwwweetterrririeeeeoorooe
14

---

[53] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Opti- mized BERT Pretraining Approach, July 2019. http: berg/abs/ii 1907 11692
arXiv:1907.1 1692 [es].
[54] Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Anthony L. Caterini, and Maksims Volkovs. TabDPT: Scaling Tabular Foundation Models on Real Data, July 2025. URL
http://arxa www.aaiwww.rggaa///oooa wwwgabb/////a/ 18164 arXiv:2410.18164 [es].
[55] Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakrish- nan, Micah Goldblum, and Colin White. When Do Neural Nets Outperform Boosted Trees on Tabular Data? Advances in Neural Information Processing Systems, 36776336-76669 Decem- ber 2023. URL httpp:: (D0d6ee6f44040099//4cee333333 Abstract- DatrtttDDDattttt... Bacrtttss__nn_eeehhhaeaaee .html httl
neurips cc/ httpc/ppee__f_eeee files /etaperr222///nee
[56] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space, September 2013. URL http:://axxi/.oorrtrt///// wttp:ivv..orrraa//1111111 [cs].
3781 arXiv: 1001 .3781
[57] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive Text Embedding Benchmark. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guisties, pages 2014-2037 2037, Dubrovnik, Croatia, May 2023. Association for Computational org/
Linguistics. doi: 10. 2023. eacl-main. main. 148/
URL. htttps://aclantholl///
[58] Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers Can Do Bayesian Inference. October 2021. URL https://openrevieer net/ forum?1d-KSugKcblff
[59] OpenAL. GPT-4 Technical Report, March 2024. httpaappp//aaai...//aaa/////////
arXiv:2303.08774 [es].
[60] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Doro- gush, and Andrey Gulin. CatBoost: unbiased boosting with categorical features. In Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL //pooced/so neurips wwtprips.cc/pppep.//. cc/paper/2018/has 14491777663335011142222222333333 -a1-:8633225554-4bbrreere Abstract html
[61] Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data, February 2025. URL http://
arxiv. wewiiv.or//oe /aab./250...2 05564] arXiv:2502.05564 [es].
[62] Ivan Rubachey, Artem Alekberov, Yury Gorishniy, and Artem Babenko. Revisiting Pretraining Objectives for Tabular Deep Learning, July 2022. URL http://arxiv.. 220
03208
arXiv:2207. .03208 [es].
[63] Xingjian Shi, Jonas Mueller, Nick Erickson, Mu Li, and Alex Smola. Benchmarking Multimodal AutoML for Tabular Data with Text Fields. August 2021. URL https:///ppenrreview net/
forum?id-g02o0aaeccH
[64] Assaf Shmuel, Oren Glickman, and Teddy Lazebnik. A Comprehensive Benchmark of Machine and Deep Learning Across Diverse Tabular Datasets, August 2024. URL http:://rxi.ii...////// org/ abs/2403 14817 arXiv:2408. 14817 [cs].
[65] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, May 2022. ISSN 1566-2535. doi: 10. 10.1016/..nfftus2222.... 202..
URL
com/ wttpp//t..o.ccccccccccaaaaaaaaaa http://ccc...ccccepppp//sssaaaaa
[66] Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition, April 2015. URL 155 [cs].
/etes//449... 1556
15

---

[67] Leslie N. Smith and Nicholay Topin. Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates, May 2018. URL http::/arxiv, org 07120
wwwp:/iwi.ogga//////oooa/ abs/
arXiv:1708.07120) [cs].
[68] Aivin V. Solatorio and Olivier Dupriez. REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers, February 2023. httthhttttapiiii http:/twwwvi.....aaaaa// /abb/23002
02041
arXiv:2302.02041 [cs].
[69] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre- Training, June 2021. WtRphhtpp//axxii..oo wrg. httgg/ab/211000 221//111066013222
arXiv:2106.00 342 [cs].
[70] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autolnt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 19, pages 1161-1170, New York, NY, USA, November 2019. A Association doi: 10.1 10.1145/33578884333777 URL
for Computing Machinery. ISBN 978-1 978-1-4503-6676-33
https: //dl. acm. org /10/////111111 1111144333333333399999333
[71] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural net- works. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2, volume 2 of NIPS' 14, pages 1104-3112, Cambridge, MA, USA, December 2014. MIT Press.
[72] Zhiqiang Tang, Haoyang Fang. Su Zhou, Taojiannan Yang. Zihan Zhong, Cuixiong Hu, Katrin Kirchhoff, and George Karypis. AutoGluon-Mulimmdaal (AutoMM): Supercharging Multimodal AutoML with Foundation Models. AutoML Conference 2024 (ABCD Track), April 2024. URL wwwp:ssw..n/.fooooe
nwt. meneefforumaee www.rrumiiddiiittttttie irStSm9wai)
[73] Zhiqiang Tang, Zihan Zhong, Tong He, and Gerald Friedland. Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Data, December 2024. URL http:://axixi org/ abs/ 16243, arXiv:2412. 16243 [cs].
[74] Gemini Team. Gemini: A Family of Highly Capable Multimodal Models, May 2025. URL pttp://arxiv..or org/ abs/ 231b. 11805 arXiv:2312. 1.80 [cs].
[75] Avijit Thawani, Jay Pujara, Pedro A. Szekely, and Filip Ilievski. Representing Numbers in NLP: a Survey and a Vision, March 2021. http::httt::::t wwwpxii..rg/aaae orgo attpaabb///1/111// 2203 13136
1xXiv:2103.13166 [cs].
[76] Andrej Tschalzev, Lennart Purucker, Stefan Lüdtke, Frank Hutter, Christian Bartelt, and Heiner Stuckenschmidt. Unreflected Use of Tabular Data Repositories Can Undermine Research Quality, March 2025. URL/htp::/ org/ aaa///b///// 09159 arXiv:2503.09159 [cs].
[77] Boris Van Breugel and Mihaela Van Der Schaar. Position: why tabular foundation models should be a research priority. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of ICML pages 48976-48993, Vienna, Austria, July 2024. JMLR.org.
[78] Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. OpenML: networked science in machine learning. SIGKDD Explor. Newsl., 15(2):49 60, June 2014. ISSN doi: 10.1145/2641190.22
2641198
[79] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention All you Need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, www.eeiiiiiinnnnnnnneeeeeeee neurips. pipaper_iilee/ppprr/////// Paper pdf
Inc., 2017. URL 3tfo22255766e111tt00c1ccllaslss
[80]
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text Embeddings by Sueee--Suerr Contrastive Pre-training, February 2024. URL http:htt///aa org/ abs//b//1112 03533 arXiv:2212.03533 [cs].
16

---

[81] Ruoxi Wang, Rakesh Shivanna. Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems. In Proceedings of the Web Conference 2021, www '21, pages 1785-1797, New York, NY, USA, June 2021, Association for Computing Machinery. ISBN 978-1-4503-8812-
7. doi: 10.1 10.1145/3442381.3555 14550778. URL httpss//: acm. prgg/oi//1..om 1145/3442381 3450078
[82] Wei- Yao Wang, Wei-Wei Du, Derek Xu, Wei Wang. and Wen-Chih Peng. A Survey on Self Self--uperrvied Learning for Non-Sequential Tabular Data. CoRR, January 2024. URL
httts://openrevviex htt.:///w....ttoooooooooddddddar
[83] Zifeng Wang and Jimeng Sun. Trans Learning Transferable Tabular Transformers Across http:p/www.........eeeraaaaaaaa.
Tables. October 2022.
[84] Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making Pre-trained Language Models Great on Tabular Prediction. The Twelfth International Conference on Learning Representations, October 2023. URL https
/ //openreview wwtp:e/oooommmddd....rcraoc.
[85] Junchen Yang, Ofir Lindenbaum, and Yuval Kluger. Locally Sparse Neural Networks for Tabular Biomedical Data. In Proceedings of the 39th International Conference on Machine Learning, pages 25123 123 25 153. PMLR, June 2022. URL http::://pocceeeaiiiiiee yang22i. html ISSN: 2640-3498.
mlr pwwss/vv6222
[86] Chao Ye, Guoshan Lu, Haobo Wang. Liyao Li, Sai Wu, Gang Chen, and Junbo Zhao. To- wards Cross-Table Masked Pretraining for Web Data Mining. May 2024. URL https
[87] Han-Jia Ye, Si- Yang Liu, Hao-Run Cai, Qi-Le Zhou, and De-Chuan Zhan. A Closer Look Deep Learning on Tabular Data. CoRR, January 2024. URL https::/ net/ forum?id-eu2cABIIH
[88] Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. A Comprehensive Survey on Pretrained Foundation Models: History from BERT to ChatGPT, May 2023. URL (ttp:/tt://artt /tr//vxxivv.. org/ hrgg/abs/3002. 2302. 09419 arXiv:2302.09419 [cs].
[89] Qile Zhou, Han-Jia Ye, Leye Wang, and De-Chuan Zhan. Unlocking the Transferability of Tokens in Deep Models for Tabular Data. October 2023. URL https:://peerreiiett net/
Forum?id=u20V(22Xvr
[90] Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. XTab: Cross-table Pretraining for Tabular Transformers. In Proceedings of the 40th International Conference on Machine Learning, pages 43181-43204. PMLR, July 2023. URL https //proceedings. mlr prwss/eessvveee
seress/v002// 120222hu2222 html ISSN: 2640-3498.
[91] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A Comprehensive Survey on Transfer Learning. Proceedings of the IEEE, 109())44376, January 2021. ISSN 1558-2256. doi: 10. 101110//PR0.22220000005555 URL https://ieeoxplore ieee arr/abberracc/ooouue///3333333
17

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We claim "TabSTAR achieves state-of ofte-of-tie of-tthe-att performance for both medium- and large- sized datasets across known benchmarks" in the abstract and "TabSTAR achieves state- of-the-art (SOTA) performance on classification benchmarks containing substantial textual content" in the introduction we describe these results detail in $5 showing TabSTAR is the top performant comparing vs leading gradient boosting decision trees (GBDTs) and leading Tabular Foundation Models (TFMs)
We claim "...its pretraining phase exhibits scaling laws in the number of datasets" the abstract and "its performance steadily improves with more pretraining data" in the intro, and show in that TabSTAR's 's performance improve as we significantly increase the number of pre-training datasets.
We claim in the introduction "TabSTAR can optimize free-text feature representations, demonstrating their clear superiority over alternative frozen embedding approaches" we describe these results in detail in $6 by showing that TabSTAR's ' performance increase greatly when finetuning the embeddings, making this the advantage over other algorithms compared in $5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, in 77 we discuss TabSTAR's limitations, such its lower performance in regression and its lack of evaluation in other scenarios like datasets without textual content.
Some more details about the regression performance elaborated $5)
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
18

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
The paper doesn't present theoretical proofs, but focuses on algorithmic development and empirical validation.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes. Our model and method is highly elaborated through the paper (see 3 and the main experiments well ($4). In addition, multiple supplemental materials cover these points in more details (e.g., Appendix A Appendix B). Furthermore, we release our code, alongside documentation.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
19

---

reproducible, they While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: See reference from I to our anonymous GitHub repository, where we provide our code and running instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/ //nips https/ccccuulcccguudcccocc wewggdddde/eeeeebbinnnme wwt.:Suuumiiiioooooooooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
$
Apppendix B and Appendix give the full experimental protocol.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
20

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results are accompanied by 95% confidence intervals (see Appendix F). This is true also for the analysis section $6 which follows the same standards.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the bars are capturing should be clearly stated (for example, train/test split, imitialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, see Appendix F.4 for for compute information and running times, for both
TabSTAR and other baselines.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:////weeereesccccpppccc//tta
Answer: [Yes]
Justification: The study relies exclusively on publicly available, de-identified datasets (OpenML, Kaggle and published benchmarks) and releases code and models under open- source licence; no human subjects or sensitive personal data are involved, fully satisfying the NeurlPS Code of Ethics.
21

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We use running example introduced in Table[an an important real use case with high impact the healthcare industry, motivated by the importance improving decision making in small-data regime. In addition, we briefly discuss other industries in $7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. degloomeent of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: TabSTAR is not generative model, and thus can't 't be used to create potentially high-risk text and images. In addition, we use publicly available datasets which don't impact individual privacy or security.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
22

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we credited them in an appropriate way, adding URLs and package names in multiple occasions throughout the paper and its supplemental materials.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should provided. For popular datasets, paw.rssi.iaasaiiiia...... thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Upon acceptance, will release several TabSTAR pretrained checkpoints, plus complete training and inference code. We believe that this research line is of huge potential and we encourage researchers to continue our line of work, also providing feasible evaluations setups for TFMs, such as default hyperparameters.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer:[NA
Justification:
The paper doesn't 't involve crowdsourcing ot human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
As the study didn't involve any study participants, this section isn't applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: While TabSTAR uses a language model at its core, this is an encoder-only model, and not generative LLM. This widely used language model, e5-small-v2 [80] is used as part of the architecture in a standard usage. Thus, this section isn' applicable to our research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM for what should or should not be described.
24

---

A
Architecture
This appendix provides additional technical details for the architecture introduced in $ First, we discuss the verbalization module; next, we formally describe the architecture step-by-step; and finally, we present selected experiments on the TabSTAR architecture.
A.1 The Verbalization Module
TabSTAR's verbalization module standardizes heterogeneous tabular inputs by converting each column, whether predictive feature target variable, into templated blocks. We first describe the detection of column types, and then detail the processing steps for each type.
Feature Detection We classify each column as either numerical, referring to quantitative values, or semantic, referring textual values including categorical and boolean fields encoded as text. We rely on heuristics involving both the primitive data type (e.g., string, float) and human annotation (e.g., OpenMI. metadata). However, real-world datasets pose challenges, as numerical features can often be stored strings (e.g., "35 years". "unknown age") or may lack inherent order (e.g., country calling codes). Leveraging LLMs for contextualized data cleaning can be a promising direction [7].
A special case is the handling of timestamp and date columns, Similarly to [40] we rely on skrub's 's Encoee to detect datetime columns and decompose each one of them into set of new features. Each extracted feature then undergoes its own processing: For example, the weekday is treated as semantic, while the total seconds since the Unix epoch is treated as numerical. Integrating date features more holistically remains an open research question.
Table 5: Illustrative verbalization of a numerical feature (Age) with 10 bins. Examples outside the range and missing values considered special bins.
Bin
Range
Example Value
Illustrative Verbalization
Lower than 18
17
Age: Lower than 18 (Quantile 0%)
18-23
20
Age: 18-25 (Quantile 0-10%)
2
23-27
25
Age: 23-27 (Quantile 10-20%)
3
27-31
29
Age: 27-31 (Quantile 20-30%)
4
31-35
33
Age: 31-35 (Quantile 30-40%)
5 35-40
38
Age: 35-40 (Quantile 40-50%)
40-45
42
Age: 40-45 (Quantile 50-60%)
7 45-51
48
Age: 45-51 (Quantile 60-70%)
51-58
55
Age: 51-58 (Quantile 70-80%)
9 58-67
63
Age: 58 67 (Quantile 80-90%)
10
67-87
83
Age: 67-87 (Quantile 90-100%)
Higher than 87
93
Age: Higher than 87 (Quantilc 100%)
Unknown
Age: Unknown Value
Numerical Features Numerical features represented by both numerical and semantic representation. For the numerical representation, given a value æ, we compute the clipped z-score z' -clipp(a p)/a, -3, 3) where μ. are the training set mean and the standard deviation, and missing values are set to 0. For the semantic representation, we build B = 10 quantile bins over the training distribution to map the value accordingly. Table 5 an illustrative example for the feature Age from our running example in Table
Semantic Features
Semantic features are sanitized (e.g., normalizing whitespaces) and verbalized using the template presented in Table Missing values are mapped "Unknown Value". just like for numerical features. If a text exceeds the model's context window (512 tokens for -5-small-v2), it is naively truncated fit it. This limitation is far more pronounced for methods that serialize the entire example into single textual sequence [84] thereby dramatically reducing the effective context size.
batps///kkrub-aataa
25

---

Target Variables The verbalization templates for the target values are prepended to every example. For classification tasks, each possible label is verbalized, while for regression we verbalize a single element consisting solely of the feature name. Employing binning strategy to treat regression as classification task future work direction, as diseussed in $7 For regression tasks, target values go through the same standardization with outlier clipping as numerical features, being used solely as the ground truth without going through the input.
Table 6: Verbalization templates for semantic features and target values.
Element Type
Verbalization Template
Predictive feature
"Predictive Feature: {feature_ "amaturr_aam"""
"Feature Value: [feature_valuee value)"
Classification target
"Target Feature: (target_name)"
"Feature Value: (target_ value)"
Regression target
"Numerical Target Feature: [targetnname]"
A.2 The Annotated TabSTAR
Table 7 describes the number of parameters per component in the TabSTAR architecture, when using e5-small-v2 [80] as the text encoder. It has approximately 47. 47.26M parameters, most of which come from the text encoder. When unfreezing 6 layers of the text encoder, about 24. 70M parameters are tuned, with the remaining 11.92M embedding parameters and 10.65M layer ones being kept frozen.
Table 7: Parameter counts for TabSTAR components
Module
Parameters
Encoding:
Semantic
33,360 23,360,000
Encoding:
Numerical
296,832
Fusion
1,774,464
Interaction
10,646,784
Prediction
1,185,794
To describe the architecture more precisely, we start by defining the dataset formally. Let D {(x,, y4) y:) denote a tabular dataset with n examples. Each example 2, = [aai..... Dsm] has m features. The target variable y; is either continuous (regression) or discrete (classification) taking one of classes. For simplicity, we describe the architecture at the example level, though all computations are actually carried out on mini-batches of size B. The batches are always drawn from a single dataset in both pretraining and finetuning, removing any need for padding.
Verbalization We denote by the number of target entries where ( C' for classification and t= 1 for regression. We then form a raw sequence of length = / + m by listing the target values, followed by the feature entries. Each element in this sequence is then verbalized into semantic string 8y and numerical value ny, set to the clipped z-score for numerical non-missing features, and zero otherwise. The example is thus represented by parallel sequences (s, n) of length
Encoding Each semantic string 8, and numerical value n, are projected into d-dimensional vector. Semantic strings are encoded with an encoder-only language model (e5-small-y) (80)). Each string tokenized, passed through the model, and pooled produce its final embedding. This process is independent between elements, i.e. the attention is at the token level within a single element. In parallel, each numeric value fed through a two-layer MI.P that first projects from to 2d dimensions, applies a ReLUU and dropout, and then projects back down to d. This produces matching d-dimensional embeddings for each of the elements, ready to be fused.
Fusion To unify semantic and numerical embeddings into a single representation, we apply a single-layer Transformer Encode 31 over each element's pair of vectors. Concretely, for each element
With 2 attention heads, ferd-forward hidden size of 4d, dropout 0.I, and ReLU activation.
26

---

we stack its d-dimensional text and numeric embeddings and feed them through the encoder layer. For every element, the attention upplied between its two representations, and we average the two outputs to produce one fused d-dimensional embedding. This yields final sequence of length and dimension d, which will serve as tokens for the Interaction block.
Interaction The fused sequence e tokens is processed by a standard Transformer Encoder with model dimension d 384, I. 6 layers, 6 attention heads per layer, feed-forward size 4d, dropout 0.1, ReLU activation and using a pre-norm configuration. Unlike in language modeling. feature ordering irrelevant, so no positional encodings are used. The encoder produces contextualized embeddings for every position, and we retain the t target embeddings for the prediction.
Prediction We initialize two identical MLP heads, one for regression and one for classification. Each them consists hidden layer size 4d (with ReLU activation) followed by a linear projection to single output. For each dataset, we choose the relevant head and process the & target token embeddings. For classification, we independently feed each one of the t= C' target tokens to the classification head to obtain score (logit) per class. Notably, the same head is shared across classes and datasets. We apply softmax over these scores, yielding probability distribution regardless of the number of classes. For regression, the single target token is projected into a real value. Note that the heads is shared between datasets, as regression outputs are always clipped z-scores.
A.3 Architecture Experiments
In this section we explore the effect of different design choices for TabSTAR's architecture. For each experiment, we only vary the parameter of interest, keeping everything else fixed. We follow the same pretraining regime as in Appendix B.I except that for computational efficiency
train only 25 epochs (instead of 50) with 128 pretraining datasets (instead of 390). We evaluate each variant relying solely pretraining performance, as approximation for downstream task performance. We acknowledge that our conclusions might depend on this limited scale, hence we discuss a subset of the experiments briefly reflect the depth of our work and inspire future research.
The Fusion Block's Mechanism For the fusion block, we consider two simpler alternatives to the attention mechanism, both of them underperforming: (1) Concatenation, by concatenating the semantic and numerical d-dimenniionl vectors into 2dlddimensional vector, and projecting them back via MLP, and (2) Multiplication, by multiplying the semantic representation directly with the numerical value in a straightforward, parameter- manner as in [83 86).
The Number of Interaction Layers We experiment with the number of encoder layers, and observe that 3 yields anecdotally worse performance than with lower parameter count. Nevertheless, we prioritize deeper network as for datasets with very complex relationships we believe that this might be beneficial. Additionally, we try 9-layer variant which performs significantly worse, while also increasing the parameter count.
Row-Level Attention We experiment with adopting the architecture proposed by SAINT [69]. which adds row-level attention to each encoder layer. Similar concepts are also employed by models that get the input the whole dataset, labels included [3 49). We run experiments with 2, 4 and 6 layers as they are roughly equivalent in parameter count to 3, and 9 layers without row-attention. We observe substantial gain, and thus prioritize the simpler solution, as row-level attention is sensitive to the batch size and adds complexity to inference time.
Verbalization We experiment with various verbalization strategies. For features, including the column name alongside the value consistently outperforms using the value alone. For target-aware tokens, we find that explicitly verbalizing all possible target values yields better performance than omitting them. contrast, adding a dataset-level description prefix to improve contextualization offers no measurable advantage. Interestingly. TabSTAR maintains high performance even when no semantic metadata is present. We see this robustness as a strength.
After rescaling to be centered around rather than O, using learned scaling factor.
27

---

B Training
In this section we elaborate the two stages of TabSTAR's training: pretraining and finetuning. presented in }
As in Appendix A.3 we summarize key pretraining experiments.
B.1 Prefraining
TabSTAR is pretrained employing supervised learning in a multi-task regime, jointly learning regression, binary and multiclass classification tasks. The parameters of the architecture are fully- shared, without any need for dataset-specific parameters. Every example during the pretraining updates all the model's weights, with the sole exception of the prediction head, for which every example uses its respective head depending on the nature of the task (classification or regression).
Sampling For computational efficiency, each dataset is subsampled once before the pretraining. At the example level, we sample up to 300,000 examples from each dataset, stratified by the target variable for classification tasks. Since we only use a fraction of each dataset for each pretraining epoch, this decision has negligible influence. In addition, we randomly sample up to 200 features per dataset. While straightforward. this decision is suboptimal as feature importance isn't taken into consideration. As this work does not focus on wide-feature datasets, we consider this trade-off acceptable. Importantly, this setup is enabled during finetuning as the TabSTAR architecture agnostic to the number of features. We split each dataset into train-validation splits (95%-5%) 23 without any need for test splits, and cap the validation set at maximum of 1, 1,000 examples used for evaluating pretraining performance.
Batching Every epoch, we randomly sample up 2,048 examples from each dataset in mini- batches of 32, and shuffle all the batches. We conduct gradient accumulation and update the model every 4 steps reduce the chances of a single update being dominated by single dataset, so the global batch size is effectively 128. Appendix B.3.1 elaborates on the effect of batch size.
Metrics Our loss function is cross-entropy for classification, and MSE for regression. With standardized targets, R² = 1 - MSE, although this equivalence is degraded by clipping targets in and apply gradient clipping to stabilize training without
preprocessing. We train with task-specific weights, with Appendix [.3.2 discussing the limitations of this approach. We use as metrics AUROC for classification and R² for each task the optimal metric value is 1. We average performance across all datasets into a single metric that reflects the pretraining performance.
Training We pretrain for 50 epochs with the OneCycleLR 667 optimizer, with warmup during the first 5 epochs (10%) and cosine annealing. Early stopping is conducted after 3 epochs without improvement on the pretraining metric. The weight decay is set to 0.001, and a max learning rate of Ir = 5 x 10-5 is applied uniformly across all layers. Appendix B.3.3 discusses experiments with differential learning rate. Pretraining running time varies depending on the number of epochs and the included datasets. The full models (390 datasets) reported in $5 train for less than 48 hours on a single NVIDIA A40 GPU (48GB memory), and we believe that this could be optimized much further.
B.2 Finetuning
We finetune downstream tasks using LoRA's implementation of the peft package We use a rank of 32, set a = 2r = 64 and dropout 0.1. We employ the same scheduler as in the pretraining phase, with the only difference being that we set Ir 0.001, and increase the patience parameter for early stopping to 5. We apply a train-test split of 90%- 10%% and sample a validation set of 10%. As opposed the pretraining, all batches are drawn from the same dataset. Therefore, we observe no effect from changing the mini-batch size when keeping the global batch size fixed to 128. We tune 1.597.440 out of TabSTAR's 47,263,87 874 parameters (3.4%), spanning all blocks of the architecture. The frozen layers of the text encoder do not receive LoRA adapters.
24
Finetuning hyperparameters are selected by pretraining TabSTAR over 256 datasets and performing grid-search over held-out set of 25 downstream tasks disjoint from the 50 datasets in the benchmark
2eWe choose only 5% for efficiency, as we use hundreds pretraining datasets. hhanps//gituubbooo///gggggggiiet
28

---

evaluated in $4 The search space is presented in Table and we observe that average performance relatively robust across this space. An interesting observation is that decreasing the number of parameters by setting = 16 mildly hurts performance, but it has upsides on memory and latency aspects, allowing a future trade-off exploration. As final note, we argue that providing strong default configuration for TFMs is crucial for evaluating them, but for real-world applications, it is still recommendable to find the best hyperparameters tailored to the downstream task.
Table 8: LoRA hyperparameter tuning grid search for TabSTAR's finetuning.
Hyper-parameter Search Space
LoRA rank (r)
16, 32, 64 0.0005, 0.001, 0.002, 0.005, 0.01
Learning Rate Dropout
O, 0.1
The only experiment in this paper where we employ full finetuning instead of LoRA is for the non-pretrained variant discussed in the analysis in $ For this variant we fully finetune the pretrained model on each downstream task. Compared the pretraining, we use 2.5 10 and increase the patience to 5. These hyperparameters are lightly tuned using the same procedure as for LoRA, and we observe that fully finetuning the model achieves comparable performance, except for small datasets, where training is more prone to overfitting.
B.3 Pretraining Experiments
In this section, we briefly elaborate on some experiments performed over TabSTAR's pretraining protocol. As in Appendix A.3 we highlight only subset of them in a high-level manner.
B.3.1 Batch Size
During pretraining, we use mini-batch size of 32, each of them drawn from single dataset. Since we train with gradient accumulation and global batch size of 128, varying the batch size affects diversity of a single model update: lower batch sizes are likely to be exposed to more datasets. We decrease the batch size to 16 and 8 and observe an improvement at the cost of slower training. An interesting direction for future work moving to mixed-datasets batches, which require more complex implementation but might benefit from more regularized learning. Such approach, however, goes against row-level attention methods and ICL, as discussed in Appendix A.3
B.3.2 Loss Weights
Pretraining the model over hundreds of datasets in multi-task regime presents a key challenge: the loss scale of each dataset can vary substantially, depending on task difficulty or task type. For example, multiclass classification task with dozens of classes will naturally yield a higher average loss than a binary task. These dynamics can also shift during training. Our default approach naively averages the loss across all datasets, which risks over-weighting tasks for potentially arbitrary reasons.
To address this, we explore two alternative weighting strategies: (1) Assigning a constant weight task type, accounting for the number of classes in classification tasks, and (2) Normalizing each dataset's contribution by the best loss achieved by CatBoost [60] when being fitted to that dataset. While these strategies better reflect task-specific characteristics, they hardly impact performance and introduce additional complexity. Notably, adjusting loss weights across tasks impacts metric interpretability, each weighting scheme implicitly optimizes different objective.
We do not explore more sophisticated methods such as learning per-datased weights, as these often require mixed- -atasett batches and introduce additional learnable parameters. We believe. however, that multi-task pretraining over tabular datasets remains an open and important research question.
B.3.3 Differential Learning Rate
TabSTAR's 's weights initialization is not balanced: the textual encoder is a pretrained embedding model while the rest of the architecture parameters are randomly initialized. To counteract this imbalance, experiment with using differential learning rates for the textual encoder layers, and experiment
29

---

with scaling it by a factor of 0.5 and of 0.75. To our surprise, this decision hurts performance, so we stick uniform learning rate across all layers.
C Training Datasets
In this appendix briefly expand on the pretraining corpus elaborated in $4 It is composed of 350 datasets, spanning all the datasets appearing in AMLB [25). OpenML-
[23]. TabZilla [55] and the ones presented by Grinsztajn [31] After deduplication this results in 152 datasets (94 classification, 58 regression). Interestingly, only 6 of these 152 datasets have free-text or high- cardinality features. We manually add datasets from OpenML 78 and Kaggle, well as from the AutoML-Benchmarr -rain 22 corpus, and achieve a total of 350 datasets, with 49 textual datasets.
Table details the 253 classification datasets and Table 10 the 97 regression ones. We elaborate the Datasel name, the number examples n, the number of features m, and the number of classes C for classification. In addition, mark datasets that belong to one of the benchmarks, and the ones that have text features. Importantly, the textual flag is quite permissive, as it includes features with relatively short texts potentially low predictive power (e.g.. people names addresses).
Table 9: The 253 Classification Datasets of the Pretraining Corpus, with their examples, features, C' classes, presence in a benchmark (B) and whether they are textual (T).
Dataset
m
C
KDDCup99
4,898,422
40
20
mimic extract los
4.155,270
17
68
V
2.875,146
16
st-police-mcrrente
2,215.023
V
physionet sepsis
1,552,210
42
2
poker-hand
1,025,009
10
Higgs
1 1,000,000
28
BAF base
1,000,000
Credit Creddt_Card______ _raud
1,000,000
7
648,493
porto-seguro
595,212
57
2
covertype
581,012
AVIDa-hIL6
573,891
aurlines
539,383
7
HolisticBias
472,991
14
albert
425,240
78
DBPedia
342,781
219
hcdr main
307,511
120
Mental Health Dataset
292,364
16
Kuzushin-49
270,912
784
49
spoken-arabic-digl
263,256
14
10
edc diabetes
253,680
21
2
skin-segmentation
245,057
LI- Laa-teee--------alllllee
233.154
38
2
Churn Telco_ Europa
190.776
Idpa
164,860
6
11
Give-Me MeM---------eeeie
150,000
149,332
4
22
social bias frames
144,649
Wikipedia Lalele
140,379
12
15
Munteraal-Deht-Rss---men
138,509
MimBoNNE
130,064
50
2
nba-shot-logs
128,069
15
Continued on next page
And the exclusion of the the,fia fifa dataset, which is included in the benchmark.
30

---

Table 9: The 253 Classification Datasets of the Pretraining Corpus.
Dataset
C
B
T
college_scorecse
124,699
117
drrg-orry
120,215
16
7
TVS Loan Default
119.528
29
road-safety
111.762
32
2
Diahetes33UUS
101,766
fars
100.968
29
8
Credit Score Classification
100,000
numera128.6
96,320
21
Run_or__ wwalk_informate
88,588
jannss
83,733
54
KDD98
82,318
477
APSFatlure
76,000
169
kick
72,983
human-choice -oriicccrccc--iiitionnn
71,579
20
2
Traftic violation
70,340
20
Fashion-MNIST
70,000
784
10
Cardiovascular-Dae hwwpaaccuuaa---sss--ttaaaaaa
70,000
mnist 784
70,000
719
10
connect-4
67,557
42
mobile_chup
66,469
63
helena
65,196
27
100
LICD
63,634
413
CIFAR 10
60,000
3,072
10
REASONER
58,497
volkert
58,310
147
10
shuttle
58,000
GTSKB-HueHist
51,839
256
43
okcupid-stem
50,789
19
KDDDCupUS-Upeeling
50,000
13,419
2
KDDDCupU_appeteeey
50,000
207
adult
48,842
14
of-guu--f-LLgends Diamond
48,651
14
Lamelnadu-llccteetty
45,781
2
20
bank Lunk-marketing
45,211
16
meta stream intervals Intervas
45,164
74
11
juggle chess
44,819
6
htw.riaalllyyeGeeeeeee---ee Datsse
41,144
8
2
Breass-cancer--peediitoo
39,998
11
Click_preddcttoonssnaa
39,948
Hotel-Keviews
38,932
electricity
38,474
nomao
34,465
118
2
Employce- Tumover-at-TECCCC
34,452
Amazon employee access
32,769
9
2
Credit-Risk-Dataan
32,581
Canerlt------eed--------------C Dlattt Dataset
30,000
23
2
tunpedia
29,819
credit risk_china
27.522
27
5
Insurance
23,548
guillemo
20,000
4,281
2
nicaardo
20,000
4,283
insurance_dame dataset
20,000
26
letter
20,000
16
26
16,825
5
43
Continued next page
31

---

Table 9: The 253 Classification Datasets of the Pretraining Corpus.
Dataset
C
B
T
NewspaperChurn
15,855
16
mozilla4
15,545
5
2
pol
15,000
26
11
eog-cye-state
14,980
14
2
Magiclelesscppe
13,376
nursery
12,958
8
onlne-shhopers-inennioo
12,330
Disaster- Iweets
11,370
4
mammography
11,183
Websites
11,055
30
2
Shanyy-DDaaaetto--PPhhi--n--- Legg
11,000
pendigits
10,992
16
10
WBCAtt
10,298
wrtuhcial-chaaace
10,218
7
10
internet usage
10,108
71
46
robert
10,000
7,200
10
dilbert
10,000
2,000
shrutime
10,000
10
Japanese Vowels
9,961
14
9,873
32
#ICO-HELOC-oleancl
9,871
23
IBRD Loans Classincation
9,215
10
Indian pines
9,144
220
8
SpeedDating
8,378
120
fabert
8,237
795
mushroom
8,124
isolet
7,797
617
26
sye_movements
7,608
23
twonorm
7,400
20
blastchar
7,043
19
musk
6,598
167
2
6,118
HMEQ Data
5,960
12
2
philippine
5,832
308
optdigits
5,620
62
10
BachChoorlHarmony
5,586
15
68
page-blocks
5,473
10
5
wall-robot-navigaiion
5,456
24
christine
5,418
1,611
phoneme
5,404
fraud
5,227
19
sylvine
5.124
20
Satellite
5,100
Multrclass Classification for Lae__Corrrrteeeate Credit
5,000
7
10
Lassonal-Loan-Modeee
5,000
churn
5,000
20
2
waveform-500
5,000
5,000
9
Heart Failure Prediction Predcttueereeeeecttooonnn
5,000
compas-two-years)
4,966
11
2
wine-
4,898
wilt
4,839
5
2
spambase
4,601
StacKOveerloo-polarity
4,423
3
Continued next page
32

---

Table 9: The 253 Classification Datasets of the Pretraining Corpus.
Dataset
C
B
T
hiva hiva_agvvnnl _aanasio
4,229
1,617
4,156
27
2
4,147
analcatdata supreme
4,052
7
10
hypothyroid
3,770
Bioresponse
3,751
1,776
nnvertet-dteertismmentts
3,279
1,558
led24
3,200
24
10
kr-vs-kp
3,196
splice
3,190
60
dna
3,186
2ma
3,153
970
2
madeline
3,140
259
2
jasmine
2,984
144
cjs
2,796
29
madelon
2,600
500
ozone-level-Shr
2,534
72
segment
2,310
16
cardiotocography
2,126
23
10
Listmattoonoo__besiii Levels
2,111
16
kcl
2,109
21
2
Corporate- Caeeit-rrrriiiiiie -aainn
2,026
micat-tactors
2,000
216
10
South Asian Churn dataset
2,000
13
mfeaa-zeenike
2,000
47
10
mffatttourier
2,000
pbcseq
1,945
18
3
steel-plates-fate
1,941
car
1,728
6
4
GAMETES Heeeroggnettr
1,600
one- chlundred-platiie tlants-textureee
1,599
64
100
autat-data
1,552
OVA Breast
1,545
10,935
2
amazon-commereo--rrviii
1,500
10,000
50
yeast
1,484
8
10
cme
1,473
9
ibm-employee--atriton
1,470
31
pc4
1,458
37
Data Science CiitneeeeNgeriiaaaa Nigeria_Telecoms. Telecoms_ _Churn
1,400
14
hepatitis c virus hev _er__tor__eeeeeeeeeeeeeeee
1,385
28
Bank--ooe--Auhhnniattoo-CCCC
1,372
baseball
1,340
16
Titanic
1,309
13
mncntal-heallt-inntcch-ss
1,259
26
2
hill-valley
1,212
100
Heari-Disease Teert-DDsease-Datttseet (Ctseeett
1,190
11
volcanoes-el
1,183
Airlines- TweeT Sweets-Sentimens
1,097
MiceProtein
1,080
cnae-9
1,080
856
solar Haae
1,058
qsarebiodeg
1,055
41
2
SOCC
1,043
13
rmftsa _sceppdteeaaaaaaalla
1,024
2
4
Continued on next page
33

---

Table 9: The 253 Classification Datasets of the Pretraining Corpus.
Dataset
C
B
T
autoUniv-aul-100
1,000
20
collins
1,000
19
30
credit-g
1,000
20
vowel
990
12
11
989
xd6
973
9
2
tokyol
959
Uc-tac--tee
958
9
Tour-and- craeessCCCsoooee------rrrhhooooo
954
acp-breast-cancer
949
01 [|ill
937
anneal
898
18
Cervical Cancer Risk Factors
858
30
vehicle
846
18
analcatdata authorship
841
70
glioma grading clinical and mutation features
839
23
analcatdata _
797
Begensburg_pedaiieee
780
55
779
12
Diabetes Dataset
768
8
748
4
2
eucalyptus
736
19
breast-w
699
9
2
Australian
690
14
soybean
683
35
19
protb
672
Student Performance
666
11
balance-scale
625
Loan- Loan-Predication
614
11
2
monks-problems-
Serthettc_contro
600
60
1lpd
583
10
micro-mass
571
1.082
20
wdbc
569
arsenic-male-hum
559
4
cylinder-bands
540
34
simmtt-moodd-simuutiioooccssss
540
18
water-trealmeni
527
36
520
16
dresses-Sales
500
12
insh
500
arrhythmia
443
262
10
wholesale-customers
440
vote
435
16
2
cars
406
chronic- chronic-kidney-oeee kidney-disease
400
25
ditterentiated thyroid
canner cancer recurrence
383
16
colic
368
26
2
breast-cancer
286
Qualitativebbnnkuutter
250
6
2
us-
245
audiology
192
57
8
bone marrow twwnsplanntnnhhtte children
187
darwin
174
450
Continued next page
34

---

Table 9: The 253 Classification Datasets of the Pretraining Corpus.
Dataset
C
B
T
lae
151
EgyptianSkulls
150
4
5
lymph
148
18
arcene
100
9,920
2
Table 10: The 93 Regression Datasets of the Pretraining Corpus, with their n examples, m features, presence a benchmark (B) and whether they are textual (2).
Dataset
π
m
T
delays delays_zurich transport
5,465,575
14
New-York- Mow-York-Citt-Bik--
4,500,000
7
USA-Airport-Datasee
3,606,803
14
New-York-laxx-
2,083,778
21
Buzzmsoccalmedizz Twitter
583,250
77
nyc-taxx-groon--cc-2011
581,835
18
b1bK-Hoidd-eevowssDaaaeeee
515,738
16
dionis
416,188
54
Yolanda
400,000
100
Allstate Claims Severity
188,318
130
Football players Fifa stats
183,142
37
black __nda__maa_________
166,821
9
medical medical_charges
163,065
3
Toothall-manager-aall
159,541
87
energy
72,000
32
v1deo transcoding
68,784
18
dating profiee
59,946
30
diamonds
53,940
9
sarcos
48,933
21
phystochemical peaeoochemical_poteinn protein
45,730
9
tried
40,768
10
2dplanes
40,768
10
40,768
10
Perth-House-PPigg Prices
33,656
17
cps&&wages
28,155
6
Ips benchmark
24,624
39
news 2aWNs__pplanty2
24,007
4
house 16H
22,784
16
health insurance
22,272
11
house sales
21,613
21
superconductivity
21,263
81
california _o___hooinnggg
20,640
8
avocado-sales
18,249
Bike_ Shar_SShring Lemand Demand
17,379
12
elevators
16,599
18
Flyyeo-Playerr
14,999
72
miami housing
13,932
15
naval propulsion plant
11,934
14
Brazihan houses
10,692
11
German Jermann-House- House- Prices
10,552
24
sultur
10,081
5
climate change_annnenee impact
10,000
14
ctd__stablitty
10,000
12
Continued on next page
35

---

Table 10: The 93 Regression Datasets of the Pretraining Corpus.
Dataset
B T
Credit-Card- Cat------Daaaett------ttteeeeeee
8,949
16
topo 2
8,885
261
yprop
8,885
212
Scoul bike sharing demand cat
8,760
13
pumadyn32nh
8,192
32
kin8nm
8,192
8
cpu activity
8,192
bank322h
8,192
32
Pollcn-Luxeemborr-19999
7,784
36
colleges
7,063
44
wind
6,574
14
QSAR-THH TID-10980
5,766
1,024
QSAR IID-11
5,742
1,024
Myanmar- Air- Air-Quality
5,122
10
Santander transaction value
4,459
4,735
HouT--HAD--rnnim---errree
4,440
114
Mercedes Benz_GGreener
Greener MMonnnattttuuuinngggg
4,209
364
abalone
4,177
8
pollen
3,848
4
space ga
3,107
6
scotch soood---iiiskkyyeeiiivsvddddddd
2,247
4
quake
2,178
3
auction verification
2,043
7
crime
1,994
126
airfoil_se sell___ee_f_ee_ee noise
1,503
5
house mouse_pricess
1,460
80
house prices nominal
1,460
79
NUA-PLAYYDRS-011-201
1,408
43
Insurance-Premiun-Daaa
1,338
6
Moneyball
1,232
14
socmob
1,156
5
MIP-2016-regression
1,090
116
geographical ongin of music
1,059
116
concrete compressive strength
1,030
8
Housshodd-moothhyeeeeccrt--y--y
1,000
9
stock
950
9
QSAR fish seh_ttxicity
908
6
cars
804
17
energy_efficiimee efffcienccy
768
8
kdd el_mnossmmll
709
8
student performance por
649
30
strikes
625
6
sensory
576
11
meta
528
21
forest fires
517
12
rmttsa ladata
508
10
boston
506
13
no2
500
7
Diabetes(scikittLeams
442
10
NBAA-2200 Si-A--KK0-player-oe playeer-d-atttt
439
14
baseball-hitte
263
22
bodyłat
252
14
Lisbon- House-Pricee
246
13
tecator
240
124
36

---

D
Benchmark Datasets
This appendix elaborates on the benchmark presented in $4 We consider all datasets proposed by AutoML Multimodal Benchmark (SHI) [63] Vectorizing (VEC) [2), and MARTE-Benchmall (CRT) 477. resulting a final set of datasets. We deduplicate datasets that appear as-is in more than one benchmark. In addition, since CARTE explores the concept of multi-table learning, they introduce highly-overlapping datasets for which we remove one variant (see 4.3 and B.2 in their paper).
Table II presents the classification datasets and Table 12 the regression ones. Each table includes an internal ID used for reference, the Dataset name, the number of examples n and of features m, and the number of classes C/ for classification 26 Finally, we also indicate the benchmark sources where each dataset appears. addition, Table 13 presents the full benchmark with a short description per dataset, and Table [4 details the datasets removed during the deduplication process. Most of the excluded datasets are regression datasets from the CARTE BeRTE-Benchmark, because of its high-overlapping nature.
Table 11: The 14 classification datasets of the benchmark, with their examples, m features, C classes, and presence in the SHI, VEC and CRT benchmarks.
ID
Dataset
C
SHI
VEC
CRT
C01
women clothing review
18,788
10
5
C02
us-accidents
7,728,394
42
4
v
C03
data scientist_salan
15,841
6
C04
imdb genre preddition
800
11
2
C05
protuct_sentiment machine hack
5,091
2
C06
google _a__a_______ question type_reeppee reason
4,863
39
5
C07 haabeei---uudd--stttuaaa-------s
17,735
11
5
C08 _ake_iob_posting22
12,725
5
2
C09 @essw_unnteeded bias 100K
100,000
40
C10 relprrvveews-datasel
10,000
5
5
C11 news channel
20,284
17
C12 wine reviews
84.123
5
30
C13 kick starter funding
86,502
9
2
C14
fhhourne_airbn)
18,316
89
10
E
Baselines
In this appendix we first discuss models excluded from the evaluation due data leakage concerns, and then cover implementation details for the baselines used in our main experiments $4
E.1 Excluded Baselines
As opposed to GDBTs or single-dataset deep learning methods, evaluating pretrained tabular models introduces additional complexity. Indeed, leakage can come in multiple forms. When LLMs are involved, there is a risk of memorization [9] and models trained on synthetic datasets (38 which try to mimic real-world distributions, can be unintentionally biased towards popular benchmarks.
While these two forms of leakage are subtle and hard detect, a more direct form must be strictly avoided: When the same dataset (or variant of it) is used during pretraining, and then it is evaluated as a downstream task. In such scenario there inevitable severe data leakage, especially when running with multiple random test splits. The rest of the section explains how both TP-BERTa (84) and CM2 [86) suffer from such contamination with respect to our benchmark. As we briefly mention in (7, we advocate for improving TFM research by encouraging models that are practical to evaluate, by releasing several versions each model, and providing default hyperparameters.
We exclude TP-BERTa from evaluation for two key reasons. First, their imple- mentation assumes that every example is treated as a serialized single sequence, which allows for
TP-BERTa
We treat ranking problems with up 10 discrete values multiclass problems.
37

---

Table 12:
The 36 regression datasets of the benchmark, with their n examples, m features, and presence in the SHI. VEC and CRT benchmarks.
ID
Dataset
n
m
SHI
VEC
CRT
RO1
used-cars cond-cars-daaee wet-dddt-tatdeeeeeee
37,814
112
RO2
second-haid -andd-han---a- mercedes- benz
16,392
7
R03
unimeplaanc-rccommentiiii
14,391
14
R04
MI/IDS-Salaries
119,628
R05
Babies-R-Ue
5,085
12
R06
employee salaries
employee_salaries
9,228
R07
spotuly-racks-datee
114,000
18
RO8
calitomia_house house price
37,951
R09
1fa
19.178
28
R10
colfce-srrap-coffeereieee
2,440
17
R11
Bike Wale
9,003
6
R12
used car- ceat-prices-iie in -nnlcssin--pakistan
72,655
9
R13
bookprice prediction
4,989
8
R14
ae ae_price_ crrlc___reiition
22,662
12
R15
Employee-remuneration
44,574
5
R16
iimty-movies-datasel
41,399
17
R17
Jice-/-mllion-compaayaaaaall
7.173.426
7
R18
museums
22,290
21
R19
viveno-wine-data
8,650
6
R20
wikiliq dataset
12,569
12
R21
seer-prolle-and-rule
3,197
24
R22
korean -drama
1,647
9
R23
videogamesales
16,598
5
R24
comato-bangalore-rettnee
41,665
15
R25
the-movies-datasel
45,460
20
R26
aba-draft-basketha
1,669
22
R27
Goodreads
3,967
14
R28
Rotten- Tomitos
7,158
15
R29
saudi-arabia uusubba--uud--eeee -ssedcarrsddttsel
8,035
12
R30
top-ramen ratings-
4.105
4
R31
31,136
R32
chocolate- coocolate-barraaungg.
1,795
8
R33
mercari price suggestion 100K LOOK
100,000
9
R34
wine- sime-prrceeon--pliis-maae
2,247
18
R35
clear-corpus
4,724
30
R36
Jc s__penne__prooee prodducss
10,860
5
a maximum length of 512 tokens, as elaborated in Appendix A. for datasets with a low amount of features and no free-text presence, around half of the datasets in our benchmark are too long for that limitation, as they either contain too many features or long free-texts. Second, TP-BERTa' 's pretraining uses datasets that appear in our evaluation set, as listed in Table of their paper (84) It is evident that several datasets overlap directly with datasets in our benchmark $4 (e.g.. 1510_fff 1368_ 3368_MMD6-B880 IMDb-Ratings 1639 [439_Melbourde
While this decision is efficient
disqualifying them for our purposes. Furthermore, we observe concerning overlap between their pretraining and downstream task datasets (e-g., airlines, sf police and diabetes). We believe that this questions the validity of their evaluation, and that such contamination poses serious challenge for the TFM community which could be substantially addressed by better tabular data repositories (76).
CM2 CM2 was pretrained over OpenTabs, compilation of more than 2,000 datasets drawn from public tabular data repositories, including OpenML. and Kaggle. While this collection is valuable, pretraining a model over these datasets compromises further evaluation of any of them. Naturally, the overlap with our benchmark here is extremely high, making infeasible to use
38

---

Table 13: Benchmark Datasets Description
ID
Description
C01
Women Clothing E-Commerce Reviews
C02
US Accidents between 2016 and 2023
C03
Indian Data Scientist Salary Prediction
C04
IMDB Movies Genre Prediction
C05
Product Sentiment Analysis
C06
Google QA Question Type Reason Explanation
C07
Michelin Guide Restaurants Awards
C08
Fake Job Posting Detection
C09
Online Social Media Comments Toxicity
C10
YELP Dataset Reviews
C11
News Channel Prediction
C12
Wine Reviews for Variety Prediction
C13
Kickstarter Funding Prediction
C14
Melbourne AirBnB Listings
R01
User cars and listing price in the website Cardekho
R02
Second-hand cars Mercedes Benz price Italy
R03
Anime-Plame Recommendation Database 2020
R04
Salaries of ML/DS Professionals Worldwide
ROS
Prices Prediction for baby product from Babies R Us website
R06
Employee Salary Montgomery County, MD
R07
Spotify Tracks Popularity
RO8
California Houses 2020 Prices
R09
FIFA 2022 Players Wages
R10
Coffee Review Rating
R11
Bike and scooters from bikewale website in India
R12
Used car prices Pakistan 2021
R13
Book Price Prediction
R14
American Eagle Retailer Price Prediction
R15
Employee Remuneration and Expenses- Vancouver
R16
FilmTV movies ataset rating
R17
Company size prediction
R18
General information on the US museums
R19
Vivino Spanish Wine Data
R20
WikiliQ Alcohol dataset (May, 2022)
R21
Tasting profiles and consumer reviews for beers
R22
Korean Dramas
R23
Video Games Sales
R24
Zomato Restaurants in Bengaluru
R25
Metadata of movies released until 2017 for box-office revenues
R26
NBA Draft Basketball Player Data 1989-2021
R27
Books ratings
R28
Rotten Tomatoes Movic Ratings
R29
Saudi Arabia Used Cars Price from Syarah Website
R30
Ramen Ratings
R31
Academic impact for Scientific Journals
R32
Chocolate Bar expert ratings
R33
Mercari Online Marketplace Product Prices
R34
Information about wines the polish market
R35
Readability scores for text passages spanning various genres and time periods
R36 JC Penney Product Prices in Retailer Website
39

---

Table 14: Excluded datasets, with their benchmark origin and reason for removal: (1) Duplicate dataset, (2) Unavailable, for datasets with inconvenient or unavailable hosting outside tabular repositories, and (3) Pretraining, for two (regression) datasets mistakenly used for the pretraining.
Dataset
Benchmark
Reason
Duplicate
google_gp qa answer __ppe
SHI
Duplicate
google qa question_tyy
news_ 2oppupoplrrrty2
SHI
Pretraining
US Presidential
VEC
Unavailable
Journal Influence
VEC
Duplicate
Nurmal-Score-SSS
Buy Buy Baby
CRT
Duplicate
Babies-R-US
Bikedekho
CRT
Duplicate
BikeWale
Journal Score JCR
CRT
Duplicate
Journal-ccoreSJR
Japanese Anime
CRT
Duplicate
aninepllnet-rccommeeattioo
Mydramalist
CRT
Duplicate
korean-drama
Prescription Drugs
CRT
Unavailable
Roger Ebert
CRT
Unavailable
US Presidential
CRT
Unavailable
Used Cars 24
CRT
Duplicate
used-car-prices-in--aasstt
Whisky
CRT
Pretraining
Wine.com
CRT
Duplicate
wine_reviews
WineEnthssiasts
CRT
Duplicate
wine_reviews
baseline. Interestingly, their repository" lists TP-BERTa method trained on a subset of OpenTabs, reinforcing that the leakage is shared between the models.
E.2 Baselines Implementation and Hyperparameters
This section outlines the implementation and hyperparameter tuning strategy used for the baselines reported in $4 While each baseline has its own model-specific preprocessing pipeline, we apply two shared preprocessing steps both TabSTAR (as detailed in Appendix [A.1] and all the baselines: (I) We perform date preprocessing by using skrub's Datetime and (2) Apply a clipped z-score transformation for target variables in regression datasets.
Textual Feature Handling CARTE natively supports textual inputs, and the TabPFN-v2 API clien does well, although its implementation details remain undisclosed. As all other baselines do not natively support free-text features 29 we preprocess these features into fixed-size embeddings using skrub's 's TextEncoder, which internally applies frozen e5-small-v2 encoder to each semantic column. This aligns with the encoder used in TabSTAR, enabling a fair comparison across models. There are, however, two key differences how TabSTAR handles these embeddings: First, the embeddings are specifically finetuned for the task, contributing significantly to its strong performance as shown in $5 and further analyzed in G.2 The second detail that skrub applies dimensionality reduction to 30
dimensions, as proposed by 32 This compressed representation performs comparably to the full embedding space, while offering improved inference efficiency.
Hyperparameter Tuning For the tuned models we use the Optuna package with a budget of hours for every and parallelizing trials 8 CPU cores. We use 5-fold cross-validation and take the best configuration selected based on this mean score. We use it then to retrain the model on the full training data.
50 with random search,
hhtps://gttubb.com wwt.il..ooommhaaaaaaea cww/Chao- Yc/CM2
https://ithuu..co/// com/Priorl abshapplaastaapnn-ellentt
CatBoost includes a built-in text module, but it underperforms compared to dense text embeddings. https:/hhppioo//nnojectootaat
40

---

E.2.1
TabPFN-v2
We run TabPFN-v2 using their API client which supports text features While the intrinsic details of their textual handling remain undocumented, it's reasonable to assume that resembles the processing we apply to GBDTs, as their model leverages ICL and their architecture has textual encoder.
E.2.2 TabICL
We run TabICL using its package with the default configuration. Since TabICL. can only be evaluated on classification tasks, we exclude it from the regression analysis.
E.2.3 TabDPT
We TabDPT using its package with the default configuration. While it achieves statc-of-thes the-art performance on some regression tasks, we encounter severe performance issues on others. Due to this inconsistency, exclude TabDPT from the reported results and in contact with the authors to investigate potential implementation issues.
E.2.4 CARTE
We run CARTE using its package which inherently performs k-fold cross-validation. After consulting with the authors, set for efficiency instead of 10. We do grid search over their recommended learning rates and we take the best-performing variant per dataset split.
E.2.5 RealMLP
We run RealMLP using its official implementation in the pytabkit package Following the advice of its authors, we disable label smoothing and optimize for cross _estsspe_opyyyyy for binary classification and I l-auc_ovr for multiclass classification, and keep all other default hyperparameters. For the tuned version, we follow the hyperparameter search space suggested by (39 as detailed in Table [5]
36
Table 15: RealMLP-Tunel Hyperparameters Search Space
Hyperparameter
Search Space
num cam_emb__ype
Choice((None, PBLD, PI., PLR))
ald_front_scall I
Choice((True, False], p=[0.6, 0.4))
logl((2e - 2, 3e - 1)
p_drop
Choice([O.0, 0.15, 0.3), p=[0.3, 0.5, 0.2)) Choice([ReLU, SELU, Mish])
act
midden_sizes wd
Choice([[256, 256, 256], [64, 64, 64, 64], [512]], p=[0.6, 0.2, 0.2))
Choice([O.0, 2e-2)) log(5(0.05, 0.5) if classification else 0.1
plr sigma
E.2.6 CatBoost
We run CatBoost using the catboost package and run the default configuration suggested by [26 by setting carly_stopping_rote roonnds 50, od poal 0.001, iterations 2000. For the tuned version, we follow the hyperparameter search suggested by [38 as detailed in Table T6]
17
We use v2.0.8, the latest version available at the time of running the experiments. 52 http:////pppp.oog/rooooo/taaaaaa phtpps:/pyypioogg/pooeee/tee
[2.5 x 10-4 5 x 10-4 7.5 10-1, 2.5 10-3, x 10-3 7.5 x 10
37 bans://oppionggpojjeccaaaooos
41

---

Table 16: CatBoost-Tuner Hyperparameters Search Space
Search Space Hyperparameter
learning_ran rate
logu(e
random_strenge _stennghh
U(1, 2.... 203
12_leaf_reg
log
bagging temperature leaf estimation iterations iterations
u(0.0,1.0)
u u(1,2,.... 203 U{100, 101, 4000}
E.2.7 XGBoost
We rum XGBoost using the xghoost package For the default configuration, we follow the suggestion of (26) and use: booster gbtree", sttpllssssppngggo_ _ = 50, n estimators 2000. For the tuned variant, we follow the hyperparameter search space suggested by [38, as shown in Table
Table 17: KGBoost-Tunel Hyperparameters Search Space
Hyperparameter
Search Space
learning_ran rate
logude=
max max_depth subsample
u(1,2,.... 10}
u(0.2,
@isample_bytree
u(0.2,
colsample_by byylevl
U(0.2,
min_ min__hhid____ _hiild_weighhtt
logu(le 16
alpha
logul(e 16
reg lambda
logu((
gamma logll(e
n_estimators
u{100, 101, 4000}
E.2.8 LightGBM
run LightGBM using the lightgbm package and its default implementation. For the tuned variant, we follow the hyperparameter search space suggested by (38), as shown in Table 18
39
Table 18: LightGBM-TTnned Hyperparameters Search Space
Hyperparameter
Search Space
num leaves
U(5, 6, ..... 50}
max_ max_depth learning_rae rate rate
4(3.4.... 20)
logi((e-3,1)
n_ d_essimators
4(50,51, 51,0,.1.... 51.6,,,....... .0000)
min_child__g weight
2,
10-1 1, 10, 102 108,, 104)
subsample
U(0.2,0.8)
colsample bytree
2((0.2,0.8
reg_alpha
reg_ lambda
10,10--..1,,
10, 20, 50, 100)
E.2.9 Random Forest
We treat Random Forest as a weak baseline to establish lower-bound reference for each dataset split. We run it with the sklearn package 40 and use its default configuration with n r_estmmatooss 100.
https///ppp..og//ooooet///ooooop 39 batpss//sikkit-earnn.oo
42

---

Table 19: Classification performance per dataset (up to 10K). Best score before rounding bolded. We report average AUROC with 95% Cls. Models: CARTE (CRT), CatBoost (CTB), TabICL (ICL), LightGBM (LGB), RealMLP (MLP), TabDPT (DPT), TabM (TBM), TabPFNv2 (PFN),
RandomForest (RF), TabSTAR (STR), XGBoost (XGB). Tuned models are marked with a '+'.
ID
CRT
CTB
CTB+
DPT
ICL
LGB
LGB+
MLP
MLP+
PFN
RF
STR
TBM
TBM-
XGB
XGB+
88.4
90.2
90.2
90.7
90.2
90.2
90.5
90.5
89.3
90.2
C01
10.3
10.3
10.4
10.3
10.4
10.4
104
10.5
10.3
10.3
10.3
10.33
1.0.3
10.33
10.4
1.04
97.3
91.4
92.3
95.2
97.2
97.3
96.4
96.6
96.3
97.9
96.8
97.2
97.2
97.6
C02
10.5
10.4
10.6
1..44
10.55
10.5
10.5
1:0.5
1:0.5
1005
1:0.4
1:0.3
10.3
82.5
82.0
81.2
85.4
81.4
79.8
79.9
82.6
82.7
82.4
77.3
83.0
83.3
83,8
80.5
82.1
1003
10.3
0066
1.0.3
1:0.3
10.33
4:03
20.4
1:0.3
4:0.3
4003
1:033
4:0.
1:0.3
4:0.3
2003
83.3
82.4
83.7
85,0
85.1
80.7
85.4
C04
11.
+1.8
+2.88
4:2.3
42.3
4.2.1
13.11
+:2.2
1:5
4:30
+:23
4:311
+2.66
422
42.0
88.5
90.6
90.9
92.2
92.9
88.2
88.4
91.0
91.4
91.2
88.0
91.3
91.1
90.7
88.1
90.3
C08
1-0.9
1.0
1.0.5
4:0.6
1.0.5
1:13
1:0.6
4:0.8
1:0.6
1:0.7
4:1.1
1.0.8
4:0.7
1.0.6
11111
1:066
80,9
81.3
82.6
73.5
74.3
81.9
82.8
75.6
77.5
87.7
73.6
87.0
79.0
79.2
81.5
83.7
C06
+1.5
1.3
+1.1
#:2.2
+1.8
+1.1
+:09
114
+14
4:055
+1.2
1:066
+1.7
11.5
+1.0
+009
90.1
90.3
90.6
89.9
86.7
88.9
89.3
90.0
90.2
89.8
85.1
91.5
91.1
91.1
87.2
89.3
C07
+00.4
+:0.3
-0,3
+0.2
+0.3
+0.4
+04
#0.3
+002
+04
+0.7
+0033
+0.3
+003
#0.6
+0.5
90.8
93.0
93.2
95.1
92_1
91.9
90.1
93.5
92.1
92.1
91.9
94.4
C08
116
+00.9
1.00
+1.4
+0.7
+0077
+:0.9
+0.9
+1.3
+111
+1.
+1.
11.
+1.5
+1.1
+007
82.4
82.6
81.3
84.3
81.1
82.9
82.6
81.3
82.5
76.4
94.0
81.8
83.6
79.4
83.2
C09
11222
+1.0
11144
4:2.5
1-0.8
+:4.1
1:1.0
1166
+122
86.7
87.0
87.2
87.7
85.7
85.9
81.7
87.6
87.6
84.6
89.1
87.5
87.3
85.4
86.8
C10
102
-02
40.2
+002
4:0.3
4:0.3
40.2
+00.2
40.4
4002
1:0.2
+0.3
1:0.3
+:033
+002
78,4
80.7
79.6
81.5
78.9
79.2
80.4
79.6
81.4
76.2
79.2
81.2
81.3
78.2
80.2
C11
10.5
+005
=0.4
2:04
-0055
+005
4004
20.4
+0.5
+0.4
+0.6
+:0.5
4:03
+:0.3
+006
+005
C12
96.0
97.5
91.7
93.8
97.8
96.7
96.7
97.6
97.6
95.3
98.3
97.8
97.9
96.6
97.3
1.0.2
10011
10.11
10..3
10.11
1:0.2
1.0.1
10.1
10.1
1.0.2
1.0.1
1:0.1
1.0.1
100.2
1.01
70,9
73.7
74.1
70.7
74.0
72.6
72.7
73.0
72.4
72.3
71.1
75.0
74.3
74.4
70.2
74.1
C13
10.8
11.1
1.0
100.7
1.0.8
1:100
11.0
100.7
11.2
11.1
1.0.7
1.0.6
1.0.7
10.88
11.11
83.5
84.0
82.4
80.4
81.9
82.3
81,9
83.2
80.1
84.0
84.8
85.4
81.4
83,2
+003
-0.
+0.3
+0.3
4..4
+00
20.5
+0.4
+0.3
+0.3
40.2
+0.3
#0.4
+0.4
F Extended Main Results
In this appendix we provide the main results for the experiment as reported in $5 As elaborated in $4 each model is evaluated on each dataset across 10 splits. Since performance scales vary between datasets, we follow the normalization approach proposed by (38), rescaling all scores to the [o, 1] range. The final reported performance for each model is the average over these normalized runs, and
we compute 95% confidence intervals using the standard normal approximation: j + 1.96 Vn"
F.1 Technical Limitations
As discussed in S TabPFN-v2 is unable to run on 4 datasets: C12, because it is a multiclass problem with more than 10 classes, and CO2, C14 and R01 because they support inference for up to 500,000 cells. Attempts to run the model over subset of the examples led to a significantly worse performance, and thus we decide not to report them to allow fair comparison.
41 Additionally, CARTE is unable to run over 15 of the datasets in the benchmark due to a known bug in their implementation for the Powerrraasformmaionn: which struggles in the presence of features with too less unique values. Furthermore, TabDPT's performance over a few regression datasets is especially low. Since focus classification tasks, exclude this model's performance this task. Finally, ReaLMLP, fails to run on RO1 in the unlimited setting due to memory constraints.
F22 Dataset Level Performance
We report AUROC for classification and R² for regression, with 95% CIs computed over the 10 runs for each dataset. Tables [9 and 20 summarize classification performance datasets with up to 10K and over 10K examples, respectively. Tables and 22 to the same for regression tasks. For conciseness, datasets are referred by their ID from Appendix
43

---

Table 20: Classification performance per dataset (above 10K). Best score before rounding bolded.
We report average AUROC with 95% Cls. Models: CARTE (CRT), CatBoost (CTB), TabICL (ICL), LightGBM (LGB), RealMLP (MLP), TabDPT (DPT), TabM (TBM), TabPFNv2 (PFN),
RandomForest (RF), TabSTAR (STR), XGBoost (XGB). Unlimited models are marked with a "!
ID
CRT
CTB
CTB!
DPT
ICL
ICL!
LGB
LGB!
MLP
MLP!
PPN
STR
STR!
TBM
TBMI
XGB
XGB!
88.4
90.7
90.9
90.5
C01
10.3
1044
10.4
10.3
10.4
10.4
10.4
104
10.3
10.4
10.3
10.3
003
10.3
10.4
10.4
10.3
97.4
98.3
92.3
95.2
96.9
97.3
98.3
96.6
98.5
97.9
98.4
97.2
98.5
97.6
98.2
CU2
10.4
10.5
1005
40.4
1-0.3
82.5
81.2
81.5
85.4
81,4
81.2
79.9
80.4
82.7
83.4
82.4
83.0
83.8
83.8
84.1
82.1
82.3
1:03
1:0.6
40.8
4.0.3
4:0.3
40.3
1:03
4:0.3
4:0.3
4:03
1.03
+:03
=0.5
4:.3
1:04
40.3
40.3
C07
86.7
91.1
91.6
10.4
4-0.3
+004
+-02
+.0.3
4-033
1:044
4:0.3
4:0.2
1044
10.4
4.03
-0.3
+003
4:0.3
10.5
1-0.5
90.8
93.2
93.1
93.2
95.1
95.3
93.2
93.4
91.9
91.9
91.3
93.5
95.1
92.1
93.9
94.4
94.4
C08
11.
1:10
1:1.2
1:144
1.0.7
1:0.6
1:0.9
1:0.9
1:1.3
1:1.
1.1
1.5.5
00.9
1:1.5
4:1.1
1.0.7
1.0.9
82.6
85,9
81.3
84.3
86.3
82.9
K5.6
81.3
84.8
82.5
94.0
96.2
83.6
86.1
83.2
85.5
C09
+1.4
+1.2
+12
+1.0
+111
1.3
#122
+:2.5
#:0.
1111
+0.8
=0.3
#:10
11.3
+122
+122
78.4
80.7
81.8
79.6
81.5
82.3
79.2
80.5
79.6
81.0
81.4
81.0
81.5
82.8
81.4
+0.5
+0.4
+0.4
+04
+005
+0.
+004
+0.5
+005
+0.3
+0.4
+0.5
=0,5
+:.3
+0.4
+0.5
+0.4
96.0
97.8
98.6
98.7
C12
+0.2
+0.1
+0.1
+0.3
+001
+0.1
+0.1
+111
+0.1
#0.1
+0.1
=0.1
+0.1
+0.1
+0.1
+0.1
70.9
74.1
76.7
70.3
74.0
76.7
72.7
75.8
72.4
76.4
72.3
75.0
77.9
74.4
77.6
74.1
76.9
C13
10.88
10.8
40.7
84.0
84.8
82.4
80.
81.3
82.
83.5
83.2
84.3
84.0
85.1
85.4
86.2
83.2
84.2
10.
+0.4
40.3
1033
203
4044
4:03
4044
4033
+0.3
003
10.3
10.3
10.4
104
Table 21: Regression performance per dataset (up to 10K). Best score before rounding is bolded. We report average Rº with 95% Cls. Models: CARTE (CRT), CatBoost (CTB), LightGBM (LGB), ReaLMLP (MLP), TabM (TBM), TabPFNy2 (PFN), RandomForest (RF), TabSTAR (STR), XGBoost
(XGB). Tuned models are marked with '+'.
ID
CRT
CTB
CTB+
LGB
LGB+
MLP
MLP+
PFN
RF
STR
TBM
TBM+
XGB
XGB+
100.0
100.0
100.0
100.0
100.0
99.9
100.0
100.0
100.0
99.8
99.8
99.9
100.0
R01
+0.00
::0.0
+0.0
10.0
-0.0
10.1
10.0
10000
10.0
10.1
10.1
10.1
10.0
98.3
98.1
98.1
98.2
98.2
98.0
98.0
99.0
97.9
98.3
R02
+1.0
10.9
+1.0
+1.0
+1.0
+1.0
+0.9
+1.0
+1.1
1..88
+0.8
+1.1
40.9
71.8
73.8
74.1
72.2
72.4
69.7
70.1
74.9
68.4
70.9
75.3
75.4
69.1
73.7
R03
+00.5
+0.44
+-0.3
+0.4
+.0.4
+-0.7
+-0.8
+0.4
+0.6
+0.8
+0.55
00.5
+-0.5
+0.5
86.2
85.1
85.7
84.6
88.7
90.3
86.1
85.1
81.5
84.0
84.1
85.8
87.5
R04
#7.4
17.1
16.5
16.2
14.4
13.2
15.22
16.8
18.7
#3.1
=3.1
47.5
13.7
93.2
89.5
90.0
87.4
88.2
90.9
91.2
92.7
86.7
93.2
92.1
92.3
87.1
90.1
RO5
10.8
11.3
11.2
11.5
#1.4
11.
11.5
10.9
11.3
11.2
#1.1
#1.1
#1.4
11.3
97.7
98.1
98.2
97.7
97.8
97.5
97.8
98.5
97.2
98.1
97.8
97.9
97.1
97.9
R06
+-0.8
+0..5
+-0.5
+0..7
+00.7
+-0.9
+-099
+0.6
+1.0
+-0.3
+0088
+0.8
+00.7
+0.7
71.6
66.9
63.4
68.9
71.1
67.9
69.6
61.9
68.8
R07
11.3
0.888
+-0.8
+1.0
+1.0
+0.8
11.3
+1.3
+1.2
+1.8
+0.7
+0.9
+1.5
+0.8
93.2
93.0
93.2
92.9
93.0
92.9
92.7
93.9
92.2
92.8
93.4
93.3
92.3
93.1
RO8
10.8
10.7
10.7
10.7
10.7
10.8
10.8
10.7
10.77
10.6
10.77
10.7
10.7
10.7
89.2
89.2
88.7
88.9
88.6
89.2
89.8
88.6
88.8
89.5
89.7
89.3
RO9
10.5
10.6
10.4
10.5
10.5
10.4
10.5
10.5
10.7
10.5
10.77
10.7
10.7
10.5
99.5
99.0
98.9
98.5
98.5
99.3
99.4
99.1
98.1
99.5
98.9
99.1
98.6
99.4
R10
+0.2
+0.6
+0.6
+0.3
+0.3
+022
+0.2
+0.3
+0.3
00.1
=0.7
+0.4
+0.3
+0.2
94.2
94.0
93.6
93.6
94.2
94.4
94.5
92.5
93.9
94.5
94.6
93.3
94.2
R11
+0.9
+0.9
+0.9
+1.0
=1.0
+1.0
+0,9
+1.3
#1.1
+0.8
-0.8
+1.1
+0.8
98.5
98.5
98.3
98.4
98.4
98.4
98.5
98.0
98.3
98.6
98.7
98.2
98.5
R12
10.3
10.2
10.3
10.2
10.3
10.2
10.2
10.3
1...2
10.2
10.2
10.3
10.2
52.8
57.6
58.2
54.7
55.0
55.4
55.1
53.8
51.8
53.4
58.7
58.4
49.0
57.5
R13
122.1
1.2.7
12.66
12.8
42.6
12.5
12.55
12.77
13.55
43.00
11.9
12.1
1.3.3
12.7
96.6
97.3
97.3
97.0
97.0
96.8
96.7
96.4
96.3
96.3
97.1
97.1
R14
10.2
1.0.1
10..1
10.1
4.01
10.1
1..22
1.0.2
10.11
4.0.1
10.1
10.1
1.0.1
10.1
79.5
79.9
77.8
77.7
76.2
77.4
79.4
77.3
78.9
80.6
79.9
76,8
80.3
R15
+0.9
+1.1
+0.9
10.9
11.
11.
+10
1.2
+1.5
-0.7
-0.7
112
+1.0
Continued page
44

---

Table 21: Regression performance per dataset (up to 10K). Best score before rounding is bolded. We report average R² with 95% Cls. Models: CARTE (CRT), CatBoost (CTB), LightGBM (LGB),
RealMLP (MLP), TabM (TBM), TabPFNv2 (PFN), RandomForest (RF), TabSTAR (STR), XGBoost (XGB), Tuned models are marked with a '+'.
ID
CRT
CTB
CTB+
LGB
LGB+
MLP
MLP+
PFN
RF
STR
TBM
TBM+
XGB
XGB+
98.7
98.7
98.7
98.7
98.7
97.6
97.8
98.8
97.7
98.7
97.9
97.9
97.7
97.8
R16
10.0
1...0
10.0
40.0
40.0
10.1
10.1
10..0
40.0
10.1
10.1
10.1
10.1
95.3
94.8
95.2
94.6
95.2
94.4
95.4
95.1
94.2
94.7
96.7
96.7
94.5
95.4
R17
+:2.0
42244
422.0
+2.3
42.0
42.4
+1.9
4.211
42..5
42.3
#1.1
1111
42.6
+1.9
98.1
98.2
98.1
98.0
97.7
97.7
97.4
98.2
98.1
97.5
98.3
R18
+0.4
+0.4
10.4
+0.4
+0.4
+0.5
10.5
+1.3
+0.4
+0.6
+0.3
=0.4
+0.8
+0.4
85.5
85.8
85.2
85.2
82.7
83.3
84.0
85.3
83.8
85.0
85.1
83,9
82.4
R19
+0.8
10.9
+0.9
+0.9
=0.8
11.
#1.1
+0.9
#111
#1.0
=0.9
#1.0
+2.0
96.0
96.2
95.7
95.8
95.3
96.0
93.9
95.7
95.6
95.9
95.8
95.3
96.2
R20
+00.5
+-0.6
+0.5
+0.5
+0.5
+-0.8
+0.5
+1.0
+0.6
40.9
4-0.7
+00.7
+-0.9
+0.5
92.3
92.5
92.6
92.3
92.3
92.0
91.7
93.3
91.8
92.3
91.6
92.1
91.6
92.6
R21
+1.2
+1.1
+1.1
+1.2
+1.2
+1.1
+1.
+1.1
+1.1
+1.0
+144
+1.2
+1.0
+1.0
45.2
45.2
42.4
44.7
43.6
44.4
43.8
39.1
39.7
47.9
48.1
36.1
46.8
R22
+:5.5
+:5.9
+5.6
+5.55
+6..4
+6.3
+5.5
+6.4
45.5
+5.55
+5.4
+7.1
+5.8
85,0
85.4
85.5
84.3
85.5
83.3
85.6
84.8
81.8
83.0
86.4
86.6
83.3
85.1
R23
11.
1.2
11.1
11.2
118
11.5
11.6
11.5
11.6
10.99
10.8
#1.2
11.2
86.0
84.0
85.8
78.8
79.5
84.6
82.8
70.3
79.7
81.8
84.9
85.3
82.9
85.9
R24
10.6
1..8
10.77
10.9
10.9
10.8
10.9
11.6
11.0
11.8
40.9
10.8
10.7
10.9
94.3
95.4
95.3
95.2
95.2
93.2
94.1
86.7
94.8
94.8
94.3
94.5
94.6
95.4
R25
+0.6
0..
+00.5
+0.6
+0.5
+.66
+0.77
+1.0
+0.55
+0.5
+0.44
-0.4
+0.6
+0.5
99.8
99.4
99.4
99.2
99.3
99.9
99.9
99.8
99.0
99.6
98.6
98.7
99.1
99.5
R26
+0.1
+0.1
+0.1
+0.2
+0.2
+011
+0.0
+0.1
+0.2
+0.1
#0.7
=0.8
+0.2
+0.1
81.9
85.2
84.8
83.3
76.7
82.0
84.0
85.5
R27
11.6
11.3
11.4
11.4
11.3
11.5
111.2
11.6
11.3
11.6
11.5
11.7
11.2
11.2
52.5
53.4
53.3
50.0
51.0
52.3
52.6
61.7
46.0
51.5
54.7
55.6
45.3
53.5
R28
12.66
1.2.7
12.88
13.0
12.9
13.22
13.00
12.1
122.7
12.9
12277
12.6
122.3
12.7
94.4
94.4
94.8
94.8
94.3
94.4
95.7
93.0
94.3
94.9
95.0
93.4
94.5
R29
0.77
+0.7
+0.7
+0.8
+0.7
+0.8
+0.6
+1.0
+0.9
+0.8
+0.7
+0,8
+0.8
23.8
22.7
24.6
22.3
23.9
15.7
18.8
20.9
23.5
15.7
22.9
23.4
17.5
25.5
R30
+4.2
#4.1
+3.9
+3.7
+4.2
#3.9
+5.4
14.8
+5.0
#5.0
#4.8
#4.2
13.7
92.1
92.1
92.0
91.6
91.6
91.5
91.5
93.2
89.9
91.7
92.4
92.6
90.3
92.0
R31
+.0.3
1...4
10.44
10.4
4.0.4
+.0.3
10.44
10.3
10.5
1.0.4
100.5
10.5
10..5
10.4
28.8
28.2
25.0
27.3
19.8
19.8
26.2
28.6
19.4
25.8
27.3
22.7
31.6
R32
1.3.3
+.5.8
16.4
+:5.7
18.88
1.5.4
1.5.9
1:5.6
15533
16.0
17.3
16.2
46.6
47.4
47.8
44.8
45.2
40.4
44.6
44.9
40.2
46.0
48.4
48.4
40.3
47.9
R33
11.5
1.7
+-1.7
+1.8
+1.6
+221
11.6
+1.66
+1.66
+1.8
+1.1
1.2
+-2.1
+16
90.9
89.9
88.6
89.1
91.1
90.9
88.9
91.6
R34
+2.5
12.1
12.9
+2.4
#3.0
11.4
11.8
13.1
+2.9
=1.7
118
13.6
12.1
85.9
84.4
84.5
83.7
83.7
83.1
84.2
85.2
81.2
85.7
84.5
84.5
81.6
84,4
R35
-0.6
10.5
10.5
10.6
10.6
10.9
10.8
10.7
10.4
10.8
10.6
-0.5
10.8
10.3
96.3
95.2
95.5
94.7
94.9
95.4
95.7
91.2
94.3
95.7
96.2
96.4
94.4
95.5
R36
+00.9
+09
+0.9
+1.0
+1.0
+111
+0.9
+1.1
+1.0
+0.8
+0066
0.6
+00.9
+0.9
E3
Head-to-head comparisons
We compare the performance TabSTAR against each of the models in head-to-head comparisons. We report win rate, which can be seen as private case of the normalized metric with only two models. We exclude failed runs when comparing to CARTE and TabPFN-v2 Table 23 shows the performance of TabSTAR against all models competing up to 10K examples, for both regression and classification, with 95% Cls over the win rate. Table 24 does the same for
E.4 Running Times and Compute Information
Hardware All baselines are evaluated using single NVIDIA A 100- GPU with 40GB memory, and 8 CPU cores of type AMD EPYC 7742 64-Core Processor. The only exclusions are TabPFN-v2,
45

---

Table 22: Regression performance per dataset (above 10K). Best score before rounding is bolded.
We report average R² with 95% Cls. Models: CARTE (CRT), CatBoost (CTB), LightGBM (LGB),
ReaLMLP (MLP), TabM (TBM), TabPFNv2 (PFN), RandomForest (RF). TabSTAR (STR), XGBoost
(XGB). Unlimited models are marked with '!'
ID
CRT
CTB
CTB!
LGB
LGBB
MLP MI.P
PFN
STR
STR!
TBM
TBM!
XGB
XGB!
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
99.8
100.0
100.0
100.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.1
+0.0
+0.0
+0.0
98.8
98.8
98.2
98.4
98.0
98.4
98.5
98.3
98.8
R02
10.9
+0.9
11.0
+1.0
+1.0
11.0
10.9
11.1
11.0
10.8
10.9
10.9
10.9
71.8
74.1
75.1
72.4
72.7
70.1
71.0
74.9
70.9
72.3
75.4
76.4
73.7
74.6
R03
10.5
10.3
10.4
10.4
10.5
10.8
10.6
10.4
10.8
10.6
10.5
10.5
10.55
10.6
85.1
91.3
84.6
91.0
90.3
91.4
86.1
81.5
91.0
84.1
91.0
87.5
91.3
R04
+7.1
+0.8
+6.2
+1.3
+3.2
+0.8
+552
+8.7
+0.7
#3.1
+0..7
+3.7
+0.6
71.6
67.8
85.2
63.4
70.1
67.4
85.0
62.2
71.1
79.9
69.6
84.0
68.8
86.1
R07
+1.3
+0.8
+0.3
+1.0
+0.5
+1.3
+0.3
1.3
+1.8
#1.8
+0.9
#1.3
+0.8
-0.7
93.2
93.2
93.8
93.0
93.6
92.7
93.5
93.9
92.8
93.3
93.3
93.9
93.1
93.9
10.8
10.7
10.6
10.7
10.5
10.8
100.7
10.77
10.6
10066
10.7
10.66
10.7
10.5
89.2
89.3
89.5
89.2
89.2
89.3
89.8
88.8
89.7
89.3
89.3
89.5
RO9
10.55
10.4
10.4
10..5
10.5
1.0.5
10.5
10.55
1.066
10.7
1.0.4
10.55
10.4
98.5
98.9
98.4
98.9
98.4
99.0
98.5
98.3
98.4
98.7
99.0
98.5
98.9
R12
+0.2
=0.1
+0.2
+0.2
+022
=0.1
#0.2
+0.2
#0.2
+0.2
+0.2
+0.2
=0.2
96.6
97.3
97.8
97.5
96.2
96.4
97.2
97.1
97.5
R14
+0.2
+0.1
=0.1
#0.1
+0.1
+0.2
#0.1
=0.2
#0.1
10.2
#0.1
#0.1
+0.1
=0.1
79.9
86.5
77.7
82.0
77.4
85.7
79.4
78.9
83.9
79.9
87.2
80.3
87.0
R15
+1.11
+00.7
40.9
1.066
11.
10.66
11.0
11.5
11.4
+0.7
1.0.8
+1.0
10.7
98.7
98.7
98.8
98.7
98.8
97.8
98.0
98.8
98.7
98.8
97.9
98.1
97.8
98.0
R16
+0.0
+0.00
4.00
+0.0
+0.0
+..1
+0.11
+0.00
40.0
+0.00
+0.1
4.0.1
40.1
+0.1
95.4
95.3
95.2
97.6
95.2
97.6
95.4
97.5
95.1
94.7
97.6
96.7
97.2
97.7
R17
+2.0
+2.00
+0.6
+2.0
+00.7
+1.9
+0.66
42211
+223
+0.6
+1.1
+-0.7
#1.
+0.6
98.1
98.2
99.0
98.1
98.8
97.7
98.4
93.1
97.4
97.7
98.1
98.7
98.3
98.9
RI8
10.4
+0.4
-0.2
#0.4
10.3
+0.5
10.4
+1.3
10.6
10.6
10.4
+0.5
4044
-0.3
96.4
96.2
96.3
95.8
95.8
96.0
95.6
93.9
95.6
95.3
9558
95.9
9692
96.3
R20
10.5
10.4
10.5
10.5
11.0
10.9
11.0
10.7
10.6
10.5
10.5
85.0
85.5
86.3
85.5
85.8
85.6
86.6
84.8
83.0
85.2
86.6
88.3
85.1
86.1
R23
+1.8
+1.1
+0.7
+1.2
+0.7
+1.5
+0.8
11.6
+1.6
+1.7
+0.8
+0.4
+1.2
+0.8
86.0
97.1
79.5
85.4
82.8
96.5
70.3
81.8
95.5
85.5
97.8
85.9
97.3
R24
+0.6
+0.7
+0.3
+0.9
+0.6
+0.9
+0.44
+1.6
+1.8
+0.77
+0.8
+0.3
+0.9
-0.3
94.3
95.3
95.9
95.2
95.7
94.1
95.0
86.7
94.8
95.1
94.5
95.9
95.4
95.8
R25
10.6
10.5
10.4
10.5
10.5
10.7
10.5
11.0
10.5
10.5
10.4
10.4
10.5
10.5
92.1
92.0
93.0
91.6
92.5
91.5
92.5
93.2
91.7
92.7
92.6
93.6
92.0
92.9
R31
10.3
10.4
10.3
10.4
10.3
10.4
10.4
10.3
10.4
10.4
10.5
10.3
10.4
10.3
46.6
47.8
55.9
45.2
49.8
44.6
54.5
44.9
46.0
57.2
48.4
57.8
47.9
55.7
R33
+1.5
+1.7
+1.1
+1.6
+1.3
+1.6
+1.
116
+1.8
11.
+1.
+12
+1.6
+1.3
96.3
95.5
95.5
94.9
94.9
95.7
95.8
91.2
95.7
95.5
96.4
96.1
95.5
95.5
R36
+0.9
+0.9
+0.9
+1.0
+1.0
+0.9
+0.9
11.1
+0.8
+0.5
10.6
+0,9
+0,9
+0.9
46

---

Table 23: Win rates of TabSTAR (up to 10K) against baselines, Win rate with 95% CI.
Model
Classification
Regression
CARTE
93.
35.7 5.9
CatBoost
80.7 + 6.6
33.6 + 4.9
CatBoost-Tunm
75.0
32.2 4.8
LightGBM
92.1 2..1:0 4.5
49.7
LightGiM-TTuned
90.7
4.1
RandomForest
9..4
66.9
RealMLP
8.41
5.24
ReaIMLP-Tune
83.6
4.2
TabDPT
7.3
6..7
TabICL
7.6
TabM
6.0
39.2 5.0
TabM-Tuned
6.0
3.04
TabPFN-v2
6.9
00.9522
XGBoost
9.14
69.4
XGBoost-Tuned
78.6
30.3 + 4.8
Table 24: Win rates of TabSTAR-Unlimite. (above 10K) against baselines. Win rate with 95% CL. Model
Classification
Regression
CARTE
98.6 + 2.8
63.5 + 7.5
CatBoost-Tuned
9.9
53.0 6.9
77.0
2
LightGBM-Tuned
98.0 2.8
69.5
BighnGBM-Tunne-UUingg
9.3
14.0
RealMLP-Tuned
97.0+3.4
7..0
RealMLP-Tuned MAMML--PTneddUnllmitee Unlimited
6.80
38.9
TabDPT
8..0
TabICL
8..0
TabICL-Unlimited
8..0
TabM-Tuned
7..8
6.72
SLMM-Tuned-Unlimiin
5.80
2..5
TabPFN-v2
9..0
6..1
TabSTAR
94.0
8.3
GGBooost-Tnnee
9.3
59.5
XGBoost-
8.70
28.5
which runs its API version, and CARTE, for which we employ several NVIDIA GeForce GTX 1080 GPUs with 11GB memory, as their high latency and lack of default hyperparameters forced us to rely on more accessible hardware.
Running Times Table 25 presents the average rumning times of the different models per dataset, with up to 10,000 examples, excluding the tuned variants, TabPFN-v2 and CARTE. TabSTAR's average running time for downstream task ranges from 59s (C04) 9,332s (R24). Although higher than single run of other baselines, these times remain far below the 14,400s (4 hours) typically required for their tuned counterparts. TabICL achieves considerably faster running times: however, their ICL-bbseed approach limits them to smaller datasets, and inference is as costly as training (see TabPFN-v2, though unreported, subject the same limitation.
$5).
F.5
TabSTAR's Cost Analysis by Number of Features
While reports average computational costs, these values vary considerably across datasets, depend- ing on factors such as the number of examples, the feature count, and the amount textual fields. In this subsection, we examine how TabSTAR's computational cost scales with the number of features.
47

---

Table 23: Average model training running time seconds, per dataset, for up to 10K examples.
ID
CatBoost LightGBM RandomForest
RealMLP
TabICL
TabSTAR
XGBoost
C01 116
35
48
148
37
397
36
C02 251
57
75
179
58
1,135
58
C03 106
147
49
553
C04 29
17
19
25
20
59
18
C05 26
16
63
13
132
C06 170
79
83
126
77
938
81
C07 163
98
186
80
1,044
C08 89
72
86
180
74
1,057
69
C09
33
128
29
414
C10 49
35
43
126
36
572
35
C11 43
22
29
121
20
521
C12 614
39
37
145
28
729
36
C13 70
55
68
163
55
551
50
C14 1,938
307
333
418
311
4,068
329
RO1 488
161
1,204
368
2,908
161
R02 28
10
45
128
283
10
R03 111
62
282
171
592
61
R04 21
17
138
240
ROS 65
20
68
90
366
23
R06 60
119
129
501
23
RO7 66
39
285
145
673
44
RO8 213
105
951
222
1,448
112
R09 12
8
56
115
604
7
R1O 102
104
255
R11 26
10
45
98
197
10
R12 34
42
133
335
RI3 78
44
145
94
484
46
R14
159
1,296
R15 38
18
102
132
390
17
R16 108
316
193
898
R17 62
54
1,110
160
261
53
RI8 124
1,705
201
553
R19 22
13
38
107
198
12
R20 70
52
401
176
705
49
R21 53
29
86
62
222
30
R22 57
31
62
52
171
33
R23 41
19
137
140
281
18
R24 461
396
650
520
9,332
392
R25 191
104
2,583
223
750
110
R26 29
12
22
30
205
11
R27 103
164
334
R28 175
92
460
158
647
97
R29
100
308
R30 26
14
47
56
114
14
R31 148
505
200
797
R32 20
11
23
31
82
11
R33
161
608
R34 66
22
66
50
122
24
R35
146
101
R36 58
38
223
146
823
39
48

---

Table 26:
Cost Analysis for TabSTAR per number features. Median training and inference times and peak memory usage on GPU and CPU, aggregated across 45 datasets.
Train
Inference
Feature Group
Time (s)
GPU (GB)
CPU (GB)
Time (s)
GPU (GB) CPU (GB)
Up to 10 30-50
85.6
1.9
4.4
0.5
1.6
4.4
228.8
3.1
4.5
1.0
1.7
4.4
100+
417.9
7.8
4.5
2.5
2.3
4.4
Table 26 summarizes both latency and memory consumption of TabSTAR, aggregated over group of up to 10 features, group of 30-50 features and group of more than 100 features. These datasets were randomly sampled from the pretraining corpus; consequently, the reported times may differ from those observed on the benchmark datasets. We observe that latency can increase by factor of 5, and GPU memory consumption by factor of 4. This behavior stems from the transformer's architecture, whose computational complexity is quadratic in the sequence length. This raises concerns for training TabSTAR over datasets with hundreds of features, and calls for improvements to expand to datasets of thousands of features.
Additionally, TabSTAR's memory consumption in classification tasks is affected by the number of classes due its target-aware tokens. We assume that for TabSTAR, the effective number of features equals the actual number of features plus the number of classes. For highly multiclass datasets, this can become limiting factor.
F.6 Number of Trials for Tuned models
The tuned models are optimized with a budget of 4 hours using CPU cores. Each trial is executed on single core, ensuring that at least trial could be completed within the allocated time. Table presents the number of hyperparameter trials per dataset.
G Extended Analysis
In this section we expand on the analysis results discussed 6
G.1 Evaluation Datasets for Analysis
All experiments are conducted on 20 datasets from the benchmark described in Appendix D Each experiment reports performance using AUROC for classification and R for regression. Each tables reports both regression and classification tasks, distinguishable by their ID. Furthermore, each experiment compares its variants to other models, excluding TabPFN-v2 and CARTE, which could not be executed on all datasets. Although this comparison covers only subset of the benchmark and should not be interpreted as conclusive, the results nonetheless offer valuable reference points.
G.2 The Role of the Encoder Unfreezing (Q1)
Table 28 shows the results for each variant of the experiment presented in $ It is evident that unfreezing the textual encoder yields significant performance boost across datasets, seen in Figure The frozen TabSTAR variant is much worse than any other baseline! Furthermore, while finetuning single layer gives significant boost, it underperforms compared to 6 unfrozen layers.
G.3 The Effect of Pretraining (Q2)
We pretrain three TabSTAR variants on nested dataset subsets size 16, 64, and 256. The 64-dataset variant contains the original 16 plus 48 datasets, and the 256-dataset variant builds those 64 by adding another 192. This cumulative design minimizes variance between variants so that performance differences reflect only the effect of increasing data volume.
While LoRA 42 powerful technique, can't be applied to a randomly initialized model. Therefore, perform full finetuning of the non non-pretrained model, explained in Appendix B.2
49

---

Table 27: Average number of hyperparameter trials per dataset. Tuned models: CatBoost (CTB),
LightGBM (LGB), RealMLP (MLP), XGBoost (XGB). Unlimited models are marked with !!.
Dataset
CTB
CTB!
LGB
LGB!
MLP
MLP!
XGB
XGB!
C01
27.1
24.4
770.2
812.0
56.0
26.7
97.7
90.0
C02
18.1
9.4
417.5
51.3
31.9
8.0
47.2
19.6
C03
18.0
15.8
312.0
338.1
37.4
25.2
42.4
39.5
C04
105.8
11283.8
642.1
458.5
C05
122.1
3373.3
98.3
228.3
C06
18.1
406.9
67.7
44.6
C07
17.3
18.5
335.6
376.6
31.8
23.7
53.4
37.7
C08
101.1
67.8
2911.4
2394.6
43.1
37.3
508.3
566.1
C09
169.0 81.3
3969.5
475.7
50.5
9.2
883.9
207.2
C10
73.7
1598.1
55.4
165.2
C11
58.3
48.5
812.4
863.4
44.8
26.0
124.1
96.3
C12
9.9
8.5
207.2
34.4
35.3
8.2
41.8
17.3
C13
85.8
27.4
1848.3
329.6
40.8
8.7
234.6
105.7
C14
8.6
8.4
50.9
36.3
14.9
9.7
13.7
11.6
ROI
14.2
12.2
278.5
169.6
11.3
32.8
15.4
R02
138.9
118.6
4463.0
5214.6
48.0
30.7
742.4
1054.5
R03
55.5
64.2
1033.6
1265.9
35.8
30.0
94.3
97.8
RO4
123.8
31.6
4309.9
714.9
44.1
8.7
1324.0
364.8
RO5
78.8
4428.5
87.9
307.0
RO6
88.8
2305.3
47.5
232.2
RO7
109.9
58.8
1370.0
344.1
42.0
8.5
145.6
72.5
RO8
30.5
30.1
568.6
122.9
21.4
10.4
65.7
47.1
R09
194.6
208.7
6391.0
5975.1
50.4
31.2
589.0
680.6
R10
53.7
2806.9
135.9
241.8
R11
142.9
4669.1
58.7
662.6
R12
126.8
45.6
2990.9
814.7
46.9
8.6
385.8
178.0
R13
80.9
2814.6
91.2
204.8
R14
59.9
67.6
1725.3
1542.6
39.5
15.2
200.6
177.0
R15
107.4
69.2
2896.1
1177.4
52.5
14.0
337.5
242.7
R16
65.8
46.3
1446.3
663.7
34.9
10.9
160.9
104.9
R17
79.7
44.6
1517.3
316.3
45.3
9.1
254.0
104.4
RI8
39.1
43.8
1572.4
1041.4
29.4
16.6
258.8
144.5
R19
117.5
4046.2
63.1
1063.2
R20
62.6
89.1
1611.5
2176.4
36.4
30.1
250.5
258.8
R21
101.6
3284.9
122.3
159.4
R22
67.8
3584.3
259.9
127.2
R23
91.8
105.4
3179.2
2773.0
53.4
27.6
420.1
314.5
R24
52.4
38.1
867.2
417.7
35.2
10.4
104.7
60.3
R25
40.4
33.5
806.9
357.6
23.6
10.8
122.4
72.0
R26
166.5
12567.7
279.4
625.0
R27
53,9
2034.2
85.3
155.5
R28
40.9
670.8
37.5
59.3
R29
135.3
7615.2
63.8
1024.3
R30
194.3
5275.3
114.8
431.7
R31
36.6
39.5
774.5
504.3
26.4
12.6
90.3
54.1
R32
143.2
6605.7
191.7
337.3
R33
69.3
30.8
1053.9
212.0
36.7
8.3
119.8
51.1
R34
69.1
3962.2
148.9
314.9
R35
65.8
3351.6
76.3
139.9
R36
108.0
120.2
2633.9
4634.0
49,9
48.0
289.9
228.8
50

---

Classification Up examples (8 datasets)
TabSTAR
TabM
Unfreeze-6
Unfreeze-3
CatBoost
Unfreeze-9
TabICL
XGBoost
RealMLP
Unfreeze-1
LightGBM
TabDPT
Default
RandomForest
Default
Unfreeze-0
Tuned
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Normalized Score
Figure 6: Comparison of normalized scores with 95% CIs for the encoder unfreezing experiment, comparing TabSTAR variants against baseline models in classification tasks.
Table 28:
Downstream performance for Q1: The Role of the Encoder Unfreezing. Results for 20 datasets with 95% CI, for varying number of unfrozen layers. The top performance score bolded first, and then all scores rounded. We report AUROC for classification and R2 for regression.
ID
0
3
9
C01 87.820.3 90.4+0.4
90.8+0.4
91.0=0.4
90.8#0.4
C02 94.9=1.2 97.8+0.3
97.8+0.3
98.1+0.3
97.7+0.3
C03 77.6=0.9 82.4+0.2
82.8+0.3
83.020.5
83.020.4
CO5 87.0=0.7 90.2+0.9
90.820.6
91.5+0.9
89.4+0.8
C07 12.1 89.1+0.9
90.640.5
91.3+0.4
91.0=0.4
C11 77.4=0.5 78.420.6
78.720.7
78.8+0.5
78.3+0.6
C12 94.4+0.2
98.3+0.1
98.3+0.1
98.3=0.1
98.2+0.1
C13 66.5+0.9 71.7+1.1
72.940.7
74.1=0.7
73.0+0.8
R02 97.0=1.7 97.9+1.2
98.0+1.1
97.9+1.2 98.0#1.1
R03 67.2=0.8 69.810.8
70.940.9
71.240.8 70.8+0.6
ROS 80.8+2.6 91.8+1.5
93.240.5
93.1 92.9+0.8
R09 88.120.7 88.720.7
89.020.5
89.010.6 88.940.8
R12 97.0+0.6 98.2+0.3
98.240.2
98.3+0.3 98.240.3
R13 37.623.6 45.742.1
51.643.2
51.4+2.9
12.1
R18 96.3+1.1 97.140.6
97.410.6
97.3+0.6
97.0+0.6
R23 79.3#2.4 81.6+1.6
82.8+1.6
83.242.2
12.
R27 81.3+1.6 81.3+1.6
81.5+1.6
81.3+1.7
81,4+1.6
R30 14.2#4.6 15.6+5.6
19.2+3.1
19.1#4.0
17.3+4.5
R33 36.0#2.0
43.6#1.5
44.3+1.4
46.0+1.8
43.5+2.2
R34
84.3+2.8 87.4+2.9
88.4+3.4
87.9+3.5
88.7+2.7
Table 29 show the normalized results for classification and regression, and Table 30 shows the dataset level results. It is evident that for most of the datasets, improvement is observed when scaling, with the 256 datasets variant winning for almost all datasets, Figure highlights this effect, showing that adding more datasets enables TabSTAR to outperform the baselines.
G.4 Numerical Verbalization (Q3)
We show the full results for the experiment in $6) with Table illustrating the verbalizations each variant. Note that we do not include exact value verbalization, since it would increase the number of unique text inputs and place extra memory demands. The two variants which integrate numerical information into the verbalization dominate the experiment, although the improvement seems to be
51

---

Table 29: Normalized score with 95% Cls by the number of datasets used during TabSTAR pretrain- ing.
Pretraining Datasets
64
256
Classification
0.352 + 0.086 0.450 = 0.084
0.558 = 0.086
0.786 + 0.076
Regression
0.338 0.073 0.395 0.068 0.642 0.066 0.811 + 0.055
Classification Up to 10K examples (8 datasets)
TabSTAR
Scale-256
TabM
CatBoost
Scale-64
TabICL
XGBoost
RealMLP
Scale-16
Scale-0
LightGBM
TabDPT
Default
RandomForest
Tuned
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.9
Normalized Score
Figure 7: Comparison of normalized scores with 95% Cls for the pretraining effect experiment, comparing TabSTAR variants against baseline models in classification tasks.
marginal for some datasets. Interestingly, some datasets significantly underperform, with the R27 dataset completely failing the task. The addition of the quantile information on top of the bin seems to have limited impact, although marginally winning the average performance. Figure 8 presents a comparison with baseline models, highlighting the importance incorporating richer numerical verbalizations beyond relying solely on the column name.
Classification Up examples (8 datasets)
TabSTAR
TabM
Numerical-Full
Runerical-Range
CatBoost
Numerical-None
TabiCL
XGBoost
ReaIMLP
LightGBM
Default
TabDPT
Default
RandomForest
Tuned
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Normalized Score
Figure 8: Comparison of normalized scores with 95% Cls for the numerical verbalization experiment, comparing TabSTAR variants against baseline models in classification tasks.
52

---

Table 30: Downstream performance for Q2: The Effect of Pretraining, Results for 20 datasets with 95% CL. for varying number of pretraining datasets. The top performance score bolded first, and then all scores are rounded. We report AUROC for classification and R for regression.
ID 0
16
64
256
C01 90.8=0.4
90.7+0.4
90.7+0.3
91.0=0.4
C02 98.0=0.4 97.440.8
97.8+0.3 98.1+0.3
C03 69.248.9 83.1+0.4
83.2+0.3
83.0+0.5
C05 90.7+0.8 90.3+1.1
90.6+0.5
91.544.99
C07 87.920.5 87.6+0.8
90.940.3
91.320.4
C11 78.240.6 77.440.9
78.020.4
78.8+0.5
C12 98.320.1 98.2+0.1
98.2+0.1
98.3+0.1
C13 74.0=0.5 73.5+0.8
73.7+1.0
74.1=0.7
R02 97.2#1.6 97.6+1.3
97.9+1.2
97.9=1.2
R03 67.3#1.9 68.5+1.0
71.2+0.7 71.2+0.8
ROS 88.0+2.8 90.6+2.2
92.2+1.0 93.1+0.7
R09 88.1+1.0 88.4+0.7
88.8+0.5
89.0#0.6
R12 97.720.3 97.940.3
98.1 98.3+0.3
R13 49.4+2.7 49.143.6
48.0+3.0
51.4+2.9
R18 95.0+2.2 94.9+1.2
96.720.6
97.320.6
R23 82.2#1.6 81.2+2.3
81.8+2.3
83.2+2.2
R27 80.9#1.7
81.3+1.6
81.5#1.6
81.3+1.7
R30 13.1+4.1 18.5+4.9
18.5+3.9
19.1+4.0
R33 45.4+2.2
42.8+3.3
45.0#1.5
46.0+1.8
83.8+4.3 R34
86.0+3.3
88.0+3.4
87.9+3.5
Table 31: Illustrative verbalization of numerical feature (Age) for the Q3: Numerical Verbalization
experiment.
Value Name
Name Bin
TabSTAR
17
Age: Numeric
Age: Lower than 18
Age: Lower than 18 (Quantile 0%)
20
Age: Numeric
Age: 18-23
Age: 18 18-23 (Quantile 0-10%)
25
Age: Numeric
Age: 23-27
Age: 23-27 (Quantile 10-20%)
29
Age: Numeric
Age: 27-31
Age: 27-31 (Quantile 20-30%)
33
Age: Numeric
Age: 31-35
Age: 31-35 (Quantile 30-40%)
38
Age: Numeric
Age: 35-40
Age: 35 (Quantile 40-50%)
42
Age: Numeric
Age: 40-45
Age: 40-45 (Quantile 50-60%)
48
Age: Numeric
Age: 45-51
Age: 45-51 (Quantile 60-70%)
55
Age: Numeric
Age: 51-58
Age: 51-58 (Quantile 70-80%)
63
Age: Numeric
Age: 58-67
Age: 58 67 (Quantile 80-90%)
83
Age: Numeric
Age: 67-87
Age: 67-87 (Quantile 90-100%)
93
Age: Numeric
Age: Higher than 87
Age: Higher than 87 (Quantile 100%)
Age: Unknown Value
Age: Unknown Value
Age: Unknown Value
53

---

Table 32: Downstream performance for Q3: Numerical Verbalization. Results for 20 datasets with 95% CI, for different verbalizations. The top performance score is bolded first, and then all scores are rounded. We report AUROC for classification and lt² for regression.
Name
Name + Bin TabSTAR
C01 91.0=0.4 91.2=0.4
91,0=0.4
C02 98.1+0.2 97.9+0.3
10.3
C03 83.3+0.4 83.3+0.3
83.0+0.5
C05 90.8+0.9 91.1+1.22
91.5++..9
C07 91.2+0.2 91.4+0.4
91.3##0.4
C11 78.2+0.5 78.240.6
78.8+0.5
C12 98.240.1 98.4+0.1
98.3+0.1
C13 71.620.7 73.8+0.9
74.1+0.7
R02 98.0+1.1 98.0+1.1
97.9+1.2
R03 67.420.7 71.320.6
71.240.8
ROS 93.1#1.1 93.2+0.8
93.1
R09 88.9#0.5 88.940.6
89.020.6
R12 98.2+0.2 98.3+0.2
98.3+0.3
R13 50.1+3.3 49.743.4
51.4+2.
R18 97.1#0.6 97.1+0.7
97.3+0.6
R23 81.9+2.4 82.7+1.9
83.2+2.2
R27 16.7+6.6
82.0+1.5
81.3+1.7
R30 16.414.4
17.745.3
19.1+4.0
R33 45.621.1
46.2+2.2
46.0+1.8
R34
88.0+3.6
88.6+2.5
87.943.5
54


Reviews:
[{"review_id": "iZ4cQeIQVZ", "reviewer": "Reviewer_Cb5W", "summary": "The paper presents TabSTAR, a novel tabular foundation model that incorporates semantically target-aware tokens into an unfrozen pretrained text encoder, enabling end-to-end fine-tuning of textual and numerical features for tabular prediction tasks. The authors curate a large pretraining corpus of 350 tabular datasets (with both classification and regression tasks), demonstrate strong empirical performance on 50 benchmarks containing free-text or high-cardinality features, and show that TabSTAR scales with more pretraining data.", "questions": "- Could you please clarify the no. of evaluated trials per model and dataset? Also, were the results of the predictions of tree-based models of the five folds averaged (bagging)?\n- Can you provide quantitative evidence isolating the benefit of a) including column names versus solely the values; b) including target tokens instead of just unfreezing the text embeddings using the values?\n- Does the fusion layer provide any gain over simple concatenation? I am unsure whether this part of the modelis important.\n- Figure 4: Looking at the right plot, it seems like the normalization was done solely on the TabStar versions, without the other models. Although the difference to the 0-unfrozen setup is quite large, the true impact on the final downstream performance cannot be assessed using this plot. The same seems to be the case for Tables 3 and 4. How would the plot look like if the other models would be included in the normalization?\n- Table 3: The gains from pretraining seem surprisingly high. WHat are your thought on why TabStar is much worse than conventional neural networks in the 'classic' training regime?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n- Clear motivation & positioning. The paper convincingly identifies the limitations of static, target-agnostic text embeddings in existing tabular ML and foundation models, particularly when handling free-text or high-cardinality categorical features.\n- Extensive related work section covering major approaches for tabular data.\n- Well-written and easy-to-follow.\n- Well-curated pretraining dataset collection and extensive efforts in avoiding contamination.\n- Thoughtful choice of the evaluated baseline models. In particular, the exclusion of models such as TP-BERTa or Tabula, are very reasonable given the severe leakage concerns and unreasonable training times. \n- Extensive appendix with all required information to reproduce the results.\n- Insightful analyses. Ablations on the number of unfrozen encoder layers, the effect of pretraining dataset count, and different numerical verbalizations are very helpful.\n\n- High novelty, as there is no comparable approach that uses \"target-aware tokens\" in tabular tasks.\n- Empirical results are substantial, covering various datasets and providing clear comparative analyses against robust baseline methods.\n\n\n### Weaknesses:\n- Evaluation of baselines using a 4h time budget without reporting the no. of successful trials per model. The average computation times per 10k features in the Appendix are a reference, but without knowing how many trials were truly evaluated it is impossible to tell whether the baselines are well-tuned or not. As they are currently presented, the conclusions are only valid under the 4h time budget. A practitioner would invest much more time to solve a single task. Therefore, the general claim 'TabSTAR achieves state-of-the-art performance' cannot be substantiated. Therefore, the claim in the abstract should either be changed to reflect that the claim only holds with a 4h time budget, or more information on the no. of successful trials per model is required, to verify that the tree-based models were truly tuned close to their full potential.\n- No classic deep learning methods (i.e. TabM or RealMLP) included. \n- Despite being highlighted, computational cost and efficiency are not deeply analyzed. Given that gradient boosting decision trees (GBDTs) are highly performant with lower computational overhead, this factor should be more explicitly addressed. How does TabStar compare to tree-based models in terms of memory efficiency and inference time?\n- Training on some of the benchmark datasets and evaluating five different models adds additional complexity to the evaluation, and, at the same time requires an additional ablation. How is the performance of a single TabStar model without using the additional training data from the downstream task benchmark?\n- While the ablations are insightful, the core innovation that is also present in the full title have not been studied in isolation. Is it really beneficial to use target-aware tokens, or is it simply enough to retrain (unfreeze) the layers without providing semantic target information?", "was_revised": true, "final_justification": "Given the strong submission and rebuttal, I keep my positive score.", "rebuttal": "Dear reviewer Cb5W,\n\nWe appreciate that you recognize the strength of the results, the novelty of our work and the clarity of our paper. We carefully considered your questions and potential weaknesses and would like to address them thematically.\n\n# Baselines\n\n> No classic deep learning methods (i.e. TabM or RealMLP) included.\n\nWe added RealMLP, LightGBM and TabICL. For brevity, we only report the normalized metric in the Classification-10K condition, but a similar behavior occurs in the unlimited one. For regression, the non-tuned based algorithms were below TabSTAR. The full results will appear in the camera-ready version.\n\nTabSTAR (0.82)  \nTabPFN-v2 (0.76)  \nCatBoost-Tuned (0.75)  \nXGBoost-Tuned (0.73)  \nCatBoost (0.7)  \nRealMLP-Tuned (0.63)   \nTabICL (0.61)   \nRealMLP (0.59)  \nLGBM-Tuned (0.54)  \nLGBM (0.51)  \nXGB (0.40)  \nCARTE (0.39)  \nRF (0.16)  \n\nThe added models underperform, as they are bounded to frozen, static embeddings. We believe it's highly unlikely that any non-semantic missing baseline could do something fundamentally better. We would like to emphasize that part of our contribution is not only SOTA performance, but also identifying a gap that hopefully will inspire competing methods to include a semantic approach, as this seems crucial for tabular datasets with text.\n\n# GBDTs Tuning\n\n> Could you please clarify the no. of evaluated trials per model and dataset?\n\nWe apologize for forgetting to report it in the paper and will add to the camera-ready paper the full details. Here is the average number of successful trials for per classification dataset for CatBoost at the 10K condition, sorted as in Table 11:  \n[27.1, 18.1, 18, 105.8, 122.1, 18.1, 17.3, 101.1, 169, 73.7, 58.3, 9.9, 85.8, 8.6].\n\nWe tuned the trees with each trial running on one of the 8 cores, letting it complete at least a single trial and then refitted the best model. For two of the datasets, it took much more than the 4hs budget. \n\n> were the results of the predictions of tree-based models of the five folds averaged (bagging)? \n\nNo, we refitted the model on the best configuration on the whole dataset, following TabPFN-v2 and CARTE. \n\n> the claim in the abstract should either be changed to reflect that the claim only holds with a 4h time budget \n\nPoint taken, we'll refine the statement.\n\n> A practitioner would invest much more time to solve a single task\n\nAgreed, but note that TabSTAR’s downstream performance could also improve, as we train a single fold with no hyperparameter tuning.\n\n>… verify that the tree-based models were truly tuned close to their full potential… \n\nWe understand your concern and we were obsessed with doing things right. We followed leading papers and found contradictions, ambiguity and no public code (until the concurrent release of TabArena). However, since for some datasets the gap is significant even vs TabPFN-v2-API, we believe our results are strong, transparent and replicable (we made the evaluation code public as well). In addition, TabSTAR is not just a model, but a pretraining framework which should improve with further scaling as we suggest in section 6. We thank you for the comments and would love to keep addressing any concerns or questions you might have on this matter during the discussion period.\n\n# Computational Costs\n\n> computational cost and efficiency are not deeply analyzed. \n\nSince we were driven by peak performance, and since TabSTAR could be optimized further with inference efficiency in mind, this was not a main focus. Nevertheless, this request came from multiple reviewers so we took it very seriously. We changed our code to meticulously log CPU/GPU memory and running times for both training and inference time, and we plan to rerun TabSTAR as well as a subset of the baselines. This analysis will complement the results section in the main paper. \n\n> Given that gradient boosting decision trees (GBDTs) are highly performant with lower computational overhead\n\nWhile true in general, we found that for tabular data with text features that require embedding models, GBDTs must use a GPU to hold this claim. Serving embeddings models with CPU-only makes both training and inference orders of magnitude slower. This claim is very important and non-trivial.\n\n> How does TabStar compare to tree-based models in terms of memory efficiency and inference time?\n\nDue to the rebuttal time constraints, we share a partial analysis of TabSTAR vs the best open-source representative for other categories: CatBoost, RealMLP and TabICL. As mentioned above, all baselines used a GPU to be competitive. While TabSTAR is GPU memory-hungry during training (3x more than others), it is memory efficient with comparative memory and latency to GBDTs. On the contrary, while TabICL has virtually no training time, its inference is prohibitively slower (~10x). As the community rapidly adapts ICL as the leading method for doing Foundation Models, the upside of no training time comes with the cost of GPU-based low-latency/high-memory inference. TabSTAR, on the contrary, is inference-time efficient.\n\n# Semantics Analysis\n\n> Can you provide quantitative evidence isolating the benefit of a) including column names versus solely the values; b) including target tokens instead of just unfreezing the text embeddings using the values?\n\nGreat question, we agree that the benefit of including target tokens could have been analyzed in the paper more deeply, although the benefit of including column names worked for many architectures (TransTab, CM2, TpBERTa, CARTE). We conducted an experiment that will be added to the camera-ready paper, and got the following results:\n\nTabSTAR (0.8404)  \nNo Target Values (0.8387)  \nNo Column Names (0.8354)  \n\"Anonymized\": Neither (0.8339)  \n\nWhile the results are on the right track, the effect is moderate; it's worth noting that many pretraining datasets have no rich semantics, so they lower the average effect. We see TabSTAR's ability to thrive even in the absence of semantic information as a strength and a robust property.\n\n> Is it really beneficial to use target-aware tokens, or is it simply enough to retrain (unfreeze) the layers without providing semantic target information?\n\nIt's worth emphasizing that TabSTAR is an architecture destined to scale, and that target-aware tokens should shine with more data, stronger LMs and less tuning. We believe that the further we go that road, the more semantic impact and world knowledge we'll witness.\n\n# Fusion Block\n\n> Does the fusion layer provide any gain over simple concatenation?\n\nIn Appendix A3 we report the gain of the fusion layer over simpler approaches. The exact (unreported) results: \n\nFusion (0.8351)  \nMultiplication (0.8280)  \nConcatenation (0.8241)  \n\nThe concatenation approach causes losing 1% on average. While multiplication is a non-parametric simpler alternative, we believe that the fusion component will scale better and lends itself to multimodality (e.g., adding image features, or richer numerical representations). We show additional experiments in our response to reviewer mFUa, under the header “Fusion vs Interaction”.\n\n# TabSTAR Eval Complexity\n\n> Training on some of the benchmark datasets and evaluating five different models adds additional complexity to the evaluation\n\nWhen training on real datasets, evaluating downstream tasks is a concern due to contamination risks. To mitigate it we released several TabSTAR models, so every pretraining dataset can be matched no a non-contaminated variant, and plan to support this mapping efforts for follow-up research evaluating TabSTAR. We believe that this step is necessary in the era of tabular foundation models, and hope to inspire practitioners, since success in benchmark is de-facto the way models are judged in academic settings.\n\n> How is the performance of a single TabStar model without using the additional training data from the downstream task benchmark?\n\nIt should probably be slightly below TabPFN-v2, but stronger than trees. Growing from 350 to 390 datasets, especially as they are textual, was important to consolidate ourselves as SOTA. We see this as a technical detail, and with a larger pretraining corpus this practice is definitely not mandatory for evaluation, but important when releasing models.\n\n# Normalized Plots\n\n> Figure 4: Looking at the right plot ... How would the plot look like if the other models would be included in the normalization?\n\nWe avoided this comparison as its done on a subset of the datasets, and thus the conclusions could be sensitive when comparing to other models. Nevertheless, we report the ranked variants from best to worse in Classification 10K and plan to include this information in Appendix G, with the appropriate disclaimers:\n\nTabSTAR: Scale 256 / Unfreeze-6 / Numerical-Full  \nNumerical-Range  \nTabPFN-v2  \nScale-64  \nUnfreeze-3  \nNumerical-None  \nCatB-Tuned  \nUnfreeze-9  \nXGB-Tuned  \nCatB  \nUnfreeze-1  \nScale-16   \nScale-0  \nRealMLP-Tuned  \nTabICL  \nRealMLP  \nLGBM-Tuned  \nLGBM  \nCARTE \nXGB  \nRF  \nUnfreeze-0  \n\nWhile we believe that strong conclusions shouldn’t be drawn from this limited comparison, it’s quite clear that both the pre-training scale and the unfreezing of layers are key to TabSTAR’s success. \n\n# Pretraining gains\n\n> The gains from pretraining seem surprisingly high. WHat are your thought on why TabStar is much worse than conventional neural networks in the 'classic' training regime?\n\nTabSTAR’s architecture is designed for transfer-learning without dataset-specific parameters, with components that aim to generalize, which can be a limited decision without pretraining. For example, the numerical encoder maps every (normalized) numerical value to a vector in the same way. Had we allowed for feature-specific mapping (like FT-Transformer) we could probably improve on the non-pretraining scenario, but this won't allow pretraining at scale. Similarly, the Fusion block might be an overkill without pretraining, but shines with enough data."}, {"review_id": "oYY7wSClaa", "reviewer": "Reviewer_1N8m", "summary": "The paper presents TabSTAR, a new foundation model designed for tabular datasets that contain text features. The central idea is to create more meaningful, task-oriented feature representations by making the model \"target-aware.\" This is achieved by feeding the model not just the input features but also tokens representing the possible target values (for classification).\n\nThe architecture involves five main steps:\n\nVerbalization: All features (numerical, categorical, and text) are converted into human-readable strings. For example, a numerical age of 45 might become \"Age: 40-50 (Quantile 50-60%)\".\n\nEncoding: A pretrained text encoder (e5-small-v2) transforms these strings into embeddings. Crucially, some layers of this encoder are unfrozen, allowing them to be fine-tuned for the specific task.\n\nFusion: Embeddings for numerical features are combined with their semantic string representations.\n\nInteraction: A Transformer encoder processes the embeddings of all features and target tokens, allowing them to interact and influence each other.\n\nPrediction: A shared prediction head uses the final representations of the target tokens to output probabilities for classification or a value for regression.\n\nTabSTAR is pretrained on 350 tabular datasets and then fine-tuned for specific downstream tasks. The authors claim that this approach achieves state-of-the-art performance on classification tasks for medium-to-large datasets containing text, outperforming Gradient Boosting Decision Trees (GBDTs) and other tabular foundation models.", "questions": "Novelty: Can you clarify the primary novelty of your verbalization and encoding method compared to prior work like TP-BERTa, which also tokenizes numerical and categorical features for a transformer? \n\nBaselines: Why were other foundation models for tabular data, such as XTab, not included in the comparison? Their inclusion seems critical for validating the claim of state-of-the-art performance in the TFM space. \n\nModel Capabilities: Since the model relies on an embedding model rather than a full LLM, what are the theoretical limits of its reasoning ability? Do you believe the performance gains come solely from adaptive embeddings, or is there evidence of more complex semantic understanding?\n\nRegression Performance: The paper notes that performance lags GBDTs on regression tasks. Beyond scaling the pretraining data, are there specific architectural weaknesses that you believe cause this gap? For instance, is the verbalization strategy inherently less effective for continuous targets?\n\nTarget-Awareness in Regression: For regression tasks, a single target token representing the target's name is used. How does this provide meaningful \"awareness\" compared to the classification case, where all possible target values are provided as input?", "limitations": "The paper rightly acknowledges some of its limitations in the discussion section. Based on the work presented, the key limitations are:\n\nWeak Regression Performance: The model is not competitive with established GBDT models on regression tasks, indicating that the current architecture is not well-suited for predicting continuous values.\n\nDependence on an Embedding Model: The model's capabilities are constrained by the pretrained text encoder it uses. It does not possess the broad reasoning and knowledge integration abilities of a true LLM.\n\nIncomplete Benchmarking: The lack of comparison against a fuller suite of contemporary tabular foundation models makes the SOTA claims difficult to fully validate.\n\nMarginal Gains in Classification: The performance improvements in classification, while present, are not overwhelmingly large, and the overlapping confidence intervals with the next-best model (TabPFN-v2) suggest the advantage may not be statistically robust across all datasets.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nFocus on Textual Features: The model is explicitly designed to handle tabular data with rich text, a common and important type of real-world data that has been historically underrepresented in tabular learning benchmarks.\n\nNovel Target-Aware Architecture: The introduction of target-aware tokens is an interesting architectural choice. It allows the model to learn representations that are specifically tailored to the prediction task and enables an architecture free of dataset-specific parameters, which is a key advantage for a general-purpose foundation model.\n\nSystematic Ablation Studies: The paper provides a good analysis of its key components, showing that unfreezing the text encoder and using detailed numerical verbalization both significantly improve performance. The results also demonstrate that performance scales with the number of pretraining datasets, supporting the foundation model approach.\n\nEmphasis on Reproducibility: The authors provide details on their experimental setup and plan to release pretrained models and code, which is a valuable contribution to the research community.\n\n\nWeaknesses\nLimited Novelty: The core method of converting tabular data into text (\"verbalization\") and processing it with a transformer-based model has been explored in prior work (e.g., TP-BERTa, TransTab). The paper does not sufficiently distinguish its approach to establish clear novelty over these existing methods.\n\nIncomplete Baselines: For a paper claiming to be a new Tabular Foundation Model (TFM), the experimental comparison is missing several important baselines from the same category, such as XTab. The comparison set is primarily GBDTs and TabPFN-v2, which doesn't fully represent the current TFM landscape.\n\nMisleading Framing around \"LLMs\": The model does not use a Large Language Model (LLM) in the conventional sense (i.e., a model with billions of parameters and emergent reasoning abilities). It uses a much smaller text embedding model (e5-small-v2). This limits its ability to perform complex reasoning or leverage the vast world knowledge associated with true LLMs. The improvements likely stem from better-tuned embeddings rather than higher-level reasoning.\n\nUnconvincing Performance Gains: The reported improvements are not consistently strong. TabSTAR is significantly outperformed by GBDTs in regression tasks. Even in its strong suit, classification, the performance gains over TabPFN-v2 are modest, with overlapping confidence intervals in the summary charts, suggesting the difference may not be statistically significant in all cases", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 1N8m,\n\nWe thank you for taking the time and effort, and for the constructive feedback. Some of your suggestions are key to improve the paper. While constructive, we believe we offer more strengths than weaknesses, and we hope to engage in a discussion to convince you so.\n\n# Novelty\n\n> Limited Novelty: ... The paper does not sufficiently distinguish its approach to establish clear novelty ... Can you clarify the primary novelty of your verbalization and encoding method compared to prior work like TP-BERTa?\n\nWhile we rely on some the foundations of TP-BERT, our work provides several strong novelties:\n\n1. We introduce target-aware tokens, a novel component which adds semantic knowledge, enables full parameter sharing and unlocks pretrain scaling. On the contrary, Tp-BERTa specifically adds parameter for every prediction head, a very undesired property when scaling the pretraining corpus to massive scale.\n2. We finetune word embeddings, as opposed to their shallow infra-feature layer which can be seen as a mere pooling that fuses numerical tokens with word embeddings. Note that most of their transformer is equivalent to our interaction component, and that their \"text encoding\" component is very shallow. This seems to be a key aspect of TabSTAR's success (section 6), and we believe Tp-BERTa's architecture is suboptimal.\n3. We suggest a novel mixed verbalization for numerical features, while Tp-BERTa uses special tokens free of semantics which don't exploit parametric knowledge of the LM.\n4. We encode features separately, while Tp-BERTa has a limited context window, being irrelevant for free text (see Appendix E1).\n\nWe hope that these arguments are convincing and we would love to keep discussing. We would also like to mention that both reviewer Cb5W and reviewer qMRw agree with the high novelty of our work.\n\n# Baselines\n\n> doesn't fully represent the current TFM landscape... The lack of comparison against a fuller suite...\n\nWe expand the baseline section to include TabICL, the second-best TFM, as well as RealMLP and LightGBM. We observe that TabSTAR significantly beats these non-semantic approaches. For brevity, we only report the normalized metric in the Classification-10K condition, and leave regression and unlimited results for the camera-ready version:\n\nTabSTAR (0.82)  \nTabPFN-v2 (0.76)  \nCatBoost-Tuned (0.75)  \nXGBoost-Tuned (0.73)  \nCatBoost (0.7)  \nRealMLP-Tuned (0.63)  \nTabICL (0.61)  \nRealMLP (0.59)  \nLGBM-Tuned (0.54)  \nLGBM (0.51)  \nXGB (0.40)  \nCARTE (0.39)  \nRF (0.16)\n\nThe added models underperform, as they are bounded to frozen, static embeddings. We believe it's highly unlikely that any non-semantic missing baseline could do something fundamentally better. We hope you'll consider our efforts extensive and would love to address any concerns regarding the extent of the evaluation. \n\n> Why were other foundation models for tabular data, such as XTab, not included in the comparison?\n\nXTab was deprecated from the AutoGluon package. To our understanding, the model presented in the paper isn't competitive, as written in the paper: *\"XTab outperforms its counterpart FTT in all scenarios thanks to cross-table pretraining, whereas CatBoost is the overall best model.\"* Furthermore, it was not included in the newly released TabArena leaderboard, although its first author is one of XTab's authors and AutoGluon core developers.\n\nWe also tried to include TabDPT, the other foundation model present in TabArena, but its results are surprisingly poor. We contacted their authors to verify the results, and we are collaborating on that matter. We would like to emphasize that we have verified with the authors of ALL the deep models in our paper (TabPFN-v2, CARTE, TabICL, RealMLP) that the baselines were run properly.\n\n# Misleading Framing around \"LLMs\"\n\n> The model does not use a Large Language Model (LLM)... This limits its ability to perform complex reasoning or leverage the vast world knowledge associated with true LLMs. The improvements likely stem from better-tuned embeddings rather than higher-level reasoning... It does not possess the broad reasoning and knowledge integration abilities of a true LLM.\n\nWe appreciate this spot-on observation and analysis, and generally agree with your intuition. We see TabSTAR as both a model and a framework: with enough data and compute, we could replace e5-small-v2 with, for example, Qwen/Qwen3-Embedding-0.6B, making the model ~20x times bigger, and equipping it with much stronger semantic understanding and reasoning abilities. This is possible thanks to our novel target-aware tokens and no dataset-specific parameters, as well as the constant improvement with the increase of the pretraining corpus. We believe that the fact TabSTAR achieves such performance even with limited data and model size is a strength, not a weakness, and we envision future work exploring these areas with industry-level budget pretraining.\n\nNevertheless, we acknowledge that the paper should be very clear and separate between the current capabilities of the released model, and the potential of the framework. It would be highly beneficial if you could pinpoint the sentences you found confusing, wrong or misleading, to ensure they are rephrased accordingly towards the camera-ready version.\n\n# Unconvincing Performance Gains\n\n> the performance gains over TabPFN-v2 are modest ... suggesting the difference may not be statistically significant in all cases...  improvements in classification, while present, are not overwhelmingly large\n\nTabPFN-v2-API is an extremely strong, closed-source model, powered by a commercial company, Prior Labs, and trained over 130M datasets. This very week Amazon released Mitra, publicly released but without the data. We believe that demanding consistent, significant, cross-dataset improvements vs emerging foundation models backed by industry-level budgets trained on much more data, will hinder progress if it becomes a condition for publishing. We believe we should focus on the potential of TabSTAR, which shows remarkable achievements in low-budget regimes. \n\nIn addition, TabPFN-v2 is limited to 10K examples and thus can't compete in realistic dataset sizes, and thus TabSTAR is the definite SOTA for these very realistic use-cases. We limited the comparison to 10K examples exactly to show that we are slightly better than TabPFN-v2 on average even for small-medium datasets, but this is an artificial limitation, derived by their constraints. We see this as a crucial point: foundation models should also focus on bigger datasets, and TabSTAR shows that it scales well.\n\n> may not be statistically robust across all datasets.\n\nWhile TabSTAR doesn't consistently beats TabPFN-v2 across all datasets, Table 18 shows the per dataset comparison with its respective confidence intervals. For some datasets (C10, C13) the gain is of 2-3%. We would like to refer again to TabArena, written by strong leading researchers: \"*We designate TabArena-Lite to be used in academic studies and find any novel model that outperforms other models on at least one dataset, even if it is not among the best on average, a valuable publication*\". We believe that TabSTAR more than fully complies with this expectation.\n\nAs a last point, we want to highlight that our work doesn't contribute only a SOTA model, but also important insights about how virtually all other algorithms miss the potential of Semantic Target-Aware Representations. This is especially relevant as we see new foundation models ignoring semantics whatsoever, and believe that our work will inspire competing models to benefit from our ideas and set new SOTA performance on the important, neglected task of tabular datasets with text features.\n\n# Regression Performance\n\n> The model is not competitive with established GBDT models on regression tasks\n\nWe agree that our regression performance is a point for improvement, and would like to pinpoint that TabPFN-v1 and TabICL were released as classification-only models, and that Tp-BERTa trained separate models for regression and classification. Nevertheless, since TabSTAR consistently improved with more regression datasets and still beats several baselines, we proudly decided to publish these results, and open-sourced our full code, as we want to inspire future research to extend our work and tackle this challenge. \n\n> Beyond scaling the pretraining data, are there specific architectural weaknesses that you believe cause this gap? ... indicating that the current architecture is not well-suited for predicting continuous values.\n\nWe discuss in the paper several potential reasons:\n\n1. Lack of data: we use ~100 regression datasets, this is probably not enough to generalize and transfer learn this complex task. Scaling the training set with millions of synthetic regression tasks (e.g. TabICL's public data) could provide a huge impact.\n\n2. Better numerical modeling: we used a naive single transformation for numerical features, which potentially limits TabSTAR's numerical capabilities. We believe this could be challenged, similarly to how \"On embeddings for numerical features in tabular deep learning\" expanded FT-Transformer. \n\n3. Regression-as-classification: discretizing the target value and using binned (target) tokens, similarly to CHRONOS and other TS FMs.\n\n> Target-Awareness in Regression: For regression tasks, a single target token representing the target's name is used. How does this provide meaningful \"awareness\" compared to the classification case, where all possible target values are provided as input?\n\nDuring pretraining, the model sees different verbalizations per regression datasets, e.g. \"Numerical Target Feature: house price\" and \"Numerical Target Feature: dog age\". This verbalization could help transfer learning similar distributions, and potentially steer the LM to provide potentially useful prior knowledge emerging from its pretraining corpus. This would be boosted when plugging a stronger LLM."}, {"review_id": "13XAH0f4HB", "reviewer": "Reviewer_mFUa", "summary": "This paper proposes TabSTAR, a tabular foundation model that integrates the semantic understanding capabilities of language models. Its core innovation lies in the introduction of target-aware tokens, where all possible target values are included as input tokens, and logits are generated for each of them.", "questions": "1. Please clarify how TabSTAR can be scaled to larger dataset volumes, especially in scenarios where labeled data is scarce. (W1)\n\n2. Provide time and memory usage analysis to assess computational efficiency. (W2)\n\n3. Include a rationale for encoder unfreezing, and describe how the number of unfrozen layers is selected. (W3)\n\n4. Add an ablation study of each component. Also consider evaluating whether the current architecture performs better than a simpler version that just stacks the two modules with equivalent Transformer layers. (W4)\n\n5. Improve the evaluation. Instead of only fine-tuning, test common unsupervised evaluation settings - such as aggregating embeddings and training XGBoost or linear probes on top. Also include comparisons with few-shot or unsupervised tabular learning methods such as SimSiam [1], SCARF [2], STAB [3], MET [4], STUNT [5], LFR [6], and TST-LLM [7]. If finishing these experiments is hard in time, at minimum, the paper should include a methodological comparison and explain why such methods were not used. (W5, W6)\n\n[1] SimSiam: Exploring Simple Siamese Representation Learning (CVPR 2021)\n\n[2] SCARF: Self-supervised Contrastive Learning Using Random Feature Corruption (ICLR 2022)\n\n[3] STAB: Self-supervised Learning for Tabular Data (NeurIPS Workshop 2022)\n\n[4] MET: Masked Encoding for Tabular Data (NeurIPS Workshop 2022)\n\n[5] STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables (ICLR 2023)\n\n[6] LFR: Self-supervised Representation Learning from Random Data Projectors (ICLR 2024)\n\n[7] TST-LLM: LLM-Guided Self-Supervised Tabular Learning With Task-Specific Pre-text Tasks (TMLR 2025)", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The problem addressed in this paper is important, and the high-level direction is well-motivated.\n\n2.The introduction of target-aware tokens is a meaningful and creative idea.\n\n**Weaknesses**\n1. The proposed method relies entirely on supervised pretraining. In practice, this raises concerns about how the model handles datasets without labels. There is a reason that language foundation models typically use next-token prediction with unlabeled data. If TabSTAR depends solely on supervised learning, scalability may be limited by labeled data availability. For example, TabPFN mitigates this issue using synthetic data. It is unclear whether TabSTAR employs a novel strategy to address this limitation.\n\n2. The method may struggle when the number of features grows large (e.g., 100+). As each feature is embedded (e.g., into 128 dimensions), memory usage increases rapidly, especially when applying Transformer layers for both fusion and interaction. This could raise scalability concerns.\n\n3. The paper repeatedly emphasizes the importance of encoder unfreezing, but offers little explanation beyond empirical evidence. A theoretical rationale would help justify this design choice. If such justification is not possible, it may be better to reduce the emphasis. Moreover, if the number of unfrozen layers is treated as a hyperparameter, the paper should clarify whether a principled selection strategy exists, or if it is simply tuned using a held-out validation set.\n\n4. Both the fusion and interaction modules process embeddings through Transformer layers. It is unclear how the paper ensures that fusion occurs during the fusion stage, and interaction occurs during the interaction stage, given the architectural similarity.\n\n5. The evaluation section underexplores the implications of pretraining. Since pretraining is the core contribution, it would be useful to assess the model under different usage paradigms beyond fine-tuning.\n\n6. The set of baselines is limited. As a pretraining-based approach, TabSTAR should be compared against recent few-shot or unsupervised tabular learning methods. Currently, none are included.", "was_revised": true, "final_justification": "After reviewing the authors' rebuttal, I have decided to raise my score. The responses were detailed and addressed most of my initial concerns:\n\n* Framing as Supervised Learning: While the authors’ perspective is understandable, the use of terms like \"pretraining\" made the approach appear close to tabular representation learning, prompting my request for comparisons to related baselines. Although I still find this boundary somewhat unclear, I acknowledge that this issue is secondary given the resolution of other major points. I encourage the authors to clarify this distinction in the paper to avoid potential confusion.\n\n* Scalability: This concern is resolved.\n\n* Encoder Unfreezing: The rationale provided is reasonable, and the authors’ observation regarding training stability would be a valuable addition to the paper.\n\n* Fusion vs. Interaction: This concern is resolved.\n\nOverall, while I retain some reservations regarding the positioning of baselines, all other concerns were thoroughly addressed. The paper has improved in clarity and scope as a result of the rebuttal, and I have updated my score accordingly.", "rebuttal": "Dear reviewer mFUa,\n\nWe thank you for taking the time to review our paper and raise weaknesses and questions, which we took very seriously. While we believe that some of your concerns are spot-on and led to fruitful additional experiments which will make the paper better, some of your questions and weaknesses confused us, and we are afraid that the paper was misunderstood and framed as a paper that belongs to a different research branch, affecting your feedback and rating accordingly. \n\n# TabSTAR: A model for supervised learning\n\n> As a pretraining-based approach, TabSTAR should be compared against recent few-shot or unsupervised tabular learning methods. Currently, none are included. Improve the evaluation ... include comparisons with few-shot or unsupervised tabular learning methods ... explain why such methods were not used. \n\nBefore we specifically address each one of your questions, we want to clarify that TabSTAR is a (pretrained) model used for tabular supervised learning, which is one of the downstream tasks of a higher-level problem of tabular representation. Very relevant papers that are strictly connected to TabSTAR are:\n\n* Why Tabular Foundation Models Should Be a Research Priority\n* TabArena: A Living Benchmark for Machine Learning on Tabular Data\n\nAs well as the papers of TabPFN, TabPFN-v2, TabICL, CARTE and Tp-BERTa. We highlight that NONE of them cites ANY of the 7 papers you mentioned. In addition, none of the other 4 reviewers asked anything that hints that these methods are closely relevant to our work. With that in mind, we will try to answer your questions.\n\n# Evaluation\n\n> Improve the evaluation... include comparisons with few-shot or unsupervised tabular learning methods ... at minimum, the paper should include a methodological comparison and explain why such methods were not used.\n\nSince we only pretrain with supervision, it's unlikely that TabSTAR could be competitive vs models that employ more suitable training objectives, like SCARF or STAB. Accordingly, to the best of our knowledge, this method aren't widely adopted for supervised tabular learning. We'll add to this distinction to the related work section, to make TabSTAR's scope clearer.\n\n# Supervised pretraining\n\n> The proposed method relies entirely on supervised pretraining. In practice, this raises concerns about how the model handles datasets without labels. There is a reason that language foundation models typically use next-token prediction with unlabeled data. If TabSTAR depends solely on supervised learning, scalability may be limited by labeled data availability. \n\nThe pretraining relies on supervised learning as the model is used for downstream tasks that are supervised as well, and thus we don't expect to encounter unlabeled data as a downstream task. Still, it is true that scaling the pretraining might be hard due to lower availability of labeled data, and we write in the discussion: \"*Data scaling might leverage self-supervised learning [59] over large-scale table corpora [17], or realistic synthetic tabular data generators [9], which have proven successful [37, 2]*\".\n\n> Please clarify how TabSTAR can be scaled to larger dataset volumes, especially in scenarios where labeled data is scarce. \n\nFor future work and data scaling, we have a few options, which are not mutually exclusive.\n1. Adding even more labeled datasets, as growing from 400 by a factor of 10x can be possible with Kaggle.\n2. Leveraging unlabeled datasets or tables in the wild by:  \na. Assigning a target variable that could be predicted by the other features.  \nb. Choosing a SSL strategy, for which we will could explore the SSL objectives from the papers you mentioned.  \n3. Creating/leveraging synthetic labeled datasets, involving LLMs due to TabSTAR's semantic nature.\n\n> For example, TabPFN mitigates this issue using synthetic data. It is unclear whether TabSTAR employs a novel strategy to address this limitation. \n\nFrom their paper: *\"Data generation: we define a generative process (referred to as our prior) to synthesize diverse tabular datasets with varying relationships between features and targets ... For each dataset, a subset of samples has their target values masked, simulating a supervised prediction problem.\"*\n\nTabPFN created synthetic datasets, which can easily be employed for supervised learning, and they would be perfect for training TabSTAR, up to the fact that they contain no semantic information. Nevertheless, it is perfectly possible that we could first pre-pretrain over TabPFN's datasets without semantics, and only then continue with semantic real datasets.\n\n# Number of features\n\n> The method may struggle when the number of features grows large (e.g., 100+) ... This could raise scalability concerns.\n\nThis concern is valid, and indeed one of our limitations. As we state in the paper, during pretraining \"*we randomly sample up to 200 features per dataset.*\" and we mention in the discussion that \"*it may struggle with memory constraints on datasets containing hundreds of features*\". Indeed, the problem you mentioned is a common one for transformer based methods, and similarly to other transformer based foundation models, TabSTAR doesn't scale well with lots of features. This could be mitigated by future work investing in reducing the feature dimensionality. \n\n> Provide time and memory usage analysis to assess computational efficiency.\n\nAs requested by other reviewers, we plan to include in the paper an analysis of runtime and memory consumption during both training and inference time. Please see the section \"Computational Costs\" in the response to reviewer Cb5W for more details.\n\nIn addition and per your request, we'll make sure to also break-down the comparison into different groups of number of features. Since memory consumption depends on many factors, we'll need to run a wide variety of datasets for strong conclusions, but we can say that TabSTAR can't really scale above 200 features in a A40 GPU and that time and memory metrics grow by a factor of 3-5x when moving from the 30-50 features bin to the 100-200 bin.\n\n# Encoder unfreezing\n\n> Include a rationale for encoder unfreezing. \n\nSince word embeddings are trained to predict masked or next token, opposite concepts might be very close to each other in the embedding space, but for tabular learning we would like to have a very different geometric structure. Given a categorical feature, like country of birth, we would ideally like to eliminate both the shared information as well as differences that are irrelevant to our target variable. Empirically, we indeed observe that the geometry varies as we keep training, building a new \"correlation space\" that is more suitable for tabular learning. An evidence for this theory is that the average distance of strings dramatically drops during training, from +0.9 for almost any pair to much lower numbers.\n\n> Describe how the number of unfrozen layers is selected\n\nIt seems that this detail is missing from the paper: for downstream, we apply LoRA on the 6 unfrozen layers from pretraining, which selection was explained in section 6, under \"The Role of the Encoder Unfreezing\".\n\n# Fusion vs Interaction\n\n> Both the fusion and interaction modules process embeddings through Transformer layers. It is unclear how the paper ensures that fusion occurs during the fusion stage, and interaction occurs during the interaction stage, given the architectural similarity. \n\nWhile the Fusion and Interaction blocks share architecture similarity, their different roles stem from their input. The Fusion block unifies the semantic and numerical vector into one vector, for each feature independently. This means that for every example, we use this block potentially multiple times. On the contrary, the Interaction block operates over all the elements (features and targets) and is applied only once. De-facto, the first block applies “intra-feature” attention and the second one applies “cross-feature” attention. Note that this means that the effective number of inputs for each block is very different (2 vs num features + target tokens). We acknowledge that this should have been clearer in the paper, and we will rephrase accordingly.\n\n> Add an ablation study of each component. Also consider evaluating whether the current architecture performs better than a simpler version that just stacks the two modules with equivalent Transformer layers.\n\nWe would first want to highlight a pretraining experiment reported in Appendix A3 (\"The Fusion Block’s Mechanism\"). In this experiment we consider simpler approaches to replace the Fusion component. The pretraining validation metric is:\n* Fusion (0.8351)\n* Multiplication (0.8280)\n* Concatenation (0.8241)\n\nThe concatenation approach causes losing 1% on average per dataset. While multiplication is a non-parametric simpler alternative, we believe that the fusion component is more flexible, will scale better and lends itself to multimodality (e.g., adding image features, or richer numerical representations).\n\nFollowing your concern, we conducted a new experiment and suggested two variants that effectively skip the fusion block:\n1. “Numeric Only”: for numerical features, we avoid using any semantic information and go directly to the interaction block with a single vector per element.\n2. “Double Representation”: instead of a separate block for fusing, we pass to the interaction encoder two elements for numerical features. Note that this approach inflates the number of \"features\", potentially by a factor of two, making it a problematic choice in terms of memory. \n\nThe pretraining performance is:\n\n* Fusion & Interaction: 0.8404\n* Numeric Only: 0.8356\n* Double Representation: 0.8249\n\nTo summarize, we claim that the fusion block has a strong rational as well as an empirical validation, and can even be seen as a novel approach. We believe that our arguments could transform this perceived weakness to a strength, and we hope that you agree."}, {"review_id": "9YYjZI8YJ1", "reviewer": "Reviewer_SJjF", "summary": "The paper proposes a tabular foundation model for datasets with textual features. Both numeric and textual features are summarized into verbalizations together with targets and pass through an encoder to build vector representations. These representations are then passed through a second interaction encoder that attends over all inputs. Finally, attended representations of targets are used to apply the regression/classification heads and generate predictions. Results on real-world datasets with textual features indicate that this approach can generalize to unseen tasks.", "questions": "-Is fine-tuning require for each downstream task? Table 3 shows the impact of pre-training but I couldn't find results where full pre-training is done but model is then applied as is without fine tuning. I think these results are important to show.\n-Why is regression performance so weak? Authors say that \"regression is not our main focus\" but there are quite a few regression datasets in pre-training yet the model is not generalizing well after. Analysis on why that happens would be useful to see.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n-The paper is well written and easy to follow. It proposes an innovative approach to represent numerical data through verbalizations of quantile binning.\n-Proposed architecture seamlessly integrates targets without making any assumptions on cardinality and LLM embeddings for textual data with end-to-end fine-tuning.\n-Experiments on real-world data show promising results on generalization against some of the leading baselines.\n\nWeaknesses:\n-Novelty is somewhat limited, other than numeric data verbalizations, the rest of the architecture are fairly standard components that have been explored.\n-Proposed model requires fine-tuning for each downstream task and is not really a \"foundation\" model like TabPFN where no finetuning is needed.\n-Experimental results are mixed, even though TabSTAR has almost 100 regression datasets in pre-training, the regression performance is  quite weak.", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer SJjF,\n\nWe thank you for reviewing our work and for acknowledging our strengths. We found your questions interesting, and we are sure that they will lead to a fruitful discussion. We believe that some of your questions should have been explained more clearly in the paper, and we plan to improve the camera-ready version accordingly. We hope that we can also highlight some of our strengths which perhaps were not noted enough, and turn some of your perceived weaknesses into actual strengths, resulting into an even higher rating of our work.\n\n# Novelty\n\n> Novelty is somewhat limited, other than numeric data verbalizations, the rest of the architecture are fairly standard components that have been explored.\n\nWe humbly believe that our work contributes a few more important novelties, and we would like to highlight them:\n\n1. Target-Aware Tokens: We introduce \"target aware tokens\", a novel component that cleverly introduces semantics about the target values and can build a representation from its early layers accordingly. This component has interesting properties both for ICL approaches, as well as for models such as XTab or Tp-BERTa which used dataset-specific prediction heads, making pretraining in scale infeasible. We would like to emphasize that each one of the other 4 reviewers specifically mentioned that this concept is creative and novel.\n\n2. Semantically Target-Aware Represenations: we prove that TabSTAR could have been a fairly limited model had it used frozen, static embeddings as an initialization point, as done by Tp-BERTa and CARTE, which are semantically-aware foundation models. We believe that this misses the potential and show it even with a small text encoder and a fairly limited amount of data. As a bonus, this actually leads to SOTA performance on classification tabular tasks with text.\n\n3. We are perhaps the only tabular foundation models which puts emphasis on free text, rather than strings. We can note that CARTE shines on categorical, high-cardinality features, and that Tp-BERTa encode examples as sequences. None of these works put multimodal-ish free-text datasets as a research priority, and we believe that true tabular foundation models should be able to properly handle these very common type of datasets which were historically neglected.\n\n4. Our architecture is designed to scale over downstream big datasets, without any special limitations. This is in contrary to the popular ICL approach which shines on small-medium datasets, but struggle as they grow. We believe that we are the only foundation model that exhibits such behavior.\n\nTo summarize, we truly believe this paper has actually high novelty and this weakness should be recognized as a strength. We hope that our arguments make sense and we'd be glad to engage in further discussion.\n\n# Foundation Model Properties\n\n> Proposed model requires fine-tuning for each downstream task\n\nThis is not necessarily a weakness, but a careful design choice with clear upsides. TabPFN and other ICL methods have the magical property of working \"out of the box\" without training, as they don't do parameter update. However, this comes with the price of having limited context size (and thus can process up to 10,000 examples). While some engineering efforts led to TabICL being able to process 100,000 examples, this method would likely not be able to scale to millions of examples. In addition, inference is much more complex and comes with a huge price of latency/memory, as every example attends the whole dataset.\n\nWe believe that this approach is not necessarily the right one, and definitely not the only valid one. Our model does finetune, yes, but its inference is similar to GBDTs in terms of latency, which can be a crucial aspect in realistic scenarios. While our training is more expensive, for most datasets we can efficiently tune it thanks to the usage of LoRA, and the fact that we achieve out-of-the-box SOTA performance without tuning hyperparameters.\n\n> is not really a \"foundation\" model like TabPFN where no finetuning is needed. \n\nWe believe that this statement is not accurate. \n\nThe term Foundation Model coined by Bommasani et al. in \"On the Opportunities and Risks of Foundation Models\", explicitly defines it in their first paragraph: *\"A foundation model is any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks; current examples include BERT\"*. This message is supported in the paper \"Why Tabular Foundation Models Should Be a Research Priority\".\n\nWhile we are used to having foundation models that require no finetuning in other modalities, it's unclear whether this property is a requisite for Tabular Foundation Models, especially as we are still in early days. \n\n> Is fine-tuning require for each downstream task? Table 3 shows the impact of pre-training but I couldn't find results where full pre-training is done but model is then applied as is without fine tuning. I think these results are important to show. \n\nTabSTAR indeed requires finetuning per downstream task, and this is a feature, not a bug. We believe that updating the parameters of the network makes it more practical for inference purposes and purposely avoided using ICL. Having said that, we believe that future versions of TabSTAR, being trained with 130M+ datasets like TabPFN and potentially leveraging larger language models, could potentially be flexible enough to both support ICL (for very small datasets) and do finetuning (for large ones), but we leave this as an exciting future area of research.\n\nWe are open to hear your thoughts about this matter, and we believe that this approach should have been presented with more emphasis in the paper, and will emphasize this at the beginning of the \"Tabular Foundation Models\" paragraph in section 2. \n\n# Model Performance\n\n> Experimental results are mixed\n\nWhile the current model of TabSTAR underperforms in regression tasks (for now), we respectfully believe that this should not shadow that the model achieves SOTA for classification, and these are excellent results. We believe that foundation models should be able to tackle both tasks, although we could easily publish a classification-only model, like TabPFN-v1 or TabICL. We had the courage to publish the regression results, and we believe but this shouldn't stand against our work.\n\n> even though TabSTAR has almost 100 regression datasets in pre-training, the regression performance is quite weak.\n\nWe would like to remind that TabPFN-v2 was trained over 130M+ datasets, carefully generated to simulate a wide range of numerical distributions. Pretraining over 100 datasets is a very modest decision, constrained by both budget and data availability. We believe that given the right data, TabSTAR could be scaled, as its architecture lends itself to that and shows consistent improvement, also for regression tasks.\n\n> Why is regression performance so weak? Authors say that \"regression is not our main focus\" but there are quite a few regression datasets in pre-training yet the model is not generalizing well after. Analysis on why that happens would be useful to see.\n\nWe acknowledge that this is a weakness and area for improvement. We discuss in the paper several potential reasons:\n\n1. Lack of data: we use ~100 regression datasets, this is probably not enough to generalize and transfer learn this complex task. Scaling the training set with millions of synthetic regression tasks (e.g. TabICL's public data) could provide a huge impact.\n\n2. Better numerical modeling: we used a naive single transformation for numerical features, which potentially limits TabSTAR's numerical capabilities. We believe this could be challenged, similarly to how \"On embeddings for numerical features in tabular deep learning\" expanded FT-Transformer.\n\n3. Regression-as-classification: discretizing the target value and using binned (target) tokens, similarly to CHRONOS and other TS FMs.\n\nTo summarize: yes, TabSTAR's performance in regression is not great, but this is only an exciting area for future research. It's unlikely to expect a new architecture at small scale to shine for every problem. To demonstrate that, we would like to quote the \"Conclusions & Future Work\" section of TabPFN-v1, which we believe is super-transparent about its weaknesses and limitations, and that papers should be valued for that spirit:\n\n*\"The TabPFN still has important limitations... which motivates work on (1) scaling up to large datasets.. (2) improved handling\nof categorical features... (3) missing values... (4) robustness to unimportant features... (9) regression tasks\"*\n\nWe hope that we have convinced you that our performance is a strength, and not a weakness, and are open to discuss it further."}, {"review_id": "xtDjXs6VqC", "reviewer": "Reviewer_qMRw", "summary": "The paper introduces a new model called TabSTAR. The model achieves competitive\nperformance compared to GBDTs, neural nets and in context learner models on\nboth regression and classification tasks. This is done via a two phase\npre-train and finetune process. In terms of architecture, an unfrozen language\nmodel is used, with layers for fusion, interaction and prediction. A novel\napproach is the use of target-aware tokens that enable the model to work with\nany number of classes. Finally, the paper justifies the architecture choices\nthrough experiments.", "questions": "- Are the final models trained with 6 unfrozen layers for the text encoder?\n- The NLP community has been using LoRA for finetuning, why the authors choose\n  to unfreeze the text encoder instead? Note that LoRA is already used for\n  finetuning stage. Figure 4 shows that 9 unfrozen layers underperforms 6\n  unfrozen layers. It seems like overfitting is a concern (validation is lower\n  but test is not). Note that the amount of text here is tiny compared to LLM's\n  pretraining dataset and things like catastrophic forgetting can be\n  concerning.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Quality\n\n- The experiments are thoroughly conducted with lots of results in the\n  Appendix that show competitive performance and justify many choices\n  made for training the model. Furthermore, the Appendix highlights some\n  efforts that did not lead to significant results, which will help the\n  community explore the space more efficiently.\n- The paper's claim about being a foundation model is not fully justified.\n  Foundation models such as TabPFN, TabDPT, and TabICL achieve their results\n  without finetuning on each dataset. This paper, however, uses five models\n  that are separately trained (contrary to the foundation model claim) and then\n  finetunes on each dataset. A fairer comparison would have been to use one\n  model and still finetune on each dataset but with baselines using\n  inference-time calibration (as done in LocaLPFN). Alternatively, more\n  emphasis should be put on the inference-time compute required for each model,\n  and that section should be in the main text, not the Appendix (F.3).\n\n## Clarity\n\n- The paper reads well and explains the ideas clearly and justifies most of\n  them with experiments.\n- The authors could touch upon the relation between number of classes and\n  performance and efficiency, since target-aware tokens scale with the number of\n  classes. This seems important as when comparing training dataset to\n  evaluation dataset, one observes the training datasets have many examples\n  with large number of classes while evaluation dataset (on which computational\n  cost/performance analysis is done) lacks such datasets.\n- In the Appendix A.3 Row-Level Attention is discussed. It seems like the\n  authors tried in context learning approach. Although, the details are\n  missing. What was the number of in context samples? Did the authors see any\n  sign of overfitting as reported in TabDPT (which motivates their use of SSL\n  methods)? This is a tangent in the Appendix and I am not raising this as a\n  negative point about the paper. But I believe this approach is of interest to\n  the community and adding the details to the Appendix would be valuable.\n\n\n## Significance\n\n- The model trains fast (<48h on A40), opening the door for community training\n  their own variant.\n- TabSTAR is good on classification tasks (possibly due to verbalization and\n  the text encoder and target-aware token being more helpful here) but not as\n  good in regression, even compared to TabPFNv2 which doesn't have per dataset\n  finetuning (on less than 10K setting).\n\n\n## Originality\n\n- The architecture is novel.\n- The method used to mix the verbalization with numeric values is novel.\n- Target-aware tokens are novel.", "was_revised": true, "final_justification": "I originally liked the paper and gave it rating 5. After discussion with authors and reading other reviews I still like the paper and maintain my score.", "rebuttal": "Dear reviewer qMRw,\n\nWe appreciate your thorough review and the hard work, as well as the warm feedback. We believe that some of your points are highly valuable and will contribute to the camera-version paper. We especially appreciate that you've gone thoroughly over the additional experiments described in the appendix, and that you found them insightful and worthy. \n\n# Foundation Model Property\n\n> The paper's claim about being a foundation model is not fully justified. Foundation models such as TabPFN, TabDPT, and TabICL achieve their results without finetuning on each dataset. \n\nThe term Foundation Model coined by Bommasani et al. in \"On the Opportunities and Risks of Foundation Models\", explicitly defines it in their first paragraph: \"A foundation model is any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks; current examples include BERT\". This message is supported in the paper \"Why Tabular Foundation Models Should Be a Research Priority\".\n\nThis is not necessarily a weakness, but a careful design choice with clear upsides. TabPFN and other ICL methods have the magical property of working \"out of the box\" without training, as they don't do parameter update. However, this comes with the price of having limited context size (and thus can process up to 10,000 examples). While some engineering efforts led to TabICL being able to process 100,000 examples, this method would likely not be able to scale to millions of examples. In addition, inference is much more complex and comes with a huge price of latency/memory, as every example attends the whole dataset.\n\nWe believe that this approach is not necessarily the right one, and definitely not the only valid one. Our model does finetune, yes, but its inference is similar to GBDTs in terms of latency, which can be a crucial aspect in realistic scenarios. While our training is more expensive, for most datasets we can efficiently tune it thanks to the usage of LoRA, and we are faster than GBDTs as we offer out-of-the-box SOTA performance without tuning hyperparameters.\n\n> This paper, however, uses five models that are separately trained (contrary to the foundation model claim) and then finetunes on each dataset. \n\nThis is inaccurate, and we were probably not clear enough. Every downstream task is only finetuned once, using only one of the 5 pretrained models. We write in the paper:  *\"To maximize the value of cross-dataset learning, instead of pretraining TabSTAR once, we create five dataset splits. Each variant is pretrained on the 350 pretraining datasets and 40 of the benchmark datasets, while the other 10 serve exclusively as its test set.\"*.\n\nWe understand the confusion and we probably not clear enough, and also our evaluation setup is indeed a bit complex. For some additional details, you could read the section \"TabSTAR Eval Complexity\" in our response to reviewer Cb5W\n\nAs a side note, please note that TabSTAR is actually underoptimized during training/inference time: both tuned baselines, TabPFN-v2 and TabICL, fit more than one model and average predictions. We consider this optimization worthy, and definitely necessary for real-world production use-cases, but we believe that foundation models should be easy to use and we preferred to avoid such internal optimizations to make the model as light as we can for future benchmarking.\n\n# Inference-time compute\n\n> Alternatively, more emphasis should be put on the inference-time compute required for each model, and that section should be in the main text, not the Appendix (F.3).\n\nWe thank you for this comment and it will be well addressed, and we will include a thorough analysis as part of the main text. To justify our rationale, we would add that we were driven by peak performance, and that TabSTAR could be optimized further with inference efficiency in mind. Nevertheless, this request come from multiple reviewers and we took it very seriously. We changed our code to meticulously log CPU/GPU memory and running times for both training and inference time. This analysis will complement the results section in the main paper.\n\nDue to the rebuttal time constraints, we can only share a partial, qualitative analysis of TabSTAR vs the best open-source representative for other categories: CatBoost, RealMLP and TabICL. We present here our two main conclusions so far:\n\n- GPU vs CPU: We found that for tabular data with text features that require embedding models, GBDTs must use a GPU to be competitive in terms of latency. Serving embeddings models with CPU-only makes both training and inference orders of magnitude slower. This claim is very important and non-trivial, as it narrows the gap from more efficient methods.\n\n- While TabSTAR is GPU memory-hungry during training (3x more than others), it is memory efficient with comparative memory and latency to GBDTs. On the contrary, while TabICL has virtually no training time, its inference is prohibitively slower (~10x). As the community rapidly adapts ICL as the leading method for doing Foundation Models, the upside of no training time comes with the cost of GPU-based low-latency/high-memory inference. TabSTAR, on the contrary, is inference-time efficient. \n\n# Multiclass performance\n\n> The authors could touch upon the relation between number of classes and performance and efficiency, since target-aware tokens scale with the number of classes. \n\nValid concern. Since this problem is very similar to the \"multiple features\" scenario as pointed by reviewer mFUa, we analyze your question with that mindset. If you consider it insufficient, we can specifically benchmark number of classes as well for the camera-ready paper.\n\nWe performed a partial analysis over groups of \"<10\", \"30-50\" and \"100-200\" features. We observe that transformer based models scale worse, a problem shared by both TabSTAR and TabICL. It seems that metrics increase up to 5x times in a multi-feature scenario, and become infeasible or too costly with 200+ datasets for TabSTAR. While this is a problem for virtually all foundation models (as they all use transformers), we agree that TabSTAR is also sensitive to the number of classes, and this is an additional limitation we should mention in the paper, and future work material.\n\n# Row-level Attention\n\n> In the Appendix A.3 Row-Level Attention is discussed. It seems like the authors tried in context learning approach. \n\nFor a long time we explored using row-level attention as presented in SAINT, i.e. batches of samples of the dataset while still doing finetuning. We never took the ICL path. While we believe that this direction is interesting, and especially appealing for small datasets, we wanted to put emphasis on big datasets to show that foundation models should scale.\n\n> What was the number of in context samples? Did the authors see any sign of overfitting as reported in TabDPT (which motivates their use of SSL methods)?\n\nAnecdotically, and here we rely on memory only as this direction was neglected, we did see signs of overfitting during pretraining, something relatively rare when shuffling between many datasets over different samples of the data. We experimented with batches of very different sizes both during pretraining and finetuning. Eventually, as we've seen limited value and too much overhead (overfitting, unclear batch size during both phases, unclear inference method) we decided to leave this path completely, and we currently believe that finetuning with row-level attention is too complex, and thus was probably not widely adopted: it should be either IID finetuning like TabSTAR, or pure ICL like TabPFN et. al.\n\n# Regression Performance\n\n> TabSTAR is good on classification tasks ... but not as good in regression, even compared to TabPFNv2 which doesn't have per dataset finetuning (on less than 10K setting).\n\nAlthough this is not explicitly stated as a weakness, and although we believe that this is definitely an area for improvement, we would like to highlight that TabPFN-v1 didn't support regression tasks and that TabPFN-v2 was trained over 130M+ datasets, carefully generated to simulate a wide range of numerical distributions. We believe that given the right data, TabSTAR could be scaled, as its architecture lends itself to that and shows consistent improvement, also for regression tasks.\n\n# Frozen Layers\n\n>Are the final models trained with 6 unfrozen layers for the text encoder?\n\nDuring pretraining, the final models unfreeze the 6 top layers. During finetuning, LoRA is applied to this layers. We neither touch the other bottom 6 layers, nor the word embeddings.\n\n> The NLP community has been using LoRA for finetuning, why the authors choose to unfreeze the text encoder instead? Note that LoRA is already used for finetuning stage. \n\nThe term \"unfrozen\" is misleading and our fault, we'll rephrase. During pretraining, these 6 layers are completely unfrozen. During finetuning, we apply LoRA, so the pretrained weights are actually frozen. We used the term unfrozen to differentiate them from the bottom 6 layers that remain completely frozen without LoRA, of from other methods that don't tune (in any way) these embeddings. But this was unclear, thanks!\n\n> Figure 4 shows that 9 unfrozen layers underperforms 6 unfrozen layers. It seems like overfitting is a concern (validation is lower but test is not). Note that the amount of text here is tiny compared to LLM's pretraining dataset and things like catastrophic forgetting can be concerning.\n\nVery valid point. We believe that with more data we'll pretrain the whole text encoder, and that when using a stronger base language model, we'll need to add stronger regularization. This is aligned with some of our thoughts for future works, thank you!"}]

Author rebuttal:

Meta-review: This paper presents a new architecture for a tabular foundation model that uses target aware tokens and Semantic Target-Aware Representations. It verbalizes all features into readable strings, encodes them with a pre-trained text embedding model, processes them with transformers, and finally uses a prediction head fine-tuned for a given dataset on the representations. Compared to existing tabular foundation models and non-foundational architectures, TabSTAR performs well on classification tasks (but poorly on regression).

The reviewers initially raised many different issues, including: limited novelty; debate on what counts as a foundation model (i.e. should fine-tuning be a requirement or allowed at all); the poor results on regression; scalability especially with number of classes and features; incomplete baselining; and missing evaluations on cost and efficiency.

Many of these issues were resolved in the discussions. Novelty was more clearly described in terms of target aware tokens and Semantic Target-Aware Representations. Scalability was discussed in comparison to other transformer-based TFMs. The authors presented additional baseline methods, both foundational and not (RealMLP, LightGBM, TabICL), although several relevant models could still be added (e.g. TabDPT a foundation model that shows strong regression performance on TabArena, TabM a non-foundation model with strong classification performance on TabArena).

One of the biggest outstanding issues is poor regression performance, which was mostly left for future improvements. However, a new foundation model that works well for classification can still be interesting on its own, as many tabular papers focus entirely on classification.

Hence, I am recommending to accept this submission. The authors must complete the baselining of their approach on the methods discussed with reviewers (atleast RealMLP, LightGBM, TabICL, TabDPT, TabM), and include the other improvements that resolved reviewers’ issues. Authors -- please make a comment when your camera-ready version is up with the completed baselining for I and the reviewers to check.
Decision: Accept (poster)