Title: TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields
Venue: NeurIPS 2025
Accepted: yes
Keywords: tabular data

tabular foundation model

deep learning
Abstract: While deep learning has achieved remarkable success across many domains, it has historically underperformed on tabular learning tasks, which remain dominated by gradient boosting decision trees. However, recent advancements are paving the way for Tabular Foundation Models, which can leverage real-world knowledge and generalize across diverse datasets, particularly when the data contains free-text. Although incorporating language model capabilities into tabular tasks has been explored, most existing methods utilize static, target-agnostic textual representations, limiting their effectiveness. We introduce TabSTAR: a Tabular Foundation Model with Semantically Target-Aware Representations. TabSTAR is designed to enable transfer learning on tabular data with textual features, with an architecture free of dataset-specific parameters. It unfreezes a pretrained text encoder and takes as input target tokens, which provide the model with the context needed to learn task-specific embeddings. TabSTAR achieves state-of-the-art performance for both medium- and large-sized datasets across known benchmarks of classification tasks with text features, and its pretraining phase exhibits scaling laws in the number of datasets, offering a pathway for further performance improvements.

TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields

Abstract

While deep learning has achieved remarkable success across many domains, it has historically underperformed on tabular learning tasks, which remain dominated by gradient boosting decision trees. However, recent advancements are paving the way for Tabular Foundation Models, which can leverage real-world knowledge and generalize across diverse datasets, particularly when the data contains free-text. Although incorporating language model capabilities into tabular tasks has been explored, most existing methods utilize static, target-agnostic textual representations, limiting their effectiveness. We introduce TabSTAR: a Tabular Foundation Model with Semantically Target-Aware Representations. TabSTAR is designed to enable transfer learning on tabular data with textual features, with an architecture free of dataset-specific parameters. It unfreezes a pretrained text encoder and takes as input target tokens, which provide the model with the context needed to learn task-specific embeddings. TabSTAR achieves state-of-the-art performance for both medium- and large-sized datasets across known benchmarks of classification tasks with text features, and its pretraining phase exhibits scaling laws in the number of datasets, offering a pathway for further performance improvements. 1

1 Introduction

In recent years, deep learning has profoundly reshaped research and practice in computer vision [50, 66, 33, 17] and natural language processing [56, 5, 79, 16, 13]. This transformation was notably accelerated by the rise of foundation models [8, 88, 4], capable of cross-modal understanding and generalization from massive pretraining across heterogeneous data sources. Importantly, they enabled an end-to-end approach that outperformed previous modular alternatives [71, 1]. Moreover, deep learning models excel at transfer learning [91], generalizing from their pretraining data to new tasks. Their strength, combined with techniques like In-Context Learning (ICL) [13] and Parameter-Efficient Fine-Tuning (PEFT) [41], has enabled rapid adaptation to new tasks with only limited labeled data.

Despite this progress, deep learning has historically lagged behind gradient-boosted decision trees (GBDTs) on tabular data [12, 15, 60], in both classification and regression tasks [65, 11, 31, 55]. The heterogeneity of tabular data, which lacks the spatial locality of images or the sequential order of text, makes it more challenging for deep models to learn. Consequently, GBDTs have remained the de facto standard for tabular learning, offering strong out-of-the-box performance, computational efficiency, and built-in inductive biases (e.g., robustness to skewed feature distributions and automatic feature selection) that make them especially well-suited to heterogeneous datasets [31]. Nonetheless, GBDTs cannot be pretrained to reuse strong representations for downstream tasks. This limitation becomes critical in low-data settings like those often found in healthcare applications [52]. Crucially, they must rely on external embedding models to process unstructured data types like text and images, yielding fixed feature representations that cannot be finetuned for a specific prediction task.

The emerging field of Tabular Foundation Models (TFMs) has begun addressing these shortcomings, introducing powerful cross-dataset learning strategies [84, 47, 38]. However, the flagship model TabPFN-v2 [38] still handles text inputs no more flexibly than conventional GBDTs. This design choice is not incidental; historically, tabular benchmarks have prioritized numerical datasets without free-text features, largely for ease of modeling and evaluation. A recent study [48] of mainstream tabular datasets benchmarks [25, 23, 87, 55] found that half of these datasets are more than 20 years old, being a poor representation of modern real-world data.

Real-world tabular datasets often include high-cardinality 2 and free-text features [14], illustrated by a toy example in Table 1. In such datasets, free-text features (e.g., Report ) carry rich semantic information critical for tasks like predicting whether a patient will be discharged from the hospital or require continued care. Yet, most models encode them in a target-agnostic manner, delegating to a generic embedding that fails to capture task-specific nuances for predicting Decision . Crucially, that same embedding would have been used for a different target variable (e.g., Treatment Cost ). Similarly, categorical features with dozens of unique values (e.g., Department ) are difficult to encode efficiently without external knowledge, making naive approaches brittle and limiting generalization. Importantly, the column names, which could guide the model toward more effective representations, are typically ignored. Addressing these limitations is crucial for developing tabular models that leverage semantic information, transfer knowledge from many datasets, and generalize across domains.

In this paper, we introduce TabSTAR : a novel Tab ular Foundation Model with S emantically T argetA ware R epresentations, designed explicitly for end-to-end handling of purely textual features. By integrating an unfrozen text encoder at its core, TabSTAR can optimize free-text feature representations, demonstrating their clear superiority over alternative frozen embedding approaches. Additionally, it introduces a novel approach of target-aware tokens , which inject semantic information about the target variable as part of the input, allowing for efficient parameter sharing and resulting in an architecture with no dataset-specific parameters (see Figure 1). TabSTAR's training is highly efficient 3 and its performance steadily improves with more pretraining data. Empirically, TabSTAR achieves state-of-the-art (SOTA) performance on classification datasets containing textual features, surpassing leading TFMs as well as GBDTs tuned for 4 hours.

2 Related Work

This section reviews prior work in supervised tabular learning. We begin with deep learning methods tailored for tabular data, which were applied to a single dataset. We then discuss cross-dataset transfer learning techniques that improve generalization by leveraging related datasets. Next, we cover the field of TFMs, which aim to generalize across diverse tasks and datasets through large-scale pretraining. Finally, we review recent work on applying large language models (LLMs) to tabular data and elaborate on existing AutoML [34] multimodal solutions. As we focus on supervised learning, we do not cover self-supervised methods [82, 6] and their downstream applications.

Deep Learning on a Single Tabular Dataset Several architectures have been proposed to enhance deep learning for tabular data [70, 45, 81, 85]. TabNet [3] and TabTransformer [43] introduced attention mechanisms into tabular deep learning, while FT-Transformer [26] and its improvement [27] jointly integrated numerical and categorical features into a transformer [79]. Other novel approaches leveraged inter-example information at inference time, with SAINT [69] proposing row-

level attention between examples, Non-Parametric Transformers [49] processing the entire dataset, including labels, in a single forward pass, and TabR [28] combining a k-nearest-neighbor mechanism with a traditional Multi-Layer Perceptron (MLP) architecture. Recent works such as TabM [29] and RealMLP [39] focused on refining MLPs without an attention component. Despite these innovations, single-dataset deep learning models have not yet convincingly outperformed GBDTs [65, 31, 64]. Furthermore, none of them addressed the challenge of modeling tabular data with rich textual features.

Cross-Dataset Transfer Learning Deep learning has been proven to shine when performing transfer learning in many machine learning domains [91]. Motivated by this success, [52, 89] proved that cross-dataset learning can boost single-dataset performance, but were limited to strict requirements such as partial overlap of feature names. To address this limitation, TransTab [83] integrated semantic understanding into feature tokenization, and XTab [90] pretrained a transformer backbone with dataset-specific parameters, proving that pretraining contributes to a stronger initialization for a downstream task. Despite their small scale, these studies demonstrated cross-dataset transfer learning's potential, laying essential groundwork for the rise of TFMs.

Tabular Foundation Models TFMs represent an emerging paradigm in tabular learning. While the definition is still evolving, we adopt the framing proposed by [77], which identifies key desired characteristics of TFMs: large-scale pretraining with adaptability to downstream tasks, mixed-type column support, cross-domain generalization, use of textual metadata, 4 and column-order invariance.

TabPFN [36] is recognized as the first TFM, and its successor TabPFN-v2 [38] currently sets the SOTA in tabular learning, becoming a popular approach for TFMs [61, 21, 54]. TabPFN-v2 was the first model to consistently outperform GBDTs on medium-sized datasets, by pretraining Bayesian Prior-Data Fitted Networks (PFNs) [58] on 130 million synthetic datasets. Using ICL at inference time, it accepts up to 10,000 examples as input and predicts without updating its weights. TabPFN-v2 inspired TabICL [61] which improved scalability, and TabDPT [54], trained on real data. However, they all rely on off-the-shelf embeddings for text features like GBDTs, limiting their performance.

CM2 [86], CARTE [47], and TP-BERTa [84] represent a shift toward semantic tabular modeling, leveraging textual signals and external knowledge at a greater scale. Unlike prior methods, these models transfer knowledge via language representations. CM2 was pretrained on over 2,000 datasets, but did not focus on free-text features and used static word embeddings without further finetuning them. CARTE encodes tables as star-shaped graphs, jointly representing features by their names and values, and applies attention over the graph to capture contextual relations. While effective for high-cardinality features, it lacks exposure to longer free-text fields during pretraining and was proven useful mainly for small datasets. TP-BERTa adapts RoBERTa [53] with intra-feature attention and a tokenization scheme that maps numerical values into discrete relative-magnitude bins, to address the weakness of language models when tokenizing numbers [75]. Although it performs well, its use of dataset-specific output layers limits scalability and complicates multi-task learning. Consequently, they trained two separate models, 5 wasting potential for better cross-dataset learning. Notably, none of these approaches finetune semantic representations during downstream task training. In our work, we demonstrate that this is critical to align textual and tabular features.

Large Language Models for Tabular Data The remarkable success of LLMs is unprecedented [13, 59]. During the past years, several research attempts have tried to combine LLMs and tabular data. One line of work focuses on using LLMs directly for tabular prediction by converting tabular data into serialized text. TabLLM [35] assessed LLMs under few-shot scenarios, while Tabula-8b [24] finetuned the Llama 3-8B model extensively on tabular data. Although useful for few-shot learning, these models are computationally expensive, 6 suboptimal for numerical features [75, 77], and potentially compromised on widely-used benchmarks due to prior exposure during training [9]. While current generations of LLMs weren't adopted for tabular learning, their emergent knowledge from their pretraining could be crucial when textual features are present [77, 20]. Additionally, LLMs can be used in multiple aspects of tabular learning, as they seem to be promising synthetic data generators [10, 68], useful data cleaners [7], and clever feature engineers [37].

Age: 40-50

Decision: Released

Report: [Free Text]

Numerical

Encoder

Interaction

Encoder

Prediction

Head

s

Stacking

Hospitalized

Decision: Hospitalized

Textual

Encoder

Numerical

Fusion

Verbalization

Encoding

Fusion

Interaction

Prediction

Multimodal AutoML Historically, textual tabular datasets have been largely overlooked in classical tabular benchmarks. However, the AutoML [34] community has made significant progress in developing multimodal solutions. In particular, AutoGluon [19] introduced the AutoML Multimodal Benchmark [63], initially focusing on text features and later evolving into AutoGluon-Multimodal [73, 72], which incorporates images as well. This powerful AutoML framework can fuse text and image foundation models with tabular models and ensemble multiple models through a meta-learning approach [22], making it one of the few systems able to refine static textual representations via joint learning. Nevertheless, this line of work should not be seen as a single model but rather as a highly optimized, production-ready system. According to the authors, it is "a collection of tricks that significantly enhance performance" [73], establishing itself as a robust approach for multimodal, multi-model tabular learning. However, this line of work remains somewhat orthogonal to the development of novel TFMs.

3 TabSTAR

In this section, we introduce TabSTAR: a Tabular Foundation Model with Semantically Target-Aware Representations. Our training framework consists of two stages: (1) Pretraining , where the model is pretrained over a corpus of tabular datasets 7 in a multi-task regime, mixing classification with regression tasks, then (2) Finetuning , where the pretrained model is further trained with LoRA [42] on a single downstream task. TabSTAR is designed to enable effective cross-dataset learning by applying supervised learning on the target variable in both stages. At its core, it uses an unfrozen encoder-only language model, which can potentially invoke world knowledge acquired during the language model pretraining. 8 The encoder is combined with a tabular-specific architecture tailored to structured data, mitigating the known limitations of language models in tabular settings [75, 77].

TabSTAR's architecture comprises five core modules: (1) Verbalization , mapping every feature into a textual representation composed of both the column name and value, with a special treatment to numerical features for full numerical precision; (2) Encoding , transforming semantic and numerical inputs into meaningful embeddings of the same dimension; (3) Fusion , integrating textual and numerical representations on each feature independently; (4) Interaction , modeling dependencies and relationships between different features through cross-feature self-attention; and (5) Prediction , where outputs are projected into a real value for regression or a probability distribution for classification. Figure 1 illustrates the architecture, Appendix A elaborates it, and Appendix B discusses the training.

A key innovation of TabSTAR is the introduction of target-aware tokens , a novel approach that integrates the target variable's identity as an input to the model. Unlike existing TFMs [38, 47, 84, 86, 90, 83], which treat the target value as a mere label, TabSTAR fuses target-awareness from the very beginning. For classification tasks, each target value is verbalized and encoded like any other feature. Then, features and target tokens interact with each other, building representations that are then used for prediction. Crucially, this target-awareness allows parameter sharing between all target tokens, which can later use a shared prediction head that maps tokens to probabilities regardless of the number of classes and their identity. By doing so, TabSTAR eliminates the need for dataset-specific components commonly found in prior work [26, 90, 84]. TabSTAR's flexible architecture effortlessly scales 9 to any dataset size, and handles any number of classes in multiclass classification tasks.

Verbalization All the features and each of the target values are processed into a sequence of elements . Numerical features are processed into two inputs: a numerical one and a semantic one. The numerical input is standardized using z-scores, with outlier clipping at ±3 standard deviations. In addition, they are verbalized using quantile discretization into 10 bins, a novel approach to mitigate the precision loss inherent in language models [75]. Appendix A.1 shows a precise example and §6 discusses different verbalization strategies. In contrast, semantic features are directly verbalized by concatenating the feature name and textual value, without any numerical representation. The target variable is also included as part of the input: In classification tasks, each of the C possible values is represented by an element, constant for every example, while the true value remains hidden. For regression tasks, a single element is verbalized, carrying only the target name. Table 1 shows a toy dataset of patient records and outcomes and Table 2 shows the verbalization for the first patient.

Encoding We employ a pretrained e5-small-v2 [80] embedding model for semantic encoding, chosen for its strong performance on the MTEB benchmark [57] with a relatively modest parameter count. By unfreezing the upper half of its layers, the representations are optimized for predicting the target variable, which leads to a significant impact on TabSTAR performance (see §6). Each verbalization element is encoded independently into a semantic representation, with attention applied between tokens within each sequence element. In parallel, we encode standardized numerical values by projecting them into the same dimension using a small MLP. For the patient in Table 2, this results in a numerical embedding for Age alongside semantic representations for each of the 5 verbalizations.

Fusion To obtain a unified representation for each sequence element, we apply a fusion block consisting of a single encoder-only transformer layer. Crucially, each numerical feature is fused independently, as the block attends only to its numerical and semantic embeddings. In our running example, the representation of Age now jointly captures both its semantic context (the fact that the value represents age) as well as its numerical value (the patient's age, 45, or 0.27 after standardization).

Interaction The fused, semantically-rich and numerically-grounded representations of all elements interact via a 6-layer Transformer encoder [79]. Each input element is now a token, with feature tokens and target tokens all attending to each other. Unlike standard language models, which integrate positional encoding, the Interaction module's inputs are order-invariant, a desideratum for TFMs, as defined by [77]. The encoder produces contextualized representations for each target value. In our example, this yields dedicated embeddings for the Release and Hospitalization target values. The role of these representations is to carry information about how likely each value is to be the true value.

Prediction TabSTAR is designed for cross-dataset learning, with shared regression and classification heads used during both pretraining and finetuning. For classification, each of the C target tokens is processed independently through the same classification head, which projects them to scores. We then apply a softmax over all the possible values to yield a probability distribution. Crucially, the fact that target tokens for every class in every dataset share the same classification head allows efficient parameter sharing, flexibly supports any number of output classes, and removes any need for dataset-specific parameters. This is not only efficient during pretraining, but also provides a better initialization for finetuning. In our example, both the Released and Hospitalized tokens go through the same classification head, which maps them from representations to logits. Applying softmax yields predicted probabilities. For regression tasks, a single target token is projected into a real value.

4 Experiments

The TabSTAR Pretraining Corpus While TabSTAR could be pretrained on a massive scale, for this work we limit ourselves to a modest pretraining corpus focusing on classification, as we believe that TabSTAR's inductive biases are best suited to shine in this task. We manually curate a pretraining corpus of 350 high-quality tabular datasets (253 classification, 97 regression), in a tedious process in which we uncover numerous duplications in the most popular tabular repositories, OpenML [78] and Kaggle, 10 as elaborated by [76]. We begin by sourcing datasets from popular benchmarks [25, 47, 23, 22, 31, 63, 55, 32], but observe that the presence of textual tabular datasets in them is very rare. Thus, we furthermore expand our corpus, focusing on classification datasets with rich semantic content. See Appendix C for more details.

Benchmark Tabular datasets with free-text have seen little prior research, and accordingly, benchmarks are rare. To address this, we compile all available datasets from three sources: (1) the AutoML Multimodal Benchmark [63], (2) the CARTE paper [47], and (3) the analysis of free-text and highcardinality features by [32]. After deduplication, our final benchmark includes 50 datasets. Despite its breadth, this collection has two key limitations: First, the benchmark is heavily skewed towards regression tasks, with 36 datasets. Secondly, 29 out of these 36 datasets were solely contributed by the CARTE benchmark, which focuses more heavily on high-cardinality features, rather than longer texts, as it was pretrained over knowledge graphs. While our main motivation is classification tasks with textual features, we decide nevertheless to evaluate on the full set of 50 datasets, although it is biased toward regression problems and high-cardinality features (see Appendix D).

Baselines We compare TabSTAR against GBDTs : Random Forest [12], LightGBM [46], XGBoost [15], and CatBoost [60]; MLPs : RealMLP [39] and TabM ; and TFMs : [29], CARTE [47], TabDPT [54], TabPFN-v2 [38], and TabICL [61] which only supports classification tasks. For several models 11 we consider both default variants as well as tuned ones, where hyperparameters are optimized separately for each task using random search with 5-fold cross-validation under a 4-hour budget on 8 CPU cores. Since the public TabPFN-v2 model does not support text, we use their closed-sourced API client. 12 For models lacking native support for textual features, we embed text using e5-small-v2 [80], allowing a fair comparison. For more details about the hyperparameters for each baseline as well as exclusion of baselines due to potential leakage concerns, see Appendix E.

Experimental Setup Each of the 50 datasets in the benchmark is evaluated with 10 random traintest splits (90% training, 10% testing), resulting in 500 runs per model. While 30 of the datasets have more than 10,000 examples, most TFMs can't effectively scale beyond it: TabPFN-v2, for example, employs ICL and thus receives as input at most 10,000 examples. While CARTE imposes no strict size cap, its tuning is slow, inefficient, and impractical for larger datasets. 13 Because of these important limitations, we consider two experiment conditions: (1) 10K : Each model is trained 14 over at most 10,000 training examples, and (2) Unlimited : We evaluate TabSTAR and the most competitive, scalable baselines on the full version of the 30 datasets. 15

The TabSTAR Training To maximize the value of cross-dataset learning, instead of pretraining TabSTAR once, we create five dataset splits. Each variant is pretrained on the 350 pretraining datasets and 40 of the benchmark datasets, while the other 10 serve exclusively as its test set. Crucially, the whole collection was carefully curated to prevent any data leakage from duplicate or overly similar datasets. As a result, each dataset is evaluated by finetuning the single pretrained variant, which excludes it from its pretraining. For finetuning, while dataset-specific hyperparameter tuning can boost performance, we believe that robust out-of-the-box defaults are essential for TFMs and their evaluation. Therefore, we use a default hyperparameter configuration that was found robust over a disjoint set of tabular datasets, as detailed in Appendix B.2.

5 Results

Weevaluate each model using AUROC (classification) and R 2 (regression) as metrics. Following [38], we normalize scores per dataset split to the [0 , 1] range, using the best and worst model performance as anchors. 16 The normalized scores are averaged across all runs, with 95% CIs. Performance for all models on both conditions is shown in Figure 2 (classification) and Figure 3 (regression). For certain datasets, we are unable to execute TabPFN-v2, CARTE, and TabDPT due to model-specific implementation issues or scalability constraints. Reported averages for these models are computed only over the datasets where evaluation is feasible. Appendix F.1 elaborates on technical limitations, Appendix F.2 on dataset-level performance, and Appendix F.3 on head-to-head comparisons.

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Normalized Score

TabSTAR

TabM

CatBoost

XGBoost

TabICL

RealMLP

LightGBM

TabDPT

CARTE

RandomForest

Default

Default

Tuned

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Normalized Score

TabSTAR

TabM

CatBoost

XGBoost

RealMLP

TabICL

LightGBM

TabDPT

CARTE

Classification - Above 10K examples (10 datasets)

10K

Unlimited

10K

Unlimited

In classification problems, TabSTAR consistently achieves SOTA performance. This is evident both when restricting the dataset size to 10,000 examples and when using larger datasets in the unlimited condition. For the 10K condition, TabSTAR achieves a 0.83 score, followed by TabM-Tuned (0.73), CatBoost-Tuned (0.69), and TabPFN-v2 (0.67). When analyzing head-to-head comparisons (see Appendix F.3), TabSTAR outperforms TabPFN-v2 (8/11 datasets), TabM-Tuned (10/14), and CatBoost-Tuned (12/14). For the unlimited condition, TabSTAR-Unlimited achieves a 0.84 score, followed by TabM-Tuned-Unlimited (0.79), and much above the rest. Importantly, unlimited variants significantly surpass the 10K ones, emphasizing the importance of models with no scaling restrictions.

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Normalized Score

CatBoost

XGBoost

CARTE

TabM

LightGBM

RealMLP

TabSTAR

RandomForest

Default

Default

Tuned

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Normalized Score

CatBoost

TabM

XGBoost

RealMLP

TabSTAR

LightGBM

CARTE

Regression - Above 10K examples (20 datasets)

10K

Unlimited

10K

Unlimited

Although regression is not our main focus, TabSTAR achieves competitive results in the 10K condition, but clearly does not set the SOTA. Surprisingly, while TabPFN-v2 is superior, it significantly underperforms compared to GBDTs which dominate this category. This emphasizes the need for better modeling of textual tabular learning, especially since TabPFN-v2 has shown remarkable performance in non-textual tabular datasets, and CARTE set the SOTA for small datasets. In the unlimited setting, TabSTAR scales well and surpasses other TFMs which cannot scale, but the gap from GBDTs remains significant. §7 discusses this limitation and suggests promising directions for future generations of TabSTAR to achieve SOTA in regression as well.

Cost Analysis Table 3 reports the computational requirements of TabSTAR and competing baselines, measured by runtime and peak memory usage, for both training and inference. To highlight the importance of GPU acceleration when text embeddings are part of the preprocessing, we include a variant of CatBoost restricted to CPU only. We exclude TabPFN-v2, whose performance is probably comparable to TabICL, since it is only accessible via an API, and CARTE, whose extensive tuning requirements and prohibitive training costs forced evaluation on less capable hardware.

We find that TabSTAR incurs modest inference costs, comparable to GPU-accelerated GBDTs, since GPU processing of text embeddings dominates runtime. This is evident from the CatBoostCPU variant, whose inference is nearly ten times slower than TabSTAR. Moreover, TabICL and TabDPT are orders of magnitude slower and consume far more memory, emphasizing the scalability limitations for ICL-based models. During training, TabSTAR is considerably slower than the baselines. However, while fitting a single GBDT is much faster, this advantage quickly diminishes in practice when hyperparameter tuning is needed or GPU acceleration for text embeddings is not available. Appendix F.4 provides dataset-level runtimes and F.5 analyzes costs as a function of the feature count.

6 Analysis

We analyze the factors contributing to TabSTAR's strong performance by addressing three key research questions: Q1: How important is the encoder language model unfreezing? Q2: Does the number of datasets during pretraining contribute to the downstream task performance? and Q3: How do different verbalization methods of numerical features impact performance?

To answer these questions, we pretrain several variants of TabSTAR for each analysis, limiting ourselves to a subset of the tabular datasets used for the main experiment (see §4). Specifically, each variant is pretrained over 256 datasets 17 including 30 datasets from our benchmark, and evaluated over the remaining 20 datasets (12 regression, 8 classification). This reduced setup allows leveraging transfer learning and exploiting our corpus, without the burden of training multiple folds per variant. Appendix G.1 lists the 20 datasets used for evaluation along with per-dataset results.

Q1: The Role of the Encoder Unfreezing We examine the effect of unfreezing different numbers of textual encoder layers during pretraining. In finetuning, we keep all base layers frozen and instead apply LoRA adapters to the same layers that were unfrozen in pretraining. 18 Figure 4 shows the validation loss during TabSTAR pretraining (left) and the normalized score on the downstream tasks (right) as a function of the number of unfrozen encoder layers. Notably, unfreezing even a single encoder layer significantly outperforms using static embeddings. Further substantial improvements are observed as more layers are tuned, with the best results achieved when unfreezing 6 layers. While unfreezing 9 layers shows lower performance, it is plausible that adding more datasets to the pretraining phase will affect this finding. See Appendix G.2 for more details.

0

20

40

Epochs

0.6

0.8

1.0

Validation Loss

0

1

3

6

9

0.00

0.25

0.50

0.75

1.00

Average Score

0 Unfrozen

1 Unfrozen

3 Unfrozen

6 Unfrozen

9 Unfrozen

Q2: The Effect of Pretraining To evaluate the impact of pretraining on TabSTAR's downstream performance, we compare a pretrained version of TabSTAR with a version that was finetuned from scratch. 19 In line with previous work [90, 86], the pretrained model performs significantly better, highlighting the critical role of transfer learning for TabSTAR's success. To further investigate the effect of the number of pretraining datasets on downstream task performance, we train two additional versions: one pretrained on 16 datasets and another on 64 datasets.

0

16

64

256

# Pretraining datasets

0.80

0.82

0.84

0.86

0.88

0.90

AUROC

Classification

0

16

64

256

# Pretraining datasets

0.70

0.72

0.74

0.76

0.78

0.80

R²

Regression

As shown in Figure 5, increasing the number of pretraining datasets consistently improved performance in both classification and regression tasks. Notably, the substantial gain in regression tasks suggests that TabSTAR's downstream performance on §5 could improve with more pretraining data, potentially reaching SOTA performance with enough scale. See Appendix G.3 for more details.

Q3: Numerical Verbalization A key challenge in integrating language models with numerical data is determining how to best represent numerical values within a linguistic framework. While some semantic tabular methods omit numerical features from the verbalization [83, 86], TP-BERTa [84] introduced Relative Magnitude Tokenization [84], which encodes numerical information through non-semantic special bin tokens. In contrast, TabSTAR injects semantic numerical information into the verbalization of numerical features, as illustrated in Table 2. To quantify the effect of our novel verbalization, we explore two thinner variants: (1) Name + Bin , which excludes the quantile information, and (2) Name , which omits numeric information entirely and verbalizes the feature name only. Appendix G.4 shows an illustrative example for each variant and presents the full results. As demonstrated in Table 4, our findings reveal that incorporating numerical information significantly enhances performance, highlighting the importance of balancing numerical precision with a representation format that aligns with the language model's parametric knowledge.

7 Discussion and Conclusion

We introduce TabSTAR, a Tabular Foundation Model with Semantically Target-Aware Representations, which integrates textual features through an unfrozen pretrained encoder. In addition, its novel target-aware tokens enable efficient cross-dataset generalization without dataset-specific parameters. Despite limited pretraining data and a relatively small text encoder [80], TabSTAR sets the SOTA in tabular classification with textual features, significantly surpassing GBDTs and leading TFMs.

Since scaling laws in data and model size have proven themselves for LLMs [44] and TabSTAR improves with the number of pretraining datasets (see §6), future work should scale TabSTAR across both model and data dimensions. For model scaling, we envision a family of model sizes, common for LLMs [30, 74, 51], that will allow a trade-off between quality and costs. Data scaling might leverage self-supervised learning [62, 82] over large-scale table corpora [18], or realistic synthetic tabular data generators [10], which have proven successful [38, 2]. At scale, it could potentially unlock few-shot learning capabilities and develop automatic feature-engineering skills [37].

Beyond scaling, TabSTAR's semantic approach has tremendous potential to explicitly include world knowledge by leveraging LLMs, which to date have had a limited impact on tabular learning. As a few motivating examples, LLMs could improve TabSTAR's numerical verbalization binning approach by providing semantically informed thresholds, or by providing explicit world knowledge that could be injected as a strong prior in small data scenarios. While these directions seem like plausible research paths, they come with a risk of data leakage due to the memorization properties of LLMs [9]. Evaluating TFMs fairly while keeping benchmarks uncontaminated would be an important enabler for tabular research. As a step in this direction, we are releasing several TabSTAR variants, each with a different dataset withheld during pretraining, ensuring that for every dataset there is a TabSTAR model that has never seen it. We urge fellow researchers to adopt this approach in their own work.

While TabSTAR sets a new bar in classification, its regression results lag behind GBDTs, which outperform other TFMs as well. This gap could be narrowed through additional scaling, by exploring regression-via-classification techniques like [38, 2], and by enriching the numerical encoder with distribution-aware statistics per feature as done by [61]. In addition, similar to other TFMs, TabSTAR may encounter memory bottlenecks on datasets with hundreds of features, and its training speed lags behind untuned GBDTs and ICL-based TFMs. Yet TabSTAR achieves GBDT-level efficiency during inference, as opposed to TabPFN-v2 and TabICL. Furthermore, TabSTAR has not been extensively evaluated in few-shot scenarios and in purely numerical datasets.

Despite these limitations, TabSTAR offers a promising pathway toward improving performance on tabular datasets with textual fields, common in industries with high social impact (e.g., healthcare, education), or with significant economic value (e.g., banking, manufacturing). In addition, its architecture lends itself to multimodality and could be extended to tabular datasets that combine numerical, textual, and image features. We believe TabSTAR paves the way for a new generation of semantically enriched, Multimodal TFMs, and invite the research community to advance this vision.

Acknowledgments and Disclosure of Funding

Roi Reichart and Eilam Shapira have been partially supported by a VATAT grant on data science.

Wethank Omri Feldman for brainstorming since the very beginning; Elad Hoffer and Ofir Lindenbaum for consulting and feedback; David Holzmüller, Lennart Purucker, Myung Kim, and Gaël Varoquaux for assisting with evaluations and benchmarks; and Frank Hutter, Noah Hollmann, Léo Grinsztajn, and the rest of the Prior Labs team for providing extensive access to TabPFN-v2's API version.

References

Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, and et al. Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin. In Proceedings of The 33rd International Conference on Machine Learning , pages 173-182. PMLR, June 2016. URL https://proceedings.mlr.press/v48/amodei16.html . ISSN: 1938-7228.

Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and Bernie Wang. Chronos: Learning the Language of Time Series. Transactions on Machine Learning Research , May 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=gerNCVqqtR .

Sercan Ö Arik and Tomas Pfister. TabNet: Attentive Interpretable Tabular Learning. Proceedings of the AAAI Conference on Artificial Intelligence , 35(8):6679-6687, May 2021. ISSN 23743468. doi: 10.1609/aaai.v35i8.16826. URL https://ojs.aaai.org/index.php/AAAI/ article/view/16826 . Number: 8.

Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundation Models Defining a New Era in Vision: A Survey and Outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence , 47(4):2245-2264, April 2025. ISSN 1939-3539. doi: 10.1109/TPAMI.2024.3506283. URL https://ieeexplore.ieee.org/abstract/ document/10834497 .

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate, May 2016. URL http://arxiv.org/abs/1409.0473 . arXiv:1409.0473 [cs].

Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-Supervised Contrastive Learning using Random Feature Corruption. October 2021. URL https://openreview. net/forum?id=CuV_qYkmKb3 .

Tommaso Bendinelli, Artur Dox, and Christian Holz. Exploring LLM Agents for Cleaning Tabular Machine Learning Datasets. March 2025. URL https://openreview.net/forum? id=RXnQPYSoun .

Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, and et al. On the Opportunities and Risks of Foundation Models, August 2021. URL https://arxiv.org/abs/2108.07258v3 .

Sebastian Bordt, Harsha Nori, Vanessa Rodrigues, Besmira Nushi, and Rich Caruana. Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models. First Conference on Language Modeling , 2024.

Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language Models are Realistic Tabular Data Generators. September 2022. URL https: //openreview.net/forum?id=cEygmQNOeI .

Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep Neural Networks and Tabular Data: A Survey. IEEE Transactions on Neural Networks and Learning Systems , 35(6):7499-7519, June 2024. ISSN 2162-2388. doi: 10.1109/TNNLS.2022.3229161. URL https://ieeexplore.ieee.org/abstract/ document/9998482 .

Leo Breiman. Random Forests. Machine Learning , 45(1):5-32, October 2001. ISSN 1573-0565. doi: 10.1023/A:1010933404324. URL https://doi.org/10.1023/A:1010933404324 .

Tom Brown, Benjamin Mann, Nick Ryder, and et al. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html .

Patricio Cerda and Gaël Varoquaux. Encoding High-Cardinality String Categorical Variables. IEEE Transactions on Knowledge and Data Engineering , 34(3):1164-1176, March 2022. ISSN 1558-2191. doi: 10.1109/TKDE.2020.2992529. URL https://ieeexplore.ieee.org/ abstract/document/9086128 .

Tianqi Chen and Carlos Guestrin. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '16, pages 785-794, New York, NY, USA, August 2016. Association for Computing Machinery. ISBN 978-1-4503-4232-2. doi: 10.1145/2939672.2939785. URL https://dl. acm.org/doi/10.1145/2939672.2939785 .

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https:// aclanthology.org/N19-1423/ .

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021. URL http://arxiv.org/abs/2010.11929 . arXiv:2010.11929 [cs].

Gus Eggert, Kevin Huo, Mike Biven, and Justin Waugh. TabLib: A Dataset of 627M Tables with Context, October 2023. URL http://arxiv.org/abs/2310.07875 . arXiv:2310.07875 [cs].

Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data, March 2020. URL http://arxiv.org/abs/2003.06505 . arXiv:2003.06505 [stat].

Xi Fang, Weijie Xu, Fiona Anting Tan, Ziqing Hu, Jiani Zhang, Yanjun Qi, Srinivasan H. Sengamedu, and Christos Faloutsos. Large Language Models (LLMs) on Tabular Data: Prediction, Generation, and Understanding - A Survey. Transactions on Machine Learning Research , March 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=IZnrCGF9WI .

Benjamin Feuer, Robin T. Schirrmeister, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, and Colin White. TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks. Advances in Neural Information Processing Systems , 37:8343083464, December 2024. URL https://proceedings.neurips.cc/paper_files/paper/ 2024/hash/97dc07f1253ab33ee514f395a82fa7cc-Abstract-Conference.html .

Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning. Journal of Machine Learning Research , 23(261):1-61, 2022. ISSN 1533-7928. URL http://jmlr.org/papers/v23/ 21-0992.html .

Sebastian Felix Fischer, Matthias Feurer, and Bernd Bischl. OpenML-CTR23 - A curated tabular regression benchmarking suite. August 2023. URL https://openreview.net/ forum?id=HebAOoMm94 .

Josh Gardner, Juan C. Perdomo, and Ludwig Schmidt. Large Scale Transfer Learning for Tabular Data via Language Modeling. Advances in Neural Information Processing Systems , 37:4515545205, December 2024. URL https://proceedings.neurips.cc/paper_files/paper/ 2024/hash/4fd5cfd2e31bebbccfa5ffa354c04bdc-Abstract-Conference.html .

Pieter Gijsbers, Marcos L. P. Bueno, Stefan Coors, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. AMLB: an AutoML Benchmark. Journal of Machine Learning Research , 25(101):1-65, 2024. ISSN 1533-7928. URL http://jmlr.org/ papers/v25/22-0493.html .

Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In Proceedings of the 35th International Conference on Neural Information Processing Systems , NIPS '21, pages 18932-18943, Red Hook, NY, USA, December 2021. Curran Associates Inc. ISBN 978-1-7138-4539-3.

Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems , NIPS '22, pages 24991-25004, Red Hook, NY, USA, November 2022. Curran Associates Inc. ISBN 978-1-7138-7108-8.

Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. TabR: Tabular Deep Learning Meets Nearest Neighbors. The Twelfth International Conference on Learning Representations, October 2023. URL https://openreview. net/forum?id=rhgIgTSSxW .

Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling, February 2025. URL http://arxiv.org/ abs/2410.24210 . arXiv:2410.24210 [cs].

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and et al. The Llama 3 Herd of Models, November 2024. URL http://arxiv.org/abs/2407.21783 . arXiv:2407.21783 [cs].

Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems , 35:507-520, December 2022. URL https://proceedings.neurips.cc/paper_files/ paper/2022/hash/0378c7692da36807bdec87ab043cdadc-Abstract-Datasets_and_ Benchmarks.html .

Léo Grinsztajn, Edouard Oyallon, Myung Jun Kim, and Gaël Varoquaux. Vectorizing string entries for data processing on tables: when are larger language models better?, December 2023. URL http://arxiv.org/abs/2312.09634 . arXiv:2312.09634 [stat].

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015. URL http://arxiv.org/abs/1512.03385 . arXiv:1512.03385 [cs].

Xin He, Kaiyong Zhao, and Xiaowen Chu. AutoML: A survey of the state-of-the-art. Knowledge-Based Systems , 212:106622, January 2021. ISSN 0950-7051. doi: 10.1016/j. knosys.2020.106622. URL https://www.sciencedirect.com/science/article/pii/ S0950705120307516 .

Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. TabLLM: Few-shot Classification of Tabular Data with Large Language Models. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , pages 5549-5581. PMLR, April 2023. URL https://proceedings.mlr.press/v206/ hegselmann23a.html . ISSN: 2640-3498.

Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second. The Eleventh International Conference on Learning Representations, September 2022. URL https:// openreview.net/forum?id=cp5PvcI6w8_ .

Noah Hollmann, Samuel Müller, and Frank Hutter. Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering. Advances in Neural Information Processing Systems , 36:44753-44775, December 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 8c2df4c35cdbee764ebb9e9d0acd5197-Abstract-Conference.html .

Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature , 637(8045):319-326, January 2025. ISSN 14764687. doi: 10.1038/s41586-024-08328-6. URL https://www.nature.com/articles/ s41586-024-08328-6 . Publisher: Nature Publishing Group.

David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned MLPs and boosted trees on tabular data. Advances in Neural Information Processing Systems , 37: 26577-26658, December 2024. URL https://proceedings.neurips.cc/paper_files/ paper/2024/hash/2ee1c87245956e3eaa71aaba5f5753eb-Abstract-Conference. html .

Shi Bin Hoo, Samuel Müller, David Salinas, and Frank Hutter. The Tabular Foundation Model TabPFN Outperforms Specialized Time Series Forecasting Models Based on Simple Features. October 2024. URL https://openreview.net/forum?id=H02X7RO3OC#discussion .

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-Efficient Transfer Learning for NLP. In Proceedings of the 36th International Conference on Machine Learning , pages 2790-2799. PMLR, May 2019. URL https://proceedings.mlr.press/v97/ houlsby19a.html . ISSN: 2640-3498.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. October 2021. URL https://openreview.net/forum?id=nZeVKeeFYf9 .

Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular Data Modeling Using Contextual Embeddings, December 2020. URL http://arxiv.org/abs/ 2012.06678 . arXiv:2012.06678 [cs].

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models, January 2020. URL http://arxiv.org/abs/2001.08361 . arXiv:2001.08361 [cs].

Liran Katzir, Gal Elidan, and Ran El-Yaniv. Net-DNF: Effective Deep Modeling of Tabular Data. October 2020. URL https://openreview.net/forum?id=73WTGs96kho .

Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://papers.nips.cc/paper_files/paper/2017/hash/ 6449f44a102fde848669bdd9eb6b76fa-Abstract.html .

Myung Jun Kim, Léo Grinsztajn, and Gaël Varoquaux. CARTE: pretraining and transfer for tabular learning. In Proceedings of the 41st International Conference on Machine Learning , volume 235 of ICML'24 , pages 23843-23866, Vienna, Austria, July 2024. JMLR.org.

Ravin Kohli, Matthias Feurer, Katharina Eggensperger, Bernd Bischl, and Frank Hutter. Towards Quantifying the Effect of Datasets for Benchmarking: A Look at Tabular Machine Learning.

Jannik Kossen, Neil Band, Clare Lyle, Aidan N Gomez, Thomas Rainforth, and Yarin Gal. Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning. In Advances in Neural Information Processing Systems , volume 34, pages 2874228756. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/ 2021/hash/f1507aba9fc82ffa7cc7373c58f8a613-Abstract.html .

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc., 2012. URL https://papers.nips.cc/paper_files/ paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html .

Barak Lenz, Opher Lieber, Alan Arazi, and et al. Jamba: Hybrid Transformer-Mamba Language Models. October 2024. URL https://openreview.net/forum?id=JFPaD7lpBD .

Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C. Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, and Micah Goldblum. Transfer Learning with Deep Tabular Models. The Eleventh International Conference on Learning Representations, September 2022. URL https://openreview.net/forum?id=b0RuGUYo8pA .

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach, July 2019. URL http://arxiv.org/abs/1907.11692 . arXiv:1907.11692 [cs].

Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Anthony L. Caterini, and Maksims Volkovs. TabDPT: Scaling Tabular Foundation Models on Real Data, July 2025. URL http://arxiv.org/abs/2410.18164 . arXiv:2410.18164 [cs].

Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When Do Neural Nets Outperform Boosted Trees on Tabular Data? Advances in Neural Information Processing Systems , 36:76336-76369, December 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ f06d5ebd4ff40b40dd97e30cee632123-Abstract-Datasets_and_Benchmarks.html .

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space, September 2013. URL http://arxiv.org/abs/1301. 3781 . arXiv:1301.3781 [cs].

Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive Text Embedding Benchmark. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 2014-2037, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.eacl-main.148. URL https://aclanthology.org/ 2023.eacl-main.148/ .

Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers Can Do Bayesian Inference. October 2021. URL https://openreview.net/ forum?id=KSugKcbNf9 .

OpenAI. GPT-4 Technical Report, March 2024. URL http://arxiv.org/abs/2303.08774 . arXiv:2303.08774 [cs].

Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: unbiased boosting with categorical features. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/hash/ 14491b756b3a51daac41c24863285549-Abstract.html .

Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data, February 2025. URL http:// arxiv.org/abs/2502.05564 . arXiv:2502.05564 [cs].

Ivan Rubachev, Artem Alekberov, Yury Gorishniy, and Artem Babenko. Revisiting Pretraining Objectives for Tabular Deep Learning, July 2022. URL http://arxiv.org/abs/2207. 03208 . arXiv:2207.03208 [cs].

Xingjian Shi, Jonas Mueller, Nick Erickson, Mu Li, and Alex Smola. Benchmarking Multimodal AutoML for Tabular Data with Text Fields. August 2021. URL https://openreview.net/ forum?id=Q0zOIaec8HF .

Assaf Shmuel, Oren Glickman, and Teddy Lazebnik. A Comprehensive Benchmark of Machine and Deep Learning Across Diverse Tabular Datasets, August 2024. URL http://arxiv.org/ abs/2408.14817 . arXiv:2408.14817 [cs].

Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion , 81:84-90, May 2022. ISSN 1566-2535. doi: 10.1016/j.inffus.2021.11.011. URL https://www.sciencedirect.com/science/article/pii/S1566253521002360 .

Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition, April 2015. URL http://arxiv.org/abs/1409.1556 . arXiv:1409.1556 [cs].

Leslie N. Smith and Nicholay Topin. Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates, May 2018. URL http://arxiv.org/abs/1708. 07120 . arXiv:1708.07120 [cs].

Aivin V. Solatorio and Olivier Dupriez. REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers, February 2023. URL http://arxiv.org/abs/2302. 02041 . arXiv:2302.02041 [cs].

Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive PreTraining, June 2021. URL http://arxiv.org/abs/2106.01342 . arXiv:2106.01342 [cs].

Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management , CIKM '19, pages 1161-1170, New York, NY, USA, November 2019. Association for Computing Machinery. ISBN 978-1-4503-6976-3. doi: 10.1145/3357384.3357925. URL https://dl.acm.org/doi/10.1145/3357384.3357925 .

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2 , volume 2 of NIPS'14 , pages 3104-3112, Cambridge, MA, USA, December 2014. MIT Press.

Zhiqiang Tang, Haoyang Fang, Su Zhou, Taojiannan Yang, Zihan Zhong, Cuixiong Hu, Katrin Kirchhoff, and George Karypis. AutoGluon-Multimodal (AutoMM): Supercharging Multimodal AutoML with Foundation Models. AutoML Conference 2024 (ABCD Track), April 2024. URL https://openreview.net/forum?id=irStSm9waW .

Zhiqiang Tang, Zihan Zhong, Tong He, and Gerald Friedland. Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Data, December 2024. URL http://arxiv.org/ abs/2412.16243 . arXiv:2412.16243 [cs].

Gemini Team. Gemini: A Family of Highly Capable Multimodal Models, May 2025. URL http://arxiv.org/abs/2312.11805 . arXiv:2312.11805 [cs].

Avijit Thawani, Jay Pujara, Pedro A. Szekely, and Filip Ilievski. Representing Numbers in NLP: a Survey and a Vision, March 2021. URL http://arxiv.org/abs/2103.13136 . arXiv:2103.13136 [cs].

Andrej Tschalzev, Lennart Purucker, Stefan Lüdtke, Frank Hutter, Christian Bartelt, and Heiner Stuckenschmidt. Unreflected Use of Tabular Data Repositories Can Undermine Research Quality, March 2025. URL http://arxiv.org/abs/2503.09159 . arXiv:2503.09159 [cs].

Boris Van Breugel and Mihaela Van Der Schaar. Position: why tabular foundation models should be a research priority. In Proceedings of the 41st International Conference on Machine Learning , volume 235 of ICML'24 , pages 48976-48993, Vienna, Austria, July 2024. JMLR.org.

Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. OpenML: networked science in machine learning. SIGKDD Explor. Newsl. , 15(2):49-60, June 2014. ISSN 1931-0145. doi: 10.1145/2641190.2641198. URL https://doi.org/10.1145/2641190.2641198 .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text Embeddings by Weakly-Supervised Contrastive Pre-training, February 2024. URL http://arxiv.org/abs/2212.03533 . arXiv:2212.03533 [cs].

Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems. In Proceedings of the Web Conference 2021 , WWW '21, pages 1785-1797, New York, NY, USA, June 2021. Association for Computing Machinery. ISBN 978-1-4503-83127. doi: 10.1145/3442381.3450078. URL https://dl.acm.org/doi/10.1145/3442381. 3450078 .

Wei-Yao Wang, Wei-Wei Du, Derek Xu, Wei Wang, and Wen-Chih Peng. A Survey on Self-Supervised Learning for Non-Sequential Tabular Data. CoRR , January 2024. URL https://openreview.net/forum?id=idHPqbNraV .

Zifeng Wang and Jimeng Sun. TransTab: Learning Transferable Tabular Transformers Across Tables. October 2022. URL https://openreview.net/forum?id=A1yGs_SWiIi .

Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making Pre-trained Language Models Great on Tabular Prediction. The Twelfth International Conference on Learning Representations, October 2023. URL https: //openreview.net/forum?id=anzIzGZuLi .

Junchen Yang, Ofir Lindenbaum, and Yuval Kluger. Locally Sparse Neural Networks for Tabular Biomedical Data. In Proceedings of the 39th International Conference on Machine Learning , pages 25123-25153. PMLR, June 2022. URL https://proceedings.mlr.press/v162/ yang22i.html . ISSN: 2640-3498.

Chao Ye, Guoshan Lu, Haobo Wang, Liyao Li, Sai Wu, Gang Chen, and Junbo Zhao. Towards Cross-Table Masked Pretraining for Web Data Mining. May 2024. URL https: //openreview.net/forum?id=9jj7cMOXQo .

Han-Jia Ye, Si-Yang Liu, Hao-Run Cai, Qi-Le Zhou, and De-Chuan Zhan. A Closer Look at Deep Learning on Tabular Data. CoRR , January 2024. URL https://openreview.net/ forum?id=eu2cABIHge .

Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT, May 2023. URL http://arxiv.org/abs/2302.09419 . arXiv:2302.09419 [cs].

Qile Zhou, Han-Jia Ye, Leye Wang, and De-Chuan Zhan. Unlocking the Transferability of Tokens in Deep Models for Tabular Data. October 2023. URL https://openreview.net/ forum?id=u2OVQ2Xvq1 .

Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. XTab: Cross-table Pretraining for Tabular Transformers. In Proceedings of the 40th International Conference on Machine Learning , pages 43181-43204. PMLR, July 2023. URL https: //proceedings.mlr.press/v202/zhu23k.html . ISSN: 2640-3498.

Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A Comprehensive Survey on Transfer Learning. Proceedings of the IEEE , 109(1):43-76, January 2021. ISSN 1558-2256. doi: 10.1109/JPROC.2020.3004555. URL https://ieeexplore.ieee.org/abstract/document/9134370 .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

We claim "TabSTAR achieves state-of-the-art performance for both medium- and largesized datasets across known benchmarks" in the abstract and "TabSTAR achieves stateof-the-art (SOTA) performance on classification benchmarks containing substantial textual content" in the introduction - we describe these results in detail in §5, showing TabSTAR is the top performant comparing vs leading gradient boosting decision trees (GBDTs) and leading Tabular Foundation Models (TFMs)

We claim "...its pretraining phase exhibits scaling laws in the number of datasets" in the abstract and "its performance steadily improves with more pretraining data" in the intro, and show in §6 that TabSTAR's performance improve as we significantly increase the number of pre-training datasets.

We claim in the introduction "TabSTAR can optimize free-text feature representations, demonstrating their clear superiority over alternative frozen embedding approaches" - we describe these results in detail in §6, by showing that TabSTAR's performance increase greatly when finetuning the embeddings, making this the advantage over other algorithms compared in §5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, in §7 we discuss TabSTAR's limitations, such as its lower performance in regression and its lack of evaluation in other scenarios like datasets without textual content. Some more details about the regression performance are elaborated at §5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper doesn't present theoretical proofs, but focuses on algorithmic development and empirical validation.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes. Our model and method is highly elaborated through the paper (see §3) and the main experiments as well (§4). In addition, multiple supplemental materials cover these points in more details (e.g., Appendix A, Appendix B). Furthermore, we release our code, alongside documentation.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See reference from §1 to our anonymous GitHub repository, where we provide our code and running instructions.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: §4, Appendix B and Appendix E give the full experimental protocol.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The results are accompanied by 95% confidence intervals (see Appendix F). This is true also for the analysis section §6, which follows the same standards.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, see Appendix F.4 for compute information and running times, for both TabSTAR and other baselines.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The study relies exclusively on publicly available, de-identified datasets (OpenML, Kaggle and published benchmarks) and releases code and models under an opensource licence; no human subjects or sensitive personal data are involved, fully satisfying the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We use as a running example introduced in Table 1 an important real use case with high impact for the healthcare industry, motivated by the importance of improving decision making in a small-data regime. In addition, we briefly discuss other industries in §7.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: TabSTAR is not a generative model, and thus can't be used to create potentially high-risk text and images. In addition, we use publicly available datasets which don't impact individual privacy or security.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, we credited them in an appropriate way, adding URLs and package names in multiple occasions throughout the paper and its supplemental materials.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Upon acceptance, we will release several TabSTAR pretrained checkpoints, plus complete training and inference code. We believe that this research line is of huge potential and we encourage researchers to continue our line of work, also providing feasible evaluations setups for TFMs, such as default hyperparameters.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer:[NA]

Justification: The paper doesn't involve crowdsourcing ot human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: As the study didn't involve any study participants, this section isn't applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: While TabSTAR uses a language model at its core, this is an encoder-only model, and not a generative LLM. This widely used language model, e5-small-v2 [80], is used as part of the architecture in a standard usage. Thus, this section isn't applicable to our research.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Architecture

This appendix provides additional technical details for the architecture introduced in §3. First, we discuss the verbalization module; next, we formally describe the architecture step-by-step; and finally, we present selected experiments on the TabSTAR architecture.

A.1 The Verbalization Module

TabSTAR's verbalization module standardizes heterogeneous tabular inputs by converting each column, whether predictive feature or target variable, into templated text blocks. We first describe the detection of column types, and then detail the processing steps for each type.

Feature Detection We classify each column as either numerical, referring to quantitative values, or semantic, referring to textual values including categorical and boolean fields encoded as text. We rely on heuristics involving both the primitive data type (e.g., string, float) and human annotation (e.g., OpenML metadata). However, real-world datasets pose challenges, as numerical features can often be stored as strings (e.g., '35 years', 'unknown age') or may lack inherent order (e.g., country calling codes). Leveraging LLMs for contextualized data cleaning can be a promising direction [7].

A special case is the handling of timestamp and date columns. Similarly to [40], we rely on skrub's DatetimeEncoder 20 to detect datetime columns and decompose each one of them into a set of new features. Each extracted feature then undergoes its own processing: For example, the weekday is treated as semantic, while the total seconds since the Unix epoch is treated as numerical. Integrating date features more holistically remains an open research question.

Numerical Features Numerical features are represented by both a numerical and a semantic representation. For the numerical representation, given a value x , we compute the clipped z-score z ′ = clip ( ( x -µ ) /σ, -3 , 3 ) where µ, σ are the training set mean and the standard deviation, and missing values are set to 0. For the semantic representation, we build B = 10 quantile bins over the training distribution to map the value accordingly. Table 5 shows an illustrative example for the feature Age from our running example in Table 2.

Semantic Features Semantic features are sanitized (e.g., normalizing whitespaces) and verbalized using the template presented in Table 6. Missing values are mapped to 'Unknown Value', just like for numerical features. If a text exceeds the model's context window (512 tokens for e5-small-v2 ), it is naively truncated to fit it. This limitation is far more pronounced for methods that serialize the entire example into a single textual sequence [84], thereby dramatically reducing the effective context size.

Target Variables The verbalization templates for the target values are prepended to every example. For classification tasks, each possible label is verbalized, while for regression we verbalize a single element consisting solely of the feature name. Employing a binning strategy to treat regression as a classification task is a future work direction, as discussed in §7. For regression tasks, target values go through the same standardization with outlier clipping as numerical features, being used solely as the ground truth without going through the input.

A.2 The Annotated TabSTAR

Table 7 describes the number of parameters per component in the TabSTAR architecture, when using e5-small-v2 [80] as the text encoder. It has approximately 47.26M parameters, most of which come from the text encoder. When unfreezing 6 layers of the text encoder, about 24.70M parameters are tuned, with the remaining 11.92M embedding parameters and 10.65M layer ones being kept frozen.

To describe the architecture more precisely, we start by defining the dataset formally. Let D = { ( x i , y i ) } n i =1 denote a tabular dataset with n examples. Each example x i = [ x i 1 , . . . , x im ] has m features. The target variable y i is either continuous (regression) or discrete (classification) taking one of C classes. For simplicity, we describe the architecture at the example level, though all computations are actually carried out on mini-batches of size B . The batches are always drawn from a single dataset in both pretraining and finetuning, removing any need for padding.

Verbalization We denote by t the number of target entries where t = C for classification and t = 1 for regression. We then form a raw sequence of length e = t + m by listing the t target values, followed by the m feature entries. Each element j in this sequence is then verbalized into a semantic string s j and a numerical value n j , set to be the clipped z-score for numerical non-missing features, and zero otherwise. The example is thus represented by parallel sequences ( s , n ) of length e .

Encoding Each semantic string s j and numerical value n j are projected into a d -dimensional vector. Semantic strings are encoded with an encoder-only language model ( e5-small-v2 [80] ). Each string is tokenized, passed through the model, and pooled to produce its final embedding. This process is independent between elements, i.e. the attention is at the token level within a single element. In parallel, each numeric value is fed through a two-layer MLP that first projects from 1 to 2 d dimensions, applies a ReLU and dropout, and then projects back down to d . This produces matching d -dimensional embeddings for each of the e elements, ready to be fused.

Fusion To unify semantic and numerical embeddings into a single representation, we apply a single-layer Transformer Encoder 21 over each element's pair of vectors. Concretely, for each element

we stack its d -dimensional text and numeric embeddings and feed them through the encoder layer. For every element, the attention is applied between its two representations, and we average the two outputs to produce one fused d -dimensional embedding. This yields a final sequence of length e and dimension d , which will serve as tokens for the Interaction block.

Interaction The fused sequence of e tokens is processed by a standard Transformer Encoder with model dimension d = 384 , L = 6 layers, 6 attention heads per layer, feed-forward size 4 d , dropout 0 . 1 , ReLU activation and using a pre-norm configuration. Unlike in language modeling, feature ordering is irrelevant, so no positional encodings are used. The encoder produces contextualized embeddings for every position, and we retain the t target embeddings for the prediction.

Prediction We initialize two identical MLP heads, one for regression and one for classification. Each of them consists of a hidden layer of size 4 d (with ReLU activation) followed by a linear projection to a single output. For each dataset, we choose the relevant head and process the t target token embeddings. For classification, we independently feed each one of the t = C target tokens to the classification head to obtain a score (logit) per class. Notably, the same head is shared across classes and datasets. We apply softmax over these scores, yielding a probability distribution regardless of the number of classes. For regression, the single target token is projected into a real value. Note that the heads is shared between datasets, as regression outputs are always clipped z-scores.

A.3 Architecture Experiments

In this section we explore the effect of different design choices for TabSTAR's architecture. For each experiment, we only vary the parameter of interest, keeping everything else fixed. We follow the same pretraining regime as in Appendix B.1, except that for computational efficiency we train only 25 epochs (instead of 50) with 128 pretraining datasets (instead of 390). We evaluate each variant relying solely on pretraining performance, as an approximation for downstream task performance. We acknowledge that our conclusions might depend on this limited scale, hence we discuss a subset of the experiments briefly to reflect the depth of our work and inspire future research.

The Fusion Block's Mechanism For the fusion block, we consider two simpler alternatives to the attention mechanism, both of them underperforming: (1) Concatenation, by concatenating the semantic and numerical d -dimensional vectors into a 2 d -dimensional vector, and projecting them back via an MLP, and (2) Multiplication, by multiplying the semantic representation directly with the numerical value 22 in a straightforward, parameter-free manner as in [83, 86].

The Number of Interaction Layers Weexperiment with the number of encoder layers, and observe that 3 yields anecdotally worse performance than 6, with lower parameter count. Nevertheless, we prioritize a deeper network as for datasets with very complex relationships we believe that this might be beneficial. Additionally, we try a 9-layer variant which performs significantly worse, while also increasing the parameter count.

Row-Level Attention We experiment with adopting the architecture proposed by SAINT [69], which adds row-level attention to each encoder layer. Similar concepts are also employed by models that get the input the whole dataset, labels included [38, 49]. We run experiments with 2, 4 and 6 layers as they are roughly equivalent in parameter count to 3, 6, and 9 layers without row-attention. We observe no substantial gain, and thus we prioritize the simpler solution, as row-level attention is sensitive to the batch size and adds complexity to inference time.

Verbalization We experiment with various verbalization strategies. For features, including the column name alongside the value consistently outperforms using the value alone. For target-aware tokens, we find that explicitly verbalizing all possible target values yields better performance than omitting them. In contrast, adding a dataset-level description prefix to improve contextualization offers no measurable advantage. Interestingly, TabSTAR maintains high performance even when no semantic metadata is present. We see this robustness as a strength.

B Training

In this section we elaborate on the two stages of TabSTAR's training: pretraining and finetuning, presented in §3. As in Appendix A.3, we summarize key pretraining experiments.

B.1 Pretraining

TabSTAR is pretrained employing supervised learning in a multi-task regime, jointly learning regression, binary and multiclass classification tasks. The parameters of the architecture are fullyshared, without any need for dataset-specific parameters. Every example during the pretraining updates all the model's weights, with the sole exception of the prediction head, for which every example uses its respective head depending on the nature of the task (classification or regression).

Sampling For computational efficiency, each dataset is subsampled once before the pretraining. At the example level, we sample up to 300,000 examples from each dataset, stratified by the target variable for classification tasks. Since we only use a fraction of each dataset for each pretraining epoch, this decision has negligible influence. In addition, we randomly sample up to 200 features per dataset. While straightforward, this decision is suboptimal as feature importance isn't taken into consideration. As this work does not focus on wide-feature datasets, we consider this trade-off acceptable. Importantly, this setup is enabled during finetuning as the TabSTAR architecture is agnostic to the number of features. We split each dataset into train-validation splits (95%-5%), 23 without any need for test splits, and cap the validation set at a maximum of 1,000 examples used for evaluating pretraining performance.

Batching Every epoch, we randomly sample up to 2,048 examples from each dataset in minibatches of 32, and shuffle all the batches. We conduct gradient accumulation and update the model every 4 steps to reduce the chances of a single update being dominated by a single dataset, so the global batch size is effectively 128. Appendix B.3.1 elaborates on the effect of batch size.

Metrics Our loss function is cross-entropy for classification, and MSE for regression. With standardized targets, R 2 ≈ 1 -MSE, although this equivalence is degraded by clipping targets in preprocessing. We train with mixed-precision and apply gradient clipping to stabilize training without task-specific weights, with Appendix B.3.2 discussing the limitations of this approach. We use as metrics AUROC for classification and R 2 , so for each task the optimal metric value is 1. We average performance across all datasets into a single metric that reflects the pretraining performance.

Training We pretrain for 50 epochs with the OneCycleLR [67] optimizer, with warmup during the first 5 epochs (10%) and cosine annealing. Early stopping is conducted after 3 epochs without improvement on the pretraining metric. The weight decay is set to 0.001, and a max learning rate of lr = 5 × 10 -5 is applied uniformly across all layers. Appendix B.3.3 discusses experiments with differential learning rate. Pretraining running time varies depending on the number of epochs and the included datasets. The full models (390 datasets) reported in §5 train for less than 48 hours on a single NVIDIA A40 GPU (48GB memory), and we believe that this could be optimized much further.

B.2 Finetuning

We finetune downstream tasks using LoRA's implementation of the peft package 24 . We use a rank of r = 32 , set α = 2 r = 64 and dropout = 0 . 1 . We employ the same scheduler as in the pretraining phase, with the only difference being that we set lr = 0 . 001 , and increase the patience parameter for early stopping to 5. We apply a train-test split of 90%-10% and sample a validation set of 10%. As opposed to the pretraining, all batches are drawn from the same dataset. Therefore, we observe no effect from changing the mini-batch size when keeping the global batch size fixed to 128. We tune 1,597,440 out of TabSTAR's 47,263,874 parameters (3.4%), spanning all blocks of the architecture. The frozen layers of the text encoder do not receive LoRA adapters.

Finetuning hyperparameters are selected by pretraining TabSTAR over 256 datasets and performing grid-search over a held-out set of 25 downstream tasks disjoint from the 50 datasets in the benchmark

evaluated in §4. The search space is presented in Table 8, and we observe that average performance is relatively robust across this space. An interesting observation is that decreasing the number of parameters by setting r = 16 mildly hurts performance, but it has upsides on memory and latency aspects, allowing a future trade-off exploration. As a final note, we argue that providing a strong default configuration for TFMs is crucial for evaluating them, but for real-world applications, it is still recommendable to find the best hyperparameters tailored to the downstream task.

The only experiment in this paper where we employ full finetuning instead of LoRA is for the non-pretrained variant discussed in the analysis in §6. For this variant we fully finetune the pretrained model on each downstream task. Compared to the pretraining, we use lr = 2 . 5 × 10 -5 and increase the patience to 5. These hyperparameters are lightly tuned using the same procedure as for LoRA, and we observe that fully finetuning the model achieves comparable performance, except for small datasets, where training is more prone to overfitting.

B.3 Pretraining Experiments

In this section, we briefly elaborate on some experiments performed over TabSTAR's pretraining protocol. As in Appendix A.3, we highlight only a subset of them in a high-level manner.

B.3.1 Batch Size

During pretraining, we use a mini-batch size of 32, each of them drawn from a single dataset. Since we train with gradient accumulation and a global batch size of 128, varying the batch size affects the diversity of a single model update: lower batch sizes are likely to be exposed to more datasets. We decrease the batch size to 16 and 8 and observe an improvement at the cost of slower training. An interesting direction for future work is moving to mixed-datasets batches, which require more complex implementation but might benefit from more regularized learning. Such approach, however, goes against row-level attention methods and ICL, as discussed in Appendix A.3.

B.3.2 Loss Weights

Pretraining the model over hundreds of datasets in a multi-task regime presents a key challenge: the loss scale of each dataset can vary substantially, depending on task difficulty or task type. For example, a multiclass classification task with dozens of classes will naturally yield a higher average loss than a binary task. These dynamics can also shift during training. Our default approach naively averages the loss across all datasets, which risks over-weighting tasks for potentially arbitrary reasons.

To address this, we explore two alternative weighting strategies: (1) Assigning a constant weight per task type, accounting for the number of classes in classification tasks, and (2) Normalizing each dataset's contribution by the best loss achieved by CatBoost [60] when being fitted to that dataset. While these strategies better reflect task-specific characteristics, they hardly impact performance and introduce additional complexity. Notably, adjusting loss weights across tasks impacts metric interpretability, as each weighting scheme implicitly optimizes a different objective.

We do not explore more sophisticated methods such as learning per-dataset weights, as these often require mixed-dataset batches and introduce additional learnable parameters. We believe, however, that multi-task pretraining over tabular datasets remains an open and important research question.

B.3.3 Differential Learning Rate

TabSTAR's weights initialization is not balanced: the textual encoder is a pretrained embedding model while the rest of the architecture parameters are randomly initialized. To counteract this imbalance, we experiment with using differential learning rates for the textual encoder layers, and experiment

with scaling it by a factor of 0.5 and of 0.75. To our surprise, this decision hurts performance, so we stick to a uniform learning rate across all layers.

C Training Datasets

In this appendix we briefly expand on the pretraining corpus elaborated in §4. It is composed of 350 datasets, spanning all the datasets appearing in AMLB [25], OpenML-CTR23 [23], TabZilla [55] and the ones presented by Grinsztajn [31]. After deduplication, 25 this results in 152 datasets (94 classification, 58 regression). Interestingly, only 6 of these 152 datasets have free-text or highcardinality features. We manually add datasets from OpenML [78] and Kaggle, as well as from the AutoML-Benchmark-Train [22] corpus, and achieve a total of 350 datasets, with 49 textual datasets.

Table 9 details the 253 classification datasets and Table 10 the 97 regression ones. We elaborate the Dataset name, the number of examples n , the number of features m , and the number of classes C for classification. In addition, we mark datasets that belong to one of the benchmarks, and the ones that have text features. Importantly, the textual flag is quite permissive, as it includes features with relatively short texts or potentially low predictive power (e.g., people names or addresses).

Continued on next page

Continued on next page

Continued on next page

Continued on next page

D Benchmark Datasets

This appendix elaborates on the benchmark presented in §4. We consider all datasets proposed by AutoML Multimodal Benchmark (SHI) [63], Vectorizing (VEC) [32], and CARTE-Benchmark (CRT) [47], resulting in a final set of 50 datasets. We deduplicate datasets that appear as-is in more than one benchmark. In addition, since CARTE explores the concept of multi-table learning, they introduce highly-overlapping datasets for which we remove one variant (see 4.3 and B.2 in their paper).

Table 11 presents the classification datasets and Table 12 the regression ones. Each table includes an internal ID used for reference, the Dataset name, the number of examples n and of features m , and the number of classes C for classification. 26 Finally, we also indicate the benchmark sources where each dataset appears. In addition, Table 13 presents the full benchmark with a short description per dataset, and Table 14 details the datasets removed during the deduplication process. Most of the excluded datasets are regression datasets from the CARTE-Benchmark , because of its high-overlapping nature.

E Baselines

In this appendix we first discuss models excluded from the evaluation due to data leakage concerns, and then cover implementation details for the baselines used in our main experiments §4.

E.1 Excluded Baselines

As opposed to GDBTs or single-dataset deep learning methods, evaluating pretrained tabular models introduces additional complexity. Indeed, leakage can come in multiple forms. When LLMs are involved, there is a risk of memorization [9], and models trained on synthetic datasets [38] which try to mimic real-world distributions, can be unintentionally biased towards popular benchmarks.

While these two forms of leakage are subtle and hard to detect, a more direct form must be strictly avoided: When the same dataset (or a variant of it) is used during pretraining, and then it is evaluated as a downstream task. In such scenario there is inevitable severe data leakage, especially when running with multiple random test splits. The rest of the section explains how both TP-BERTa [84] and CM2 [86] suffer from such contamination with respect to our benchmark. As we briefly mention in §7, we advocate for improving TFM research by encouraging models that are practical to evaluate, by releasing several versions of each model, and providing default hyperparameters.

TP-BERTa We exclude TP-BERTa from our evaluation for two key reasons. First, their implementation assumes that every example is treated as a serialized single sequence, which allows for

a maximum length of 512 tokens, as elaborated in Appendix A.1. While this decision is efficient for datasets with a low amount of features and no free-text presence, around half of the datasets in our benchmark are too long for that limitation, as they either contain too many features or long free-texts. Second, TP-BERTa's pretraining uses datasets that appear in our evaluation set, as listed in Table 6 of their paper [84]. It is evident that several datasets overlap directly with datasets in our benchmark §4 (e.g., 1510_fifa , 1368_IMDb-Ratings , 1639_Melbourne ), disqualifying them for our purposes. Furthermore, we observe a concerning overlap between their pretraining and downstream task datasets (e.g., airlines , sf police and diabetes ). We believe that this questions the validity of their evaluation, and that such contamination poses a serious challenge for the TFM community which could be substantially addressed by better tabular data repositories [76].

CM2 CM2 was pretrained over OpenTabs , a compilation of more than 2,000 datasets drawn from public tabular data repositories, including OpenML and Kaggle. While this collection is valuable, pretraining a model over these datasets compromises further evaluation of any of them. Naturally, the overlap with our benchmark here is extremely high, making it infeasible to use as a

baseline. Interestingly, their repository 27 lists TP-BERTa as a method trained on a subset of OpenTabs, reinforcing that the leakage is shared between the models.

E.2 Baselines Implementation and Hyperparameters

This section outlines the implementation and hyperparameter tuning strategy used for the baselines reported in §4. While each baseline has its own model-specific preprocessing pipeline, we apply two shared preprocessing steps to both TabSTAR (as detailed in Appendix A.1) and all the baselines: (1) We perform date preprocessing by using skrub's DatetimeEncoder , and (2) Apply a clipped z-score transformation for target variables in regression datasets.

Textual Feature Handling CARTE natively supports textual inputs, and the TabPFN-v2 API client 28 does as well, although its implementation details remain undisclosed. As all other baselines do not natively support free-text features, 29 we preprocess these features into fixed-size embeddings using skrub 's TextEncoder , which internally applies a frozen e5-small-v2 encoder to each semantic column. This aligns with the encoder used in TabSTAR, enabling a fair comparison across models. There are, however, two key differences in how TabSTAR handles these embeddings: First, the embeddings are specifically finetuned for the task, contributing significantly to its strong performance as shown in §5 and further analyzed in §G.2. The second detail is that skrub applies dimensionality reduction to 30 dimensions, as proposed by [32]. This compressed representation performs comparably to the full embedding space, while offering improved inference efficiency.

Hyperparameter Tuning For the tuned models we use the Optuna package 30 with random search, with a budget of 4 hours for every run and parallelizing trials on 8 CPU cores. We use 5-fold cross-validation and take the best configuration selected based on this mean score. We use it then to retrain the model on the full training data.

E.2.1 TabPFN-v2

We run TabPFN-v2 using their API client which supports text features. 31 While the intrinsic details of their textual handling remain undocumented, it's reasonable to assume that it resembles the processing we apply to GBDTs, as their model leverages ICL and their architecture has no textual encoder.

E.2.2 TabICL

Werun TabICL using its package 32 with the default configuration. Since TabICL can only be evaluated on classification tasks, we exclude it from the regression analysis.

E.2.3 TabDPT

We run TabDPT using its package 33 with the default configuration. While it achieves state-of-the-art performance on some regression tasks, we encounter severe performance issues on others. Due to this inconsistency, we exclude TabDPT from the reported results and are in contact with the authors to investigate potential implementation issues.

E.2.4 CARTE

We run CARTE using its package, 34 which inherently performs k -fold cross-validation. After consulting with the authors, we set k = 5 for efficiency instead of 10 . We do grid search over their recommended learning rates, 35 and we take the best-performing variant per dataset split.

E.2.5 RealMLP

We run RealMLP using its official implementation in the pytabkit package. 36 Following the advice of its authors, we disable label smoothing and optimize for cross_entropy for binary classification and 1-auc_ovr for multiclass classification, and keep all other default hyperparameters. For the tuned version, we follow the hyperparameter search space suggested by [39] as detailed in Table 15.

E.2.6 CatBoost

We run CatBoost using the catboost package 37 and run the default configuration suggested by [26] by setting early _ stopping _ rounds = 50 , od _ pval = 0 . 001 , iterations = 2000 . For the tuned version, we follow the hyperparameter search suggested by [38] as detailed in Table 16.

E.2.7 XGBoost

We run XGBoost using the xgboost package. 38 For the default configuration, we follow the suggestion of [26] and use: booster = ' gbtree ' , early _ stopping _ rounds = 50 , n _ estimators = 2000 . For the tuned variant, we follow the hyperparameter search space suggested by [38], as shown in Table 17.

E.2.8 LightGBM

We run LightGBM using the lightgbm package. 39 and its default implementation. For the tuned variant, we follow the hyperparameter search space suggested by [38], as shown in Table 18.

E.2.9 Random Forest

We treat Random Forest as a weak baseline to establish a lower-bound reference for each dataset split. We run it with the sklearn package 40 and use its default configuration with n _ estimators = 100 .

F Extended Main Results

In this appendix we provide the main results for the experiment as reported in §5. As elaborated in §4, each model is evaluated on each dataset across 10 splits. Since performance scales vary between datasets, we follow the normalization approach proposed by [38], rescaling all scores to the [0 , 1] range. The final reported performance for each model is the average over these normalized runs, and we compute 95% confidence intervals using the standard normal approximation: ˆ µ ± 1 . 96 ˆ σ √ n .

F.1 Technical Limitations

As discussed in §5, TabPFN-v2 is unable to run on 4 datasets: C12, because it is a multiclass problem with more than 10 classes, and C02, C14 and R01 because they support inference for up to 500,000 cells. Attempts to run the model over a subset of the examples led to a significantly worse performance, and thus we decide not to report them to allow a fair comparison.

Additionally, CARTE is unable to run over 15 of the datasets in the benchmark due to a known bug 41 in their implementation for the PowerTransformation , which struggles in the presence of features with too less unique values. Furthermore, TabDPT's performance over a few regression datasets is especially low. Since we focus on classification tasks, we exclude this model's performance on this task. Finally, RealMLP, fails to run on R01 in the unlimited setting due to memory constraints.

F.2 Dataset Level Performance

We report AUROC for classification and R 2 for regression, with 95% CIs computed over the 10 runs for each dataset. Tables 19 and 20 summarize classification performance on datasets with up to 10K and over 10K examples, respectively. Tables 21 and 22 to the same for regression tasks. For conciseness, datasets are referred by their ID from Appendix D.

F.3 Head-to-head comparisons

We compare the performance of TabSTAR against each of the models in head-to-head comparisons. We report win rate, which can be seen as a private case of the normalized metric with only two models. We exclude failed runs when comparing to CARTE and TabPFN-v2. Table 23 shows the performance of TabSTAR against all models competing up to 10K examples, for both regression and classification, with 95% CIs over the win rate. Table 24 does the same for TabSTAR-Unlimited.

F.4 Running Times and Compute Information

Hardware All baselines are evaluated using a single NVIDIA A100-SXM4 GPUwith 40GB memory, and 8 CPU cores of type AMD EPYC 7742 64-Core Processor . The only exclusions are TabPFN-v2,

which runs on its API version, and CARTE, for which we employ several NVIDIA GeForce GTX 1080 Ti GPUs with 11GB memory, as their high latency and lack of default hyperparameters forced us to rely on more accessible hardware.

Running Times Table 25 presents the average running times of the different models per dataset, with up to 10,000 examples, excluding the tuned variants, TabPFN-v2 and CARTE. TabSTAR's average running time for a downstream task ranges from 59s (C04) to 9,332s (R24). Although higher than a single run of other baselines, these times remain far below the 14,400s (4 hours) typically required for their tuned counterparts. TabICL achieves considerably faster running times; however, their ICL-based approach limits them to smaller datasets, and inference is as costly as training (see §5). TabPFN-v2, though unreported, is subject to the same limitation.

F.5 TabSTAR's Cost Analysis by Number of Features

While §5 reports average computational costs, these values vary considerably across datasets, depending on factors such as the number of examples, the feature count, and the amount of textual fields. In this subsection, we examine how TabSTAR's computational cost scales with the number of features.

Table 26 summarizes both latency and memory consumption of TabSTAR, aggregated over a group of up to 10 features, a group of 30-50 features and a group of more than 100 features. These datasets were randomly sampled from the pretraining corpus; consequently, the reported times may differ from those observed on the benchmark datasets. We observe that latency can increase by a factor of 5, and GPU memory consumption by a factor of 4. This behavior stems from the transformer's architecture, whose computational complexity is quadratic in the sequence length. This raises concerns for training TabSTAR over datasets with hundreds of features, and calls for improvements to expand to datasets of thousands of features.

Additionally, TabSTAR's memory consumption in classification tasks is affected by the number of classes due to its target-aware tokens. We assume that for TabSTAR, the effective number of features equals the actual number of features plus the number of classes. For highly multiclass datasets, this can become a limiting factor.

F.6 Number of Trials for Tuned models

The tuned models are optimized with a budget of 4 hours using 8 CPU cores. Each trial is executed on a single core, ensuring that at least one trial could be completed within the allocated time. Table 27 presents the number of hyperparameter trials per dataset.

G Extended Analysis

In this section we expand on the analysis results discussed in §6.

G.1 Evaluation Datasets for Analysis

All experiments are conducted on 20 datasets from the benchmark described in Appendix D. Each experiment reports performance using AUROC for classification and R 2 for regression. Each tables reports both regression and classification tasks, distinguishable by their ID. Furthermore, each experiment compares its variants to other models, excluding TabPFN-v2 and CARTE, which could not be executed on all datasets. Although this comparison covers only a subset of the benchmark and should not be interpreted as conclusive, the results nonetheless offer valuable reference points.

G.2 The Role of the Encoder Unfreezing (Q1)

Table 28 shows the results for each variant of the experiment presented in §6. It is evident that unfreezing the textual encoder yields a significant performance boost across datasets, as seen in Figure 6: The frozen TabSTAR variant is much worse than any other baseline! Furthermore, while finetuning a single layer gives a significant boost, it underperforms compared to 6 unfrozen layers.

G.3 The Effect of Pretraining (Q2)

We pretrain three TabSTAR variants on nested dataset subsets of size 16, 64, and 256. The 64-dataset variant contains the original 16 plus 48 new datasets, and the 256-dataset variant builds on those 64 by adding another 192. This cumulative design minimizes variance between variants so that performance differences reflect only the effect of increasing data volume.

While LoRA [42] is a powerful technique, it can't be applied to a randomly initialized model. Therefore, we perform full finetuning of the non-pretrained model, as explained in Appendix B.2.

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Normalized Score

TabSTAR

TabM

Unfreeze-6

Unfreeze-3

CatBoost

Unfreeze-9

TabICL

XGBoost

RealMLP

Unfreeze-1

LightGBM

TabDPT

RandomForest

Unfreeze-0

Classification - Up to 10K examples (8 datasets)

Default

Default

Tuned

Table 29 show the normalized results for classification and regression, and Table 30 shows the dataset level results. It is evident that for most of the datasets, improvement is observed when scaling, with the 256 datasets variant winning for almost all datasets. Figure 7 highlights this effect, showing that adding more datasets enables TabSTAR to outperform the baselines.

G.4 Numerical Verbalization (Q3)

We show the full results for the experiment in §6, with Table 31 illustrating the verbalizations in each variant. Note that we do not include an exact value verbalization, since it would increase the number of unique text inputs and place extra memory demands. The two variants which integrate numerical information into the verbalization dominate the experiment, although the improvement seems to be

Classification - Up to 10K examples (8 datasets)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Normalized Score

TabSTAR

Scale-256

TabM

CatBoost

Scale-64

TabICL

XGBoost

RealMLP

Scale-16

Scale-0

LightGBM

TabDPT

RandomForest

Default

Default

Tuned

marginal for some datasets. Interestingly, some datasets significantly underperform, with the R27 dataset completely failing the task. The addition of the quantile information on top of the bin seems to have limited impact, although marginally winning on the average performance. Figure 8 presents a comparison with baseline models, highlighting the importance of incorporating richer numerical verbalizations beyond relying solely on the column name.

Classification - Up to 10K examples (8 datasets)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Normalized Score

TabSTAR

TabM

Numerical-Full

Numerical-Range

CatBoost

Numerical-None

TabICL

XGBoost

RealMLP

LightGBM

TabDPT

RandomForest

Default

Default

Tuned

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new model called TabSTAR. The model achieves competitive\nperformance compared to GBDTs, neural nets and in context learner models on\nboth regression and classification tasks. This is done via a two phase\npre-train and finetune process. In terms of architecture, an unfrozen language\nmodel is used, with layers for fusion, interaction and prediction. A novel\napproach is the use of target-aware tokens that enable the model to work with\nany number of classes. Finally, the paper justifies the architecture choices\nthrough experiments.\n\nSTRENGTHS AND WEAKNESSES:\n## Quality\n\n- The experiments are thoroughly conducted with lots of results in the\n  Appendix that show competitive performance and justify many choices\n  made for training the model. Furthermore, the Appendix highlights some\n  efforts that did not lead to significant results, which will help the\n  community explore the space more efficiently.\n- The paper's claim about being a foundation model is not fully justified.\n  Foundation models such as TabPFN, TabDPT, and TabICL achieve their results\n  without finetuning on each dataset. This paper, however, uses five models\n  that are separately trained (contrary to the foundation model claim) and then\n  finetunes on each dataset. A fairer comparison would have been to use one\n  model and still finetune on each dataset but with baselines using\n  inference-time calibration (as done in LocaLPFN). Alternatively, more\n  emphasis should be put on the inference-time compute required for each model,\n  and that section should be in the main text, not the Appendix (F.3).\n\n## Clarity\n\n- The paper reads well and explains the ideas clearly and justifies most of\n  them with experiments.\n- The authors could touch upon the relation between number of classes and\n  performance and efficiency, since target-aware tokens scale with the number of\n  classes. This seems important as when comparing training dataset to\n  evaluation dataset, one observes the training datasets have many examples\n  with large number of classes while evaluation dataset (on which computational\n  cost/performance analysis is done) lacks such datasets.\n- In the Appendix A.3 Row-Level Attention is discussed. It seems like the\n  authors tried in context learning approach. Although, the details are\n  missing. What was the number of in context samples? Did the authors see any\n  sign of overfitting as reported in TabDPT (which motivates their use of SSL\n  methods)? This is a tangent in the Appendix and I am not raising this as a\n  negative point about the paper. But I believe this approach is of interest to\n  the community and adding the details to the Appendix would be valuable.\n\n\n## Significance\n\n- The model trains fast (<48h on A40), opening the door for community training\n  their own variant.\n- TabSTAR is good on classification tasks (possibly due to verbalization and\n  the text encoder and target-aware token being more helpful here) but not as\n  good in regression, even compared to TabPFNv2 which doesn't have per dataset\n  finetuning (on less than 10K setting).\n\n\n## Originality\n\n- The architecture is novel.\n- The method used to mix the verbalization with numeric values is novel.\n- Target-aware tokens are novel.\n\nQUESTIONS:\n- Are the final models trained with 6 unfrozen layers for the text encoder?\n- The NLP community has been using LoRA for finetuning, why the authors choose\n  to unfreeze the text encoder instead? Note that LoRA is already used for\n  finetuning stage. Figure 4 shows that 9 unfrozen layers underperforms 6\n  unfrozen layers. It seems like overfitting is a concern (validation is lower\n  but test is not). Note that the amount of text here is tiny compared to LLM's\n  pretraining dataset and things like catastrophic forgetting can be\n  concerning.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a tabular foundation model for datasets with textual features. Both numeric and textual features are summarized into verbalizations together with targets and pass through an encoder to build vector representations. These representations are then passed through a second interaction encoder that attends over all inputs. Finally, attended representations of targets are used to apply the regression/classification heads and generate predictions. Results on real-world datasets with textual features indicate that this approach can generalize to unseen tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n-The paper is well written and easy to follow. It proposes an innovative approach to represent numerical data through verbalizations of quantile binning.\n-Proposed architecture seamlessly integrates targets without making any assumptions on cardinality and LLM embeddings for textual data with end-to-end fine-tuning.\n-Experiments on real-world data show promising results on generalization against some of the leading baselines.\n\nWeaknesses:\n-Novelty is somewhat limited, other than numeric data verbalizations, the rest of the architecture are fairly standard components that have been explored.\n-Proposed model requires fine-tuning for each downstream task and is not really a \"foundation\" model like TabPFN where no finetuning is needed.\n-Experimental results are mixed, even though TabSTAR has almost 100 regression datasets in pre-training, the regression performance is  quite weak.\n\nQUESTIONS:\n-Is fine-tuning require for each downstream task? Table 3 shows the impact of pre-training but I couldn't find results where full pre-training is done but model is then applied as is without fine tuning. I think these results are important to show.\n-Why is regression performance so weak? Authors say that \"regression is not our main focus\" but there are quite a few regression datasets in pre-training yet the model is not generalizing well after. Analysis on why that happens would be useful to see.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes TabSTAR, a tabular foundation model that integrates the semantic understanding capabilities of language models. Its core innovation lies in the introduction of target-aware tokens, where all possible target values are included as input tokens, and logits are generated for each of them.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The problem addressed in this paper is important, and the high-level direction is well-motivated.\n\n2.The introduction of target-aware tokens is a meaningful and creative idea.\n\n**Weaknesses**\n1. The proposed method relies entirely on supervised pretraining. In practice, this raises concerns about how the model handles datasets without labels. There is a reason that language foundation models typically use next-token prediction with unlabeled data. If TabSTAR depends solely on supervised learning, scalability may be limited by labeled data availability. For example, TabPFN mitigates this issue using synthetic data. It is unclear whether TabSTAR employs a novel strategy to address this limitation.\n\n2. The method may struggle when the number of features grows large (e.g., 100+). As each feature is embedded (e.g., into 128 dimensions), memory usage increases rapidly, especially when applying Transformer layers for both fusion and interaction. This could raise scalability concerns.\n\n3. The paper repeatedly emphasizes the importance of encoder unfreezing, but offers little explanation beyond empirical evidence. A theoretical rationale would help justify this design choice. If such justification is not possible, it may be better to reduce the emphasis. Moreover, if the number of unfrozen layers is treated as a hyperparameter, the paper should clarify whether a principled selection strategy exists, or if it is simply tuned using a held-out validation set.\n\n4. Both the fusion and interaction modules process embeddings through Transformer layers. It is unclear how the paper ensures that fusion occurs during the fusion stage, and interaction occurs during the interaction stage, given the architectural similarity.\n\n5. The evaluation section underexplores the implications of pretraining. Since pretraining is the core contribution, it would be useful to assess the model under different usage paradigms beyond fine-tuning.\n\n6. The set of baselines is limited. As a pretraining-based approach, TabSTAR should be compared against recent few-shot or unsupervised tabular learning methods. Currently, none are included.\n\nQUESTIONS:\n1. Please clarify how TabSTAR can be scaled to larger dataset volumes, especially in scenarios where labeled data is scarce. (W1)\n\n2. Provide time and memory usage analysis to assess computational efficiency. (W2)\n\n3. Include a rationale for encoder unfreezing, and describe how the number of unfrozen layers is selected. (W3)\n\n4. Add an ablation study of each component. Also consider evaluating whether the current architecture performs better than a simpler version that just stacks the two modules with equivalent Transformer layers. (W4)\n\n5. Improve the evaluation. Instead of only fine-tuning, test common unsupervised evaluation settings - such as aggregating embeddings and training XGBoost or linear probes on top. Also include comparisons with few-shot or unsupervised tabular learning methods such as SimSiam [1], SCARF [2], STAB [3], MET [4], STUNT [5], LFR [6], and TST-LLM [7]. If finishing these experiments is hard in time, at minimum, the paper should include a methodological comparison and explain why such methods were not used. (W5, W6)\n\n[1] SimSiam: Exploring Simple Siamese Representation Learning (CVPR 2021)\n\n[2] SCARF: Self-supervised Contrastive Learning Using Random Feature Corruption (ICLR 2022)\n\n[3] STAB: Self-supervised Learning for Tabular Data (NeurIPS Workshop 2022)\n\n[4] MET: Masked Encoding for Tabular Data (NeurIPS Workshop 2022)\n\n[5] STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables (ICLR 2023)\n\n[6] LFR: Self-supervised Representation Learning from Random Data Projectors (ICLR 2024)\n\n[7] TST-LLM: LLM-Guided Self-Supervised Tabular Learning With Task-Specific Pre-text Tasks (TMLR 2025)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents TabSTAR, a new foundation model designed for tabular datasets that contain text features. The central idea is to create more meaningful, task-oriented feature representations by making the model \"target-aware.\" This is achieved by feeding the model not just the input features but also tokens representing the possible target values (for classification).\n\nThe architecture involves five main steps:\n\nVerbalization: All features (numerical, categorical, and text) are converted into human-readable strings. For example, a numerical age of 45 might become \"Age: 40-50 (Quantile 50-60%)\".\n\nEncoding: A pretrained text encoder (e5-small-v2) transforms these strings into embeddings. Crucially, some layers of this encoder are unfrozen, allowing them to be fine-tuned for the specific task.\n\nFusion: Embeddings for numerical features are combined with their semantic string representations.\n\nInteraction: A Transformer encoder processes the embeddings of all features and target tokens, allowing them to interact and influence each other.\n\nPrediction: A shared prediction head uses the final representations of the target tokens to output probabilities for classification or a value for regression.\n\nTabSTAR is pretrained on 350 tabular datasets and then fine-tuned for specific downstream tasks. The authors claim that this approach achieves state-of-the-art performance on classification tasks for medium-to-large datasets containing text, outperforming Gradient Boosting Decision Trees (GBDTs) and other tabular foundation models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nFocus on Textual Features: The model is explicitly designed to handle tabular data with rich text, a common and important type of real-world data that has been historically underrepresented in tabular learning benchmarks.\n\nNovel Target-Aware Architecture: The introduction of target-aware tokens is an interesting architectural choice. It allows the model to learn representations that are specifically tailored to the prediction task and enables an architecture free of dataset-specific parameters, which is a key advantage for a general-purpose foundation model.\n\nSystematic Ablation Studies: The paper provides a good analysis of its key components, showing that unfreezing the text encoder and using detailed numerical verbalization both significantly improve performance. The results also demonstrate that performance scales with the number of pretraining datasets, supporting the foundation model approach.\n\nEmphasis on Reproducibility: The authors provide details on their experimental setup and plan to release pretrained models and code, which is a valuable contribution to the research community.\n\n\nWeaknesses\nLimited Novelty: The core method of converting tabular data into text (\"verbalization\") and processing it with a transformer-based model has been explored in prior work (e.g., TP-BERTa, TransTab). The paper does not sufficiently distinguish its approach to establish clear novelty over these existing methods.\n\nIncomplete Baselines: For a paper claiming to be a new Tabular Foundation Model (TFM), the experimental comparison is missing several important baselines from the same category, such as XTab. The comparison set is primarily GBDTs and TabPFN-v2, which doesn't fully represent the current TFM landscape.\n\nMisleading Framing around \"LLMs\": The model does not use a Large Language Model (LLM) in the conventional sense (i.e., a model with billions of parameters and emergent reasoning abilities). It uses a much smaller text embedding model (e5-small-v2). This limits its ability to perform complex reasoning or leverage the vast world knowledge associated with true LLMs. The improvements likely stem from better-tuned embeddings rather than higher-level reasoning.\n\nUnconvincing Performance Gains: The reported improvements are not consistently strong. TabSTAR is significantly outperformed by GBDTs in regression tasks. Even in its strong suit, classification, the performance gains over TabPFN-v2 are modest, with overlapping confidence intervals in the summary charts, suggesting the difference may not be statistically significant in all cases\n\nQUESTIONS:\nNovelty: Can you clarify the primary novelty of your verbalization and encoding method compared to prior work like TP-BERTa, which also tokenizes numerical and categorical features for a transformer? \n\nBaselines: Why were other foundation models for tabular data, such as XTab, not included in the comparison? Their inclusion seems critical for validating the claim of state-of-the-art performance in the TFM space. \n\nModel Capabilities: Since the model relies on an embedding model rather than a full LLM, what are the theoretical limits of its reasoning ability? Do you believe the performance gains come solely from adaptive embeddings, or is there evidence of more complex semantic understanding?\n\nRegression Performance: The paper notes that performance lags GBDTs on regression tasks. Beyond scaling the pretraining data, are there specific architectural weaknesses that you believe cause this gap? For instance, is the verbalization strategy inherently less effective for continuous targets?\n\nTarget-Awareness in Regression: For regression tasks, a single target token representing the target's name is used. How does this provide meaningful \"awareness\" compared to the classification case, where all possible target values are provided as input?\n\nLIMITATIONS:\nThe paper rightly acknowledges some of its limitations in the discussion section. Based on the work presented, the key limitations are:\n\nWeak Regression Performance: The model is not competitive with established GBDT models on regression tasks, indicating that the current architecture is not well-suited for predicting continuous values.\n\nDependence on an Embedding Model: The model's capabilities are constrained by the pretrained text encoder it uses. It does not possess the broad reasoning and knowledge integration abilities of a true LLM.\n\nIncomplete Benchmarking: The lack of comparison against a fuller suite of contemporary tabular foundation models makes the SOTA claims difficult to fully validate.\n\nMarginal Gains in Classification: The performance improvements in classification, while present, are not overwhelmingly large, and the overlapping confidence intervals with the next-best model (TabPFN-v2) suggest the advantage may not be statistically robust across all datasets.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents TabSTAR, a novel tabular foundation model that incorporates semantically target-aware tokens into an unfrozen pretrained text encoder, enabling end-to-end fine-tuning of textual and numerical features for tabular prediction tasks. The authors curate a large pretraining corpus of 350 tabular datasets (with both classification and regression tasks), demonstrate strong empirical performance on 50 benchmarks containing free-text or high-cardinality features, and show that TabSTAR scales with more pretraining data.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n- Clear motivation & positioning. The paper convincingly identifies the limitations of static, target-agnostic text embeddings in existing tabular ML and foundation models, particularly when handling free-text or high-cardinality categorical features.\n- Extensive related work section covering major approaches for tabular data.\n- Well-written and easy-to-follow.\n- Well-curated pretraining dataset collection and extensive efforts in avoiding contamination.\n- Thoughtful choice of the evaluated baseline models. In particular, the exclusion of models such as TP-BERTa or Tabula, are very reasonable given the severe leakage concerns and unreasonable training times. \n- Extensive appendix with all required information to reproduce the results.\n- Insightful analyses. Ablations on the number of unfrozen encoder layers, the effect of pretraining dataset count, and different numerical verbalizations are very helpful.\n\n- High novelty, as there is no comparable approach that uses \"target-aware tokens\" in tabular tasks.\n- Empirical results are substantial, covering various datasets and providing clear comparative analyses against robust baseline methods.\n\n\n### Weaknesses:\n- Evaluation of baselines using a 4h time budget without reporting the no. of successful trials per model. The average computation times per 10k features in the Appendix are a reference, but without knowing how many trials were truly evaluated it is impossible to tell whether the baselines are well-tuned or not. As they are currently presented, the conclusions are only valid under the 4h time budget. A practitioner would invest much more time to solve a single task. Therefore, the general claim 'TabSTAR achieves state-of-the-art performance' cannot be substantiated. Therefore, the claim in the abstract should either be changed to reflect that the claim only holds with a 4h time budget, or more information on the no. of successful trials per model is required, to verify that the tree-based models were truly tuned close to their full potential.\n- No classic deep learning methods (i.e. TabM or RealMLP) included. \n- Despite being highlighted, computational cost and efficiency are not deeply analyzed. Given that gradient boosting decision trees (GBDTs) are highly performant with lower computational overhead, this factor should be more explicitly addressed. How does TabStar compare to tree-based models in terms of memory efficiency and inference time?\n- Training on some of the benchmark datasets and evaluating five different models adds additional complexity to the evaluation, and, at the same time requires an additional ablation. How is the performance of a single TabStar model without using the additional training data from the downstream task benchmark?\n- While the ablations are insightful, the core innovation that is also present in the full title have not been studied in isolation. Is it really beneficial to use target-aware tokens, or is it simply enough to retrain (unfreeze) the layers without providing semantic target information?\n\nQUESTIONS:\n- Could you please clarify the no. of evaluated trials per model and dataset? Also, were the results of the predictions of tree-based models of the five folds averaged (bagging)?\n- Can you provide quantitative evidence isolating the benefit of a) including column names versus solely the values; b) including target tokens instead of just unfreezing the text embeddings using the values?\n- Does the fusion layer provide any gain over simple concatenation? I am unsure whether this part of the modelis important.\n- Figure 4: Looking at the right plot, it seems like the normalization was done solely on the TabStar versions, without the other models. Although the difference to the 0-unfrozen setup is quite large, the true impact on the final downstream performance cannot be assessed using this plot. The same seems to be the case for Tables 3 and 4. How would the plot look like if the other models would be included in the normalization?\n- Table 3: The gains from pretraining seem surprisingly high. WHat are your thought on why TabStar is much worse than conventional neural networks in the 'classic' training regime?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}