Table Foundation Models: on knowledge pre-training for tabular learning
Authors: Myung Jun Kim
Félix Lefebvre
Gaëtan Brison
Alexandre Perez-Lebel
Gaël Varoquaux
Venue: Accepted by TMLR ()

Abstract: Table foundation models bring high hopes to data science: pre-trained on tabular data to embark knowledge or priors, they should facilitate downstream tasks on tables. One specific challenge is that of data semantics: numerical entries take their meaning from context, *e.g.*, column name. The traditional approach combines column-specific data preparation with tree-based models that adapt to column specificities. Pre-trained neural networks that jointly model column names and table entries have recently boosted prediction accuracy. While these models outline the promises of world knowledge to interpret table values, they lack the convenience of popular foundation models in text or vision. Indeed, they must be fine-tuned to bring benefits, come with sizeable computation costs, and cannot easily be reused or combined with other architectures. Here we introduce TARTE, a foundation model that transforms tables to knowledge-enhanced vector representations using the string to capture semantics. Pre-trained on large relational data, TARTE yields representations that facilitate subsequent learning with little additional cost. These representations can be fine-tuned or combined with other learners, giving models that push the state-of-the-art prediction performance and improve the prediction/computation performance trade-off. Specialized to a task or a domain, TARTE gives domain-specific representations that facilitate further learning. Our study demonstrates an effective approach to knowledge pre-training for tabular learning.

Published in Transactions on Machine Learning Research (08/2025)
Table Foundation Models: on knowledge pre-training for
tabular learning
Myung Jun Kim
myung.kim@inrio.ff
SODA Team, Inria Sacloy
Félix Lefebvre
fekk.lefeebere@hhrri..p
SODA Team, Inria Suclay
Gaëtan Brison
gb2764 edu
Hi! PARTS, Institut Polytechnique de Paris
New York University
Alexandre Percz-Lebel
dle/@fondamentallccc
SODA Tram, Inria Suclay
Pundamental Technologies
Gaël Varoquaux
gael, www.uuuurumirra...i...u
SODA Team, Inria Saclay Probabi.ai
Reviewed OpenReview:
Abstract
Table foundation models bring high hopes to data science: pre-trained tabular data to embark knowledge priors, they should facilitate downstream tasks tables. One specific challenge is that of data semantics: numerical entries take their meaning from context, e.g., column name. Pre-trained neural networks that jointly model column names and table entries have recently boosted prediction accuracy. While these models outline the promises of world knowledge to interpret table values, they lack the convenience popular foundation models in text or vision. Indeed, they must be fine-tuned to bring benefits, come with sizeable computation costs, and cannot casily be reused combined with other architectures. Here introduce TARTE, foundation model that transforms tables to knowledge-enhancet vector representations using the string to capture semantics. Pre- trained on large relational data, TARTE yields representations that facilitate subsequent learning with little additional cost. These representations can be finc-tuned or combined with her learners, giving models that push the stale-of-the-ar. prediction performance and improve the prediction/computaaae domain, TARTE gives domain-speciffc representations that facilitate further learning, Our study demonstrates effective approach to knowledge pre-training for tabular learning.
performance trade-off. Specialized
task or
1 Introduction: promising, but limited, foundation models for tabular learning
Tables, that often contain an organization's precious data, come with specific challenges to machine learning, as they contain columns of different types and nature. Until recently, doep Icarning brought little benefits over tree- based models for typical tables (Shwartz-Zit Ziv & Armon, 2022; Grinsztajn et al., 2022; McElfresh al., 2024). This is in contrast with images, signals, or text, where the latest advances driven by

---

Published in Transactions on Machine Learning Research (08/2025)
foundation models: neural networks pre-trained on a large amount of background data that can be adapted and reused for a great variety of tasks (Bommasani et al., 2021). Pivotal to this vision was BERT (Devlin et al., 2019), showing that repurposing transformer- based backbones (Vaswani et al., 2017) could boost many natural language tasks. Early signs of similar breakthroughs are visible for tabular data. Using pre-trained transformers on synthetic numerical lables, TabPFN outperforms traditional approaches for classífication and regression, and can perform data generation and density estimation (Hollmann et al., 2023; 2025).
Beyond modeling numbers, another important challenge of tabular learning is data semantics. Human beings use string in table entries column names to understand the table. Progress in table understonding models leverages these strings (Zhang et al., 2024). But for most tabular learning models, including the TabPFN lamily, the strings are challenging, Such models operate on numbers and they demand that the data scientist convert all columns to numerical representations, a crucial and often tedious operation. Rather, strings can be an opportunity to bring world knowledge in pre-trained models for tables (Kim et al., 2024; Yang et al., 2024). But such existing pre-trained models cannot casily be reused and come with large operational costs, as they require costly fine-tuning to oulperform tree-based models.
Indeed, applications must navigate trade-offs between prediction accuracy and operational costs (Bernardi et al., 2019; Paleyes et al., 2022). One benefit of deep learning in vision or text has been to reduce operational complexity via model reuse across tasks (Zhai et. al., 2019). Mature foundation models have pushed much further this reuse and operational convenience, as they can easily be specialized with little downstream data (Bommasani et al., 2021). However, their computational cost for pre-training but also inference is a real concern (Varoquaux et al., 2024), as illustrated by the stock-market turmoil (Saul, 2025) created by the announcement of the efficient. DeepSeek-R1 model (Guo et al., 2025). Table foundation models hold the same promise and face the samn peril. Capturing strings and column namnes enables them to model multiple tables without matching columns (Kim et al., 2024; Yang et al., 2024). This potential is compromised by their compute cost: Kim et al. (2024) report times 200x slower than XGBoost, a very strong baseline. Idcally, repurposing a foundation model should be less, not more, costly than starting from scratch.
Here we introduce TARTE (Transformer Augmented Representation of Table Entries); a pre-trained tabular model. TARTE uses knowledge pre-training capture associations between strings and numbers. Corre- sponding representations facilitate downstream lcarning, fine-tuned or reused as such in combination with other models. In both settings, it gives predictors that outperform the best baselines, based on trees or neural networks. Like recent tabular models, TARTE builds on the playbook of foundation models: broad pre-training to bake in implicit priors that help for a wide variety of tasks. But TARTE takes it much further: we show that its knowledge pre-training does capture information easy to reuse, unlike prior models of strings and numbers where the benefits come from fine-tuning rather than pre-training.
In section 2, we analyze the challenges that tabular data pose to foundation models, and the progress overcoming them across time. We present the TARTE model in section 3: the architecture, transformer that models string or numerical entries, enriched by column names; the pre-training, on large knowledge base enriched with numerical attributes from Wikidata; different post-training options, fine-tuned or frozen, combined with another models. Section 4 gives an extensive empirical study. We first show how, given a downstream table, various post-training strategies of TARTE improve the state of the art in tabular learning for different trado-offs between prediction accuracy and computational cost. We study both a small-sample regime, from n 32 to 1 124, and mid-sized tables, n 000. We then study the factors of success of knowledge pre-training, showing that it needs diverse and rich pre-training data and works best on complex tables with strings similar to the pre-training data. Finally, we show that TARTE can also be specialized to a domain, enabling form of transfer learning. Section 5 ends with a discussion and conclusion.
2 The unfolding of tabular foundation models
Compared to neural networks, tree-based models have a lead start for tabular learning: their inductive biases mathh well the properties of tabular data (Grinsztaju et al., 2022). Progress in dedicated neural architectures has recently been closing the gap for largo-enough datasets (Borisov et al., 2022; Ye et al., 2024). But tabular foundation models bring the promise of benefits for data of small to moderate size. van Breugel & van der Schaar (2024) argue they should be research priority, calling for developing properties important to tabular
2

---

Published in Transactions on Machine Learning Research (08/2025)
applications such as cross-dataset modeling well as handling tables with different types, e.g., numbers and strings. Recent progress on this agenda has required overcoming many table-specific challenges.
PFNs: learning priors for numerical tables TabPFN (Hollmann et al., 2023) brings to labular learning key ingredients of the success of foundation models: modeling context with transformers. The PFN (prior fitted network) is pre-trained over many datasets chosen to match the domain of interest, here tabular learning. The prediction is "in context": the training set is given as context in the transformer, which uses it to complete the query in a forward pass. As pre-training requires huge amount of datasets, these must be synthetic, computed with sophisticated random processes. Improving architecture and data generation, the followup TabPFNv2 (Hollmann et al,, 2025), and related works (Qu et al., 2025; Liu & Yc, 2025; den Breejen & Yun, 2024), leads to reliably outperforming tree-based models on purely numerical tables. Much ongoing work improves this type of approach, e.g., with better post-training to specialize a model on downstream data (Thomas et al., 2024; Feuer et al., 2024; Koshil et al., 2024; den Breejen et al., 2024, etc).
Modeling varying schemas and data semantics The agenda of table foundation models implics Icarn- ing across tables and model reuse, and as a consoquence modeling tables with different "schemas", different columns with different. information. Such setting breaks traditional machine-lerriing models used on tables, which need correspondence in columns to form features. Given tables with varying number of columns, pre-trained transformers are useful again (even without the in-context learning of PFNs) to construet joint representations ions from a varying number of inputs (Zhu et. al., 2023; Chen et al., 2024). Going further, models should idcally use the data semantics of columns. For instance, two columns may contain numbers, but one being age and the other weight. Capturing these semantics is important, if only to bridge related information across tables. Column names help. A variety of transformer- based models have tackled learning across tables with different columns by adding the columns names to the data used as input: Wang & Sun (2022) learn and transfer across various clinical-trial datasets (and followups, Yang et al., 2024; Spinaci al., 2024). But, without broad pre-training, Lransformer-based models do not outperform tree-based models in general.
Language models as tabular learners Large language models (LLMs) are the epitome of models cap- turing much via broad pre-training, including world knowledge. They work with free-flowing text, not constrained by a schema, and understand the corresponding semantics. They can be adapted to tables e.g., by turning rows into sentences (Dinh et al., 2022; Hegselmann et al., 2023). Dedicated fine-tuning turns LLMs to tabular learners, best performers in very few shot settings (Wang et al., 2023; Gardner et al., 2024). based approaches on tables have met more success for tasks beyond tabular learning, such as table understanding, question answering, recognizing columns or entries (Herzig et al., 2021; Zhang et al., 2024).
Pre-trained models of text and numbers The road to build foundation models for tabular learning has stretched between focusing on numbers, as the TabPFN literature, and adapting language models, that deal naturally with the strings in the tables. A body of work shows that tabular learning needs modeling strings but also numbers as such, rather than relying on the tokenization of LLMs. Yan et al. (2024) integrates an LLM but retrain on tables with discretized numerical features. Further from LLMs, Yang et al. (2024) use the column name, cell value, and data lype as inputs to a transformer, pre-trained over many tables. CARTE (Kim et al., 2024) reliably outperforms tree- based models on small datasets by pre-training on relational data graph transformer with an attention mechanism that combines the column name with string encodings or numbers. These models, however, rely on fine-tuning and incur large computational costs.
Reusing table foundation models Language or vision models such as BERT (Devlin al., 2019), CLIP (Radford et al., 2021), or the many followups ended up being called "foundation model" because their case of reuse and specialization to many application-speeciic models. The huge popularity of specializing models is visible for instance on the huggingface hub, that hosts more than million models, many of which are derived by reusing already pre-trained models, to offset pre-training costs. LLMs can even give backbones for table predictors, as mentioned above. Reuse and transfer is also high stakes for tabular data (Levin et al., 2023), and here modeling across tables with unmatched columns is particularly useful, to bring in data with different schema. This ease of specialization and reuse has not really been demonstrated for tabular data. Maybe the closest result comes from CARTE (Kim et al., 2024) which demonstrates benefits of specializing to
3

---

Published in Transactions on Machine Learning Research (08/2025)
Release
Production
Country
Language
Popularity
Revenue
Runtime
layers
Norm
Jumanji
1995.12.15
Tristar Pictures United S States
English
21.94694
262,797,248
104
Feed Forward
Add Noom
heads
A.I.
2001.06.29
Dream Works United States
English
13.24936
235,926,552
146
Matmul
Softmax
LMO Language Model DC) Danetine comvertor
Matmul & Scale
Readout [?]
LMCTitie)
(Mellaa)
(Producttion
IM (Revenue)
M((untimm)
Linear (Col) Pr(E)
D(2001 06. 29)
LM (Revenne)
A (Iuuatimm))
AM(AL.)
[Mr(rramm Works)
Linear (Cell) py(K)
LM (Releasse))
235, 926, 552
146
TARTE Input (Z)
TARTE Transformer
Figure 1: Transformer-based architecture of TARTE. TARTE models
in table set of
column(() E) cell(X) pairs. Given tabular data with multiple data types, TARTE maps the representation of column and cell values to the same dimension using language model (LMM. From the mapped input, the transformer takes in a combination of both column and cell information to contextualize the cell content.
domain defined by topic, but tables from different sources. However, it uses a computationally expensive joint learning, where the model must be fine-tuned on the various datasets. To achieve full benefits of table foundation models, designing modols that can be easily specialized to given domains or tasks is a priority.
3 TARTE: A backbone for knowledge pre-training
TARTE is easily-reusable pre-trained model that encodes dala semanlics across helerogeneous tables by pre-training from large knowledge bases. This section details the main components of TARTE: (1) a transformer-based architecture that models the data semantics of table entries via the dependencies between columns and cells; (2) knowledge pre-training from rich background information stored in large knowledge bases; (3) effective post-training to reuse knowledge pre-training in diverse downstream tabular tasks.
3.1 A transformer-based architecture
As with the success in LLMs (Devlin et al., 2019; Achiam et al., 2023, ...) and pre-trained models for (abular data (e.g., Hollmann et al., 2025; Kim et al., 2024), TARTE builds upon a variant of the transformer architecture (Vaswani et al., 2017). Designing such a variant entails 1) a suitable transformation of input data vector representations, for instance word tokenization and positional encoding in LLMs, 2) neural architecture based on the self-attention mechanism to capture the complex dependencies across inputs.
To learn across tables, a central challenge is to find common representation of tables across heterogeneous datasets. Tables store diverse and incongruous information, despite the structured layout of rows and columns. To name a few challenges, tables represent information with different number columns, data types (e.g., numerical or discrete), and naming conventions (e.g., France or FR). Combining column name with cell values enables representing diverse columns (scc appendix A.1). Diverse data types call for tokenization of strings (Wang & Sun, 2022) or using language model (Yan et al., 2024, discretizing numerical values). Additionally, Kim et al. (2024) represents each row in a table with graph, where each cells and columns are represented with nodes and edges, respectively, with embeddings of strings (Fast Mikolov et al., 2017).
TARTE borrows from Kim et al. (2024) the modeling of columm(() cell(X) pairs but loosens the graph structure. Figure/ 1 shows the modeling process of a table entry to a suitable input for the transformer. Given a table of & columns with multiple data types, the i-th row set XX)] in which all the components are mapped to the same dimension d using a language model. More specifically, with column names (ey) 1 datetime converter, D(-), and language model, L.M.),, have
4

---

Published in Transactions on Machine Learning Research (08/2025)
has
located
website
known for Mona
Batch Sample
Louvre
louvre.fr
(In-batch) Negatives
Louvre (Input)
TARTE Transformer
has name Louvre
Readout(1)
WIKIDATA
Knowledge Extraction
located in Paris
Linear (x(FF)
"Paris' 'London'
Contrastive Loss
Data curation Enrichment
website louvre.fr
yagox knowledge
Linear (C (Cell)
Px
known for
Positive pairs
Large Knowledgebases
Figure 2: TARTE Pre-training. We extract. facts from knowledge bases and replicate the input structure in Figure Then, batches are constructed with positive samples by replacing parts of information (e.g., Paris to London). TARTE is trained with contrastive learning using in-batch negatives (Chen et al., 2020b). The logos modified from YAGO and Wikidata, used under CC BY 4.0 and CCO, respectively.
E, (e,) X; 2) Ej e ERd if xf is numerical,
ER,
x; LM(x)) xj = D(af) E ERd if z) datetime.
ER" if 2) is categorical/string
The language model enables TARTE to work with an open set of vocabulary, without requiring any inter- vention on string entries, hence bypassing the complex column or entity matching problems. It paves the way for TARTE to bring data semantics across tables (sce subsection 4.3).
To obtain the transformer input Z E (k++ for the i-th row, we simply add embeddings of linearly mapped column and cell information, Pe(E') and px px(X), and stack learnable vector T R4:
z stackkT; + px
for column ll,,..,,
where pl-) = Linear LReeaa(Re and T E Rd works as the readout element (similar to the [CLS] token in Devlin et al. (2019)). The processed input Z then fed to an encoder-ban baseed transformer with the typical multi-head self-attention and feed-forward module (Vaswani et al., 2017).
3.2 Knowledge pre-training from large knowledge bases
Figure 2 summarizes the overall pre-training process of TARTE. TARTE learns data semantics of heteroge- neous tables by pre-training on large knowledge bases, containing millions of real-world facts.
Pre-train data Capturing tabular knowledge requires pre-training from diverse tabular data. While the best. models for numerical data use synthetic data generation (Hollmann et al., 2023; 2025), Kim et al. (2024) have pushed the idea of pre-training from knowledge bases, due to their resemblance to tabular data. However, the high level of curation (particularly for YAGO3, Mahdisoltani et al., 2013) leads to datasets that lacks the diversity of relational (column) information found in real-world tables (see appendix A.3).
Going beyond Kim al. (2024), TARTE expands the coverage of pre-train data by combining two large knowledge bases, YAGO4.5 (Suchanek et al., 2024) and Wikidata (Vrandečíć & Krötzsch, 2014). In essence, YAGO4.5 is cleaned version of Wikidata with simplified taxonomy and much fewer properties to facilitate automated reasoning. To build our datasct, we restrict YAGO4.5 to entitics with Wikipedia page as these come with abundant information, valuable for pre-training. YAGO4.5 includes relatively little numerical information compared to downstream tables, and lacks diversity in relational information, especially for numerical triples. To address these shortcomings, we enrich with numerical facts (numbers and dates) from the more comprehensive but noisier Wikidata. The resulting knowledge base describes over 5.5 million entities and includes 30 million facts with 687 distinct relations (see appendix A.3).
5

---

Published in Transactions on Machine Learning Research (08/2025)
Preprocessing the pre-training data The curated dataset is set of triples with 'head', 'relation', and 'tail', in which and I. describe h. For example, fact 'Louvre is located in Paris' is represented as r,)) where the elements are 'Louvre', 'is located in', and 'Paris', respectively (Figure 2). To enable pre- training, it requires additional steps of preprocessing for each data type of strings, numbers, and datetimes. For strings, we build a look-up table of embeddings built from language model. In particular, we choose Fest (Mikolov et al., 2017): text entries in tables mostly contain one or a few words, too short for LLMs. For numerical values, including datetimes converted into fractional years, we perform relation-wise power transformation (Yeo & Johnson, 2000). Power transform has shown to be effective in numerous tabular learning studies, including pre-trained models (Hollmann et al., 2023; 2025; Kim et al., 2024).
Batch sampling and castrvv loss To construct a batch for TARTE pre-training, we first select. N,, entities and extract related facts for each. For example, if "Louvre" is selected, we extract its corresponding information, such as location, website, and its well-known art Mona Lisa (sce Figure 2). In most cases, cach entity would have different number of associated facts. However, there only a fixed number of columns within a given table. Thus, we trim the number of related information for each entity to have a fixed number of facts across the batch. To enable contrastive learning, we also include positive samples, which are generated by replacing one or two of the facts with different information for each entity (see appendix A.4).
The embedding of a given table row is assembled with a linear readout T from the output of the transformer. We then apply the contrastive learning framework of Chen et al. (2020b), in which other entities inside the batch are considered as negatives. In contrast to the widely-used cosine similarity, we take the Gaussian kernel with median distance as the bandwidth. We then use the InfoNCE contrastive loss (Oord et al., 2018).
Model specifications The transformer architecture of TARTE is specified as follows: We set three self- attention layers with 24 multi-head attentions, 768 hidden dimension, and 2 2048 feed-forward dimension per layer. For the projection layers for contrastive learning, we set two linear layers with hidden and output dimensions of 2 048 and 768, respectively. The resulting model contains over 25 million trainable parameters.
3.3 Learning with the backbone: fine-tuned, or frozen, combined with another model
Given downstream tables', the knowledge backbone of TARTE facilitates learning through various post- training paradigns: fine-tuning, reused as a frozen featurizers, combined with other models.
Fine-tuning specific task To fine-tune TARTE, we replace the projection layers for contrastive learning with three layers of p(:) = Linear(neelll (BeLU(LayerrNorr(()) that focuses on task-specific settings. The model is trained end-to-end with parameters of the transformer layers kept. frozen. Additionally, we adopt bagging (Breiman, 1996), which has been shown to be useful in several works for neural networks (e.g., Kim et al., 2024; Holzmüller et al., 2024). For this, we train multiple models on different train vrainaialiattion splits used for early-stopping, and average the outputs from each model to form predictions.
TARTE as a table featurizer with frozen backbone Similar to the sentence-traansfrmmess (Reimers & Gurevych, 2019) on LLMs, TARTE can be used to generate meaningful embeddings for table entries. The preprocessed input of a downstream table is passed through the frozen backbone of TARTE, and the embedded representation of the readout element T (similar to the [CLS] token in LLMs) can be used with any machine learning model as a pipeline to make predictions on unseen data.
Boosting a complementary model As TARTE is pre-trained from large knowledge bascs, the em- beddings of the TARTE featurizer potentially hold implicit background information. Such prior knowledge can be useful, especially in few-shot settings; but when the original table provides sufficient information for learning, the background information, on its own, becomes less useful for inference (see Figure 3). Yet, we argue that the implicit prior information continues to be useful when combined appropriately with a com- plementary tabular model. To accompany the prior to tabular models, we formulate a boosting strategy:
'Dwnssreamm tables are preprocessed as for pre-training (subscction 3.2): Fast string embedding, power transformation numbers, datetime columns. We also experimented with various stacking approaches, but they did not bring the marked benefits of boosting.
6

---

Published in Transactions on Machine Learning Research (08/2025)
the base tabular model with the original table is ensembled with model that fits the (train) residuals of the base model with TARTE embedded features. For efficiency, we use TARTE with a Ridge regression.
Specializing to domain A paramount aspect of foundation models is to enable repurposing and spe- cializing to domain. For table foundation models, the requirement goes alongside with cross-table modeling van Breugel & van der Schaar (2024), to be able to draw from different tables. Finc-tuning across different tables within the same domain can work, but with a hefty cost (Kim et al., 2024). Rather, a specialized model that easy to re-adjust would be ideal, e.g., to compensate data drifts.
Here, the ability of TARTE as table fcaturizer comes in handy. Avoiding a costly joint-learning (e.g., Kim et al., 2024), TARTE can readily extract embeddings from fine-tuned models of related tables. The domain specialized representations can then be incorporated with the boosting strategy: the residuals are sequentially fitted with domain specialized representations. Boosting reuses the implicit information from tables within the same domain, embarking domain specialized predictions for downstream tables. For multiple source tables, the same process is repeated for each source table, a multi-step boosting.
3.4 Differences to the CARTE approach
While TARTE draws from CARTE (Kim et al., 2024), we will see that its representations capture much more knowledge from pre-training, providing value without fine-tuning, Multiple differences improve pre-training,
First, TARTE avoids the graph structure used in CARTE. This graph structure, used as a data representation across tables, creates a bottleneck that limits the flow of information between inputs in the self-attention layers (Alon & Yahav, 2021). For example, representing the entry in Figure 1 with the graph structure of Kim et al. (2024) masks the attention between 'Country' U.S. and 'Language' English. Indeed, in CARTE, these inputs are connected only via the center node, row-summary token. On contrary, TARTE keeps all such relations in single attention mechanism, providing context to capture data semantics.
Second, TARTE is trained as a rather shallow architecture of three layers, reflecting that the input for the transformer works in a column-level setting. For both CARTE and TARTE, the columns work similarly as the context- context-window for LLMs, in which the row-dimension of the imput is determined by the number of columns. In many cases, the number of columns in tables are relatively limited, and thus the models are more prone to the problem of oversmoothing (Chen et al., 2020a; Nguyen et al., 2023) with deeper architectures.
Third, TARTE reflects better downstream tasks by handling different data types and better pre-training data sources (see subsection 3.2 and appendix B.1), carefully preprocessing the batch samples, such as trimming to match the number of columns and limiting redundant relations. In addition, TARTE carcfully controls the training procedures (see appendix A.4).
4 Empirical study: TARTE improves prediction and speed, and is reusable
4.1 Experimental set up: tabular learning
Datasets and methods We use the benchmark from Kim et al. (2024): 40 regression and 11 classifica- tion d These tables come with informative columns and discrete entries. For comparing mothods, evaluate post-training paradigms of TARTE (subsection 3.3) and the best performers in related bench- marks: the leading gradient-boosted tree models, XGBoost (XGB, Chen & Guestrin, 2016) and CatBoost (Prokhorenkova et al., 2018); neural-network model, RealMLP (Holzmüller et al., 2024); pre-trained mod- els, TabPFNv2 (Hollmann et al., 2025) and CARTE (Kim el al., 2024) with fine-tuning and boosting (CARTE-B-); simple linear model Ridge from scikit-learn (Pedregosa et al., 2011).
In terms of data preparation, most models provide native handling of diverse data types. However, for models that explicitly require numerical tables, we rely on heuristics provided by the TableVectorizer (Tab functionality of the skrub software. While TabPFNv2 handles readily the input tables, we optionally combine it with the TableVectorizer, for better handling of high-cardinality string and datetime columns.
3 Datasets available
wttp://accecccataattttteeaaaaeea co/ http///aattteeeeeeaeeee /irriiiiiiaaa soda/ carte- benchmark
7

---

Published in Transactions on Machine Learning Research (08/2025)
Regression
Classification
1
TabVec: skrub's Tablevectorizer
TARTE-B: TARTE-BoOst
XGB: XGBoost
Normalized score
0.8
Normalized score
0.8
FT: Finetuned
0.6
0.6
Models (ordered by
Models (ordered n=1024)
TARTE-B-
TabVec:
TARTE-B
TabVec-
TabPFNV2
RealMLP
TabPFNv2
RealMLP
TARTE-FT
TARTE
TARTE FT
TARTE Ridge
CARTE-B-
TabPFNv2
CARTE-FT
TabVec-Ridge
0.4
TabPFNv2
Ridge
TARTE B B-XGB
CatBoost
CARTE-FT
TARTE-XOE XGB
TabVec-
CatBoast
TahPFNv2
TARTE
TabPFNv2
Tabvec-Ridge
TabVec.
TabPFNv2
TabPFNv2
TabPFNv2
0.2
TabVec-XGB
0.2
TGbVec-XGB
32
64
128
256
512
1024
32
64
128
256
512
1024
Number of training samples
Number of training samples
Critical difference diagram
= 32
Critical difference diagram
n 1024
10
CatBoost
TARDE-B-Ta
TabPFNv2
TabVec XGB
TARTE-FT
TabVec-Ridge Ridge
TARTE-FT
TabVec-Ridge
TARTE-B-XGB
CatB oost
CARTE-B-Tane B-bb--B-T
TARTE-XGB
SabVec-RealMLL
TARTE XGB
CARTE-FT
TabPFNv2
TabVec-TabPFNv2
TARTE-Ridge
TabVec-TabPFNV2
TARPE-TabPFNV2
CARTE-B-Taie B-TabPFNv2
TARTE-TabPFNv2
CARTE-FT
TARTE-Ridge
TabVec-ReaIMLP
TabVec-XGo
Prediction/ computation Pareto diagram
Prediction/computation Pareto diagram
TARTE-B-TabPFNV2
0.8
TARTE- B-bbb-22
CARTE-B-
Normalized score
Normalized score
TabPFNv2
CARTE-FT
TARTE-FT
TabVec- TabPFNv2
TARTE
TARTE-FT
0.8
TabVec-
XGBLA
TabVec
TabPFNv2
TARTE-B-GGR
RealMLP
0.6
TARTE
CARTE-B- TabPFNv2
TARTE-
TabVec-
XGB
Rea MLP
TabVec-
Ridge
0.6
Ridge
TARTE-TabPFNVV
CARTE-FT
TARTE-
TabPFNv2
TARTE XGB
TabPFNv2
TARTE-XGB
0.4
CatBoost
0.4
TabVec- Ridge
TabVec-XGB
Ridge
CatBoost
TabPFNv2
3.6s
36s
6mn
3.65
365
6mn
1hr
Total runtime (log-scale)
Total runtime (log-scale)
Figure 3: TARTE performs best for learning on small tables Top: Learning curve for normalized prediction scores for regression and classification. In general, pre-trained models perform better, with variants of TARTE surpassing all baseline models. Middle: Critical difference diagram of average rank at = 32 and 024. Bottom: Pareto diagrams normalized prediction scores with respect to total runtime (log-scale). Efficient base models bring runtime benefits, and TARTE brings additional performance gains.
For TARTE, we abbreviate the variants as follows:
TARTE- FT:
Fine-tuning TARTE downstream tables.
TARTE- TARTE as table fcaturizer. We consider Ridge, XGB, and TabPFNy2 as prediction models. TARTE- B Boosting scheme with TARTE embedded features. We consider stale-of-the-art tabular models, TabPFNv2 and XGB, as base models where TableVectorizer used for data preparation.
Specifie details experiment settings (e.g., hyperparameter selection) are presented in appendix B.1.
8

---

Published in Transactions on Machine Learning Research (08/2025)
TARTE-B-
TARTE-FT
TabPFNv2
TARTE-B-
Normalized score
CARTE-B-
XGB
TabPFNv2
CARTE-
0.8
TabPFNv2
TabVec-
Tabvec- XGB
ReaIMLP
Tabvec
12
10
8
6
2
TARTE-Ridge
CatBoost
TabVec-Ridge
TARTE-FT
TabPFNv2
CARTE-FT
TARPE-Ta
TARTE-Ridge
BARTE-B-TabPFINE
0.6
TARTE-TabPFNv2
TARTE-B-XGB
CatBoost
TabVec- MroVec-RealMLP BARTE-B-TabPFNY2
TabPFNv2
TabVec-TabPFNv2
0.4
TabVec-Ridge
TabVec-XGB
1hr Total runtime (log-scale)
36s
6mn
Results for 10000 Left: Pareto diagram Right: critical difference diagram of average rank Fine-tuned TARTE and TARTE boosting continues to surpass the basclines, but fine-tuning TARTE brings more benefits than on smaller data.
Figure 4:
4.2 TARTE boosts learning, and can be computationally efficient
On small tables: few-shot learning Figure 3 (top) shows learning curves of normalized prediction scores function of sample sizes. The scores are normalized across all train sizes per dataset, with 1 as the best and 0 as the worst performing model. Pre-trained models (TARTE, CARTE, and TabPFNv2) generally perform better, with variants of TARTE as the best performing models, regardless of the sample sizes. Critical difference diagrams" (Figure 3, middle) show that the benefits brought by TARTE are significant.
Considering computation-tinn time and TabPFNv2 form the Pareto frontier. When compute time is important, models that avoid fine-tuning Tabvec-TabPFNv22 gain advantage, especially with larger (e.g., 1 1024).
time trade-offs, Pareto diagrams (Figure 3, bottom) show that variants of TARTE
(as
In general, boosting a base model with pre-trained embeddings (TARTE-B B and CARTE-BI B) improves pre- diction. However, as TARTE is better pre-trained (see subsection 4.3), its embeddings complement better the base models, bringing prior information acquired from knowledge pre-training. The best performances require a strong base tabular model, as with TabPFNv2 and XGB. On the oer hand, TARTE FT does not have such requirements, but comes with computation burdens, with extensive hyperparameter tuning". Finally, TARTE fcaturizer is good table preparator: compared to TabVec (dashed lines), it markedly increase performance, bringing, for instance, ridge to good position.
On larger tables, TARTE helps both for prediction and scalability Figure 4 summarizes prediction accuracy and runtime costs at n = 10000. Similar to small tables, pre-training schemes are the Parelo frontier, with TARTE-E and TARTE B outperforming baselines. TARTE-B B continues to blend well with base models, TabPFNv2 and XGB. Fino-tuning TARTE brings more prediction benefits, but a heavy cost with a 14-fold runtime increase compared to TARTE B TabPFNv2. TARTE B TabPFNv2 gives both performance and scalability: compared to TabVec-TabPPNS the prior from knowledge pre-training improves performance at small compute cost mm).
On more numerical tables, comparing to TabLLM We evaluate TARTE-FF and TARTE- B over nine datascts presented in TabLLM (Hegselmann et al., 2023). The datasets contain larger fraction of numerical columns with lower cardinality in categorical columns (Kim et al., 2024). Figure shows methods comparison, as a critical difference diagram. Tn addition to four baselines from Hegselmann et. al. (2023)
*Critical difference diagrams display average rank across models with crossbars depicting the two having statistically significant difference based on Conover post hoc after Friedman test. pairwise significance (Conover, 1999). 6 Due the computation costs, the search space for FARTE-FT limited (see Table 3)

---

Published in Transactions on Machine Learning Research (08/2025)
Figure 5: Comparison baselines on more numerical tables. For datasets with higher fraction of mumerical columns (from Hegselmann et al. (2023)), TARTE-B contin- ues to help base models for prediction.
7
6
5
4
3
2
TabLLM
TARTE-B-TabPFNE TabVec-TabPFNr2
XGB
CARTE-FT
TabPFNv2
TARTE-BB B--GB
TARTE-FT
7
4 3
CARTE YAGO3
TARTE
Enriched YAG04.5
CARTE Enriched YAGO4.5
TARTE
Remove datetime Enriched YAG04.5
TARTE MMiHHash
Random weights
TARTE
Remove column names Enriched YAG04.5
CARTE Random weights
TARTE
YAG03
CARTE New sampling
Enriched YAG04.5
TARTE Random weights
Figure 6: Ablating architecture, pre-training, and preprocessing components. ridge is fitted with embeddings from different schemes: random weights (no pre-training); replacement FastTexx with skrub's "MinHash" encoder; TARTE and CARTE pre-trained with YAGO3 and Euriched YAGO4.5; CARTE with TARTE pre-training schemes (New sampling); TARTE without datetime detection or column information.
and Kim et al. (2024) (in dashed lines), include two TabPFNv2 variants considered in this study, While TabPFN tends to performs better, TARTE B can help the base models, even on more numerical tables.
4.3 Good knowledge pre-training, adapted to downstream tasks, is important
Better knowledge pre-training, better performance TARTE embeddedings boost prediction. But what drives this boost? Ts it the inductive bias of the architecture, or is the pre-training on the knowledge base actually important? To answer this question, we investigate TARTE pre-training on small tables (n = 32 to 1 024), where the effect of TARTE is most eminent. Figure 6 presents the critical difference diagram for Ridge fitted with embeddings from different schemes. Here, CARTE with "New sampling" corresponds CARTE architecture with TARTE pre-training schemes, and "MinHash" replaces FastText by MinHash encoding, that only captures morphological similarities of strings (Cerda & Varoquaux, 2020).
First, comparing MinHash to TARTE reveals the importance of Fast'Text. A simple use of FastText with random (non pre-trained) weights already exhibit relatively strong performances. As Fest captures semantic similarities, its combination with a suitable transformer architecture (subsection 3.1 or CARTE) forms an inductive bias of smoothness: similar tables have similar representations.
Going further, proper combination of pre-train data and procedures, the architecture, and preprocessing of tables improves knowledge pre-training (TARTE with Enriched YAG04.5). For CARTE representations used without fine-tuning, pre-trained representations perform less well compared to random weights, regardless of the pre-train data: these representations lack out-of-the-box re-usability. TARTE pre-training schemes CARTE (New sampling) helps but the performance does not match that of TARTE, highlighting the various sources of improvements through TARTE (see subsection 3.2). In addition, without datetime detection or column information, properly pre-trained weights still provide competitive represontations. This suggests that TARTE can perform well even on tables without meaningful column names.
Which downstrcam tables benefit from TARTE? Tables are very heterogencous, and not all may correspond to learning lasks that benefit. from the knowledge embarked in TARTE. We explore which char- acteristics of a downstream table leads to a performance boost with TARTE. Table 1 provides multivariate analysis of various factors (datasct meta-features) that affect the prediction on downstream tables (lincar model explaining the performance boost of TARTE on top of Tab Vec-Rige).
We find that TARTE works less well on tables with long string entries (likely a limitation of FastText embedings, as discussed appendix A.2). However, brings more benefits for tables with many strings considered as inliers the pre-training source (using a One-Class SVM fitted on the embeddings of the pre-training strings define inlier score). TARTE also works better with fewer low-cardinality columns
10

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Factors success for TARTE. Coefficients and their confidence intervals of a linear model explain- ing the improvement of TARTE Ridge over TabVec Ridge across datascts (R-squared: 0.22). Negative values mean that greater values for the corresponding meta-feature are associated to less benefits of TARTE. Performance decrease for long strings suggest a limitation of the language model, FasText. Importance inlier probability shows the need for rritig to cover well downst.ream lerminology.
Dataset feature
Coef.
CI(0.025, 0.975]
Avg. string length
-0.35
[-0.40, -0.31
Avg. inlier prob.
0.29
[0.26, 0.33
# datetime cols Avg. string sim.
0.28
[0.24, 0.31
0.18
[0.13, 0.22
# low card. cols
-0.18
-0..23, -0.14]
# numerical cols
0.13
[0.09, 0.16)
# high card. cols
0.06
[0.02, 0.10
log (train-size)
0.04
[0.00, 0.08)
TARTE-B- TARTE- B-TabPFNV2
TabPFNv2
(DS)
TabVec-
(Single)
FT
12
10
8
Normalized score
0.7
TabPFNV2 (Single)
T-RTE-B- XGR (SingleT
(MT)
TabVec-XGB(MT)
TARTE-BB-TbbPFNN2(0S
TARTE- B-BBB
(DS)
CatBoost(Single)
BURTEEBBTTbDPFNN2((Sigggg)
CARTE FS
CatBoost(MT)
CARTE FT(MT)
TARTE- Ridge (Single)
TARTE-Ridge Ridge
TabVec-XGB(Single
SabVee--TbbFNv2(SIin)
0.6
(DS)
TabVec-TabPFNv2(MT)
TARTE-B-XGB(DS)
Tabvec- TNb2 (MT)
TARTE-Ridge(DS)
TARTE-B-XGO XXXBBGiggl)
(SSe--))dge(Single)
CARTE-FT(C (ARTE-FT(Single)
Tab (Single)
0.5
CatBoost(MM
DS: Trained with Domain-Specialized features MT: Trained on Multi-Tables
CatBoost (Single)
VGGVM)
36s
6mn
1hr
Single: Trained on Single table
Total runtime (log-scale)
Figure 7: Domain specialization from a single source - Left: Pareto diagram Right: criti- cal difference diagram of average rank. 'DS' and 'MT" denote Domain-Specialized and Multi-Tables schemes, respectively. Models that blends inference from the target table with representations tuned on the source improve (TARTE B and CARTE). TARTE B TabPFNv2 gives the best predictions.
(trees shine on these), with many datetime columns, with strings similar across rows, with more numerical or high-cardinality columns, and with more data. Overall, we find that TARTE brings benefits on more complex tables, with many complex strings, but particularly so these look like strings seen during pre-training.
4.4 Specializing to domain: fine-tuned TARTE gives representations that transfer
Experimental set up and baselines We investigate whether reusing TARTE representations from fine- tuned models facilitates subsequent learning. We evaluate TARTE Ridge and TARTE B with TabPFNv2 and XGB. Here, TARTE-Ridge denotes Ridge model fitted on fine-tuned TARTE embeddings and TARTE- B sequentially fits the residuals with domain specialized representations (see subsection 3.3). As baselines, we consider top-performing tabular models that provide native handling of missing values. Indeed, even with differing columns across tables, these models can be fitted with stacked table. We also include CARTE multi-table (CARTE-MT), the only baseline, to our knowledge, that learns across tables without correspondences (Kim et al., 2024). Note that the setup for TARTE and bascline models is slightly different. Baselines fit all tables, sources and target, jointly; while the TARTE model does not need access to the source tables to be reused for transfer. We consider each domain one after the other, and for each, we consider every combination of source tables and a target table: source tables to specialize the model, and target table evaluate the corresponding learner. As datasets, we select from the previous set tables group of tables within the same domain, but acquired in different settings (Kim et al., 2024). See appendix B.3 for details.
TARTE improves with domain specialized representations Figure presents the improvements and runtime costs for single source table. Here, the abbreviations 'DS' and 'MT' denote Domain-Specialiver and
11

---

Published in Transactions on Machine Learning Research (08/2025)
Improvement over BARTE-B-TabPFNV2 (Single)
Figure 8: Domain specialization from multiple sources: comparison between TARTE and CARTE. Improvements from the best performing model without any source information, TARTE- B TabPFNv2, with re-
0.02
Train size
0.02
32
0.01
Includes
64
train time
128
sources (20mn)
256
0.01
2 21mn
m1mm 30s
spect to the total runtime across different train sizes. While both TARTE and CARTE improve with multiple sources, TARTE is far more efficient. In contrast to CARTE,
Models
TARTE-B-
TabPFNv2 (DS)
0.00
CARTE-FT
(MT)
TARTE can reuse the fitting on the source tables (here, 20 mn on average).
24mn 38mn 1hr
Total runtime (log scale)
Multi-Tables, respectively. Out of top-performing tabular models, those that actually benefit are TARTE- B, CARTE, and CatBoost. Here, most of the runtime of TARTE is driven by the training time of the source model. The cost can be amortized we are given a new target table: TARTE can readily reuse the domain-specialized models, which gives an advantage over baselines that requires a new fitting of the model.
TARTE stays efficient with multiple sources We evaluate every possible source-targe combinations with TARTE-B- (abbFFN22())) and CARTE-MT. Figure 8 gives the improvements of respective models compared to the best performing model single tables (TARTE B TabPFNv2), with respect to the runtime. Both models benefit from source tables, with larger gains for TARTE- B TabPFNy2(DS). Compared to Kim et al. (2024), the baseline here is much more performant, and thus it more difficult to improve. Concerning the runtime; CARTE-MT requires pairwise fitting for each source, markedly increasing compute costs even for small increase train-size. TARTE, however, is more efficient: the runtime of TARTE dominated by the training time of source tables (20 mn). Therefore, reusing domain-specialized models imposes far less compute cost, putting TARTE in better position for transfer within the specific domain.
5 Discussion and conclusion: A knowledge backbone that can be reused
Foundation models have changed the landscape of machine learning because they facilitate huge variety of applications and can efficiently lead to specialized models. Table Foundation Models are progressing rapidly, one line of work improving table-level tasks such as table understanding and question answering (Zhang et al., 2024; Li et al., 2024), another geared toward tabular learning, i.e., row-level predictive analytics (Hollmann et al., 2025; Kim el al., 2024). In the latter, the reamstreem lask comes with strong supervision. Here, developing foundation models requires extra work to fulfill the promises of pre-training and use
Knowledge pre-training that helps tabular learning Our study shows the value of pre-training to acquire world knowledge for tabular learning (Figure 6). Current table understanding models clearly leverage world knowledge, but it is less visible in tabular learning ones. Table understanding models are typically variants of LLMs, generative, pre-trained an large textual corpora, and often used via prompting and thus in context. learning. For tabular learning, however, post-training for the downstream tasks is typically further away from the next-token prediction used to injoct world knowledge LLMs. Indood, for tables with many rows, a good tabular learner often needs to aggregate information across these rows, opposed to retrieval of a few rows which suffices for typical sable-understanding tasks. But language models do not leverage long contexts uniformly (Liu et al., 2024). By matching pre-training tasks to downstream tasks, the TabPFN literature has managed to train suitable in-context models, valuable for tabular learning. However, this has required synthetic datasets that do not bring world knowledge.
Powerful post-training used for tabular learning can compensate for suboptimal pre-training. In fact, models without pre-training (such as XGBoost) are very strong baselines. Also, our results (Figure 6) show that CARTE, pretrained model for tabular learning, achieved its good performance thanks to post-training,
12

---

Published in Transactions on Machine Learning Research (08/2025)
namely fine-tuning. A drawback of requiring sophisticated post-training is the compute cost. Investing in pre-training, as with TARTE, enables to use simple downstream learners, Ridge, which becomes a very strong baseline with TARTE (On small, but also mid-sized data: figures 3 and 4). Pre-training cost is then "amortized" by making downstream learning and inference cheap.
A re-usable backbone improves predictions and decreases costs TARTE's backbone is suited to many post-training approaches. Well pre-trained, its transformer- architecture gives a good encoder for complex tables that can be easily re-used and specialized as a backbone. Tt can be plugged into any learner, possibly cached to facilitate operations. Fine-tuning domain gives best prediction. But different choices of post-training (using TARTE only data preparation for subsequent learning, boosting, or fine-tuning) explore different trade-offs in prediction performance versus computational costs. Using TARTE improves the "Parcto optimality": more prediction performance for the same computational cost. This approach is scalable and beneficial even for larger tables. Finally, while strings (in column names and cell entries) are central to TARTE, the encoder also leads to state-of-the-art performance on more numerical tables.
A popular aspect of foundation models is their ability be specializod, as visible from the number of fine- luned models on hugging face (e.g., 1 566 derived from Roberta, as of May 2025, hugginglace). A fine-tuned TARTE can be readily reused for multiple applications in the given domain, improving downstream prediction and computational performance. Such fine-tuning followed by an independent reuse is to be contrasted with most prior successes of transfer in tabular learning, built on joint lcarning and thus increasing operational costs rather than decreasing them. On the contrary, reusing a domain-speciaalzzed version of TARTE uses less computational resources than starting from scratch or fitting a cross-table model.
The agenda of table foundation models calls for repurposable and reusable models that are as powerful and easy to apply to new tables possible. Our study makes step is this direction, showing how to pretrain to capture world knowledge and reuse it for tabular learning.
Acknowledgements
The authors acknowledge the support in part by the French Agence Nationale de la Recherche under Grant ANR-20-CHIA-ONE (Learnl). We also would like thank the anonymous reviewers for constructive com- ments and valuable suggestions, which have significantly improved the quality of the manuscript.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint xrxi:22303.0877 2023.
Uri Alon and Eran Yahav. On the bottlencek of graph neural networks and its practical implications. In International Conference on Learning Representations, 2021. URL http::://pperreeeee...o http:///ppmmereeeeeeeeeertrreeee fet/oooum?idd 1800Ph0CVH2.
Lucas Bernardi, Themistoklis Mavridis, and Pablo Fstevez. 150 successful machine learning models: 6 lessons learned at booking, com. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery data mining, pp. 1743-1751, 2019.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108. 07258, 2021.
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Hang, Martin Pawelczyk, and Gjergji Kasnoci. Deep neural networks and tabular data: survey. IEEE transactions on neural networks and learning systems, 2022.
Leo Breiman. Bagging predictors. Machine learning, 24:123-144 -14--,100111 1996.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Patricio Cerda and Gaäl Varoquaux. Encoding high-cardinality string categorical variables. IEEE Transac- lions on Knowledge and Dala Engineering, 34(3):1164-1177 2020.
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoot problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 8438-3445 2020a.
Jintai Chen, Zhen Lin, Qiyuan Chen, and Jimeng Sun. Cross-table pretraining towards a universal function space for heterogeneous tabular data. arXiv preprint 2XXi:22406000281, 2024.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd aom sigkdd international conference on knowledge discovery and data mining, pp. 785-7944 2016.
Ting Chen, Simon Korublith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive Icarning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
William Jay Conover. Practical nonparumetric slatistics, volume 350. john wiley & sons, 1999.
Alexis Cvetkov-Iliev, Alexandre Allauzen, and Gaël Varoquaux. Relational data embeddings for feature enrichment with background information. Machine Learning, 112(2):687-720, 2023.
Felix den Breejen and Se- Se-Young Yun. Attic: A new architecture for tabular in-context learning transformers. 2024.
Felix den Breejen, Sangmin Bac, Stephen Cha, and Se- So-Youg Yun. Finc-tuned in-context learning trans- formers are excellent tabular data classifiers. arXiv preprint arXiv:2405. 133966 2024.
Jacob Devlin, Ming- Wei Chang, Lee Kenton, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NaacL-HLI, 2019.
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dim- itris Papailioponlos, and Kangwook Lec. Lift: Language-interfaeed fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 35:11763 11784, 2022.
Benjamin Feuer, Robin Tibor Schirrmeister, Valcriia Cherepanova, Chinmay Hogde, Frank Hutter, Micah Goldblum, Niv Cohen, and Colin White. Tunetables: Context optimization for scalable prior-data fitted networks. 0rXiv:2402.11133 2024.
Joshua P Gardner, Juan Carlos Perdomo, and Ludwig Schmidt. Large scale transfer learning for tabular data via language modeling. In NeurIPS, 2024.
Léo Grinsztajn, Edouard Oyallon, and Gaäl Varoquaux. Why do tree-based models still outperform deep learning on typical labular data? Advances in neural information processing systems, 35:507-555 520, 2022.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement lcarning. arXiv preprint arXiv:2501. 12948, 2025.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag, Tablim: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pp. 5549 5581. PMLR, 2023.
Jonathan Herzig, Thomas Mueller, Syrine Krichene, and Julian Fisenschlos. Open domain question answering over tables via dense retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 512-519, 2021.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. Tn ICLR, 2023.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirmmesserr, and Frank Hutter. Accurate predictions on small data with a tabular founda- tion model. Nature, 637(8045):319 326, 2025.
David Holamüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned mips and boosted troes on tabular data. arXiv preprint arXie:2407.04441, 2024.
huggingface. List of fine-tuned models from facebookai/roberttee base. http:://aaggingffcc..o//oodllll other=base_monee _aseeeemmmll:___ : finetunnee Taa.ee/aaaaaao///orrrrraaaaaaeaa
Myung Jun Kim, Leo Grinsztajn, and Gael Varoquaux. Carte: Pretraining and transfer for tabular learning. In ICML, 2024.
Mykhailo Koshil, Thomas Nagler, Matthias Feurer, and Katharina Eggensperger. Towards localization via data embedding for tabpfn. In NeurIPS 2024 Third Table Representation Learning Workshop, 2024.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateck Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233-30223 2022.
Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, and Micah Goldblum. Transfer learning with deep tabular models. In The Fleventh International Conference on Learning Representations, 2023.
Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. Table-gpt: Table fine-tuned gpt for diverse table tasks. Proceedings of the ACM on Management of Data, 2(3):1-28, 2024.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2024.
Si-Yang Liu and Han-Jia Ye. Tabpfn unleashed: A scalable and effective solution to tabular classification problems. Xixiv preprint ur xrx:22502.000 02227, 2025.
Farzanch Mahdisoltani, Joanna Biega, and Fabian M Suchanck. Yago3: A knowledge base from multilingual wikipedias. In CIDR, 2013.
Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems, 36, 2024.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. Advances in pre-training distributed word representations. arXiv preprint drXiv::718.09405, 2017.
Tam Nguyen, Tan Nguyen, and Richard Baraniuk. Mitigating omoothhn in transformers via regularized nonlocal functionals. Advances in Neural Information Processing Systems, 36:80233 80256, 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807 807.018877.09748, 2018.
Andrei Paleyes, Raoul-Gabriel Urma, and Neil D Lawrence. Challenges in deploying machine learning: a survey of case studies. ACM computing surveys, 55(6):1 29, 2022.
Fabian Pedregosa, Gaäl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-loarn: Machine lcarning in python. the Journal of machine Learning research, 12:2825 2830, 2011.
Liudmila Prokhorenkova, Gleb Gusev, Alcksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Jingang Qu, David Holzmüller, Gaäl Varoquaux, and Marine Le Morvan. TabicI: A tabular foundation model for in-context learning on large data. arXiv preprinl arXi::2502. 05504, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748 8763, PMLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 11 2019. URL https:///axxii orgg/bs//////.... 10084.
Derek Saul. Biggest market loss in history: Nvidia stock sheds nearly $600 billion as DeepSeek shakes AI darling, Jan 2025. URL shw.tocc--seed---eeereeeee hatpa/aswwyaataaatttoteadaaaaaa-
Ravid Shwartz-Ziv Ziv and Amitai Armon. Tabular data: Doep learning is not all you need. Information Fusion, 81:84 90, 2022.
Marco Spinaci, Marek Polewezyk, Johannes Hoffart, Markus C Kohler, Sam Thelin, and Tassilo Klein. Portal: Scalable tabular foundation models via content-specific lokenization. In NearIPS 2024 Third Table Representation Learning Workshop, 2024.
Fabian M Suchanek, Mehwish Alam, Thomas Bonald, Lihu Chen, Pierre-Henri Paris, and Jules Soria. Yago 4.5: A large and clean knowledge base with a rich taxonomy. In Proceedingg of the 47th International ACM SIGIR Conference on Research and Development Information Retrieval, pp. 131 140, 2024.
Valentin Thomas, Junwei Ma, Rasa Hosseinzadeh, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony Caterini. Retrieval &amp; fine-tuning for in-conlext tabular models. In Advances in Newral Information Processing Systems, 2024.
Boris van Breugel and Mihaela van der Schaar. Position: Why tabular foundation models should be a research priority. In ICML, 2024.
Gaël Varoquaux, Alexandra Sasha Luccioni, and Meredith Whittaker. Hype, sustainability, and the price of the bigger-is-better paradigm in ai. arXiv preprint arXiv:2409. 14160, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78-88 85,88-852222 2014.
Ruiyu Wang, Zifeng Wang, and Jimeng Sun. Unipredict: Large language models are universal tabular predictors. arXiu preprint arXio:2310. 032666, 2023.
Zifeng Wang and Jimeng Sun. Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35:2902-201 2915, 2022.
Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great tabular prediction. In The Twelfth International Conference on Learning Representations, 2024.
Yazheng Yang, Yuqi Wang, Guang Liu, Ledell Wu, and Qi Liu. Unitabe: A universal pretraining protocol for tabular foundation model in data science. In The Twelfth International Conference Learning Represenlations, 2024.
Han-Jia Ye, Si- Yang Liu, Hao-Run Cai, Qi-Le Zhou, and De- De-Chan Zhan. A closer look deep learning on tabular data. Xixiv preprint or 007:24007000566 2024.
16

---

Published in Transactions on Machine Learning Research (08/2025)
In-Kwon Yeo and Richard A Johnson. A new family of power transformations to improve normality symmetry. Biometrika, 87(4):954-00) 959, 2000.
Andrew Zhai, Hao- Yu Wu, Eric Tzeng, Dong Huk Park, and Charles Rosenberg. Learning unified embed- ding for visual search at pinterest. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 2412-2420, 2019.
Tianshu Zhang, Xiang Yue, Yifei Li, and Huan Sun. Tablellama: Towards open large generalist models for tables. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Compulational Linguislies: Human Language Technologies (Volume 1: Long Pupers), pp. 6024-6044) 2024.
Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: cross-table pretraining for tabular transformers. In Proceedings of the 40th International Conference on Machine Learning, pp. 43181 43204, 2023.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Table Comparison pre-train datasets for TARTE and CARTE
Data
Used in # Entities
# Relations
# Facts
Total: 65
Total: 18 108 790
YAGO3
CARTE
0227 996
Categorical: 38
Categorical: 14 826 655
Numerical: 27
Numerical: 3 282 135
Total: 97
Total: 23 457263 263
YAGO4.5
5 5455 689
Categorical: 72
Categorical: 16 296 953
Numerical: 25
Numerical: 7 7160 310
Total: 687
Total: 30 262 472
YAGO4.5 Enriched
TARTE
5776 475
Categorical:
Categorical: 16 296 953
Numerical: 615
Numerical: 13 9655519
A Further details on backbone and pre-training
A.1 Modeling with column - cell pairs: context-aware for transformers
Given a table, TARTE models with a set of column cell pairs and combine the embeddings to form the input of the transformer. The column information is crucial to supplement context for the transformers (Kim et al., 2024). For instance, the entry (e.g., 'A.I') the content of the table (e.g., 'movics") would be difficult to undersland without the column information (see Figure 1). For knowledge bases of the pretrain data, this corresponds to modeling relations, which has been pivotal to knowledge embedding models (Cvetkov-Iliev et al., 2023). Moreover, the column embeddings serve as medium to combine different data types for a uniform processing of entries.
A.2 Language model used in TARTE
We use Fast embeddings (Mikolov et al., 2017), which hold some limitations for long string entries (subsection 4.3). Yet, string (text) entries in tables mostly contain only a few words: across 51 datasets, the median ratio of unique entries with more than 10 words 0.025. An interesting alley to explore more sophisticated language models, possibly using different language models depending on different structure of strings (e.g., simple words or sentences).
A.3 Pre-train data
Table gives the statistics of the pre-trained datasets used by TARTE and CARTE. They both describe approximately the same number of entities those that have a Wikipedia page (Suchanek et al., 2024). How- ever, the pre-train dataset for TARTE is enriched and diversified with more numerical information: YAGO4.5 Enriched, used in TARTE, describes almost twice many numerical facts and, most importantly, includes far more diverse propertics (615 distinct relation types against 27 for YAGO3).
A.4 Pre-training procedures
The batch size is set 512 in which 256 entities randomly selected with additional positive for each. The lotal number of steps for training is 200 000, with the AdamW optimizer and the cosine scheduler. The learning rates were set as Irmin = 10-5 Immaz = 10 of warm-up over the first 2 000 stops, followed by a linear decay in learning rate schedule. The probability for all dropout layers was set as 0.1.
Training specifications
To enable contrastive learning, we construct batch samples by generating positive sam- ples that replace parts of information (for instance, replacing Paris with London, see Figure 2). While this may be counterintaitive, it helps the pre-train model to embed representations that is geared for capturing
Batch sampling
18

---

Published in Transactions on Machine Learning Research (08/2025)
smoothness (similar rows having similar target value) in table. When trimming relations have fixed number of related information across entities for batch, we consider the number of unique relations for each entity, For example, well-known city can have many number of relations 'has neighbor', which dominate the facts about the city. This creates the difficulty in resembling the imput data with a table, and thus we trim the relations such that there are not. many duplicate relations for an entity.
Contrastive loss with Matryoshka embeddings For the contrastive learning, embedding scheme (Kusupati ct al., 2022) to provide various pretrained embededdings of reduced dimen- sions. The Matryoshka embeddings attach several linear projection layers of different dimensions in parallel, calculate contrastive loss for each dimension, and aggregate the losses to backpropagate through the whole network. For TARTE pro-training, the dimension set was {64, 128, 256, 512, 768}.
the Matryoshka
B Details on downstream experiments
B.1 Training procedures
Data preparation We use TARTE embeddings of dimension, = 768 except for TabPFNv2, which we set d == 256 since the maximum feature-size that TabPFNv2 can handle is 500. For TableVectorizer from the skrub, categorical columns are differently encoded depending on the cardinality (number of categories): Columns with low cardinality are one-hot encoded while those with high cardinality are encoded using the Camma-Poisson encoder (Cerda & Varoquaux, 2020). For models without native handling missing values, we imputed with the mean for numerical features, and treated as another category for categorical features.
Hyperparameter optimization We perform hyperparameter selection, except for TabPFNv2. For finc- luned models, TARTE FT and CARTE FT, we use custom grid-search hyperparameter optimization function tailored with the bagging from different eain/validation splits. It identifies the optimal hyperpa- rameters based on mean validation loss. For Ridge regression, we RidgeCV from scikit-learn (Pedregosa et al., 2011), which performs an efficient Leave-Oned Caare-OOn-Oree Out Cross-Valee Validation to select the hyperparameters. For rest of the models, we run 5-fold cross-validation over 100 random search iterations. Table 3 shows the hyperparameter spaces for each method. Most of the spaces were adapted from Grinsztaju et al. (2022) and Holzmüller al. (2024), except for the fine-tuned models, which were adapted from Kim et al. (2024).
Additional details For evaluation single tables, the train-size for each tables varied from 32, 64, 128, 256, 512, 1 0244, and 10000; remaining data set the test set. Out of 51 datasets, 27 datasets contain sufficient data points for the train-size of 10000. The performance was mcasured with R² score for regression and the Area Under Receiver Operating Curve (AUROC) for classification tasks. For the runtime, it measures the total time for data preparation, hyperparameter optimization, and prediction. Overall, the results were recorded on 10 different train/test tost splits for each datasct.
B.2 Details on multivariate analysis
The inlier score is obtained by fitting a One-Class SVM on the embeddings of strings present in the pre-train data. Considering the size of the pre-train data, we train SCDOneClassSVM from scikit-learn (Pedregosa et al., 2011), that solves linear One-Class SVM using stochastic gradient descent. In regards to the categor- ical columns, we follow the heuristics from TableVectorizer in skrub package, with the criterion of high cardinality set 40.
B.3 Experimental set-up for domain specialization
We consider the few-shot settings with the train-size on the target table varied from 32, 64, 128, and 256. The splits were set as same as that singletables to enable comparable results. For each baseline, some additional details were considered.
19

---

Published in Transactions on Machine Learning Research (08/2025)
Table 3: Hyperparameter space for models considered in the study.
Methods
Parameters
Grid
Learning rate Batch size
[1, 2.5, 5, 7.5, 10] x le for 10000 else 256
TARTE-FT
Learning rate
[1, 2.5, 5, 7.5, 10] le=4
CARTE FT
Batch size
16 for 10000 else 256
Ridge
Alpha
[le" le-,, let, 1ca]
Num. estimators
1000
Max depth
Uniformint [2, 6]
Learning rate
LogUniform [le
Bagging temperature
Uniform [0, 1]
Iz-leaf regularization
LogUniform [1, 10]
CatBoost
One hot max size Random strength Leaf estimation iterations od wait
UniformInt [0, 25]
UniformInt [1, 20]
UniformInt [1, 20]
300
od lype
'lter'
Num. estimators
1000
Max depth
Uniformint [2, 10]
Learning rate
LogUniform [le
Min child weight
LogUniform [1, 100]
Subsample
Uniform [0.5, 1]
XGBoost
Colsample by level
Uniform [0.5, 1]
Colsample by tree
Uniform [0.5, 1]
Gamma
LogUniforu [lc [1c-8,
Lambda
LogUniform [1, 4]
Alpha
LogUniform [1e [1e-5 100]
Farly stopping rounds
300
Num estimators
Uniformint [50, 250]
Max depth
[None, 2, 3, 1]
Max features
[sqrt, log2, None, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)
GandomForest
Min samples leaf Bootstrap
LogUniform [1.5, 50.5]
[True, False]
Min impurity decrease
[0, 0.01, 0.02, 0.05]
Num layers
UniformInt [1, 4]
Layer size
UniformInt [16, 1024]
Dropout
Uniform [0, 0.5]
MLP
Learning rate
LogUniform [le 1e-
Weight decay
LogUniform [le 1e
Batch size
[16, 32]
TARTE:
The runtime of TARTE models include the training time of source tables. For each source table, we fine-tuned the pre-trained TARTE with the associated task. No hyperparameter tuning was performed, with the batch size and the learning rate set 256 and 5e respectively.
CARTE-MT: We follow Kim et al. (2024) for the set-up. The runtime includes training without source (including hyperparameter optimization) and joint learning of targot-source pairs (Kim al., 2024). Other baselines: For models that stack the target and a source table, the maximum size of the training data (including both the target and the source table) was set as 10000, which is the boundary for TabPFNv2, and larger size would incur overfitting the source (Kim et al., 2024). Moreover, the column names matched through manual inspection for best possible performances.
20

---

Published in Transactions on Machine Learning Research (08/2025)
Datasets For datasets from similar domain, adapt those used in multi-table learning in Kim et al. (2024). These are tables within the same domain, but acquired in different settings (for instance from multiple sources). We have 12 domains: Wine prices (4 tables), Wine ratings (3 tables), Beers (2 tables), Used cars (5 tables), Films (2 tables), Dramas (2 tables), Anime (2 tables), Baby products (2 tables), Bike sales (2 lables), Employee remunerations (3 tables), Restaurant ratings (3 tables), Journal scores (2 tables).
C Hardware specifications
The hardware specifications for pretraining and downstream tasks are as follows. For the pre-training of TARTE, single NVIDIA A40 (48GB) gpu was used. The downstream experiments was run on 32 cores of CPU except for TabPFNv2 variants for all n, and TARTE-FT and CARTE-FF at 10000, in which we used gpus. The hardware chosen based availability.
GPUs: NVIDIA V100 (32GB VRAM), A40 (40GB / 48GB VRAM)
CPUs: AMD EPYC 7742 64-Core Processor, AMD EPYC 7702 64-Core Processor (512GB RAM),
Intel(R) (R) CPU E5-2660 v2, Intel(R) Xeon(R) Gold 6226R CPU (256GB RAM)
D Implementation of TARTE
The implementation of TARTE is available at https: attps://githut
ai.
E Extended results
E.1 Results on small tables
Figure 9 presents the results for all train-sizes in few-shot setting. We include additional baselines of Ran- dom Forest (RF), the classical Multilayer Perceptron (MLP), and TARTE-CatBoost using the default hyperparameters. TARTE, through various post-training schemes, outperforms the baselines regardless of train-sizes. In addition, TARTE featurizer perforrsms TableVectorizer (except for TabPFNv2) for given learner in small training samples (n < 256). For TARTE TabPFNv2, it underperforms likely due to that the dimension of TARTE embeddings was 256 for TabPFNv2 and the resulting data representations do not have a distribution matching to that of synthetic tables from TabPFNv2 pre-training.
E.2 Results on single source domain specialization
Figure shows improvements for using a single source table. Regardless of the train-sizes, TARTE-B provides benefit to the base models. Compared a CARTE MT with the joint learning scheme, TARTE shows the benefit of having a reusable model. Once specialized models available, TARTE can readily benefit without requiring complex refitting of the model, placing TARTE in better position for transfer.
21

---

Published in Transactions on Machine Learning Research (08/2025)
0.8
TARTE-B-TabPFNYZ
Tab
TARTE- Ridge
TARTE-B XGB
TARTE-FT
TabPFNv2
Normalized score
--RTE-BB TabPENv2
TabVec- ReaiMLP
n
= 32
0.6
TabVec
TARTE-RF
CARTE-FT
16
14
12
10
6
TabVec-
TARTE- TabPFNV2
Ridge
TARTE- CatBoost
TARTE- XGB
TabVec- MLPP
TARTE-B BARTE-B-TAbRRE B-TaPFNNv2
TBPFNN22
TabVec-XGB
TARTE-MLE
TARTE-FT
CatBoost
CatBoost
TARTE-B B-XGB
0.4
TabVec- XGB
ReVVec-RealMLP
TabVec-Ride Ridge
TabVec-TabPFNV
TARTE-MLP
TARTE-CatBoOSE CatBoost
TARTE
CARTE-B-TabPER B-TabPFNv2
TabPFNv2
TARTE-Ridge
TARTE- AGB
CARTE-FT
Tab
CARTE-TabPFIN
TabVec-MLP
TARTE- RF
3.6s
36s
6mn
Total runtime (log-scale)
TARTE-FT
B-Tbb---
0.8
TabVec- TabPFNv2
CARTE B-TbbP
CARTE-
Normalized score
TARTE-
Ridge
TARTE-B
TabVec- RealMLP
TARTE- XGB
FT
64
TARTE-
TabPFNV2 TARTE-RF
16
14
12
10
8
6
0.6
TabVec- Ridge
TARTE- CatBoost
TARTE-MLP
TabVec-MLP
TARTE-FT FT
TabPFNv2
TabVec-XGB
TARTE-B BASTE-B-TAbREE TabPFNv2
Tab
TabVec- TabPFNv2
TabVec-RF
CatBoost
CatBoost
BARTE-B-TaPPER
TARTE-Ridhe BRRTE-Ridge
0.4
TARTE CatBoost
TARTE-B B-XBB
TabPFNv2
CARTE-F FT
TabVec-XGB
TARTE-XGB
TARTE- TabPFNv2
TARTE-RF
TabVec-MLP
TabVec-Ridge Ridge
3.65
36s
6mn
Total runtime (log-scale)
SARTE-B-TabPFNV/
TabVec- TabPFNv2
CARTE-B TabPFNv2
TARTE-FT
Normalized score
0.8
TARTE-
TARTE-B-XGB
CARTE-FT
128
Ridge
TARTE- TabPFNv2
RealMLP
TabVec-
TARTE- CalBoost
0.6
Ridge
TARTE-RE TARTE-
TabVec-MLP
TARTE-B- T
TabVec-RF
CatBoost
TabVec-XGB
TARTE- MLP
TARTE-FT
PARTE-B-TabPFNO TabVec-TabPFNV2
CatBoost
TabPFNv2
TARTE-MLP
TabVec-RF
CARTE-FT
TabVec- XGB
TabPFNv2
TARTE-B-XGB
TARTE-RF
TARTE- RidTE-Ridge
0.4
TARTE-XGE
TARTE- TabFFV22
TARTE CatBsttt
Tab
TabVec Ridge
3.6s
365
6mn
Total runtime (log-scale)
Figure 9: Results for small tables Left: Pareto diagram Right: critical difference diagram of average rank TARTE surpasses the baselines, and act effective table preparator, especially for small number of training samples (n < 256).
22

---

Published in Transactions on Machine Learning Research (08/2025)
TARTE-B-TaBPFNY2
CARTE-B-
TabPFNv2
TARTE-FT
Teb
Normalized score
0.8
TabPFNv2
TARTE-B-XGB
CARTE-FT
256
Vec
TARTE-
Ridge
TARTE- TabPFNv2
RealMLP
TARTE-
16
14
12
10
4
TabVec-XGB
TARTE- CatBoost
XGB
TabVec-MLP TARTE-MLP
T TARTE-B-TabPFNE N-bPPFF22
0.6
TabVect
Ridge
RFbVec-RF
TARTE-RF
TARTE-FT
PARTE-B-TabPFEE
CotB
TARTE-RF
CatBoost
TabPFNv2
TabPFNv2
TabVec-RF
TARTE-B
BARTE-B-XGB
TARTE-MLP
TabVec-XGB
TARTE-Ridge
0.4
TabVec-Ride
TabVec-ReaiMin
TARTE CatBoost
TORTE-TabPFNV2
TabVec-MLP
3.65
365
6mn
1hr
Total runtime (log-scale)
TARTE-B-TaDPFNY2
CARTE-B- TabPFNv2
TARTE-FT
TabVec-
Normalized score
0.8
TabPFNv2
TARTE-B-XGB
CARTE-FT
512
TARTE-
TabVec-XGB
RealMLP
TabVec-
10
Ridge
0.6
TARTE- TabPFNv2
TabVec-
TARTE-
MLP
Catboost
TARTE-
TabVec-
TabPFNv2
TARTE-RF
TARTE-B BARTE-B-TabPEEE BTTbPFNV2
TARTE-FT
B-TAPPFNv2
TabVec-
CatBoost
TARTE-
XGB
TARTE-MLP
CatBoost
TaDVec-TabPFNY2
CARTE-FT
Ridge
TARTE-RF
TabVec-Rid Rid
TARTE-B-XGG
Tab RF
TabVec ReaiMLP
0.4
TabPFNv2
TARTE CatBoost
PARTE-TabPFNN
TARTE- EGB
TARTE-RIdg
TabVec-MLP
TabVec XGB
3.6s
36s
6mn
1hr
Total runtime (log-scale)
TARTE-B-TabPFNV2
CARTE-B- TabPFNv2
TARTE-FT,
CARTE-FT
TabVec- TabPFNv2
aoomalized score
T-RTE-B-
1024
0.8
XGB
Teb
XGB
Rea/MLP
TabVec-
16
14
12
10
4
0.6
TARTE-
Ridge
ATARTE-TaDFFNw2
TabVec-RF
TARTE-XGB
TabVec-MLP
TabPFNv2
TARTE-RF
TARTE-FT
PARTE-B-TabPFNOG
B-bbP--22
CatBoost
TARTE-
TabVec-Ridge
CARTE-FT
TARTE MLP
CatBoost
TAP
TabVec-TabPFNV2
MARTE-CatBooS
TARTE-B-XGB
TabVec- Ridge
TARTE-RF
CatBoost
TabVec-XGB
0.4
TARTE- AGB
TabVec RealML
TabPFNv2
TLP
TARTE- Ridge
Tab
TARTE- TabPFNv2
3.6s
36s
6mn
1hr
Total runtime (log-scale)
Continued.
23

---

Published in Transactions on Machine Learning Research (08/2025)
32
TARTE-B- TabPFNv2 (DS)
64
TARTE-B-
TabPFNv2(DS)
TARTE-B- TabPFNV2
TabVec-
TARTE-B- TabPFNv2
0.7
(Single)
1ARTE-BXXGB
TabPFNV2
(Single)
(Single)
Normalized score
TabVec-
TARTE-B- XGB (inglla)
CARMEEFT
Normalized score
ARTE-8 XGBISingla
TARTE-B XGB(DS)
TabPFNv2 (Single)
0.6
TARTE- Ridge (Single)
TARTE-
TARTE- Ridge (Single)
0.6
CARTE-FT (Single)
Ridge (DS)
Vec- (MT)
0.5
Tvv (MT)
CARTE-FT (Single)
TabVec-XGB (Single)
CatBoost (MT)
(MT)
0.4
0.4
TabVec-XGB
CatBoost (Single)
TabVec-XGB (MT)
CatBoost (Single)
TabVec-XGB (MT)
(Single)
-2mn
6mn
6mn
6mn
Total runtime (log-scale)
Total runtime (log-scale)
CatBoost(Singie))
(-TabFNW22(DS)
TGBVSi-X)
TaBVec-XGB(MT)
TARTE-B- (
TabVec (GB(MT)
CARTE FT(MT)
CatBoost(MTT)
TARTE B 3GGGBDDS)
CatBoost(Single)
B-TabPENN2(Single)
TGB(iccXG
FT(MT)
CatBoost(MM)
TARTE- (-TEEB-XGB(DS)
CARTE (T(ingle)
TARTE В XGB((Snglle)
TMBVecTTabPFNv2(MT)
TabVec- TabPFNN2((Siiglle)
TabVecTabPFNv22MT) TARTE Ridge(Si
TARTE- Ridge)))
CARTE FT(Single)
TARTE (xXGB(Single)
TabVec TabPFiv2inii)
TARTE Ridge(DS)
TARTE Ridge(Singie)
n = 128
TARTE-B- TvbPFNv2 (Single)
256
TARTE-B. TabPFNv2 (Single)
TARTE-B TabPFNv2 (DS)
TabPFNvZ(DS)
TabVec- TabPFNV2 (Single)
TabPFNv2 (Single)
Normalized score
0.8
TARTE-B-
XGBIDS
CARIE MT
Normalized score
TARTE-B-XGB (DS)
CARTE-FT
TARTE-B- KGB(Single)
0.8
TARTE-B XGB(Single)
(M))
CARTE-FT(Single)
CARTE-FT (Single)
TARTE-
0.6
Ridge (Single)
Ridge(DS
TARTE-
TARTE- Ridge (Single)
TARTE-Ridge
(DS)
0.6
TabPFNv2
Tab TabPFNv2 (MT)
(MT)
TabVec-XGB (Single)
TabVec-XGB (Single)
CatBoost
CatBoost (ingle)
0.4
Tab Veec-XGG((M)
0.4
CatBoost (Single)
XGB MTT
CatBoost (MT)
36s
2mn
6mn
~19mn
36s
6mn
1hr
Total runtime (log-scale)
Total runtime (log-scale)
10
TGBVec
B.TabPFNV2(DS))
TabVec-XGB(MT)
CatBoost(Single)
CARTE-FT(MT)
CatBoost(MT)
TARTE 2.TablPNN2(Singla)
MRRTEBBTTbBPFNV((Singgl
CatBoost(Single)
TGBISnnX))
mabVa:TabPFFN2(SSngg
TabVec (SB(igle)
TabVec-Tabe Nvb
TabVe:TabbFNW222
TARTE BxXGB(DS)
Ridge(DS)
TARTE-B-XBB(DS)
TARTE- Ridge(DS)
CARTE-FTIC (ATTE-T(Singgee
TARTE Ridga()
LARTE-FT(Sine)
TARTE-Rldge(Single
TARTE-B-xGB((ingle)
TARTE BxXGB(Siggle)
Figure 9: Results for domain specialization with single source table Regardless of the train-sizes, TARTE-B B can provide benefits with domain specialization. Once domain specialized models are available. TARTE can readily benefit without requiring complex refitting with the source tables.
24


Reviews:
[{"review_id": "gnIlMlfKIP", "reviewer": "Reviewer_GrFF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. TARTE demonstrates that knowledge pre-training can directly improve downstream performance. This contrasts with most existing methods that require fine-tuning.\n2. The model supports multiple deployment modes allowing flexible trade-offs between accuracy and computational cost.\n3. The paper shows that TARTE achieves Pareto-optimal performance, balancing runtime with predictive quality. It is also shown to be reusable, saving training cost in multi-task or transfer settings.\n4. The authors provide an in-depth study of factors influencing pretraining effectiveness, including string similarity, column types, and data heterogeneity. This adds clarity to the model's practical applicability.\n\nWeaknesses:\n1. The FastText-based encoding has known limitations with long or complex string entries, and TARTE inherits this weakness.\n2. The effectiveness of TARTE may diminish on domains with unfamiliar terminology or structure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces TARTE (Transformer Augmented Representation of Table Entries), a pretrained model for tabular data that leverages knowledge pre-training to learn effective representations of both string and numerical entries in tables. Unlike prior tabular models that mainly benefit from fine-tuning, TARTE demonstrates that pretraining alone can yield strong, reusable representations for a wide range of downstream tasks. The key contributions are:\n1. A novel Transformer-based architecture for tabular data encodes cell values enriched with column names, leveraging large-scale factual and numerical data during pre-training. This enables the model to capture semantic associations between entries without requiring access to original pre-training tables during downstream tasks, supporting both fine-tuning and frozen usage in diverse downstream applications.\n2. Extensive empirical validation over small- and mid-sized tables, demonstrating that TARTE consistently outperforms state-of-the-art baselines (e.g., tree-based and neural models) in both accuracy and efficiency. Detailed analysis of pretraining effectiveness, showing that TARTE performs best when downstream tables contain rich and diverse string-based features that overlap with the pretraining corpus.\n3. Demonstration of TARTE's transferability, highlighting its ability to specialize to new domains via domain-specific finetuning or feature reuse, in line with foundation model paradigms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weakness part.\n\nThe authors may also consider larger-scale datasets, and include the results of some non-pretrain methods such as FT-T, ModernNCA, RealMLP, and TabR.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aZQoBAyvLb", "reviewer": "Reviewer_iJVz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper has clear presentation and is easy to follow.\n\n2.\tThe paper conducts abundant experiments and analyzes in which scenarios TARTE has more gains than other methods. This gives a comprehensive picture of the capability boundary of TARTE.\n\n3.\tThe proposed method achieves a good balance between accuracy and efficiency.\n\nWeaknesses:\n\n1.\tIn the paragraph “TARTE as a table featurizer,” the paper claims that “TARTE can be used to generate meaningful embeddings for table entries.” However, there are no further details on how to generate those embeddings. It is suggested that the authors elaborate on this part.\n\n2.\tIt is also suggested that the authors provide more detailed descriptions on how “boosting a complementary model” is done.\n\n3.\tIn Fig. 3, it seems that boosting with TARTE consistently outperforms other methods (including those variants of TARTE). However, there are no baseline boosting methods using other pretrained tabular models (e.g., CARTE). The authors are suggested to explain this issue. If boosting with other pretrained tabular models is feasible, these results should be included in the comparison. If not, please clarify the reason.\n\n4.\tIt is not explained why in Fig. 3 the boosting variant TARTE-B-TabPFNv2 consistently outperforms the finetuning variant TARTE-FT. The authors are encouraged to explain where this performance gain comes from, and when boosting tends to perform worse than finetuning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new pre-training scheme named TARTE for tabular data. TARTE models each row in a table as a pair containing the embeddings of the column name and the cell value. It is pre-trained on large knowledge bases using rational data and is able to benefit downstream tasks through several different approaches, including finetuning, serving as a table featurizer, and boosting another complementary model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested changes:\n1.\tPlease give a brief introduction to what a critical difference diagram is and what message it conveys. This would make the experiment section more self-contained\n\nFor other requested changes, please refer to the Weaknesses section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "I6jNW8zHAx", "reviewer": "Reviewer_FSQi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nOverall, the paper is well written quite easy to follow. I appreciate the authors in drafting a well written manuscript! Some key strengths:\n* TARTE effectively bridges the gap between tabular learning and foundation models by incorporating data semantics through pre-training.\n* Strong empirical validation across diverse datasets (classification and regression, varying sizes and data types).\n* Multiple post-training options (fine-tuning, feature extraction, boosting) make TARTE adaptable to diverse use cases\n\n### Weaknesses\n* A majority of the architectural novelty comes from the baseline model CARTE, so the paper is more about how to modify CARTE to better world considerations like efficient computation and better latency.\n* See Requested changes", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduced TARTE, a foundational model for tabular data. The primary highlight is knowledge pre-training to generate reusable vector representations of table rows. A prior model CARTE serves as a primary baseline, but the new model allows generalizable and reusable baselines reducing the need for domain specific fine tuning. The architecture is a transformer-based model at its core, which jointly processes column names and cell values, enabling context-aware representations. supporting several downstream paradigms (e.g., featurization, boosting, domain transfer). Empirical evaluations show that TARTE outperforms tree-based and neural baselines including TabPFNv2 and CARTE above, especially in few-shot and domain-transfer scenarios, while also offering favorable compute-performance trade-offs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* It is my understanding that CARTE and TARTE have different pre-training datasets. It seems possible to me that the differences in performance entirely stem from this fact. This should also be mentioned in Section \"3.4 Differences to the CARTE approach\", as it seems like a key difference.\n* On a similar note, it might be beneficial to include some ablation studies on architecture or preprocessing steps, as far as computational resources allow. This will certainly strengthen the paper.\n* How were the source and target tables selected/determined?\n* On page 2, \"we first showing how\" -> \"we first show how\"", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)