# Knowledge-Rich Embeddings for Tabular Learning

Félix Lefebvre, Myung Jun Kim, Gaël Varoquaux

► To cite this version:

Félix Lefebvre, Myung Jun Kim, Gaël Varoquaux. Knowledge-Rich Embeddings for Tabular Learning. AITD 2025 - Workshop on AI for Tabular Data, Dec 2025, Copenhagen, Denmark. hal-05418034

HAL Id: hal-05418034

https://inria.hal.science/hal-05418034v1

Submitted on 15 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Knowledge-Rich Embeddings for Tabular Learning

Félix Lefebvre

SODA Team, Inria Saclay
felix.lefebvre@inria.fr

Myung Jun Kim

SODA Team, Inria Saclay

Gaël Varoquaux

SODA Team, Inria Saclay
Probabl, France

Abstract

Tables have their own structure, calling for dedicated tabular learning methods with the right inductive bias. These methods outperform direct applications of language models, which struggle with the heterogeneous features typical in tables, such as numerical data or dates. Yet, many tables contain text that refers to real-world entities, and most tabular learning methods ignore the external knowledge that such strings could unlock. Which knowledge-rich representations should tabular learning leverage? While large language models (LLMs) encode implicit factual knowledge, knowledge graphs (KGs) share the relational structure of tables and come with the promise of better-controlled knowledge. Studying tables in the wild, we assemble 105 tabular learning datasets comprising text. We find that knowledge-rich representations from LLMs or KGs boost prediction and, combined with simple linear models, markedly outperform strong tabular baselines. Larger LLMs and larger KGs both provide greater gains. On datasets where all entities are linked to a KG, LLMs and KG models of similar size perform similarly, suggesting that the benefit of LLMs over KGs is to solve the entity linking problem. Our results highlight that external knowledge is a powerful but underused ingredient for advancing tabular learning.

# 1 Introduction: background knowledge for tabular learning

Tabular data is prominent in machine learning, often containing text entries that refer to real-world entities such as company names, drugs, or locations. While methods like gradient-boosted decision trees and modern table foundation models excel with numerical data [Chen and Guestrin, 2016, Hollmann et al., 2025], the text processing is relegated to a preprocessing step that typically relies on superficial encodings (e.g., character n-grams), discarding the rich, real-world knowledge embedded in these strings. This is a missed opportunity, as external knowledge could significantly boost predictive power, especially in low-data regimes.

A scalable way to inject this knowledge is to use vector representations pretrained on large-scale sources [Cvetkov-Iliev et al., 2023, Grinsztajn et al., 2023, Lefebvre and Varoquaux, 2025]. Two main paradigms exist for this: knowledge graphs (KGs) and large language models (LLMs). General-purpose KGs [Bollacker et al., 2008, Vrandečić and Krötzsch, 2014, Suchanek et al., 2024] offer structured, curated facts, but their use is hampered by incompleteness and the difficult "symbol grounding" problem of linking messy text to canonical entities [Mendes et al., 2011]. In contrast, LLMs implicitly encode vast world knowledge from web-scale text and can embed any string, effectively sidestepping the entity linking challenge. However, this comes at the cost of reliability, as their knowledge is statistical, not factual, and prone to hallucination [Ji et al., 2023].

This raises a critical question: which source of knowledge is more effective for tabular learning? To investigate this, we conduct a large-scale empirical study on 105 tabular datasets, assembled from three diverse sources. We compare knowledge-rich representations from LLMs and KG models, evaluating their impact on downstream prediction tasks. Our key findings are:
1. **Good representations matter more than sophisticated downstream tabular learners:** Both LLM and KG embeddings, when paired with a simple linear model, outperform strong tabular baselines that use superficial text encodings.

2. **Scale brings gains:** Performance improves with the size of both LLMs and KGs.

3. **Entity linking is the key bottleneck:** On a subset of tables where entities are pre-linked to a KG, KG embeddings perform on par with LLMs of similar size. This suggests that the primary advantage of LLMs is not superior knowledge, but their ability to implicitly solve the entity linking problem.

## 2 Related work

**Tabular learning with text features** Tabular learning has long been dominated by gradient-boosted decision trees [Chen and Guestrin, 2016], but recent deep learning approaches, including table foundation models [Hollmann et al., 2025, Ma et al., 2024, Qu et al., 2025], now often outperform them [Erickson et al., 2025]. A shared limitation, however, is the absence of specific handling of text features, which are typically vectorized using superficial methods like TF-IDF that ignore the semantics and external knowledge embedded in the strings.

To address this, recent work has explored using external knowledge sources. One approach leverages LLMs by serializing table rows into text to be processed by an LLM [Hegselmann et al., 2023, Gardner et al., 2024]. An alternative paradigm uses KGs to pretrain tabular models [Kim et al., 2024, 2025]. Prior comparative studies have shown that embeddings from language models outperform traditional substring-based encoders, especially in diverse-entry and low-data regimes [Grinsztajn et al., 2023, Kasneci and Kasneci, 2024]. However, these works do not provide a direct comparison between knowledge sourced from LLMs versus structured KGs, a gap our study aims to fill.

**Learning representations from knowledge sources** Knowledge can be extracted from KGs by learning embeddings from the graph structure [Bordes et al., 2013, Yang et al., 2014, Trouillon et al., 2016, Sun et al., 2019] or by using language models to encode the textual descriptions of entities and relations [Wang et al., 2021, Saxena et al., 2022]. A related line of work refines general-purpose LLMs on knowledge-base data to improve their factual grounding and performance on knowledge-intensive tasks [Sun et al., 2020, Feng et al., 2023]. Our work evaluates and contrasts these different representation strategies in the context of tabular learning.

## 3 Methodology: a benchmark for table background knowledge

**105 tabular datasets** We assemble a diverse benchmark from three sources: TextTabBench [Mráz et al., 2025], CARTE [Kim et al., 2024], and WikiDBs [Vogel et al., 2024], covering regression, binary, and multi-class classification tasks (Table 1). To focus on text-based knowledge, we remove all numerical columns and apply standard preprocessing (see Appendix A), ensuring each dataset has at least 1,050 rows. For a controlled comparison between LLMs and KGs, we identify a subset of 15 tables where entries are unambiguously linked to Wikidata5M [Wang et al., 2021]. This allows us to evaluate pure KG models in a setting where the entity linking problem is solved. We also create smaller KG versions by filtering out low-degree entities and retaining the largest connected component, to study the impact of KG size (Table 2).

<table><caption>Table 1: Task distribution across sources of tables.</caption><thead><tr><th>Source</th><th>b-clf</th><th>m-clf</th><th>reg</th><th>Total</th></tr></thead><tbody><tr><td>TextTabBench</td><td>5</td><td>2</td><td>10</td><td><strong>17</strong></td></tr><tr><td>CARTE</td><td>11</td><td>0</td><td>40</td><td><strong>51</strong></td></tr><tr><td>WikiDBs</td><td>1</td><td>21</td><td>15</td><td><strong>37</strong></td></tr></tbody><tfoot><tr><th>Total</th><td><strong>17</strong></td><td><strong>23</strong></td><td><strong>65</strong></td><td><strong>105</strong></td></tr></tfoot></table>

<table><caption>Table 2: Knowledge graph datasets. All graphs use the same 822 relations.</caption><thead><tr><th></th><th># entities</th><th># triples</th><th>deg.</th></tr></thead><tbody><tr><td>Wikidata5M</td><td>4.6M</td><td>20.6M</td><td>-</td></tr><tr><td>Wikidata3M</td><td>3.2M</td><td>15.5M</td><td>3</td></tr><tr><td>Wikidata2M</td><td>2.1M</td><td>11.5M</td><td>4</td></tr><tr><td>Wikidata1M</td><td>1.1M</td><td>6.8M</td><td>6</td></tr><tr><td>Wikidata500k</td><td>0.5M</td><td>3.1M</td><td>9</td></tr></tbody></table>

**Evaluation pipeline** Figure 4 shows our pipeline to evaluate representations on downstream tasks. For each dataset, we sample 1,024 training rows, and 1,024 testing rows (or all remaining rows if fewer are available), and average results over 10 random seeds.
**Embedding models** We compare a wide array of representations:

* **Non-pretrained baseline:** TF-IDF followed by SVD, via the Skrub library.

* **Pure LLMs:** A diverse set including Llama 3 [Dubey et al., 2024], Qwen3 [Zhang et al., 2025], RoBERTa [Liu et al., 2019], and others to study the effect of scale and architecture.

* **Hybrid LLM+KG models:** Models that refine LLMs on relational data, such as ERNIE 2.0, KGT5, and TabuLa-8B, each time compared against their corresponding base LLM.

* **Pure KG models:** For linked tables, we use classic KG embeddings (DistMult, TransE, ComplEX, and RotatE) trained on Wikidata5M and its subsets.

**Table serialization and downstream estimators** We serialize each table row into a natural language prompt (e.g., "The <col_a> is <val_a>. The <col_b> is <val_b>.", details in Appendix A), enabling LLMs to generate context-aware embeddings. These embeddings are then used to train three representative tabular learners: Ridge regression, XGBoost, and the TabPFNv2 foundation model. For the latter two, we use PCA to reduce dimensionality to *d* = 300 and *d* = 500, respectively.

# 4 Results: knowledge representations for tabular learning

## 4.1 Knowledge-rich representations boost tabular learning

**Good representations matter more than advanced tabular models** Our central finding is that text representation quality is paramount. As shown in Figure 1a, a simple linear model (Ridge) fed with high-quality embeddings from modern LLMs consistently outperforms sophisticated tabular learners like XGBoost and TabPFNv2 that use superficial text encodings. This suggests that for tables

<figure><img src="image_2.png" /><figcaption>Figure 1: Performance gain of the various knowledge-rich representations compared to a non-pretrained baseline – **a.** Comparisons including three tabular learners: Ridge, XGBoost, and TabPFNv2; absolute scores. – **b.** Relative improvements to non-pretrained string representations, when using a ridge model as a tabular learner; normalized scores (0 is 10% worse, 1 is the best score observed). – Appendix Figure 5 gives critical difference diagrams across all methods and datasets.</figcaption></figure>
rich in text, the primary performance bottleneck is not the downstream learning algorithm but the semantic poverty of the input features.

**Modern tabular models struggle with high-dimensional embeddings** Interestingly, the performance benefit of advanced tabular learners disappears when they are given knowledge-rich embeddings. XGBoost and TabPFNv2, when paired with LLM embeddings, underperform the simple Ridge model (Figure 1a). A likely cause is the need for aggressive dimensionality reduction (PCA) to make these embeddings compatible with the tabular learners, which may discard valuable information. This could also come from a mismatch between the inductive biases of current tabular models and the dense, rotationally-invariant nature of modern text representations [Grinsztajn et al., 2022].

**The benefit of refining LLMs on relational data is unclear** We find no consistent evidence that refining LLMs on relational data (hybrid models) improves performance. As seen in Figure 1b, while ERNIE 2.0 shows a clear gain over its RoBERTa base, KGT5 does not improve upon T5, and TabuLa-8B is on par with its Llama base. The effectiveness of such refinement appears highly dependent on the specific model and pretraining strategy, warranting further investigation.

## 4.2 LLMs vs. KGs: entity linking is the bottleneck

To isolate the value of knowledge from the challenge of entity linking, we evaluate performance on a subset of 15 tables where all text entries are pre-linked to Wikidata5M entities. This allows a direct comparison between pure LLM embeddings and pure KG embeddings.

**Scale is key** Figure 2 shows a clear trend for both LLMs and KGs: bigger is better. Performance improves steadily with the size of the LLM and the size of the KG used for training embeddings. For KGs, smaller graphs cover fewer entities, leading to a sharp performance drop as entity coverage decreases. LLMs exhibit a softer degradation, as they can still generate a (less informed) embedding for any string.

**When linking is solved, KGs match LLMs**

Crucially, on these fully linked tables, the best KG embedding models perform on par with LLMs of a similar parameter count (Figure 2). This suggests that the primary advantage of LLMs in the general setting is not that they encode superior knowledge, but that they implicitly solve the difficult "symbol grounding" problem of linking messy, real-world text to canonical entities.

When this problem is removed, the structured knowledge from KGs is just as effective. This also implies that for these tasks, the advanced language understanding capabilities of LLMs beyond entity recognition provide little additional benefit.

<figure><img src="image_3.png" /><figcaption>Figure 2: Comparing pure-KG to pure-LLM approaches on matched tables. Performance scales with model size for both. On fully linked data, KG embeddings match LLMs of similar size.</figcaption></figure>

## 5 Conclusion

Our large-scale study demonstrates that for tabular learning with text, the quality of text representations is crucial. Knowledge-rich embeddings from LLMs or KGs bring more gains than complex downstream models. Thus, the focus should shift from the learning algorithm to the quality of the input features. When entity linking is provided, pure KG models are just as powerful as LLMs of similar size, suggesting that the main advantage of LLMs for tabular data is not superior knowledge, but their ability to bridge the gap between unstructured text and canonical entities. The clear benefit of scale points to a crucial direction for future research: next-generation tabular foundation models should be built upon large language models to leverage their powerful entity linking and knowledge encoding capabilities. Tapping into massive knowledge bases like the full Wikidata could unlock significant performance gains and lead to more powerful and versatile tabular learning systems.
References

Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In *Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining*, pages 785–794, 2016.

Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. *Nature*, 637(8045):319–326, 2025.

Alexis Cvetkov-Iliev, Alexandre Allauzen, and Gaël Varoquaux. Relational data embeddings for feature enrichment with background information. *Machine Learning*, 112(2):687–720, 2023.

Léo Grinsztajn, Edouard Oyallon, Myung Jun Kim, and Gaël Varoquaux. Vectorizing string entries for data processing on tables: when are larger language models better? *arXiv preprint arXiv:2312.09634*, 2023.

Félix Lefebvre and Gaël Varoquaux. Scalable feature learning on huge knowledge graphs for downstream machine learning. *Advances in Neural Information Processing Systems*, 38, 2025.

Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In *Proceedings of the 2008 ACM SIGMOD international conference on Management of data*, pages 1247–1250, 2008.

Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. *Communications of the ACM*, 57(10):78–85, 2014.

Fabian M Suchanek, Mehwish Alam, Thomas Bonald, Lihu Chen, Pierre-Henri Paris, and Jules Soria. Yago 4.5: A large and clean knowledge base with a rich taxonomy. In *Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval*, pages 131–140, 2024.

Pablo N Mendes, Max Jakob, Andrés García-Silva, and Christian Bizer. Dbpedia spotlight: shedding light on the web of documents. In *Proceedings of the 7th international conference on semantic systems*, pages 1–8, 2011.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. *ACM computing surveys*, 55(12):1–38, 2023.

Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony L Caterini. Tabdpt: Scaling tabular foundation models. *arXiv preprint arXiv:2410.18164*, 2024.

Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. Tabicl: A tabular foundation model for in-context learning on large data. *arXiv preprint arXiv:2502.05564*, 2025.

Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. Tabarena: A living benchmark for machine learning on tabular data. *arXiv preprint arXiv:2506.16791*, 2025.

Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In *International conference on artificial intelligence and statistics*, pages 5549–5581. PMLR, 2023.

Josh Gardner, Juan C Perdomo, and Ludwig Schmidt. Large scale transfer learning for tabular data via language modeling. *Advances in Neural Information Processing Systems*, 37:45155–45205, 2024.

Myung Jun Kim, Leo Grinsztajn, and Gael Varoquaux. Carte: Pretraining and transfer for tabular learning. In *ICML*, 2024.

Myung Jun Kim, Félix Lefebvre, Gaëtan Brison, Alexandre Perez-Lebel, and Gaël Varoquaux. Table foundation models: on knowledge pre-training for tabular learning. *arXiv preprint arXiv:2505.14415*, 2025.
Gjergji Kasneci and Enkelejda Kasneci. Enriching tabular data with contextual llm embeddings: A comprehensive ablation study for ensemble classifiers. arXiv preprint arXiv:2411.01645, 2024.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. *Advances in neural information processing systems*, 26, 2013.

Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575, 2014.

Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In *International conference on machine learning*, pages 2071–2080. PMLR, 2016.

Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197, 2019.

Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. *Transactions of the Association for Computational Linguistics*, 9:176–194, 2021.

Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. Sequence-to-sequence knowledge graph completion and question answering. arXiv preprint arXiv:2203.10321, 2022.

Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for language understanding. In *Proceedings of the AAAI conference on artificial intelligence*, volume 34, pages 8968–8975, 2020.

Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge card: Filling llms' knowledge gaps with plug-in specialized language models. arXiv preprint arXiv:2305.09955, 2023.

Martin Mráz, Breenda Das, Anshul Gupta, Lennart Purucker, and Frank Hutter. Towards benchmarking foundation models for tabular data with text. arXiv preprint arXiv:2507.07829, 2025.

Liane Vogel, Jan-Micha Bodensohn, and Carsten Binnig. Wikidbs: A large-scale corpus of relational databases from wikidata. *Advances in Neural Information Processing Systems*, 37:41186–41201, 2024.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. *arXiv e-prints*, pages arXiv-2407, 2024.

Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.

Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? *Advances in neural information processing systems*, 35: 507–520, 2022.

Mehdi Ali, Max Berrendorf, Charles Tapley Hoyt, Laurent Vermue, Sahand Sharifzadeh, Volker Tresp, and Jens Lehmann. Pykeen 1.0: a python library for training and evaluating knowledge graph embeddings. *Journal of Machine Learning Research*, 22(82):1–6, 2021.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. *Journal of machine learning research*, 21(140):1–67, 2020.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. *arXiv preprint arXiv:2212.03533*, 2022.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. *arXiv preprint arXiv:2205.01068*, 2022.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In *Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations*, pages 38–45, 2020.

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. *Transactions of the association for computational linguistics*, 5:135–146, 2017.
<figure><img src="image_4.png" /><figcaption>Figure 3: Statistics distribution across sources.</figcaption></figure>

## A More details on the experiments

**Dataset selection and properties** TextTabBench and CARTE are established benchmarks for tabular learning, providing real-world tables with varied text features, from short entity names to longer descriptions. Each table is associated with a predefined prediction task (regression, binary, or multi-class classification). WikiDBs is a large corpus of over 1.6 million semi-synthetic tables generated from Wiki data. To create meaningful tasks from this source, we first filtered for tables with at least 1,200 rows, then manually curated a subset of 37 tables for which we could define a relevant prediction problem. Table 1 summarizes the final distribution of tasks across the three sources, and Table 6 gives general properties of the tables. Further details on each individual dataset are provided in Table 7, Table 8, and Table 9.

Figure 3 gives statistics about table sizes, proportion of missing values, and mean column cardinality.

**Data preprocessing** We adopt the original preprocessing from TextTabBench and CARTE. For WikiDBs, we apply a procedure similar to TextTabBench. We also ensure that multi-class classification tasks have at most 10 classes, each with at least 105 samples. For all 105 datasets, we then apply the following preprocessing pipeline: (1) we remove all numerical columns to focus our study on text-based knowledge; (2) we log-transform regression targets with wide-ranging distributions; (3) we downsample majority classes in multi-class problems to create balanced datasets; and (4) we discard any table with fewer than 1,050 rows post-processing to ensure sufficient data for evaluation. We also exclude one

dataset from TextTabBench with excessively long text entries that exceed the context limits of some of our baselines.

<table><caption>Table 3: Task distribution across sources, for linked tables.</caption><thead><tr><th>Source</th><th>b-clf</th><th>m-clf</th><th>reg</th><th>Total</th></tr></thead><tbody><tr><td>CARTE</td><td>0</td><td>0</td><td>3</td><td><strong>3</strong></td></tr><tr><td>WikiDBs</td><td>1</td><td>3</td><td>8</td><td><strong>12</strong></td></tr><tr><td><strong>Total</strong></td><td><strong>1</strong></td><td><strong>3</strong></td><td><strong>11</strong></td><td><strong>15</strong></td></tr></tbody></table>

<table><caption>Table 4: Embedding dimensions for the different baseline models.</caption><thead><tr><th>Model</th><th>Dimension</th></tr></thead><tbody><tr><td>TF-IDF + SVD</td><td>30 per column</td></tr><tr><td>FastText</td><td>300</td></tr><tr><td>TARTE</td><td>768</td></tr><tr><td>Llama-3.2-1B</td><td>2048</td></tr><tr><td>Llama-3.2-3B</td><td>3072</td></tr><tr><td>Llama-3.1-8B</td><td>4096</td></tr><tr><td>TabuLa-8B</td><td>4096</td></tr><tr><td>Qwen3-0.6B</td><td>1024</td></tr><tr><td>Qwen3-4B</td><td>2560</td></tr><tr><td>Qwen3-8B</td><td>4096</td></tr><tr><td>RoBERTa (base, large)</td><td>768, 1024</td></tr><tr><td>ERNIE 2.0 (base, large)</td><td>768, 1024</td></tr><tr><td>e5-v2 (small, base)</td><td>384, 768</td></tr><tr><td>T5 (small, base)</td><td>512, 768</td></tr><tr><td>KGT5 (small, base)</td><td>512, 768</td></tr><tr><td>OPT-1.3B</td><td>2048</td></tr><tr><td>Knowledge card</td><td>2048</td></tr></tbody></table>

**Experiments on linked tables** We have 15 linked tables, 4 for classification and 11 for regression. Details on these tables are provided in Table 3.

For the KG embedding models (DistMult, TransE, ComplEx and RotatE), we use $d = 300$ for the embedding dimension, and train them for 100 epochs with a batch size of 8192 and a learning rate of $10^{-3}$, and use the default parameters of their PyKEEN implementation [Ali et al., 2021].
For KGs smaller than Wikidata5M (see Table 2), some rows of the linked tables are not matched to the KG. In that case, after embedding the rows corresponding to matched entities, we impute missing values using the mean along each column. If no row at all is matched in a table, we simply replace the missing values with zeros.

The number of parameters reported in Figure 2 for KG models is the number of entities in the KG multiplied by the embedding dimension *d*.

**Extracting embeddings from LLMs** We generated sentence-level embeddings from the serialized rows using the SentenceTransformer framework [Reimers and Gurevych, 2019], which provides a unified interface for a wide range of transformer-based models. We used it to extract representations from the following models: Llama 3 [Dubey et al., 2024], Qwen3 [Zhang et al., 2025], RoBERTa [Liu et al., 2019], T5 [Raffel et al., 2020], e5-v2 [Wang et al., 2022], OPT [Zhang et al., 2022], TabuLa [Gardner et al., 2024], ERNIE 2.0 [Sun et al., 2020], Knowledge Card [Feng et al., 2023], and KGT5 [Saxena et al., 2022] families, using pretrained checkpoints available on the Hugging Face Hub [Wolf et al., 2020]. For TARTE [Kim et al., 2025] and FastText [Bojanowski et al., 2017], we directly use the output embeddings of the model.

**Table serialization** To generate embeddings from LLMs, we serialize each table row into a natural language prompt. Following Gardner et al. [2024], we use the format: "The <col_a> is <val_a>. The <col_b> is <val_b>. What is the value of <target>?". For KGT5, we adapt the prompt to better match its pretraining format: "<col_a> | <val_a>. <col_b> | <val_b>. Predict: <target>". Constructing the embeddings across multiple columns (as opposed to the study of Grinsztajn et al. [2023]) is important because it enables the context (column name, other entries on the same row) to inform the representation, e.g. leading to disambiguate "Cambridge; UK" from "Cambridge; Massachusetts" in a table with columns "city; country".

**Embedding dimensions** Table 4 reports the embedding dimensions for the different baseline models used.

<table><caption>Table 5: Search space for XGBoost hyperparameters.</caption><thead><tr><td>Hyperparameter</td><td>Distribution</td><td>Range</td></tr></thead><tbody><tr><td>n_estimators</td><td>Integer</td><td>[50, 1000]</td></tr><tr><td>max_depth</td><td>Integer</td><td>[2, 6]</td></tr><tr><td>min_child_weight</td><td>Log-uniform</td><td>[1, 100]</td></tr><tr><td>subsample</td><td>Uniform</td><td>[0.5, 1.0]</td></tr><tr><td>learning_rate</td><td>Log-uniform</td><td>[10<sup>-5</sup>, 1]</td></tr><tr><td>colsample_bylevel</td><td>Uniform</td><td>[0.5, 1.0]</td></tr><tr><td>colsample_bytree</td><td>Uniform</td><td>[0.5, 1.0]</td></tr><tr><td>gamma</td><td>Log-uniform</td><td>[10<sup>-8</sup>, 7]</td></tr><tr><td>reglambda</td><td>Log-uniform</td><td>[1, 4]</td></tr><tr><td>alpha</td><td>Log-uniform</td><td>[10<sup>-8</sup>, 100]</td></tr></tbody></table>

<table><caption>Table 6: Aggregated features of tabular datasets across sources. The cardinality is computed on 1,024 rows.</caption><thead><tr><td></td><td>TextTabBench</td><td>CARTE</td><td>WikiDBs</td></tr></thead><tbody><tr><td># columns</td><td>15.65</td><td>6.76</td><td>6.73</td></tr><tr><td>cardinality</td><td>286.36</td><td>371.44</td><td>463.70</td></tr><tr><td>string length</td><td>975.29</td><td>298.80</td><td>203.62</td></tr><tr><td>string similarity<sup>1</sup></td><td>0.16</td><td>0.10</td><td>0.08</td></tr></tbody></table>

**Metrics and score normalization** We evaluate performance using the R2 score for regression and the ROC-AUC score for classification. To aggregate results across datasets of varying difficulty, we normalize scores for each dataset and random seed. Following Grinsztajn et al. [2022], we establish a normalized scale where the best-performing model scores 1 and the model at the 10th performance percentile scores 0. Other models' scores are mapped to this [0, 1] range via an affine transformation. For regression, we clip scores at 0 to mitigate the impact of poor-performing outliers.

**Uncertainty estimation** To account for statistical variability, we repeat each experiment 10 times with different random seeds. The error bars in our result figures represent the standard error of the mean across these runs.

**XGBoost hyperparameter tuning** For the XGBoost estimator, we perform hyperparameter optimization via a randomized search with 100 iterations. We use 5-fold cross-validation, repeated 5 times on the training set, to evaluate each hyperparameter configuration. The detailed search space is provided in Table 5.

<footnote n="1">¹ cosine similarity of TF-IDF across rows</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 4: An overview of our evaluation pipeline. For each dataset, we sample training and test sets. We then serialize the rows and use the embedding **model** to generate a vector representation for each row. Finally, we train a tabular learning estimator to evaluate these embeddings.</figcaption></figure>

**Overview of the evaluation pipeline** Figure 4 summarizes our evaluation pipeline.

# B Additional results

## B.1 Runtime analysis

The benefits of leveraging external knowledge come at a computational cost. Table 10 details the average runtimes for embedding generation and estimator fitting (ridge) across different embedding models and training sizes. As expected, larger models introduce a significant computational overhead. For instance, generating embeddings with an 8-billion-parameter LLM is, on average, over 100 times slower than using the non-pretrained baseline. This highlights the trade-off between predictive performance and the computational resources required for knowledge integration.

<table><caption>Table 7: Overview of TextTabBench datasets used in our benchmark. Table statistics after preprocess-
ing.</caption><thead><tr><th>Dataset</th><th>Task</th><th># rows</th><th># columns</th><th># classes</th><th># linked rows</th></tr></thead><tbody><tr><td>Diabetes</td><td>b-clf</td><td>17,000</td><td>5</td><td>2</td><td>-</td></tr><tr><td>Job Frauds</td><td>b-clf</td><td>1,732</td><td>12</td><td>2</td><td>-</td></tr><tr><td>Kickstarter</td><td>b-clf</td><td>18,720</td><td>10</td><td>2</td><td>-</td></tr><tr><td>Lending Club</td><td>b-clf</td><td>11,254</td><td>13</td><td>2</td><td>-</td></tr><tr><td>Osha Accidents</td><td>b-clf</td><td>3,598</td><td>16</td><td>2</td><td>-</td></tr><tr><td>Customer Complaints</td><td>m-clf</td><td>1,384</td><td>9</td><td>4</td><td>-</td></tr><tr><td>Spotify</td><td>m-clf</td><td>10,000</td><td>4</td><td>10</td><td>-</td></tr><tr><td>Airbnb</td><td>reg</td><td>3,818</td><td>33</td><td>-</td><td>-</td></tr><tr><td>Beer</td><td>reg</td><td>2,914</td><td>6</td><td>-</td><td>-</td></tr><tr><td>California Houses</td><td>reg</td><td>11,349</td><td>14</td><td>-</td><td>-</td></tr><tr><td>Covid Trials</td><td>reg</td><td>1,165</td><td>14</td><td>-</td><td>-</td></tr><tr><td>Insurance Complaints</td><td>reg</td><td>37,484</td><td>9</td><td>-</td><td>-</td></tr><tr><td>IT Salary</td><td>reg</td><td>1,253</td><td>17</td><td>-</td><td>-</td></tr><tr><td>Mercari</td><td>reg</td><td>12,000</td><td>5</td><td>-</td><td>-</td></tr><tr><td>San Francisco Permits</td><td>reg</td><td>183,794</td><td>13</td><td>-</td><td>-</td></tr><tr><td>Stack Overflow</td><td>reg</td><td>19,427</td><td>90</td><td>-</td><td>-</td></tr><tr><td>Wine</td><td>reg</td><td>1,281</td><td>13</td><td>-</td><td>-</td></tr></tbody></table>
<figure><img src="image_6.png" /><figcaption>Figure 5: Critical difference diagram across all data sources and methods.</figcaption></figure>

## B.2 Overall model ranking

Figure 5 presents a critical difference diagram comparing the mean ranks of all embedding methods when paired with a Ridge predictor. It also includes the performance of more advanced estimators on non-pretrained representations for context.

## B.3 Performance analysis by data source

Figure 6 illustrates the relative improvements of knowledge-rich representations over non-pretrained ones, broken down by data source. The benefits of external knowledge vary with dataset characteristics; tables from WikiDBs and CARTE, which are more knowledge-intensive, gain more from these representations than those from TextTabBench.

Table 6 also shows that TextTabBench table entries are less varied than the ones of CARTE and WikiDBs. This probably adds regularities that non-pretrained features combined with TabPFNv2 can leverage, explaining their better performance on these datasets.
<table><caption>Table 8: Overview of CARTE datasets used in our benchmark. Table statistics after preprocessing.</caption><thead><tr><th>Dataset</th><th>Task</th><th># rows</th><th># columns</th><th># classes</th><th># linked rows</th></tr></thead><tbody><tr><td>Chocolate Bar Ratings</td><td>b-clf</td><td>2,218</td><td>7</td><td>2</td><td>-</td></tr><tr><td>Coffee Ratings</td><td>b-clf</td><td>1,670</td><td>9</td><td>2</td><td>-</td></tr><tr><td>Michelin</td><td>b-clf</td><td>6,774</td><td>6</td><td>2</td><td>-</td></tr><tr><td>NBA Draft</td><td>b-clf</td><td>1,550</td><td>5</td><td>2</td><td>-</td></tr><tr><td>Ramen Ratings</td><td>b-clf</td><td>3,726</td><td>5</td><td>2</td><td>-</td></tr><tr><td>Roger Ebert</td><td>b-clf</td><td>2,668</td><td>6</td><td>2</td><td>-</td></tr><tr><td>Spotify</td><td>b-clf</td><td>41,096</td><td>8</td><td>2</td><td>-</td></tr><tr><td>US Accidents Severity</td><td>b-clf</td><td>20,930</td><td>10</td><td>2</td><td>-</td></tr><tr><td>Whisky</td><td>b-clf</td><td>1,788</td><td>7</td><td>2</td><td>-</td></tr><tr><td>Yelp</td><td>b-clf</td><td>60,088</td><td>9</td><td>2</td><td>-</td></tr><tr><td>Zomato</td><td>b-clf</td><td>60,302</td><td>8</td><td>2</td><td>-</td></tr><tr><td>Movies</td><td>reg</td><td>7,224</td><td>8</td><td>-</td><td>7,095</td></tr><tr><td>US Accidents Counts</td><td>reg</td><td>22,623</td><td>7</td><td>-</td><td>14,697</td></tr><tr><td>US Presidential</td><td>reg</td><td>19,857</td><td>7</td><td>-</td><td>13,221</td></tr><tr><td>Anime Planet</td><td>reg</td><td>14,391</td><td>7</td><td>-</td><td>-</td></tr><tr><td>Babies R Us</td><td>reg</td><td>5,085</td><td>5</td><td>-</td><td>-</td></tr><tr><td>Beer Ratings</td><td>reg</td><td>3,197</td><td>6</td><td>-</td><td>-</td></tr><tr><td>Bikedekho</td><td>reg</td><td>4,786</td><td>6</td><td>-</td><td>-</td></tr><tr><td>Bikewale</td><td>reg</td><td>8,992</td><td>6</td><td>-</td><td>-</td></tr><tr><td>Buy Buy Baby</td><td>reg</td><td>10,718</td><td>5</td><td>-</td><td>-</td></tr><tr><td>Cardekho</td><td>reg</td><td>37,813</td><td>14</td><td>-</td><td>-</td></tr><tr><td>Clear Corpus</td><td>reg</td><td>4,724</td><td>11</td><td>-</td><td>-</td></tr><tr><td>Company Employees</td><td>reg</td><td>10,941</td><td>8</td><td>-</td><td>-</td></tr><tr><td>Employee Remuneration</td><td>reg</td><td>35,396</td><td>3</td><td>-</td><td>-</td></tr><tr><td>Employee Salaries</td><td>reg</td><td>9,211</td><td>7</td><td>-</td><td>-</td></tr><tr><td>Fifa22 Players</td><td>reg</td><td>18,085</td><td>10</td><td>-</td><td>-</td></tr><tr><td>Filmtv Movies</td><td>reg</td><td>41,205</td><td>7</td><td>-</td><td>-</td></tr><tr><td>Journal JCR</td><td>reg</td><td>9,615</td><td>5</td><td>-</td><td>-</td></tr><tr><td>Journal SJR</td><td>reg</td><td>27,931</td><td>10</td><td>-</td><td>-</td></tr><tr><td>Japanese anime</td><td>reg</td><td>15,535</td><td>12</td><td>-</td><td>-</td></tr><tr><td>K-Drama</td><td>reg</td><td>1,239</td><td>9</td><td>-</td><td>-</td></tr><tr><td>ML/DS salaries</td><td>reg</td><td>10,456</td><td>8</td><td>-</td><td>-</td></tr><tr><td>Museums</td><td>reg</td><td>11,467</td><td>15</td><td>-</td><td>-</td></tr><tr><td>Mydramalist</td><td>reg</td><td>3,400</td><td>11</td><td>-</td><td>-</td></tr><tr><td>Prescription Drugs</td><td>reg</td><td>1,714</td><td>6</td><td>-</td><td>-</td></tr><tr><td>Rotten Tomatoes</td><td>reg</td><td>7,158</td><td>11</td><td>-</td><td>-</td></tr><tr><td>Used Cars 24</td><td>reg</td><td>5,918</td><td>7</td><td>-</td><td>-</td></tr><tr><td>Used Cars Benz Italy</td><td>reg</td><td>16,391</td><td>6</td><td>-</td><td>-</td></tr><tr><td>UsedCars.com</td><td>reg</td><td>4,009</td><td>9</td><td>-</td><td>-</td></tr><tr><td>Used Cars Pakistan</td><td>reg</td><td>72,655</td><td>5</td><td>-</td><td>-</td></tr><tr><td>Used Cars Saudi Arabia</td><td>reg</td><td>5,507</td><td>8</td><td>-</td><td>-</td></tr><tr><td>Videogame Sales</td><td>reg</td><td>16,410</td><td>5</td><td>-</td><td>-</td></tr><tr><td>Wikiliq Beer</td><td>reg</td><td>13,461</td><td>8</td><td>-</td><td>-</td></tr><tr><td>Wikiliq Spirit</td><td>reg</td><td>12,275</td><td>6</td><td>-</td><td>-</td></tr><tr><td>Wina Poland</td><td>reg</td><td>2,247</td><td>13</td><td>-</td><td>-</td></tr><tr><td>Wine.com Prices</td><td>reg</td><td>15,254</td><td>7</td><td>-</td><td>-</td></tr><tr><td>Wine.com Ratings</td><td>reg</td><td>4,095</td><td>7</td><td>-</td><td>-</td></tr><tr><td>WineEnthusiasts Prices</td><td>reg</td><td>120,975</td><td>9</td><td>-</td><td>-</td></tr><tr><td>WineEnthusiasts Ratings</td><td>reg</td><td>129,971</td><td>9</td><td>-</td><td>-</td></tr><tr><td>WineVivino Price</td><td>reg</td><td>13,834</td><td>6</td><td>-</td><td>-</td></tr><tr><td>WineVivino Rating</td><td>reg</td><td>13,834</td><td>7</td><td>-</td><td>-</td></tr></tbody></table>
<table><caption>Table 9: Overview of WikiDBs datasets used in our benchmark. Table statistics after preprocessing.</caption><thead><tr><th>Dataset</th><th>Task</th><th># rows</th><th># columns</th><th># classes</th><th># linked rows</th></tr></thead><tbody><tr><td>CC Authors</td><td>b-clf</td><td>16,224</td><td>8</td><td>2</td><td>1,302</td></tr><tr><td>Defenders</td><td>m-clf</td><td>18,610</td><td>11</td><td>10</td><td>8,700</td></tr><tr><td>Philosophers</td><td>m-clf</td><td>4,230</td><td>9</td><td>10</td><td>1,656</td></tr><tr><td>US Music Albums</td><td>m-clf</td><td>3,270</td><td>11</td><td>10</td><td>2,180</td></tr><tr><td>Artist Copyrights</td><td>m-clf</td><td>2,000</td><td>10</td><td>10</td><td>-</td></tr><tr><td>Artworks Catalog</td><td>m-clf</td><td>1,210</td><td>9</td><td>10</td><td>-</td></tr><tr><td>Forward Players</td><td>m-clf</td><td>1,400</td><td>11</td><td>10</td><td>-</td></tr><tr><td>Geographers</td><td>m-clf</td><td>1,130</td><td>10</td><td>10</td><td>-</td></tr><tr><td>Historic Buildings</td><td>m-clf</td><td>27,980</td><td>7</td><td>10</td><td>-</td></tr><tr><td>Islands</td><td>m-clf</td><td>19,650</td><td>4</td><td>10</td><td>-</td></tr><tr><td>Kindergarten Locations</td><td>m-clf</td><td>2,790</td><td>7</td><td>3</td><td>-</td></tr><tr><td>Magic Narratives</td><td>m-clf</td><td>1,062</td><td>5</td><td>9</td><td>-</td></tr><tr><td>Museums</td><td>m-clf</td><td>9,550</td><td>5</td><td>10</td><td>-</td></tr><tr><td>Noble Individuals</td><td>m-clf</td><td>1,400</td><td>10</td><td>10</td><td>-</td></tr><tr><td>Notable Trees</td><td>m-clf</td><td>1,408</td><td>5</td><td>8</td><td>-</td></tr><tr><td>Parish Churches</td><td>m-clf</td><td>1,350</td><td>5</td><td>10</td><td>-</td></tr><tr><td>Sculptures</td><td>m-clf</td><td>3,720</td><td>7</td><td>10</td><td>-</td></tr><tr><td>Spring Locations</td><td>m-clf</td><td>5,930</td><td>3</td><td>10</td><td>-</td></tr><tr><td>State Schools</td><td>m-clf</td><td>2,800</td><td>4</td><td>10</td><td>-</td></tr><tr><td>Scientific Articles</td><td>m-clf</td><td>2,760</td><td>14</td><td>10</td><td>-</td></tr><tr><td>Sub Post Offices</td><td>m-clf</td><td>1,530</td><td>4</td><td>10</td><td>-</td></tr><tr><td>Transport Stations</td><td>m-clf</td><td>4,640</td><td>9</td><td>10</td><td>-</td></tr><tr><td>Business Locations</td><td>reg</td><td>16,821</td><td>5</td><td>-</td><td>16,438</td></tr><tr><td>Dissolved Municipalities</td><td>reg</td><td>13,462</td><td>7</td><td>-</td><td>1,656</td></tr><tr><td>Geopolitical Regions</td><td>reg</td><td>1,114</td><td>7</td><td>-</td><td>1,066</td></tr><tr><td>Historical Figures</td><td>reg</td><td>11,260</td><td>12</td><td>-</td><td>2,134</td></tr><tr><td>Municipal District Capital</td><td>reg</td><td>1,658</td><td>6</td><td>-</td><td>1,267</td></tr><tr><td>Poets</td><td>reg</td><td>60,240</td><td>11</td><td>-</td><td>21,564</td></tr><tr><td>Territorial Entities</td><td>reg</td><td>36,717</td><td>8</td><td>-</td><td>34,189</td></tr><tr><td>WWI Personnel</td><td>reg</td><td>30,675</td><td>12</td><td>-</td><td>16,227</td></tr><tr><td>Artworks Inventory</td><td>reg</td><td>10,635</td><td>6</td><td>-</td><td>-</td></tr><tr><td>Drawings Catalog</td><td>reg</td><td>63,130</td><td>9</td><td>-</td><td>-</td></tr><tr><td>Eclipsing Binary Stars</td><td>reg</td><td>297,934</td><td>7</td><td>-</td><td>-</td></tr><tr><td>Registered Ships</td><td>reg</td><td>4,644</td><td>7</td><td>-</td><td>-</td></tr><tr><td>Research Articles</td><td>reg</td><td>6,962</td><td>7</td><td>-</td><td>-</td></tr><tr><td>Research Article Citations</td><td>reg</td><td>4,115</td><td>10</td><td>-</td><td>-</td></tr><tr><td>Ukrainian Villages</td><td>reg</td><td>21,355</td><td>4</td><td>-</td><td>-</td></tr></tbody></table>
<table><caption>Table 10: Average runtimes (in seconds) for embedding extraction and ridge fitting, for varying train set sizes.</caption><thead><tr><th rowspan="2">Models</th><th colspan="3">Train-size</th></tr><tr><th>64</th><th>256</th><th>1 024</th></tr></thead><tbody><tr><td>TabuLa-8B</td><td>124 ± 141</td><td>145 ± 166</td><td>216 ± 258</td></tr><tr><td>Llama-3.1-8B</td><td>119 ± 133</td><td>140 ± 157</td><td>209 ± 247</td></tr><tr><td>Llama-3.2-3B</td><td>43 ± 51</td><td>51 ± 60</td><td>76 ± 93</td></tr><tr><td>Llama-3.2-1B</td><td>18 ± 20</td><td>21 ± 24</td><td>32 ± 37</td></tr><tr><td>Qwen3-8B</td><td>120 ± 144</td><td>140 ± 169</td><td>210 ± 262</td></tr><tr><td>Qwen3-4B</td><td>65 ± 82</td><td>76 ± 96</td><td>114 ± 150</td></tr><tr><td>Qwen3-0.6B</td><td>12 ± 13</td><td>14 ± 15</td><td>21 ± 22</td></tr><tr><td>Knowledge-card</td><td>25 ± 29</td><td>30 ± 34</td><td>45 ± 53</td></tr><tr><td>OPT-1.3B</td><td>23 ± 29</td><td>27 ± 34</td><td>40 ± 53</td></tr><tr><td>ERNIE-large</td><td>8 ± 6</td><td>10 ± 7</td><td>15 ± 9</td></tr><tr><td>RoBERTa-large</td><td>8 ± 6</td><td>10 ± 7</td><td>14 ± 9</td></tr><tr><td>ERNIE-base</td><td>5 ± 4</td><td>6 ± 4</td><td>8 ± 6</td></tr><tr><td>RoBERTa-base</td><td>4 ± 3</td><td>5 ± 3</td><td>7 ± 4</td></tr><tr><td>KGT5</td><td>5 ± 3</td><td>6 ± 4</td><td>8 ± 5</td></tr><tr><td>T5-base</td><td>5 ± 6</td><td>7 ± 7</td><td>9 ± 11</td></tr><tr><td>KGT5-small</td><td>3 ± 3</td><td>4 ± 3</td><td>6 ± 4</td></tr><tr><td>T5-small</td><td>4 ± 3</td><td>4 ± 3</td><td>6 ± 4</td></tr><tr><td>TARTE</td><td>4 ± 4</td><td>5 ± 5</td><td>8 ± 6</td></tr><tr><td>FastText</td><td>2 ± 4</td><td>3 ± 4</td><td>4 ± 6</td></tr><tr><td>Non-pretrained</td><td>0.5 ± 0.7</td><td>1 ± 1</td><td>2 ± 2</td></tr></tbody></table>
<figure><img src="image_7.png" /></figure>

Average rank measured at *n* = 1024

a. Normalized performance gain for Ridge over non-pretrained representations

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

Average rank measured at *n* = 1024

b. Normalized performance gain for Ridge over non-pretrained representations

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

Average rank measured at *n* = 1024

c. Normalized performance gain for Ridge over non-pretrained representations

<caption>Figure 6: Relative improvements to non-pretrained string representations, when using a ridge model as a tabular learner. For each source, larger models consistently yield better performances: **a. TextTabBench** **b. CARTE** **c. WikiDBs.**</caption>