Large Scale Transfer Learning for Tabular Data  via Language Modeling

Authors: josh  gardner
juan c perdomo
ludwig  schmidt
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 21:36:18

Abstract: Tabular data – structured, heterogeneous, spreadsheet-style data with rows and columns – is widely used in practice across many domains. However, while recent foundation models have reduced the need for developing task-specific datasets and predictors in domains such as language modeling and computer vision, this transfer learning paradigm has not had similar impact in the tabular domain. In this work, we seek to narrow this gap and present TABULA-8B, a language model for tabular prediction. We define a process for extracting a large, high-quality training dataset from the TabLib corpus, proposing methods for tabular data filtering and quality control. Using the resulting dataset, which comprises over 2.1B rows from 4.2M unique tables, we fine-tune a Llama 3-8B large language model (LLM) for tabular data prediction (classification and binned regression) using a novel packing and attention scheme for tabular prediction. Through evaluation across a test suite of 329 datasets, we find that TABULA-8B has zero-shot accuracy on unseen tables that is over 15 percentage points (pp) higher than random guessing, a feat that is not possible with existing state-of-the-art tabular prediction models (e.g. XGBoost, TabPFN). In the few-shot setting (1-32 shots), without any fine-tuning on the target datasets, TABULA-8B is 5-15 pp more accurate than XGBoost and TabPFN models that are explicitly trained on equal, or even up to 16× more data. We release our model, code, and data along with the publication of this paper.

Main paper: 
 our contributions: 
 This paper has the following three main contributions:TABULA-8B, a Tabular Prediction Model: We build TABULA-8B (Tabular Llama 3 -8B), a model for prediction on tabular data. On an evaluation suite consisting of 329 tables drawn from five tabular benchmarks, TABULA-8B has zero-shot accuracy 17 percentage points (pp) above random guessing. In the few-shot setting (1-32 examples) , TABULA-8B is 5-15 pp more accurate than state-of-the-art methods (XGboost, TabPFN, CatBoost) that are trained on equal number of shots, and these methods require 2-8× more data to achieve the performance of our model. TABULA-8B outperforms a variety of strong tabular baselines and even commercial LLMs such as Claude Instant and Claude 3 Sonnet.T4 -A Large Scale, High Quality Training Dataset: We build and release The Tremendous TabLib Trawl (T4), a filtered collection of 4.2M unique tables (consisting of over 2.1B rows, a total of 100B tokens) from TabLib [13] . We detail the recipe used to construct T4, including a suite of methods for filtering web-scale tabular data at several levels (table, row, column), removing unwanted information such as PII and code, and selecting unsupervised prediction targets from these tables.Open-Source Release: As part of our publication, we release all relevant infrastructure (code, models, and data) with the hopes that the community will build on our work. We provide high-quality, efficient implementations of data pre-processing and model training pipelines, including our new row-causal tabular masking (RCTM) attention and packing scheme for training on tabular data. We also share the code used to filter T4 from TabLib, enabling future work that extends our dataset construction methodology.Our work is concerned with prediction models for tabular data. We define both below.Tabular Data: For our purposes, tabular data has three main properties. (i) Structured: It consists of elements with a "key-value" structure, often represented as a table with keys (or "headers") representing column names, and rows that consist of values. (ii) Heterogeneous: The values are of mixed types, including numeric, boolean, categorical, ordinal, text, date/time, etc. Missing values may be present. (iii) Exchangeable: The ordering of rows and columns in the dataset is arbitrary. In particular, any permutation of the rows, or columns, still represents the same tabular dataset.Prediction Task Definition: The main focus of this work is prediction on tabular data. In tabular prediction, the goal is to predict the value y of a specific target column for a row in a dataset using the key-value pairs x from all other columns. More specifically, we focus on classification tasks where values y for the target column belong to a finite set C. Binned regression tasks, in which a real-valued y is discretized into a finite set of numeric values (as in [59] ) also fit this definition. 
related work: 
 Our work builds on a line of foundation modeling, tabular data prediction, and natural language processing research. Given space constraints, here we focus on the most closely related literature.Transfer Learning and Foundation Models: The idea of building general purpose models via autoregressive next-token prediction on large scale datasets was pioneered in a series of papers in natural language processing [6, 11, 36, 40, 57] . These results have since led to the development of foundation models capable of solving diverse tasks in other modalities including vision [38, 63] , audio [39, 66] , code [22, 42] , time series [10, 18, 21] , and graphs [62] , as well as multi-modal models [53, 55] . Our work also build upon on the demonstrated capacity of transformers to perform few-shot or in context-learning [6, 17] , which entails making predictions on examples from a previously unseen dataset, given only a few labeled examples from that task.Large-Scale Dataset Curation: The construction of large, high-quality datasets has emerged as one of the most critical, and challenging, issues in the development of transferable models. Several major milestones in this space [6, 40, 41, [53] [54] [55] [56] stand out in their effort spent curating and cleaning web-scale datasets -often while using a model architecture and training recipe that only slightly differs from prior work. This has led to a number of modality-specific methods for large-scale dataset curation; for example, the use of heuristics [41] and model-based quality scoring to select high-quality text data [6, 9, 56] ; methods for selecting aligned audio-transcript pairs for speech [39] ; or the use of CLIP scores to filter for aligned image-text pairs [46] . However, to the best of our knowledge no prior work has developed analogous methods for tabular data. This lack of large-scale training data has been a critical bottleneck toward the development of tabular foundation models.Models for Tabular Prediction: Despite the fact that deep learning methods are now the norm in domains such as computer vision or NLP, methods based on gradient-boosted decision trees (GBDTs) [5, 7, 37] continue to be at or near state-of-the-art in tabular prediction [20] . Drawing upon recent breakthroughs in other modalities, the field has now developed deep learning-inspired approaches [19, 52] that are competitive with tree-based models in-distribution, where models are trained and evaluated on the same data, but the benefits of such approaches relative to GBDTs appears to be limited in practice [20, 33] . In particular, [25] introduces TabPFN, a transformer model for tabular data that outperforms XGBoost in certain regimes [33] and is capable of making predictions on unseen datasets (with some constraints on dataset size and label space; see D.2). A related recent work, CARTE [30] , explores the use of graph-based architectures for tabular transfer, based on key-value encodings and pretrained on a large knowledge graph.Several recent works [12, 23, 59, 66] have explored fine-tuning LLMs on individual tables, or on small collections of tables (< 200). The main idea in this closely related line of work is to reduce classification to next-token prediction by first serializing a row as text (see Figure 2b for an illustration) and then training an LLM to predict the serialized labels. These studies demonstrate that this LLM approach is often competitive with trees or tabular deep learning methods in-distribution [12, 23, 59 ]. However, in cases where models were evaluated out-of-distribution, they were less accurate than SOTA methods trained on these held-out tables [65] . Our work builds on this promising line of work. Relative to these prior efforts, we specifically address the (1) lack of large-scale and training data; and (2) the inability of exiting methods to be competitive when evaluated out-of-distribution. 
tabula-8b -model design and training: 
 Our overall approach is to fine-tune the pretrained Llama 3-8B language model [54] on tabular prediction tasks using a new web-scale corpus, T4. We use Llama 3-8B as our starting point since it is a high-quality, open-source model trained on over 15T tokens that demonstrates strong performance on a diverse set of downstream tasks [54] , particularly at its relatively modest size (which makes fine-tuning, inference, and deployment more accessible).Serialization and Tabular Language Models: As discussed previously, our methodology extends ideas pioneered in previous work [12, 23, 59, 65] demonstrating how LLMs can be trained to perform tabular prediction tasks by serializing rows as text, converting the text to tokens, and then using the same loss function and optimization routines used in language modeling. Serialization refers to the procedure of converting a row of data into text, for instance by concatenating substrings of the form "the <key> is <value>". Prior works investigated the impact of different serialization formats [ 12, 23, 51] , demonstrating that performance is largely insensitive to the exact mapping (e.g. using "{ <key>: <value> }") and other strategies do not improve upon this "the <key> is <value>" structure.We adopt a similar serialization strategy, illustrated in Figure 2b . Given a row of data from a table, the corresponding serialization has three main parts: (i) a prefix containing a prompt (always "Predict the value of <target column name>") followed by a list of possible label values ("val1 || . . . || valNumClasses ||), (ii) the example consisting of all key, value pairs for the columns used as features, and (iii) a suffix prompting the model with a question ("What is the value of <target column name>?") again followed by the possible labels. For multiple-shot samples, we concatenate their serializations. We introduce three special tokens into the Llama 3 vocabulary to ensure these sequences are properly tokenized: ||, to delimit answer choices; 
 to denote the end of an input sequence (the last token before the targets or model generation begin); and 
 to indicate the end of a completion.Training Procedure: We train TABULA-8B using a standard language modeling setup where the model is trained to minimize the cross-entropy over the sequence of target tokens. We only compute loss over the subsequence of target tokens: the tokens starting after the 
 token, up to and including 
. This objective focuses training on learning the desired target label, as in [12, 23, 59] , rather than developing a broader generative model of tabular data as in [65] .Relative to prior studies on tabular prediction with LLMs, our work has one main methodological innovation. We introduce an efficient attention masking scheme, row-causal tabular masking (RCTM), tailored to few-shot tabular prediction whereby the model is allowed to attend to all previous samples from the same table in a batch, but not to samples from other tables (this is sometimes referred to as "cross-contamination" in the language modeling literature [31] ). However, by appropriately masking out values, RCTM also enables packing examples into the same batch (as effectively zero padding is required during training despite the large variance in the size of each tokenized table or row), thereby increasing model throughput (see Figure 2a ). Taken together, these have the effect of training the model to use multiple "shots" during training and mitigates the potential loss of few-shot learning capabilities that has been observed to occur during fine-tuning [29, 61, 64] .The RCTM masking structure is shown in Figure 2a . Lower-triangular blocks correspond to rows from the same table that are present in the batch. This is similar to the "in-context pretraining" method from [48] , except that (i) our procedure encourages the model to aggregate information across multiple rows of a given table, rather than attending across documents, and (ii) our procedure only trains the model to predict the target tokens, not the input features. We show that RCTM has a drastic impact on few-shot performance through an ablation experiment (see Section F.1).Training Details: The final model is trained for 40k steps with a global batch size of 24 (with sample packing, this is roughly equivalent to a global batch size of 600 rows of tabular data). The modelFigure 3 : Sketch of dataset generation pipeline. 627M tables from TabLib [13] are filtered by applying rules at the table, row, and column level. Then, for each table, we identify valid and high-quality prediction targets in an unsupervised manner and use the results for training TABULA-8B. sees roughly 8B tokens during training; we note that is less than 10% of the 100Btokens in T4, and less than one one thousandth of TabLib itself. We fully fine-tune all model parameters, as opposed to parameter-efficient fine-tuning, since full-fine tuning consistently benefits from scale [24, 64] . Reproducibility details are given in Appendix B. 
dataset construction: building the tremendous tablib trawl (t4): 
 Beginning from a web-scale corpus of raw data (TabLib), we apply various filters to produce a high-quality subset, and transform the results into a set of prediction tasks for training. As the result of this procedure, we produce T4 (The Tremendous Tablib Trawl), which we release with this paper.Original Raw Data Source: TabLib [13] is a publicly-available dataset consisting of 627M tables extracted from two main sources: Common Crawl and Github (see [13] for more details). Due to its scale and diversity, TabLib presents a unique opportunity for training foundation-scale models on the tabular data. However, like other web-scale datasets, the vast majority of its contents are low quality and not suitable for training. For instance, TabLib contains numerous system logs with inscrutable statistics, tables of software documentation, and call sheets with personally identifiable information (PII). To the best of our knowledge, no previous work has addressed the task of filtering TabLib into a usable training set, and no publicly-available models have been trained on this corpus.Filtering Strategies: Filtering large collections of raw data to extract a higher-quality subset is an essential component in the development of foundation models [e.g. 41], yet to date, no previous work has addressed this core issue for tabular data. Filtering a web-scale dataset like TabLib into a usable subset of high-quality tables is critical to leverage its diversity and scale, but also raises unique challenges specific to tabular data, such as missing data, web "content" that is formatted as HTML tables that does not satisfy our definition of tabular data, and PII. To turn TabLib into a usable training set, we develop a set of filtering methods to identify high-quality tables for prediction. Conceptually, our filtering occurs at three levels, each applied sequentially: tables (entire tables are removed from the pool), columns (individual columns are removed from a table), and rows (rows are removed from a table).Similar to previous approaches [38, 40, 41, 54, 56] , we use a mix of heuristics and rule-based methods to remove low-quality sources from our pool. We present the full list of our filtering rules in Section A. At a high level, our emphasis across all filtering strategies is to: (1) remove non-tabular data (such as text or PDFs incorrectly identified as tabular data during TabLib's collection), (2) ensure the safety of chosen tables (e.g. by removing PII), and (3) find sources with high semantic content (e.g. by removing tables with too many missing values). As part of this filtering process we develop and apply simple methods for deduplication, English language filtering, filtering for missing data, PII removal, code removal, and more.Unsupervised Task Selection: As described in Section 1.2, we focus on methods for tabular prediction: predicting the value of a target column given the values of all other columns for an instance. Therefore, as part our data pipeline we develop new methods for selecting, in an unsupervised fashion, which column is the target column for each table in the corpus. Selecting targets of prediction for tabular data at scale is an under-explored problem. Prior work in this space operated on at most a few hundred tables and used either a combination of expensive queries to commercial LLMs or manual curation to identify tabular prediction targets [59, 65] . However, when operating on hundreds of millions of distinct tables with potentially no associated metadata, these strategies are not feasible.For each table, we select a prediction target programmatically by first identifying a subset of columns that are suitable for prediction according to various heuristics, and then choosing a specific column at random from this set. The exact list of heuristics to arrive at this set is presented in Appendix A. Amongst others, these include excluding candidate columns if: the column name is numeric, it has only one unique value, or it has unique values for every row (excluding numeric columns).Final T4 Dataset Summary: Running this entire filtering process (from raw data to serialized examples ready for training) on all 70TB of TabLib using our open-sourced implementation takes about 4 hours on a CPU cluster. It yields a total of 4.2M tables, which equates to a table filtering rate of approximately 97.91%. Additional descriptive statistics for the dataset are given in Appendix A.3. The resulting dataset contains over 2.1B rows (approximately 100B Llama 3 tokens) for training of the downstream model, and occupies roughly 2TB compressed on disk. We note that 100B tokens is larger than the total number of tokens TABULA-8B sees during training. Therefore, the model sees each distinct table at most once during training, and our pipeline could be scaled up to support larger models or longer training runs. 
evaluation methodology: 
 We evaluate the transfer learning performance of TABULA-8B on a diverse set of established benchmarks previously considered in prior work (see Section 5.2 for a list). For each dataset, we use the predefined prediction target from the original benchmark. Due to computational constraints, we evaluate TABULA-8B on up to 128 test examples for each dataset and number of shots k.The term "few-shot" is unfortunately overloaded. It is used both to refer to models that make predictions on instances never seen during training, and to models that directly train on these examples before predicting on unseen samples. We do not fine-tune our model on test examples, in contrast to [23, 59] . Our methodology only requires performing forward passes through the network to generate predictions and avoids the need for computationally-expensive gradient updates. In zero-shot evaluations, given a row of a dataset along with the corresponding set of columns and possible labels, we first serialize the row into the same format used during training, and feed it into the model to generate a prediction following the 
 token. For few-shot evaluations, we perform the same procedure, except that we preprend the serialized "shots" as in Figure 2b .In contrast to methods like XGBoost that directly predict likelihoods of a set of labels, language models output likelihoods over a set of tokens (128k in the case of Llama 3). For each evaluation dataset, the values in the label set (e.g. "sun, rain, snow" in Figure 2b ) can consist of a sequence of many individual tokens from this large vocabulary. Here, we use open-vocabulary (or "openended") accuracy [1, 8] as the main evaluation metric for our model. In this setup, once the model is prompted with a serialized example, it is allowed to generate an arbitrary sequence of tokens. Once it produces the 
 token, the generated text is then directly compared to the correct completion. Only an exact match, including the terminating 
 token, is counted as accurate. This is more challenging than closed-vocabulary evaluation, where the model is only rated on assigning the highest probability to the correct completion from a predetermined set.We evaluate our model's predictive performance across a collection of 329 publicly-available tabular datasets drawn from five tabular benchmarks (see Appendix H for a full list). These include:UniPredict Benchmark (169 datasets) [59] : We use the "supervised" subset of 169 datasets from the recently-introduced UniPredict benchmark. These are high-quality tabular datasets with generally informative column names and a mix of both categorical and continuous targets, drawn directly from Kaggle. While the model introduced in Wang et al. [59] was trained and tested on separate splits of these datasets, we only use them for testing. We make corrections to several datasets with targets erroneously treated as categorical in the original benchmark, described in Section 5.2.Grinsztajn Benchmark (45 datasets) [20] : The Grinsztajn benchmark is a curated suite of datasets consisting of numeric and categorical features. This dataset is notable in that the original study by Grinsztajn et al. found that gradient boosted decision trees (GBDTs) consistently outperformed deep learning-based methods on these tasks.AutoML Multimodal Benchmark (AMLB) (8 datasets) [49] : A suite of tables which include one or more free-text fields (such as an Airbnb description, or a product review). The benchmark is considered challenging for tree-based methods due to the non-standard text-based features. However, it also poses a challenge for LLMs since some columns can contain highly variable lengths of text.OpenML CC-18 Benchmark (72 datasets) [2] : The OpenML Curated Classification Benchmark was created by applying filtering rules to extract a high-quality subset from the OpenML platform. The rules include: no artificial data sets, no subsets of larger data sets nor binarizations of other data sets, no data sets which are perfectly predictable by using a single feature or a simple decision tree.OpenML CTR-23 Benchmark (35 datasets) [14] : The OpenML Curated Tabular Regression (CTR) Benchmark is a curated set of tables for regression drawn from OpenML. The curation process is similar to that of OpenML-CC18, for regression tasks. We note that, being primarily intended for the evaluation of AutoML methods, the OpenML benchmarks are notable for lacking informative column names (i.e. names such as "Var1, Var2, . . . " are common in OpenML benchmark datasets).We transform all regression tasks into a 4-class classification based on quartiles, as in [59] (see Appendix A.2 for details). Many datasets contain rows with missing data. We leave these as-is and do not remove any data. On a computational note, some datasets contain a large numbers of features and performing k-shot evaluations on these datasets at large k can exceed the model's context window. Therefore, in few-shot evaluations, we always report results for the subset of datasets where k shots fit into the model's context window, for the entire specified range of k. We provide more details on the evaluation datasets in Appendix D.1 and report per-dataset results in Appendix H.When inspecting TABULA-8B's performance, we compare against the following baselines: Llama 3-8B [54] : This is the base model from which TABULA-8B is fine-tuned. Comparing to the base model isolates the effects of the fine-tuning process. It also controls for any contamination of evaluation datasets that may be contained in pretraining data for Llama 3 (the exact training data for Llama 3 are not currently disclosed). We return to this point in Section 5.6.XGBoost [7] : XGBoost is a supervised learning gradient-boosted decision tree (GBDT) method. It is widely considered to be highly competitive in tabular prediction tasks [15, 20, 33] .TabPFN [25] : This a transformed-based hypernetwork pretrained to reflect a set of inductive biases germane to tabular data. TabPFN is thus considered especially effective for few-shot learning [25, 33] .Whenever possible, we perform hyperparameter tuning on XGBoost and TabPFN in order to maximize their performance. See Appendix D.2 for further details on baseline implementation and tuning. We also provide results comparing to additional supervised baseline models, and to commercial LLMs, in Section E.2.We present our main experiments evaluating the transfer learning ability of TABULA-8B in Figure 4 . As a whole, TABULA-8B demonstrates strong transfer performance across the broad range of tasks.In the zero-shot regime (seen in the left-most point for each plot in Figure 4 ) -where the model is presented with no further information about the target dataset except for the serialized key-value pairs and set of possible labels for a single row -TABULA-8B is between 5 to 25 pp more accurate than a random baseline and 50 pp more accurate than the base Llama 3 model. This illustrates one of the key benefits of using language models for tabular prediction: after fine-tuning, TABULA-8B can leverage semantic information contained in the serialized data to make high-quality predictions.While XGboost and TabPFN are not capable of zero-shot prediction, this behavior has been observed in the original Llama 3 model [54, 57] . However, in our evaluations, Llama 3 performs below random guessing in the zero-shot setting. We hypothesize that the base Llama 3 model requires a small number of samples to understand the input-output format and task (as indicated by the large leap in Llama 3 performance from 0 → 1 shot).In the few-shot setting, where each method additionally sees a small number of labeled examples, TABULA-8B's performance steadily improves with the number of shots. In the regime of 1 to 32 The final plot (lower right) shows curves separately over decontaminated vs. potentially-contaminated evaluation tasks (see Section 5.6); we find no impact on our overall findings due to contamination (and performance on tasks which may be in our training set is lower on average, across all models).shots, it outperforms state-of-the-art models (XGBoost and TabPFN) that are directly trained (and hyperparameter tuned) on each specific dataset by 5-20pp. Once we evaluate performance on 32, 64, or 128 shots (see Figure 7b ), this gap begins to diminish, but the number of datasets that can fit > 32 shots into the 8192-token context window is both small and a relatively biased sample (due to their small number of features). TABULA-8B is consistently 10 to 20pp above the Llama 3-8B base model for the full range of shots, highlighting the benefits of our training procedure on T4.Improvements in Sample Efficiency: As discussed previously, the main benefit of transferable models is that they reduce the amount of data necessary to achieve good performance on new tasks. For instance, as seen in Figure 4 , TABULA-8B only needs one shot to achieve 60% average accuracy on UniPredict tasks. However, both TabPFN and XGBoost only reaches 60% accuracy after 16 shots. Therefore, TABULA-8B reduces the amount of data necessary to achieve 60% accuracy by 16 fold relative to XGBoost annd TabPFN. We refer to this statistic as the relative sample efficiency (see D.3). TABULA-8B in general achieves higher accuracy than the benchmarks using less data. Hence, the relative sample efficiency is always > 1 (the exact ratio varies across benchmarks).Impact of Informative Column Headers: As shown in Figure 4 , while TABULA-8B generally has higher accuracy than the baselines, this accuracy gap varies across benchmarks and the number of shots. For instance, for the UniPredict benchmark -which was specifically constructed to include datasets with semantically-meaningful column headers [59] -the gap to supervised baselines is much larger than in the OpenML benchmarks, which tend to have less semantically-meaningful column names. If meaningful column headers are absent, the model still performs well (matching or outperforming XGBoost at shots k ≤ 8), but its advantage over these strong baselines is lessened. We investigate this effect in further detail with a controlled experiment in Section F.2.Robustness to Column Ordering. Apart from evaluating TABULA-8B's transfer learning ability, we also investigate its robustness and the degree to which performance is affected by the order in which columns are presented (serialized); this order invariance is cited as a necessary attribute of tabular foundation models in [58] . We present these experiments in Appendix F.4. Our results demonstrate that changing column order does not alter performance in a statistically significant way, but that there may be a small (∼ 1pp) drop which we hypothesize is due to manual ordering of tabular columns in certain benchmark datasets which sometimes reflects a more "natural" ordering.Robustness to Feature Dropout. Language models may be uniquely susceptible to small changes in the downstream data; for example, the removal of specific features may affect language models' prediction performance more than traditional supervised methods. We conduct an ablation study to assess the behavior of TABULA-8B as columns are removed from the test data. We assess removal both in order of descending and ascending importance. The results of these experiments, in Appendix F.3, demonstrate that TABULA-8B's performance declines at a similar rate to an XGBoost model trained directly on the subset of features.Robustness to Column Header Removal. Another potential risk of tabular language models is that, while these models are able to utilize the semantic information in column names, the model may also be overly reliant on the presence of informative column names. In Appendix F.2 we conduct an ablation study to assess this. The results in Appendix F.2 demonstrate that there is a small decline in performance when column headers are removed (replaced with uninformative headers), but that TABULA-8B still outperforms baselines across all numbers of shots. We believe that this drop in performance is commensurate to the loss in information when column headers are eliminated.Importance of Row-Causal Tabular Mask. We evaluate the impact of the attention masking scheme introduced and described in Section 3. We conduct an ablation study, replacing this component of the model with a sample-wise causal attention (the same form of attention used during standard language model training, where attention across documents is prevented). Our results, detailed in Appendix F.1 and Figure 11 , illustrate that this modification is central to the few-shot learning capabilities of TABULA-8B: when our mechanism is replaced with sample-wise attention the resulting model does not demonstrate few-shot learning capacity, and its performance degrades for k ≥ 16 (see Figure 11 ).Influence of the Base LLM. We conduct an ablation study of the base LLM to evaluate how TABULA-8B improves as the base LLM improves. In particular, we rerun our main training pipeline described in Section 3, but using LLama 1 and 2 as the initial language model instead of LLama 3. These results, provided in Section F.6, demonstrate that TABULA-8B improves along with the performance of the underlying base model. Taken together, these results highlight how the primary contribution of the paper is not the specific model we produce, as much as it is the methodology we present for generating tabular predictors from base language models. As LLMs continue to improve, so will the tabular models that are produced by applying our training methodology to new LLMs.Given that T4 consists of 4.2Mtables sourced from public data sources (Common Crawl, Github) and that our evaluations are also comprised of public benchmarks, we investigate the extent and possible impact of data contamination -that is, training datasets that are part of the evaluation suite.In Section G, we explain our methodology to test for the potential presence of benchmark datasets in T4. Using a conservative identification strategy based on column matching (likely to include false positives). We find at most one-third of benchmark tables may occur at least once in the training set. When training large-scale models for transfer learning, it is not always clear a priori what the eventual application domains will be. Therefore, we believe that it is an important research question to understand the extent to which contamination may affect performance, as contamination may be difficult to prevent in some cases. Initial foundation modeling efforts in non-tabular domains adopted a similar approach, and found mixed or no impact from overlap [38, 40] .We evaluate the impact of contamination in our experimental setup by evaluating TABULA-8B separately on "potentially contaminated" vs. uncontaminated tables. Our results are shown in the bottom right plot of Figure 4 , as well as in Figures 17 and 18 . Summarizing, we find no clear evidence that contamination affects model performance on the test suite, or that transfer ability is affected by contamination. In fact, as seen in Figure 4 , the gap between TABULA-8B and XGBoost is in fact larger if we restrict evaluation to the benchmark tables which we verify are not in T4. In addition to verifying that our results continue to hold over a diverse set of tables which we know the model did not see in training, it also shows that having some amount of potential contamination did not upwardly bias our estimate of TABULA-8B's transfer learning ability. We hypothesize that the observed gap in Figure 4 is due to our conservative duplication procedure being more likely to flag datasets with generic or common column names, which also leads to worse baseline performance on these tasks.We present more comprehensive investigation on the effects of contamination in Appendix G. 
discussion: 
 Limitations: TABULA-8B has several limitations. First, TABULA-8B has a limited context window of 8192 tokens. This restricts the number of examples that can be utilized for few-shot learning, as well as the additional information (such as text context or extended feature descriptions) that are available to the model. We expect that this limitation will be eased as the availability of longer-context models grows [e.g. 53]. Second, TABULA-8B has 8B parameters, which makes serving and inference expensive and limits the environments it may be deployed in. Lastly, given that it uses a pretrained LLM as a base-model and fine tunes on a web-scale corpora of historical datasets that likely contain various social biases, TABULA-8B introduces new potential fairness considerations that are not present when using preexisting supervised methods such as XGBoost. We hope that by open sourcing the model, data, and code, we might enable future research addressing these important directions.Future Work: Our work on transfer learning for tabular data is the first its kind at this scale, and there are several avenues for future research. These can be coarsely categorized as either improvements (of the existing dataset and model) or extensions (deeper investigations into the model and data itself).On the improvements side, we see several promising directions. These include improvements in tabular data filtering (this has been the main axis of improvement in recent generations of language models); scaling the model + data + compute; exploring the use of inference-time strategies to improve prediction (such as self-consistency [60] , prompt ensembling [1, 38] , or in-context example selection [43] ); and introducing extra information during both training and inference, such as contextual information or samples from different, but related, tables.On the extensions side, we hope that our work opens avenues toward deeper understanding of tabular foundation models including: understanding potential biases or unwanted behavior with respect to sensitive features (features such as race, age, and gender are common in tabular datasets); using tabular foundation models to address small-sample problems which might be aided by a high-quality pretrained model (such as in the Fragile Families Challenge [44] ); and extending this approach to new tasks beyond prediction, such as data generation, explanation, data wrangling, and more.7 Accessing Open-Source Code, Data, and Model Weights TabLib Preprocessing Code: A Python module for filtering TabLib, tabliblib, along with scripts and configurations used to perform the filtering, are available at https://github.com/ mlfoundations/tabliblib.We provide rtfm, a Python module used to train TABULA-8B, perform inference and evaluation, and process data, at https://github.com/mlfoundations/ rtfm.T4 Dataset: The T4 dataset is available via public credentialized access on Hugging Face datasets at https://huggingface.co/datasets/mlfoundations/t4-full. Because the dataset is derived from TabLib, users must first obtain permission to access TabLib at https://huggingface. co/datasets/approximatelabs/tablib-v1-full.Evaluation Datasets: The full evaluation suite used to evaluate TABULA-8B is available via Hugging Face Datasets at https://huggingface.co/datasets/mlfoundations/ tabula-8b-eval-suite. Each dataset includes: a CSV file containing the raw data; a TableShift [16] FeatureList JSON object; and a YAML file with associated metadata.Model Weights: TABULA-8B weights are available via Hugging Face at https://huggingface. co/mlfoundations/tabula-8b.As discussed previously, in order to generate T4, we filter TabLib [13] at the table, column and row level. Having filtered down the tables, we then programatically select a target column for prediction according to another set of heuristics. The remaining columns are used as features, but not as prediction targets. We select only a single target column for each table.The following tables describe the entire set of heuristics we use as filters. A precise implementation may be found as a part of our software release associated with this paper. Our pipeline involves a language identification step; for language detection, we make use of the fasttext library [27] . 5Level Name Description Motivation / Hypothesis The TabLib process used to scrape the tables can result in tables with "headers" that are actually just rows of data. One indicator of this is very long headers (e.g. a text column that ends up as a header).Drop columns with names that are numeric.TabLib's parsing removed tables with all-numeric headers "for most file formats". This means that (a) some formats were missed, and (b) tables with many numeric headers and even one non-numeric header were still included. Our procedure for target column selection is based on the rules described in the above tables. Given a table, we consider all columns to be potentially valid targets unless they do not satisfy one of the listed criteria.Once target candidates are identified for a table, we choose a single candidate at random and use this as the prediction target. When there are both continuous and categorical candidates present, we choose a categorical candidate with probability p = 0.9 and a continuous candidate with probability 1 -p. This decision reflects our qualitative observation that our selection method tends to produce higher-quality categorical columns than numeric columns, and has the effect of showing the model classification tasks more often than (binned) regression tasks during training.In the case where we select a continuous target, we discretize it into a discrete set by selecting a number of quantiles uniformly at random over the interval [3, 8] , and then discretizing the target value into these columns. We serialize the resulting quantiles as "less than 1," "between 1 and 2.5", "greater than 2.5" etc.Even after filtering, the tables in our pool contain columns which may not be meaningful prediction targets (for example, timestamps or UIDs). For a given table, we propose and apply a series of heuristics for identifying suitable candidates for target columns. These heuristics include excluding columns if: the name is numeric; only one unique value; has unique values for every row (excluding numeric columns); any row has a value longer than 256 characters; the column is of a date or timestamp type. We provide a detailed list of the exact target selection rules in Section A.2, and an implementation in our code. We do not drop such columns from the table; columns not meeting these criteria are simply kept as predictors but will never be used as prediction targets.Once target candidates are identified for a table, we choose a single candidate at random and use this as the prediction target. When there are both continuous and categorical candidates present, we choose a categorical candidate with probability p = 0.9 and a continuous candidate with probability 1 -p. This decision reflects our qualitative observation that our selection method tends to produce higher-quality categorical columns than numeric columns, and has the effect of showing the model classification tasks more often than (binned) regression tasks during training.This section provides some basic descriptive statistics of the final dataset, shown in Figures 5 and 6 . Figure 5a shows that T4 represents a wide variety of data types across its tables, but that data are primarily represented as float, int, and object (string/categorical) data types. Figure 5b shows that, while most tables in T4 have little to no missing data, tables can have as much as 10% of data missing (a maximum threshold enforced by heuristics described above).Figure 6 provides a sense of the "shape" of tables in T4, showing that roughly 30% of tables have 64 columns, and roughly 30% have 1, 000 columns, the minimum and maximum number of rows set by our heuristic filters. In contrast, nearly all tables have fewer than 50 columns, with only a very small fraction (< 0.01%) having 500 or more columns (in practice, rows from tables with 500 columns would almost never fit inside the context window of T4 after serialization and tokenization).Our data processing implementation uses Ray Datasets6 to process tables in parallel. Our pipeline utilizes TabLib's existing content_hash feature to read only a set of unique tables; this avoids performing an expensive deduplication step during online processing. The shards of TabLib are processed in chunks to avoid extra overhead due to very large collections in Ray. Our pipeline includes certain additional optimizations: for example, in TabLib, data is stored as serialized Arrow bytes (and the deserialized bytes cannot easily be passed between stages of the Ray pipeline); as a result, our pipeline avoids repeatedly deserializing these bytes and only does so at a single filtering step, and at the write step.Our preprocessing implementation is provided as a separate library, tabliblib, along with the release of this paper and in the supplementary material.TABULA-8B is trained for 40 thousand steps with a global batch size of 24. We use a peak learning rate of 1e -5 warmed up over 10% of the total training to the peak learning rate, and then use a cosine decay schedule to zero. We do not use weight decay. We found that our model was quite sensitive to the selection of the learning rate, and that evaluation loss during training correlated strongly with performance on downstream tasks.Our final training run for TABULA-8B took approximately 6 days on a single node of 8 NVIDIA 80GB A100 GPUs on a commercial cloud provider. For our TabLib filtering and XGboost experiments, we used an academic CPU cluster. Our evaluations were distributed across two academic GPU clusters consisting of NVIDIA 40GB A40 GPUs and NVIDIA A100 GPUs. As a rule of thumb, evaluating the model on a single dataset over a grid of 8 values for k (number of shots) consumes around 4 GPU-hours. We estimate that the total number of GPU hours used across training, evaluation, and development is approximately 5, 000 -10, 000.In this section we provide more detail on the benchmarks datasets in our evaluation suite. We do not preprocess the datasets (no normalization, one-hot encoding, etc), as many datasets have been filtered for specific properties, and some of the downstream methods (e.g. TabPFN) perform best when preprocessing is not applied [25] .We use the "supervised" subset of 169 datasets from the recently-introduced UniPredict [59] benchmark, obtained through correspondence with the authors. These are high-quality tabular datasets with generally informative column names and a mix of both categorical and continuous targets, drawn directly from Kaggle. The datasets are prostprocessed in [59] using a commercial LLM; however, we do not have access to the results of this postprocessing and instead use the datasets exactly as they are obtained from the Kaggle API. Note that while the model introduced in [59] was trained and tested on separate splits of these datasets, we only use them for testing.We obtain the complete set of tasks, and the corresponding target columns for each task, for UniPredict, via correspondence with the authors of UniPredict. However, we found that several tasks in the benchmark were incorrectly labeled as categorical when, in fact, these tasks were continuous (this is likely due to the use of an LLM to determine the target columns and their attributes in [59] ). We manually verify the correct target type (continuous vs. categorical) for each of the 169 datasets, and apply corrections to TODO datasets in the benchmark. The exact set of benchmarks, target columns, and target type (continuous vs. non-continuous) used in our paper are provided in the supplementary material.We note that our results are not directly comparable to the original results in UniPredict. This is for at least two reasons: (1) due to the modifications described above, where there are errors present in the original categorization of the targets (continuous vs. non-continuous) and (2) because some of the Kaggle datasets in UniPredict are continuously updated (e.g. stock datasets) and the data or access dates are not reported in [59] . We do provide our full evaluation suite, including all UniPredict datasets, in the supplementary material in order for future comparisons to our work.The Grinsztajn benchmark [20] is a curated suite of 45 datasets consisting of numeric and categorical features. This dataset is notable in that the original study by Grinsztajn et al. found that gradient boosted decision trees (GBDTs) consistently outperformed deep learning-based methods on these tasks. The benchmark is comprised of a mix of classification and regression tasks; for all regression tasks, we apply the discretization method used in UniPredict [59] and discretize the targets into quartiles.The AutoML Multimodal Benchmark [49] 7 is a suite of tables which include one or more free-text fields (such as an Airbnb description, or a product review). The benchmark is considered challenging for tree-based methods due to the non-standard text-based features. However, it also poses a challenge for LLMs since some columns can contain highly variable lengths of text.The OpenML Curated Classification Benchmark [2] was created by applying filtering rules to extract a high-quality subset from the OpenML platform. The rules include: no artificial data sets, no subsets of larger data sets nor binarizations of other data sets, no data sets which are perfectly predictable by using a single feature or a simple decision tree.The OpenML Curated Tabular Regression (CTR) Benchmark [14] is a curated set of tables for regression drawn from OpenML. The curation process is similar to that of OpenML-CC18, for regression tasks. As in [59] , we convert the continuous regression targets into a finite set of discrete labels.We remove the solar_flare task from the benchmark, as 82% of the observations have the same regression target value (0) and thus we cannot apply our quartile-transformation method.For the supervised learning baselines (XGBoost, TabPFN, CatBoost, Logistic Regression), we conduct 10 independent trials, drawing separate training sets of size k shots, for each value of k. We use the full remaining dataset as the test set.Hyperparameter tuning: For each of the 10 independent trials, we tune the hyperparameters of the model. For XGBoost, we conduct 10 iterations of hyperparameter tuning using the HyperOpt hyperparameter optimization library and the hyperparameter grid defined in [16] . For TabPFN and L2-regularized Logistic Regression, we conduct a full grid search (since there is only a single hyperparameter).We use the pretrained Llama 3 model available on Hugging Face. For this model, we do not modify the tokenizer (i.e. by adding special tokens that are used by TABULA-8B), but the serialized data format is identical to the format seen by our model during training (Figure 2b ).For every dataset and number of shots, we tune the hyperparameters according to the grid from [16] . We use 10 iterations of the adaptive hyperparameter tuning method HyperOpt on this grid with 3-fold cross-validation, whenever the number of samples is greater than or equal to 3; when the number of samples is less than 3, we use the default settings.TabPFN [25] is a hypernetwork that predicts the parameters of a neural network that can be used to classify the data. TabPFN is a pretrained Transformer model that takes a training dataset as input, and produces the parameters of a network as output; that network is then used to classify the test data. TabPFN is widely considered to be among the state-of-the-art methods for prediction on tabular data [25, 33] , and has been shown to be especially effective for few-shot learning.We use the official TabPFN implementation 8 .TabPFN has one tunable hyperparameter, the number of model predictions that are ensembled with feature and class rotations (N_ensemble_configurations in the TabPFN codebase). We sweep over all values in the range [3, 2 • d] where d is the number of features. As noted in the package documentation, when N_ensemble_configurations > 2 • d for a binary classification task, no further averaging is applied.The official implementation of TabPFN has three limitations relevant to our experiments. First, TabPFN is limited to 100 features. 9 As a result, when the number of features is greater than 100, we use TabPFN's feature subsampling, which randomly selects 100 features. Second, TabPFN cannot be trained on datasets that have more than 10 input classes. We do not report the results of experiments where TabPFN cannot be trained on at least one of the 10 random iterates of each value of k. Third, TabPFN cannot be trained on fewer than |C| examples, where C is the set of potential classes.For two classifiers f, g, let N D (f, α) and N D (g, α) denote the number of samples required for the classifiers to reach a performance level α on data D. The relative efficiency of f relative to g on dataset D at level α is equal toEQUATIONWe use the default generation settings of the Llama 3 Hugging Face model. 10 . This includes: temperature of 0.6, top-p 0.9. We do not tune these generation settings.In this section, we provide additional results for larger numbers of shots not provided in the main text.Figure 7 provides extended results for both the baseline models, and for TABULA-8B.As shown in Figure 7 , all models tend to improve with more shots. However, on the subset of datasets that can be used for 64-and 128-shot learning, we observe a narrower gap between TABULA-8B and baselines. We hypothesize that this is due to the fact that there is a selection bias: only datasets with small numbers of features and short column headers are candidates for 128-shot learning (due to the limited context window size of TABULA-8B). As a result, this biases those evaluations away from semantically-rich datasets where we expect TABULA-8B to excel.In this section, we provide comparisons to additional baselines not included in the main text. These include supervised baselines (Logistic Regression, CatBoost), and commercial LLMs (variants of Claude 11 ). Figure 7 : 7a: Results for 32-shot tasks, with extended curves for baselines. 7b, 7c: Results for 64and 128-shot tasks. All models continue to improve as k increases, but the gap between methods may narrow. However, the nature of the datasets that can be used for 64-shot learning with TABULA-8B are considerably different -fewer features, with shorter, less semantically-meaningful column names -which may downwardly bias the observed performance of TABULA-8B with large k.For the Logistic Regression and Catboost baselines, we follow the same hyperparameter tuning and evaluation procedure described in the main text. These results, along with our main results, are presented in Figure 8 . We conduct additional comparisons to strong commercial LLMs. In particular, we compare TABULA-8B to two variances of Anthropic's Claude models: (1) Claude Instant, a fast, relatively small model likely to be on the same order of magnitude of compute as TABULA-8B (the exact parameter counts of all Claude models are not publicly disclosed); and (2) Claude 3 Sonnet, a highly performant instruction-tuned LLM likely to be larger than TABULA-8B both in terms of parameter count and total training compute. Due to the cost of accessing these commercial models, we conduct this evaluation on a subset of our evaluation suite.Figure 9 shows that TABULA-8B significantly outperforms both models. Figure 9 shows two particularly interesting results: first, both Claude models show little to no improvement with increasing number of shots. This is also consistent with the behavior of the base Llama 3 model. We hypothesize that this behavior demonstrates the value of explicit training for few-shot learning, and likely highlights the gap betweenn more generalized, task-agnostic training of these models relative to the task-specific training of TABULA-8B. Second, the commercial models perform worse than any other baseline, on average, beyond 3 shots (below which most baselines perform similarly). We hypothesize that this is due, at least in part, to the instruction-following training of the Claude models; no other set of models in our comparison undergo this second post-training phase likely to be utilized in the Claude training pipeline. This instruction-following may improve models alignment with user intentions but could decrease their ability to explicitly learn from data. One concern with tabular LLMs is their perceived inability to represent numeric data: since language models represent all data as tokens in a learned embedding space, it may be more challenging for language models to learn the complex relationships between numeric features which are required for many classification tasks.In order to provide evidence of TABULA-8B's performance on numeric data, in this section we report two different slices of our main results. First, we present results on the subset of our evaluation tasks that contain at least one numeric column (int or float data type). This excludes tables that are strictly non-numeric. Second, we present results on only the subset of our evaluation tasks that contain entirely numeric data. We note that both of these subsets exclude tables with purely textual dataprecisely the datasets where we might expect language models to perform strongly.The results on these two subsets of our evaluation suite are shown in Figure 10 .Figure 10a shows that, on the evaluation subset where all tables contain at least one numeric column, TABULA-8B still outperforms baselines across all numbers of shots. Figure 10a demonstrates that the mere presence of numeric features does not erode the performance of TABULA-8B relative to existing SOTA baseline methods.Figure 10b shows that, on the evaluation subset where all tables contain only numeric columns, TABULA-8B performs on par with existing SOTA baselines but generally only matches their performance. This result is perhaps unsurprising, as numeric-only data is the most advantageous setting for GBDT models and TabPFN (as GBDTs can directly learn splits over numeric values, and TabPFN is exclusively trained on numeric features). However, the ability of TABULA-8B to match these strong baselines, while exceeding them on non-numeric data and possessing capabilities no other baseline possesses (such as zero-shot prediction and transfer), is an indication of its strength and potential utility as a general tabular classifier. Figure 10 : Few-shot curves for TABULA-8B on the subset of evaluation tasks that contain at least one numeric column (10a) and on the subset of evaluation tasks that contain entirely numeric columns (10b).We conduct a series of additional ablation studies to investigate robustness to column ordering, robustness to feature dropout, robustness to header removal, the impact of our causal masking procedure, and the impact of our data filtering procedure. We note that other aspects of our pipeline, such as the target selection procedure and the individual parameters of several of our TabLib processing heuristics, also affect the quality of our resulting model, but a comprehensive evaluation of these individual decisions is left to future work. We describe each ablation study below.In this section we conduct an ablation study of the row-causal table masking (RCTM) procedure used to train our model. To summarize the masking procedure (also described in Section 3 and Figure 2a ): we explicitly allow the model to attend across samples within the same table in a batch. We hypothesize that this structure will encourage few-shot learning and will mitigate catastrophic forgetting which could cause the base model's few-shot capabilities to deteriorate during fine-tuning.To do this, we design a controlled experiment. Both arms of the experiment use 10% of the compute of TABULA-8B (trained for 4k steps) but are otherwise identical. In one run, we remove the RCTM strategy described in Figure 2a , replacing it with a per-sample causal attention mask (the model is not allowed to attend to any samples besides the target sample, regardless of which table they are derived from). We evaluate both models over the full test suite (all benchmark tasks).The results of this study are shown in Figure 11 . Our proposed masking scheme improves the models' ability to attend across samples, while removing this masking causes the model not to learn from additional shots (for k ≤ 8) and to deteriorate as the number of shots grows. This figure Results of an ablation study comparing a model trained without our novel row-causal tabular masking (RCTM) scheme (described in Section 3 and illustrated in Figure 2a ) vs. a baseline compute-matched version of TABULA-8B. RCTM improves the models' ability to attend across samples, while removing RCTM causes the model not to learn from additional shots (for k ≤ 8) and to deteriorate as the number of shots grows. This result demonstrates the potential loss of few-shot capabilities during fine-tuning if it is not explicitly encouraged by the fine-tuning task. also demonstrates the potential loss of few-shot capabilities during fine-tuning if it is not explicitly encouraged by the fine-tuning task -but also that these capabilities can be maintained or improved over the base model if they are a part of the fine-tuning task.A potential disadvantage of a language modeling approach to tabular data prediction is that language models may be particularly reliant on semantically-informative column "headers" (column names, the keys in the tabular key-value structure), in contrast to traditional supervised learning methods which do not utilize the headers at all. This risk has been noted in previous work; for example, UniPredict [59] suggests an approach that uses a commercial LLM to "rewrite" the headers of a table to make them more informative. In this work, however, we seek to avoid expensive preprocessing and use only the provided headers from our training data.To understand this sensitivity to the semantic quality of the headers, we conduct a controlled experiment to test the effect of removing informative column headers from a dataset. Specifically, we do the following: starting from a benchmark with high-quality headers (UniPredict), we replace the original headers with "X1", "X2", . . . and the target with "Y". Then, we evaluate TABULA-8B on the data. We do not alter the data itself; only the feature names are replaced.The results of this study are shown in Figure 12 . We highlight a few key findings from these results. First, for small number of shots, the semantically-meaningful headers provide a performance benefit: for instance, in the 16-shot subset of Figure 12 , semantically-meaningful headers provide a consistent accuracy gain of 3-5pp. Second, TABULA-8B can still outperform supervised baselines even without these headers: for example, Figure 12 shows that TABULA-8B still outperforms all baselines on the benchmark even without semantically-meaningful headers. This finding is further supported by our results on the OpenML benchmarks (CC18, CTR23) in Figure 4 ; these datasets also tend to have uninformative headers. Third, we observe that the utility of semantically-meaningful headers decreases as the number of shots increase. For example, at 32 shots, the performance with and without the original headers is effectively identical. We hypothesize that, as the number of shots grows, the model is increasingly utilizing the values provided in the shots (and their distribution) and is less reliant on the keys for providing information about the task.Collectively, the results of this ablation study suggest that TABULA-8B is robust to the semantic content of the headers, and that TABULA-8B is capable of providing effective tabular data predictions even in the absence of rich column headers. Ablation Study -Column Headers (UniPredict 32-shot Subset)Figure 12 : Results of column header ablation study described in Section F.2. For low numbers of shots, there tends to be a positive effect from informative headers. However, as the number of shots increases, the utility of the headers decreases.A potential risk of using language models for tabular data prediction may be their brittleness: language models, particularly in the few-shot setting, are known to be sensitive to details which should be irrelevant to the task difficulty, including order of examples [3, 32] , whitespace [47] , and prompt formatting [26, 45, 47] . Here, we are particularly interested in probing robustness to the removal of features. Some supervised models, including XGBoost, can be trained in a way that allows them to handle missing features at inference time, at a cost of slightly decreased predictive accuracy due to the loss of information. However, whether language models possess similar characteristics is unknown.We design an experiment to test how TABULA-8B's zero-and few-shot performance degrades as features are removed from the evaluation datasets. First, on the training split of each dataset, we train a single XGBoost model using the same hyperparameter tuning procedure used for our baselines, and we extract the feature importance for all features in the dataset according to this model. Next, we evaluate TABULA-8B on each dataset, removing the top k features for k ∈ [1, 5] . These features are removed in either descending or ascending order of importance ("important first" and "important last", respectively). For comparison, we also train and evaluate XGBoost models. For these XGBoost models, we set 1/k fraction of the data to missing, uniformly at random. Then we train a hyperparameter-tuned model on this data and evaluate it on clean test data where the top-k features are set to missing (no other data is set to missing in the test data); this allows us to naturally leverage XGBoost's robustness to missing data.This experiment is conducted on the same random sample of 32 datasets described in Section F. 4 . The results of this study are shown in Figure 13 . Across 0-,4-,16-,and 32-shot evaluations, TABULA-8B shows a similar or favorable rate of decline in performance, relative to XGBoost, as the number of removed features increases (as indicated by the similar slope of the lines). We note that the XGBoost models have better absolute performance because these are full-shot models trained on the full training split; we use these models to compare the rate of decrease in accuracy as dropout increases, not to compare the accuracy itself. The results in Figure 13 also provide further evidence of when TABULA-8B may be favorable to standard supervised learning methods: namely, when the amount of missing data at test time is large. Finally, Figure 13 suggests that TABULA-8B is not brittle with respect to the features present in our evaluation datasets; removing these features causes only the expected drop in performance (and removing unimportant features is even associated with an increase in performance relative to the full feature set when the number of shots is larger than 8). Robustness Study -Feature DropoutFigure 13 : Feature dropout ablation study results. We compare the few-shot performance of TABULA-8B to the performance of an XGBoost model trained on the full training split of each dataset, progressively removing features at evaluation time. Features are removed in order of decreasing ("important first") or increasing ("important last") variable importance (see Section F.3 for details). TABULA-8B's performance decreases at a rate consistent with XGBoost.Recent work has suggested that invariance to column ordering is an important property of tabular foundation models [58] . In this subsection, we conduct a controlled experiment to assess the sensitivity of our model to the ordering of the columns in the dataset.To assess this, we conduct the following experiment. We first choose a random subset of 32 tasks from our evaluation suite (we exclude the tasks from AMLB due to their relatively irregular structure as discussed above since our goal is to compare performance on relatively standard tabular data). For each task, we evaluate the model on a permuted version of the original data: that is, we randomly permute the columns, but otherwise leave the data unchanged. Due to computational constraints, we conduct this evaluation only at a coarse grid of (0, 8, 16, 32) shots. We compare the accuracy on these tasks to the accuracy on the original data. We use the same TABULA-8B model for both the permuted and non-permuted evaluations.The results of this study are shown in Figure 14 . We observe a small drop (with Clopper-Pearson intervals overlapping at all points) after permuting the columns, but the general shape and rate of increase of the model under both cases is quite similar. We hypothesize that this drop is due to the fact that many tabular datasets, including those from our evaluation benchmark (which are drawn from manually-curated sources such as Kaggle, OpenML, and UCI Machine Learning Repository), have manually-selected column orderings that slightly improve prediction performance. This might include, for example, a "date" preceding the rest of the columns, or a "high" and "low" column located near each other in a stock dataset. These feature relationships, we hypothesize, can make it easier for models to pool information between related features and may contribute to the small drop observed on permuting the columns. We note that this sensitivity to order has been observed for language models in other contexts [32, 34] .Our results broadly show that TABULA-8B maintains consistent performance above baselines even under feature permutation. However, they also suggest that the sensitivity to prompting and formatting that affects LLMs in other contexts [32, 34] could possibly affect tabular LLMs. We believe further research into this issue is necessary, and may indeed point toward future, more effective methods for leveraging feature ordering to improve model performance.We conduct a controlled experiment to assess the impact of our filtering strategies. In particular, we compare a dataset filtered according to the strategies described in Section 4 to a "minimally-filtered" Ablation Study -Permuted Features (32-Shot Subset)Figure 14 : Results of column permutation study described in F.4. We randomly permute the columns for a randomly-selected subset of 32 tasks, and evaluate TABULA-8B on the permuted data. We hypothesize that the slight drop in model performance is due to manually-crafted and semantically meaningful feature orderings in the data. However, our results broadly show that TABULA-8B maintains consistent performance above baselines even under feature permutation.TabLib dataset. We use a "minimally-filtered" dataset rather than an unfiltered dataset because applying no filtering could potentially result in a small number of very large tables dominating training. For the minimally-filtered baseline, we only apply the max-row count filter, max column filter, and max header length filter; the latter two help ensure that the resulting serializations are not too long so that the model is still able to perform few-shot training.The results of this study are shown in Figure 15 . While our filtering strategy has a positive impact at larger numbers of shots (k ≥ 16), there is no impact evident at k < 16, with the minimally-filtered baseline performing similarly. We hypothesize that this is a reflection of the relatively limited additional filtering performed by the rest of our pipeline relative to the "minimal" baseline (which consists mainly of language filtering, PII and code filtering, and heuristics to remove excessive amounts of missing data). Additionally, given the clear impact of improvements in data quality for language model pretraining (e.g. [54] ), we hypothesize that further refinements of our filtering pipeline would be likely to achieve further gains over minimal filtering. Finally, we emphasize that some of our filtering strategies (in particular, the PII detection) were designed to improve the safety of the model, not the quality, and we believe that some form of safety filtering should still be used regardless of its downstream effects.In this section, we evaluate the improvements of TABULA-8B due to improvements in the underlying base language model. In order to do so, we train variants of TABULA-8B which are identical to the final version, except we use Llama 1 and Llama 2 as the base language models. This allows us to investigate the improvements in TABULA-8B as the underlying language model improves. We compare the Llama 1 and Llama 2 variants to a compute-matched Llama 3 variant (we use only 10% of the compute relative to our final model, as in our other ablation studies, but also compare to the 100%-compute TABULA-8B for reference). We note that, due to the smaller context sizes of the Llama 1 and Llama 2 models, we ensure the comparisons are example-matched; that is, we train Llama 2 (context size 4096) for 2x the update steps relative to Llama 3 (context size 8192), and train Llama 1 (context size 2048) for 4x the update steps. This ensures that all models see roughly the same number of tokens or examples during fine-tuning.The results of this study are shown in Figure 16 . We only report the results up to 8 shots to allow for fair comparisons across all models, as Llama 1's context size of 2048 is 4x smaller than TABULA-8B and can only fit up to 8 shots for many tasks. Figure 16 shows the clear improvement from better base models: as the underlying base models improve (Llama 1 → Llama 2 → Llama 3), the fine-tuned model also improves. These results provide hope that further improvements in language modeling could also lead to gains in tabular models. Identifying contamination in tabular data is complex. As mentioned above, tabular data is invariant to permutations of rows and columns; as a result, the "same" tabular dataset could appear in a number of different permutations of its respective rows or columns. As a result, so-called "exact" deduplication methods are likely to be imperfect for tabular data (we also perform exact deduplication in our filtering step, so at most each individual dataset should only appear once in T4).When assessing the impact of duplication on our downstream evaluations, we are particularly concerned about the possibility of a model overfitting to the overall features and distribution of a dataset, not individual points in that distirbution -a model could learn unwanted information about a 1 : Counts of potentially contaminated tables according to "fuzzy" and "strict" procedures described. Note that "fuzzy" decontamination is stricter and is more likely to generate both true and false positives when checking for contamination. test set, for example, by observing points strictly from the training set of an i.i.d. split. As a result, we focus on eliminating datasets which have the same schema as a proxy for a dataset; we do not search for individual data points within that schema.We propose two levels of searching for contamination; we refer to these methods as "fuzzy" and "strict". In fuzzy deduplication, we search for whether every column name in an evaluation dataset is present in the training dataset. In strict deduplication, we further add the condition that the number of columns must be identical. Note that we do not compute a matching directly between the columns of the two datasets, as checking for membership is much more efficient than checking for a compatible mapping between columns and our datasets can be up to 512 columns. Note that fuzzy deduplication will potentially exclude more datasets at the risk of potentially more false positives, since strict deduplication only adds conditions to the fuzzy deduplication. Fuzzy is therefore a more conservative deduplication mechanism than strict.We search over all 1.55M tables in T4 and apply both "fuzzy" and "strict" checks. Some descriptive metrics for this search are shown in Table 1 .It is perhaps expected that, in most cases, benchmark tables pass our T4 filtering procedure and make it into T4 (one notable exception is AMLB, where most tables likely fail to pass our rules which filter for cells with large numbers of characters). Indeed, these are public benchmarks designed for learning on tabular data, and TabLib includes a significant component of datasets sourced from GitHub; many users likely work with these datasets and have uploaded them to GitHub.In this section we investigate the impact of potential contamination in our training data. In particular, we investigate whether the number of repetitions of a dataset in the training data is correlated with the downstream performance on that task, and we assess performance on "potentially contaminated" vs. "decontaminated" tasks.Figure 18 shows the relationship between the number of potential contamination instances for each dataset, and the TABULA-8B accuracy on that dataset. We find no clear relationship between this potential contamination and downstream performance, and believe that this reflects, at least in part, the conservative nature of our contamination test, which is likely to have may false positives for datasets with generic column names that may occur frequently in TabLib (for example, columns "Date", "Open", "Close" are common among stock datasets; columns "v1" "v2" are common generic variable names).Additionally, we believe that other considerations likely explain the lack of correlation between (potential) contamination in the training data and downstream benchmark berformance. For example:• TABULA-8B does not train on the full corpus; datasets that appear a small number of times may in fact not be seen during training. • Prior work has suggested mixed impact of contamination [e.g. 6, 40]; contamination does not always improve performance and can sometimes reduce performance. • We are only fine-tuning the model which can be thought of as an alignment step. It is possible that contamination in pretraining affects this more and that fine-tuning is less susceptible to memorization. Figure 17 : 17a: Results curves on the subset of tasks identified as potentially contaminated according to our fuzzy decontamination procedure. 17b: Results curves on the subset of tasks not identified as potentially contaminated according to our fuzzy decontamination procedure. Figure 18 : Contamination rates vs. accuracy across varying numbers of shots. We find no clear relationship between contamination in the training set and downstream task performance.• Due to our use of deduplication, if a dataset does recur, it is not identical -so the model never really sees an identical table more than once unless we make multiple passes over the training set or sample with replacement.• Due to our use of row-level deduplication and random shuffling, the exact evaluation datasets in the same order are unlikely to ever be seen by the model even if provided during training; this may be a guard against memorization.We also separately report our results on the "possibly contaminated" vs. "decontaminated" subset of our evaluation suite, in Figure 17 . Figure 17 shows that our model (and all baseline models) tend to perform better on the decontaminated subset. We hypothesize that this reflects a few factors. In particular, datasets in the "decontaminated" subset are likely to have unique column names. This "uniqueness" likely correlates positively with semantic quality; our model will tend to perform better on such datasets. Furthermore, we hypothesize that this reflects the strictness of our fuzzy decontamination check: it is likely that many of the tables flagged as "potentially contaminated" are false positives (our manual inspection confirmed this in many cases, although it is difficult to verify that two shuffled tabular datasets are identical; we leave such an analysis to future work).We provide results for TABULA-8B on each individual task, along with the accuracy of a randomclass predictor (which is equivalent to 1 / number of classes) in this section. For the complete results for all values of shots and all baselines, see the supplementary material. Note that 'NA' results for TABULA-8B indicate that the serialized data with a given value of shots k exceeds the model's context window size.TabuLa We provide a Model Card for TABULA-8B, as outlined in [35] .Person or organization developing model: This model was developed by the authors of this paper. Organizations providing computational support are listed in the Acknowledgements, but this model is not officially developed as part of any organization. The author affiliations are listed on the first page of this paper.Model date: This paper describes the May 2024 version of TABULA-8B.Model version: This paper describes version 1.0 of TABULA-8B.Model type: TABULA-8B is an autoregressive language model, identical in architecture to Llama 3 [54] .Information about training algorithms, parameters, fairness constraints or other applied approaches, and features: Our training procedure is described in Section 3. Our procedure for dataset construction, which includes methods for removing sensitive PII, is described in Sections 4 and A.Paper or other resource for more information: This paper is the primary resource for information about TABULA-8B. Implementation details can also be found at the open-source code release associated with the project.Citation details: See the first page of this paper.License: The model uses the Meta Llama 3 license (see https://llama.meta.com/llama3/ license/).Where to send questions or comments about the model: Send questions or comments directly to the corresponding authors, or file issues on the project git repo.The dataset consists of a deterministically filtered subset of the TabLib dataset.The data consists of tables drawn from Github and CommonCrawl, as initially captured by the TabLib authors.Is there a label or target associated with each instance?Not by default. As part of our work, we select a target at random from filtered subset of the columns for each dataset. See A.2.Yes, many tables contain missing values for certain rows and columns.Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)?In some cases, the tables have informative column headers describing the relationships between features for individual rows.Are there recommended data splits (e.g., training, develop-ment/validation, testing)?Not by default. We implement these as part of our training.Are there any errors, sources of noise, or redundancies in the dataset?We deduplicate the T4 dataset so that each table appears at most once. However, as is the case with most internet scale datasets, many of the tables contain noisy values whose correctness we do not manually inspect.Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?It is self-contained.Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)?We do our best to remove any kind of data that might be considered confidential or personally identifying. If someone finds tables with confidential information that still remain, we would appreciate if they contact us so we might remove them.Does the dataset contain data that, if viewed directly, might be offen-sive, insulting, threatening, or might otherwise cause anxiety?It is possible that there are tables with information that might be anxiety inducing. We do not explicitly filter for this type of information, but believe it is not common in our dataset.Does the dataset identify any subpopulations (e.g., by age, gender)?The instances (table rows) in the data represent a variety of entities, and the majority of these do not represent persons. However, for the subset of tables where each row does represent an individual person, it is possible that the dataset does identify subpopulations.Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset?While we aim to reduce obviously personally identifying data, we do not use techniques like differential privacy to formally defend against reidentification attacks.Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orienta-tions, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)?We aim to remove this kind of information.If the dataset relates to people, are there applicable limits on the re-tention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)?Will older versions of the dataset continue to be supported/hosted/maintained?If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?Others are free to build on the dataset as long as they adhere to the original terms of use put forth by [13] .• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). Justification: Our research did not involve human subjects. When developing and releasing TabLib we pay special attention to only use public datasets that do not have any personally identifying or otherwise sensitive information. We discuss the potential harms and pitfalls of using natural language descriptions of columns, and large pretrained models for tabular prediction in Section 6. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Yes, we discuss these in Section 6. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We assess that fine-tuning of the Llama 3 model on publicly-available data poses no larger risk than the Llama 3 model itself, which is already open sourced and likely pretrained on data from the same sources as TabLib (Common Crawl, GitHub). Our model release is also required to adhere to the terms of use and Acceptable Use Policy 12 of the Llama 3 model, which includes prohibiting the use of the model for violating the law or others' rights, engaging in or facilitating harassment, processing, disclosing, generating, or inferring health, demographic, or other sensitive personal or private information about individuals without rights and consents required by applicable laws, and other harmful uses. We note that both the TabLib dataset [13] (from which our dataset is extracted), and also the original data used to create TabLib (extracted from public Common Crawl and Github data), are publicly available. As noted in the original release of TabLib [13] , TabLib does appear to contain some personally identifying information (such as names, email addresses, and phone numbers), although at least some of this data may be synthetic. We take steps to aggressively remove tables containing PII from our released dataset (described in Section A), removing any table where we detect PII. As a result, our released subset of TabLib, which we refer to as T4, may indeed be safer than the original TabLib and may improve the safety of downstream models trained on it (relative to training on TabLib). Furthermore, the release of our data processing code will both enable transparency into the dataset creation process, but will also enable future work improving the safety of tabular data derived from TabLib and other similar sources.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.For links to all code, data, and model, see Section 7.38th Conference on Neural Information Processing Systems (NeurIPS2024).https://www.mlfoundry.comwww.gauss-centre.euhttps://www.approximatelabs.comhttps://pypi.org/project/fasttext-langdetect/https://docs.ray.io/en/latest/data/overview.htmlhttps://github.com/sxjscience/automl_multimodal_benchmark/tree/mainhttps://github.com/automl/TabPFNhttps://github.com/automl/TabPFN/blob/main/tabpfn/scripts/transformer_ prediction_interface.py#L105https://huggingface.co/meta-llama/Meta-Llama-3-8Bhttps://www.anthropic.com/claudehttps://llama.meta.com/llama3/use-policy/ 
: 
 Transfer learning -the ability of a model to accurately solve prediction tasks on data it was not trained on -is one of the defining hallmarks of recent foundation models in domains such as vision [38] and language [6] . Among their many advantages, transferable models expand the scope of problems that can be tackled via machine learning by reducing the need for curated, task-specific models and datasets. Such models also can provide both absolute performance and sample-efficiency gains over task-specific models when applied to new tasks [38, 41, 53] .In this work, we introduce a new model and dataset for large-scale transfer learning on tabular data. Tabular, spreadsheet-style data underlies applications in healthcare, finance, government, and the natural sciences [4, 16, 50] .Yet, despite the potential impacts of transferable foundation models for tabular data [58] , the core practices of machine learning on tabular data have remained largely unchanged. The prevailing paradigm is still to train single-task models (e.g., XGboost [7] ) using a fixed schema on data from the same distribution on which the model will be deployed.Here, we aim to bridge this gap. We introduce TABULA-8B, a language model for tabular prediction which can flexibly solve classification tasks across unseen domains, including where data is scarce. Our methodology expands the scope what is possible in these settings, thereby democratizing access to prediction in low-resource contexts and providing state-of-the-art, training-free transfer learning on any tabular data. Since the model only requires a forward pass to perform inference on the target data, it also avoids the privacy or computational considerations that arise in other approaches that require fine-tuning on local, and potentially sensitive, datasets.In particular, given a small number of examples (shots), and without any fine-tuning on the task, TABULA-8B outperforms state-of-the-art gradient-boosted decision trees and tabular deep learning methods that are explicitly trained on the target data (see Figure 1 ). Furthermore, TABULA-8B is capable of zero-shot prediction, a behavior which is not possible using these prior methods. To enable these results, we build a new dataset for tabular prediction, The Tremendous TabLib Trawl (T4), that allows us to scale up training by several orders of magnitude (10,000× more data) relative to previous work. 
 
 Appendix:  


Decision: This paper presents a language model designed for tabular data prediction. It is trained using a large, high-quality dataset (T4) from the TabLib corpus, which contains over 800 million rows from 1.5 million unique tables. The model can readily be used for few-shot and zero-shot learning and achieve much better accuracy than baselines.

Reviewers raised several questions including the experiment design and concerns of low novelty. After rebuttal and discussion, several reviewers increase their scores.

Reviews: review: paper_summary: paper_summary: The paper introduces TABULA-8B, a language model designed for tabular data prediction. The authors detail the creation of a large, high-quality dataset (T4) from the TabLib corpus, containing over 800 million rows from 1.5 million unique tables. TABULA-8B is fine-tuned from the Llama 3-8B model using techniques for tabular prediction. Extensive evaluation shows that TABULA-8B outperforms state-of-the-art models like XGBoost and TabPFN in both zero-shot and few-shot settings. The authors also discuss the robustness of TABULA-8B, its efficient attention masking scheme, and the potential impact of data contamination.
strength_weakness: strength_weakness: The primary contribution of this article lies in constructing a meticulously curated large-scale corpus, which significantly aids in the application research of large language models (LLMs) in the domain of tabular data. **This can transform into an excellent benchmark paper. However, I think it is not suitable for the main track** for the following reasons:

- Regression tasks are crucial for tabular data prediction, but the paper converts the regression task into a four-class classification task.
- The evaluation of models is limited to the zero-shot and few-shot levels. Many of the datasets used in the benchmarks have data volumes far exceeding the few-shot range. Additionally, few-shot tasks for tabular data introduce significant randomness (e.g., the performance of xgboost models trained on data from different samples can vary greatly), necessitating a sufficient number of repeated samplings.
- The method primarily involves converting tabular data into textual data for NLP tasks. Tabular data tasks lean more towards numerical reasoning rather than text generation. Without leveraging external models, language models will face performance bottlenecks in full-shot scenarios.
- There is a need for more comparison with other relevant work on language models for tabular data prediction, such as CAAFE[1], TP-BERTa[2], FeatLLM[3], and TabLLM[4].

[1] Large language models for automated data science: Introducing caafe for context-aware automated feature engineering. NeurIPS, 2023

[2] Making pre-trained language models great on tabular prediction. ICLR, 2024

[3] Large language models can automatically engineer features for few-shot tabular learning. ICML, 2024

[4] Tabllm: Few-shot classification of tabular data with large language models. AISTATS, 2023
questions: questions: Please see the weakness.
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: The paper presents a framework in which it curates a large collection of tabular datasets, fine-tunes a language model that can readily be used for few-shot and zero-shot learning.
strength_weakness: strength_weakness: - It would be interesting to see some results with larger train-size.
- It is difficult to grasp the importance of few-shot and zero-shot learning for application perspectives. Some examples may aid the understandings.
- It would be interesting to see CatBoost and linear models or random forests to be included as baselines.
- It may be important to include a simple screening process in which the downstream datasets are not included the fine-tuning of language model step (as the paper concentrates on 'unseen' tables). 
- The supplementary section can be cleaned.
questions: questions: - What are the some scenarios of zero-shot and few-shot learning in real-world applications?
- Are there any screening process for leakage of downstream datasets?
- How does the model compare in terms of computation time for the downstream tasks? I would guess it would be the inference time, since it does not fine-tune on the specific target dataset.
- What are some difficulties of (or reasons for not employing) fine-tuning for specific target dataset?
- How does the model handle missing values? Is it simply ignored in the serialization step or is there a special token to handle them?
- One possible advantage of using language models for pretraning is that it provides a general representation across tables. In this sense, it can be used for transfer learning or domain adaptation across the tables. Would this model be applicable for such cases? (for instance, wine in France and wine in Italy).
limitations: limitations: Yes, the authors have addressed the limitations.
paper_summary: paper_summary: The paper introduces TABULA-8B, a specialized large language model for tabular data prediction tasks. It details a comprehensive process for extracting a high-quality, large-scale dataset from the TabLib corpus, labeled as T4, which comprises 1.5 million unique tables and over 800 million rows of data. By fine-tuning the Llama 3-8B large language model on the T4 dataset using a novel packing and attention scheme, TABULA-8B achieves superior performance in tabular data prediction, including classification and binned regression tasks. Through extensive evaluation on a suite of 300 datasets, TABULA-8B demonstrates a groundbreaking zero-shot accuracy that exceeds random guessing by over 15 percentage points, outperforming existing state-of-the-art models like XGBoost and TabPFN. In few-shot settings (1-32 shots), without any fine-tuning on target datasets, TABULA-8B achieves 5-15 percentage points higher accuracy than these models, even when they are trained on up to 16 times more data. Additionally, TABULA-8B's capability to perform zero-shot learning marks a significant advancement over previous methods.
strength_weakness: strength_weakness: 1. The contribution of this paper is not novel enough. Although a new dataset T4 is proposed, it is extracted and filtered from the existing public corpus and does not generate challenging data. Besides, block-causal attention mask is also a common attention method. Despite the impressive results of the TABULA-8B model, the innovation of its contribution remains to be improved.
2. Additional baseline experiments could be added to confirm the high performance of TABULA- 8B. See Q1.
3. The paper could conduct a more detailed analysis of the training and test table data, introduce the length distribution of the table, and whether it is a relational or non-relational table. While the paper proposes a series of filtering rules to ensure data quality, these rules primarily target overall table features such as row count and column count. There is a lack of evaluation for the internal data quality of tables, such as whether the data is too simple or close to real-world data, can lead to inconsistent data quality. See Q2.
4. As shown in Figure 5, due to the limitation of TABULA-8B context window, the examples that can be used for few-shot learning are limited. With the increase of the number of k samples, the effect of all models is improved, and when more than 64 samples are used, the effect of XGBOOST is almost the same as that of TABULA-8B. Considering the expensive training and reasoning cost of TABULA 8B, it cannot completely replace XGBOOST at present, which can still be improved.
questions: questions: Q1: Although XGBOOST is widely considered to be highly competitive in table prediction tasks, with the development of LLMs, there are also many LLMs applications in table prediction tasks, such as UniPredict[52] mentioned in the paper. Could you compare the results with these latest works? In addition, the large language model llama-8B is used as the baseline in the paper. How do other large language models such as GPT-3.5 and GPT-4 perform on this task? Does TABULA-8B perform better than large language models such as GPT-3.5 and GPT-4? 
Q2: Are all the tables mentioned in the paper relational tables? A big percentage of tables are non-relational in the real world. Could TABULA-8B achieve the excellent prediction effect on non-relational tables? 
Q3: In appendix F.1, in line 745, it appears that the sentence is incomplete and there is unfinished content.
limitations: limitations: The author has given a detailed explanation of the limitations in the paper.
paper_summary: paper_summary: This paper presents a large tabular prediction model based on large language models (LLMs). The model was trained on 800M rows from 1.5M tables, hence achieves high zero-shot accuracy on unseen tables. To make it work, the authors made extensive data collection, cleaning, and filtering to build a large-scale tabular datasets, based on TabLib. The yielded model was shown to outperform XGBoost and
strength_weakness: strength_weakness: - The paper overstates its novelty in applying transfer learning to tabular prediction tasks and fails to discuss several critical prior studies, including references [1, 2, 3, 4], among others.

- The evaluation does not include comparisons with any existing cross-table transfer learning baselines, which could provide a more comprehensive assessment of the proposed method's efficacy.

- The model predicts values for target columns without providing confidence scores or probabilities for each class, making it challenging to trust and use in real-world scenarios.

- There is a potential data leakage issue that needs careful attention, as the evaluation data might have been included in the large-scale web-crawled training dataset.



[1] Wang, Z., & Sun, J. (2022). Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35, 2902-2915.

[2] Zhu, B., Shi, X., Erickson, N., Li, M., Karypis, G., & Shoaran, M. (2023). Xtab: Cross-table pretraining for tabular transformers. arXiv preprint arXiv:2305.06090.

[3] Zhang, T., Wang, S., Yan, S., Li, J., & Liu, Q. (2023). Generative table pre-training empowers models for tabular prediction. arXiv preprint arXiv:2305.09696.

[4] Ye, C., Lu, G., Wang, H., Li, L., Wu, S., Chen, G., & Zhao, J. (2023). CT-BERT: learning better tabular representations through cross-table pre-training. arXiv preprint arXiv:2307.04308.
questions: questions: Pls refer to the weakness section.
limitations: limitations: n/a

Comments: title: Follow up response to Reviewer mEH8 [3/3]
Follow up response to Reviewer mEH8 [2/3]
Follow up response to Reviewer mEH8 [1/3]
Please engage the discussion.
Request from authors to review author responses
Follow-up: Claude Evaluation Results
Follow up - additional baseline results request by reviewer
comment: Dear Author,

Thanks for your detailed reply.

Although I still think the issue of "applying it to regression tasks" and "comparing it to more LLM-related work" still exists. (I’m not suggesting that you need to perform comparisons on numerous datasets; you can select some representative datasets for comparison.) **I decide to raise my score to 5, showing my encouragement.**

I hope the authors can make improvements in these two aspects, especially with respect to regression tasks that require precise numerical reasoning. This will make the contribution of the paper more comprehensive.

Regarding the Kaggle dataset I mentioned, it is an example of a case where I hope you can address the potential failures of LLMs. 

I am very eager to see this paper improved into a comprehensive LLM-tabular work.

Best wishes,

The reviewer
Additionally, we acknowledge that there is some ambiguity between D&B track and main track papers. Indeed, even the D&B call for papers acknowledges this (https://neurips.cc/Conferences/2023/CallForDatasetsBenchmarks), and it is possible that our work could also be a fit for this track. However, we emphasize, as previously, that the main contribution is not strictly “a new dataset, benchmark, or other work that falls into the scope of the track” (to quote the D&B frequently asked questions), and so we feel it is best suited for the main track.

# 3. Regression tasks

The reviewer claims that our method would be computationally infeasible to extend to regression tasks.

 First, our paper is focused on classification where, as the reviewer points out, we make a significant advance.

Second, we do not agree with the reviewer’s claim that our method “cannot be considered a practical solution for regression tasks.” Practicality is a subjective judgment that depends on context; we do not believe that any highly-effective prediction method can be dismissed purely on these grounds. The existence of many widely-used LLMs with size equal or greater than our method seems to indicate that, for at least some applications, users consider access to computationally-intensive models to be a practical solution. We believe there are likely to be users who consider significant improvements in predictive performance achieved by our model to be worth the computational cost, including in settings where they may perform multiple forward passes to iteratively refine predictions. As we mention in the supplement, prediction currently takes roughly one second for a single sample; even a 10x increase in this latency (due to repeated predictions at higher granularity for a specific example) would still put the model on par with many commercial LLMs, which have response latency in the range of seconds.

# 4. Comparison to recent works leveraging LLMs

As part of our author response, we do compare against state-of-the-art commercial LLMs like Claude that have been pre-trained on huge amounts of data. Our method significantly outperforms both Claude variants evaluated, as acknowledged by the other reviewers.  We encourage the reviewer to check the rebuttal PDF for these results.

The reviewer specifically mentioned TP-BERTa, which we discuss in the bullets below. 

* As we mentioned in our initial response, **TP-BERTa requires fine-tuning on every individual dataset. Due to this, we do not consider this method to be an applicable baseline.** As we mention in the previous response, TP-BERTa would require 2,568 individual fine-tuning runs of our 8B parameter model for evaluation (even without accounting for hyperparameter tuning or multiple runs to account for randomness over the selected shots).  We feel that this scale is not feasible for a baseline that is not directly comparable to our work. 

* **TP-BERTa is missing the prediction code to do inference/evaluation.** An issue flagging this has been open on the repository for over one month https://github.com/jyansir/tp-berta/issues/3 , which currently makes reproducing the TP-BERTa inference/evaluation procedure impossible. 

* **TP-BERTa does not support multiclass classification**, which are a significant component of our evaluations and would again make it not directly comparable to our model.

# Conclusion

Once again, we are extremely grateful for their thoughtful engagement with the work and their expert assessment and recommendations for improving it. We appreciate the reviewer’s commitment to constructive dialogue – your suggestions will considerably improve the paper.
**Table A: Accuracy on tables containing 1 or more numeric columns (random baseline: 0.331; average over 266 tasks; Clopper-Pearson confidence intervals are width ≤0.01):**

| Num. Shots | TabuLa-8B | Llama 3 8B (no fine-tuning) | XGBoost trained + tuned on k samples | TabPFN on k samples |
|------------|-----------|----------------------------|--------------------------------------|---------------------|
| **0**          | 0.492     | 0                          | N/A                                  | N/A                 |
| **1**          | 0.535     | 0.376                      | 0.403                                | N/A                 |
| **2**          | 0.551     | 0.406                      | 0.423                                | 0.351               |
| **3**          | 0.563     | 0.414                      | 0.420                                 | 0.397               |
| **4**          | 0.569     | 0.423                      | 0.433                                | 0.424               |
| **8**          | 0.598     | 0.436                      | 0.494                                | 0.503               |
| **16**         | 0.623     | 0.459                      | 0.57                                 | 0.58                |

**Table B: Accuracy on tables composed entirely of numeric columns (random baseline: 0.437; average over 51 tasks; Clopper-Pearson confidence intervals are width ≤0.025):**

| Num. Shots | TabuLa-8B | Llama 3 8B (no fine-tuning) | XGBoost trained + tuned on k samples | TabPFN on k samples |
|------------|-----------|----------------------------|--------------------------------------|---------------------|
| **0**          | 0.486     | 0                          | N/A                                  | N/A                 |
| **1**          | 0.535     | 0.45                       | 0.521                                | N/A                 |
| **2**          | 0.551     | 0.483                      | 0.555                                | 0.448               |
| **3**          | 0.563     | 0.498                      | 0.54                                 | 0.511               |
| **4**          | 0.565     | 0.505                      | 0.555                                | 0.518               |
| **8**          | 0.592     | 0.518                      | 0.579                                | 0.588               |
| **16**         | 0.619     | 0.546                      | 0.637                                | 0.637               |

  * 1c. **Additional dataset:** The reviewer provides a link to an additional Kaggle dataset. This is useful context regarding the types of data the reviewer feels would be useful, thank you! This dataset (54 feature columns, 98.1% numeric) closely resembles our evaluation suite's composition (93.8% numeric columns).  Please refer to the results on tables containing 1 or more numeric columns (since this table is not entirely numeric) above, which show that our model achieves SOTA zero- and few-shot performance, outperforming the baselines, on tables of similar composition. We believe that performance on these high-quality benchmarks is a more reliable indicator than performance on a single Kaggle dataset.

# 2. Main track vs. datasets and benchmarks

We are glad that the reviewer acknowledges how our method does indeed improve upon the state-of-the-art for tabular prediction, and is working to ensure that our paper is published in the correct venue. We share this objective.

We believe that there is a strong and established precedent that works like ours which introduces a new core methodology (i.e, a web-scale dataset and new training recipe for tabular data) and significantly expands what’s possible on a fundamental ML problem (tabular prediction) should appear in the main track.

Please see our response above with numerous examples of similar papers that appeared in the main track of NeurIPS, ICML, and ICLR. The reviewer says that “superior performance alone is not a sufficient condition to make a significant contribution to the main track”; however, we feel (and all reviewers, this one included) have acknowledged that our work does more than simply present “superior performance” for an existing method. As a further example, consider Kadra et al., “Well-tuned simple nets excel on tabular datasets”, NeurIPS main track 2021 – which is also a purely empirical tabular data study that shows that a simple, pre existing method (MLP with carefully tuned regularization) achieves superior performance with no new model or algorithm.
Thank you to the reviewer for your thoughtful engagement with our work. The reviewer expresses reservations that we ourselves were curious about when we started this work– the capacity of LLMs to model numeric data; comparison to the strongest possible baselines – and we take these concerns seriously. Indeed, we designed our experiments to address such questions, using rigorous evaluation at a scale beyond any other recent tabular prediction method. We believe that the reviewers’ questions and concerns can be addressed by focusing on different subsets of our results (e.g., tables with numeric data), and that doing so will further improve the paper’s presentation and discussion.

Since we will refer to our evaluation results below, please recall that our evaluations cover 330 tables comprising five high-quality tabular benchmarks (OpenML-CC18, OpenML-CTR23, Grinsztajn, UniPredict (which is comprised entirely of quality-curated Kaggle datasets), and the AutoML Multimodal Benchmark (AMLB)). These benchmarks have been previously proposed and vetted by the tabular data modeling community and are widely used across many studies as performance references for classification and regression on tabular data.

 We offer specific responses to each concern in the reviewer’s comment below.

  # 1. **Performance on numeric data** 

We share the reviewer’s understanding that numeric data is a fundamentally important component of tabular data. However, we believe that the reviewers’ response does not  reflect the research goals of our work, nor the strength of our results. We provide a few specific responses regarding numeric data below.

  * 1a. **Objective of this work:** The goal of our work is to demonstrate an end-to-end recipe for transfer learning on tabular data prediction tasks. All reviewers (including mEH8) appear to agree that our work considerably outperforms the current state-of-the-art (SOTA) in the few-shot tabular prediction setting and enables zero-shot prediction not possible with existing tabular models. It is not our goal to demonstrate that LLMs “fully handle complex numerical reasoning tasks in tabular data” – this objective, while important, is both out of scope for the current work and currently difficult to assess. Improving the SOTA on established benchmarks is the most common way to empirically demonstrate progress on research problems in AI. Our results show that we achieve this over both classical methods (such as XGBoost and TabPFN) and LLM-based models (such as variants of Claude). Popular methods like XGBoost and TabPFN are also widely impactful, yet only address prediction problems. 

  * 1b. **Results on numeric features:** The reviewer states that “The method's outperforming results are insufficient to demonstrate that treating numbers as text enables LLMs to fully handle complex numerical reasoning tasks in tabular data.” In the submitted version of the paper, we do not differentiate between datasets that contain numeric data and those that do not – we thank the reviewer for pushing us to make this distinction, and we discuss our evaluation results on numeric data below.

    * (i) **Numeric data is prevalent in our evaluation datasets:** We analyzed the distribution of data types across all columns in our evaluation suite. The distribution is as follows. float: 6,637 columns (33.6% of columns),  int: 11,881 columns (60.2% of columns), object: 1,201 columns (6.1% of columns),  bool: 15 columns (<1% of columns). In total, 318 tables (95.8%) contain one or more numeric columns, and 78 tables (23.5%) contain only numeric columns. **Numeric data is thus by far the most prevalent type of data across our evaluation suite, comprising 93.8% of the columns across our evaluation tables** (33.6% float + 60.2% int). These benchmarks thus reflect the reviewers' emphasis on numeric data.
    * (ii) **Results on tables with numeric data:** To further characterize our model’s performance on numeric data, we provide two additional views of our results below. First, we give the performance on tables in our evaluation suite which contain at least one numeric column (int or float dtype). Second, we give the performance on tables which contain *only* numeric columns. Our results indicate that our model still outperforms all baselines on tasks containing numeric data (related to the previous point). Furthermore, the results show that our model is competitive with (matching or outperforming) baselines even on tables that contain *entirely* numeric data – a setting that advantages TabPFN and XGBoost which can operate in continuous space.

*(continued in following post due to character limits)*
Thanks for your responses. Some of my concerns remain as follows:

1. The method's outperforming results are **insufficient to demonstrate that treating numbers as text enables LLMs to fully handle complex numerical reasoning tasks in tabular data**. The improvement could stem from various factors, such as carefully curated datasets during LLM training or the high-quality text transformation of the test datasets. There are many tabular datasets that can't be reasoned by text, such as pump-sensor-data (https://www.kaggle.com/datasets/nphantawee/pump-sensor-data).  **LLMs are unable to make reasonable predictions on these numerical datasets based solely on text**.

2. Similarly, superior performance alone **is not a sufficient condition to make a significant contribution to the main track**. TransTab has introduced a general transfer learning approach, CAAFE employs LLMs for feature engineering, and TabPFN has pioneered the application of PFN families in tabular data, all of which hold a unique position in the field of tabular data.

3. Repeatedly narrowing the "bins" based on the models' predictions for regression tasks is a theoretical approach, but it is essentially still a classification method. but achieving the desired precision requires considerable computational resources. I think this approach **cannot be considered a practical solution for regression tasks**.

4. Since large language models (LLMs) are being used, it would be **unfair not to compare them with recent works that also leverage LLMs**. While a comparison across all benchmarks isn't necessary, I believe a fair comparison on several datasets is essential. For instance, as far as I know, fine-tuning TP-BERTa does not incur an unbearable cost.


In summary, the paper's main contribution lies in the meticulous preparation of the dataset and the training of the corresponding language models, which are helpful for LLM tabular prediction. **All the aforementioned concerns represent challenges for LLMs in tabular prediction. However, compared to other methods that serialize tabular data into text for prediction, this paper does not offer sufficient breakthroughs. In terms of field contribution and application scope, I think it may not be entirely suitable for the main track.**
Dear AC and authors,

Thank you for the reminder. As I am currently working, I will reply within eight hours.
Dear Reviewer 3ko2 and mEH8, 

Please read the authors' response and give your comments!

Thanks
Thanks for your reply. I've improved the score to 7.
Dear reviewers,

Thank you once again for your thoughtful and insightful engagement with our work during the review process. We sincerely believe that each reviewers' expert feedback has improved our work considerably, and that the additional results and discussion generated during the review process have only strengthened the clarity of our paper's results in zero- and few-shot transfer learning for tabular data and clarified the significance of its contributions in this area.

Given that the author response window is closing soon, we would like to follow up on our previous responses to encourage reviewers to review our detailed responses. Please let us know if there is additional clarification that we can provide while the author dialogue window is still open. Several reviewers requested additional results or clarifications, which have been provided in our responses, and we would greatly appreciate your review of our responses and the opportunity to continue to provide clarification as time allows. Additionally, if our responses have properly addressed your concerns, please consider adjusting your score to reflect your current assessment of our paper's contributions.

Warmly,

Paper authors
Thank you for your engagement during the dialogue window, and for your thoughtful consideration of both our initial submission and rebuttal/additional results. We are glad that our response and our additional results answer your concerns from the perspectives of "novelty, baseline experiment, data statistics, etc.", and that you found the results "comprehensive."

We appreciate your increase in the score. If there are further clarifications we can provide before the dialogue window closes, please let us know.
I appreciate the authors' detailed feedback. Thank you for answering my concerns in detail from the perspectives of novelty, baseline experiment, data statistics, etc. Although I think there are still some limitations in terms of novelty, the experiment is comprehensive, and the results are very detailed. I will raise the score.
We are following up on our previous comments with results comparing our model to Claude, a frontier AI model from Anthropic.  In particular, the reviewer asked:

> Does TABULA-8B perform better than large language models such as GPT-3.5 and GPT-4?

We provide results comparing our model to both Claude 3 Sonnet and Claude Instant, in order to provide comparisons to both very strong commercial LLMs (Claude 3 Sonnet) and mid-tier commercial LLMs (Claude Instant) at two different price-performance ratios. **Simply put, the answer to the reviewers’ question is “yes” – our model outperforms both versions of Claude across all values of shots evaluated.** Please find the results curves in the author rebuttal PDF. 

Below, we describe the Claude experiments in slightly more detail and provide a more detailed interpretation. However, we emphasize that the main takeaway of this experiment is to provide clear evidence that our model outperforms strong commercial LLMs. Again, we are grateful to the reviewer for suggesting these additional experiments and believe that they will considerably strengthen the paper!

# Experiment Design - Claude 3 Sonnet and Claude Instant Evaluation

We perform the same procedure for both Claude 3 Sonnet and Claude Instant. For each model, we serialize inputs in a format identical to that used in the paper. During evaluation we use 128 randomly-selected samples at every value of k (number of shots) and from every table (random shots are also selected IID for every sample in the same manner as the experiments in the paper). However, we also do the following in accordance with Claude’s training and recommended usage:
1. Follow the “Human:” “Assistant:” format recommended in the Claude user guide.
2. Add a prompt describing the general task and the input/output format. This prompt is the same for every dataset.

Claude models are rate-limited. As such, due to the very large number of evaluations required (330 benchmark datasets at 1, 2, 3, 4, 8, 16, 32 shots, all with 128 samples each), we were not able to complete the evaluations on our entire benchmark suite prior to the closure of the rebuttal window. However, we provide the complete current set of results (74 total benchmark tasks at 16 shots, 46 of which are also 32-shot) which we believe provide a strong representative sample from our evaluation suite. We commit to adding the complete set of results to the camera-ready version of the paper.

# Results Interpretation

The results (shown in the rebuttal PDF) show the following:
* **TabuLa outperforms both Claude models across all numbers of shots evaluated.** Our model achieves significantly higher few-shot accuracy (shown by the nonoverlapping confidence intervals) at every point evaluated on the curves. TabuLa outperforms Claude 3 Sonnet by 10-20% everywhere.
* **Stronger commercial LLMs also have stronger few-shot performance.** Claude 3 Sonnet performs better than Claude Instant, which reflects the larger model capacity and (likely) larger training data of Claude 3 Sonnet. This also reflects their relative rankings according to various benchmark metrics and is a useful sanity check of our overall evaluation approach.
* **Commercial LLMs do not always improve monotonically with more shots.** It has been noted in the literature (for example, GPT-2 and GPT-3 papers) that more shots do not always improve the performance of generalist LLMs in few-shot learning. Our models’ capacity to monotonically improve with more shots is a key advantage of our training procedure. Our results also show that the two Claude models evaluated demonstrate this behavior – the performance actually tends to level off after 4-8 shots, with no further improvements.
* **RLHF may drive the observed differences in behavior.** It is widely known that commercial LLMs (likely including all Claude and ChatGPT models) undergo a post-training procedure where the model is aligned to human preferences using reinforcement learning from human feedback (RLHF). The Claude models are the *only* such models in our study. This suggests (but does not prove) that RLHF may be a factor driving the different performance of the Claude models vs. other models (and particularly the non-RLHF models like TabuLa and Llama 3 7B base model). It is possible that RLHF decreases models’ ability to learn our task in-context. Again, this contrast demonstrates the strength of our approach.

# Prompt Used

You are performing a classification task on a tabular dataset.
Below you will be provided with a few rows of tabular data, with the label for each row.
The final row in the tabular dataset will not contain a label.
Your task is to predict the label for that row, using the choices provided.

Instructions:
* Predict the target label for the final row.
* Always choose one of the provided classes.
* Only return one of the provided classes; do not return any other text.

Data:
Following up, in our rebuttal PDF we are also including results curves which include, in addition to the baselines from the original text, a linear model (logistic regression) and CatBoost. We use the same procedure for these baselines as described in the paper: specificaly, we conduct 10 completely separate sampling iterations for every dataset and number of shots, and we evaluate on the full test set for each dataset. For logistic regression, we conduct a complete grid search over the L2 regularization parameter over a grid of 51 values (50 regularization values plus no regularization). For catboost, due to the computational expense of training and the time limitations of the response window, we use the default hyperparameters. (Note that recent work has shown that the default hyperparameters of CatBoost are at or near SOTA for tabular classification [7] and that the difference due to tuning from CatBoost tends to be less than the difference between NN-GBDT algorithm selection [8], indicating that tuning does not tend to change the relative rankings of CatBoost vs. NN models).

These results show the following:
* Logistic regression and CatBoost are both outperformed by TabPFN and XGBoost at nearly all points along the curves.
* Logistic regression has surprisingly competitive performance, particularly in the range of 4-16 shots.
* CatBoost performs less well across the benchmark, and is the lowest-performing baseline (vs. logistic regression, XGBoost, and TabPFN). We hypothesize that this may be due to the frequency of numeric data across our tasks. We also note that few-shot evaluations of CatBoost are rare or nonexistent in the literature, and the few-shot performance of CatBoost is simply not known -- it may be the case that CatBoost requires relatively larger datasets to reach peak performance. Furthermore, this is connsistent with studies that have found XGBoost to outperform CatBoost [9] (although we note that the relative performance of these two varies across studies and likely reflects nuances in the data and experimental setups across studies).

We are grateful to the reviewer for suggesting these additional baselines, and will add them to the paper! We believe that these results also provide further evidence of the effectiveness of our model -- even these additional strong baselines do not outperform our proposed method. We hope that this further supports the reviewers' connclusion that our model achieves "strong performance in zero-shot and few-shot settings."

References
* [7] Gorishniy, Yury, et al. "Revisiting deep learning models for tabular data." Advances in Neural Information Processing Systems 34 (2021): 18932-18943.
* [8] McElfresh, Duncan, et al. "When do neural nets outperform boosted trees on tabular data?." Advances in Neural Information Processing Systems 36 (2024).
* [9] Kadra, Arlind, et al. "Well-tuned simple nets excel on tabular datasets." Advances in neural information processing systems 34 (2021): 23928-23941.
_**Large shot behavior vs XGBoost.**_

The goal of our paper is not to outperform XGBoost when the number of shots is large, but rather when the training set is small. We believe that tabular prediction in low data regimes is a fundamentally important problem. This is particularly true in settings like health, or education, where an institution (i.e a school or hospital) wants to develop a predictor for their specific, local population but lacks a large historical database of cases. 

In this regime, our method significantly outperforms classical, SOTA algorithms like XGBoost that cannot leverage large scale pre-training datasets. Further, we note that TabPFN is also very limited in what data it can be used on (no more than 10 classes, no more than 100 features, limited overall context window) but has been highly impactful as a tabular few-shot learning method (and our method in fact even outperforms TabPFN, without having these limitations). 

L745: Thank you for pointing this out, we will fix this. We meant to restate our definition of tabular data here as presented in L67-72 in Section 1.2.


_**References:**_

- [1] Wang et al. UniPredict: Large Language Models are Universal Tabular Classifiers, 2023
- [2] Llama AI Team, Meta. The Llama 3 Herd of Models, 2024
- [3] Hegselmann et al, TabLLM: Few-shot classification of tabular data with large language models. AISTATS, 2023
- [4] Raffel et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transforme. JMLR 2020
Thank you for reading our manuscript in detail and providing a constructive review. We’re very encouraged to hear how you believe that our methodology and results “will likely set a new standard for future research” in this area. Please see our responses to your concerns below, and thank you in advance for your dedication to this discussion period. 

_**Novelty.**_

It is true that our paper does not propose a new architecture or optimization algorithm. However, the goal of a scientific paper is not necessarily methodological novelty but generating new knowledge and insights, which you can do by combining an existing method with new training data for instance. This has especially been the case with many landmark results in AI in recent years (e.g GPT3.5, Llama 3). 

To quote from the Llama 3 technical report [2], “Llama 3 uses a standard, dense Transformer architecture. It does not deviate significantly from Llama and Llama 2 in terms of model architecture; our performance gains are primarily driven by improvements in data quality and diversity as well as by increased training scale.” Despite not presenting new algorithms these research endeavors have undoubtedly had massive impact. 

Like these breakthrough results, we performed careful ablation experiments with attention masks and optimization routines that empowered our results, and spent most of our effort in designing robust and scalable procedures that increase training data by four orders of magnitude relative to previous tabular language modeling papers  [1,3]. The main novelty of our work is that we are the first to outperform state of the art models like XGBoost on unseen datasets in the few shot regime without any finetuning on the target data.

Furthermore, providing a high-quality dataset that is “extracted and filtered from the existing public corpus” is also a contribution that can both be extremely impactful, and is in line with previous breakthrough efforts in other domains (e.g. the C4 dataset [4], for example, is a high-quality filtered version of the public Common Crawl and is widely used in SOTA language models). 

_**Baselines.**_ 

We are grateful for the reviewers’ excellent suggestion to compare our method to state-of-the-art generalist LLMs such as GPT3.5 and GPT-4.  We will evaluate the performance of Claude 3.5-Sonnet, a current SOTA commercial LLM that provides a strong balance between price and performance, on our benchmark datasets, and is capable of zero- and few-shot prediction with no fine-tuning on the target datasets. We will report back on these results in the next few days. Thank you again for the suggestion and we look forward to sharing the results! 

In terms of other deep learning baselines, comparing to methods like Unipredict is out of scope since there is not currently an open-source model for us to use for evaluations (the current UniPredict repository is minimal, and it says “The official repository will be released after the review process finishes” and that training from scratch using the current code “ is not recommended” [1]). More importantly, UniPredict requires fine-tuning on every downstream target dataset, a limitation we explicitly aim to avoid. Please recall that our method does not require any gradient updates on the downstream tables, we only need to perform a forward pass at inference time – this is a key differentiator from many previous works.

_**T4 Summary Statistics and Relational vs Non Relational Tables.**_

We are grateful for the reviewers’ in-depth interest in understanding data quality – this is also important to us! Perhaps the reviewer can clarify what they mean by a relational vs non-relational table, in the context of our data and task setup. Each table in our evaluation suite is a standalone table. Different tables refer to different entities and are in that sense non-relational. We agree with the reviewer that performance on unseen and unrelated tables is the gold standard for transfer learning and this is the core philosophy behind our choice of evaluation set. 

We agree with the reviewer that more detailed summary information about the T4 dataset would be useful! Thank you for this request. We provide broad summary statistics about T4 in Figure B of the pdf of the author response. These include histograms of # of rows and columns per table, as well as the various distributions of data types.
_**Handling Missing Values.**_ 

Thank you for the question. This is another key advantage of our method. To perform inference on rows with missing values, we do not perform any imputation or preprocessing. For missing or null values, we simply retain the value (e.g. ‘nan’ or ‘None’) in the serialized data. No special token is used. This allows the model to directly learn to process and represent missing or null data during its training, without any additional preprocessing.

_**Learning General Representations.**_ 

Yes, absolutely! Thank you for raising this. This is one of the main motivations behind our work and one of the core strengths of the Tabula model. It is precisely because of its ability to leverage large scale pretraining on diverse datasets (wine in france) that tabula can learn meaningful representations and generate good predictions on downstream datasets. One core piece of evidence is its ability to predict labels zero shot much better than what is information-theoretically possible by random guessing. See Figure 1 in the paper.

_**References:**_

- [1] JC Perdomo, The Relative Value of Prediction in Algorithmic Decision Making; ICML 2024
- [2] See this PR for scikit-learn to add the feature, which has been in progress for over five years: https://github.com/scikit-learn/scikit-learn/pull/12866
- [3] Alayrac et al. Flamingo: a Visual Language Model for Few-Shot Learning. Neurips 2022.
- [4] Wang et al. Generalizing from a Few Examples: A Survey on Few-Shot Learning. ACM 2020
- [5] Radford et al. Language Models are Unsupervised Multitask Learners. 2022
- [6] Brown et al. Language Models are Few-Shot Learners. 2023
_**Screening Downstream Datasets.**_  

In Section 5.6 and Appendix G, we describe how we tested for the possibility of downstream datasets being included in the pretraining set. Please see the paper for a precise description, but to test whether a table in the test set appears in the training set, we check whether there exists a table in T4 that contains the same set of columns and column names as a subset of its columns.  

Using this conservative procedure, which is likely to over report the true number of tables in the test set that appear in T4, we report results on the subset of tables from our eval set which we know are not on the training set. We report the results of these evaluations in Figure 8 and the bottom right subplot of Figure 4. As mentioned in the paper, we find that Tabula performs better on these tables that contain no overlap with the training set: both in an absolute sense (the average accuracy is higher) and also in a relative sense (the gap to XGBoost is higher than on the possibly contaminated set). We conclude from these results that overfitting to the training set is not a concern. We note that this finding is in line with previous high-profile evaluations of dataset contamination, including the original GPT2 and 3 papers [5,6]. 

_**Supplementary Section.**_ 

Thank you for the suggestion (and for reviewing the supplementary material). We will invest effort in improving the clarity of the appendix. In addition to incorporating various revisions proposed by other reviewers, we would be happy to incorporate any specific edits the reviewer feels are appropriate. We believe that the extensive supplementary results (e.g. extra ablation studies, detailed per-dataset results, etc.) are important to our paper and we will conduct a thorough revision for the camera-ready to maximize the clarity of the supplement.

_**Inference time.**_ 

One of the core advantages of our approaches relative to previous deep learning methods for tabular data, is that we only need to perform a single forward pass through the model to get a prediction, we do not need to fine tune on the target datasets. On average, it takes about 1s to do a forward pass through the model on a single 40GB A100 GPU. 

This is significantly faster than fine-tuning even a much smaller model in terms of wall clock time. It is comparable to the time it takes to fit an XGBoost model. However, making inferences (predictions) using a trained XGBoost is significantly cheaper (predicting on a single example using XGBoost takes about .001s).  Holistically speaking, the overall inference time costs of one method vs the other depend on the total size of the dataset.
Firstly, thank you for taking the time to review our paper and bringing up a number of interesting comments that we respond to below. We appreciate your dedication to the discussion period.

_**Larger train size.**_ 

Our model is trained on 8b tokens, which after various performance optimizations, took 6 days on a node of 8 80GB A100s. Due to financial and computational limitations, we could not afford to train for longer. We agree however that it is interesting and valuable to understand how scale, and the quality of the initial model, affects performance. We also believe that the open-source release of our data, model, and pretraining code will enable future work in this direction.

To answer this question, regarding how train size affects performance while respecting our compute and time constraints, we follow the identical training procedure for the final Tabula8b model, except that we train for 1/10th as long (process 90% fewer tokens). We present these in Figure A in the pdf included in the general author response.By comparing Tabula-8b to the compute matched baseline that sees fewer examples, we see that increasing training size by 10x improves performance but only slightly. Hopefully this answers your question. 

As an additional experiment, we also inspect how the final performance of the model varies as we change the initial language model from llama 3 to llama 1 and 2. We train llama 1 and 2 models using our methodology for the same number of tokens as the Tabula 8b compute-matched baseline. 

This experiment highlights one of the core advantages of our proposal. More than proposing a specific algorithm, our paper develops a core methodology that adapts language models to tabular classification tasks. As open source LLMs continue to improve, so will the downstream performance of models that are adapted via our procedure -- as evidenced by the large gap between the final performance Llama 1 and the Tabula 8b compute matched baseline. By releasing our comprehensive software suite, we also empower the broader community to further develop this technology.

_**Importance of Zero and Few Shot Learning.**_ 

Prediction on tabular data in low data or few shot regimes is an important problem whenever we want to localize models to specific domains. This is often the case in regimes like health, finance, or education, where an institution (i.e a school, bank, or hospital)  wants to develop a predictor for their specific population but lacks a large historical database of similar cases. Our work shows how by leveraging Tabula-8b, one can significantly expand the scope of what’s possible relative to fitting standard methods like XGBoost on that data. Importantly, small improvements in prediction may be enough to solve important resource allocation problems, like vaccine distribution or targeted cash transfers [1]. Better few shot models democratizes access to prediction in low-resource, data-poor domains. 

Furthermore, few shot learning already has proven to be extremely impactful in other modalities like vision and language [3]. Please see this ACM article for a broader survey regarding the impact zero and few shot learning has had in practice [4].

_**Other Baselines.**_ 

Thank you for the suggestion. We are currently running these experiments and will add logistic regression and catboost as baselines (we note that CatBoost in particular has emerged as a very strong tabular classifier in the full-shot regime; random forest tends to be less competitive). We will include these baselines in a follow-up post before the close of the dialogue window. We do not compare to Random Forest as it does not support categorical features [2] due to the computational complexity of high-cardinality categories. Hence, it could not be evaluated on a large subset of our datasets.
Thank you for taking the time to read our paper, provide constructive comments and insightful connections to related works that will improve the quality of our manuscript. We’re very encouraged to see you appreciate the value of our scaling efforts to build models that can achieve “high zero-shot accuracy on unseen tables”. We respond to your comments below:

_**Relationship to Prior Work and Baselines.**_ 

Thank you for bringing these important and relevant papers to our attention. We will certainly add them and discuss them in our updated manuscript. While relevant, we believe these papers provide clearly distinct and complementary perspectives. In particular, 

- Xtab Zhu et al 2023 [2]: Xtab provides a novel method for cross training on many tables, however it requires explicitly fine tuning a featurizer and projection head on each downstream evaluation dataset. As such, it falls outside the scope of our benchmark comparisons since we are interested in deep learning methods that do not require training on the test task (e.g only require a forward pass). 

- Zhang et al 2023 [3]: This paper provides an interesting method for data augmentation on tabular datasets, rather tabular prediction per se. It can be deployed in addition to our model to improve performance, but we do not feel that it is an appropriate baseline for our model, as it only provides a method for augmentation, not an end-to-end prediction method. 

- TransTab, Wans & Sun 2022 [1] & Ye et al 2023 [4]: As is the case with Xtab, the methodology proposed by these authors requires fine tuning the model on every downstream dataset. It falls into a different class of methods and it’s not evident from the original works that one can scale up pre-training their model to millions of tables.  

In terms of other cross table learning baselines, we compare against the base Llama 3 model which is trained on 15T tokens. We are currently also adding a comparison against the few shot performance of the latest Claude model. We would be happy to include additional baselines capable of zero- and few-shot prediction on tabular data, if pretrained models are available along with instructions on how to apply the models on new data; please feel free to point us to other existing high-quality, open-source implementations we may have missed so that we may include them in our results.

_**Class Probabilities**_

Since our classifier uses an LLM, it can be extended to provide class probabilities in a straightforward fashion. Language models do this natively. Given the serialized example, we can compute the likelihood with which the model will complete the prompt using each of the possible labels. These would be the class probabilities. This has been shown to be effective in other few shot learning papers that use LLMs (see e.g [6]) and we can add this functionality to final version of the paper. 

_**Potential Data leakage.**_ 

In Section 5.6 and Appendix G, we describe how we tested for the possibility of downstream datasets being included in the pretraining set. Please see the paper for a precise description. To test whether a table in the test set appears in the training set, for each table, we check whether there exists a table in T4 that contains the same set of columns and column names.  

Using this conservative procedure, which is likely to over report the number of tables in the test set that appear in T4, we report results on the tables from our eval set which we confidently know are not on the training set. We report the results of these evaluations in Figure 8 and the bottom right subplot of Figure 4. 

As mentioned in the paper, we find that Tabula performs better on these tables: both in an absolute sense (the average accuracy is higher) and also in a relative sense (the gap to XGBoost is higher than on the possibly contaminated set). We conclude from these results that contamination does not appear to be a concern. We note that this finding is in line with previous high-profile evaluations of dataset contamination, including the original GPT2 and 3 papers [5,6]. 

_**References**_

- [1] Wang, Z., & Sun, J. (2022). Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35, 2902-2915.
- [2] Zhu, B., Shi, X., Erickson, N., Li, M., Karypis, G., & Shoaran, M. (2023). Xtab: Cross-table pretraining for tabular transformers. arXiv preprint arXiv:2305.06090.
- [3] Zhang, T., Wang, S., Yan, S., Li, J., & Liu, Q. (2023). Generative table pre-training empowers models for tabular prediction. arXiv preprint arXiv:2305.09696.
- [4] Ye, C., Lu, G., Wang, H., Li, L., Wu, S., Chen, G., & Zhao, J. (2023). CT-BERT: learning better tabular representations through cross-table pre-training. arXiv preprint arXiv:2307.04308.
- [5] Radford et al. Language Models are Unsupervised Multitask Learners. 2022
- [6] Brown et al. Language Models are Few-Shot Learners. 2023
- [7] Awadalla et al. OpenFlamingo. 2023
_**References**_

- [1] Large language models for automated data science: Introducing caafe for context-aware automated feature engineering. NeurIPS, 2023
- [2] Making pre-trained language models great on tabular prediction. ICLR, 2024
- [3] Large language models can automatically engineer features for few-shot tabular learning. ICML, 2024
- [4] Tabllm: Few-shot classification of tabular data with large language models. AISTATS, 2023
- [5] Wang, Ruiyu, Zifeng Wang, and Jimeng Sun. "Unipredict: Large language models are universal tabular predictors." arXiv preprint arXiv:2310.03266 (2023).
- [6] Fang, Xi, et al. "Large language models (LLMs) on tabular data: Prediction, generation, and understanding-a survey." (2024).
- [7] Wen, Xumeng, et al. "From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models." arXiv e-prints (2023): arXiv-2310.
- [8] Yang, Yazheng, et al. "Unleashing the Potential of Large Language Models for Predictive Tabular Tasks in Data Science." arXiv preprint arXiv:2403.20208 (2024).
- [9] Zhang, Han, et al. "Towards foundation models for learning on tabular data." arXiv preprint arXiv:2310.07338 (2023).
- [10] Torgo, L. and Gama, J., 1996. Regression by classification. 
- [11] Wang, Zifeng, and Jimeng Sun. "Transtab: Learning transferable tabular transformers across tables." Advances in Neural Information Processing Systems 35 (2022): 2902-2915.
- [12] Zhu, Bingzhao, et al. "XTab: cross-table pretraining for tabular transformers." Proceedings of the 40th International Conference on Machine Learning. 2023.
- [13] Hollmann, Noah, et al. "TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second." The Eleventh International Conference on Learning Representations.
_**Randomness in Evaluation**_

We share the reviewers’ concern about reliable evaluation and took careful steps to ensure that our estimates were robust to the potential issues related to random selection which the reviewer correctly identifies. We kindly remind the reviewer that, for every baseline method, we conduct 10 completely independent trials at every number of shots for every dataset; we also always evaluate on the full remainder of the data for testing, which allows for very large test sets. The gaps between our method and XGBoost hold robustly across a suite of over 300, independently evaluated, tabular benchmark datasets – an evaluation pool much larger than any of the related works mentioned below (e.g. TabLLM [4] uses less than 25 tables and [5] Unipredict evaluates their model on less than 70). 

As such, we believe these estimates provide a strong reliable signal of performance. We also kindly remind the reviewer that 95% Clopper-Pearson intervals are shown in all of the curves in our paper (in most cases they are extremely narrow due to the large test sets used, which is an indication of the low degree of statistical uncertainty of our point estimates); the intervals for TabuLa-8B indicate a high degree of statistical confidence that its performance is not equivalent to any of the baseline methods.  If the reviewer has a specific statistical test that they would like us to perform that they believe would provide further clarity into this question, please let us know. 

_**Regression vs. classification:**_ 

We agree that regression is an important task in tabular prediction! This is part of why we include binned regression tasks in our approach. This is well motivated for a number of reasons. First, it follows the precedent set by prior work in the tabular LLM space that bins real valued targets, e.g.  [5]. Second, as has been classically observed by the learning theory community, one can reduce regression to classification [10]. That is, any algorithm capable of solving classification tasks can also be used to solve regression tasks by using binary search on a series of binned regression tasks. Once again, our method is directly compatible with performing more detailed regression inference out of the box – one can simply repeatedly narrow the “bins” based on the models’ predictions, fit a fixed sample, and make regression predictions to an arbitrary degree of precision – perhaps an advantage over a more rigid regression approach which would only allow for a fixed precision of the outputs. We will more clearly highlight this potential future direction in the discussion and future work sections. We also note that our release of the code, evaluation suite, and pretrained model will enable other researchers to conduct detailed further experiments in this direction as well.  

_**Benchmark vs. Main Track:**_ 

The reviewer states that “This can transform into an excellent benchmark paper. However, I think it is not suitable for the main track”. We are glad the reviewer acknowledges our papers’ contribution. However, we strongly disagree that the paper is not suited for the main track.  A new, fully open model and dataset which enable, as the reviewer states, “superior performance in zero-shot and few-shot learning scenarios, outperforming SOTA models like XGBoost and TabPFN'' is a contribution in line with prior works which have appeared in the main track of NeurIPS and other top AI conferences. For example, TransTab [11] and CAAFE [1] appeared in the NeurIPS main track, [2] appeared in the ICLR main track (spotlight), [4] appeared in AISTATS main track, [3] and [12] in ICML main track. TabPFN [13] also appeared in ICLR main track (notable paper - top 25%) and, as the reviewer notes, our method substantially outperforms TabPFN. We note that [1], [2], [3], and [4] also all use existing pretrained LLMs as the backbone of their models. While not identical to the current work, we believe these papers make comparable contributions in the area of cross-table or few-shot classification for tabular data, and that our models’ “superior performance in zero-shot and few-shot learning scenarios” also justifies a contribution most relevant to the main track, not a benchmark. 

We also kindly note that our paper does not propose, or attempt to conduct, a benchmarking study of existing algorithms; we simply aggregate a large set of high-quality tabular benchmarks and use them to compare TabuLa to other relevant methods from the literature.
Thank you for taking the time to carefully read our paper and provide constructive comments that will improve the quality of our manuscript. We also appreciate your recognition of how “TABULA-8B demonstrates superior performance in zero-shot and few-shot learning scenarios”, as well as your belief that our paper will encourage further work in this area. We respond to your comments below, and thank you in advance for your dedication to the review process.

_**Relationship to prior work.**_

We would like to explicitly state that two key differentiators of our approach relative to the related works highlighted by the reviewer are scale and finetuning-free transfer of our method. With respect to scale, TabuLA is trained on four orders of magnitude more tabular data than any existing tabular LLM, including TP-BERTa (202 tabular datasets), TabFM [9] (115 datasets), and UniPredict (169 datasets). Our training set consists of 4.2 million datasets. With respect to training-free transfer, many existing tabular LLMs perform fine-tuning directly on the target dataset “shots”. This includes TabLLM (except in the zero-shot case) and UniPredict. In contrast, our method performs zero training on any downstream task on any setting in the paper. We believe these substantial differences mean that comparisons to these methods are not always informative – especially considering the significant cost of the downstream fine-tuning – and we kindly request that the reviewer consider these differences throughout our discussion. We compare to prior work in more detail below:

- CAAFE [1] and FeatLLM [3]: Both of these papers provide feature engineering and data augmentation methods that can then be used by a base tabular prediction model like logistic regression ([1] uses TabPFN as the downstream classifier). They do not provide models for tabular prediction per se. As such, both of the methods are largely complementary/orthogonal to our work. One could in principle use these methods to engineer more features for a specific table, and then feed this new table into Tabula.

- TP-BERTa [2] and TabLLM [4]: Both of these methods require explicitly fine-tuning a large model on every downstream/eval dataset. This is a significant limitation that has been noted in prior works [6, 7] that we explicitly aim to overcome with our method which only requires a forward pass at test time (no gradient updates). To provide further context, to evaluate TabLLM on 321 benchmark datasets with (0, 1, 2, 3, 4, 8, 16, 32) shots, this would require (321 * 8 = 2568) individual fine-tuning runs of the T-few model (even without performing hyperparameter tuning), which is not computationally feasible. 

_**Numerical reasoning**_ 

Numeric features are indeed an important aspect of tabular data. However, we believe one of the core contributions of our work is to show that LLMs can indeed outperform classical SOTA baselines by treating numbers as text. This simplicity and power is a feature-- not a bug-- that allows us to scale up training to millions of tables, in ways previous work could not. The OpenML benchmarks and a large subset of the Grinsztajn datasets consist primarily of numeric features and our model outperforms TabPFN and XGBoost by a significant margin on these datasets. Please see Figure 4. 

_**Full-Shot vs Few Shot**_ 

The primary focus of our work is to develop new methods of learning that expand the scope of what’s possible in data scarce regimes. That is, the goal of our paper is not to outperform XGBoost when the number of shots is large, but rather when the training set is small. We believe that tabular prediction in low data regimes is a fundamentally important problem. This is particularly true in settings like health, or education, where an institution (i.e a school or hospital) wants to develop a predictor for their specific, local population but lacks a large historical database of cases. We believe that it is clear from our results that Tabula-8B significantly expands the scope of what is possible.