UniTabE: A Universal Pretraining Protocol for Tabular Foundation  Model in Data Science
Authors: Yazheng Yang
Yuqi Wang
Guang Liu
Ledell Wu
Qi Liu
Venue: ICLR 2024 poster (2024)
Keywords: Pre-training Tabular Encoder
Pre-training
Heterogeneous Tabular Data
Classification and Regression
Deep Learning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: This study explores pre-training over tabular data with the proposed UniTabE method, addressing challenges in diverse table structures and knowledge transfer.

Abstract: Recent advancements in Natural Language Processing (NLP) have witnessed the groundbreaking impact of pretrained models, yielding impressive outcomes across various tasks. This study seeks to extend the power of pretraining methodologies to facilitating the prediction over tables in data science, a domain traditionally overlooked, yet inherently challenging due to the plethora of table schemas intrinsic to different tasks. The primary research questions underpinning this work revolve around the establishment of a universal pretraining protocol for tables with varied structures, the generalizability and transferability of learned knowledge across tasks, the adaptation to diverse downstream applications, and the incorporation of incremental columns over time. In response to these challenges, we introduce UniTabE, a straightforward yet effective method designed to process tables in a uniform manner, devoid of constraints imposed by specific table structures. UniTabE's core concept relies on representing each basic table element with a module, termed TabUnit. This is subsequently followed by a Transformer encoder to refine the representation. Moreover, our model is designed to facilitate pretraining and finetuning through the utilization of free-form prompts. In order to implement the pretraining phase, we curated an expansive tabular dataset comprising approximately 13 billion samples, meticulously gathered from the Kaggle platform. This research primarily centers on classification and regression tasks involving tabular data, and conducts rigorous experimental testing and analyses to validate the effectiveness of our methodology. The experimental results demonstrate UniTabE's superior performance against several baseline models across a multitude of benchmark datasets. This, therefore, underscores UniTabE's potential to significantly enhance the semantic representation of tabular data, thereby marking a significant stride for tabular data analysis.

Published as a conference paper at ICLR 2024
UNITABE: A UNIVERSAL PRETRAINING PROTOCOL FOR TABULAR FOUNDATION MODEL IN DATA SCIENCE
Yazheng Yang', YuQi Wang', Guang Liu', Ledell Wu Liu
The University of Hong Kong". Beijing Academy of Artificial Intelligence', Creatify All
Pokfulam Road, Hong Kong, China", Zhongguaneun East Road, Beijing. China', California, United States (yangyazh, v @econnect....................... hku. hk, 1 Ligguagg0sai. wttp:n/aaaaaa........mao cn.
dedell@ccee 1eeleleeeiiiiiii fy. ai, liugi 0cs. hku. hk
ABSTRACT
Recent advancements in Natural Language Processing (NLP) have witnessed the ground- breaking impact of pretrained models, yielding impressive outcomes across various tasks, This study seeks to extend the power of pretraining methodologies to facilitating the pre- diction over tables data science, domain traditionally overlooked, yet inherently chal- lenging due to the plethora of table schemas intrinsic to different tasks. The primary research questions underpinning this work revolve around the establishment of a universal pretraining protocol for tables with varied structures, the generalizability and transfer- ability of learned knowledge across tasks, the adaptation to diverse downstream applica- tions, and the incorporation of incremental columns over time. In response to these chal- lenges, we introduce UniTabE, a straightforward yet effective method designed to process tables in a uniform manner, devoid of constraints imposed by specific table structures. UniTabE's s core concept relies on representing each basic table element with a module, termed TabUnit. This is subsequently followed by Transformer encoder refine the representation. Moreover, our model is designed to facilitate pretraining and finetuning through the utilization of free-form prompts. order implement the pretraining phase, we curated an expansive tabular dataset comprising approximately 13 billion samples, meticulously gathered from the Kaggle platform. This research primarily centers on clas- sification and regression tasks involving tabular data, and conducts rigorous experimental testing and analyses to validate the effectiveness of our methodology. The experimen- tal results demonstrate UniTabE's superior performance against several baseline models across a multitude of benchmark datasets. This, therefore, underscores UniTabE's poten- tial to significantly enhance the semantic representation of tabular data, thereby marking significant stride for tabular data analysis.
1 INTRODUCTION
Tabular data, characterized by its structured arrangement of rows and columns, represents foundational element within the field of data science, finding extensive practical utility in diverse domains. It facilitates a spectrum of real-world applications, such as stock market prediction, real estate price forecasting, and credit level estimation, etc. In the context of data analysis, modeling, and decision-making, classification and regression, the predominant tasks of predicting over tabular data, play a central role across diverse industries, attracting considerable attention from the research community.
However, several significant challenges have been identified within this domain: 1) The prevailing focus in the literature centers on bolstering the capabilities of powerful model architectures, often at the expense of relatively simple approaches to feature processing, such as adopting textualization [Song et al. 2019 Herzig
ct al. 2020 Yin et al. 2020a Huang et al. 2020 Gorishniy et al. 2021]. This approach, while effective
certain contexts, tends to overlook the intrinsic nature and numerical significance of tabular data, particularly numerical values. This omission
impede the model's capacity extract meaningful insights from such data, highlighting potential gap in current methodologies [Eisenschlos et al., 2021: Herzig et al. 2020). 2) Recent approaches have turned finetuning large language models (LLMs) pretrained on natural language processing (NLP) datasets, such as TaBERT (Yin et al.] 2020a). Tapas (Herzig et al. 2020 etc. However, LLMs do not inherently excel in handling textualized tabular data, which fundamentally differs from natural

---

Published as a conference paper at ICLR 2024
language texts. Furthermore, these methods often employ simplistic textualization techniques mentioned above that can limit their effectiveness. 3) Prior work exploring pretraining solely on large-scale tabular datasets and assessing transferability is limited. Existing attempts [Wang and Sun 2022] have primarily focused relatively small number of datasets from the same domain, which may not sufficiently validate the model's adaptability across diverse contexts. 4) Existing neural network methods often fall short in per- formance compared eXtreme Gradient Boosting (XGBoost) [Chen and Guestrin) 2016] in data science applications, where XGBoost's high predictive accuracy, efficiency, and flexibility have contributed to its widespread adoption in industry. 5) Many existing approaches require strict consistency in table structures between training and testing data, posing challenges when minor modifications to table structures are nec- essary. This limitation becomes particularly relevant when applying trained models to tables with additional columns, common scenario in practical applications. For example, in clinical trials, incremental columns are collected across different phases.
To address these challenges, we introduce UniTabE, a straightforward yet efficient framework designed to process tables uniformly while accommodating flexible table structures. UniTabE adopts fine-grained ap- proach to feature engineering by processing each cell in the table individually. Motivated by the success of pretraining in NLP, we explore large-scale pretraining over tables in data science, assessing the knowledge transferability and application scalability. We have collected expansive tabular dataset from Kaggle, com- prising approximately 13 billion examples spanning diverse domains. This dataset enables our large-scale pretraining over tabular data. In order accommodate diverse pretraining and finetuning tasks within a unified framework, we introduce a universal training protocol. This protocol involves the utilization of an aulo-regressive decoder, coupled with adaptable free-form prompts. The decoder performs reasoning on the high-level semantic representation produced by the encoder, functioning as an adaptable module capable of task-specific customization through the application of task-specific prompts. To ensure the preservation of a significant portion of pretrained knowledge within the encoder that serves as the foundational module de- signed for tabular data, we employ a shallow decoder. This deliberate choice compels the encoder to assume the primary responsibility for absorbing the intricacies of learning during pretraining. To comprehensively evaluate the effectiveness of our method, we conduct extensive experiments under different scenarios in- cluding predictive tasks on prominent benchmark datasets, filling in missing values, zero-shot prediction, adaptation to incremental tables, and integration of the neural semantic representation with XGBoost, etc.
Our contributions are follow:
We introduce an innovative architectural framework, UniTabE, specifically designed to offer metic- ulous feature processing tailored to tabular data. Furthermore, through the incorporation of free- form prompts into our model, we augment its scalability to an extensive array of tasks for down- stream applications.
We have built an extensive tabular dataset for large-scale pretraining. We introduce an efficient framework for both pretraining and finetuning, optimized to leverage the full potential of our col- lected dataset.
With comprehensive experiments, substantiate the feasibility of pretraining tabular data, underscore the transferability of knowledge acquired, and highlight the substantial performance enhancements it affords in downstream tasks. Our experimental findings also elucidate the practical efficacy of our approach in scenarios such as missing value handling, zero-shot prediction, and adaptability to incremental column structures. Moreover, our method outperforms XGBoost across wide spectrum of benchmark datasets, demonstrating its superiority.
In Sec. 2 we cover related work. Sec. introduces the model architecture of our proposed UniTabE. Sec. gives detailed description of our pretraining and finetuning strategies. Finally, Sec. pirooviees the results of experimental studies which verify the effectiveness of our method.
2 RELATED WORK
2.1 TABLE DATA FEATURIZATION
Tables encompass various data types, and process this tabular data with neural models, conventional methods typically convert each data format into a continuous space using individual strategies. For instance, texts are processed with tokenization and word embedding, while images are handled using image patch
2

---

Published as a conference paper at ICLR 2024
Tabuday Semandi Repussertation
Typo Entbodding
V' V4 ys
Vis
Date Type
Culu Name Vooke"
Ferward
Narra
Cokmmn
Pase Layer
Mobi-Head Attention
Shallow Autoregresiee becader
Cokern Naree Enteodrrg
Linking Laper
You
Colore Valan Errbescing:
Pronpt Tokone
Target Tokess
Cell
Call Ind! Veston
The left part delineates the operational procedure of TabUnit module in processing individual cells. The right part provides an overview of our UniTabE architecture. "n" denotes the number of cells in each example, "Q" denotes the length of prompt, while "T" here represents the length of target. A shallow decoder is applied offering adaptability to a spectrum of diverse downstream tasks.
Figure 1:
embedding. However, prior research tends to simplify this process by treating numerical values as text and then applying the same embedding strategy, which invariably disrupts the original recording structure and numerical meanings [Eisenschlos et al. 2021 Herzig et al., 2020]. Given the prevalence of numerical values
in tables, this area has garnered increasing attention Gooishnny et al.) [022). To enhance the representation of numerical values, MATE [Eisenschlos al., 2021] and TaPas [Herzig al. 2020) introduced a ranking embedding based on numeric rank, which relies on comparisons. Further, TUTA Wang et al. 2021] applied additional numerical features, such as magnitude, precision, the first digit, and the last digit, to distinguish numbers from text. Gorishmiy et al. [2022] attempted to train the embedding vector for numbers. [Wang] and Sun [2022] suggested categorizing tabular data into three distinct types: textual, categorical, and nu- merical. Their model, TransTab, concatenates columns of the same type into text sequence, with column names, column values, and different columns separated by a space. After the concatenation, these three text sequences are fed into the embedding layer individually.
2.2 PRETRAINING TABLE MODELS
In recent years, the pretraining of language models (LMs) over vast text corpora has led to noteworthy hancements in performance for a variety of downstream tasks. This success has stimulated a growing body of work that focuses pretraining and adapting LMs specifically for tabular data. The prevailing method employed by these studies is to fine-tune LMs that have been pretrained on NI.P datasets, such as BERT (De-
vlin al. 2018). BART [Lewis et al., 2019], etc. Typically, this training utilizes the Masked Language
Model (MLM) objective, as evidenced by models like Tabtrr [Huang et al. 2020]. TaBERT (Yin
ct al. 20206 Tabnet [Arik and Plister [02T], Saint (Somepalli et al., 2021], and so on. Liu et al. [022
proposed a modality transformation that converts tabular data into textual data using a basic lexicon and ordered syntax before feeding it into the pretrained LMs. They then finetuned their model using the same MLM objective. However, LMs pretrained tabular data differs fundamentally from natural language texts. The efficacy of finetuned LMs without archi- tectural modifications has largely been confined to text data far. But making modifications might lead to undesired outcomes like catastrophic forgetting knowledge learned from natural language corpora [Chen ct al. 2020 Bavarian et al. 2022). In this work, our focus lies pretraining large model from scratch on tabular data. In line with previous work, we construct our model upon the renowned Transformer encoder, utilizing sulf-supervised objective similar to MIM, which masks parts of the model input and then pre- dicts the masked content. Contrary to previous work, we abstain from textualizing the tabular data using a simplistic strategy. Instead, we introduce TabUnit module designed to process the basic element of a table independently, leading to an improved modeling of tabular data.
natural texts do not perform optimally as the textualized
3 UNITABE ARCHITECTURE
In this section, we present the architecture of our model, which is composed of three primary components: the TabUnit, the Encoding Layer, and Shallow Decoder. Our model employs the TabUnit module serving the foundational feature processor for varying data types before leveraging the Transformer's encoder for
3

---

Published as a conference paper at ICLR 2024
further encoding. Through adopting the setting of prompts and integrating a decoder, our model becomes adaptable wide range of tasks.
TabUnit Module As depicted on the left side of Figure I we propose the use of an unified module, called TabUnit in this work, for modeling the basic element of tabular data. To mitigate the influence of table structure, we treat each cell in a table as a key-value pair representing the column name and the column value respectively. In our implementation, we concatenate the vector of column name and vectors of cell value into a sequence as the cell's representation. To get the column name vector, the tokens of the column name Xen are first passed into the embedding module:
X = ( (xa))
(I)
ddt
(xxa)(())))
where Emb represents the embeddings consisting of word embedding and positional embedding. X.n. is the vector after mean pooling across the dimension token sequence. We map data types Xat into integers (e.g. numerical-+0, categorical--1, and textual-2) and subsequently employ an embedding module EmbDT to obtain the data type embedding vector x Here, the Embpr is specifically designed to help the model adeptly handle diverse data formats, particularly in instances where columns share the same/similar name but contain values in different data formats. For example, the values in the "salary" column of a table in a downstream task might be numerical, while those in the corresponding column of another table could be textual ge. high income, medium income, and low income, etc). In some way, such situation may cause confusion for the neural networks. In order to remedy this problem, our model integrates the data type information into Xen via a fuse layer relied on the gate mechanism:
Idt = S Jixa + byı)) V (X x (1
(2)
9dt Xen Idt * Xdtt
where Wft, by: and fft are trainable parameters. Note that the ratio gat is calculated only conditioning on Xat- Theoretically, integrating an equal amount of data type information into the column name representation across columns of the same data type is reasonable, opposed to computing the fusing ratio gut based on both X.n and Xat that will lead to adding varied amount of data type information. Tokens in each cell are also passed into the embedding module:
(x X x2 _......,
x.....,,--11 x = Emb((= T
2
cm. 2g-11)
(3)
Where the embedding module Emb(.) is shared to carry out embedding for column names and column values, q here denotes the length of the cell value. Given the orderless nature of self-attention within the Transformer encoder, it becomes challenging for the model to learn the connection between the column name and its value when all cells are concatenated into a sequence. As a result, we introduce a Linking Layer to establish the relationship within each (column name, cell value) pair. We employ gated function to weave the information from the column name into its corresponding value. Thus guaranteeing vectors coming from the same pair have higher weights in the self-attention operation make the model aware of the association between column name and values.
Sigmoid Vin ReLUU(WT. + bux)
x Ven
where Wik. bi and Vik are learnable parameters. We employ a to ensure that an equal amount of column name information integrated into all value vectors. This allows the model recognize which parts of vectors are values corresponding to specific column. To avoid having the value information overshadowed by the column name information, we only apply the multiplication operation to a and Ven- Overall, the Tabunit can be briefly formulated as:
XTU = {Ven, Ceu
v2-1) fradunst (Xde, Xens Xev)
(5)
where Xac. Xon and Xex denote the data type indicator, tokens of column name and tokens of column value, respectively. The concatenation of column name vector and value vectors treated as the inner representation of tabular cell. In our implementation, all cells are processed in parallel.
Encoding Layer We concatenate the representations of all cells, and attach a trainable [CLS] vector the head of such sequence. We leverage the Transformer encoder as the encoding layer:
(hats, ho h1 ...,, .,.....h""- N11) = [Enn(Vds X X1 Xtu..., X
(6)
4

---

Published as a conference paper at ICLR 2024
where n is the number of cells, and N is the length after concatenating all cells' representations.
Shallow Decoder During pretraining, want to encourage the encoder to store most of the learned knowl- edge. Hence, we adopt Long Short-Tema Memory network (LSTM) [Hochreiter and Schmidhuber [997]
the weak decoder. Specifically, the hidden state of [CLS] token and the prompt are passed the decoder to compute the initial state our decoder:
tyo V1 -Q-1 = faann( (WT Emb( Lvo :3 :........ ...,,6-11)) W2 (X X XTU })
(7)
Vp Σ yxp(vily.)
(8)
exp(v
Vstate (Vp, hds)
b
2017 W. W2. Vı, W. and b, are train- able parameters. Here vp represents the weighted average of attention states the prompt. The embedding layer of decoder also share the same parameter as that one TabUnit further reduce parameters of the decoder. The target sequence of tokens are generated by the decoder step by step conditioned on the initial state and previously produced token.
where the fatta denotes the dot product attention Vaswani et al.
4 PRETRAINING & FINETUNING
In this section, will commence by presenting the pretraining objectives that consist of multi-cell-masking and contrastive learning, followed by an elaboration on the finetuning methodologies.
4.1 PRETRAINING OBJECTIVE
Multi-Cell Celti-elll-Maskin. Previous research in NLP pretraining utilized self-supervised tasks within datasets provide pretraining signals, such as predicting next token, generating masked spans of texts, and determining the subsequent sentence, etc. These studies have shown that unlabeled data can aid in the learning of seman- tic representation. In this work, we also adopt the mask-then then-predict approach to facilitate eef-ssupr training. practical applications, filling in the entire content of a cell is more useful than merely filling in part of the cell content. As such, we treat each cell as the basic masked unit, as opposed to the token level in NI.P pretraining. We randomly replace the content of the masked cells with the special token, [MASK]. We also [MASK] as the default content for cells whose values are missing.
For each example, we arbitrarily mask columns and train the model to predict the masked content. Often, there may be multiple missing values in downstream tasks. Therefore, we also train the model under condi- tions where several values are missing, to familiarize the model with such situations. The number of masked cells varies during training. The prompt template for pretraining is set to "fill in missing value, <column name> :", which specifies the precise masked column to predict. The details regarding the number of ran- domly masked cells and their corresponding probabilities will be elaborated in $ 5.22 An illustration of the Our model trained with the optimization objective
multi-celll-msking objective is presented in Figure 5 of the maximum log likelihood estimation.
Contrastive Learning. Expect for predicting values of masked cells that focuses on the global information of the table, we can also concentrate on local properties. In various applications, table row represents sample and its columns are features. A natural self-supervise opti- mization signal would be distinguishing the differences across rows. Prior work [Khosla et al.) 2020 Tian et al.) 2020 in computer vision treated the augment of original image as the positive sample, and augments of other im- ages as negative ones. For instance, the augment can be a block of the original image. Inspired by prior work, regard& a subset of row as the compared unit. In Figure 2 block B2 is positive example, B3 and B4 are anchor. B3 and B4 come from different row that is not identical negative examples when B1 is treated We utilize the cosine similarity to measure the closeness between blocks. The training objective is maximizing the similarity between anchor and positive block while
C1
C2
C3
C4
C5
C6
C8
B1
B4
Figure 2: Demonstration of contrastive learning.
(B1, B2) is positive pair, while (B1, B3) and (B1, B4) are negative pairs.
to the row containing the given anchor.
5

---

Published as a conference paper at ICLR 2024
Table Statistics of our pretraining dataset. "Avg# NC". "Avg# CC" and "Avg# TC" signify the average counts of numerical columns, categorical columns, and textual columns, respectively, within each table. The lower section of the table exhibits the Top-5 domains most prominently featured our dataset.
Domains
# Domains
Tables
Examples
Avg# NC
Avg# CC
Avg# TC
ALL
303
283K
13B
28.7
0,4
7.7
Investing
71K
IB
29.33
0.02
1.58
Time Series
65K
1B
6.47
0.02
2.27
Finance
52K
773M
37.57
0.04
1.46
Economics
47K
488M
40.34
0.01
1.27
Games
32K
430M
23.37
0.66
3.66
minimizing it between anchor and negative block. We adopt the overlap between B1 and B2 to guarantee such supervised signal will not make the optimization collapse contributing the robustness for various tables.
4.2 FINETUNING FORMULATION
Filling in Missing Value Prediction. When finetuning our model for downstream tasks, can consider the target as an additional column of the table. The model is then tasked with predicting the masked values of this target column using the same prompt as during pretraining. Thanks to the decoder, UniTabE is capable of generating textual and numerical targets. For example, the trained model is suited for classification and regression tasks, as well as predicting missing values in tables. our implementation, we also support constrained generation. This feature is particularly beneficial for classification tasks, as the model only needs to predict from a small subset of the vocabulary.
Finetuning with Task-specific Prompt. Apart from those tasks where we treat the target the masked column of the table, there are tasks that require the model to perform reasoning over the table and other inputs. For instance, in table question answering (TableQA) tasks, the model needs to produce an answer conditioned on the provided table and question. The prompt used these tasks may include the task instruc- tion and the question, appropriately formatted.
Numerical Categorical
60.5
59.2
36.5
57.4
Mallttlllalaaoo Lall Malilaa
201
3.5
604
37.2
4.1
3.7
(a) Domains Distribution
(b) Data Type Distribution
Distribution visualization. The left part (a) demonstrates the distribution of domains and the number of tables in each domain. Please magnify the figure as some captions are small. The right part shows the proportion (cell level) of different data types in train/dev/test splits.
Figure 3:
5 EXPERIMENTS AND ANALYSES
In this section, we will commence by introducing the pretraining dataset collected from Kaggle. Subse- quently, we will provide detailed exposition of our implementation. Finally, will engage in extensive discussion of the experimental results and analyses, encompassing an assessment of performance in Kaggle benchmarks, public benchmarks, zero-shot prediction, and related aspects.
6

---

Published as a conference paper at ICLR 2024
Table 2:
Performance comparison 12 Kaggle tasks coming from different domains. "UniTabE scratch"
is trained from scratch without pretraining. The result of each method is the average score across five runs.
Classification (AUC f)
Regression (R2 f)
Method/Dataset
PID
RWQ
HFP
HIC
EPL
LDP
MIP
GPP
RSP
CCL
MMP
HPA
XGBoost
0.79
0.67
0.89
0.85
0.73
0.50
0.54
0.64
0.98
0.78
0.71
0.49
TransTab-LSTM
0.81
0.56
0.74
0.85
0.73
0.50
0,50
0.55
0.64
0.65
0.51
0.47
UniTabE scratch
0.76
0.57
0,61
0.85
0.73
0.52
0.53
0.76
0.99
0.72
0.82
0.54
UniTabE finetune
0.83
0.66
0.81
0.86
0.78
0.53
0.75
0.99
0.99
0.96
0.87
0.58
Table Evaluation results with AUC on public tabular datasets, We conducted five training iterations for
each method and subsequently computed the average scores across these runs.
Method/Dataset
CG
CA
DS
AD
CB
BL
10
Avg.
XGBoost
0.78
0.93
0.54
0.91
0.87
0.82
0.71
0.79
NODE (Popov et al.
2019
0.65
0.85
0.58
0.90
0.70
0.83
0.67
0.74
AutoInt Song et al.
2019
0.77
0.93
0.57
0.91
0.82
0.84
0.72
0.79
Tapas [Herzig et al.
2020
0.77
0.92
0.63
0.91
0.87
0.84
0.70
0.81
TaBERT Yin et al.
2020a
0.72
0.88
0.54
0.90
0.71
0.82
0.66
0.75
TabTransformer Fuang et al. 2020
0.76
0.92
0.56
0.91
0.82
0.82
0.71
0.79
FT-Transformer Gorishmy et al.
2021
0.77
0.93
0.58
0.91
0.85
0.84
0.71
0.80
TabNet [Arik and Phster
2021
0.49
0.48
0.48
0.90
0.53
0.79
0.62
0.61
TUTA Wang et al.
2021
0.74
0.92
0.61
0.81
0.74
0.83
0.65
0.76
TransTab Wang and Sun
2022
0.73
0.86
0.52
0.90
0.80
0.71
0.73
0.75
TransTab-LSTM,
0.70
0.85
0.56
0.90
0.72
0.83
0.73
0.76
TabPFN Hollmann et al.
2022
0.79
0.93
0.62
0.88
0.78
0.81
0.70
0.79
GANDALF [Joseph and Raj
2023
0.78
0.92
0.63
0.91
0.84
0.84
0.71
0.80
Llama2 13B [Touvron et
2023
0.77
0.92
0.65
0.91
0.84
0.83
0.68
0.80
UniTabl: scratch
0.76
0.93
0.62
0.91
0.85
0.84
0.74
0.81
UniTabE + XGBoost
0.79
0.93
0.60
0.91
0.88
0.83
0.74
0.81
UniTabE finetune
0.79
0.94
0.66
0.91
0.88
0.84
0.76
0.83
5.1 PRETRAINING DATASET
As there are no large-scale, high-quality tabular datasets available for pretraining, we have collected our own dataset by crawling from Kaggle. We downloaded CSV tables, omitting empty columns. Specifically, we constructed initial keywords for each domain and extended the set of keywords using WordNet under the same topic. We then searched and downloaded tables using these keywords. For tables originating from the same Kaggle dataset, we attempted to join tables using primary and foreign keys. This process resulted in a 7TB dataset containing 13 billion tabular examples. Statistics about the pretraining data are presented in Table [ We also present the statistics of the Top-5 domains in this table. Figure 3 shows the distribution of domains and the proportion of cells of different data types for each split of our dataset. We tried to split our dataset in such way to maintain similar distribution of data types and domains.
5.2 IMPLEMENTATION DETAILS
We train our UniTabE with 32 A100 GPUs in the distributed way. Our model implemented with the PyTorch v1.12. The Transformer encoder used the backbone of our model is borrowed from the hug- gingface "transformers" module. During training, the learning rate is set to be 1e-5, the batch size 64. We Adam [Kingma and Ba 2015) as the optimizer with =099 3,=0.999 and = We train three variants of our model, Uni UniiTablin Eaase UniTabEEargee and UniTabEEterrge The hidden size and embedding 768. Its encoder is the 12 layers of self-attention stack with 12 heads. UniTabEaarge consists of 24 layers encoder, 16 attention heads. Its hidden size and embedding size are both 1024. UniTabEztarge the 48 layers' version of UniTabE More details about implementation of baselines and experimental setting, please refer to Appendix A.3
size for UniTabEsase are both
5.3 RESULTS & ANALYSES
Many practical tabular tasks in the field of data science fall into the categories of classification regres- sion. As such, we evaluate the effectiveness of our pretrained model on these types of tasks. Details about baselines are attached in the Appendix A.l
7

---

Published as a conference paper at ICLR 2024
Kaggle Benchmarks. We selected a diverse set of tasks comprising 6 representative classification assign- ments and 6 regression tasks from Kaggle. To ensure fair and unbiased evaluation, deliberately cluded these specific tabular datasets from our pretraining corpus. This precaution was taken to prevent the pretrained model from gaining undue familiarity with the target columns, thereby avoiding any potential bias in the results. The statistical metrics pertaining to these datasets are delineated Table The comparison of results among methods is presented in Table 2 On most of datasets, notice that our method outperforms XGBoost that is widely used in industry due to its promising performance. Trans Tann-TTM is the model that we equip TransTab with the same decoder as ours. Its vital differences from ours consist of the feature processing and gate mechanism integrated with the Transformer encoder. Compared with our UniTabE achieves higher scores in both classification and regression tasks. It demonstrates the superior- ity of our table feature processor, TnbUnit, as opposed to the table textualization strategy of TransTab. This is indicative of the efficacy of our model's approach to handling and understanding the nuanced structure of tabular data.
Public Benchmarks. Since those 12 tasks coming from Kaggle might have the similar domains to our pretraining data, we also use other public tabular datasets (CG, CA, DS, AD, CB, BL and I0) to further evaluate the efficacy of our method. These datasets are all binary classfication tasks. The statistical data pertaining these datasets is presented in Table We compare our method against previous start-of-the -f--oo---trrrrrr art (SOTA) methods for tabular classification on these datasets. Table presents experimental results on these datasets. Similar to those results in 12 Kaggle tasks, method also achieves impressive results indicating its superiority in learning the intrinsic semantic information of tabular data.
Table 4:
The accuracy of zero-shot classification.
Method/Dataset
CG
CA
DS
AD
CB
BL
10
UniTabE finetune
0.75
0.89
0.62
0.86
0.78
0.80
0.94
Random Initial
0.30
0.41
0.50
0.54
0.41
0.26
0.06
Zero-Shot
0.70
0.56
0.58
0.76
0.57
0.73
0.94
Zero-shot Prediction. Table 4 presents the results of zero-shot prediction. The "Random Initial" ap- proach, which does not load the pretrained parameters, exhibits inferior performance. In contrast, our pre- trained model demonstrates strong performance, even without fine-tuning on the target datasets. indicating its promising generalizability. These results also suggest that UniTabEE acquires certain degree of high-level reasoning capabilities through extensive large-scale pretraining, which contributes to its robust performance in zero-shot settings.
Adaption of Incremental Columns. We want to investi- gate the scalability of various models when confronted with an increasing number of columns. To simulate this scenario, remove k columns from the original training set and train models using the modified data. Subsequently, we perform inference using the unaltered test set. We conduct parative analysis hetween our methodology and Trans'aah LSTM, representative approach that necessitates congru- ence in table structure between the training and testing data. The results of this experiment are displayed in Table 5 Thanks to the flexibility afforded by the TabUnit component of UniTabE, our model exhibits adaptability to the introduction of new columns with relatively minor performance deterioration.
Table 5:
The percentage () of AUC drop facing with incremental columns.
Method/Drop k TransTab-LSTM
1
7.5
11.7
UniTabE scratch
5.3
8.4
UniTabE finetune
3.5
5.8
Model Size Analysis. We want see the impact of the size of UniTabE on performance across different size of dataset. tables for each dataset size are reserved and not included in the pretraining dataset. Here "small" means datasets whose example numbers do not surpass 2000, and "medium" means datasets of which the example numbers are between 2000 and 20000, while "large" denotes those datasets whose data num- bers exceed 20000. The results presented in Figure 4 indi- cate that for larger finetuning datasets, larger models (like UniTabEEzterge tend to perform better. However, for smaller datasets, the performance of larger models tends to decrease. We notice that UniTabbEaagge balance size for different
56-
Small Datner
Dassei
Large Damue:
54.
10-
18
Figure 4: BLEU scores illustrating the generation of textual values across various model sizes and dataset sizes.
8

---

Published as a conference paper at ICLR 2024
datasets. Hence, we apply the UniTabEtarge to compare the performance with baselines in subsequent experiments.
Ablation Analysis. 6 the results from vari- ablation studies. We notice that the omission of the fuse layer from the TabUnit module leads to a decline in per- formance, suggesting the significance of incorporating data type information into the cell representation. Similarly, re- moving the linking layer, responsible for associating column names with column values, makes the model oblivious to the distinction between vectors of column names and values. This is confirmed by notable decrease in the AUC score, at- testing to its efficacy. We further expel the contrastive learn- ing and multi-cell masking objectives separately. A signif- icant performance decline is observed when the former is removed, indicating its vital role in discerning the intrinsic differences within the table. The reduced performance fol- lowing the removal of the multi-cell masking objective fur- ther emphasizes its importance in tabular pretraining. Upon evaluating models pretrained with varying masking ratios, masking ratio of 0.15 yields an average AUC score of 0.83, proving better for the given tasks, while ratios of 0.2 and 0.3 appear inordinate. Excessive masking can impede accu- rate prediction of masked values and hinder differentiation between positive and negative pairs in contrastive learning. Furthermore, substituting UniTabE's 1-layer LSTM decoder with a I-layer Transformer decoder showed the former to be more suitable for the tasks hand. Models with an increasing number of LSTM decoder layers exhibit declining trend in performance, suggesting a simpler decoder enables the encoder to retain more knowledge, leading to enhanced semantic representation of tabular data and superior performance. Additional exploration on the text generation capa- bilities of models with varying decoder layers is presented in Appendix A.5 The task involves querying the model about reasons for loan applications. The results shown in Table indicate that models with additional decoder layers achieve improved BLEU scores, suggesting that more complex decoders absorb a greater portion of the pretrained knowledge, potentially detracting from the encoder. This can subsequently reduce performance in classification tasks that are reliant on the semantic representation of tabular data generated by the encoder.
Table Ablation studies. The average AUC score computed across 7 datasets(CG I0). "MCM" and "CL" denote multi-cell masking and contrastive learning objective.
Method
Avg.
UniTabE
0.83
w/o Fuse Layer
0.77
w/o Linking Layer
0.75
w/o Fuse&Linking
0.72
w/o MCM
0.81
w/o CL
0.79
MCM rate: 0.3
0.8T
MCM rate: 0.2
0.81
MCM rate: 0.1
0.82
3L Decoder
0.8T
6L Decoder
0.79
12L Decoder
0.78
TI. Transformer Decoder
00.8
Learned Feature XGBoost. We are interested in examining the synergistic effects of combining the semantic representation derived from UniTabE with traditional machine learning algorithms, such XG- Boost. We integrate the original features with the generated representation vector, and use this composite data as input to XGBoost. The outcomes are displayed Table 3 These results indicate that the neural feature acts as a beneficial supplement, leading to slight performance enhancement in most instances.
Fill in Missing Value Analysis. Furthermore, we assess the efficacy of our approach concerning capacity to fill in missing values, with the corresponding results displayed in Table TO] The improvement in perfor- mance provides additional experimental support for the effectiveness of our pretrained model in addressing missing values, thereby affirming its potential utility in applications related to data completion, data recovery and tabular data synthesis tasks.
CONCLUSIONS
In this research, we investigate the challenge of pretraining large-scale models specifically on tabular data. We demonstrate the feasibility and scalability of such pretraining. To address the inherent difficulties in pretraining over tabular data, we introduce UniTabE, flexible approach capable of conducting fine-graines feature processing as well as adapting to different tasks with single framework. We have also collected a substantial dataset. comprising 13B examples spanning various domains, for pretraining purposes. To ascertain the effectiveness of our methodology, we carry out extensive experiments across 19 downstream tasks, with the outcomes confirming the superiority of our approach. Additionally, explore issues related to the practical application of our method, including zero-shot prediction. adaptation to incrementally added columns, and combination learned representation and traditional machine learning method (XGBoost).
9

---

Published as a conference paper at ICLR 2024
REFERENCES
S. Ó. Arik and T. Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6679-6687, 2021.
M. Bavarian, H. Jun, N. Tezak, J. Schulman, C. McLeavey, J. Tworek, and M. Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207. 22tii::2227..22555, 2022.
S. Chen, Y. Hou, Y. Cui, W. Che, T. Liu, and X. Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. arXiv preprint arXiv:2004. 12651, 2020.
T. Chen and C. Guestrin. Xghoost: A scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /81000888 2018.
J. M. Eisenschlos, M. Gor, T. Müller, and W. W. Cohen. Mate: Multi-view attention for table transformer efficiency. arXiv preprint arXiv:2109.043 2021.
Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34: 18932-1838, 2021.
Y. Gorishniy, I. Rubachev, and A. Babenko. On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 35:24991-25004, 2022.
J. Herzig, P. K. Nowak, T. Müller, F. Piccinno, and J. M. Eisenschlos. Tapas: Weakly supervised table parsing via pre-training, arXiv preprint urXiv:2004.02349 2020.
S. Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 9(8):1335 1997.
N. Hollmann, S. Müller, K. Eggensperger, and F. Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848, 2022.
X. Huang, A. Khetan, M. Cvitkovic, and Z. Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012..0667, 2020.
M. Joseph and H. Raj. Gandalf: Gated adaptive network for deep automated learning of features, 2023.
J. Kasai, N. Pappas, H. Peng. J. Cross, and N. A. Smith. Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation. arXiv preprint arXiv:2006. 10369, 2020.
P. Khosla, P. Teterwak, C. Wang, A. Sama, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 13:18661-18673, 2020.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceed- ings, 2015.
6980
X. Kong, A. Renduchintala, J. Cross, Y. Tang, Gu, and X. Li. Multilingual neural machine translation with deep encoder and multiple shallow decoders. arXiv preprint urXiv:2206.02079 2022.
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer: Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. arXiv preprint arXiv:1910.11
2019.
Y. Li, Y. Lin, T. Xiao, and J. Zhu. An efficient transformer decoder with compressed sub-layers. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13315-13323, 2021.
G. Liu, Yang, and L. Wu. Ptab: Using the pre-trained language model for modeling tabular data. arXiv preprint arXiv:2209.08066 08060, 2022.
S. Popov, S. Morozov, and A. Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arxiv preprint 190.vv1900..6:1 2019.
10

---

Published as a conference paper at ICLR 2024
G. Somepalli, M. Goldblum, A. Schwarzschild, C. B. Bruss, and Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342, 2021.
W. Song, C. Shi, Z. Xiao, Z. Duan, Y. Xu, M. Zhang, and J. Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th ACM international conference on information and knowledge management, pages 1161-1170, 2019.
X. Sun, T. Ge, F. Wei, and H. Wang. Instantaneous grammatical enror correction with shallow aggressive decoding. arXiv preprint arXiv:2106.0000 04970, 2021.
Tian, C. Sun, B. Poole, D. Krishman, C. Schmid, and P. Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-688 2020.
H. Touvron, L. Martin, K. Stone, Albert, A. Almahairi, Y. Babaei, N. Bashlykov, Batra, Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv2230.009288, 2023.
A. Vaswani, N. Shazeer, Parmar, Uszkoreit, Jones, A. N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Z. Wang and J. Sun. Transtab: Learning transferable tabular transformers across tables. arXiv preprint 200.2220,.09328, 2022.
Z. Wang, H. Dong, R. Jia, J. Li, Z. Fu, S. Han, and D. Zhang. Tuta: tree-based transformers for gener- ally structured table pre-training. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1780-1790, 2021.
P. Yin, G. Neubig, W. tau Yih, and S. Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Annual Conference of the Association for Computational Linguistics (ACL), July 2020a.
P. Yin, G. Neubig, W.t Yih, and S. Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. arXiv preprint ar 2rXiv:2005.083/4 2020b.
11

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 BASELINES
XGBoost [Chen and Guestrin 2016
Despite large number of proposed neural network architectures for tabular data, the performance gap between them and the "shallow" ensembles of decision trees, like XGBoost, often remains significant in industry. We use it as baseline for classification tasks. Implemented based on the XGBoost package.
TransTab [Wang and Sun 2022] also builds upon the Transformer encoder. It splits data into three categories: numerical, binary and categorical. Before being processed by the encoder, columns of the same data type are textualized into a text by inserting space among columns, column names and column values. For instance, textual columns will be organized as the form: "column0 name column0_ value column name columnl _valle.... ... columnk name columnk_ valuee. addition, the numerical values are represented through multiplying with a training vector.
TransTab-LSTM In order to equip TransTab with the ability of text generation for the pretraining objective of multi-cell masking, we integrate the LSTTM decoder of our model into TransTab. Thus, we compare the performance of different tabular data featuring strategies.
Linear Regress is also used to compare the performance for regression tasks. We adopt the Scikit-learn implementation linear regression in the experiments.
Tapas Herzig et al. 2020] - exxennds the pretrained BERT to encompass the encoding of tabular data. This extension is realized through the joint pre-training process that encompasses both textual segments and tabular data extracted from Wikipedia. TAPAS is method devised for addressing questions posed over tables, and notably, it operates without the generation of explicit logical forms. The training procedure for TAPAS is conducted under the paradigm of weak supervision, and its predictive mechanism involves the selection of specific cells within tables, optionally accompanied by the application of an associated aggregation operator to the chosen cell selections.
TabPEN Hollmann et al. 2022
TabPFN, denoting a Prior-Date Fitted Network (PFN), undergoes offline training as one-time process, aimed approximating Bayesian inference. This approximation is applied synthetic datasets generated from a prior distribution that incorporates principles rooted in causal reasoning. encompassing a vast array of structural causal models, with an inherent preference for simplicity. It employs in-context learning (ICL), acquiring the capability to make predictions based on sequences of labeled ex- amples within the input context, all while obviating the need for subsequent parameter updates. This model operates without necessitating hyperparameter tuning, while showcasing competitive performance against contemporary state-of oate--of-hheaatte art classification methodologies.
TaBERT (Yin et al. 2020a] 8 TaBERT is a pretrained language model designed for learning joint represen-
tations of natural language utterances and structured tables. TaBERT can be integrated as a substitute for the original encoder within semantic parser, facilitating the computation of representations for both utterances and table columns.
Llama2 13B [Tovrron et al. 2023 The recent advanced and versatile language model that offers enhanced capabilities, increased language support, and improved performance across a wide range of natural language processing tasks. This state-of-thh of-the-art model leverages the latest advancements in deep learning to provide comprehensive linguistic insights.
TabTransformer (Huang et al. 2020 The TraTransformer architecture is based the Transformer, with its primary objective being the acquisition of effective contextual embeddings for categorical features of tabular data. Its experiments reveal the resilience of contextual embeddings derived from the Tab Tarnrassfrmee
aebboost.sklcary
https//gggttbbooo/////aaaa////nt http:////momooooooccoceesesetcna hrmi#module--kllaaa lincar_model
Shtpss/shhhibbbccomeeeeeboerssss 1-1--hh
hhapss//etthbb.com//uudraaaaaaaa http://nnwottttraaaarrrereeerree pytorch
12

---

Published as a conference paper at ICLR 2024
Statistics of datasets in downstream tasks. "# NC", ; "# CC", "# TC" and "# Examples" denote the number of numerical columns, categorical columns, textual columns of the table and the total number of examples in the dataset, respectively.
Table 7:
Name
# NC
# CC
# TC
Examples
Pima Indians Diabetes (PID)
ur]]
768
Red Wine Quality (RWQ)
[rrl]
11
11
1599
Heart Failure Prediction (HFP) [r]]
299
Health Insurance Cross Scil (HIC)
3 381109
Eligibility Prediction for Loan (EPL)
[r]]]
614
Loan Default Prediction (LDP)
[ur]]
24
7
67463
Medical Insurance Payout (MIP) [rl
1338
Gold Price Prediction (GPP)
url
80
1718
Reliance Stock Price (RSP)
[r]
10
6205 5
Credit Card Limit Prediction (CCL)
url
400
Miami Housing Prediction (MMP)
url
14
3
13932
House Prices Advanced (HPA)
[rr]]
41
1460
Credit-G (CG) [ur]]
11
1000
Credit-Approval (CA) [rr]
690
Dress-Sales (DS) [ur]]
0
11
500
Adult (AD) [url]
48842
Cylinder-Bandee Bands (CB) [ur]]
13
18
540
Blastchar (BL) [ur]]
7043
Insurance-co (1O)
[r]]
83
2
5822
in the face of both missing values and noisy data, thereby enhancing interpretability and demonstrating their robustness.
TT-Transformer Gorishniy et al., 2021 9 An adaptation involves the incorporation of the Transformer architecture specifically tailored for tabular data processing. A distinct Feature Tokenizer is employed transform individual columns into continuous vector spaces. Numerical values are subjected to multipli- cation with training vector, while other non-numeric data tokenized as text, utilizing word embedding techniques for further processing.
GANDALF [Joseph and Raj 2023 deep learning framework, denoted as the Gated Adaptive Network for Deep Automated Learning of Features (GANDALF), which demonstrates its promissing performance, interpretability, and computational efficiency in the context of tabular data analysis. GANDALF incorporates a special tabular processing unit characterized by gating mechanism and integrated feature selection capa- bilities, termed the Gated Feature Learning Unit (GFLU), serving as a fundamental component for feature representation learning.
NODE [Popov et al. 2019 The Neural Oblivious Decision Ensembles (NODE) constitute an emerging deep learning architecture expressly engineered for heterogeneous tabular data. The NODE architecture extends the concept of ensembles composed of oblivious decision trees, harnessing the advantages inher- ent in both end-to-end gradient-based optimization and the potent capabilities of multi-layer hierarchical representation learning.
TabNet [Arik and Pfister 2021 TabNet represents an interpretable canonical deep learning architecture designed specifically for the represention of tabular data. Its core mechanism employs sequential attention to systematically determine the pertinent features consider each decision-making juneture. This strategic feature selection not only enhances interpretability but also optimizes the learning process by allocating learning capacity to the most salient features.
AutoInt (Song al. 2019 AutoInt is automated mechanism designed for the autonomous acquisition high-order feature interactions inherent in input attributes. performs joint mapping of numerical and textual features onto a shared, ceduced-dimensiooall space. Subsequently, a multihead self-attentive
10 https:///hthuboommmydddrrrrrrrre
aevisiting-mmoes
tabular
13

---

Published as a conference paper at ICLR 2024
Table 8:
Ablation results on public tabular datasets (CG ~ 10).
Method/Dataset
CG
CA
DS
AD
CB
BL
10
Avg.
UniTabE finetune
0.79
0.94
0.66
0.91
0.88
0.84
0.76
0.83
w/o Fuse Layer
0.74
0.84
0.60
0.90
0.80
0.83
0.69
0.77
w/o Linking Layer
0.69
0.84
0.61
0.90
0.72
0.81
0.69
0.75
w/o Lune&ninking
0.65
0.85
0.61
0.84
0.65
0.77
0.65
0.72
w/o MCM
0.77
0.92
0.58
0.91
0.88
0.84
'0.5
0.81
w/o CL
0.74
0.84
0.60
0.90
0.90
0.83
0.69
0.79
MCM rate: 0.3
0.79
0.92
0.65
0.90
0.84
0.83
'0.55
0.81
MCM rate: 0.2
0.78
0.92
0.64
0.90
0.82
0.85
0.73
0.81
MCM rate: 0.1
0.79
0.93
0.65
0.91
0.85
0.83
0.75
0.82
3L Decoder
0.75
0.93
0.63
0.90
0.86
0.83
0.76
0.81
6L Decoder
0.74
0.90
0.62
0.89
0.84
0.83
0.73
0.79
12L Decoder
0.74
0.89
0.60
0.90
0.82
0.81
0.69
0.78
IL Transformer Decoder
00.8
0.93
0.62
0.91
0.87
0.84
0.73
0.80
3L Transformer Decoder
0.73
0.88
0.59
0.86
0.83
0.83
0.72
0.78
3L Transformer Decoder
0.72
0.86
0.68
0.85
0.77
0.82
0.69
0.77
12L Transformer Decoder
0.74
0.86
0.60
0.89
0.79
0.81
0.69
0.77
neural network, featuring residual connections, is employed to explicitly capture feature interactions within this low-dimensional representation. Through incorporating multiple layers of the multi-head self-attentive neural network, the model accommodates the modeling diverse orders feature combinations derived from the input attributes.
A.2 DATASETS
The statistical characteristics of each dataset involved in the downstream tasks are provided in Table 7 The table includes the dataset names, along with their corresponding abbreviations, the "Name" column. Additionally, we have included hyperlinks to access each dataset directly within the table for reference. We have deliberately selected these datasets, representing diverse domains (e.g. finance, medicine, healthcare, real estate, credit analysis, etc.), with the primary objective of conducting comprehensive and thorough evaluation of our methodology.
A.3 EXPERIMENTAL DETAILS
The UniTabE is pretrained with ten steps of gradient accumulation. The two pretraining objectives. multi- cell-masking and contrastive learning, are carried out interactively. We adopt the hyperparameter to control [0. 1] for each cell. If q S p. then the corresponding cell will be masked. Masking single cell is used as the backup option in some cases, such as there is no any cell to mask. is set to be 0.15 in this work. We follow prior work [Wang and Sun 2022], the overlap ratio used in this work is set be 50% while conducting contrastive learning, We skip those tables whose column numbers are less than two during training with contrastive learning objective. Furthermore, employ random shuffling procedure to alter the arrangement of columns within the table during the pretraining phase. The batch size is set as while finetuning our model in downstream tasks. The finetune learning rate is 1e-6. We use the same optimizer settings that used during pretraining. For compared baselines, we use the official optimal settings and train these models on different datasets. We then average scores five runs for all model for fair comparison.
the overall proportion of cells to mask. The program generates random number
A.4 ABLATION STUDIES
Table 8 presents the detailed ablation results different datasets.
A.5 TABLEQA CASE STUDY
We have observed a diminishing performance trend in classification tasks as we increase the number of decoder layers in our model. This phenomenon raises the suspicion that a strong decoder may inadvertently
14

---

Published as a conference paper at ICLR 2024
Comparison of text generation ability. The BLEU score between ground truth and generated text. The results reveal that more pretrained knowledge are stored in decoder its layer number grows.
Table 9:
# Layers/Decoder
LSTM
Transformer
IL
24.72
24.91
3L
27.00
27.42
6L
28.82
29.14
12L
29.54
30.16
Table 10: Performance comparison in filling in missing value. "mean/mode" uses the average in numerical column as prediction, and takes the most common text as prediction for textual column.
Regression (MAE J)
Text Generation (BLEU f)
Method/Dataset
CG
CA
DS
AD
CB
BL.
CG
CA
DS
AD
CB
BI.
mean/mode
0.81
0.61
0.84
0.78
0.73
0.84
62
62
43
Lincar Regress
0.64
0.58
0.91
0.45
0.60
0.51
GPT-3
0.58
0.57
0.69
0.64
0.61
0.46
48
66
39
73
71
82
UniTabE scratch
0.49
0.56
0.83
0.69
0.72
0.38
35
63
28
67
18
75
UniTabB finetune
0.40
0.51
0.61
0.43
0.51
0.22
59
76
51
80
85
92
divert some of the pretrained knowledge away from the encoder, potentially compromising the encoder's s capacity for representation. In contrast, a weaker decoder places the onus of modeling knowledge on the encoder during pretraining. Consequently, an encoder trained in collaboration with shallow decoder tends yield better semantic representations. To assess the efficacy of text generation, we constructed table question answering (table QA) dataset based on the CG dataset. In this dataset, we omitted the "purpose" column from the original data and employed the prompt "why the person applies for the loan?" for finetuning our UniTabE model. Table A.5 presents comparative analysis of results across varying decoder sizes. Notably, we observe that the BLEU score exhibits an ascending trend as the number of decoder layers increases. Given the observation that pretrained models equipped with a powerful decoder demonstrate marginally inferior performance in semantic representation, we have opted for the design featuring shallow decoder, aiming to prioritize the storage of acquired knowledge within the encoder, opposed relying on a powerful decoder or applying the balanced encoder-decoder architecture. This phenomenon aligns with findings in prior research [Kasai et al. 2020 (Sun al. 2021 Li al.. 2021 Kong et al., 2022 which further inspired us to take such architecture. For instance, all them adopted Transformer as backbone with shallow decoder. Extensive experiments of Kasai et al.] (2020) and Kong al. [2022] demonstrated competitive performance between the single-layer decoder integrating with strong encoder and the balanced encoder- decoder depth in Neural Machine Translation (NMT). In addition, the LSTM decoder underperforms the Transformer decoder in the generation of target text, underscoring the suitability of the LSTM decoder for deployment weaker component within our model.
A.6 FILL IN MISSING VALUE ANALYSIS
We evaluate the model's capability fill in missing values, utilizing the Mean Absolute Error (MAE) metric for numerical columns and the BLEU score for textual predictions. For comparative analysis, we also fine-tune and test the GPT-3 model. The results, outlined in Table [0] indicate significant improvement over other baseline models. This substantial gain in performance further corroborates the efficacy of our pretrained model in handling missing values, demonstrating its potential for applications in data completion and recovery tasks.
15

---

Published as a conference paper at ICLR 2024
Input:
psvelfilm
RGUt
RCUt
pout
Timestamp
ACM
pavolfilm
psvelfilm
0.88085
-0.02716
-0.46049
906.37207
5.924
136122816.0
288624243.0
5521.72217
1.10703
-0.00098
0.55356
-4164.50488
[MASK]
136124253.0
288622077.0
4150:10059
0.97426
-0.04228
0.62234
-2770.6145
5.428
136124555.0
288621753.0
3317.0603
0.92573
0.04246
-0.57147
549.31641
288807277.
5569 54053
-0.45747
136123573.0
0.01218
6621.72217
288625307.0
6749:7251
0.79271
6.116
4116.43945
0.648
136116864.0
0.01286
288602476.0
3050.28613
-0.08359
0.76895
Prompt:
fill in the missing value, Timestamp:
Target: 5 5.6
Figure 5: Demonstration of multi cell-masking.
A.7 DEMONSTRATION OF MULTI-CELL-MASKING
Figure illustrates the pretraining example of pulti-cell-maskingg We use the prompt template "fill in missing value, <column name> :" to specify the precise masked column to predict.
A.8 KAGGLE DATA LICENSE AND COPYRIGHT
Our utilization Kaggle's dataset for UniTabE pretraining adheres meticulously to Kaggle's terms of and licensing agreements. The publicly available dataset complies with Kaggle's licensing terms, and any specific requirements imposed by dataset providers have been respected. The dataset was col- lected through the official Kaggle API (https://githubb.co wwaebb.co//aaaa Kaggle/kaggle-api,, specifying the license option as "gpl" and "odb". We exclusively collected data under licenses such as GNU General Public License (GPL) and Open Data Commons series (e.g.. ODbL, PDDL, ODC-By), ensuring compatibility for open- sourcing our model. For example, the "sales-from cense (ntps://opendatacommoss org//icceses/ooblll.... -0/), allowing a worldwide, royalty-free, non-exclusive license for data use. Our approach aligns with copyright laws, and we uphold ethical considerations, prior- itizing responsible and fair data use within permitted scope and respecting Kaggle's data privacy policies. We intend to include information on data licensing, copyright, and ethical considerations in the final version our paper if is accepted.
Silias--room-ifffeen-----e different--sorres dataset is obtained under the ODbL li-
16


Reviews:
[{"review_id": "yQ8kn72mof", "reviewer": "Reviewer_3wag", "summary": "The paper proposes a transformer model for table classification and regression. The authors curated a large set of Kaggle datasets for training and evaluation. The central component of the proposed architecture is the TabUnit, which embeds data type, column name and column value respectively, and fuses as well as links the embeddings via gates. The authors propose to use a Transformer encoder and LSTM as decoder. The model is pre-trained by different strategies, namely cell and column masking and siamese triplet loss based on (partial-) row contents. Fine-tuning then entails predicting missing columns for a new dataset specialisied prompts for tasks such as question answering. The authors trained the Transformer on a self-collected dataset comprising 13 billion tables. The approach is evaluated for standard, fine-tuned as well as combined version (with XGBoost). Part of the evaluation is done for Kaggel tasks (12), where the approach is compared to XGBoost and TransTab. The rest of the evaluation is performed for 7 public datasets, where a larger amount of competitors is compared against. The results show that the proposed approach often achieves superior performance in terms of AUC/R2. In addition, multiple ablation studies also highlight, among others, robustness of the approach towards missing data.", "questions": "Will get code / the trained model be made available?\n\nWhy was the evaluation only in part conducted with all competitors?\n\nWhat are the most challenging datasets you tested the method on?\n\nWhat are other advantages of the proposed approach next to pure predictive performance? Is the method's robustness superior to the other competitors as well?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The chosen topic of the paper is highly relevant to the venue, as tabular machine learning is, as argued in the paper, often overlooked but highly important in industrial applications.\n\nThe proposed approach to learn designated embeddings for data type, column name and values within the TabUnit is sensible and is different to existing approaches. In addition, the authors collect a large dataset for training their Transformer. It would be value adding to open-source such a model.\n\nThe work incorporates a large number of competitors in part of the evaluation (on open datasets): The authors include more than 10 baseline approaches of recent papers to compare the performance of their approach on a selected sample of open datasets. The chosen baselines, here, are well-chosen. The authors also provide ablation studies wrt to ther model design, highlighting the implications of omitting parts of the architecture. Lastly, the predictive performance of the approach is often better than the competitors.", "weaknesses": "It is unclear why only part of the evaluation was done for all Transformer-based competitors and why no regression results for a XGBoost regressor have been generated. It would have been highly insightful to have the same evaluation as for the open datasets.\n\nIt would also have been insightful to compare to a representative AutoML approach in order to see the boundaries of the approach. Of course, it does not have to be assumed that the model has to beat the best tuned models, but it should come close enough. \n\nWhile the results are promising, the gap between the competitors is often rather small. This is not to reduce the quality of the predictive performance, as the task at hand is very difficult and competitive, it just shows that competitors are already strong. \n\nWhile part of the evaluation is exhaustive in terms of baseline approaches, the evaluation contains relatively little open datasets compared to some of the competing papers. It would be beneficial to add more thorough design decisions to why the datasets have been chosen or why they are sufficiently representative.\n\nThe ablation study with respect to robustness is very interesting and highlights another benefit of the method - being able to deal with missing data. It would be intersting to know if this result holds over the other competitors too.\n\nA minor point (as many competitors were looked at): It would have intersting to see the performance of TUTA, as similar to the proposed approach, more refined embeddings are learned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BTQIIV4rr8", "reviewer": "Reviewer_es5d", "summary": "This paper presents UniTabE - a universal pretraining method for tabular data that consists of three components: tabular unit decomposition, encoding, and decoding. Compared to prior work, a key novelty of this model is that it is trained from scratch, and it involves innovations in terms of developing a TabUnit splitter and a simple decoder. The compilation of a large dataset for model training is also an interesting undertaking. The evaluation section provides a large number of experiments, showing the improvement brought by UniTabE on fine-tuned and zero-shot classification, regression tasks, and missing data prediction, whereas a series of ablations show the benefit of each aspect of the method.", "questions": "Q1) Given that a pretty large amount of data from Kaggle is used for pretraining of UniTabE, I would appreciate it if the authors could comment on the aspect of data licensing, copyright, and ethics of use. Also, what would the use of this model as a foundational model for tables entail given the specific tables used for pretraining?\n\nQ2) Can the authors motivate the reason for selecting certain baselines for each experiment, especially for tables 4 and 5?\n\nQ3) How does UniTabE handle tables with more complicated structures, where the organization of columns and rows is less obvious?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1) The paper is well-written and easy to follow. It provides information that should suffice to reproduce the presented method.\n\nS2) The methodological contributions are clearly indicated and well-justified by the paper. \n\nS3) The paper contributes a novel dataset and set of models, that should be interesting to practitioners working with tables.\n\nS4) The experimental section shows the benefit of this method against competitive baselines. Ablation studies are provided to ensure that the improvement comes from the expected method components.\n\nS5) To my knowledge, the undertaking to create such a foundational model for tables is novel. I agree with the authors that it can be very impactful given the amount of tabular data online.", "weaknesses": "W1) The paper lacks a discussion of the ethical and legal aspects of the data, including biases, copyright, and licensing.\n\nW2) The proposed method has a strong performance against the baselines consistently, but the baseline choices are inconsistent across the experiments. Can the authors motivate the reason for selecting certain baselines for each experiment, especially for Tables 4 and 5?\n\nW3) The paper talks about generalization to tables with more columns, but the generalization to tables with more complicated structures (e.g., multi-row and multi-column blocks) is not discussed. I am wondering how are such tables handled by the UniTabE.\n\nMinor:\n* Figure 3a is impossible to read - please enlarge the figure for the follow-up version, and ideally provide more guidance/commentary about it in the paper.\n* Typo: \"Expect for \" -> \"Except for\"\n* The commentary of Table 3 (page 9) is unclear, as it refers to \"a slight performance enhancement\", even though the two versions seem to perform more-or-less on par.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PTOvHDjbcQ", "reviewer": "Reviewer_KBNh", "summary": "The paper delves into the application of large language models (LLM) to tabular data, presenting a novel approach utilizing the transformer model.\n\nTo address the unique characteristics of tabular data, the authors introduce \"TabUnit\", a method to amalgamate data type, column name, and cell values into comprehensive cell vectors. These vectors are then processed through transformer-based encoders and fed into a shallow LSTM decoder to regenerate tokens for the target label column.\n\nUpon pre-training the UniTabE model using 13B samples from Kaggle, the authors fine-tuned it on 12 chosen classification and evaluation datasets. The results reveal a significant improvement over prior methods that lack pre-training.", "questions": "1. Table 3 Variants:\nThe descriptions of the three UniTabE variants presented in Table 3—namely scratch, +XGBOOST, and fine-tune—appear to be absent. Notably, UniTabE scratch seems to rival the performance of UniTabE fine-tune and even surpasses UniTabE+XGBOOST in many instances. Given that UniTabE has been exposed to 13B tabular samples, this outcome raises questions. What factors might account for this similarity in performance, especially between the scratch and fine-tuned versions? Moreover, would broadening the evaluation across a more diverse set of datasets provide deeper insights into the tangible benefits of UniTabE pre-training?\n\n2. Llama2 13B Training:\nRegarding Llama2's tabular prediction training, how exactly was the model trained to handle tabular data predictions?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Quality:\nThe paper's quality stands out as it presents a model pre-trained on an impressively large tabular prediction dataset encompassing 13B samples. The authors conducted a meticulous evaluation of the model with contemporary tabular prediction baselines.\n\n2. Significance:\nThe results offered by the authors are particularly enlightening. While they tout the enhanced performance derived from pre-training on expansive tabular data, a closer examination (as reflected in Table 3) indicates only a slight uplift in downstream task performance post-fine-tuning, especially given the model's exposure to 13B samples. This subtle observation underscores the intricate challenges posed by tabular data prediction, taking into account its inherent heterogeneity and noise. It's pivotal to recognize that these findings could potentially delineate a performance benchmark for similar BERT-inspired methods in tabular prediction. This could galvanize researchers to delve deeper into innovative tabular modeling paradigms and re-evaluate data processing strategies for tabular prediction.", "weaknesses": "1. Evaluation:\nGiven the vastness of the training data, with 13B samples spanning 303 diverse domains, the evaluation datasets chosen seem somewhat narrow in scope. The comprehensive variety of training domains calls for an equivalently diverse evaluation to genuinely test the model's versatility and adaptability. Authors should consider widening their evaluation scope by leveraging an even more diverse array of tabular datasets.\n\n2. Results:\nWhile Table 3 does indicate that UniTabE marginally outperforms other baselines, the incremental gain appears to be disproportionate to the extensive computational overhead of the pre-training phase. It's noteworthy that the performance between a ground-up UniTabE (from scratch) and a pre-trained plus fine-tuned UniTabE are remarkably similar, prompting questions about the efficiency of the elaborate pre-training process.\n\n3. Method:\nThe paper's tab unit processing technique, although novel in its implementation, seems to draw heavily from precedents like TransTab and FT-Transformer. Additionally, the supervised strategies, such as multi-cell masking and contrastive learning, bear significant resemblance to methodologies detailed in prior research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)