PTaRL: Prototype-based Tabular Representation Learning via Space Calibration
Authors: Hangting Ye
Wei Fan
Xiaozhuang Song
Shun Zheng
He Zhao
Dan dan Guo
Yi Chang
Venue: ICLR 2024 spotlight (2024)
Keywords: Tabular data
Deep neural networks
Tabular representation learning
Prototype learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: To construct prototype-based projection space and learn the disentangled representation around global tabular data prototypes.

Abstract: Tabular data have been playing a mostly important role in diverse real-world fields, such as healthcare, engineering, finance, etc.
With the recent success of deep learning, many tabular machine learning (ML) methods based on deep networks (e.g., Transformer, ResNet) have achieved competitive performance on tabular benchmarks. However, existing deep tabular ML methods suffer from the representation entanglement and localization, which largely hinders their prediction performance and leads to  performance inconsistency on tabular tasks.
To overcome these problems, we explore a novel direction of applying prototype learning for tabular ML and propose a prototype-based tabular representation learning framework, PTaRL, for tabular prediction tasks. The core idea of PTaRL is to construct prototype-based projection space (P-Space) and learn the disentangled representation around global data prototypes. Specifically, PTaRL mainly involves two stages: (i) Prototype Generating, that constructs global prototypes as the basis vectors of P-Space for representation, and (ii) Prototype Projecting, that projects the data samples into P-Space and keeps the core global data information via Optimal Transport. Then, to further acquire the disentangled representations, we constrain PTaRL with two strategies: (i) to diversify the coordinates towards global prototypes of different representations within P-Space, we bring up a diversifying constraint for representation calibration; (ii) to avoid prototype entanglement in P-Space, we introduce a matrix orthogonalization constraint to ensure the independence of global prototypes. 
Finally, we conduct extensive experiments in PTaRL coupled with state-of-the-art deep tabular ML models on various tabular benchmarks and the results have shown our consistent superiority.

Published as a conference paper at ICLR 2024
PTARL: PROTOTYPE-BASED TABULAR REPRESENTA- TION LEARNING VIA SPACE CALIBRATION
Hangting Ye', Wei Fan', Xiaozhuang Song', Shun Zheng', He Zhao', Dandan Guol,, , Yi Chang School of Artificial Intelligence, Jilin University' University of Oxford The Chinese University of Hong Kong, Shenzhen' Microsoft Research* CSIRO's Data61 yeht2118@mails jlu. edu. en, wei. fan@wrh. ac. u'k xiaozhuangsongie wink....... cuhk. en, shun zheng@mi crosoft. com, he. zhao@ieeee org, (guodandan, yi richangg fil eew.euuuuuuuuueeeeem cn
ABSTRACT
Tabular data have been playing a mostly important role in diverse real-world fields, such as healtheare, engineering, finance, etc. With the recent success of deep learning, many tabular machine learning (ML) methods based on deep networks (e.g.. Transformer, ResNet) have achieved competitive performance on tabular benchmarks. However, existing deep tabular MI. methods suffer from the rep- resentation entanglement and localization, which largely hinders their prediction performance and leads to performance inconsistency come these problems, we explore a novel direction of applying prototype learn- ing for tabular ML and propose prototype- based tabular representation learning framework, PTARL, for tabular prediction tasks. The core idea PTARL is to construct prototype-based projection space (P-Space) and learn the disentangled representation around global data prototypes. Specifically, PTARL mainly in- volves two stages: (i) Prototype Generation, that constructs global prototypes the basis vectors of P-Space for representation, and (ii) Prototype Projection, that projects the data samples into P-Space and keeps the core global data information via Optimal Transport. Then, to further acquire the disentangled representations, we constrain PTARL with two strategies: (i) to diversify the coordinates towards global prototypes of different representations within P-Space, we bring up diver- sification constraint for representation calibration; (ii) to avoid prototype entangle- ment in P P-Space, independence of global prototypes. Finally, we conduct extensive experiments PTARL coupled with ttate-f-thheart deep tabular ML models on various tabular benchmarks and the results have shown our consistent superiority.
tabular tasks. To over-
introduce matrix orthogonalization constraint to ensure the
INTRODUCTION
Tabular data, usually represented as tables in a relational database with rows standing for the data samples and columns standing for the feature variables (e.g., categorical and numerical features), has been playing more and more vital role across diverse real-world fields, including healthcare (Her-
nandez al. 2022). engineering (Ye et al.] (003). advertising (Frosch al.] (01O), finance (Assefa
et al. 2020p. etc. Starting from traditional machine learning methods (e.g., linear regression (Su
al. 2012), logistic regression (Wright 1995)) to tree-based methods (e.g. XGBoost (Chen &
Guestrin 2016), LightGBM (Ke et al. 2017)), tabular machine learning has received broad atten-
tion from researchers for many decades.
More recently, with the great success of deep networks in computer vision (CV) (He et al. 2016) and natural language processing (NLP) (Devlin et al. 2018). numerous methods based on deep learning accomplish tabular prediction tasks (Song
have been proposed for tabular machine learning (ML)
et al. 2019
Huang et al.) 2020 Gorishniy et al. 2021 Wang et al. 2021). For example, (Song
al. 2019 proposed AutoInt based transformers.
(Gorishniy al. 2021 further improved
*Corresponding authors.

---

Published as a conference paper at ICLR 2024
AutoInt through better token embeddings, and (Wang al. 2021 proposed DCN V2 that consists of an MLP-like module and a feature crossing module.
While the recent deep learning solutions have performed competitively on tabular benchmarks (Gor- ishniy et al. 202T; Shwartz-Ziv Ziv & Armon 2022), there still cxists an performance inconsistency tabular tasks: existing state-of-the-art deep tabular ML models (e.g.. FT- Transformer (Gorishniy et al. 2021). ResNet (Gorishniy et al., (021)) cannot perform consistently well on different tasks, such as regression, classification, etc. We investigate the learned patterns of deep tabular ML models and identify two inherent characteristic hindering prediction: (i) rep- resentation entanglement: the learned representations of existing deep tabular methods are usually entangled and thus cannot support clear and accurate decision maaiing, and (ii) representation local- ization: each data sample are represented distinctively, making the global data structures over data samples are overlooked.
in their predictions
To better overcome the aforementioned challenges, we explore the direction of applying prototype learning for tabular modeling, and accordingly we propose PTARL, prototype-based tabular rep- resentation learning framework for tabular ML predictions. The core idea of PTARL is to develop protolype-based projection spuce (P-Space) for deep tabular ML models, in which the disentan- gled representation!" around pre-defined global prototypes can be acquired with global tabular data structure to enhance the tabular predictions. Specifically, our PTARL mainly involves two stages, (i) Prototype Generating and (ii) Representation Projecting. In the first stage, we construct K global prototypes for tabular representations, each of which is regarded the basis vector for the P-Space to stimulate disentangled learning for more global data representations. We initialize the global pro- totypes with K-Means clustering (Hartigan & Wong 1979) facilitate the efficiency of prototype learning. In the second stage, we project the original data samples into P-Space with the global prototypes to learn the representations with global data structure information. To learn the global data structure, we propose a shared estimator to output the projected representations with global prototypes; besides, we propose to utilize Optimal Transport (Peyré et al. 2017) to jointly optimize the learned representations in P-Space with global prototypes and original representations by deep tabular models, to preserve original data structure information.
In addition to employing global prototypes, we propose two additional strategies to further dis- entangle the learned representations in PTARL: (i) Coordinates Diversifying Constraint motivated by contrastive learning that diversifies the representation coordinates of data samples in P-Space to represent data samples in a disentangled manner, and (ii) Matrix Orthogonalization Constraint that makes the defined global prototypes in P-Space orthogonal with each other to ensure the in- dependence of prototypes and facilitate the disentangled learning. In brief, our contribution can be summarized follows:
We investigated the learned patterns of deep tabular models and explore novel direction of ap- plying prototype learning for tabular machine learning to address representation entanglement and localization.
We propose a model-agnostic prototype-based tabular representation learning framework, PTARL for tabular prediction tasks, which transforms data into the prototype- based projection space and optimize representations via Optimal Transport.
We propose two different strategies, the Coordinates Diversifying Constraint and the Matrix Or- thogonalization Constraint to make PTARL learn disentangled representations.
We conducted extensive experiments in PTARL coupled with state-of- oef-the-rrt (SOTA) deep tabular ML models on various tabular benchmarks and the comprehensive results along with analysis and visualizations demonstrate our effectiveness.
2 RELATED WORK
Deep Learning for Tabular machine learning, Starting from statistical machine learning methods (e.g., linear regression (Su et al. 2012). logistic regression (Wright 1995)) to tree-based methods
In our paper, "disentangled representations" means the representations more separated and discrimina- tive for supervised tabular modeling tasks, which is different from disentanglement deep generative models.
2

---

Published as a conference paper at ICLR 2024
(e.g. XGBoost (Chen & Guestrin 2016), LightGBM (Ke al. 2017)), traditional machine learn- ing methods are broadly used for tabular machine learning. More recently, inspired by the success of of deep learning in computer vision (CV) (He et al. 2016) and natural language processing (NIP) (Devlin et al., 2018), numerous methods based on deep learning have been proposed for tab-
ular machine learning to accomplish tabular prediction tasks (Song et al. [209 Huang al. 2020
Gorishniy et al.] |021 Wang et al. 2021). Among these works, [Wang et al. (2021) proposed DCN
V2 that consists of an MLP-like module and feature crossing module; AutoInt (Song et al. 2019)
leveraged the Transformer architecture to capture inter-column correlations; FT- Ti-Transformer (Gor-
ishniy al., (021) further enhanced AutoInt's performance through improved token embeddings; ResNet for tabular domain (Gorishniy et al.] (02T)) also achieved remarkable performance. How- ever, these methods may fail to capture the global data structure information, and are possibly af- fected by the representation coupling problem. Therefore, they cannot perform consistently well on different tasks, c.g. regression and classification. Recently, another line of research has tried to use additional information outside target dataset to enhancing deep learning for tabular data. TransTab Wang & Sun 2022) incorporates feature name information into Transformer to achieve cross table learning. XTab (Zhu et al., 2023) pretrains Transformer a variety of datasets to enhance tabular deep learning. Different from this line, PTARL does not need additional information outside target dataset. Note that PTARL, as general representation learning pipeline, is model-agnostic such that it can be integrated with many of the above deep tabular ML models to learn better tabular data representations.
Prototype Learning. Typically, a prototype refers to a proxy of class and is computed as the weighted results of latent features of all instances belonged to the corresponding class. Prototype- based methods have been widely applied in a range machine learning applicaitons, like computer vision (Yang et al., 2018 Li et al., 202 I Nauta et al. 2021 Zhou et al. 2022), natural language processing Huang et al. 2012 Devlin et al. 2018: Zalmout & Li 2022). In the field of CV, pro- totype learning assigns labels to testing images by computing their distances to prototypes of each class. This method has been proven to make model to be more resilient and consistent when dealing with few-shot and zero-shot scenarios (Yang al. (018)) Likewise, in the field of natural language processing (NLP), taking the mean of word embeddings as prototypes for sentence representations has also demonstrated robust and competitive performance on various NLP tasks.
The mentioned approachs generally employ the design of prototype learning to facilitate the sharing of global information across tasks, enabling rapid adaptation of new tasks (Huang al.] 2012 Hoang et al./ [2020] Li et al. 2021 Zhou et al. 2022). Similarly. in tabular deep learning. the global information of data samples is crucial for inferring labels of each data sample (Zhou et al. 2020 Du et al. 2021). This inspired us to incorporate prototype learning into our proposed framework for capturing global information and leveraging it to enhance the tabular learning performance.
3 BACKGROUND
Notation. Denote the i-th sample as (I,, yi). where x; = (ax 22.(00) ((att) E X represents numerical and categorical features respectively and Y/ W is the corresponding label. A tabular dataset D = (X,Y) is a collection of n data samples, where X = [== and Y = {y: He- We use Drreen to denote training set for training, Dvel to denote validation set for early stopping and hyperparameter tuning, and Drest to denote test set for final evaluation. Note that in this paper we consider deep learning for supervised tabular prediction tasks: binary classification Y = {0, 1).
(num)
multiclass classification Y = =1,,.,,c) c} and regression Y =. R. The goal is to obtain an accurate deep tabular model ((:6): : X Y trained on Drrain.
Optimal Transport. Although Optimal Transport (OT) possesses rich theoretical foundation, we focus our discussion solely on OT for discrete probability distributions, please refer to (Peyré et al,, 2017) for more details. Let us consider p and as two discrete probability distributions over an arbitrary space E Rdd which can be expressed as p = Σn= a,o2, and q = Σ byōy.- In this case, a E Σ" and b E Σm" where Σ" represents the probability simplex in R". The OT distance between p and g is defined as:
OT(p,4) = -min C), TeH((p-))
(1)
3

---

Published as a conference paper at ICLR 2024
where (:-) is the Frobenius dot dot-product and C 20 is the transport cost matrix constructed by Cjj = Distance(z; y;). The transport probability matrix T E 1220 which satisfies 1l(p, q) {T T. bj. أ= Túj = a.). is learned by minimizing (T, C). Directly optimizing Eq.
often comes at the cost of heavy computational demands, and OT with entropic regularization is introduced allow the optimization at small computational cost in sufficient smoothness (Cuturi
2013
4 PROPOSED METHOD: PTARL
Layers
Lavers
Layers
Figure 1: The visualization of representations of deep network w/o and w/ PTARL with varying model layer depths.
4.1 MOTIVATION AND OVERALL PIPELINE
In the context of tabular data, the intrinsic heterogeneity presents a challenge for achieving satisfac- tory performance using deep models. As shown in the first subfigure of Fig. [ the latent representa- tions learned by the FTT-Tranffrrmer (Gorishniy et al. 202T) (one of the SOTA deep tabular models) on a binary classification dataset Adult (Kohavi et al. 1996) are entangled. To validate whether is caused by the limitation of model capacity, we gradually increase "TT-Transformer' ' layer depths and visualize the corresponding latent representation by T-SNE (Van der Maaten Hinton 2008). As shown in the first row of Fig. with sequential increase in the number of model layers, we could observe the representation entanglement phenomenon has not be mitigated while gradually aug- menting the model capacity. In addition, our empirical observations indicate that this phenomenon also exists for other deep tabular models and we recognize that the representation entanglement the inherent limitation of deep models in tabular domain. Moreover, the learned representations also lack global data structure information, which failed to model the shared information among the data instances. Compared other domains like CV and NL.,, especially in the heterogeneous tabular domain, samples overlook the statistical global structure information among the total dataset would drop into representation localization. Furthermore, recent researches (Gorishniy al. [021]] Ziv & Armon 2022) show that different types of data may require varying types of deep models (e.g. Transformer based and MLP based architecture).
To address the aforementioned limitations of deep models for the tabular domain, we apply proto- type learning into tabular modeling and propose the prototype-based tabular representation learning (PTARL) framework. Note that PTARL, as general representation learning framework, is model- agnostic such that can be coupled with any deep tabular model F(:: to learn better tabular data representations in our redefined Prototype-based Projection Space, which is the core of PTARL. In the following, we will elaborate on the specific learning procedure our PTARL in Section 4.2 then, we will provide two constraints to further constraint PTARL for representation calibration and prototype independence in Section 4.3 As shown in the second row Fig. D with PTARL, the latent space is calibrated to make the representation disentangled. Fig, gives overview of the proposed PTARL. Before the introduction of PTARL, we first present the formal definition of the crase projection space as follows:
4

---

Published as a conference paper at ICLR 2024
Fradiction
Prodiction
Legotd:
Projection Head (a(:0p)
Projection Read (2(:0))
LEbi
Pcocatspar
Hidden hepcesentation Gy(K.0y)
2,'4 %
Representation
Joototypes orthopmasitntiio
Projection Representation
Brotengee
Blackbox Backbone
Gy(:0)
Calibration
(uLa Pratsport
Bata:X.Y
Blackbox Backbone
Gy(:0)
6)(x:9)
The PTARL framework. The original representation of each sample by backbone would be pushed forward to the corresponding projection representation by minimizing the Optimal Transport Distance. The sentences "coordinates diversifying" and "prototypes orthogonalization" correspond to two constraints for representation disentanglement.
Figure 2:
Definition 1. Prototype-based Projection Space (P-Space). Given set of global prototypes = {,3, E g-K'd xd where K is the number of prototypes and 8k is the representation of define the P-Space
the k-th prototype with d as the hidden dimension, consisting of the global prototypes as the basis vectors for representation. For example, given representation in P-Space with coordinates denoted in P-Space can be formulated as Ex= Bk.
projection space
[r"]K k== IR Kx1,, each representation
4.2 PROTOTYPE-BASED TABULAR REPRESENTATION LEARNING (PTARL)
PTARL consists of the following stages: (i) prototype generating, that constructs global pro- totypes of P-Space, and (ii) representation projecting, that projects ((:;0)+ representations into P-Space to capture global data structure information via optimal transport.
Global Prototypes Initialization. To start with, let us rewrite the deep tabular network F(: 0) as Gn Gn ()), where Gy(:;y)) is the backbone and G) 0x) is the head parameterized by 0y and 0 respectively. To obtain global prototypes initialization, we first train F(::0) by:
min task Y)
min L'(G,, 0y.0%
On),,
where L' is the loss function, and then propose applying K-Means clustering (Hartigan & Wong 1979) to the output of the trained backbone (y;X;0)):
Σ min [[G 0y) - -; TCI, subject to j 1 = (E(0.))
min Ke
where 1x E RRK centroid matrix C' would be served as the initialization of prototypes B to guide the stage (ii) train- ing process of PTARL. The advantage prototypes lies in (a) enabling the preservation of global data structure information from the trained F(:0) in stage (i), despite the presence of noise, and (b) facilitating faster convergence of the stage (ii) training process compared to random initialization.
vector of ones, ys is the centroid index and C is the centroid matrix. The
using K-Means to generate the initialization of global
Representation Projecting with Global Data Information via Optimal Transport. After con- structing the global prototypes B. we re-initialize the parameters of F(=;) and start to project F(::))s representation into P-Space via PTARL to capture global data structure information. To obtain the projection of i-th instance representation Gz(z;; By). with learnable to calculate its coordinate r; by (G,( 0y); Mathematically. our formulation
shared estimator o(::7)
for the i-th projection representation distribution takes the following form: Q: EK r. 5 On the other hand, the i-th latent representation G (y) could be viewed as the empirical distribution over single sample representation: P: dc, (x,;9))))
Since all samples are sampled from the same data distribution, it is reasonable to assume that there exists shared structural information among these samples. To capture the shared global data structure
5

---

Published as a conference paper at ICLR 2024
information, formulate the representation projecting the process of extracting instance- wise data information by G (2 8y) to P.. and then pushing P. towards Qi to encourage each prototype 3& to capture the shared global data structure information, process facilitated by leveraging the optimal transport (OT) distance:
min projeccing (X, B) min 1 Σ OT(P; Qi) min B T2
min (, Ci), TEEI(P...
(4)
where C the transport cost matrix calculated by cosine distance and T is the transport probability matrix, please refer to 33ction[3oor more details about OT. We provide the more detailed explanation of the optimization process of Eq. in Appendix A.8 In contrast to the latent space of F(; 0), the P-Space consists of explicitly defined basis vectors denoted as global prototypes, thus it is more accurate to give predictions on representations within P-Space. We incorporate the label information by re-defining Eq.
K
min Maakk((
min 0
1 n.
L r Br:00)) On),
Σ Rr
(5)
i=1
By representing each projected representation using the shared basis vectors (global prototypes B) within the P-Space, we can effectively model this global data structure information to effectively solve one of the deep tabular network's s inherent characteristic: sample localization. The representa- tion entanglement problem still exists within P-Space, and we will introduce the designed constraints to make the representation disentangled the following section.
4.3 CONSTRAINTS FOR PTARL VIA COORDINATES DIVERSIFYING AND PROTOTYPE MATRIX ORTHOGONALIZATION
P-Space Coordinates Diversifying Constraint. Directly optimizing the above process could po- tentially result in collapse of coordinates generating, which entails all projection representations exhibiting similar coordinates within P-Space. To alleviate it, we design a constraint to diversifying the coordinates towards global prototypes within P-Space separate the representation into several disjoint regions, where representations with similar labels are in the same regions. Specifically, for classification, we categorize samples within the same class as positive pairs (Khosla et al.) 2020), while those from different classes form negative pairs. In the context of regression, we segment the labels within a minibatch into sub-intervals indicated by 1 + log(n,), where ny is bach- size. Samples residing within the same bin are regarded as positive pairs, while those from distinct bins are regarded as negative pairs, thereby facilitating the formation of pairings. This process is achieved by minimizing:
1h
78
exp (cos(ra,r))
Ldiversifying
Σ
1 (1r,3) E positive pair) log
(6)
Σラ exp ccos(T;,r,))
This constraint is motivated by contrastive learning (CL) (Khosla et al. 2020). Distinct from con- ventional contrastive learning methods in tabular domain that directly augment the variety of sample representations (Wang& & Sun 2022. Bahri et al., 2021). diversifies the coordinates of latent rep- resentations within P-Space to calibrate the entangled latent representations. In this context, the use of prototypes as basis vectors defines a structured coordinate system in the P-Space, thereby facilitating the enhancement of generating disentangled representations among samples, opposed to directly optimizing their representations. To improve computational efficiency, practically, we randomly select 50% of the samples within a minibatch. We posit that this approach can mitigate computational complexity while makes it casier to approximate the model to the optimized state.
Global Prototype Matrix Orthogonalization Constraint. Since the P-Space is composed of set of global prototypes, to better represent the P P-Space, these global prototypes should serve as the basis vectors, with each prototype maintaining orthogonal independence from the others. The presence of interdependence among prototypes would invariably compromise the representation efficacy of these prototypes. To ensure the independence of prototypes from one another, the condition of orthogonality must be satisfied. This mandates the following approach:
[M
min L Lorthogonalicaaioo ( = min
+ maxx0, K |
M
2

---

Published as a conference paper at ICLR 2024
where M [0, [0, and M, = (3. 3,)11. The first term
I forces the M be sparse, I
i.e. any element M, = || cos (3,, 8,)11 I to be close to O (3, and Bj is orthogonal) or 1, while the second term motivates M|] - K, i.e., only K elements close to Since Ma cos( (3;, B:) 1, Vi E f1.2.... .... }, it would force My = || cos(3; (3) [1, Vi E f1, K},2 + to be 0, i..
each prototype maintains orthogonal independence from the others.
5
EXPERIMENT & ANALYSIS
5.1 EXPERIMENT SETUP
Datasets. As described before, it is challenging for deep networks to achieve satisfactory perfor- mance due to heterogeneity, complexity, and diversity of tabular data. In this paper, we consider variety of supervised tabular deep learning tasks with heterogeneous features, including binary classification, multiclass classification, and regression. Specifically, the tabular datasets include: Adult (AD) (Kohavi et al.) (196)) Higgs (HI) (Vanschoren al. (2014). Helena (HE) (Guyon al.) 2019). Jannis (JA) (Guyon et al.) 2019). ALOI (AL) (Geusebrock et al. (2005). California housing (CA) (Pace& & Barry] 1997). The dataset properties summarized in Appendix A. We split each dataset into training, validation and test set by the ratio of 6:2:2. For the data pre- nee-rocesssngg please
refer to Appendix for more details.
Baseline Deep Tabular Models. As the PTARL is a model-agnostic pipeline, we include 6 main- stream deep tabular models test PTARL's applicability and effectiveness to different predictors with diverse architectures, which are as follows: MLP (Taud & Mas 2018), ResNet (He et al.
2016), SNN (Klambauer et al. (017)) DCNV2 (Wang et al. 2021). AutoInt (Song et al. 2019), and FT-Transformer (Gorishniy et al. 202T). More details could be found in Appendix A.2
PTARL Details. The PTARL is a two-stage model-agnostic pipeline that aims to enhance the performance of any deep tabular model F-::0) 0) without altering its internal architecture. The first stage is to construct the core of PTARL, i.e. P-Space, that consists of set of global prototypes B. The number of global prototypes K is data-specific, and we set K to the ceil of log(N), where N the total number of features. The estimator (d::7), which is used to calculate the coordinates of representations within P-Space, is a simple 3-layer fully-com collneccnde MLP. To ensure faimess, in the second stage of training, we inherit the hyperparameters of F((:0) (the learnable would be re-initialized). We provide the PTARL workflow in Appendix A..3 Following the common practice of previous studies, we use Square Error (MSE) (lower is better) to evaluate the regression tasks, Accuracy (higher is better) to evaluate binary and multiclass classification tasks. To reduce the effect of randomness, the reported performance is averaged over 10 independent runs.
5.2 EMPIRICAL RESULTS
PTARL generally improves deep tabular models' performance. From Table we can observe that PTARL achieves consistent improvements over the baseline deep models in all settings. achieves more than 4% performance improvement for all settings, whether using Accuracy or RMSE as the evaluation metric. In addition, we conduct Wilcoxon signed-rank test (with o = 0.05) [Woolson] 2007) to measure the improvement significance. In all settings, the improvement of PTARL over deep models is statistically significant at the 95% confidence level. This demonstrates the superior adaptability and generality of PTARL to different models and tasks. In addition, the results also indicate that there is no deep model that consistently outperforms others on all tasks, i... the universal winner solution does not exist, which is aligned with the findings in previous works (Gorishniy et al., 2021).
Ablation results. We further conduct ablation study to demonstrate the effectiveness of key com- ponents of PTARL. Specifically, we conduct a comparison between the deep model coupled with PTARL and three variants: (i) PTARL w/o O, that removes the global prototype matrix orthogonal- ization constraint, (ii) PTARL w/o O, D, that further removes the P-Space coordinates diversifying constraint and (iii) w/o PTARL, that is identical to directly training deep models by Eq. 2 The results in Table 2 show that the removal of any of the components degrades the performance of PTARL. The comparison between PTARL w/o O, D and w/o PTARL indicates that by projecting deep tabular model's representation into P-Space, the shared global data structure information are
7

---

Published as a conference paper at ICLR 2024
Table I: Tabular prediction performance of PTARL over different deep tabular models for different tasks. "f" represents higher evaluation metric is better for classification, "J" represents lower evaluation metric is better for regression. The best results are highlighted bold. "Win" represents number datasets that one scheme achieves the best.
ML.P
+PTaRL
DCNV2
+PTaRL
SNN
+PTaRT.
ResNet
+PTaRL
AutoInt
+PTaRL
FT-Trans
+PTaRL.
AD
0.825
0.868
0.826
0.867
0.825
0.859
0.813
0.862
0.823
0.871
0.827
0.871
HI 0.681
0.723
0.681
0.731
0.69
0.724
0.682
0.729
0.685
0.738
0.687
0.738
HE
0.352
0.396
0.34
0.389
0.338
0.394
0.354
0.399
0.338
0.396
0.352
0.397
JA 7
0.672
0.71
0.662
0.723
0.689
0.732
0.666
0.723
0.653
0.722
0.689
0.738
A
0.917
0.894
0.955
CA 0,518
0,489
0.502
0,465
0.898
0.631
0.537
0,498
0.507
0.464
0,486
0,448
Win
0
0
0
6
0
Table 2: Ablation results on the effects of different components PTARL. The experiment is conducted on FT-Transformer. The best results are highlighted in bold.
AD
HI
HB f
JA
AL
CA
Win
FT-Transformer- PTARL
0.871
0.738
0.397
0.738
0.97
0.448
PTARL w/o O
0.859
0.722
0
0.725
0.96
0.453
0.704
0.369
Wo PTARI.
0.827
0.687
0.352
0.689
0.924
0.486
captured by global prototypes to solve the representation localization to enhance the representation quality. In addition, diversifying the representation coordinates in P-Space and orthogonalizing the global prototypes of P-Space could both enable the generation of disentangled representations to al- leviate the representation entanglement problem. Besides, we also provide additional ablation tests, including validating the effectiveness of K-Means the global prototypes initialization method and the effectiveness of Optimal Transport (OT) as the distribution measurement in Table 3 The result of initialization method is aligned our explanation for the advantage of K-Means in Section 4.2) In addition, the result of distribution measurement indicates that compared to Manhattan distance and Euclidean distance, OT measures the minimum distance between two distributions through point by point calculation, which could better capture the data structure between two distributions. Due to the space limit, we leave more details and results about the above ablation study to Appendix
Computational efficiency and sensitivity analysis. In paper, while Optimal Transport (OT) demonstrates strong computational capabilities for measuring the minimum distance between two distributions, it does not significantly increase computational complexity. We include the computa- tional efficiency details in Appendix A..5 In addition, we also incorporate the sensitivity analysis for the weights of different loss functions in Appendix [A.6] Since the core of PTARL is the constructed P-Space, we further explore PTARL's performance under different global prototypes number in Appendix A.6
5.3 PTARL PROCEDURE ANALYSIS AND VISUALIZATION
PTARL enables the generation of disentangled representation. Fig. 3 shows the learned repre- sentation of deep model F(::;) f) w/ and w/o PTARL. on binary and multiclass classification tasks. The first row shows that different deep models suffer from the representation entanglement problem and this is aligned with our motivation. With PTARL, representations within P-Space are separated into several disjoint regions, where representations with similar labels are in the same regions. This
Table 3: Ablation results the effects of different prototype initialization and distribution measurement in PTARL. The experiment is conducted on FT-Transformer. The results are highlighted in hold.
Initialization
AD
HI 1
HE
JA
AL +
CA L
(Ours)
0.397
0.738
PTARL w/ Random
0.863
0.721
0.389
0.729
0.951
0.462
Distribation measurement
AD
1 0.738
HE
JA
AL
L
PTARL w/ OT (Ours)
0.871
0.397
0.738
0.97
0.448
PTARL w/ Manhatian distance
0.859
0.729
0.382
0.721
0.953
0.454
PTARL w/ Euclidean distance
0.862
0.713
0.390
0.724
0.949
0.459
8

---

Published as a conference paper at ICLR 2024
MIP
FT-Trandtormer
MIP
FT-Tranformme
Figure 3: Visualization of learned representations of deep tabular models w/ and w/o PTARL.
demonstrates PTARL' ability generate disentangled representations for tabular deep learning. PTARL enables P-Space coordinates diver- sifying. To validate whether the learned coor- dinates of representation within P-Space have been diversified, we average the coordinates of data points which are belong to the same category and visualize them in Fig. With- out the Diversifying Constraint, coordinates of different categories appear similar, but incor- porating the constraint enhances the diversity between category coordinates. PTA RL en- ables the orthogonalization of global proto- types within P-Space. To explore the struc- ture of constructed P-Space, we visualize the relation between any (3,,,3, (3)) by calculating || (Os( B1)]11 in Fig. 5 Each prototype main- tains orthogonal independence from the others, which demonstrates the effectiveness of proto- type matrix orthogonalization constraint.
Grod Proouges Figure 4: P-Space coordinates diversifying visualiza- tion of FT-Trassformer Diversifying Constraint (D). The first column see- ond column correspond to the average coordinates val- ues of two different categories, the third column repre- sents the difference of the two categories.
Gobe Potmos Inte: Olda hrop
HI w/o and w/ Coordinates
11:00
60
3:0
0.0
0.90
1.00
1.00
022
1.8
8:2
0.90
0.00
0.00
3.6
66
0.5
8:77
1:81
1.m
0.34
631
0.0
1.59
0.91
3.79
700
1.00
0/71
0.00
8.00
8:08
0.59
0.00
0.00
1.00
0.90
0.25
0.00
W/ ortoggonatiwe
W/ otthogoneization
aethogeraltxtion
Figure 5: Global prototypes orthogonalization visualization of MLP two different tasks.
CONCLUSION
In this paper, we have investigated the learned patterns of deep tabular models and identified two inherent representation challenges hindering satisfactory performance, i.c. sample localization and representation entanglement. To handle these challenges, we proposed PTARL,, a prototype-base tabular representation learning pipeline that can be coupled with any deep tabular model to enhance the representation quality. The core the PTARL is the constructed P-Space, that consists set of global prototypes. PTARL mainly involves two stages, i.e. global prototype generation and projecting representations into P-Space, to capture the global data structure information. Besides, two constraints are designed disentangle the projected representations within P-Space. The em- pirical results on various real world tasks demonstrated the effectiveness of PTARL for tabular deep learning. Our work can shed some light developing better algorithms for similar tasks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (No.61976102, No.U19A2065, No.62306125) and Key R&D Program of the Ministry of Science and Technology, China (2023YFF0905400,
REFERENCES
Jason Altschuler, Jonathan Niles Weed, and Philippe Rigollet. Near-linear time approximation al- gorithms for optimal transport via sinkhorn iteration. Advances in neural information processing systems, 30, 2017.
Samuel A Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert E Tillman, Prashant Reddy, and Manuela Veloso. Generating synthetic data in finance: opportunities, challenges and pitfalls. Proceedings of the First ACM International Conference Al in Finance, pp- 1-8, 2020.
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervieed contrastive learning using random feature corruption. arXiv preprint arXiv:2106.15147 2021.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Lenaic Chizat, Pierre Roussillon, Flavien Léger, François- Xavier Vialard, and Gabriel Peyré. Faster wasserstein distance estimation with the sinkhorn divergence. Advances in Neural Information Processing Systems, 33:2257-2269, 2020.
Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. Advances in neural information processing systems, 26, 2013.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXivv/810..44 2018.
Lun Du, Fei Gao, Xu Chen, Ran Jia, Junshan Wang, Jiang Zhang, Shi Han, and Dongmei Zhang. Tabulamet: neural network architecture for understanding semantic structures of tabular data. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp- 322-331, 2021.
Dominick L Frosch, David Grande, Derjung M Tarn, and Richard L Kravitz. A decade of contro- versy: balancing policy with evidence in the regulation of prescription drug advertising, American Journal of Public Health, 100(1):24-32 2010.
Jan-Mark Geusebrock. Gertjan Burghouts, and Amold WM Smeulders. The amsterdam library of object images. International Journal of Computer Vision, 61:103-112, 2005.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:1 14418993--8 2021.
Isabelle Guyon, Lisheng Sun Hosoya, Mare Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengy- ing Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michèle Sehag, al. Analysis of the automl challenge series. Automated Machine Learning, 177, 2019.
John Hartigan and Manchek A Wong. Algorithm as 136: k-means clustering algorithm. Journal 1979.
of the royal statistical society. series e (applied statistics),
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Mikel Hernandez, Gorka Epelde, Ane Alberdi, Rodrigo Cilla, and Debbie Rankin. Synthetic data generation for tabular health records: systematic review. Nearocomputing, 493:28-44 45, 2022.
10

---

Published as a conference paper at ICLR 2024
Nghia Hoang, Thanh Lam, Bryan Kian Hsiang Low, and Patrick Jaillet. Learning task-agnostic embedding of multiple black-box experts for multi-task model fusion. International Conference Machine Learning. 1 0144282-42922 2020.
Eric H Huang, Richard Socher, Christopher Manning, and Andrew Y Ng. Improving word rep- resentations via global context and multiple word prototypes. Proceedings of the 50th annual meeting of the association for computational linguistics (Volume I: Long papers), pp- 873-882, 2012.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06677 2020.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightghm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances neural information procexsing systems, 30, 2017.
Ron Kohavi et al. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. KDD, 96:202-207, 1996.
Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. Adap- tive prototype learning and allocation for few-shot segmentation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8334-8343, 2021.
Meike Nauta, Ron Van Bree, and Christin Seifert. Neural prototype trees for interpretable fine- grained image recognition. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14933-144 14943, 2021.
Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics Probability Letters, 33 (3):291-299 297, 1997.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Center for Research in Eco-
nomics and Statistics Working Papers, pp. 2017-86, 2017.
Ravid Shwartz. and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang- Autoint: Automatic feature interaction learning via self-attentive neural networks. Proceedings of the 28th ACM international conference on information and knowledge management, pp. 1161- 1170, 2019.
Xiaogang Su, Xin Yan, and Chih-Ling Tsai. Linear regression. Wiley Interdisciplinary Reviews: Computational Statistics, 4(3):275-294 2012.
Hind Taud and JF Mas. Multilayer perceptron (mlp). Geomatic approaches for modeling land change scenarios, pp. 451-455, 2018.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49-66 60, 2014.

---

Published as a conference paper at ICLR 2024
Ruoxi Wang. Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. Proceedings of the web conference 2021. pp. 1785-1797, 2021.
Zifeng Wang and Jimeng Sun. Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35:2902-2915, 2022.
Robert F Woolson. Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials, pp. 1-3, 2007.
Raymond E Wright. Logistic regression. American Psychological Association, 1995.
Hong-Ming Yang, Xu-Yao Zhang, Fei Yin, and Cheng-Lin Liu. Robust classification with convo- lutional prototype learning. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1474-3482, 2018.
Hangting Ye, Zhining Liu, Xinyi Shen, Wei Cao, Shun Zheng, Xiaofan Gui, Huishuai Zhang, Yi Chang, and Jiang Bian. Uadb: Unsupervised anomaly detection booster. arXiv preprint arXiv:2306.01999,, 2023.
Nasser Zalmout and Xian Li. Projotyperepresennatiins for training data filtering in weakly- supervised information extraction. pp. 467-474, 2022.
Mengyu Zhou, Wang Tao, Ji Pengxin, Han Shi, and Zhang Dongmei. Table2analysis: Modeling and recommendation of common analysis patterns for multi-dimensional data. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):320-328, 2020.
Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmen- tation: A prototype view. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2582-2593, 2022.
Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. arXiv preprint arXiv:2305. 200502300.000000 2023.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
In this section, we provide details of the datasets, baseline methods, the implementation our method, comprehensive experimental results and visualizations. More detailed information is avail- able at https:://gggtiiie com
A.1 DATASETS DETAILS.
Data pre-processing. Due to the property of neural networks, data pre-processing is important, especially for tabular data. To handle categorical features, we adopt integer encoding scheme, where each category within a column is uniquely mapped to an integer to index the embedding in lookup table. Furthermore, maintain consistent embedding dimensions for all categorical features. For numerical features, we apply column- normalization method. In regression tasks, also apply the normalization the labels. To ensure fair comparisons, adhere to identical preprocessing procedures across all deep networks for each dataset. Following (Gorishniy et al.) 2021), we use the quantile transformation from the Scikit-learn library (Pedregosa al. 2011). We apply Standardization to HE and AL. The latter one represents image data, and standardization is common practice in computer vision.
Table Tabular data properties. Accuracy used for binary and multiclass classification, RMSE denotes Root Mean Square Error for regression.
AD
HI
HE
JA
AL
CA
Objects 48842
98050 28, 0 2
65196
83733 54, 0
108000
20640
Numerical, Categorical
27,
128,
8,0
Classes metric
2
100 Accuracy
1000
Accuracy
Accuracy
Accuracy
Accuracy
RMSE
A.2
BASELINE DEEP TABULAR MODELS DETAILS.
MLP Taud & Mas
2018
DCNV2 WWang al.
2021). Consists of MLP-like module and the feature crossing module (a combination of linear layers and multiplications).
SNN Kllambauer et al. 2017). An MLP-like architecture with the SELU activation that enables training deeper models.
ResNet (He al. 2016). The key innovation is the use of residual connections, also known as skip connections or shortcut connections. These connections enable the network to effectively train very deep neural networks, which challenging before due the vanishing gradient problem. In this paper, we use the ResNet version introduced by (Gorishniy et al., 2021).
AutoInt (Song et al., 2019). Transforms features to embeddings and applies a series of attention based transformations to the embeddings.
FT- Trannffffrrer (Gorishniy et al. 2021). FT- FT-Transformeri is introduced by (Gorishniy et al. 2021 further improved Autolnt through better token embeddings.
A.3 PTARL WORKFLOW.
13

---

Published as a conference paper at ICLR 2024
Algorithm PTARL algorithm workflow.
Input: Input data D = {X, Y). deep tabular model F(:;)) Gn. (G) coordinates
estimator ((::7)
1: Phase 1: prototype generating
2: Train F(:; by min taak (X, Y) ming, 0, L(G, (G; 0,), ,) (Eq.2)
3: Obtain global prototypes B = {Bx1 K E RKxd xx through applying K-Means clustering to the
output of the trained backbone (x;)00g)
4: Phase 2: representation projecting
5: Re-initialize the parameters of F(::;) Gn. (G)(::0):0k)
6: while B, 0 0n, 7 has not converged do
7: Sample minibatch of size n, from D
8: for + to n2 do
9:
Obtain i-th instance representation distribution by P: 6,
Calculate i-th instance projection representation coordinates r; =
11: Obtain i-th instance projection representation distribution: Q: = Σk= r dg.
Calculate the OT distance between P. and Q;: OT(P,, Q.) mi Q.) (Ti C,)
13: end for
Average the OT distance within the minibatch compute Eprojecting (X,B)
Ene OT(P,, Qi) ng E m.) C₁) (Eq.4)
Compute Liask (X, Y) Re E d'(Gr( Br; 0x). y:) (Eq.5)
Random select 50% of the samples within minibatch to compute Laiverss
-=1 1(11, E positive pair) log exp (cos(ri.ry))
Σ.e Ene exp (cos(r:.75)) (Eq.6)
17:
Compute Lortthogoaaizzation (B)
M A
max(O, K [|M]))) (Eq.7)
18:
Update B, f 0n. by minimizing Lorthogonalization through gradient descent
Liask + Lprojecting + Ldiversifying +
19: end while
20: return Gý, Gh, B.B
14

---

Published as a conference paper at ICLR 2024
A.4 COMPARISON WITH BASELINE DEEP NETWORKS.
Table 5: Ablation results on the effects of different components in PTARL.
AD f
HI f
HE
JA 1
AL1 1
CA 4
Win
MLP + PTARL
0.868
0.723
0.396
0.71
0.964
0.489
PTARL w/o
0.856
0.711
0.381
0.703
0.955
0.496
PTARL w/o O
0.838
0.696
0.364
0.683
0.937
0.511
w/o PTARL ResNet PTARL
0.825
0.681
0.352
0.672
0.917
0.518
0.862
0.729
0.399
0.723
0.964
0.498
5
PTARL w/o O PTARL w/o O w/o PTARL FT-Transformer + PTARL
0.848
0.712
0.383
0.707
0.949
0.493
0.83
0.698
0.368
0.69
0.933
0.518
0.813
0.682
0.354
0.666
0.919
0.537
0.871
0.738
0.397
0.738
0.97
0.448
PTARL O PTARL w/o O, D w/o PTARL
0.859
0.722
0.383
0.725
0.96
0,453
0.841
0.704
0.369
0.702
0.94
0.466
0.827
0.687
0.352
0.689
0.924
0.486
A.5 COMPUTATIONAL EFFICIENCY DETAILS.
To approximate the optimal transport (OT) distance between two discrete distributions of size n, the time complexity bound scales n² log(n) e² reach e-accuracy with Sinkhorn's 's algorithm, as demonstrated by (Chizat et al. 2020 Alischuler et al.) 2017). In this paper, for each instance I; we pushing G (zz; 0y)'s representation distribution P; the corresponding projection representation distribution Q, by minimizing their OT distance. Thus the instance-wise time complexity bound scales O R log( ()) where is set to the number of global prototypes. The number of global prototypes K is data-specific, and we set K to the ceil of log(N), N), where N is the total number of features. While OT demonstrates strong computational capabilities for measuring the minimum distance between two distributions, it does not significantly increase computational complexity in our setting.
We also compare the computational cost of PTARL single GTX 3090 GPU. We report the computational cost (s) of PTARL per training epoch on different datasets. As shown Table [ coupling deep models with PTARL produces better performance all datasets with acceptable cost.
Table 6: Computational cost (s) per training epoch for PTARL.
AD
HI
HE
JA
AL
CA
MLP + PTARL phasel
0.230
0.482
0.312
0.402
0.520
0.107
MLP + PTARL phase2
1.030
2.435
0.909
1.204
1.469
0.543
DCNV2 + PTARL
phase
0.436
1.000
0.653
0.840
1.081
0.210
DCNV2 PTARL
phase2
1.217
2,831
1.159
1.485
1.934
0.599
PTARL
p
0.257
0.506
0.343
0.446
0.555
0.117
SNN + P.
phase2
1.025
2.371
0.926
1.162
1.528
0.533
ResNet + PTARL phasel
0.413
0.919
0.616
0.783
1.019
0.196
ResNet + P phase2
1.131
2.697
1.167
1.451
1.908
0.568
AutoInt + PTARL phasel
0.655
2.278
1.394
3.575
7.278
0.304
AutoInt PTARL phase2
2.009
9.976
5.578
22.884
23.444
0.707
FT FI-Trrasffrmmr- PTARL phasel
0.828
2.367
1.489
3.130
9.194
0.418
FT-Transformer + PTARL phase2
1.579
4.176
1.976
3.798
9,962
0.804
15

---

Published as a conference paper at ICLR 2024
A.6 SENSITIVITY ANALYSIS.
The inftuence of loss weights. Altogether, the proposed PTARL aims to minimize the following objective function in stage 2:
Lusk (X,Y) + prrjeccting (X, [) + Ldivers([ying(() + ( dorthogonalization (B)
The weights of above losses
1.0, 0.25, 0.25 and 0.25 respectively and the weights fixed in all settings. We provide the sensitivity analysis for the loss weights conducted on FT-Transformer in Table 7 The results indicate that PTARL robust the loss weights.
Table 7: Sensitivity analysis for loss weights.
Loss weights
AD t
HI
HE
JA 1
AL f
CA
1.0, 0.25, 0.25, 0.25
0.871
0.738
0.397
0.738
0.97
0.448
1.0, 0.25, 0.25, 0.2
0.871
0.740
0.398
0.737
0.97
0.449
1.0. 0.25, 0.25, 0.15
0.870
0.739
0.397
0.737
0.97
0.448
1.0, 0.25, 0.25, 0.1
0.869
0.737
0.395
0.736
0.969
0.450
1.0, 0.25, 0.25, 0.05
0.864
0.734
0.391
0.731
0.964
0.451
1.0, 0.25, 0.25, 0.0
0.859
0.722
0.383
0.725
0.96
0.453
1.0, 0.25, 0.25, 0.25
0.871
0.738
0.397
0.738
0.97
0.448
1.0, 0.25, 0.15, 0.25
0.869
0.737
0.397
0.736
0.969
0.451
1.0, 0.25, 0.05, 0.25
0.869
0.737
0.395
0.734
0.969
0.452
1.0, 0.25, 0.0, 0.25
0.861
0.724
0.379
0.722
0.961
0.456
The influence of global prototype num- ber to PTARL. Since the core of PTARL the constructed P-Space, we further ex- plore PTARL's performance under different global prototypes number. Fig. 6 shows the results of average performance of PTARL and its variants over all deep models. The PTARL 's performance gradually improves with global prototypes number increasing, and finally reaches stable level. Thus it is reasonable to set the number of prototypes K the ceil of log(N), where N the total number of features.
TE
betie)
3344
Fial 2.30
Pars (aniiig
0.4
FLEbOr
rumber
Figure 6: Tabular prediction performance of the variants of PTARL with various numbers of prototypes. The results averaged over all baseline deep tabular models under binary classification (AD), multiclass classification (HE) and regression (CA).
A.7 ADDITIONAL RESULTS.
Table The influence of DNN depths to the performance of PTARL.
Datasets
Models
layers 3
layers =
layers =
layers
MLP
0.680
0.707
0.675
0.669
MLP+ PTARL
0.719
0.730
0.729
0.732
HI f
FT-Transformer
0.687
0.719
0.709
0.691
FT-Transformer PTARL
0.738
0.742
0.741
0.737
MLP MLP+ + PTARL
0.670
0.715
0.704
0.689
0.708
0.732
0.728
0.733
JA t
FT-Transformer
0.689
0.716
0.729
0.709
FT-Transformer + PTARL
0.738
0.741
0.745
0.742
MLP MLP + PTARL
0.522
0.513
0.509
0.524
0.491
0.480
0.482
0.479
CA
FT-Transformer
0.486
0.476
0.472
0.478
FT-Transformer PTARL
0.448
0.446
0.442
0.443
16

---

Published as a conference paper at ICLR 2024
A.8 DETAILED EXPLANATION OF THE OPTIMIZATION PROCESS OF EQ.
The Optimal Transport (OT) problem is usually to find the most cost-effective way to transform a given distribution to another distribution, which typically achieved by calculating the specified transportation plan that minimizes the total transportation cost, while the minimized cost is usually called OT distance. In our paper, we minimize the distance between original representation distri- bution over each sample P. by deep tabular models and the corresponding projection representation distribution Q, in P-Space with global prototypes, in order to preserve original data information (of P.) in Qt We follow the typical setting of OT problem to first estimate the transport plan to obtain the OT distance between P; and Qi Then, the obtained OT distance is further used as loss function to jointly learn the two representations.
Specifically, after initializing the global prototypes B of P-Space, we project the original data sam- ples into P-Space to learn the representations with global data structure information. To better illus- trate the optimization process, we revise the Eq. 4 in the original paper to make it more readable. In Eq. 4, the i-th sample representation by deep tabular model is denoted as Gz fy). the em- pirical distribution over this sample representation is P. = 6G (x,,0))))) the projection representation
distribution is denoted as: Q, = Ek= rf d... where r; is coordinates. To capture the shared global data structure information, we formulate the representation projecting as the process of extracting instance-wise data information by G (x=;; ) to P, and then pushing P. towards Q: to encourage each prototype B& to capture the shared global data structure information, a process achieved by minimizing the OT distance between P; and Qi- The OT distance between P. and Q; could first be calculated by: OT(P;, Q;)
Ci), where Cik the average OT distance between P; and Qi over train sets could be viewed as loss function Eprojecting (X, be further optimized: min Eprojecting (X, B) min 1 n "/=1 OT(P, Qt) = ming, 8 >>= min,,, (T,, C2), we use gradient descent to update 7 7,
cos(GG 0y), Bx),
A.9 DETAILED DESCRIPTION OF THE global data structure information AND sample localization.
In the context of any tabular dataset, we have observed global data structure information comprises two different components: (i) the global feature structure and (ii) the global sample structure.
Considering the feature structure, traditional and deep tabular machine learning methods utilize all features or a subset of features as input, allowing them to model inherent interactions among features and thereby acquire comprehensive global feature structure. In addition, there also exists the sample structure given a tabular dataset. Traditional methods (e.g., boosted trees) can effectively model the overarching relationships between data samples. Specifically, in XGBoost, the dataset undergoes partitioning by comparing all the samples, with each node of a decision tree representing specific partition, and each leaf node corresponding to predictive value. The iterative splitting of nodes during training empowers decision trees in XGBoost to learn the distribution of all the samples across distinct regions of the data space, capturing global sample structure.
However, we note that deep tabular machine learning methods typically rely on batch training to obtain data representations within batch. These methods do not explicitly consider the structure between samples within a batch. Furthermore, they fail to capture the global structure between samples across different batches. This limitation presents challenges in comprehensively capturing global data distribution information, consequently impeding overall performance.
Our methods rebuild the representation space with global prototypes (P-Space) in the first stage. Then in the second stage, the original data representation by deep tabular machine learning methods projected into P-Space to obtain projection representation with global prototypes. On the one hand, by minimizing the Optimal Transport distance between the two representations, we could rep- resent each sample with global prototypes, and in the meanwhile encode the global feature structure in learning global prototypes, considering backbone models can inherently learn the interactions among features. On the other hand, the global prototypes are learned by directly modeling all the data samples and thus the complex data distribution could be obtained by global prototypes to cap- ture the global sample structure. Therefore, PTARL is able to capture both the feature and sample structure information by prototype learning. Considering previous deep tabular machine learning
17

---

Published as a conference paper at ICLR 2024
methods only acquire the representations limited by the batch training, the concept of sample localization to encapsulate this limitation.
A.10
GLOBAL PROTOTYPES ORTHOGGONALIZATION VISUALIZATION.
CA
1.0
1.0
HE
1.0
1.00
0.09
3.07
0.30
0.20
1.00
0.07
0.09
0.00
0.03
orchogonaliztiion
1.00
0.18
0.03
0.5
0.09
1.00
1.00
0.40
0.27
0.07
1.00
0.04
0.07
0.02
0.7
0.6
0.6
0.18
1.00
1.00
0.21
1.00
0.08
0.5
0.4
0.30
0.60
1.00
0.03
1.00
0.16
0.2
0.2
1.0
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.8
orthogonaliztionn
0.00
1.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
1:00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
044
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
1.00
0.2
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
2.00
Figure 7: Global prototypes orthogonalizatiion visualization of MLP.
CA
HE
1.0
1.0
1.0
1.00
0.34
0.13
0.29
0.29
1.00
0.29
0.33
0.24
0.23
1.00
0.39
0.03
orthogonalizttion
0.8
0.8
0.34
1.00
9.02
0.38
0.01
0.29
1.00
0.31
0.19
0.08
0.6
0.6
0.39
1.00
0.33
0.02
1.00
0.00
0.87
0:33
0.31
1.00
0.02
0.10
0..4
0.4
0.29
0.30
1.00
0.03
0.19
0.02
1.00
0.37
0.04
0.2
0.2
2.00
0.00
0.00
0.00
0.00
orthogonaliztiion
1.00
11.00
0.00
0.8
0.00
1.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
1:00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.4
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
2.00
Figure 8: Global prototypes orthogonalization visualization of ResNet.
18

---

Published as a conference paper at ICLR 2024
0.9
0.35
0.9
0.8
0.8
1.00
0.64
1.00
0.18
orthogonal
0.7
0.49
0.25
0.6
0.5
0.5
0.64
1.00
0.51
0.33
0.26
0.4
0.5
0.2
0.2
1.0
1.0
1.00
0.00
0.00
0.00
0.00
2.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
6.8
0.8
0.8
0.00
1.00
9.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
wi orthogona
0.6
0.6
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.4
0.4
0.00
1.00
0.00
0.00
0.00
1.00
0.00
0.2
0.2
Figure 9: Global prototypes orthogonalizztton visualization of Frrrrasfformrr..
19


Reviews:
[{"review_id": "ZHCtPZD6vv", "reviewer": "Reviewer_7NUi", "summary": "This paper introduces several techniques to improve the performance of neural networks on tabular data. The study demonstrates that deep tabular models often face issues related to representation entanglement and the loss of global structure. To address these challenges, the paper proposes the construction of a prototype-based projection space with two carefully designed constraints aimed at decoupling the projected representations.", "questions": "1. How is equation (4) optimized? Compared to the traditional OT problem, it includes \\theta_f as a variable to be optimized.\n2. Could you provide more technique details about the workflow of the PTARL algorithm (Algorithm 1)?\n3. The illustration (Fig. 2) could benefit from improvement as it currently lacks clarity in depicting the overall pipeline. For instance, there are two blocks labeled \"Hidden Representation\"; could you clarify the distinction between them? Additionally, the three sentences on the right side of the figure require further explanation for better understanding.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The suggested representation learning pipeline can be integrated into various deep tabular models.\n- Figure 1 clearly illustrates that the phenomenon of representation entanglement has not been mitigated as the model capacity is gradually increased.\n- The primary concept for enhancing representation involves using weighted prototypes to approximate the original mapped features. This idea is indeed intriguing.", "weaknesses": "- The illustration (Fig. 2) does not clearly depict the overall pipeline; it still remains unclear.\n- More details of the optimization process could be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zt97633uml", "reviewer": "Reviewer_z5Xh", "summary": "The existing deep tabular ML models suffer from the representation entanglement and localization. To address this, the authors explore a novel direction of applying prototype learning  framework. The proposed framework involves to construct prototype-based projection space and learn the disentangles representation around global data prototypes.\n\nThe proposed method contains two stages: prototype generating and prototype projecting. The former is to constructs global prototypes as the basis vectors of projection space for representation, and the latter is to project the data samples into projection space and keeps the core global data information via optimal transport. The authors show the efficiency of the proposed method with various benchmarks.", "questions": "I wonder if the authors believe that the proposed method can be applied to generative models.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The proposed approach is novel and the experimental results are impressive.", "weaknesses": "It would be great if the authors apply the proposed method to recent deep models for tabular representation, such as SAINT [1].\n\n[1] Saint: Improved neural networks for tabular data via row attention and contrastive pre-training, NeurIPS workshop 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gyo8Lffp4p", "reviewer": "Reviewer_GXco", "summary": "This paper presents PTaRL, a model-agnostic method to enhance deep-learning methods for tabular data prediction. The method inserts a sound prototype learning step after the penultimate layer of any DNN to alleviate the issue of representation entanglement and localization. The results show improvement across several architectures and datasets.", "questions": "The whole purpose of using the prototype seems to be for capturing “global data structure information” so as to avoid “sample localization”. However, after reading Section 4, I am still unclear what “global data structure information” really is. Could the authors provide a more explicit definition and description of it? Similarly for “sample location.”\n\nI feel Figure 4 is a much better example of disentanglement than Figure 1, because I still see substantial overlap in the bottom row of Figure 1.\n\nI don’t quite understand how Figure 5 shows diversification.\n\nI was wondering where the boosted performance sits in the literature as a whole. The paper shows DNN and DNN + PTaRL. Do these now match the performance of XGBoost and other state-of-the-art tree-based methods? How much better are they compared to older methods, such as kernel prototype classification and regression? Is it possible to apply PTaRL on different DNN depths to show the contribution of deep-learning representation vs the contribution of the prototype representation? I think knowing the answer to the first question (related to XGBoost) will be very useful. I can understand if the authors feel the other questions are more distracting than useful, since the paper has a focus on enhancing deep-learning approaches.\n\nOverall, the paper is quite well rounded. The problems, solutions, implementations, and results all work together well.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problems identified (representation entanglement and localization) and the results (including the ablation study) are convincing.\n- The various steps — the joint optimization of DNN representation, prototypes, and projection coordinate plus the two constraints — are intuitive. \n- The paper reads smoothly.", "weaknesses": "- The concepts of “global data structure information” and “sample location” are not very clear, at least not as concretely demonstrated as entanglement and orthogonality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)