—Representation learning stands as one of the critical machine learning techniques across various domains. Through the acquisition of high-quality features, pre-trained embeddings significantly reduce input space redundancy, benefiting down- stream pattern recognition tasks such as classification, regression, or detection. Nonetheless, in the domain of tabular data, feature engineering and selection still heavily rely on manual intervention and explanation, leading to time-consuming processes and necessi- tating domain expertise. In response to this challenge, we introduce EConTab, an explainable deep automatic representation learning framework with regularized contrastive learning. Agnostic to any type of modeling task, EConTab constructs an asymmetric autoencoder based on the same raw features from model inputs, producing low-dimensional representative embeddings. Specifically, regularization techniques are applied for raw feature selection and contrastive learning is leveraged to distill the most pertinent information for downstream tasks. Meanwhile, model explanation is demonstrated through feature weights and SHAP- value based model explainer. Experiments conducted on extensive real-world datasets substantiate the framework’s capacity to yield substantial and robust performance improvements. Furthermore, we empirically demonstrate that pre-trained embeddings can seamlessly integrate as easily adaptable features, enhancing the performance of various traditional methods such as XGBoost and Random Forest. Index Terms —Representation Learning, Contrastive Learn- ing, Tabular Data, Regularization, Feature Engineering, Model Explanation I. I NTRODUCTION Over the past decade, representation learning has achieved remarkable advancements in domains such as computer vision and natural language processing, fundamentally transforming how we extract meaningful insights from both image and text data. However, several critical industries, including healthcare [1]–[4], manufacturing [5]–[8], agriculture [9]–[11] and various engineering fields [12]–[22], still heavily rely on structured tabu- lar data. Researchers traditionally leverage domain expertise for feature selection and uncertainty quantification [23]–[27]. It’s commonly believed that traditional tree-based models, equipped with carefully crafted features, can automatically discern the importance of features and their interactions without the need of additional tuning. Despite its effectiveness, manual feature engineering presents some challenges. Crafting high-quality features for tabular data ∗These authors contributed equally to this work.is labor-intensive and doesn’t ensure optimal performance. Feature selection typically involves iterative experimentation, with extensive consumption of resources and time. Moreover, a significant limitation lies in the explainability of models trained on such engineered features. Understanding how and why a model makes predictions is crucial for trust and adoption. To address these challenges, recent research efforts have aimed to leverage deep representation learning to streamline feature engineering in tabular data while enhancing model explainability. However, the integration of tabular data with the remarkable success seen in deep learning across other domains encounters distinctive obstacles. Unlike text data, which inherently consists of discrete tokens, or images, where pixels exhibit spatial correlations, tabular data comprises a diverse mixture of continuous, categorical, and ordinal values. These values can demonstrate complex interdependencies and correlations, introducing layers of complexity to the modeling process. Additionally, unlike the structured nature of images or the sequential nature of text, tabular data lacks inherent positional information necessary to capture intrinsic meanings or learn explicit representations. In this paper, we proposed EConTab, a transformer-based framework to automatically generate high-quality embeddings as features for classification improvement and establish a comprehensive model explanation process. Our framework consists of an asymmetric autoencoder (AE) architecture, which is able to extract the most critical information from raw features to provide substantial performance improvement and robustness for downstream classification tasks. Moreover, EConTab can be effectively trained in both self- and supervised modes. This adaptability ensures the model to perform well across various training scenarios, irrespective of the availability of labeled data. The contributions are summarized as follows: •We introduced a transformer-based automatic feature engineering framework designed to be task-agnostic, featuring scalability and adaptability. •We developed an innovative autoencoder architecture incorporating regularization and contrastive learning to enrich the feature learning process. •We conducted a comprehensive empirical study on various public datasets that demonstrates the superiority of the proposed work in performance lift and robustness. •We demonstrated that representative embeddings extracted from raw features can serve as readily applicable features, seamlessly augmenting the performance of various con- ventional classification models such as logistic regression and tree-based models, etc., and achieve straightforward model explanation. II. R ELATED WORK A. Classical Models Various traditional machine-learning methods have been developed for tabular data classification and regression tasks. Logistic Regression (LR) [28] and Generalized Linear Mod- els (GLM) [29] are the prominent choices to model linear relationships. Decision Trees (DT) [30] sets the fundamental options for tree-based models. Furthermore, various ensemble methods based on DT, such as XGBoost [31], Random Forest [32], CatBoost [33], and LightGBM [34] are developed. These ensemble methods are highly favored in industry for their capacity to capture intricate non-linear relationships, improve interpretability, and effectively manage diverse feature types, encompassing null values and categorical features. B. Deep Learning Models A notable trend in current research has been focusing on applying deep learning techniques to tabular data [35]. This movement has led to the emergence of various neural architectures, each tailored to improve performance within the tabular data domain. These architectures can be broadly classified into several categories [5], [36], [37]. First, supervised models like ResNet [38], SNN [39], AutoInt [40], and DCN-V2 [41], have been well-known to harness the power of neural networks and improve the handling of tabular data. Second, innovative hybrid approaches like NODE [42], GrowNet [43], TabNN [44], and DeepGBM [45] could seamlessly integrate decision trees with neural networks, resulting in end-to-end training. Third, transformer-based methods such as TabNet [46], TabTransformer [47], and FT-Transformer [36] allow models to learn from attention-spanning features and data samples. Finally, representation learning methods such as VIME [48], SCARF [49], SAINT [50] and SwitchTab [51] are gaining prominence, emphasizing effective information extraction through self- and semi-supervised learning techniques. These approaches align seamlessly with the growing emphasis on representation learning in the field. C. Self- and Semi-supervised Representation Learning In computer vision, deep representation learning methodolo- gies have emerged as potent tools, capitalizing on self- and semi- supervised training paradigms [52]–[56]. These methodologies exhibit a dichotomy, falling into two distinct categories of innovation. The first category of deep representation learning methods is rooted in generative models, particularly autoen- coders [57]. A striking exemplar within this genre is the Masked AutoEncoder (MAE) architecture introduced by [58]. MAE features an asymmetric encoder-decoder architecturepurposefully crafted for the extraction of embeddings from images. Impressively, the framework demonstrates the capa- bility to capture spatiotemporal information [59] and extends seamlessly to various domains such as 3D space [60] and multiple scales [61]. Notably, akin masking strategies, prevalent in the Natural Language Processing (NLP) community [62], have also been transposed into the tabular data landscape [46], [47]. Furthermore, VIME [48] presents a method reminiscent of MAE in the tabular data context. VIME perturbs and encodes each data sample within the feature space through the involvement of two estimators. Subsequently, these estimators use decoders to reconstruct both a binary mask and the original, uncorrupted data samples, demonstrating versatility in information extraction. The second category predominantly revolves around the contrastive learning paradigm and strategically employs data augmentation techniques. Within this domain, prominent mod- els harnessed momentum-update strategies [63], [64], embraced the concept of large batch sizes [65], incorporated stop-gradient operations [66], spatiotemporal information [67], or even introduced an online network tasked with predicting the output of a target network [68]. Notably, these concepts, initially designed for image data, have gracefully transcended into the arena of tabular data. An exemplar of such adaptation is found in SCARF [49], which ingeniously incorporates the principles of SimCLR [65] to pre-train the encoder. This pre-training procedure employs a subset of feature corruption as a pivotal data augmentation method. Furthermore, the work of [50] exemplifies a contrastive framework tailored to tabular data, introducing SAINT, computing both column- and row-wise attentions. D. Regularization Regularization techniques, pivotal in machine learning and statistical modeling, mitigate overfitting and enhance gener- alization by introducing penalty terms into the loss function. Early approaches such as Ridge Regression, which applies L2 regularization to linear models [69], and Lasso Regression [70], which implements L1 regularization, paved the way for modern regularization methods. The Elastic Net [71] combines these approaches to strike a balance between feature selection and coefficient shrinkage, while Dropout [72] and Batch Nor- malization [73] cater specifically to neural networks, fostering robust and generalized representations. Other techniques like early stopping [74] and weight decay [75] further complement the regularization arsenal. Bayesian approaches introduce probabilistic frameworks, such as Bayesian regression [76] and Gaussian Processes [77], integrating prior beliefs and data likelihood. Recent trends encompass adversarial training [78] to enhance model robustness and graph regularization techniques [79] for graph-based data modeling tasks. As machine learning continues to advance, regularization remains vital for model generalization and robustness. E. Model Explanation Model explanation in machine learning [80] is a critical area of research aimed at making the behavior of complex models transparent, understandable, and accountable. Various tech- niques have been designed to elucidate how predictive models reach their conclusions. Traditional models [81] can leverage decision tree structures and calculate feature importance to understand the decision weights. For deep learning models, to unravel the black-box nature and allow better comprehension of model decisions, various explanation methods [82] are de- veloped to make specific aspects of the internal representations and information flow in neural network interpretable by humans. Among the methods, SHapley Additive exPlanations (SHAP) values [83] stand out with foundation in cooperative game theory, offering a robust and principled approach to attribute the impact of each feature on model predictions. III. M ETHOD In this section, we present EConTab, our comprehensive approach for tabular data self- and supervised representation learning. First, we outline the process of the regularization method. Second, we formulate the feature corruption process. The self-supervised training process is illustrated in the third sub-section, without knowing the task labels. The fourth sub-section elucidates our novel supervised training method, wherein we leverage labels for contrastive learning. Finally, we expound on our utilization of pre-trained encoders and embeddings to improve downstream tasks and develop the model explanation process. A. Regularization We apply regularization [24], [84] on the input layer by introducing a penalty term λ∥W∥pinto the loss function, whereWrepresents the input weights, λis the regularization parameter and pis the specific norm for the penalty. The idea behind is to prevent similar features from weighing too much in loss objective and to learn more robust representation, especially when highly correlated features are present. For example, if we can reconstruct features A, B, and C with only feature A, then B and C should be assigned less weights. B. Feature Corruption It’s common for the generative-based representation approach to use data augmentation techniques to generate robust feature embeddings. One of the most promising approaches is feature corruption, which has also been used in this paper to enhance our model’s performance. Considering the original dataset X ⊆RM, given any tabular data point xi, we have its j-th feature as xij, where xi= (xi1, xi2, ..., x iM), j⊆M, with Mrepresenting the dimension of features and idenoting the sample index. In our approach, for each sample, we stochastically select tfeatures from the pool of Mfeatures and replace them with corrupted features denoted as c. To elaborate, we generate cfrom the distribution bXij, where bXijrepresents the uniform distribution over Xij= xij:xi∈ X .C. Self-supervised Learning Self-supervised learning of EConTab aims to learn informa- tive representations from unlabeled data (Algorithm 1). For each of the two data samples, x1andx2, we apply input weights and add feature corruption to obtain corrupted data. Then we encode the corrupted data using an encoder, f, resulting in two features, z1andz2. The decoder dwill decode the learned embeddings to reconstruct ˆx1andˆx2respectively, from where we can define the reconstruction loss Lreconstruction for two samples x1andx2 as the mean squared error (MSE) between input features and reconstructions, shown as: Lreconstruction =1 MMX j=1(x1j−ˆx1j)2+1 MMX j=1(x2j−ˆx2j)2. (1) Algorithm 1 Self-supervised learning Require: unlabeled data X ⊆ RM, batch size B, encoder f, decoder d, mean squared error (MSE), input weights W⊆RM, regularization parameter λand specific norm for penalty p. for two sampled mini-batch x1 i, y1 i B i=1⊆ {X ,Y} and x2 i, y2 i B i=1⊆ {X ,Y}do for each sample x1 iandx2 i, apply input weights x1 i=x1 iW,x2 i=x2 iW, fori∈[B] apply feature corruption, define the corrupted feature as: ˘x1 iand ˘x2 i, fori∈[B] data encoding: z1 i=f(˘x1 i),z2 i=f(˘x2 i), fori∈[B] data reconstruction: ˆx1 i=d(z1 i),ˆx2 i=d(z2 i), fori∈[B] define reconstruction loss Lreconstruction = MSE(x1 i,ˆx1 i)+MSE(x2 i,ˆx2 i) define penalty as λ∥W∥p update encoder fand decoder dto minimize Lreconstruction and λ∥W∥pusing RMSProp. end for Therefore, the loss function for self-supervised learning can be defined as: Lself=Lreconstruction +λ∥W∥p, (2) D. Supervised Learning with Labels We further improve the pre-training process through super- vised learning with labels to take advantage of labeled data, as shown in Figure 1. In self-supervised learning, we only compute the MSE between reconstructed data and original data as the reconstruction loss Lreconstruction . With labels introduced, we can pose additional constraints to the encoded embeddings z1andz2. One is for label prediction to compute the prediction loss (illustrated by classification loss Lclassification through the context). To be specific, z1andz2are fed to the same multi- layer perceptron (MLP) that maps from the embedding space to the label space. We can also define the cross-entropy loss for classification task as: Lclassification =−(y1log(ˆy1) +y2log(ˆy2)), (3) where ˆy1andˆy2are predicted labels computing a MLP, i.e., ˆy1=MLP(z1)andˆy2=MLP(z2). 1) Contrastive Loss: We further introduce the contrastive lossLcontrastive in the loss function by forming contrastive pairs ( z1,z2) of embeddings in the bottleneck layer with respect to the classification labels ( y1,y2). With this constraint, the model is enforced to maximize the similarity between embeddings with the same label and minimize the similarity between embeddings with different labels, thus capturing the discriminative features for the classification labels and better aligning with downstream tasks. Algorithm 2 formally defines the contrastive loss in the proposed model, which is a variation from the original contrastive learning [85] and relevant to these extensions [65], [86], [87]. Algorithm 2 Contrastive Loss for supervised learning with labels Require: data embeddings Zfrom unlabeled data X ⊆RM, binary labelsY ⊆R, batch size B, encoder f, decoder d, contrastive loss margin m, distance function D(·) for two sampled mini-batch z1 i, y1 i B i=1⊆ {X ,Y} and z2 i, y2 i B i=1⊆ {X ,Y}do for each sample embedding z1 iandz2 i, define contrastive loss: fori= 1 toBdo ify1 i=y2 ithen y12 i = 1 for the pair (z1 i, z2 i) //z1 iis deemed similar to z2 i else y12 i = 0 for the pair (z1 i, z2 i) //z1 iis deemed dissimilar to z2 i end if di=D(z1 i, z2 i)// calculate the distance of embeddings pair ci= ( y12 i)1 2d2 i+ (1 −y12 i)1 2max(0 , m−di)2 // calculate the contrastive loss of the pair end for Lcontrastive =1 BPci update encoder fand decoder dto minimize Lcontrastive using RMSProp. end for During the optimization stage, we combine the two additional losses with the self-supervised learning loss Lselfand define the supervised learning loss function Lsupas follows: Lsup=Lself+α∗ L classification +β∗ L contrastive , (4) where αandβare used to seek balance among multiple losses and set to 1 as default, respectively. E. Downstream Fine-Tuning Drawing inspiration from established representation learning paradigms [49], [63]–[65], we embrace an end-to-end fine- tuning strategy for the pre-trained encoder ffrom EConTab, utilizing the complete labeled dataset. This approach entails the seamless integration of the encoder with an additional linear layer, thereby granting the flexibility to unlock and adapt all its parameters to align with the specific requirements of downstream supervised tasks. Additionally, we can harness the potential of the salient feature sas a versatile plug-and-play embedding. Through the fusion of zwith its original counterpart x, we construct enriched data points. This innovative approachserves to amplify inherent data characteristics, thereby assisting in the establishment of distinct decision boundaries. As a result, we anticipate notable enhancements in classification tasks when employing the concatenated features as the input for conventional models like Random Forest or LightGBM. F . Model Explanation There are three major considerations in model explanation for EConTabs. First, the feature weights are learnable through training with regularization, which could explain the importance of input features for overall training tasks. Second, the model structure, especially the encoder part, could be further explained through Deep SHAP [88], which could better understand how the latent representations are formulated from inputs. Third, when latent representations are used in downstream tasks as features, the feature importance could be leverage to explain the final predictions. IV. E XPERIMENTS AND RESULTS In this section, we present the results of our extensive experiments conducted on diverse public datasets to highlight the effectiveness of our proposed method, EConTab. This section is structured into two parts for clarity and compre- hensiveness. In the first part, we provide essential details regarding the experiments. This includes information about the public datasets for experiments, the preprocessing steps applied to these datasets, the architecture of our models, and specific training procedures. This transparency ensures the reproducibility of our findings. In the second part, we assess the performance of EConTab through various empirical studies. We conduct a thorough comparison between EConTab and mainstream deep learning methods as well as traditional methods. Meanwhile, we show- case the versatility of EConTab using it as an automatic feature engineering tool. Specifically, we demonstrate how EConTab can enhance the performance of traditional models such as XGBoost, Random Forest, and LightGBM by seamlessly integrating its salient features as plug-and-play embeddings, as shown in Figure 2. This strategy simplifies the feature engineering process and eliminates additional complexity in traditional models training. In addition, we also conducted ablation studies and illustrate the model explanation process. A. Preliminaries for Experiments 1) Public Datasets: We evaluate the performance of ECon- Tab on a standard benchmark from [50], including Bank (BK) [89], Blastchar (BC) [90], Arrhythmia (AT) [91], Arcene (AR) [92], Shoppers (SH) [93], V olkert (VO) [94] and MNIST (MN) [95]. Five of the datasets focus on binary classification, and two of them focus on multi-class classification tasks. Importantly, the datasets employed in our experiments exhibit significant diversity. They encompass a wide range of characteristics, including varying sample sizes, ranging from 200 to 495,141 samples, and feature dimensions spanning from 8 to 784, encompassing both categorical and numerical features. Among these datasets, some exhibit missing data, while others are 𝐹𝑒𝑎𝑡𝑢𝑟𝑒 𝐶𝑜𝑟𝑟𝑢𝑝𝑡𝑖𝑜𝑛𝐸𝑛𝑐𝑜𝑑𝑒𝑟(𝑓) 10De𝑐𝑜𝑑𝑒𝑟(𝑑)𝑀𝑆𝐸 𝐼𝑛𝑝𝑢𝑡 𝑊𝑒𝑖𝑔ℎ𝑡𝑠 𝑊𝐼𝑛𝑝𝑢𝑡 𝐿𝑎𝑏𝑒𝑙𝑠y10001110Contrastive LossClassification LossReconstruction LossPenalty Input Weights𝐸𝑚𝑏𝑒𝑑𝑑𝑖𝑛𝑔𝑠 𝑍Fig. 1: Proposed AE architecture with contrastive loss and input weights regularization Tabular Data Embeddings Downstream Tasks Representation Learning Existing modelsFine-tuning Plug-and-play Fig. 2: Illustration of usages of pre-trained encoders and embeddings. 1) The first option could be to fine-tune the pre-trained encoder directly for downstream tasks. This option usually achieves the optimal results but needs additional computation. 2) The second option is to concatenate the pre-trained embeddings with the original datasets, which requires no additional training and computation but still benefits the downstream tasks with considerable improvements in evaluation metrics. complete, and there is a mix of well-balanced datasets as well as those presenting highly skewed class distributions. This diversity allows us to comprehensively evaluate the performance and robustness of our proposed approach across a spectrum of real-world data scenarios. 2) Preprocessing of Datasets: To handle categorical features, we employ a backward difference encoder as described in [96]. Addressing the issue of missing data, we take a two- step approach. Initially, we remove any features that lack values across all samples. Subsequently, for the remaining missing values, we apply distinct imputation strategies based on the feature type. Numerical features are imputed using the mean value, while categorical features are filled with the most frequent category observed within the dataset. Moreover, we ensure data uniformity by employing a min-max scaler for dataset scaling. In cases involving image-based data, we flatten the images into vectors, treating them akin to tabular data. Thisapproach aligns with established practices found in prior works such as [48] and [50]. 3) Model Architectures: The EConTab model architecture features a transformer-based shared network with three layers and two attention heads. This architecture is tailored for processing input data with a dimensionality determined by the shape of the training dataset. Additionally, the decoder remains a one-layer network with a sigmoid activation function. In the downstream fine-tuning stage, we add a linear layer after the encoder fto accommodate classification or regression tasks as needed. 4) Training Details: The EConTab model is trained with a batch size of 128 over 1000 epochs, employing a learning rate of 0.0001. Gaussian masking is applied to the input data with a masking ratio of 0.3. The model’s output dimension is set to half of the input data dimension. A contrastive loss with a margin of 2 is used during training, along with L2 normalization. Additionally, a regularization coefficient of 0.01 is applied to introduce a penalty term based on the L2 norm of the standard deviation of the Gaussian mask. During training, data is divided into two batches, and various loss components, including feature reconstruction loss, classification loss, contrastive loss, and regularization penalty, are computed to guide the optimization process. These training configurations ensure effective representation learning while controlling model behavior. 5) Metrics: Given that the majority of the tasks in our analysis involve binary classification, we employ the AUROC (Area Under the Receiver Operating Characteristic curve) as our primary metric for assessing performance. AUROC effectively quantifies the model’s ability to distinguish between the two classes in the dataset. However, for the two multi-class datasets, VO and MN, we utilize accuracy on the test set as the metric for comparing performance. B. Results on the Benchmarks We show performance comparisons using chosen datasets and present the summarized results in Table I. These results Dataset size 45211 7043 452 200 12330 58310 518012 Feature size 16 20 226 783 17 147 54 Dataset BK BC AT AR SH VO ⋆ MN⋆ Raw Feature ( x) ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Distilled Feature ( s) ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Logistic Reg. 0.907 0.907 0.909 0.892 0.892 0.895 0.862 0.864 0.866 0.916 0.914 0.918 0.870 0.871 0.873 0.539 0.540 0.543 0.899 0.902 0.905 Random Forest 0.891 0.892 0.894 0.879 0.880 0.884 0.850 0.856 0.861 0.809 0.809 0.811 0.929 0.928 0.930 0.663 0.665 0.669 0.938 0.938 0.942 XGboost 0.929 0.928 0.930 0.906 0.903 0.906 0.870 0.871 0.883 0.824 0.822 0.826 0.925 0.925 0.927 0.690 0.690 0.692 0.958 0.959 0.963 LightGBM 0.939 0.933 0.939 0.910 0.909 0.912 0.887 0.888 0.907 0.821 0.822 0.825 0.932 0.933 0.936 0.679 0.680 0.682 0.952 0.953 0.954 CatBoost 0.925 0.928 0.932 0.912 0.910 0.914 0.879 0.880 0.889 0.825 0.827 0.833 0.931 0.932 0.935 0.664 0.665 0.670 0.956 0.958 0.968 MLP 0.915 0.919 0.920 0.892 0.893 0.898 0.902 0.904 0.908 0.903 0.904 0.904 0.887 0.887 0.890 0.631 0.631 0.636 0.939 0.940 0.940 VIME 0.766 - - 0.510 - - 0.653 - - 0.610 - - 0.744 - - 0.623 - - 0.958 - - TabNet 0.918 - - 0.796 - - 0.521 - - 0.541 - - 0.914 - - 0.568 - - 0.968 - - TabTransformer 0.913 - - 0.817 - - 0.700 - - 0.868 - - 0.927 - - 0.580 - - 0.887 - - EConTab(Self-Sup.) 0.908 - - 0.898 - - 0.873 - - 0.887 - - 0.920 - - 0.619 - - 0.956 - - EConTab(Sup.) 0.929 - - 0.913 - - 0.907 - - 0.918 - - 0.931 - - 0.680 - - 0.968 - - “−”indicates the experiments are not applicable for the corresponding methods to demonstrate the benefits of plug-and-play embeddings. TABLE I: Comparison of different methods on the classification tasks. For each method and dataset, we report three categories 1) raw features only, 2) salient features only, 3) plug-and-play features. The best results are shown in Bold , second-best results are Underlined . Columns added with ⋆are multi-class classification tasks, reporting accuracy. The other results of binary classification tasks are evaluated with AUROC. encompass evaluations employing both traditional models and more recent deep-learning techniques. In the majority of cases, EConTab exhibits remarkable improvements, outperforming all baseline methods and reaffirming its superiority across a range of datasets and scenarios. However, it is important to note that, on BK, SH, and VO datasets, EConTab achieved suboptimal results when compared to the best models. This observation aligns with previous research conclusions that the tabular domain presents unique challenges, with no single method universally excelling [36]. Nonetheless, EConTab still gives the best performance over all of the deep-learning-based models and the second-best results over all of the methods. Meanwhile, this outcome warrants further investigation to uncover the specific factors contributing to this variation in performance. C. Results as Plug-and-Play Embeddings As previously mentioned, EConTab has learned features that can significantly impact the decision boundaries in clas- sification tasks. In the plug-and-play setting from Figure 2, our experimental results demonstrate the immense value of integrating these salient features with the original data as additional features. To be more specific, the performance of traditional models obtains relatively marginal improvement with only distilled features, as shown in the light gray columns of Table I. While the improvement is relatively modest, it aligns with our expectations. The potential absence of original information in this scenario results in a less substantial performance boost. Larger gains without fine-tuning come from the concatenation of original and distilled features. Notably, this integration enhances the performance of every method, leading to improvements in evaluation metrics (e.g., AUROC) across various datasets, as shown in the dark gray columns of Table I. D. Ablation Studies In this section dedicated to ablation studies, we delve into the crucial components of EConTab, assessing the significance of the parameter, i.e., feature corruption rate. Our analysis encom- passes all the datasets listed in Table I, employing consistentdata preprocessing and optimization strategies throughout the experiments. In Table II, we thoroughly examine the most advantageous feature corruption ratio. After extensive analysis, we find that the optimal corruption ratio is approximately 0.3. Therefore, we’ve adopted this value as the default for all previously reported experiments. However, it’s important to emphasize that this chosen ratio may not always be the best fit for every dataset. Additionally, we’ve noticed interesting patterns in the datasets themselves. Datasets with more complex features, like VO or MN, tend to benefit from larger corruption ratios because they often contain redundant features. This observation aligns with previous research discussed in [81] regarding tabular data. On the flip side, for datasets with simpler, lower-dimensional features like BC, using smaller corruption ratios in our experiments might lead to better results. Ratio 0.0 0.1 0.2 0.3 0.4 0.5 0.6 BK 0.918 0.920 0.928 0.929 0.922 0.917 0.881 BC 0.889 0.897 0.906 0.913 0.910 0.901 0.896 AT 0.889 0.894 0.901 0.905 0.903 0.890 0.884 AR 0.904 0.911 0.913 0.918 0.915 0.909 0.901 SH 0.902 0.914 0.924 0.931 0.920 0.909 0.904 VO⋆ 0.667 0.674 0.676 0.680 0.681 0.670 0.663 MN⋆ 0.935 0.942 0.951 0.959 0.959 0.941 0.932 TABLE II: Ablation of corruption ratio. Columns added with ⋆are multi-class classification tasks, reporting their accuracy. The other results of binary classification tasks are evaluated with AUC. E. Model Explanation Illustration To illustrate how the model explanation can be conducted, we provide visualizations on all three major steps, including input feature weights, encoder SHAP values, and downstream embedding feature importance. We use BK data as an example, which has 16 features and 10 of them are categorical. After one-hot encoding on categorical features and , the input feature dimension becomes 52. With input feature weights regularization, the contribute of each input feature on overall training process can be cleanly identified, as shown in Figure 3a. We could further compare the feature correlation based on the learned weights. From Figure 3b, the top 20 features with higher weights tend to be less correlated with each other and represent more independent contributions. In the contrast, bottom 20 features with lower weights are having various correlations among each other. This could be used to unveil the shadow on how the features are learned through their interrelationships. 0123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051 Feature Index0.00.20.40.60.81.0Weight (a)Input feature weights after fine-tuning (b)Feature correlation comparison by weights Fig. 3: Input feature determination through weights regularization Using SHAP values for EConTab encoder could better understand how each input feature is contributing to the formation of each latent embedding. As shown in Figure 4, the mean SHAP values for top features are visualized and decomposed across all latent embeddings, for both pretraining only and after fine-tuning. Once the latent embeddings are used in downstream models (e.g., XGboost), the feature importance can be calculated and compared against other features if used in a plug-and-play manner, as shown in Figure 4c. V. C ONCLUSION As we observe the evolution of potent representation learning techniques tailored for different types of data from computer vision and natural language processing, we embark on a journey to extend their remarkable performance into new domains, such as tabular data. Drawing inspiration from related endeavors that address this challenge from the vantage points of contrastive learning and generative modeling, we present EConTab — an innovative self- and supervised framework designed for representation learning and feature distillation. The features learned through EConTab exhibit superior performance in downstream tasks, obviating the need for extensive exploration of hand-crafted features. Furthermore, these features manifest as discernible, low-dimensional representations that seamlessly enhance the capabilities of various traditional models with explainability. We hold a strong conviction that this research marks a pivotal milestone in the pursuit of more representative, efficient, and structured representations for tabular data. 0.0 0.1 0.2 0.3 0.4 0.5 Mean SHAP valueFeature 31Feature 25Feature 21Feature 44Feature 51Feature 24Feature 32Feature 29Feature 30Feature 41 E5 E3 E1 E2 E4 E0(a)Pretraining only 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 Mean SHAP valueFeature 49Feature 31Feature 29Feature 48Feature 30Feature 41Feature 51Feature 50Feature 4Feature 32 E5 E2 E1 E4 E0 E3 (b)Pretraining and Finetuning E5 E2 E1 E4 E0 E30.00.20.40.60.81.0Percentage of importance (c)Feature ImportanceFig. 4: Mean SHAP values for each input feature to contribute on each latent embedding