Title: D2R2: Diffusion-based Representation with Random Distance Matching for Tabular Few-shot Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Classification

Few-shot learning

Tabular data

Representation learning

Diffusion models

Self-supervised learning
Abstract: Tabular data is widely utilized in a wide range of real-world applications. The challenge of few-shot learning with tabular data stands as a crucial problem in both industry and academia, due to the high cost or even impossibility of annotating additional samples. However, the inherent heterogeneity of tabular features, combined with the scarcity of labeled data, presents a significant challenge in tabular few-shot classification. In this paper, we propose a novel approach named Diffusion-based Representation with Random Distance matching (D2R2) for tabular few-shot learning. D2R2 leverages the powerful expression ability of diffusion models to extract essential semantic knowledge crucial for denoising process. This semantic knowledge proves beneficial in few-shot downstream tasks. During the training process of our designed diffusion model, we introduce a random distance matching to preserve distance information in the embeddings, thereby improving effectiveness for classification. During the classification stage, we introduce an instance-wise iterative prototype scheme to improve performance by accommodating the multimodality of embeddings and increasing clustering robustness. Our experiments reveal the significant efficacy of D2R2 across various tabular few-shot learning benchmarks, demonstrating its state-of-the-art performance in this field.

D2R2: Diffusion-based Representation with Random Distance Matching for Tabular Few-shot Learning

Abstract

Tabular data is widely utilized in a wide range of real-world applications. The challenge of few-shot learning with tabular data stands as a crucial problem in both industry and academia, due to the high cost or even impossibility of annotating additional samples. However, the inherent heterogeneity of tabular features, combined with the scarcity of labeled data, presents a significant challenge in tabular few-shot classification. In this paper, we propose a novel approach named Diffusion-based Representation with Random Distance matching (D2R2) for tabular few-shot learning. D2R2 leverages the powerful expression ability of diffusion models to extract essential semantic knowledge crucial for denoising process. This semantic knowledge proves beneficial in few-shot downstream tasks. During the training process of our designed diffusion model, we introduce a random distance matching to preserve distance information in the embeddings, thereby improving effectiveness for classification. During the classification stage, we introduce an instance-wise iterative prototype scheme to improve performance by accommodating the multimodality of embeddings and increasing clustering robustness. Our experiments reveal the significant efficacy of D2R2 across various tabular few-shot learning benchmarks, demonstrating its state-of-the-art performance in this field.

1 Introduction

Learning with a large set of data with only a few labeled samples is an essential requirement for industry and academia, primarily due to the high cost of annotating samples. However, accurately classifying new data with a scantily labeled training set and an unlabeled training set poses a formidable challenge, as the effectiveness of statistical and modern deep learning systems in supervised learning heavily relies on the large size of the labeled set [36]. This underscores the need for research in few-shot learning in situations where labeled data is scarce. For example, in one-shot learning, each class of the training set contains only one labeled sample, and the goal is to classify new test samples. While few-shot learning has garnered considerable attention in computer vision (CV) [9, 21] and natural language processing (NLP) [30, 31], it remains relatively under-explored in the field of tabular data. Nevertheless, few-shot learning holds great importance in the context of tabular data, as limited labeled tabular data is inherently common in many real-world applications, such as fraud detection [7], disease diagnosis [40], and social science [27]. However, modeling with limited labeled tabular data presents three significant challenges.

The first challenge is due to the scarcity of labeled training samples, which leads to a deficient understanding of the test class. In this case, semantic knowledge, encompassing general and lowfrequency information not tied to any particular task, can be highly beneficial in few-shot learning [39, 52, 56]. This is because representing a class with limited labeled samples inherently involves ambiguity, necessitating the use of semantic knowledge acquired from unlabeled ones to refine the class definition and address this ambiguity. Although for CV [21] and NLP [31], semantic knowledge can be effectively derived based on the learned spatial structure patterns between pixels or tokens, it is more challenging to extract semantic knowledge for tabular data, since they typically lack local relationships between columns.

The second challenge arises from the diverse characteristics of tabular features. Tabular data comprises numerical and categorical features. Numerical features contain ordinal values but have multiple modes, while categorical features present distinct and incomparable values. Many existing statistical and deep neural network models struggle to effectively handle these mixed data types. This challenge underscores the importance of simultaneously modeling continuous and categorical features.

The third challenge pertains to the utilization of prototype classification with tabular data. Numerous approaches suggest employing average embeddings of the labeled support set for classification [49, 11, 28]. However, the limited labeled support samples lead to large variance and imprecision of the prototypes, highlighting the need for more information from unlabeled samples. Additionally, embeddings from the same class may exhibit multimodal behavior (see Figure 3 for examples), which means that using the average embeddings of one class could result in inaccurate classification results.

In addressing the aforementioned challenges of few-shot learning, significant efforts have been made over the past decade. One approach employs a meta-learning scheme. A common method is to generate a pseudo-label for each unlabeled data and then train the model using pseudo-label [25, 43, 28]. Other approaches utilize self-generated tasks, such as CACTUs [19], UMTRA [22] and Meta-GMVAE [24]. However, despite the effectiveness of prior works on image datasets, we find that applying each method to the tabular domain is highly non-trivial. These methods often assume uniformity in feature types and rely on strong spatial and sequential relationships among features, which do not hold for tabular data. For instance, augmentation techniques are readily applicable to images, where spatial relationships between pixels can be leveraged, as shown by UMTRA [22]. However, such techniques are difficult to apply to tabular data, which lacks these structural patterns [4]. STUNT [28] is designed for tabular data using a proxy-based approach. However, this methods may struggle to acquire useful semantic knowledge due to the disparities between the constructed tasks and the target task, leading to diminished performance for testing. Moreover, it applies the same methods to both numerical and categorical features, neglecting the unique information in different types of data. This oversight hampers the model's effectiveness. Recent studies have showed that self-supervised learning can leverage unlabeled datasets to acquire semantic knowledge and transferable representations of images [16] and languages [14]. However, these approaches heavily rely on augmentation schemes. For tabular data, the heterogeneous feature types pose challenges for augmentation, and only a few methods have been proposed for tabular data. State-of-the-art methods along this line include VIME [55], SCARF [4], SubTab [47].

In this work, we propose a novel framework named Diffusion-based Representation with Random Distance Matching (D2R2) for tabular few-shot learning. D2R2 extracts representations by training a designed conditional diffusion model and aligning the distances from various random projection spaces. Subsequently, we introduce the instance-wise iterative prototype to conduct few-shot classification, which addresses the multimodal behavior of embeddings and enables the creation of a robust classifier. Finally, we design an unsupervised validation scheme to address the absence of the labeled validation set for hyperparameter selection. Different from meta-learning and self-supervised learning methods, D2R2 does not relies on self-generated tasks or augmentation schemes, but instead uses a general training process. The D2R2 framework offers the following advantages:

Considering the first challenge mentioned earlier, we avoid to rely on proxy methods or augmentation methods, which are limited by the lack of local relationships in tabular data. Instead, we create an information bottleneck for extracting semantic knowledge, named D2R2, which leverages the strong expressiveness of the diffusion model and distance information from pairwise comparison. Firstly, we modify the diffusion model to serve as a representation extractor. The powerful expressiveness of diffusion model enables the extraction of semantic knowledge crucial to denoising. And the information bottleneck induced by conditional information, obtains low-frequency information over

details. This approach outperforms alternative representation learning and meta-learning methods, as demonstrated by our experiments. Moreover, the representations exhibit robustness due to the introduction of designed diffusion noise schedule and subsequent denoising process, which reinforces their stability against tabular column perturbations.

Nevertheless, relying solely on diffusion-based representations may not provide sufficient distance information to build a subsequent prototype classifier, which classifies test samples based on their distance between prototypes. Hence, we introduce a random distance matching (RDM) loss during the training of the diffusion model to obtain the distance information. One intuition of applying RDM is that if the class structure can be characterized by certain distance information, the embeddings should preserve such distance structure. Another intuition is that if two samples belong to the same class in the ground truth, their embeddings should appear similar from any perspective. Random distance matching achieves these ideas by projecting data into diverse random spaces, which encompass class structure information. Consequently, if two data points are close in multiple random spaces, they are highly likely to belong to the same class. Besides, two distinct Random Distance matching are designed for continuous and discrete features based on their characteristics, respectively. In this way, we can simultaneously model numerical and categorical features within one model, addressing the second challenge mentioned earlier. Overall, our newly designed D2R2 training process not only extracts high-quality semantic knowledge for few-shot learning but also reveals distance information for prototype classification, while remaining adaptive to mixture types of tabular data.

Moreover, during the classification phase, we predict the class of a test sample to be the same as the class of the nearest prototype. Given that the embedding of a single class may demonstrate multimodality, we introduce a novel instance-wise iterative prototype. This approach tackles the issue of multimodal behavior by creating several prototypes within one class and improving the robustness of prototypes by iterative refinement, resolving the third challenge previously mentioned.

Lastly, to address the absence of a labeled validation set in unsupervised learning for hyperparameter selection, we devise a validation scheme by generating pseudo-labels for the unlabeled dataset using soft k-means clustering of raw features.

Our contribution is summarized as follows:

To the best of our knowledge, we are the first to propose a specifically designed diffusion method to learn semantic knowledge for tabular data.

We propose an innovative framework, D2R2, to extract representations in tabular few-shot learning, which is built upon the designed diffusion process and random distance matching. D2R2 not only captures high-quality semantic knowledge, but also incorporates distance information for the subsequent prototype classifier. Moreover, it adapts well to mixture types of tabular data.

To further improve few-shot classification performance, we introduce a novel classifier with instance-wise iteration prototypes. This classifier is able to construct highly accurate and stable prototypes, while also revealing the multimodal behavior of a single class.

We conduct extensive experiments to evaluate our framework for tabular few-shot learning, comparing our method with 15 state-of-the-art literature baselines across nine datasets. Among various tabular datasets, D2R2 outperforms other baselines by a significant margin.

2 Related work

Supervised and Semi-supervised learning . Some supervised classifiers have strong ability to learn from limited samples. Methods along this line include CatBoost [35], TabPFN [18], a transformerbased network designed to make predictions on a small tabular dataset; k-nearest neighbor classifier (kNN) [32], the nearest neighbor prototype classifier. However, they are inadequate in few-shot learning because they are not intended to learn the semantic knowledge. Semi-supervised learning frameworks are designed to improve model generalization by creating a strong connection between limited labeled samples and unlabeled samples, including Mean Teacher (MT) [46], Interpolation Consistency Training (ICT) [48], and Meta Pseudo Labels (MPL) [33]. In semi-supervised learning, a model trains on labeled data and then predicts labels for unlabeled data. However, this method requires more labeled data than few-shot learning contexts.

Few-shot meta-learning . Few-shot learning aims to train models to adapt to downstream tasks with minimal labeled examples. This can be accomplished by employing meta-learning techniques on related tasks, enabling the acquisition of prior knowledge that can be utilized to solve new tasks [50]. Existing deep few-shot methods can be divided into three main categories. First, optimizationbased meta-learning methods [5, 15, 26, 29] devise a proficient optimization strategy that adapts to downstream tasks effectively. Second, metric-based methods [13, 30, 34, 51] concentrate on latent space to derive meaningful feature embeddings, and then make predictions based on the similarity between support and query embeddings. Third, data generative strategies emphasize creating more varied samples to train a more precise classifier. Most research on few-shot meta-learning focuses on NLP and CV tasks, such as CACTUs [19] and UMTRA [22], while a small subset tackles tabular few-shot learning. STUNT [28] meta-learns generalizable knowledge from self-generated tasks on an unlabeled tabular training set. However, these methods may generate ineffective semantic knowledge because of the gap between the self-generated tasks and the testing task.

Self-supervised learning . Our research addresses scenarios where an unlabeled training set and a small labeled support set ( K -shot) are used to predict class labels for a test query set, as detailed in Section 3 of our paper. Self-supervised learning is particularly effective at developing robust representations from unlabeled data [8, 37]. These approaches focus on pre-training the representation by utilizing domain-specific inductive biases, like the spatial relationships in images. Notably, prior research has demonstrated the effectiveness of self-supervised learning in few-shot scenarios relative to meta-learning techniques [12]. Chen et al. [10] added an autoencoder in the diffusion process and Yang et al.[53] introduces a latent Denoising Autoencoder architecture where the learned representations are used for Denoising Autoencoder, but the Autoencoder reconstruction process is less suitable for tabular data as discussed in STUNT [28]. Other works on self-supervised learning schemes rely on augmentation schemes. It is unclear how to extend such methods to the tabular domain due to the heterogeneous characteristics of tabular datasets. Moreover, for tabular data, multimodal and categorical features make augmentation difficult, and few effective methods have been proposed in tabular data setting. Recent state-of-the-art tabular learning augmentation techniques include masking cell: VIME [55], constructing subsets: SubTab [47], and contrastive learning: SCARF [4]. However, they fail to deliver substantial performance enhancements for few-shot tabular learning in our experiments. Rather, we train the unlabeled dataset through an unsupervised diffusion framework that does not rely on the effect of augmentation.

3 Problem definition

In this paper, we explore few-shot learning for tabular classification.Tabular data refers to the dataset organized in tables, which is a structured format that presents information in rows and columns. Such data can be represented as D = { x i } n i =1 ⊂ R d consisting of n instances and d dimensional features. Each data instance x i = ( x 1 i , x 2 i , ..., x d i ) may or may not hold strong relationship among features. The features in tabular data typically vary, comprising both numerical and categorical labels.

Our study on tabular few-shot learning focus on a very typical scenario in this field, adhering to the definition outlined in STUNT [28]. We have an unlabeled training set D u = { x u i } N u i =1 and a limited labeled support set S = { ( x s i , y s i ) } N S i =1 , x s i ∈ R d and y s i ∈ { 1 , 2 , ..., C } represent inputs and class labels, respectively. We assume that N u ≫ N S . Our goal is to predict the class labels of testing query set Q = { x q i } N Q i =1 . In the N -way K -shot setting, the classification is conducted with N targeted classes and each class in the support set has K labeled samples. Such scenarios are common in critical applications like credit risk assessment [7] and diagnosing patients with rare diseases [40].

4 Methodology

In this section, we introduce the overall design of the framework Diffusion-based Representation learning with Random Distance matching (D2R2) in detail. In order to leverage the semantic knowledge extraction capabilities of diffusion models, D2R2 employs a conditional diffusion model on the unlabeled dataset to learn an embedding space (Section 4.1). In order to enhance the clustering ability of the embedding space, facilitating subsequent classification, we modify the training process of the diffusion model, in order to ensure that the distances between instances align across the embedding space and various projected spaces. (Section 4.2). Figure 1 illustrates aforementioned

Random Distance Matching Loss

Numerical features

Categorical features

Random Projection

.....

.....

ConditionalDiffusionModel

ReconstructionLoss

∈p(xp t, Zo(xo))

DenoisingProcess

xT ~ N(0,1)

training process. In the classification stage, we construct a novel instance-wise iterative classifier to predict the testing samples on the learned embedding space (Section 4.3). Finally, due to the absence of a validation set for above unsupervised embedding space learning, we propose to use pseudo-label validation on the unlabeled dataset to select hyperparameters (Section 4.4). A summary of the training algorithm of D2R2 is presented in Appendix A.

4.1 Diffusion-based representation Learning

Diffusion-based generative models [42, 17, 44, 45] are latent variable models that use a Markovian noising and parameterized denoising process to model the data distribution thus generate realistic samples. Details of diffusion models are provided in the appendix B. The diffusion process for a given sample x 0 is defined as x t = √ ¯ α t x 0 + √ 1 -¯ α t ϵ , where ϵ ∼ N (0 , I ) , ¯ α t = ∏ t t ′ =1 (1 -β ′ t ) , β t is a pre-defined variance schedule, and the timestep t is known. The true noise is ϵ ( x t , t, x 0 ) = 1 / √ 1 -¯ α t ( x t - √ ¯ α t x 0 ) . Following DDPMs [17], the denoising model can be implemented by neural networks with learnable parameters ϕ by directly predicting the noise ϵ :

Diffusion models can be extended to conditional generative models ϵ ϕ ( x t , t, c ) by inserting the conditional information c to generate specific class samples [45, 1].

We now present the diffusion-based representation learning part of D2R2. We employ the diffusion models in a distinct way different from generation. We modify the diffusion model and repurposing it to act as representation extractor. Specifically, we represent the trainable embedding function as z θ : R d → R p , where z θ ( x 0 ) maps the noiseless input x 0 to the latent space of dimension p . Formally, the reconstruction loss for the diffusion-based representation learning is

where ϵ ϕ is a trainable noise model, and ˜ x 0 is an augmentation version of the noiseless data input x 0 . In our method, we use Gaussian noise perturbation ˜ x 0 = x 0 + σ N (0 , 1) . For numerical features we set σ i to be the standard deviation of numerical features in D u , and for categorical (one-hot) features, we use a fixed σ i to smooth the one-hot indicator values.

Reasons that the designed diffusion model can extract semantic knowledge come from twofold. Firstly, the diffusion model with powerful expressiveness encodes the information needed for denoising. Specifically, in conditional diffusion models, the noise reconstruction loss E t, x 0 , x t || ϵ ϕ ( x t , t, c ) -ϵ || 2 2 trains the noise prediction function ϵ ϕ ( x t , t, c ) to predict the true noise ϵ ( x t , t, x 0 ) given the noisy

sample x t , the knowing t and the condition information c . If c = x 0 , we could expect ϵ ϕ can almost perfectly recover ϵ . By replacing the conditional information c by a function z θ ( x 0 ) that maps to an embedding space with lower dimension than x 0 , we introduce an information bottleneck to the noise reconstruction process. This forces z θ to extract effective information for denoising from x 0 , leading to representation learning through the noise reconstruction loss (2) .

From another perspective, it is pointed out the noise reconstruction loss (2) can be expressed equivalently as the weighted noisy score matching loss E t, x 0 , x t [ λ ( t ) || s ϕ -∇ x t log p σ ( t ) ( x t | x 0 ) || 2 2 ] [1], where the weights λ ( t ) = σ 2 ( t ) are determined by the noise scale σ (1) < σ (2) < ... < σ ( T ) . The choice of noise scale σ ( t ) controls the granularity of the embedding function. We focus on larger timesteps thus extract the low-frequency semantic information rather than details.

4.2 Random distance matching

The acquired latent representation z θ from equation (2) is utilized for subsequent classification tasks, which heavily rely on distance information between the clusters within the embedding space. Although z θ ( x 0 ) learned through conditional diffusion models already exhibits clustering properties [1] to a certain extent, the learned embeddings do not prioritize capturing distance information between class structures, potentially hindering the effectiveness for downstream classification tasks. Taking inspiration that representations can be learned by training neural networks to predict distances in a randomly projected space [49], we propose the random distance matching (RDM) loss to align pairwise distances between the embedding space and a randomly projected space R r . Here we suppose that if two samples belong to the same class in the ground truth, their embeddings should be close to each other, which is reflected by the randomly projected features from any perspectives.

Specifically, we consider the random linear projections W ∈ R r × d with each element w i,j sampled i.i.d. from a fixed distribution. The RDM loss is as follows:

where x 0 , x ′ 0 are any two unlabeled data points sampled from D u and d is a metric. Since the dimensions and the scales of the two matching spaces might not be the same, we choose d to be the cosine distance.

Furthermore, in order to handle hybrid tabular data types, which involve both numerical and categorical features, we sample random projections from various distributions according to the specific feature types. We consider samples with d num numerical features and d cat categorical features. For numerical features, we sample a projection W num ∈ R r 1 × d num from symmetric uniform distribution: w i,j ∼ i.i.d. Unif ( -A,A ) ; for categorical features, we sample a projection W cat ∈ R r 2 × d cat from Bernoulli distribution: w i,j ∼ i.i.d. Bernoulli ( p ) . Considering both types of features, our random linear projection is defined as:

where [ x 0 ] num, [ x 0 ] cat are the numerical and categorical parts of x 0 , respectively.

Overall, we define a novel diffusion-based representation learning loss as follows:

where α is a hyperparameter to balance the noise prediction loss and RDM loss. We train ϕ and θ to minimize L D2R2 over the unlabeled dataset D u . The trained embedding function z θ ( x 0 ) is used as the representation function for the downstream classification tasks. Loss function L D2R2 considers learning of embeddings from two different angles. Diffusion models possess strong generative capabilities, compelling the reconstruction loss L recon ( ϕ, θ ) to ensure that the learned embedding z θ captures high-quality semantic information, which significantly influences the data distribution. The random distance loss L rdm ( θ ) adjusts the embedding to accommodate the distances between data points in a random projection space, thereby improving the clustering capability of the embedding space, rendering it suitable for subsequent classification tasks.

4.3 Instance-wise iterative prototype

Given the trained embedding function z θ ( x ) and the labeled support set S = { ( x s i , y s i ) } N S i =1 , we define the instance-wise prototype of support samples in the embedding space as c i = z θ ( x s i ) , where i = 1 , 2 , .., N s . We can predict the label of a query sample to be the same as the nearest prototype.

X

X

(a)2-shot classification

(b)Center-wise prototype

(c)Instance-wise prototype

The rationale for employing instance-wise prototypes instead of center-specific prototypes [41, 11], which averages the K -shot embeddings inside the j -th class as c j = 1 K ∑ y s i = j z θ ( x s i ) , is that embeddings from the same class may exhibit multimodality, while center-specific prototypes can only represent a uni-modal pattern, which might result in incorrect classification results (see Figure 2 for an illustration). On the other hand, in the few-shot learning scenarios, the scarcity of support samples brings large variance of prototypes, leading to unstable and imprecise classification outcomes. Drawing inspiration from the soft k-means algorithm [38], we leverage the weighted average of the query embeddings and support embeddings to create robust prototypes iteratively, referred to as the instance-wise iterative prototypes.

At the beginning, we initialize each prototype as the embedding of each support instance as c (0) i = c i = z θ ( x s i ) , where i = 1 , 2 , .., N s . At the l -th iteration step, the probability that the query x ∈ Q belonging to the i -th prototype is:

Then we update the prototypes at the ( l +1) -th iteration based on the weighted average of query embeddings:

After the L -th iteration, we predict the class of a query sample x to be the same class of the nearest prototype:

4.4 Pseudo-label validation

One challenge in the proposed unsupervised learning lies in the lack of a labeld validation set for hyper-parameter tuning. For example, in 1-shot classification, the only one labeled sample is used for training, with no additional labeled samples available for validation. We tackle this problem by generating pseudo labels for the unlabeled dataset using soft k-means of raw features.

Formally, we sample a validation set D val ⊂ D u . During each validation process, we randomly sample K ′ points { x k } K ′ k =1 ⊂ D val, which are regarded as pseudo support samples, forming K ′ distinct classes, i.e., S val = { ( x s k , y s k = k ) } K ′ k =1 . Weregard the remaining unlabeled samples Q val = D val \S val as pseudo query samples. Next, pseudo-labels are generated for Q val by applying instance-wise iterative prototype classification (Section 4.3) to the raw feature space, and those pseudo-labels serve as the 'ground-truth' labels for the query set Q val. Subsequently, we employ D2R2 on the validation set {S val , Q val } to predict the 'ground-truth' labels of Q val. We assess the validation performance to select hyperparameters for the training iteration.

5 Experiments

5.1 Experimental settings

For all the datasets, we randomly split 80% of the data for training and the remaining 20% for testing. As for N -way K -shot scenario, the support set is constructed by randomly selecting N × K samples, with K samples from each of the N classes from the training sets. Additionally, 20% of the training data is utilized for validation and hyperparameter tuning. We use one-hot encoding for categorical features and min-max scaling for numerical features, except for income data, for which we use standardized scaling (see Appendix D.2 for details). All experiment settings of baselines and D2R2 are the same as STUNT [28] for fair comparison. Details are provided in Appendix D.3 2 .

Datasets . We select nine datasets from the OpenML-CC18 benchmark [3, 6] to validate the performance of D2R2. Table 3 shows a summary of the datasets. Optdigits, mfeat-karhunen, diabetes contain and breast only numerical features; dna and mfeat-pixel consist of only categorical features; income, cmc and nomao contain both numerical and categorical features. A summary of the dataset information is provided in Appendix Table D.1.

Baselines . In order to assess the efficacy of our D2R2 framework, we compare the performance of D2R2 with four types of baselines, whose details are provided in Appendix E.

Supervised learning methods. We compare with established supervised learning methods such as CatBoost [35], k-nearest neighbors (kNN) classifier [32] according to the prototype, and TabPFN [18], a transformer-based network designed to make predictions on a small tabular dataset.

Semi-supervised learning methods. Such methods include Mean Teacher(MT) [46], Interpolation Consistency Training (ICT) [48], Pseudo-Label [25] and Meta Pseudo Labels (MPL) [33].

Few-shot meta-learning methods. We consider recent state-of-the-art meta-learning approaches such as CACTUs [19], UMTRA [22], SES [54] and STUNT [28]. STUNT is designed for tabular data while others are designed for image data, whose network structures are modified for tabular data[28].

Self-supervised learning methods. We compare with the state-of-the-art self-supervised methods for tabular data, including VIME[55], SubTab [47], SCARF [4], TabTransformer [20]. We utilize representations acquired from those models to conduct Center Prototype Classification [11].

5.2 Overall evaluation results

We compare our framework D2R2 with other state-of-the-art supervised, semi-supervised, selfsupervised and meta-learning methods. The classification accuracy of all methods is presented in Table 1. In particular, we carry out experiments under two few-shot settings: N -way 1-shot and N -way 5-shot. We report the mean accuracy across 100 random seeds.

We note that our D2R2 framework significantly outperforms baseline methods across diverse datasets. A Wilcoxon signed-ranks test is employed (Appendix D.5) to further demonstrate the statistical significance of comparison. When comparing the accuracy of the D2R2 with that of the best baseline on all datasets, Wilcoxon's P-value is below 0.05, significantly indicating the effectiveness of D2R2.

In the case of the diabetes, optdigit and karkunen datasets, where all features are numerical, STUNT method outperforms other baselines, while D2R2 demonstrates substantial improvement. In the high-dimensional dataset optdigit, D2R2 improves accuracy by 39% compared to the supervised method Catboost, 23% compared to the semi-supervised method Mean Teacher, 5% compared to the meta-learning method STUNT, and 22% compared to other self-supervised methods like SubTab. On the other hand, for pixel and dna datasets, which only contain categorical features, meta-learning such as UMTRA and SES are almost ineffective, while D2R2 still demonstrates superior accuracy, outperforming all other methods in both one-shot and five-shot settings. Similarly, for the large size dataset income, encompassing both numerical and categorical features, our framework consistently achieves a high classification accuracy of 72.08%, surpassing the leading baseline STUNT by 19%. In summary, it can be concluded that D2R2 demonstrates a robust capability to address few-shot

tabular data classification tasks, regardless of the number of shots, dataset size, feature dimension, or the proportion of categorical features.

Wespeculate that the superior performance of D2R2 is attributed to the effective learned representation of inputs along with the instance-specific prototypes. Upon visual examination of D2R2's embeddings in Figure 3, we observe clustering characteristics from point clouds, which demonstrating the embeddings' effectiveness for classification. Besides, we note the presence of multimodality in the embeddings, supporting the rationale for introducing the instance-wise prototypes.

5.3 Ablation study

We conduct an ablation study to demonstrate the efficacy of three components in D2R2, namely the diffusion representation (DR), random distance matching (RDM), and the instance-wise iterative prototype (IP). We compare the complete D2R2 design (DR+RDM+IP) with frameworks that exclude one or two of these components. We also conduct the center-specific prototype version of D2R2 (D2R2-c), which utilizes average embeddings of K -shot samples as prototype. The results of the ablation study are presented in the Table 2.

Based on the results, D2R2 surpasses all other variants, indicating that all three components collectively contribute to its superior performance. Specifically, removing the diffusion representation results in a significant degradation in performance, highlighting the crucial contribution of diffusion representation in capturing semantic knowledge. We also observe that DR+RDM shows the second best performance, highlighting the importance of incorporating random distance matching in the diffusion training process. Comparing with a center-specific prototype version of D2R2 (D2R2-c), we observe that the instance-wise prototypes leads to improvement results in optdigit and dna.

29-

福

20-

0=

1

-20

-

-20

Optdigits

Karkunen

Dna

Income

6 Conclusion

In this paper, we introduce D2R2, an innovative framework to address few-shot tabular challenges. The core idea of D2R2 is to utilize the strong expressive abilities of diffusion model, along with a random distance matching to construct representation learners. This method captures the semantic knowledge of unlabeled data and generate effective embeddings for downstream classification tasks, meanwhile adapting to the mixture feature types of the tabular data. Additionally, to accommodate multimodalities of embeddings, we devise the instance-wise iterative prototype classifier using labeled data. Furthermore, a pseudo-label validation scheme is designed for hyper-parameter selection. The superior performance of D2R2 is presented on diverse datasets, demonstrating the effectiveness of D2R2 over other baselines. We hope that our work will inspire new avenues for research in the field of diffusion model representation learning on tabular datasets.

References

Korbinian Abstreiter, Sarthak Mittal, Stefan Bauer, Bernhard Schölkopf, and Arash Mehrjou. Diffusion-based representation learning. arXiv preprint arXiv:2105.14257 , 2021.

Sercan Ö Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 6679-6687, 2021.

Arthur Asuncion and David Newman. Uci machine learning repository, 2007.

Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. arXiv preprint arXiv:2106.15147 , 2021.

Sungyong Baik, Janghoon Choi, Heewon Kim, Dohee Cho, Jaesik Min, and Kyoung Mu Lee. Meta-learning with task-adaptive loss function for few-shot learning. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9465-9474, 2021.

Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael Gomes Mantovani, Jan N van Rijn, and Joaquin Vanschoren. Openml benchmarking suites. In Proceedings of the NeurIPS 2021 Datasets and Benchmarks Track . 2021.

Longbing Cao. Ai in finance: challenges, techniques, and opportunities. ACM Computing Surveys (CSUR) , 55(3):1-38, 2022.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv:1904.04232 , 2019.

Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning. arXiv preprint arXiv:2401.14404 , 2024.

Yudong Chen, Chaoyu Guan, Zhikun Wei, Xin Wang, and Wenwu Zhu. Metadelta: A metalearning system for few-shot image classification. In AAAI Workshop on Meta-Learning and MetaDL Challenge , pages 17-28. PMLR, 2021.

Zitian Chen, Subhransu Maji, and Erik Learned-Miller. Shot in the dark: Few-shot learning with no base-class labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2668-2677, 2021.

Hao Cheng, Joey Tianyi Zhou, Wee Peng Tay, and Bihan Wen. Graph neural networks with triple attention for few-shot learning. IEEE Transactions on Multimedia , 2023.

Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. Cert: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766 , 2020.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9729-9738, 2020.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848 , 2022.

Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. arXiv preprint arXiv:1810.02334 , 2018.

Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678 , 2020.

Dahyun Kang, Heeseung Kwon, Juhong Min, and Minsu Cho. Relational embedding for few-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8822-8833, 2021.

Siavash Khodadadeh, Ladislau Boloni, and Mubarak Shah. Unsupervised meta-learning for few-shot image classification. Advances in neural information processing systems , 32, 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Dong Bok Lee. Meta-gmvae: Mixture of gaussian vaes for unsupervised meta-learning. 2021.

Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , volume 3, page 896. Atlanta, 2013.

Junjie Li, Zilei Wang, and Xiaoming Hu. Learning intact features by erasing-inpainting for fewshot classification. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 8401-8409, 2021.

Mario Molina and Filiz Garip. Machine learning for sociology. Annual Review of Sociology , 45:27-45, 2019.

Jaehyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, and Jinwoo Shin. Stunt: Few-shot tabular learning with self-generated tasks from unlabeled tables. arXiv preprint arXiv:2303.00918 , 2023.

Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 , 2018.

Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se-Young Yun. Boil: Towards representation change for few-shot learning. arXiv preprint arXiv:2008.08882 , 2020.

Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems , 34:11054-11070, 2021.

Leif E Peterson. K-nearest neighbor. Scholarpedia , 4(2):1883, 2009.

Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11557-11568, 2021.

Mihir Prabhudesai, Shamit Lal, Darshan Patil, Hsiao-Yu Tung, Adam W Harley, and Katerina Fragkiadaki. Disentangling 3d prototypical networks for few-shot concept learning. arXiv preprint arXiv:2011.03367 , 2020.

Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems , 31, 2018.

Guo-Jun Qi and Jiebo Luo. Small data challenges in big data era: A survey of recent progress on unsupervised and semi-supervised methods. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(4):2168-2187, 2020.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint arXiv:1803.00676 , 2018.

Eli Schwartz, Leonid Karlinsky, Rogerio Feris, Raja Giryes, and Alex Bronstein. Baby steps towards few-shot learning with multiple semantics. Pattern Recognition Letters , 160:142-147, 2022.

KShailaja, Banoth Seetharamulu, and MA Jabbar. Machine learning in healthcare: A review. In 2018 Second international conference on electronics, communication and aerospace technology (ICECA) , pages 910-914. IEEE, 2018.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems , 30, 2017.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems , 30, 2017.

Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for self-supervised representation learning. Advances in Neural Information Processing Systems , 34:18853-18865, 2021.

Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Arno Solin, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. Neural Networks , 145:90-106, 2022.

Hu Wang, Guansong Pang, Chunhua Shen, and Congbo Ma. Unsupervised representation learning by predicting random distances. arXiv preprint arXiv:1912.12186 , 2019.

Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur) , 53(3):1-34, 2020.

Jiangtao Xie, Fei Long, Jiaming Lv, Qilong Wang, and Peihua Li. Joint distribution matters: Deep brownian distance covariance for few-shot classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7972-7981, 2022.

Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O O Pinheiro. Adaptive cross-modal few-shot learning. Advances in Neural Information Processing Systems , 32, 2019.

Xingyi Yang and Xinchao Wang. Diffusion model as representation learner. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 18938-18949, 2023.

Han-Jia Ye, Lu Han, and De-Chuan Zhan. Revisiting unsupervised meta-learning via the characteristics of few-shot tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):3721-3737, 2022.

Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems , 33:11033-11043, 2020.

Baoquan Zhang, Xutao Li, Yunming Ye, Zhichao Huang, and Lisai Zhang. Prototype completion with primitive knowledge for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3754-3762, 2021.

A Algorithm

Algorithm 1 Training D2R2

Require: Unlabeled dataset D u = { x i } N u i =1 , perturbation scale σ , batch size B , diffusion horizon T , noise schedule { β t } T t =1 , { α t } = { 1 -β t } , { ¯ α t } = { ∏ t t ′ =1 α t ′ } , number of numerical features d num, number of categorical features d cat, feature dimension d = d num + d cat, learning rates γ 1 , γ 2 , hyperparameters: α, p, A .

Initialize: noisy network ϵ ϕ , embedding network z θ

while not done do Sample two batchs of unlabeled data x , x ′ ∼ D u . Sample a perturbation copy ˜ x ∼ x + σ N (0 , 1) . Sample timestep t ∼ 1 , 2 , ..., T and noise ϵ ∼ N (0 , 1) . Compute the noisy sample x t = √ ¯ α t x + √ 1 -¯ α t ϵ . L recon ( ϕ, θ ) = || ϵ ϕ ( x t , t, z θ (˜ x )) -ϵ || 2 2 . Sample matrix W n ∈ R d × r 1 with ( W n ) ij ∼ i.i.d. Unif ( -A,A ) . Sample matrix W c ∈ R d × r 2 with ( W c ) ij ∼ i.i.d. Bernoulli ( p ) . Compute W x = Concat ( W n [ x ] num , W c [ x ] cat ) , Compute W x ′ = Concat ( W n [ x ′ ] num , W c [ x ′ ] cat ) , L rdm ( θ ) = || cosine ( z θ ( x ) , z θ ( x ′ )) -cosine ( W x , W x ′ ) || 2 , ϕ ← ϕ -γ 1 B · ∇ ϕ L recon ( ϕ, θ ) . θ ← θ -γ 2 B · [ ∇ θ L recon ( ϕ, θ ) + α ∇ θ L rdm ( θ )] . end while return Embedding function z θ .

Algorithm 2 Pseudo-label Validation

Require: Unlabeled validation set D val = { x i } N v i =1 , number of pseudo-classes K ′ , number of iteration I

Create instance-wise iterative prototype classifier f ( ·| , S , Q ) using the raw feature vectors. Create pseudo-label in Q ← { ( x i , f ( x i |S )) | x i ∈ Q} .

return Pseudo support set S , pseudo query set Q

B Diffusion Models

Diffusion-based generative models [42, 17, 44, 45] are latent variable models that use a Markovian noising and parameterized denoising process to model the data distribution. The forward noising process of horizon T follows a pre-defined variance schedule β 1 , ..., β T that encodes data distribution x 0 ∼ p ( x ) :

and assume the prior q ( x t ) ∼ N (0 , I ) . The reverse denoising process is modeled as p ϕ ( x 0: T ) := T

N ( x t ; 0 , I ) ∏ t =1 p ϕ ( x t -1 | x t ) with learnable parameters ϕ . Following DDPMs [17], the denoising model can be implemented by neural networks directly predicting the noise ϵ :

where x t = √ ¯ α t x 0 + √ 1 -¯ α t ϵ, ϵ ∼ N (0 , I ) and ¯ α t = ∏ t t ′ =1 (1 -β ′ t ) . Diffusion models can be straightforwardly extended to conditional generative models ϵ ϕ ( x t , t, c ) by inserting the conditional information c [45, 1].

C Random Linear Projection

We generate random matrices from different distributions to cater for the numerical features and categorical features separately. In order to integrate the projections derived from numerical and categorical spaces, it is necessary to select A (for Uniform ( -A,A ) ) and p (for Bernoulli ( p ) ) in a consistent manner to equitably consider the information from heterogeneous type. In this paper, we adjust the random weights according to the consistent activation variance, that is we set A = √ 3 p (1 -p ) and treat p as a hyperparameter, and re-weight the matrices according to the number of columns, i.e., W num ← W num / √ d num , W cat ← W cat / √ d cat. Such choice of parameters preserve the variance of the row-sum of projection matrices . Additionally, we sample W num ∈ R d num × d num and W cat ∈ R d cat × d cat to get rid of the need to treat r 1 , r 2 in Algorithm (1) as hyperparameters. We leave the study of different random matrices to the future work.

D Experiment details

D.1 Dataset Information

D.2 Data normalization

For income dataset, we apply standardized scaling to each numerical feature x i :

And we employ min-max scaling numerical features of other datasets:

D.3 Hyperparameter Details

We employ fully connected layers with identical layer counts and hidden dimensions to model the diffusion noise and the embedding function. The training procedure for the diffusion model follows the same approach as in DDPM [17]. The configuration for the diffusion model and neural networks (NN) are as following:

In our experiments, we search three hyperparameters during the validation process.

All other experiment settings of D2R2 and other baseline are the same as STUNT [28] for fair comparison.

D.4 Choosing Noise Levels

The timestep t = 1 , ..., T indicates noise levels in diffusion models. In our experimental analysis, we have noted a trend where performance sees an improvement with an increase in t during the initial

stages. However, this trend of performance enhancement plateaus after the first few timesteps, which tends to stabilize with minimal fluctuations, as shown in Table 6. As t approaches T , noise levels rise, thus effective denoising requires that all information about x 0 be thoroughly encoded, which contains the most information about the data class. We focus on larger timesteps thus extract the low-frequency semantic information rather than details. Thus in our experiments, we selected the last step T for all datasets.

D.5 Wilcoxon Test

The Wilcoxon signed-ranks test is a statistical method used to compare the means of two related groups when assumptions of normal distribution are not met. Particularly useful for paired data, it involves ranking the absolute differences between paired observations and calculating a test statistic for assessing significance. The procedure includes the following:

Calculate the differences: D i = X i -Y i , where X i and Y i are paired observations.

Rank | D i | from the smallest to the largest, resolving ties by averaging ranks.

Calculate signed ranks: Assign positive ranks to positive differences and negative ranks to negative differences.

Calculate the test statistic W , which is the sum of the positive or negative ranks, depending on which sum is smaller.

Compare W to critical values: Determine significance by comparing W to critical values from the Wilcoxon signed-ranks table or using statistical software.

In summary, the test helps identify if a significant difference exists between the groups' means, even when data doesn't meet assumptions for parametric tests.

D.6 Model Complexity

The model complexity is manageable through our framework's utilization of two neural networks: the embedding network and the noise prediction network, both of which are demonstrated as 3-layer MLPs in our paper. These settings (hidden dimensions, embedding dimensions and model structure)

can be adjusted to meet different needs. In terms of computational efficiency, The main timeconsuming factor after the embedding model is the calculation of the instance-wise iterative prototype. For an N -way K -shot problem with L iterations and a query set of size Q , the computation complexity is O ( NKLQ ) . In few-shot learning, where N,K,L are small, the computation complexity is linear in the size of the query set.

E Baseline Details

we compare with supervised learning methods such as CatBoost [35], k-nearest neighbors (kNN) [32], which denotes the nearest neighbor classifier according to the prototype of the input data, and TabPFN [18], which is a transformer that that solves small tabular classification problems.

Mean Teacher (MT) [46] is a semi-supervised learning framework designed to enhance model generalization by establishing a robust interaction between labeled and unlabeled samples. This framework draws inspiration from the "teacher-student" training approach, where the "teacher" refers to a model averaging multiple student model predictions, while the "student" is the model being trained.

Interpolation Consistency Training (ICT) [48] uses MT framework. While ICT aims to improve the model performance by augmenting additional data points through interpolation, with the expectation that the model's predicted values align consistently with the interpolated labels.

Meta Pseudo Labels [33] continually adjusts the teacher model based on feedback from the student model's performance on labeled datasets. Both teacher and student models are trained concurrently, with the teacher model learning to generate improved pseudo-labels while the student model learns from these pseudo-labels.

VIME[55] is a form of self-supervised learning that derives valuable representations by intentionally corrupting random features and subsequently predicting the corrupted location. We utilize VIME representations for conducting k-nearest neighbor classification.

SubTab [47] employs three effective pretext task losses, namely, reconstruction loss, contrastive loss, and distance loss, as part of its self-supervised learning methodology. We utilize SubTab representations for conducting k-nearest neighbor classification.

SCARF [4] takes a SimCLR-like [8] contrastive loss between the sample and its corrupted version. We utilize SubTab representations for conducting k-nearest neighbor classification.

TabTransformer [20] is built upon self-attention based Transformers. The Transformer layers transform the embeddings of categorical features into robust contextual embeddings to achieve high prediction accuracy in tabular context. Experiments shows that TabTransformer performs better than TabNet [2]. We utilize TabTransformer representations for conducting k-nearest neighbor classification.

CACTUs [19] creates episodic tasks by dividing the features extracted from an earlier-trained unsupervised feature embedding network using various objective functions, and then trains the few-shot learner on these tasks.

UMTRA [22] employs a domain-specific data augmentation approach to create synthetic tasks for the meta-learning stage. In doing so, the episodic tasks that are formed are constrained by the data augmentation strategy.

SES [54] considers the augmented sample as part of the same pseudo-class and implements robust strategies for unsupervised meta-learning, which include ample episodic sampling, challenging mixed supports, and task-specific projection heads.

F Limitations

Our approach is tailored to the specific characteristics of tabular data, including data scarcity, multiple modalities, mixed feature types, and sensitivity to changes in column order. Thus it is only effective in scenarios involving tabular data, and may not work well for CV and NLP tasks.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: By clearly stating the main claims in the abstract and introduction, the reader is able to understand the purpose and significance of the paper, as well as the specific contributions and scope of the research.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: By openly discussing these limitations, we are aware of the potential shortcomings.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Due to the complex architecture of diffusion models, it is typically hard to analyze the theoretical behaviors, such as convergence rates, of our proposed model and other diffusion-based models. Thus, we leave it for future work.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully disclose all the information needed to reproduce the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all the training and test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports statistical significance of the experiments by Wilcoxon signed-ranks test.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provide sufficient information on the computer resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Discussing both potential positive and negative societal impacts of the research is important for providing a comprehensive evaluation.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The assets used in the paper have been explicitly mentioned and received proper acknowledgment from the owners.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We propose a new model in the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: There is no potential risk incurred by study participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach named Diffusion-based Representation with Random Distance matching (D2R2) for tabular few-shot learning. It leverages the powerful expression ability of diffusion models to extract essential semantic knowledge crucial for the denoising process. During the training process of the designed diffusion model, it introduces a random distance matching to preserve distance information in the embeddings, thereby improving the effectiveness of classification. During the classification stage, it introduces an instance-wise iterative prototype scheme to improve performance by accommodating the multimodality of embeddings and increasing clustering robustness. Experiments demonstrate the effectiveness of the proposed method. The main contributions of this paper are:\n\n- It may be the first to propose a specifically designed diffusion method to learn semantic knowledge for tabular data.\n- It proposes an innovative framework, D2R2, to extract representations in tabular few-shot learning.\n- It introduces a novel classifier with instance-wise iteration prototypes to further improve few-shot classification performance.\n\nSTRENGTHS:\n- The problem studied in this paper is interesting and valuable.\n- This paper is well written and in good sharp, which is easy to follow.\n- Experimental results are promising and can validate the effectiveness of the proposed method.\n\nWEAKNESSES:\n- In my opinion, the description of the shortcomings of existing tabular few-shot learning methods in the introduction is not sufficient. Specifically, the authors should further emphasize the problems the heterologous feature types bring that SOTA tabular few-shot learning methods ignore, to demonstrate the necessity and innovation of the proposed method.\n- On the basis of the first issue, the introduction of diffusion models should be based on the drawbacks of existing problems, rather than solely on their strong expressiveness. The authors should point out the motivation for the leverage of diffusion models.\n- The emergences of semi-supervised learning and self-supervised learning mentioned in this paper are somewhat abrupt. The authors should explain the relationship between them and tabular few-shot learning, as well as their relevance to the D2R2 proposed in this paper.\n\nQUESTIONS:\n1. The authors should further emphasize the problems the heterologous feature types bring that SOTA tabular few-shot learning methods ignore, to demonstrate the necessity and innovation of the proposed method.\n2. The authors should point out the motivation for the leverage of diffusion models.\n3. The authors should explain the relationship between semi-supervised learning, self-supervised learning, and tabular few-shot learning, as well as their relevance to the D2R2 proposed in this paper.\n4. Tabular few-shot learning seems not to emphasize the variation of categories in the training (base ) and test (novel) datasets, does this indicate that it ignores the requirement for the generalization of the algorithm?\n\nLIMITATIONS:\nThe authors point out the limitations of this work. No obvious potential negative societal impact remains.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a new diffusion-based representation learning method for tabular data, namely D2R2, specifically for few-shot learning. It is the first paper to use the diffusion model for tabular data representation learning. The method trains a conditional diffusion model with a combined loss of vanilla diffusion reconstruction loss and a random distance match loss to learn an embedding function $z_{\\theta}$. $z_{\\theta}$ encodes a representation of the data with different noise levels. The resulting embedding is used as the conditional information to the diffusion model. Further, random linear projections are performed on the original data, and the random distance match loss is to preserve distances in the embedding space and the random projection space. Two distinct projections are used for numerical and categorical data. After training, instance-wise iteration prototypes are generated for more accurate and stable few-shot learning. The paper conducts experiments on 7 datasets from OpenML-CC18, showing superior performances than a variety of baselines.\n\nSTRENGTHS:\n1. The paper introduced the first diffusion-based method for tabular few-shot representation learning.\n2. The proposed method has superior performance compared to SOTA methods.\n3. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. Even though D2R2 is the first method to utilize the diffusion model for representation learning, it is not the first paper to analyze/utilize the diffusion model for representation learning. The paper lacks discussion on related works in related areas such as [1] [2] [3].\n2. After training, the paper only extracts features of clean tabular data during few-shot learning, while the embedding function is learned for different noise levels. There is no analysis of the effect of different noise levels as input to the embedding function for few-shot learning.\n3. The method is limited to only tabular data, such as the specific projection head design for numerical and categorical data.\n[1] Xinlei Chen, Zhuang Liu, Saining Xie, Kaiming He. Deconstructing Denoising Diffusion Models for Self-Supervised Learning. 2024.\n[2] Sarthak Mittal, Korbinian Abstreiter, Stefan Bauer, Bernhard Schölkopf, Arash Mehrjou. Diffusion Based Representation Learning. 2023.\n[3] Xingyi Yang, Xinchao Wang. Diffusion Model as Representation Learner. 2023.\n\nQUESTIONS:\n1. See weakness 1. What's the relationship and key novelty in the method compared to existing methods?\n2. Will other noise levels be more useful for few-shot learning? Or will a combination of different noise levels be better?\n\nLIMITATIONS:\nLimitation is discussed and the paper states to have discussed potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method, Diffusion-based Representation and Random Distance Matching (D2R2), to address the challenge of few-shot learning on tabular data. This approach leverages the robust representational capacity of diffusion models to extract essential semantic knowledge from tabular data, thereby enhancing the performance of downstream classification tasks. Additionally, the paper presents the Random Distance Matching (RDM) loss function, which preserves distance information within embedding vectors during diffusion model training, further boosting classification performance. During the classification phase, the authors propose an instance-based iterative prototype scheme to accommodate the multimodal characteristics of the embedding vectors, thus improving clustering robustness. Extensive experiments conducted on various few-shot learning benchmarks for tabular data demonstrate that D2R2 surpasses current benchmark methods. In conclusion, this paper introduces an innovative few-shot learning method for tabular data, achieving significant advancements in extracting effective semantic representations and improving classification performance.\n\nSTRENGTHS:\n-\tThe use of diffusion models to extract effective semantic representations from tabular data is a novel approach that overcomes the limitations of traditional supervised learning methods.\n-\tThe introduction of the Random Distance Matching (RDM) loss function, which preserves distance information within the embedding vectors during diffusion model training, further enhances classification performance.\n-\tExtensive experiments conducted on multiple few-shot learning benchmarks demonstrate that D2R2 outperforms the latest benchmark methods, validating the effectiveness of the proposed method.\n-\tThe paper provides comprehensive explanations and descriptions of the various components of the method, including the expressive power of the diffusion model, the role of the RDM loss function, and the principles of the iterative prototype scheme, making the methodology and process clear and understandable.\n\nWEAKNESSES:\n-\tThe paper lacks a clear description of the input format for tabular data. Specifically, it is unclear whether the tabular data is input in text form or as images. Clarification on this point is essential, as the input format may significantly impact performance.\n-\tThe paper utilizes a diffusion model to enhance the semantic representational capacity of tabular data. However, it remains uncertain whether the diffusion model is effective across all types of tabular data. Detailed demonstrations and explanations of how the model captures and enhances these semantic features are needed to substantiate its broad applicability.\n-\tThe authors introduce a diffusion model that differs from existing methods. It is crucial to provide additional information regarding the model's parameter count and FLOPs (floating point operations per second) to allow for a thorough comparison of its computational efficiency with that of other models. This would help in understanding the trade-offs between performance gains and computational costs.\n\nQUESTIONS:\nPlease see the Weakness section.\n\nLIMITATIONS:\nThe author has already provided a description of the limitations, but it needs to be further elaborated.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper proposed a few-shot learning framework for tabular data by designing a diffusion based semantic knowledge encoder and introducing a random distance matching mechanism to preserve distance information in the embeddings. During classification, an instance-wise iterative prototype scheme is utilized to improve performance by accommodating the multimodality of embeddings and increasing clustering robustness. Extensive experiments on multiple datasets show that  D2R2 achieves state-of-the-art performance compared to other schemes.

STRENGTHS:
1. The proposed approach that combines the diffusion model with random distance matching for tabular few-shot learning is relatively unexplored compared to classical few-shot learning tasks.
2. The paper is well-written and organized, with detailed explanations of the challenges, methodology, and experimental results.

WEAKNESSES:
1. The feature dimensions of the benchmark datasets are relatively low, which leaves the the scalability to large datasets or those with extremely high dimensionality not fully explored.
2. The pseudo-label validation scheme for hyperparameter selection relies on the assumption that the clustering of raw features can provide a reliable proxy for true labels, which might not always hold in practice.

QUESTIONS:
1. How does the proposed method scale with larger datasets or those with higher feature dimensions? Are there any practical limitations or considerations for scaling?
2. Could the authors provide more empirical evidence or analysis on the pseudo-label validation scheme, such as when the clustering might not accurately reflect the true class structure?

LIMITATIONS:
No potential negative societal impact is observed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17