D2R2: Diffusion-based Representation with Random Distance Matching for Tabular Few-shot Learning
Authors: Ruoxue Liu
Linjiajie Fang
Wenjia Wang
Bingyi Jing
Venue: NeurIPS 2024 poster (2024)
Keywords: Classification
Few-shot learning
Tabular data
Representation learning
Diffusion models
Self-supervised learning
Primary area: deep_learning_architectures
TL;DR: We propose a novel framework for tabular few-shot learning, comprising a diffusion-based model with random distance matching for representation learning, and an instance-wise iterative prototype scheme for few-shot classification.

Abstract: Tabular data is widely utilized in a wide range of real-world applications. The challenge of few-shot learning with tabular data stands as a crucial problem in both industry and academia, due to the high cost or even impossibility of annotating additional samples. However, the inherent heterogeneity of tabular features, combined with the scarcity of labeled data, presents a significant challenge in tabular few-shot classification. In this paper, we propose a novel approach named Diffusion-based Representation with Random Distance matching (D2R2) for tabular few-shot learning. D2R2 leverages the powerful expression ability of diffusion models to extract essential semantic knowledge crucial for denoising process. This semantic knowledge proves beneficial in few-shot downstream tasks. During the training process of our designed diffusion model, we introduce a random distance matching to preserve distance information in the embeddings, thereby improving effectiveness for classification. During the classification stage, we introduce an instance-wise iterative prototype scheme to improve performance by accommodating the multimodality of embeddings and increasing clustering robustness. Our experiments reveal the significant efficacy of D2R2 across various tabular few-shot learning benchmarks, demonstrating its state-of-the-art performance in this field.

D2R2: Diffusion-based Representation with Random Distance Matching for Tabular Few-shot Learning
Ruoxue Liu HKUST rliuaj@connect. ust.hk
Linjiajie Fang HKUST
Wenjia Wang HKUST (GZ) and HKUST wenj wenjiawang@ust. hk
1fangaddconnect. ust.hk
Bing-Yi Jing SUSTech
ingbyysustechh edu.
Abstract
Tabular data is widely utilized a wide range of real-world applications. The challenge of few-shot learning with tabular data stands crucial problem in both industry and academia, due to the high cost or even impossibility of annotat- ing additional samples. However, the inherent heterogeneity of tabular features, combined with the scarcity of labeled data, presents a significant challenge in tabular few-shot classification. this paper, we propose novel approach named Diffusion- Representation with Random Distance matching (D2R2) for tabu- lar few-shot learning. D2R2 leverages the powerful expression ability of diffusion models to extract essential semantic knowledge crucial for denoising process. This semantic knowledge proves beneficial few-shot downstream tasks. During the training process of our designed diffusion model, we introduce random distance matching to preserve distance information the embeddings, thereby improving effectiveness for classification. During the classification stage, introduce an instance-wer wise iterative prototype scheme to improve performance by accommodat- ing the multimodality of embeddings and increasing clustering robustness. Our experiments reveal the significant efficacy of D2R2 across various tabular few-shot learning benchmarks, demonstrating its state-of- offte-of-hh-arn performance in this field.
Introduction
Learning with large set of data with only few labeled samples is essential requirement for industry and academia, primarily due to the high cost of annotating samples. However, accurately clas- sifying new data with scantily labeled training challenge, as the effectiveness of statistical and modern deep learning systems supervised learning heavily relies on the large size of the labeled set [36]. This underscores the need for research in few-shot learning in situations where labeled data is scarce. For example, in one-shot learning, each class of the training set contains only labeled sample, and the goal is classify
and unlabeled training set poses formidable
test samples.
While few-shot learning has garnered considerable attention in computer vision (CV) [9, 21] and natural language processing (NLP) [30, 31], remains relatively under-explored the field of tabular data. Nevertheless, few-shot learning holds great importance in the context of tabular data, as limited labeled tabular data inherently common in many real-world applications, such as fraud detection [7], disease diagnosis [40], and social science [27]. However, modeling with limited labeled tabular data presents three significant challenges.
Equal contribution. Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The first challenge is due to the scarcity of labeled training samples, which leads to a deficient understanding of the test class. In this case, semantic knowledge, encompassing general and low- frequency information not tied to any particular task, can be highly beneficial in few-shot learning [39, 52, 56]. This is because representing a class with limited labeled samples inherently involves ambiguity, necessitating the use of semantic knowledge acquired from unlabeled ones to refine the class definition and address this ambiguity. Although for CV [21] and NLP [31], semantic knowledge can be effectively derived based on the learned spatial structure patterns between pixels or tokens, it is more challenging to extract semantic knowledge for tabular data, since they typically lack local relationships between columns.
The second challenge arises from the diverse characteristics of tabular features. Tabular data comprises numerical and categorical features. Numerical features contain ordinal values but have multiple modes, while categorical features present distinct and incomparable values. Many existing statistical and deep neural network models struggle to effectively handle these mixed data types. This challenge underscores the importance of simultaneously modeling continuous and categorical features.
The third challenge pertains to the utilization of prototype classification with tabular data. Numerous approaches suggest employing average embeddings of the labeled support set for classification [49, 11, 28J. However, the limited labeled support samples lead to large variance and imprecision of the prototypes, highlighting the need for more information from unlabeled samples. Additionally, embeddings from the same class may exhibit multimodal behavior (see Figure 3 for examples), which means that using the average embeddings of one class could result in inaccurate classification results.
In addressing the aforementioned challenges of few-shot learning, significant efforts have been made over the past decade. One approach employs a meta-learning scheme. A common method is to generate a pseudo-label for each unlabeled data and then train the model using pseudo-label [25, 43, 28]. Other approaches utilize self-generated tasks, such as CACTUs [19], UMTRA [22] and Meta-GMVAE [24]. However, despite the effectiveness of prior works on image datasets, we find that applying each method to the tabular domain is highly non-trivial. These methods often assume uniformity in feature types and rely on strong spatial and sequential relationships among features, which do not hold for tabular data. For instance, augmentation techniques are readily applicable to images, where spatial relationships between pixels can be leveraged, as shown by UMTRA [22]. However, such techniques are difficult to apply to tabular data, which lacks these structural patterns [4]. STUNT [28] is designed for tabular data using proxy-based approach. However, this methods may struggle to acquire useful semantic knowledge due the disparities between the constructed tasks and the target task, leading to diminished performance for testing. Moreover, it applies the same methods to both numerical and categorical features, neglecting the unique information in different types of data. This oversight hampers the model's s effectiveness. Recent studies have showed that self-supervised learning can leverage unlabeled datasets to acquire semantic knowledge and transferable representations of images [16] and languages [14]. However, these approaches heavily rely on augmentation schemes. For tabular data, the heterogeneous feature types pose challenges for augmentation, and only a few methods have been proposed for tabular data. State-of-ther ohette-rrtt methods along this line include VIME [55], SCARF [4], SubTab [47].
In this work, we propose novel framework named Diffusion-basee Representation with Random Distance Matching (D2R2) for tabular few-shot learning. D2R2 extracts representations by training a designed conditional diffusion model and aligning the distances from various random projection spaces. Subsequently, we introduce the instance-wise iterative prototype to conduct few-shot clas- sification, which addresses the multimodal behavior of embeddings and enables the creation of robust classifier. Finally, we design an unsupervised validation scheme to address the absence of the labeled validation set for hyperparameter selection. Different from meta-learning and self superviedd learning methods, D2R2 does not relies on self generrene tasks or augmentation schemes, but instead uses general training process. The D2R2 framework offers the following advantages:
Considering the first challenge mentioned earlier, we avoid to rely on proxy methods or augmentation methods, which are limited by the lack of local relationships in tabular data. Instead, we create an information bottleneck for extracting semantic knowledge, named D2R2, which leverages the strong expressiveness of the diffusion model and distance information from pairwise comparison. Firstly, we modify the diffusion model to serve as a representation extractor. The powerful expressiveness of diffusion model enables the extraction of semantic knowledge crucial to denoising. And the information bottleneck induced by conditional information, obtains www-frequency information over
2

---

details. This approach outperforms alternative representation learning and meta- meta-learnng methods, as demonstrated by our experiments. Moreover, the representations exhibit robustness due to the introduction of designed diffusion noise schedule and subsequent denoising process, which reinforces their stability against tabular column perturbations.
Nevertheless, relying solely on diffusion-based representations may not provide sufficient distance information to build subsequent prototype classifier, which classifies test samples based on their distance between prototypes. Hence, we introduce a random distance matching (RDM) loss during the training of the diffusion model obtain the distance information. One intuition of applying RDM that if the class structure can be characterized by certain distance information, the embeddings should preserve such distance structure. Another intuition is that if two samples belong to the same class in the ground truth, their embeddings should appear similar from any perspective. Random distance matching achieves these ideas by projecting data into diverse random spaces, which encompass class structure information. Consequently, if two data points are close in multiple random spaces, they are highly likely to belong to the same class. Besides, two distinct Random Distance matching are designed for continuous and discrete features based their characteristics, respectively. In this way, we can simultaneously model numerical and categorical features within one model, addressing the second challenge mentioned earlier. Overall, our newly designed D2R2 training process not only extracts high-quality semantic knowledge for few-shot learning but also reveals distance information for prototype classification, while remaining adaptive to mixture types of tabular data.
Moreover, during the classification phase, we predict the class of a test sample to be the same as the class of the nearest prototype. Given that the embedding of single class may demonstrate multimodality, we introduce novel instance-wisg iterative prototype. This approach tackles the issue of multimodal behavior by creating several prototypes within one class and improving the robustness of prototypes by iterative refinement, resolving the third challenge previously mentioned.
Lastly, to address the absence of a labeled validation set in unsupervised learning for hyperparameter selection, we devise validation scheme by generating pseudo-labels for the unlabeled dataset using soft k-means clustering of raw features.
Our contribution is summarized follows: To the best of our knowledge, we are the first to propose specifically designed diffusion method to learn semantic knowledge for tabular data.
We propose an innovative framework, D2R2, to extract representations in tabular few-shot learning. which built upon the designed diffusion process and random distance matching. D2R2 not only captures high-quality semantic knowledge. but also incorporates distance information for the subsequent prototype classifier. Moreover, it adapts well to mixture types of tabular data.
To further improve few-shot classification performance, we introduce novel classifier with instance-wist wise iteration prototypes. This classifier is able to construct highly accurate and stable prototypes, while also revealing the multimodal behavior of a single class.
We conduct extensive experiments to evaluate our framework for tabular few-shot learning, com- paring our method with 15 state-of-the-art literature baselines across nine datasets. Among various tabular datasets, D2R2 outperforms other baselines by a significant margin.
2 Related work
Supervised and Semi-supervised learning. Some supervised classifiers have strong ability to learn from limited samples. Methods along this line include CatBoost [35], TabPFN [18], a transformer- based network designed to make predictions on a small tabular dataset; k-nearest neighbor classifier (kNN) [32], the nearest neighbor prototype classifier. However, they are inadequate in few-shot learning because they are not intended learn the semantic knowledge. frameworks are designed to improve model generalization by creating strong connection between limited labeled samples and unlabeled samples, including Mean Teacher (MT) [46], Interpolation Consistency Training (ICT) [48], and Meta Pseudo Labels (MPL) [33]. In semi-supervised learning, a model trains on labeled data and then predicts labels for unlabeled data, However, this method requires more labeled data than sho-shot learning contexts.
supirvupe learning
3

---

Few-shot meta-learning. Few-shot learning aims to train models to adapt to downstream tasks with minimal labeled examples. This can be accomplished by employing meta-learning techniques on related tasks, enabling the acquisition of prior knowledge that can be utilized to solve new tasks [50]. Existing deep few-shot methods can he divided into three main categories. First, optimization- based meta-learning methods [5, 15, 26, 29] devise a proficient optimization strategy that adapts to downstream tasks effectively. Second, metric-based methods [13, 30, 34, 51] concentrate on latent space to derive meaningful feature embeddings, and then make predictions based on the similarity between support and query embeddings. Third, data generative strategies emphasize creating more varied samples to train more precise classifier. Most research on few-shot meta-learning focuses on NLP and CV tasks, such as CACTUs [19] and UMTRA [22], while a small subset tackles tabular few-shot learning. STUNT [28] meta-leams generalizable knowledge from self- self-gene tasks on an unlabeled tabular training set. However, these methods may generate ineffective semantic knowledge because of the gap between the self-generated tasks and the testing task.
Self-supervised learning. Our research addresses scenarios where an unlabeled training set and a small labeled support set (K-shot) are used to predict class labels for test query set, as detailed in Section 3 of our paper. Self-supervised learning is particularly effective at developing robust representations from unlabeled data [8, 37J. These approaches focus on pre-training the representation by utilizing domain- speciffc inductive biases, like the spatial relationships in images. Notably, prior research has demonstrated the effectiveness of self- supervisdd learning in few-shot scenarios relative to meta- veeta-lerrnng techniques [12]. Chen et al. [10] added an autoencoder in the diffusion process and Yang et [53] introduces latent Denoising Autoencoder architecture where the learned representations are used for Denoising Autoencoder, but the Autoencoder reconstruction process is less suitable for tabular data as discussed in STUNT [28]. Other works on self-supervised learning schemes rely on augmentation schemes. It unclear how to extend such methods to the tabular domain due to the heterogeneous characteristics of tabular datasets. Moreover, for tabular data, multimodal and categorical features make augmentation difficult, and few effective methods have been proposed in tabular data setting. Recent state-of-the-arr tabular learning augmentation techniques include masking cell: VIME [55], constructing subsets: SubTab [47], and contrastive learning: SCARF [4]. However, they fail to deliver substantial performance enhancements for few-shot tabular learning in our experiments. Rather, we train the unlabeled dataset through an unsupervised diffusion framework that does not rely on the effect of augmentation.
3 Problem definition
In this paper, we explore few-shot learning for tabular classification. Tabular data refers to the dataset organized in tables, which is a structured format that presents information in rows and columns. Such data can be represented as D- = {æ: H C Rd consisting of n instances and d dimensional features. Each data instance a; = (x: x ...., xx) may or may not hold strong relationship among features. The features in tabular data typically vary, comprising both numerical and categorical labels.
Our study on tabular few-shot learning focus on a very typical scenario in this field, adhering to the
N. definition outlined in STUNT [28]. We have an unlabeled training set D., = [x and a limited labeled support set S = {({] v() N xf E IRd and yf E (1,2,..., C} represent inputs and class labels, respectively. We assume that N, >> Ng. Our goal is to predict the class labels of testing query set Q {x]] [x]]] No.
the N-way K-shot setting, the classification is conducted with N targeted classes and each class in the support set has K labeled samples. Such scenarios are common in critical applications like credit risk assessment [7] and diagnosing patients with rare diseases [40].
4 Methodology
In this section, we introduce the overall design of the framework Diffusion-based Representation learning with Random Distance matching (D2R2) detail. In order to leverage the semantic knowledge extraction capabilities of diffusion models, D2R2 employs conditional diffusion model on the unlabeled dataset to learn an embedding space (Section 4... order to enhance the clustering ability of the embedding space, facilitating subsequent classification, we modify the training process of the diffusion model, in order to ensure that the distances between instances align across the embedding space and various projected spaces. (Section 4.2). Figure I illustrates aforementioned
4

---

Random Distanoe Matching Loss
Celgorcs
Randoon Projecton
Z
Conditional Diftusion Model cp 1, Zr(āa))
Reconstrutionn
Danaising Process
-F0001
Figure 1: The diagram depicts the training process of the embedding space. Specifically, after noise perturbation, each instance undergoes mapping via an embedding function 2g. The embedding is then incorporated into a conditional diffusion model for noise prediction. The parameters of the noise model Eg and the embedding function 24 are concurrently optimized using the reconstruction loss. Additionally, the instance is subjected to mapping through random linear projection into an alternate metric space. Two distinct Random projections are generated for numerical and categorical features, respectively. We align the distances in the embedding space and the random projection space using the RDM loss to ensure that the embedding function effectively preserves distance information, which is beneficial for downstream classification tasks.
training process. In the classification stage, we construct a novel instance-wise iterative classifier to predict the testing samples the learned embedding space (Section 4.3). Finally, due to the absence of a validation set for above unsupervised embedding space learning, we propose to use pseudo-label validation the unlabeled dataset select hyperparameters (Section 4.4). A summary of the training algorithm of D2R2 is presented in Appendix A.
4.1 caatusion-baseed representation Learning
Diffusion-bar based generative models [42, 17, 44, 45] are latent variable models that use Markovian noising and parameterized denoising process model the data distribution thus generate realistic samples. Details of diffusion models are provided in the appendix B. The diffusion process for given sample is defined 3, is a pre-defined variance schedule, and the timestep t is known. The true noise is <(x,, t. x,) = 1/vI- (x, Vāxx)). Following DDPMs [17], the denoising model can be implemented by neural networks with learnable parameters & by directly predicting the noise e:
Varxo vI a,,e, dze, where
(0, I), âe II 3%),
CUDPM (b) E1,xo [xc,(x,,)) t) cl2
(1)
Diffusion models can be extended to conditional generative models ex(xe, () by inserting the conditional information to generate specific class samples [45, 1].
We now present the diffusion-baan based representation learning part of D2R2. We employ the diffusion models in distinct way different from generation. We modify the diffusion model and repurposing act as representation extractor. Specifically, we represent the trainable embedding function as Rd + R", where 24 (X)) maps the noiseless input Xy to the latent space of dimension Formally, the reconstruction loss for the diffusion- based representation learning is
Lrecon (6. 0) Ex,o, (x,, (x,, =0 (80)) - -el/2-
where Eg is a trainable noise model, and &o is an augmentation version the noiseless data input Xo- In our method, we use Gaussian noise perturbation X, Xa + o.N(0, 1). For numerical features set o; to be the standard deviation numerical features in D.. and for categorical (one-hot) features, use fixed d, smooth the one-hot indicator values.
Reasons that the designed diffusion model can extract semantic knowledge come from twofold. Firstly, the diffusion model with powerful expressiveness encodes the information needed for denoising. Specifically, in conditional diffusion models, the noise reconstruction loss Ex,xo,, b...x./eo@ (xe, (,,, - trains the noise prediction function (x,, to predict the true noise E(X,, Xo) given the noisy

---

sample Xt, the knowing t and the condition information If = Xo, we could expect ea can almost perfectly recover e. By replacing the conditional information e by a function 2g (x²)) that maps to an embedding space with lower dimension than Xo, we introduce an information bottleneck to the noise reconstruction process. This forces 2p to extract effective information for denoising from xo, leading to representation learning through the noise reconstruction loss (2).
From another perspective, it is pointed out the noise reconstruction loss (2) can be expressed cquivalently as the weighted noisy score matching loss xe. [A(t]]|s> - Vx, log Po(() xx) [1], where the weights X(t) = of are determined by the noise scale (1) < a < < (T). The choice of noise scale a (t) controls the granularity of the embedding function. We focus on larger timesteps thus extract the low-frequent wwwfrreqeecyy semantic information rather than details.
4.2 Random distance matching
The acquired latent representation 26 from equation (2) is utilized for subsequent classification tasks, which heavily rely on distance information between the clusters within the embedding space. Although 2g (xo) learned through conditional diffusion models already exhibits clustering properties [1] to a certain extent, the learned embeddings do not prioritize capturing distance information between class structures, potentially hindering the effectiveness for downstream classification tasks. Taking inspiration that representations can be learned by training neural networks to predict distances in randomly projected space [49], we propose the random distance matching (RDM) loss to align pairwise distances between the embedding space and randomly projected space R". Here we suppose that if two samples belong to the same class in the ground truth, their embeddings should be close to each other, which is reflected by the randomly projected features from any perspectives.
Specifically, we consider the random linear projections W Rrxd with each element Way sampled i.t.d. from a fixed distribution. The RDM loss is as follows:
(3) where Xo, X are any two unlabeled data points sampled from Du and d is a metric. Since the dimensions and the scales of the two matching spaces might not be the same, choose d to be the cosine distance.
Crdm (0) Ex [wlld((z)
2e (x66) (x))))))))
d( Xo. Wx
Furthermore, in order to handle hybrid tabular data types, which involve both numerical and cate- gorical features, we sample random projections from various distributions according to the specific feature types. We consider samples with dmum numerical features and dea categorical features. For numerical features, we sample projection W m E I x from symmetric uniform distribution: wij ri.idd Unif(-A, A); for categorical features, we sample projection Wan. E grex from Bernoulli distribution: Wi,j r..id.. Bernoulli(p). Considering both types features, our random linear projection is defined as:
Wxo = concat Wnnm ( [x0]mm Weat Wear|Xo]aan [Xo]aat),
where [xo] mum [xo]ant are the numerical and categorical parts of xo, respectively.
Overall, we define a novel diffusion-based representation learning loss as follows:
(5) where a is hyperparameter to balance the noise prediction loss and RDM loss. We train & and @ to minimize LD222 over the unlabeled dataset D,,- The trained embedding function 2p (xp)) is used as the representation function for the downstream classification tasks. Loss function LD222 considers learning of embeddings from two different angles. Diffusion models possess strong generative capabilities, compelling the reconstruction loss (cb 0) ensure that the learned embedding ze captures high- high-quality semantic information, which significantly influences the data distribution. The random distance loss Lydm(t) adjusts the embedding to accommodate the distances between data points in random projection space, thereby improving the clustering capability of the embedding space, rendering it suitable for subsequent classification tasks.
LDzR2(0, 0) = Lreson (Ø, 0) + a Leam(()
4.3 Instance-wise iterative prototype
Given the trained embedding function =x(x) and the labeled support set S = ) we define the ristance-wise prototype of support samples the embedding space as 6; = =n (x²) where
i = 1,,,.., Ng. We can predict the label of a query sample to be the same as the nearest prototype.

---

(e) 2-shot clastittatioon
[b) Center prototype
(c) letee prototype
Figure 2: An illustration of the rationale behind instance- wise prototype. (a): In 2-shot scenarios, shapes represent the ground-truth classes in the embedding space, while the gray-colored objects await classification. (b): If the embedding of the circle class is not unimodal, averaging the prototypes leads erroneous center suggestions and fails classify the embeddings. (c): Considering instance-wise prototypes, each prototype contributes to the classification of nearby embeddings without generating erroneous centers.
The rationale for employing instance-wiss prototypes instead of center-specific prototypes [41, 11], which averages the K-shot embeddings inside the j-th class as Cj = K Σy 20 (x{) is that embeddings from the same class may exhibit multimodality, while center-specifii prototypes can only represent a uni-modal pattern, which might result in incorrect classification results (see Figure 2 for an illustration). On the other hand, in the few-shot learning scenarios, the scarcity of support samples brings large variance of prototypes, leading to unstable and imprecise classification outcomes. Drawing inspiration from the soft k-means algorithm [38], we leverage the weighted average of the query embeddings and support embeddings to create robust prototypes iteratively, referred to as the instance- wise iterative prototypes.
(0) At the beginning, we initialize each prototype as the embedding of each support instance as c = 20 (x),, where = 1, 2.... N,. At the 1-th iteration step, the probability that the query x E Q belonging to the i-th prototype is:
Pe (2) (1)=;)) = exp(-1lzo(x) -; 1l3/T)
(6)
Ex,es exp( (spp--l/z (x) C3 112/T)
Then we update the prototypes at the (1 + 1)-th iteration based on the weighted average of query embeddings:
(1) + + Pe (ixx zx
(+1)
(1) 1 1+ Ex P/ (i S)
After the L-th iteration, we predict the class of a query sample x to be the same class the nearest prototype:
i arg max Pa (ilx;
(L)
(8)
4.4 Pseudo-label validation
One challenge in the proposed unsupervised learning lies in the lack of labeld validation set for hyper-parameter tuning. For example, in 1-shot classification, the only one labeled sample is used for training, with no additional labeled samples available for validation. We tackle this problem by generating pseudo labels for the unlabeled dataset using soft k-means of raw features.
Formally, we sample a validation set Drai C Du. During each validation process, we randomly sample K' points {xx
k= D which regarded as pseudo support samples, forming K' distinct classes, i.e., Sau = [(x] yf = k)] We regard the remaining unlabeled samples Qval D Sya as pseudo query samples. Next. pseudo-labels are generated for Qval by applying instance-wise iterative prototype classification (Section 4.3) to the raw feature space, and those pseudo-labels serve as the "ground-truth" labels for the query set Quai- Subsequently, we employ D2R2 on the validation set (Svat, } to predict the "ground-truth" labels of Q We assess the validation performance to select hyperparameters for the training iteration.
7

---

5 Experiments
5.1 Experimental settings
For all the datasets, we randomly split 80% of the data for training and the remaining 20% for testing. As for N-way K'shot scenario, the support set is constructed by randomly selecting N x K samples, with K samples from each of the N classes from the training sets. Additionally. 20% of the training data is utilized for validation and hyperparameter tuning. We use one-hot encoding for categorical features and min-max scaling for numerical features, except for income data, for which we use standardized scaling (see Appendix D.2 for details). All experiment settings of baselines and D2R2 are the same as STUNT [28] for fair comparison. Details are provided in Appendix D.3?.
Datasets. We select nine datasets from the OpenML-CC18 benchmark [3. 6] to validate the per- formance of D2R2. Table shows a summary of the datasets. Optdigits, mfeat-karhunen, diabetes contain and breast only numerical features; dna and mfeat-pixel consist of only categorical features; income, emc and nomao contain both numerical and categorical features. summary of the dataset information is provided in Appendix Table D.I.
Baselines. In order assess the efficacy of our D2R2 framework, we compare the performance of D2R2 with four types of baselines, whose details are provided Appendix E.
i. Supervised learning methods. We compare with established supervised learning methods such as CatBoost [35], k-nearest neighbors (kNN) classifier [32] according to the prototype, and TabPFN [18], transformer-based network designed to make predictions on small tabular dataset.
ii. Semi-supervised learning methods. Such methods include Mean Teacher(MT) [46], Interpolation Consistency Training (ICT) [48], Pseudo-Label [25] and Meta Pseudo Labels (MPL) [33].
ii Few-shot meta-learning methods. We consider recent state-of-the-an meta-learning approaches such as CACTUs [19]. UMTRA [22], SES [54] and STUNT [28]. STUNT is designed for tabular data while others are designed for image data, whose network structures are modified for tabular data[28].
iv. Self-supervised learning methods. We compare with the state-of-the-ar self-supervised methods for tabular data, including VIME[55], SubTab [47], SCARF [4), TabTransformer [20]. We utilize representations acquired from those models to conduct Center Prototype Classification [11].
5.2 Overall evaluation results
We compare our framework D2R2 with other state-of-the-art supervised, semi-supervised, self- supervised and meta-learning methods. The classification accuracy of all methods presented in Table 1. In particular,
carry out experiments under two few-shot settings: NNway 1-shot and N-way 5-shot. We report the mean accuracy across 100 random seeds.
We note that our D2R2 framework significantly outperforms baseline methods across diverse datasets. Wilcoxon signed-ranks test is employed (Appendix D.5) further demonstrate the statistical significance of comparison. When comparing the accuracy of the D2R2 with that of the best baseline on all datasets, Wilcoxon's 's P-value is below 0.05, significantly indicating the effectiveness of D2R2.
In the case of the diabetes, optdigit and karkunen datasets, where all features are numerical, STUNT method outperforms other baselines, while D2R2 demonstrates substantial improvement. In the high-dimensional dataset optdigit, D2R2 improves accuracy by 39% compared to the supervised method Catboost, 23% compared to the semi-supervised method Mean Teacher, 5% compared the meta-lcarning method STUNT, and 22% compared to other self-supervised methods like SubTab. On the other hand, for pixel and dna datasets, which only contain categorical features, meta-learning such as UMTRA and SES are almost ineffective, while D2R2 still demonstrates superior accuracy, outperforming all other methods in both one-shot and five-shot settings. Similarly, for the large size dataset income, encompassing both numerical and categorical features, our framework consistently achieves a high classification accuracy of 72.0%%, surpassing the leading baseline STUNT by 19%. In summary, it can be concluded that D2R2 demonstrates robust capability to address few-shot
Code available https: https:///githutl Cww.ubb..oo//aooleeo http://aroo-coooddee pow.oudddpjject/DDR...

---

Table 1: Reported test accuracy is the mean value across 100 random seeds. Asterisked (+) baselines
refer to the reported scores 2TUNT(28). Bold number and the underlined number denote highest score and the second best score, respectively. Empty data is either because the dataset exceeds the input data dimension limits of TabPFN there are no reported scores in STUNT [28].
Method
eme
diabetes
dna
income
karkunen
optdigits
pixel
nomao
brest
#shot=1
CatBoost
36.03
56.74
39.15
57.55
53.24
58.30
54.74
63.62
69.71
TabPFN
35.37
53.35
46.02
55.74
KNN
35.39
58.50
42.20
51.45
54.61
65.60
60.79
63.51
71.87
Mcan Teacher(*)
35.58
58.05
46.58
60.63
54.57
66.10
61.02
64.23
71.92
ICT(*)
36.53
58.08
46.55
61.83
58.37
69.12
60.88
Pscudo-Label(")
34.97
57.03
44.26
60.52
49.44
61.50
56.12
62.39
69.92
MPL(*)
35.13
57.39
44.22
60.85
47.66
61.52
56.01
64.28
71.33
SubTab
36.23
58.22
46.98
62.45
50.22
62.01
60.34
67.63
72.94
VIME
35,90
58.99
51.23
61.82
59.81
69.26
63.28
64.75
70.11
SCARF
35.39
55.64
57.86
57.94
60.96
63.93
68.90
75.32
RTDL
34.34
58.15
47.99
53.61
58.25
62.78
62.87
68.33
76.38
UMTRA(*)
35,46
57.64
25.13
57.23
49.05
49.87
34.26
SES(")
34.59
59.97
39.56
56.39
49.19
56.30
49.19
69.52
74.89
CACTUS(*)
36.10
58.92
65.93
64.02
65.59
71.98
67.61
71.49
75.24
STUNT(")
37.10
61.08
66.20
63.52
71.20
76.94
79.05
71.54
76.92
D2R2
42.88
63.94
68.00
75.82
72.08
81.13
81.34
79.47
77.69
#shot=5
CatBoost
39.89
64.51
60.20
67.99
77.94
83.07
83.38
75.32
77.06
TabPFN
38.31
64.06
76.59
81.68
KNN
37.65
65.61
61.16
62.19
80.08
84.16
84.75
73.78
79.43
Mcan Teacher(*)
37.73
65.45
61.47
67.05
81.08
86.66
85.24
74.78
81.26
ICT(*)
38.09
65.47
63.37
70.13
84.58
87.01
86.12
Pseudo-Label(**
37.49
64.46
60.06
66.26
78.60
83.71
82.94
72.87
78.91
MPL(*)
37.47
64.51
59.65
67.61
77.85
83.70
82.39
73.20
79.54
SubTab
39.81
68.26
62.49
72.14
70.88
83.27
80.41
76.15
82.74
VIME
39.83
67.64
71.29
72.19
19.42
83.21
85.24
74.96
85.81
SCARF
37.75
68.66
62.75
66.09
69.96
85.67
81.32
77.65
84.42
RTDL
37.59
64.27
45.49
64.92
60.43
82.58
76.13
73.61
79.66
UMTRA(*)
38.05
64.41
25.08
65.78
67.28
73.29
$1.32
SES(")
39.04
66.61
52.25
68.27
74.80
78.46
74.80
76.50
84.73
CACTUS(*)
38.81
66.79
81.52
72.03
82.20
85.25
78.33
86.90
STUNT(")
40.40
69.88
79.18
72.69
85.45
88.42
89.08
81.49
86.82
D2R2
43.39
73.52
82.38
76.02
84.96
90.73
91.06
82.69
88.27
Table 2: Ablation study of three components: diffusion representation (DR), random distance matching (RDM) and instance- -wie iterative prototype (IP). We also conduct the center-specific prototype version of D2R2 (D2R2-c). We report the test accuracy (%) under 100 random seeds.
Dataset
RDM
DR
DR+RDM
RDM+IP
DR+IP D2R2-c
D2R2
opt (I-shot)
49.05
72.38
77.41
26.66
76.87
81.13
dna (1-shot)
45.43
57.14
61.29
26.03
56.17
68.00
eme (1-shot)
35.50
35.19
38.69
34.90
34.48
42.88
opt (5-shot)
70.26
88.64
89.61
51.31
89.32
87.12
90.73
dna (5-shot)
47.72
71.84
73.03
32.16
79.24
81.39
82.38
eme (5-shot)
36.04
35.62
40.81
36.27
35.74
43.39
43.39

---

tabular data classification tasks, regardless of the number of shots, dataset size, feature dimension, or the proportion of categorical features.
We speculate that the superior performance of D222 is attributed to the effective learned representation of inputs along with the instance-specifie prototypes. Upon visual examination of D2R2's embeddings in Figure 3, we observe clustering characteristics from point clouds, which demonstrating the embeddings" effectiveness for classification. Besides, note the presence of multimodality the embeddings, supporting the rationale for introducing the instance-wise prototypes.
5.3 Ablation study
We conduct ablation study demonstrate the efficacy of three components in D2R2, namely the diffusion representation (DR), random distance matching (RDM), and the instance-wise iterative prototype (IP). We compare the complete D2R2 design (DR+RDM+IP) with frameworks that exclude one or two of these components. We also conduct the center-specific prototype version of D2R2 (D2R2-c), which utilizes average embeddings of K-shot samples prototype. The results of the ablation study are presented in the Table 2.
Based on the results, D2R2 surpasses all other variants, indicating that all three components col- lectively contribute to its superior performance. Specifically, removing the diffusion representation results in significant degradation in performance, highlighting the crucial contribution of diffusion representation in capturing semantic knowledge. We also observe that DR+RDM shows the second best performance, highlighting the importance of incorporating random distance matching in the diffusion training process. Comparing with a center-specific prototype version of D2R2 (D2R2-c), observe that the instance- wise prototypes leads to improvement results in optdigit and dna.
Optdigits
Karkunen
Income
Figure The t-SNE visualizations depict the D2R2 representations, with point clouds illustrating the embeddings of 1000 randomly selected samples, color-coded based on their respective class labels. We observe that the embeddings exhibit multimodal patterns. For instance, in the dna dataset, the red class is distributed in both the bottom right corner and the top left corner.
6 Conclusion
In this paper, we introduce D2R2, an innovative framework to address few-shot tabular challenges. The core idea of D2R2 is utilize the strong expressive abilities of diffusion model, along with random distance matching to construct representation learners. This method captures the semantic knowledge of unlabeled data and generate effective embeddings for downstream classification tasks, meanwhile adapting to the mixture feature types of the tabular data. Additionally, to accommodate multimodalities of embeddings, we devise the instance- wise iterative prototype classifier using labeled data. Furthermore, a pseudo-label validation scheme is designed for hyper-paramen parameter selection. The superior performance of D2R2 presented on diverse datasets, demonstrating the effectiveness of D2R2 over other baselines. We hope that our work will inspire new avenues for research in the field of diffusion model representation learning on tabular datasets.
Acknowledgements
We would like thank AC and reviewers for their valuable comments the manuscript. Bingyi Jing's s research is partly supported by NSFC 12371290.
10

---

References
[1] Korbinian Abstreiter, Sarthak Mittal, Stefan Bauer, Bernhard Schölkopf. and Arash Mchrjou. Diffuusin--bie representation learning. arXiv preprint arXiv:2105.14257, 2021.
[2 Sercan Ô Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceed- ings of the AAAI conference on artificial intelligence, volume 35, pages 6679-6687, 2021.
[3] Arthur Asuncion and David Newman. Uci machine learning repository. 2007.
[4] Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. arXiv preprint arXiv:2106.152 15147, 2021.
[5] Sungyong Baik, Janghoon Choi, Heewon Kim, Dohee Cho, Jaesik Min, and Kyoung Mu Lee. Meta-learning with task-adaptive loss function for few-shot learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9465-9474, 2021.
[6] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael Gomes Mantovani, Jan N van Rijn, and Joaquin Vanschoren. Openml benchmark- ing suites. In Proceedings of the NeurIPS 2021 Datasets and Benchmarks Track. 2021.
[7] Longbing Cao. Ai finance: challenges, techniques, and opportunities. ACM Computing Surveys (CSUR), 55(3):1-38, 2022.
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[9] Wei-Yu Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer 2019.
look at few-shot classification. arXiv preprint
[10] Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning. arXiv preprint arXiv:2401.14404 2024.
[11] Yudong Chen, Chaoyu Guan, Zhikun Wei, Xin Wang, and Wenwu Zhu. Metadelta: A meta- learning system for few-shot image classification. In AAAI Workshop on Meta- Leta-Learning and MetaDL Challenge. pages 17-28. PMLR, 2021.
[12] Zitian Chen, Subhransu Maji, and Erik Learned- Shot in the dark: Few-shot learning with no base-class labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2668-2677, 2021.
[13] Hao Cheng, Jocy Tianyi Zhou, Wee Peng Tay, and Bihan Wen. Graph neural networks with triple attention for few-shot learning. IEEE Transactions Multimedia, 2023,
[14] Hongchao Fang. Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. Cert: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766 2020.
[15] Chelsea Finn, Pieter Abbeel, and Sergey Levine. agnosttg meta-learning for fast adap- tation of deep networks. In International conference machine learning, pages PMLR, 2017.
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-668 68510,6681111 2020.
[18] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: transformer that solves small tabular classification problems in second. arXiv preprint arXiv:2207.0/888 2022.
[19] Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning arXiv preprint arXiv:1810.02334 2018.

---

[20] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.0667,, 2020.
[21] Dahyun Kang. Heeseung Kwon, Juhong Min, and Minsu Cho. Relational embedding for few-shot classification. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8822-8833, 2021.
[22] Siavash Khodadadch, Ladislau Boloni, and Mubarak Shah. Unsupervised meta- eeta-learning for few-shot image classification. Advances in neural information processing systems, 32, 2019.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6988, 2014.
[24] Dong Bok Lee. Meta-gmvae: Mixture of gaussian vaes for unsupervised meta-learning. 2021.
[25] Dong-Hyun Lee et Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896. Atlanta, 2013.
[26] Junjie Li, Zilei Wang, and Xiaoming Hu. Learning intact features by erasing-inpainting for few- shot classification. Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 8401-8409, 2021.
[27] Mario Molina and Filiz Garip. Machine learning for sociology. Annual Review of Sociology, 45:27-45, 2019.
[28] Jachyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, and Jinwoo Shin. Stunt: Few-shot tab- ular learning with self-generated tasks from unlabeled tables. arXiv preprint arXiv:2303.009/8 2023.
[29] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint 18X3. 0003,0299, 2018
[30] Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Young Yun. Boil: Towards representation change for few-shot learning. arXiv preprint arXiv:2008. 08882, 2020.
[31] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems, 34:1 34:11054-11000 11070, 2021.
[32] Leif Peterson. K-nearest neighbor. Scholarpedia, 4(2):1883, 2009.
[33] Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 11557-11568, 2021.
[34] Mihir Prabhudesai, Shamit Lal, Darshan Patil, Hsiao- Yu Tung. Adam W Harley, and Katerina Fragkiadaki. Disentangling 3d prototypical networks for few-shot concept learning. arXiv preprint arXiv:2011.03367, 2020.
[35] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances neural information processing systems, 31, 2018.
[36] Guo-Jun Qi and Jiebo Luo. Small data challenges in big data era: A survey of recent progress on unsupervised and semi- semi-supervised methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(4):216-2215 2020.
[37] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18))9, 2019.
[38] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenen- baum, Hugo Larochelle, and Richard $ Zemel. Meta-learning for sumi-supervised few-shot classification, arXiv preprint arXiv.1803.00677 2018.
12

---

[39] Eli Schwartz, Leonid Karlinsky, Rogerio Feris, Raja Giryes, and Alex Bronstein. Baby steps towards few-shot learning with multiple semantics. Pattern Recognition Letters, 160:142-147, 2022.
[40] K Shailaja, Banoth Seetharamulu, and MA Jabbar. Machine learning in healthcare: A review. In 2018 Second international conference on electronics, communication and aerospace technology (ICECA), pages 910-914. IEEE, 2018.
[41] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[42] Jascha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265 PMLR, 2015.
[43] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances neural information processing systems, 33:596-608 2020.
[44] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[45] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1. 2020.
[46] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averagen con- sistency targets improve supervupereieed deep learning results. Advances in neural information processing systems, 30, 2017.
[47] Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for self-supervised representation learning. Advances in Neural Information Processing Systems, 34:18853-1111 188655 2021.
[48] Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Arno Solin, Yoshua Bengio, and David Laze Interpolation consistency training for semi-supervised learning. Neural Networks, 145:90- 106, 2022.
[49] Hu Wang, Guansong Pang, Chunhua Shen, and Congbo Ma. Unsupervised representation learning by predicting random distances. arXiv preprint arXiv.1912.12186. 2019.
[50] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
[51] Jiangtao Xie, Fei Long, Jiaming Lv, Qilong Wang, and Peihua Li. Joint distribution matters: Deep brownian distance covariance for few-shot classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7972-7981, 2022.
[52] Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro 0 Pinheiro. Adaptive cross-modal few-shot learning. Advances in Neural Information Processing Systems, 32, 2019.
[53] Xingyi Yang and Xinchao Wang. Diffusion model as representation leamnr. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18938-18949, 2023.
[54] Han-Jia Ye, Lu Han, and De-Chuan Zhan. Revisiting unsupervised meta-learning via the char- acteristics of few-shot tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4333)3372--3777, 2022.
[55] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33: 13:11003-1104 2020.
[56] Baoquan Zhang. Xutao Li, Yunming Ye, Zhichao Huang, and Lisai Zhang. Prototype completion with primitive knowledge for few-shot learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3754-3762, 2021.
13

---

A
Algorithm
Algorithm Training D2R2
Require:
Unlabeled dataset Du, = {x:]i= perturbation scale a, batch size B, diffusion horizon T. noise schedule {Be) {ar} = (1- 3e), { [[I Orp) number of numerical features dnum. number of categorical features deat. feature dimension d daum + deat. learning rates Y/, 72. hyperparameters: a, A.
Initialize: noisy network <g. embedding network 20
while not done do
Sample two batchs of unlabeled data x' ~ Du. Sample a perturbation copy X ~~ I + o.(00, 1). 1,2,... ..... 7 and noise (O, 1).
Sample timestep
Compute the noisy sample X2 = Va,x vI aec.
Lrecm (0, 0) ||< =0())) cB..
Sample matrix W,, E Raxm with (Wa)is wind Unif(- A, A).
Sample matrix W. Rdxrz with (W.)y Midd Bernoulli( (p)
Compute Wx x = Concat( W [x] mum: We[ (x)c))
Compute Wx Wx' Concat (W,, [x' Ium: Wa (V'xx ccm):
Lidm ()) = cosine (x), 2x()))
cosine Wx, Wx))l
VoLrocon (b, 0).
- 0 B WaCrooon (1b, 0) + a (0)).
end while
return Embedding function 2g.
Algorithm 2 Pseudo-label Validation
Require: Unlabeled validation set D (xi);; number of pseudo-classes K", number of iteration /
for i = 1,. do
Sample K" points {x E Dall-
Form support set S = {xx, K"
Form query set Q D {x;}
Create instance-wise iterative prototype classifier f(-),S, Q) using the raw feature vectors. Create pseudo-label in {(x,, (5))) E Q).
end for
return Pseudo support set S, pseudo query Q
B Diffusion Models
Diffusion-basee generative models [42, 17, 44, 45] are latent variable models that use a Markovian noising and parameterized denoising process model the data distribution. The forward noising process of horizon T follows pre-defined variance schedule $I...., 3r that encodes data distribution Xa ~ p(z):
(xx xxx))xx-1==> = - 1,
and assume the prior q(x:) ~ (O. The reverse denoising process is modeled as px(Xo))) := (x;; 0, I) IT (x (x.) with learnable parameters ø. Following DDPMs [17], the denoising model be implemented by neural networks directly predicting the noise
LDDPM (0) -E,,xo |/ (xx,1) 1) E
(10)
where Xo,x0 + vI - aye, No I) and o; straightforwardly extended to conditional generative models (x,,, I by inserting the conditional information e [45, 1].
T B)). Diffusion models can be
14

---

C Random Linear Projection
We generate random matrices from different distributions to cater for the numerical features and categorical features separately. In order to integrate the projections derived from numerical and categorical spaces, it necessary to select (for Uniform(-24 A, A)) and p (for Bernoulli(p)) in consistent manner to equitably consider the information from heterogeneous type. In this paper, we adjust the random weights according to consistent activation variance, that is we set /3p(1 - p) and treat p as a hyperparameter, and re-weight the matrices according to the number of columns, i.c., Wnm f Wnum/ dnum: Went f Wear/ siu//vdes deat. Such choice of parameters preserve the variance of the row-sum of projection matrices Additionally, we sample Wmmm RR x dan and Wan E Rdax get rid of the need to treat the study of different random matrices to the future work.
in Algorithm (1) as hyperparameters. We leave
D Experiment details
D.1 Dataset Information
Table 3: Datasets information.
The table shows the instances number, the number of features (numerical features, categorical features) and the number of classes of each dataset.
Dataset
# Instances
# Features (num.. cate.)
# Classes
optdigit
5620
64 (64,0)
10
karkunen
2000
64 (64.0)
10
diabetes
768
8 (8,0)
pixel
2000
240 (0,240)
dna
3186
180 (0,180)
eme
1473
9 (2,7)
income
48842
14 (6,8)
nomao
34465
118 (90,28)
brest
286
24481 24481(24481, (24481,0)
D.2 Data normalization
For income dataset, we apply standardized scaling each numerical feature zy;
24 ((++)
32
(11)
o (x;)
And employ min-mat max scaling numerical features of other datasets:
min x4))
(12)
max (2:c)
min
D.3 Hyperparameter Details
We employ fully connected layers with identical layer counts and hidden dimensions model the diffusion noise and the embedding function. The training procedure for the diffusion model follows same approach as DDPM [17]. The configuration for the diffusion model and neural networks (NN) are as following:
In experiments, search three hyperparameters during the validation process.
All other experiment settings of D2R2 and other baseline are the same as STUNT [28] for fair comparison.
D.4 Choosing Noise Levels
The timestep = 1, 1...., indicates noise levels in diffusion models. our experimental analysis, we have noted a trend where performance sees an improvement with increase in t during the initial
15

---

Table 4: Model configurations.
Parameter
Setting
Description
T
10
Diffusion timesteps.
3
vp [45]
Noise schedule.
Hidden dims
512
Dimension of dense layers.
Layers
3
Number of dense layers.
Optimizer
Adam [23]
Batch Size
256
Learning Rate
0.0003
Table 5: Hyperparameter search range
Parameter
Range
Description
OK
linespace(O.1] 0.1, 5)
RDM weight.
d₂
(5, 10, 20, 40, 80)
Embedding dimension
linespace(O.1, 0.1, 2)
Temperature in equ. (6)
stages. However, this trend of performance enhancement plateaus after the first few timesteps, which tends stabilize with minimal fluctuations, shown in Table As / approaches T, noise levels rise, thus effective denoising requires that all information about op be thoroughly encoded, which contains the most information about the data class. We focus larger timesteps thus extract the low-frequency semantic information rather than details. Thus in our experiments, we selected the last step for all datasets.
Table
Performance Across Different Timesteps
Dataset optdigits optdigits
Shot
First Step
Middle Step
80th percentile Step
Last Step
Average of All Steps
72.39
79.21
80.93
81.13
80.66
84.57
88.67
91.73
90.73
89.66
cme
1
37.19
42.26
42.18
42.88
42.74
emc
36.28
42.73
42.89
43.39
42.37
D.5
Wilcoxon Test
The Wilcoxon signed-ranks test a statistical method used to compare the means of two related groups when assumptions normal distribution involves ranking the absolute differences between paired observations and calculating a test statistic for assessing significance. The procedure includes the following:
met. Particularly useful for paired data,
Calculate the differences: D; X; Y,, where X; and Y, paired observations.
ii Rank [Dj] from the smallest the largest, resolving ties by averaging ranks.
iii. Calculate signed ranks: Assign positive ranks to positive differences and negative ranks to negative differences.
iv. Calculate the test statistic W, which is the sum of the positive or negative ranks, depending on which sum is smaller.
V. Compare W critical values: Determine significance by comparing W to critical values from the Wilcoxon signed-ranks table or using statistical software.
In summary, the test helps identify a significant difference exists between the groups' means, even when data doesn't 't meet assumptions for parametric tests.
D.6 Model Complexity
The model complexity is manageable through our framework's utilization of two neural networks: the embedding network and the noise prediction network, both which are demonstrated as 3-layer MLPs in paper. These settings (hidden dimensions, embedding dimensions and model structure)
16

---

can be adjusted to meet different needs. In terms of computational efficiency, The main time- consuming factor after the embedding model the calculation of the instance-wise iterative prototype. For an N-way K-shot problem with L. iterations and a query set of size Q, the computation complexity O(NKTQ) In few-shot learning, where N, K, are small, the computation complexity is linear in the size of the query set.
E Baseline Details
we compare with supervised learning methods such as CatBoost [35], k-nearest neighbors (kNN) [32], which denotes the nearest neighbor classifier according to the prototype of the input data, and TabPEN [18], which a transformer that that solves small tabular classification problems.
Mean Teacher (MT) [46] is semi-supervised learning framework designed to enhance model generalization by establishing a robust interaction between labeled and unlabeled samples. This framework draws inspiration from the "teacher-student training approach, where the "teacher" refers to a model averaging multiple student model predictions, while the "student" is the model being trained.
Interpolation Consistency Training (ICT) [48] uses MT framework. While ICT aims to improve the model performance by augmenting additional data points through interpolation, with the expectation that the model's predicted values align consistently with the interpolated labels.
Meta Pseudo Labels [33] continually adjusts the teacher model based on feedback from the student model's performance on labeled datasets. Both teacher and student models trained concurrently, with the teacher model learning to generate improved pseudo-labels while the student model learns from these pseudo-labels.
VIME[55] is a form of self-super learning that derives valuable representations by intentionally corrupting random features and subsequently predicting the corrupted location. We utilize VIME representations for conducting k-nearest neighbor classification.
SubTab [47] employs three effective pretext task losses, namely, reconstruction loss, contrastive loss, and distance loss, as part of its self-supervised learning methodology. We utilize SubTab representations for conducting k-nearest neighbor classification.
SCARF [4] takes SimCLR-like [8] contrastive loss between the sample and its corrupted version. We utilize SubTab representations for conducting k-nearest neighbor classification.
TabTransformer [20] is built upon self-attention based Transformers. The Transformer layers trans- form the embeddings of categorical features into robust contextual embeddings to achieve high prediction accuracy in tabular context. Experiments shows that Tab Tabrransformer performs better than TabNet [2]. We utilize TabTransformer representations for conducting k-nearest neighbor classification.
CACTUS [19] creates episodic tasks by dividing the features extracted from an earlier-trained unsupervised feature embedding network using various objective functions, and then trains the few-shot learner on these tasks.
UMTRA [22] employs a domain-speciffs data augmentation approach to create synthetic tasks for the meta-learning stage. In doing so, the episodic tasks that are formed are constrained by the data augmentation strategy.
SES [54] considers the augmented sample part of the same pseudo-class and implements robust strategies for unsupervised meta-learning, which include ample episodic sampling, challenging mixed supports, and task-specific projection heads.
F Limitations
Our approach is tailored to the specific characteristics of tabular data, including data scarcity, multiple modalities, mixed feature types, and sensitivity to changes in column order. Thus it is only effective in scenarios involving tabular data, and may not work well for CV and NLP tasks.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: By clearly stating the main claims in the abstract and introduction, the reader is able to understand the purpose and significance of the paper, as well the specific contributions and scope of the research.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
By openly discussing these limitations, we are aware of the potential short-
comings.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [NA]
Justification: Due to the complex architecture of diffusion models, it is typically hard to analyze the theoretical behaviors, such convergence rates, of proposed model and other diffusion-based models. Thus, we leave it for future work.
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
19

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: (CodeSummissionPoo
/ //niss. wwtpcccpppcc.aecccccccccccccetlt htt///uddss//ee// http:/ppww.ggcppe///////eeeeea/e
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
The paper specifies all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper reports statistical significance of the experiments by Wilcoxon signed-ranks test.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provide sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the Caubllc/Ethicscuideee
NeurlPS Code of Ethics attps://neurs:
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussing both potential positive and negative societal impacts of the research is important for providing a comprehensive evaluation.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The assets used in the paper have been explicitly mentioned and received proper acknowledgment from the owners.
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We propose new model in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: There is no potential risk incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "y2BaKTfCet", "reviewer": "Reviewer_wij1", "summary": "The paper proposed a few-shot learning framework for tabular data by designing a diffusion based semantic knowledge encoder and introducing a random distance matching mechanism to preserve distance information in the embeddings. During classification, an instance-wise iterative prototype scheme is utilized to improve performance by accommodating the multimodality of embeddings and increasing clustering robustness. Extensive experiments on multiple datasets show that  D2R2 achieves state-of-the-art performance compared to other schemes.", "questions": "1. How does the proposed method scale with larger datasets or those with higher feature dimensions? Are there any practical limitations or considerations for scaling?\n2. Could the authors provide more empirical evidence or analysis on the pseudo-label validation scheme, such as when the clustering might not accurately reflect the true class structure?", "limitations": "No potential negative societal impact is observed.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed approach that combines the diffusion model with random distance matching for tabular few-shot learning is relatively unexplored compared to classical few-shot learning tasks.\n2. The paper is well-written and organized, with detailed explanations of the challenges, methodology, and experimental results.", "weaknesses": "1. The feature dimensions of the benchmark datasets are relatively low, which leaves the the scalability to large datasets or those with extremely high dimensionality not fully explored.\n2. The pseudo-label validation scheme for hyperparameter selection relies on the assumption that the clustering of raw features can provide a reliable proxy for true labels, which might not always hold in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments!\n\n**Q1. Does the proposed method scale with larger datasets or those with higher feature dimensions?**\n\nIn our experiments, we utilized seven datasets that are widely used in tabular data research, as referenced in popular papers [28], [2], and [54] in our paper. Among these, the \"income\" dataset has the most samples (48842), and the \"pixel\" dataset has the highest feature dimension (240).\n\nTo further investigate the scalability of our method, we added two new datasets to our experiments. The \"nomao\" dataset has a larger size of 34465 samples, and the \"breast\" dataset has a significantly higher feature dimension of 24482. The following table shows their results. Our method consistently outperforms other baselines. These results will be included in the updated version of our paper, demonstrating that our method scales well to larger and high dimensional datasets.\n\n| Dataset | Shot   | CatBoost | KNN  | SubTab | VIME | SCARF | RTDL | STUNT | D2R2 |\n|---------|--------|----------|------|--------|------|-------|------|-------|------|\n| nomao   | 1-shot | 0.636    | 0.635 | 0.676  | 0.647 | 0.689 | 0.683 | 0.715 | 0.794 |\n|         | 5-shot | 0.753    | 0.737 | 0.761  | 0.749 | 0.776 | 0.736 | 0.814 | 0.826 |\n| brest   | 1-shot | 0.697    | 0.718 | 0.729  | 0.701 | 0.753 | 0.763 | 0.769 | 0.776 |\n|         | 5-shot | 0.770    | 0.794 | 0.827  | 0.858 | 0.844 | 0.796 | 0.868 | 0.882 |\n\n\n**Q2. Could the authors provide more empirical evidence or analysis on the pseudo-label validation scheme？**\n\n\nIn the few-shot learning scenario, particularly in 1-shot classification, there are no labeled samples available for validation, which leads most researchers to rely on fixed parameters. However, we argue that a better set of hyper-parameters can always be identified for a specific dataset. To this end, we generate pseudo-labels for a validation set using the fundamental principles of the soft k-means algorithm. Firstly, achieving complete consistency between the validation labels and the true labels is indeed challenging. Nonetheless, existing studies have demonstrated satisfactory performance despite minor discrepancies between the validation and test sets [1]. It is important to note that we have consciously avoided over-tuning the hyper-parameters on the validation set, limiting our adjustments to only three parameters, as detailed in the appendix. Secondly, we posit that raw features possess inherent clustering properties, and the soft k-means method has been proven effective and widely adopted in various contexts, as supported by references [3][4]. Our empirical evidence further substantiates this claim.\n\nIn our experiments, the accuracy of pseudo-labels was evaluated on several datasets, all showing relative high accuracy. This leads us to believe that pseudo-labels serve as a relative reliable proxy for true labels during validation. For the dataset \"income\" , where accuracy was comparatively lower compared to the classes number, we conducted experiments without using a validation set but with fixed hyper-parameters. The results indicated that selecting parameters using a pseudo-label validation set achieved better performance than using fixed hyper-parameters. The robustness of our method, demonstrated across multiple datasets, enables our approach to be applicable to a wide variety of tabular datasets. \n\n\nTable 1: Accuracy of pseudo-labels\n| Dataset   |DNA | Income | Karkunen | Optdigits | Nomao | Breast |\n|-----------|-----|--------|----------|-----------|-------|--------|\n| Accuracy  |0.37| 0.52   | 0.53     | 0.47      | 0.59  | 0.63   |\n\n\n\n\nTable 2: Fixed hyper-parameters VS Tuned hyper-parameters\n|       | $\\alpha$ | $d_z$ | $\\tau$ | Accuracy |\n|-------|----------|-------|--------|-------|\n| Fix   | 0.1      | 10    | 0.1    | 0.742 |\n| Fix   | 0.1      | 80    | 0.1    | 0.755 |\n| Fix   | 0.5      | 10    | 0.1    | 0.739 |\n| Fix   | 0.5      | 80    | 0.5    | 0.721 |\n| Fix   | 0.9      | 10    | 0.1    | 0.732 |\n| Fix   | 0.9      | 80    | 0.9    | 0.743 |\n| Tune  | 0.775    | 10    | 0.325  | 0.758 |\n\n\n[1]Peng X, Usman B, Kaushik N, et al. Visda: The visual domain adaptation challenge[J]. arXiv preprint arXiv:1710.06924, 2017.\n[3]Ferraro M B, Giordani P. Soft clustering[J]. Wiley Interdisciplinary Reviews: Computational Statistics, 2020, 12(1): e1480.\n[4]Singh V K, Tiwari N, Garg S. Document clustering using k-means, heuristic k-means and fuzzy c-means[C]//2011 International conference on computational intelligence and communication networks. IEEE, 2011: 297-301.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZQFsFFE9w0", "reviewer": "Reviewer_93Kt", "summary": "The paper introduces a novel method, Diffusion-based Representation and Random Distance Matching (D2R2), to address the challenge of few-shot learning on tabular data. This approach leverages the robust representational capacity of diffusion models to extract essential semantic knowledge from tabular data, thereby enhancing the performance of downstream classification tasks. Additionally, the paper presents the Random Distance Matching (RDM) loss function, which preserves distance information within embedding vectors during diffusion model training, further boosting classification performance. During the classification phase, the authors propose an instance-based iterative prototype scheme to accommodate the multimodal characteristics of the embedding vectors, thus improving clustering robustness. Extensive experiments conducted on various few-shot learning benchmarks for tabular data demonstrate that D2R2 surpasses current benchmark methods. In conclusion, this paper introduces an innovative few-shot learning method for tabular data, achieving significant advancements in extracting effective semantic representations and improving classification performance.", "questions": "Please see the Weakness section.", "limitations": "The author has already provided a description of the limitations, but it needs to be further elaborated.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-\tThe use of diffusion models to extract effective semantic representations from tabular data is a novel approach that overcomes the limitations of traditional supervised learning methods.\n-\tThe introduction of the Random Distance Matching (RDM) loss function, which preserves distance information within the embedding vectors during diffusion model training, further enhances classification performance.\n-\tExtensive experiments conducted on multiple few-shot learning benchmarks demonstrate that D2R2 outperforms the latest benchmark methods, validating the effectiveness of the proposed method.\n-\tThe paper provides comprehensive explanations and descriptions of the various components of the method, including the expressive power of the diffusion model, the role of the RDM loss function, and the principles of the iterative prototype scheme, making the methodology and process clear and understandable.", "weaknesses": "-\tThe paper lacks a clear description of the input format for tabular data. Specifically, it is unclear whether the tabular data is input in text form or as images. Clarification on this point is essential, as the input format may significantly impact performance.\n-\tThe paper utilizes a diffusion model to enhance the semantic representational capacity of tabular data. However, it remains uncertain whether the diffusion model is effective across all types of tabular data. Detailed demonstrations and explanations of how the model captures and enhances these semantic features are needed to substantiate its broad applicability.\n-\tThe authors introduce a diffusion model that differs from existing methods. It is crucial to provide additional information regarding the model's parameter count and FLOPs (floating point operations per second) to allow for a thorough comparison of its computational efficiency with that of other models. This would help in understanding the trade-offs between performance gains and computational costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments!\n\n**Q1. The paper lacks a clear description of the input format for tabular data.**\n\nIn our study, \"tabular data\" refers to the dataset organized in tables, which is a structured format that presents information in rows and columns. It is defined as $D = \\{\\boldsymbol{x} _ i\\} _ {i=1}^n\\subset \\mathcal{R}^{d}$ consisting of $n$ instances (rows) and $d$ dimensional features (columns). Each data instance $\\boldsymbol{x} _ i = (x_i^1, x_i^2, ... , x_i^d)$ may or may not hold strong relationship among features. The feature types of tabular data can be a mixtures of numerical numbers or categorical indicators. For example, the \"income\" dataset classifies individuals' incomes based on features like \"age\" (numerical)，\"education\" (categorical) and so on. Table 3 in Appendix provides a detailed information on each dataset.\n\nTabular data is widely used in practical applications, and few-shot learning is particularly important in the context of tabular data, as limited labeled tabular data is inherently common in many real-world applications, such as financial fraud detection [1], disease diagnosis [2], and social science [3]. Different from CV and NLP, tabular data contains a mixture of numerical and categorical features; tabular data lacks spatial and sequential relationships between columns, making it more challenging to extract semantic knowledge; tabular column order can be arbitrary permuted without affecting the tabular information; tabular embeddings within the same class may exhibit multiple modes, as demonstrated in Figure 3 of our manuscript. The significance and challenges of tabular few-shot learning motivate us to conduct this work.\n\n\n**Q2. Whether the diffusion model is effective across all types of tabular data?**\n\nFrom the experimental analysis, our datasets (seven in the paper and two additional datasets in the response to Review wij1's Q1) involve various types of tabular data. These include varying sample sizes (from 768 to 48842), feature dimensions (from 8 to 24482), feature types (numerical, categorical or the mixture of both), and number of classes (from 2 to 10). Our experimental results demonstrate the effectiveness of our proposed method and its scalability across various types of those tabular data.\n\nFrom a theoretical analysis perspective, reasons that the designed diffusion model can extract semantic knowledge come from twofold. Firstly, the diffusion model with powerful expressiveness encodes the information needed for denoising. Specifically, in conditional diffusion models, the noise reconstruction loss $\\mathbf{E} _ {t, \\mathbf{x} _ 0, \\mathbf{x} _ t} ||\\epsilon_\\phi(\\mathbf{x} _ t, t,c) - \\epsilon||^2_2$ trains the noise prediction function $\\epsilon_\\phi(\\mathbf{x} _ t, t,c)$\nto predict the true noise $\\epsilon(\\mathbf{x} _ t, t, \\mathbf{x} _ 0)$ given the noisy sample $\\mathbf{x} _ t$, the knowing $t$ and the condition information $c$. If $c = \\mathbf{x} _ 0$, we could expect $\\epsilon_\\phi$ can almost perfectly recover $\\epsilon$. By replacing the conditional information $c$ by a function $z_\\theta(\\mathbf{x} _ 0)$ that maps to a embedding space with lower dimension than $\\mathbf{x} _ 0$, we introduce an information bottleneck to the noise reconstruction process. This forces $z_\\theta$ to extract effective information for denoising from $\\mathbf{x} _ 0$, leading to representation learning through the noise reconstruction loss.\n\n**Q3. what is model's computational costs？**\n\nThe model complexity is manageable through our framework's utilization of two neural networks: the embedding network and the noise prediction network, both of which are demonstrated as 3-layer MLPs in our paper. These settings (hidden dimensions, embedding dimensions and model structure) can be adjusted to meet different needs. In terms of computational efficiency, The main time-consuming factor after the embedding model is the calculation of the instance-wise iterative prototype. For an N-way K-shot problem with L iterations and a query set of size Q, the computation complexity is O(NKLQ). In few-shot learning, where N, K, L are small, the computation complexity is linear in the size of the query set. In the updated version, we will include the above discussion following your valuable comments.\n\n[1]West J, Bhattacharya M. Intelligent financial fraud detection: a comprehensive review[J]. Computers & security, 2016, 57: 47-66.\n\n[2]Kim S J, Choi S J, Jang J S, et al. Innovative nanosensor for disease diagnosis[J]. Accounts of Chemical Research, 2017, 50(7): 1587-1596.\n\n[3]Hicks D. The four literatures of social science[J]. Handbook of quantitative science and technology research: The use of publication and patent statistics in studies of S&T systems, 2004: 473-496.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vkMDmKEMbB", "reviewer": "Reviewer_rjfR", "summary": "The paper proposed a new diffusion-based representation learning method for tabular data, namely D2R2, specifically for few-shot learning. It is the first paper to use the diffusion model for tabular data representation learning. The method trains a conditional diffusion model with a combined loss of vanilla diffusion reconstruction loss and a random distance match loss to learn an embedding function $z_{\\theta}$. $z_{\\theta}$ encodes a representation of the data with different noise levels. The resulting embedding is used as the conditional information to the diffusion model. Further, random linear projections are performed on the original data, and the random distance match loss is to preserve distances in the embedding space and the random projection space. Two distinct projections are used for numerical and categorical data. After training, instance-wise iteration prototypes are generated for more accurate and stable few-shot learning. The paper conducts experiments on 7 datasets from OpenML-CC18, showing superior performances than a variety of baselines.", "questions": "1. See weakness 1. What's the relationship and key novelty in the method compared to existing methods?\n2. Will other noise levels be more useful for few-shot learning? Or will a combination of different noise levels be better?", "limitations": "Limitation is discussed and the paper states to have discussed potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduced the first diffusion-based method for tabular few-shot representation learning.\n2. The proposed method has superior performance compared to SOTA methods.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. Even though D2R2 is the first method to utilize the diffusion model for representation learning, it is not the first paper to analyze/utilize the diffusion model for representation learning. The paper lacks discussion on related works in related areas such as [1] [2] [3].\n2. After training, the paper only extracts features of clean tabular data during few-shot learning, while the embedding function is learned for different noise levels. There is no analysis of the effect of different noise levels as input to the embedding function for few-shot learning.\n3. The method is limited to only tabular data, such as the specific projection head design for numerical and categorical data.\n[1] Xinlei Chen, Zhuang Liu, Saining Xie, Kaiming He. Deconstructing Denoising Diffusion Models for Self-Supervised Learning. 2024.\n[2] Sarthak Mittal, Korbinian Abstreiter, Stefan Bauer, Bernhard Schölkopf, Arash Mehrjou. Diffusion Based Representation Learning. 2023.\n[3] Xingyi Yang, Xinchao Wang. Diffusion Model as Representation Learner. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments!\n\n**Q1. What's the relationship and key novelty in the method compared to existing methods?**\nExisting papers about representation learnings are all designed for image data. Paper [1] introduces a \"latent Denoising Autoencoder\" (LDAE) architecture where the learned representations are used for Denoising Autoencoder. Unlike LDAE, we learn a latent z that supports image diffusion process, rather than Autoencoder reconstruction process, which is less suitable for tabular data as discussed in [4]. The success of the representations in paper [2] depends on the quality of the generated images, meaning accurate image generation is crucial for performance. Generating high-quality data is especially difficult for tabular data due to its varied features and the absence of strong spatial or sequential relationships [5]. Our approach, however, does not rely on data quality for results, making our model more effective with tabular data. Paper [3] extracts representations from noisy data at a specific time step t in the diffusion process, using a student model trained with task labels. Our method differs by deriving representations directly from the original feature space without requiring a student model, eliminating the need for labels during training. This is especially critical for tabular few-shot learning where labeled data is scarce.\n\n\nMoreover, different from existing papers, our work effectively modify diffusion models according to tabular data characteristics, including heterogeneous features, lack of strong spatial and sequential relationships between features, the influence of column permutation, and multimodality within the same class. We design a conditional diffusion process with distance matching to extract representations from original space, which is not influenced by the new generated samples' quality. Besides, tabular columns can be arbitrarily permuted without altering the underlying information, thus models for tabular data need to be robust to such permutations. We perturbed original data with two distinct types of relatively small noises for numerical and categorical input respectively (Section 4.1), which enhanced the robustness of the performance. Moreover, deriving representations solely from the \"diffusion-only\" model is not effective enough, as shown in our manuscript's Ablation study (Table 2). Therefore, we align the comparison of pairwise distances between the embedding space and a randomly projected space with the diffusion training process, using different projection matrices for various feature types. Additionally, we propose instance-wise iterative prototype to address multimodality for tabular data.\n\n[4]Nam J, Tack J, Lee K, et al. Stunt: Few-shot tabular learning with self-generated tasks from unlabeled tables[J]. arXiv preprint arXiv:2303.00918, 2023.\n\n[5]Xu L, Skoularidou M, Cuesta-Infante A, et al. Modeling tabular data using conditional gan[J]. Advances in neural information processing systems, 2019, 32.\n\n**Q2. Will other noise levels be more useful for few-shot learning? Or will a combination of different noise levels be better?**\n\nThe timestep t = 1, ..., T indicates noise levels in diffusion models. In our experimental analysis, we have noted a trend where performance sees an improvement with an increase in t during the initial stages. However, this trend of performance enhancement plateaus after the first few timesteps, which tends to stabilize with minimal fluctuations, as shown in the following table. We speculate that this is because, as t approaches 0, the noise levels decrease, potentially causing only the fine-grained details to be lost. Hence, the representation learns to keep the information needed to recover these fine-grained details. On the contrary, as t approaches T, noise levels rise, and the mutual information between xt and x0 begins to decrease. In these situations, effective denoising requires that all information about x0 be thoroughly encoded, which contains the most information about the data class. Thus, as stated in line 200 of our manuscript, \"We focus on larger timesteps thus extract the low-frequency semantic information rather than details.\" In our experiments, we selected the last step (T) for all datasets to ensure stable performance. We will include a discussion on this observation in the updated version of our manuscript.\n\n| Dataset   | Shot | First Step | Middle Step | 80th Percentile Step | Last Step | Average of All Steps |\n|-----------|------|------------|-------------|----------------------|-----------|----------------------|\n| optgidits | 1    | 72.39      | 79.21       | 80.93                | 81.13     | 80.66                |\n|           | 5    | 84.57      | 88.67       | 91.73                | 90.73     | 89.66                |\n| cmc       | 1    | 37.19      | 42.26       | 42.86                | 42.88     | 42.74                |\n|           | 5    | 36.28      | 42.73       | 43.04                | 43.39     | 42.37                |\n\nIn the updated version, we will include the above discussion following your valuable comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mg1hXwoA6I", "reviewer": "Reviewer_m3dm", "summary": "This paper proposes a novel approach named Diffusion-based Representation with Random Distance matching (D2R2) for tabular few-shot learning. It leverages the powerful expression ability of diffusion models to extract essential semantic knowledge crucial for the denoising process. During the training process of the designed diffusion model, it introduces a random distance matching to preserve distance information in the embeddings, thereby improving the effectiveness of classification. During the classification stage, it introduces an instance-wise iterative prototype scheme to improve performance by accommodating the multimodality of embeddings and increasing clustering robustness. Experiments demonstrate the effectiveness of the proposed method. The main contributions of this paper are:\n\n- It may be the first to propose a specifically designed diffusion method to learn semantic knowledge for tabular data.\n- It proposes an innovative framework, D2R2, to extract representations in tabular few-shot learning.\n- It introduces a novel classifier with instance-wise iteration prototypes to further improve few-shot classification performance.", "questions": "1. The authors should further emphasize the problems the heterologous feature types bring that SOTA tabular few-shot learning methods ignore, to demonstrate the necessity and innovation of the proposed method.\n2. The authors should point out the motivation for the leverage of diffusion models.\n3. The authors should explain the relationship between semi-supervised learning, self-supervised learning, and tabular few-shot learning, as well as their relevance to the D2R2 proposed in this paper.\n4. Tabular few-shot learning seems not to emphasize the variation of categories in the training (base ) and test (novel) datasets, does this indicate that it ignores the requirement for the generalization of the algorithm?", "limitations": "The authors point out the limitations of this work. No obvious potential negative societal impact remains.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The problem studied in this paper is interesting and valuable.\n- This paper is well written and in good sharp, which is easy to follow.\n- Experimental results are promising and can validate the effectiveness of the proposed method.", "weaknesses": "- In my opinion, the description of the shortcomings of existing tabular few-shot learning methods in the introduction is not sufficient. Specifically, the authors should further emphasize the problems the heterologous feature types bring that SOTA tabular few-shot learning methods ignore, to demonstrate the necessity and innovation of the proposed method.\n- On the basis of the first issue, the introduction of diffusion models should be based on the drawbacks of existing problems, rather than solely on their strong expressiveness. The authors should point out the motivation for the leverage of diffusion models.\n- The emergences of semi-supervised learning and self-supervised learning mentioned in this paper are somewhat abrupt. The authors should explain the relationship between them and tabular few-shot learning, as well as their relevance to the D2R2 proposed in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments!\n\n**Q1.**\n\nWe plan to add more explanations about shortcomings of existing methods as follows. \n\nFirstly, tabular data comprises heterologous features, which underscores the importance of simultaneously modeling continuous and categorical features. The current SOTA method STUNT[28] in our paper, and other existing few-shot learning methods designed for images or texts, treat all features as the same type, neglecting the unique information in different features. Failing to address heterogeneous features hinders the model's ability to capture complex data patterns and relationships. Moreover, the heterogeneous features make tabular data lacking in straightforward augmentation techniques, which are easily implemented in images, as demonstrated by UMTRA[21].\n\nSecondly, unlike images and texts, tabular data lacks strong spatial and sequential relationships between features. While STUNT[28] generates pseudo-labels based on the assumption that substitutable features, such as \"occupation\" acting as a proxy for \"income\", may not universally apply to tabular data. Besides, the arbitrary permutation of columns in tabular data highlights the model's robustness, which is ignored by existing methods.\n\nThirdly, the influence of column permutation and multimodal behavior within same class are all ignored by existing methods. \n\n**Q2.**\n\nWe will add more explanations about motivations as follows.\n\nExisting methods failing to address issues in Q1. thus we propose a novel approach, D2R2, to tackle above challenges in tabular data. Considering the first and second challenges above, we avoid to rely on proxy methods or augmentation methods, which are limited by the lack of relationships or augmentation techniques in tabular data. Instead, we create an information bottleneck for extracting semantic knowledge, named D2R2, which leverages the strong expressiveness of the diffusion model and distance information from pairwise comparison. Notably, we handle numerical and categorical data types separately. For example, we introduce two distinct noises and generate two distinct random projections for different feature types. Adding small noise to the input also enhances the model's robustness. \n\nFor the third challenge in Q1, we propose an instance-wise iteration prototypes classifier, which can construct stable prototypes while revealing the multimodal behavior within same class.\n\n**Q3.**\n\nBased on the unique characteristics of tabular data, as detailed in Q4, we address scenarios where there is an unlabeled training set and a very limited labeled support set (K-shot) to help predict class labels for a testing query set. This follows the problem definition in [28] and is stated in Section 3 of our paper. Such scenarios are common in critical applications like credit risk assessment[1] and diagnosing patients with rare or new diseases[2]. Based on our research, three types of methods can address these scenarios: few-shot learning, semi-supervised learning, and self-supervised learning. Meta-learning is one of the most common techniques in few-shot learning. The SOTA method [28] for few-shot tabular learning in our paper follows this scheme. On the other hand, our method innovates within the prototype scheme by enhancing embedding generation and prototype classification. In semi-supervised learning, a model is trained on labeled data then to predict the unlabeled data, where predicted labels are treated as true labels in further training rounds. This method aligns with our problem definition; however, it requires more labeled data compared to few-shot learning. Self-supervised learning excels in generating robust representations. Since our approach involves generating representations, we compare our method with the SOTA self-supervised learning methods[54,4,47] in the tabular domain. The mentioned methods serve to position our work within the broader context of existing methodologies that address similar problems. We have already outlined the relevant methods in Section 2. We will include the above more detailed discussion in the updated version. \n\n[1]Chen N, Ribeiro B, Chen A. Financial credit risk assessment: a recent review[J]. Artificial Intelligence Review, 2016, 45: 1-23.\n[2]Cereda D, Tirani M, Rovida F, et al. The early phase of the COVID-19 outbreak in Lombardy, Italy[J]. 2003.09320, 2020.\n\n**Q4.**\n\nOur problem setting follows the definition provided in [28], where there is no variation of categories between the training (base) and test (novel) datasets, but the training set is unlabeled, which is a more common scenario in tabular data. Tabular datasets consist of numerical or categorical inputs that have specific and explicit meanings, where classes are well-defined and consistent across the dataset. it is rare to encounter new categories based solely on existing features. New categories usually emerge only with the introduction of new features, which is different from domains like image or text data, which have spatial and sequential relationships. For example, new image categories can be effectively classified based on the learned spatial structure patterns between pixels. Our study focus on the most common scenario in tabular few-shot learning, as mentioned in Section 3 of our paper and supported by the problem setting in [28]. \n\nHowever, we also conducted experiments to test the generalization of our method regarding the variation of categories in the training (base) and test (novel) datasets. For the dataset for 10-class classification 'optdigit', we randomly removed one classes from the train set and validated the effectiveness of the test results. The experimental results are as follows, demonstrating the robustness and effectiveness of our method.\n\n| Scenario      | shot=1 |shot=5 |\n|---------------|--------|--------|\n| Remove Class #1| 75.21| 87.37|\n| Remove Class #2| 72.15 | 84.63|\n| Remove Class #6| 77.16| 88.62|\n| Non-Remove    | 81.13| 90.73|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a diffusion-based representation learning method for tabular few-shot learning. It receives four weak accept after the response. It merits, including the interesting idea, good writing, and sufficient experiments, are well recognized by the reviewers. For the concerns about the motivation and unclear technique details, the authors well address them in the response. I agree with the reviewers and think this work should be accepted. Please also incorporate these details in the revised manuscript.
Decision: Accept (poster)