Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Later

## Abstract

Abstract The growing success of deep learning in various domains has prompted investigations into its application to tabular data, where deep models have shown promising results compared to traditional tree-based methods. In this paper, we revisit Neighborhood Component Analysis (NCA), a classic tabular prediction method introduced in 2004, designed to learn a linear projection that captures semantic similarities between instances.
We find that minor modifications, such as adjustments to the learning objectives and the integration of deep learning architectures, significantly enhance NCA’s performance, enabling it to surpass most modern deep tabular models. Additionally, we introduce a stochastic neighbor sampling strategy that improves both the efficiency and predictive accuracy of our proposed ModernNCA — sampling only a subset of neighbors during training, while utilizing the entire neighborhood during inference.
Extensive experiments demonstrate that our ModernNCA achieves state-of-the-art results in both classification and regression tasks across various tabular datasets, outperforming both tree-based and other deep tabular models, while also reducing training time and model size.

## 1 Introduction

Tabular data, characterized by its highly structured format of rows and columns representing individual examples and specific features, is prevalent in domains such as healthcare [] and e-commerce []. Inspired by the success of deep neural networks in fields like vision and language [, , ], numerous deep models have been developed to handle tabular data, capturing complex feature interactions and achieving promising results [, , , , , , , , ].

In contrast to parametric approaches like MLPs that make predictions directly, non-parametric methods such as nearest neighbors utilize the entire training dataset and leverage relationships between instances for tabular prediction []. Metric learning, rather than relying on raw features, aims to learn a distance metric corresponding to an embedding space where semantically similar instances are close together and dissimilar ones are far apart [, , , ]. These improved distance metrics facilitate the discerning process of non-parametric/neighborhood-based methods and are particularly beneficial when dealing with a large number of classes [].
classes [].

Although metric learning approaches perform well compared to vanilla nearest neighbor methods, they often struggle to match the predictive power of tree-based nonlinear approaches like Gradient Boosting Decision Trees (GBDT) [, ] on tabular data. However, the recent successes of deep metric learning in diverse domains such as image recognition[, , , ], person re-identification [, ], and recommendation systems [, ] suggest its untapped potential for tabular datasets. Moreover, specially designed MLP architectures and weight regularization strategies have enabled simple parametric models like MLPs to excel on many tabular datasets [, ], indicating that similar improvements could also benefit neighborhood-based methods.

Figure: Figure 1: Performance-Efficiency-Size comparison between ModernNCA on classification (a) and regression (b) datasets. Representative tabular prediction methods, including the classical methods (in green), the parametric deep methods (in blue), and the non-parametric/neighborhood-based deep methods (in red), are investigated, based on their records over all datasets in Table 1 and Table 2. The average rank among these eight methods is used as the performance measure. We calculate the average training time (in seconds) and the size of the model (denoted by the radius of the circles, where the larger the circle, the bigger the model). ModernNCA achieves high training speed compared to other deep tabular models and has a lower model size. TabCon is another proposed baseline. ModernNCA outperforms other methods in two cases.
Refer to caption: x1.png

This leads us to a pivotal question: “Can we improve metric learning with modern deep learning techniques to achieve competitive performance in tabular prediction tasks?”

In this paper, we revisit the classical metric learning approach, Neighborhood Component Analysis (NCA), originally proposed by Goldberger et al. [] in 2004. NCA optimizes the prediction accuracy of the target instance based on its neighbors in a linearly projected space.
With simple modifications to NCA, our enhanced method, ModernNCA, effectively addresses tabular prediction tasks.

In particular, our ModernNCA introduces a learning objective tailored for both classification and regression tasks. We further explore a deep neural network architecture that projects input instances into a latent embedding space in a nonlinear manner. Given that NCA’s prediction relies on the set of neighbors, which increases the batch size and computational burden during training, we employ a Stochastic Neighborhood Sampling (SNS) strategy.
SNS randomly selects a subset of the training set as neighbors during training while utilizing the entire training set in the test phase. This strategy improves not only the training efficiency but also the generalization capability of the model. Our experiments validate that ModernNCA outperforms most existing tabular prediction methods, including both tree-based and deep learning approaches, across various datasets. Figure 1 demonstrates that ModernNCA balances training efficiency (with lower training time compared to other deep tabular models), generalization ability (with higher average accuracy), and model size.
The contributions of our paper are:

- •
We revisit the classical metric learning approach NCA and explore ways to improve the model’s performance with modern deep learning techniques.
- •
Our proposed ModernNCA is capable of both classification and regression tasks. An additional sampling strategy improves the training efficiency and the generalization ability of the model.
- •
Extensive experiments across multiple datasets demonstrate that ModernNCA serves as a strong deep tabular baseline for tabular prediction tasks.

## 2 Related Work

Learning with Tabular Data.
Tabular data is a common format across various applications such as click-through rate prediction [] and time-series forecasting []. Tree-based methods like XGBoost [], LightGBM [], and CatBoost [] have proven effective at capturing feature interactions and are widely used in real-world applications.
Since changes in model families and hyper-parameters can significantly affect a model’s generalization ability [], some strategies focus on automated model selection and hyper-parameter tuning to optimize performance across different tasks [, ].

Deep Tabular Data Learning.
Recognizing the ability of deep neural networks to learn feature representations from raw data and make nonlinear predictions, recent methods have applied deep learning techniques to tabular models [, , , , , , , ]. For instance, deep architectures such as residual networks and transformers have been adapted for tabular prediction [, ]. Moreover, data augmentation strategies have been introduced to mitigate overfitting in deep models [, , ].
Deep tabular models have demonstrated competitive performance across a wide range of applications, and pre-trained deep tabular predictors have shown the ability to generalize to downstream tabular tasks [, , , , , , ]. However, researchers have observed that deep models still face challenges in capturing high-order feature interactions as effectively as tree-based models [, ].

Metric Learning.
Methods like K Nearest Neighbors (KNN) make predictions based on the entire training set. The choice of distance metric critically determines the set of neighbors and affects the discriminative ability of the model [, ].
Metric learning learns a projection to bring similar instances closer together and push dissimilar ones farther apart, which further yields improved classification and regression results of KNN [, , , ].
In addition to linear metrics, nonlinear metrics are often implemented using multiple metrics [, , ], local composition [, , , ], or kernel methods [, ]. Initially focused on tabular data, metric learning has become a valuable tool when integrated with deep learning techniques across various domains such as image recognition [, , , ], person re-identification [, ], and recommendation systems [, ].
Recently, TabR [] incorporated metric learning with transformers [] to enhance tabular predictions through neighborhood-based methods. Despite promising results, the extensive computational burden of neighborhood selection and the complicated architecture limit TabR’s practicality. Based on the classical method NCA [], we propose a simple deep tabular baseline that maintains efficient training speeds without sacrificing performance.

## 3 Preliminary

In this section, we first introduce the task learning with tabular data. We then provide a brief overview of metric learning approaches for tabular data, specifically on the NCA method.

### 3.1 Learning with Tabular Data

A labeled tabular dataset is formatted as $N$ examples (rows in the table) and $d$ features/attributes (columns in the table). An instance ${\bm{x}}_{i}$ is depicted by its $d$ feature values. There are two kinds of features: the numerical (continuous) ones and categorical (discrete) ones.
Given $x_{i,j}$ as the $j$-th feature of instance ${\bm{x}}_{i}$, we use $x_{i,j}^{\textit{\rm num}}\in\mathbb{R}$ and ${\bm{x}}_{i,j}^{\textit{\rm cat}}$ to denote numerical (*e.g*., the height of a person) and categorical (*e.g*., the gender of a person) feature values of an instance, respectively.
The categorical features are usually transformed in a one-hot manner, *i.e*., ${\bm{x}}_{i,j}^{\textit{\rm cat}}\in\{0,1\}^{K_{j}}$ and the index of value 1 indicates one of the $K_{j}$ categorical values. We assume the instance ${\bm{x}}_{i}\in\mathbb{R}^{d}$ w.l.o.g. and will explore other encoding strategies later.
Each instance is associated with a label $y_{i}$, where $y_{i}\in[C]=\{1,\ldots,C\}$ in a multi-class classification task and $y_{i}\in\mathbb{R}$ in a regression task.

Given a tabular dataset $\mathcal{D}=\{({\bm{x}}_{i},y_{i})\}_{i=1}^{N}$, we aim to learn a model $f$ on $\mathcal{D}$ that maps ${\bm{x}}_{i}$ to its label $y_{i}$. We measure the quality of $f$ by the joint likelihood on $\mathcal{D}$, *i.e*., $\max_{f}\prod_{({\bm{x}}_{i},y_{i})\in\mathcal{D}}\Pr(y_{i}\mid f({\bm{x}}_{i}))$. The objective could be reformulated in the form of negative log-likelihood of predicted labels:

$$ $\min_{f}\;\sum_{({\bm{x}}_{i},y_{i})\in\mathcal{D}}-\log\Pr(y_{i}\mid f({\bm{x }}_{i}))=\sum_{({\bm{x}}_{i},y_{i})\in\mathcal{D}}\ell(y,\;\hat{y}_{i}=f({\bm{ x}}_{i}))\;.$ (1) $$

The discrepancy between the predicted label $\hat{y}_{i}$ and the true label $y_{i}$ could be measured by the loss $\ell(\cdot,\cdot)$, *e.g*., cross-entropy. We expect the learned model $f$ is able to extend its ability to unseen instances sampled from the same distribution as $\mathcal{D}$. $f$ could be implemented with classical methods such as SVM, MLP, or tree-based approaches.

K Nearest Neighbors (KNN) is one of the most representative non-parametric tabular models for classification and regression — making predictions based on the labels of the nearest neighbors [, ]. In other words, the model $f$ makes predictions via $f({\bm{x}}_{i},\mathcal{D})$.
Given an instance ${\bm{x}}_{i}$, KNN calculates the distance between ${\bm{x}}_{i}$ and other instances in $\mathcal{D}$. Assume the $K$ nearest neighbors are $\mathcal{N}({\bm{x}}_{i};\mathcal{D})=\{({\bm{x}}_{1},y_{1}),\ldots,({\bm{x}}_
{K},y_{K})\}$.
then, the label $y_{i}$ of ${\bm{x}}_{i}$ is predicted based on those labels in the neighbor set $\mathcal{N}({\bm{x}}_{i};\mathcal{D})$. For classification task $\hat{y}_{i}$ is the majority voting of labels in $\mathcal{N}({\bm{x}}_{i};\mathcal{D})$ while is the average of those labels in regression tasks.

### 3.2 Metric Learning on Tabular Data

The distance ${\rm dist}({\bm{x}}_{i},{\bm{x}}_{j})$ in KNN determines the set of nearest neighbors $\mathcal{N}({\bm{x}}_{i};\mathcal{D})$, which is one of its key factors. Instead of using the Euclidean distance, the Mahalanobis distance between a pair $({\bm{x}}_{i},{\bm{x}}_{j})$ with metric $\bm{M}$ is:

$$ ${\rm dist}_{\bm{M}}({\bm{x}}_{i},{\bm{x}}_{j})=\sqrt{({\bm{x}}_{i}-{\bm{x}}_{j })^{\top}\bm{M}({\bm{x}}_{i}-{\bm{x}}_{j})}.$ (2) $$

The matrix $\bm{M}$ is assumed to be a semi-definite one.
${\rm dist}^{2}_{\bm{M}}$ becomes the Euclidean distance (denoted by ${\rm dist}({\bm{x}}_{i},{\bm{x}}_{j})$) when we set the metric $\bm{M}$ as the identity matrix.
$\bm{M}$ can be decomposed into projections $\bm{M}=\bm{L}\bm{L}^{\top}$ with $\bm{L}\in\mathbb{R}^{d\times d^{\prime}}$ so that the Mahalanobis distance becomes the Euclidean distance in the space projected by $\bm{L}$.
Most distance metric learning methods learn the metric $\bm{M}$ or $\bm{L}$ in two stages. Side information in the form of pairs or triplets is constructed first based on labels, guiding the similarity and comparison relationship between instances [, , , ]. With the help of the learned metric, KNN then makes more accurate final predictions.

Neighborhood Component Analysis (NCA).
NCA is a one-stage approach that directly links the projection $\bm{L}$ with the posterior probability that an instance ${\bm{x}}$ is classified as the class $y$ []:

$$ $\Pr(\hat{y}=y\mid{\bm{x}},\mathcal{D},\bm{L})=\sum_{{{\bm{x}}_{i}\in\mathcal{D }\land y_{i}=y}}\frac{\exp\left(-\operatorname{dist}^{2}(\bm{L}^{\top}{\bm{x}} ,\;\bm{L}^{\top}{\bm{x}}_{i})\right)}{\sum_{{\bm{x}}_{j}\in\mathcal{D},{\bm{x} }_{j}\neq{\bm{x}}_{i}}\exp\left(-\operatorname{dist}^{2}(\bm{L}^{\top}{\bm{x}} ,\;\bm{L}^{\top}{\bm{x}}_{j})\right)}\;.$ (3) $$

Therefore, the posterior that an instance ${\bm{x}}$ belongs to the class $y$ depends on its similarity (measured by the negative squared Euclidean distance in the space projected by $\bm{L}$) between its neighbors from class $y$ in $\mathcal{D}$.
Instead of considering all instances in the neighborhood equally, the prediction in Equation 3 mimics a soft version of KNN, where all instances in the training set are weighted (nearer neighbors have more weight) for the final decision. Similar strategies have also been explored in domains like few-shot learning [].

## 4 ModernNCA

In this section, we describe the learning objective, the architecture, and the training strategy of our improved version of NCA, *i.e*., ModernNCA (abbreviated as M-NCA).

Learning Objective.
We use a nonlinear transformation $\phi$, which maps the input ${\bm{x}}_{i}$ into a latent space with dimensionality $d^{\prime}$. Assume the label $y_{i}$ is a continuous value in a regression task and in a one-hot form in a classification task. Then we modify the Equation 3 as

$$ $\hat{y}_{i}=\sum_{{{\bm{x}}_{i}\in\mathcal{D}}}\frac{\exp\left(-\operatorname{ dist}(\phi({\bm{x}}),\;\phi({\bm{x}}_{i}))\right)}{\sum_{{\bm{x}}_{j}\in \mathcal{D},{\bm{x}}_{j}\neq{\bm{x}}_{i}}\exp\left(-\operatorname{dist}(\phi({ \bm{x}}),\;\phi({\bm{x}}_{j}))\right)}y_{i}\;.$ (4) $$

Equation 4 makes predictions that similar instances (based on their distance in the embedding space mapped by $\phi$) will have close outputs.
In the classification scenario, Equation 4 generalizes Equation 3 and predicts the label of the target instance with a weighted average of its neighbors in each one of the $C$ classes. $\hat{y}_{i}$ is a probability vector which can be used as $\{\Pr(\hat{y}=y\mid{\bm{x}},\mathcal{D},\phi)\}_{y\in[C]}$ in this case.
In the regression scenario, the prediction of an instance is the weighted sum over the scalar labels of its neighborhood.
By integrating
Equation 3 with Equation 1, we set the loss $\ell$ in Equation 1 as the negative log-likelihood loss for classification, while the mean square error for regression problems.
By minimizing Equation 1, the embedding $\phi$ pulls the same-class instances in the training set $\mathcal{D}$ close and pushes other instances away. In the test phase, neighbors of the test instance in the whole training set $\mathcal{D}$ are utilized to make predictions.

Architectures.
We set the mapping $\phi$ as a linear layer by default, *i.e*., $\phi({\bm{x}}_{i})=\text{Linear}({\bm{x}}_{i})$, containing a linear projection and a bias. We further improve the ability of ${\bm{x}}_{i}$ with multiple nonlinear layers.

In particular, we define the one-layer nonlinear mapping as a sequential of multiple operators [], namely, the one-dimensional batch normalization [], linear layer, ReLU activation, dropout [], and another linear layer. In other words, the input ${\bm{x}}_{i}$ will be transformed by

$$ $g({\bm{x}}_{i})=\text{Linear}\left(\text{Dropout}\left(\left(\text{ReLU}\left( \text{Linear}\left(\text{BatchNorm}\left({\bm{x}}_{i}\right)\right)\right) \right)\right)\right)\;.$ (5) $$

Zero or more layers of blocks are applied in addition to the original linear layer to implement the final mapping.
The final $\phi$ applies an additional batch normalization to calibrate the embedding. Empirical results indicate that batch normalization outperforms other normalization strategies such as layer normalization [] in learning a latent embedding space.

Stochastic Neighborhood Sampling.
Stochastic gradient descent is usually applied to optimize deep neural networks — a batch of instances is sampled, and the batch-wise loss is calculated for back-propagation.
However, pairwise distances need to be calculated during the training process, where the distances between the instance in the batch and the entire training set $\mathcal{D}$ are considered to estimate the loss function Equation 4, introducing a high computational burden.

To accelerate the training speed of ModernNCA, we propose a Stochastic Neighborhood Sampling (SNS) strategy, where a subset $\hat{\mathcal{D}}$ of the training set $\mathcal{D}$ is randomly sampled as the candidate instances and distances with which are calculated. In other words, $\hat{\mathcal{D}}$ substitutes $\mathcal{D}$ in Equation 4 and only the labels in $\hat{\mathcal{D}}$ can be utilized to predict the label of a given instance. The model searches neighbors in the entire training set $\mathcal{D}$ during the inference stage.

We empirically observed that the SNS not only increases the training efficiency of ModernNCA since fewer examples are utilized for back-propagation but also SNS improves the generalization ability of the learned metric. The improvements mainly come from the fact that the mapping $\phi$ is learned on more difficult prediction tasks during the training, so the $\phi$ adapts to the noisy and unstable neighborhoods in the test scenario. The influence of sampling ratio and other sampling strategies are investigated in the experiments.

Remark1: Relationship with other deep tabular methods. A recent deep tabular model TabR [] also takes advantage of the nearest neighbor and predicts in a neighborhood-based manner. However, there are three main differences between our ModernNCA and TabR.
First, our approach is much more concise than TabR, where the latter employs both a deep neural network and a Transformer-like architecture to search the entire training set. Our ModernNCA validates that with the prediction strategy in Equation 4, only a linear layer can achieve competitive results in most cases.
Second, ModernNCA is more efficient than TabR, requiring less training time and memory. This efficiency stems from both its simpler architecture and the stochastic sampling strategy.
In addition, instead of using an instance-specific prediction as in TabR, ModernNCA learns an embedding space and makes predictions in a more effective manner.
Figure 1 demonstrates the advantages of ModernNCA in terms of performance, efficiency, and model size when compared to TabR.

Remark 2: Another way to learn the deep tabular transformation. Our ModernNCA leverages NCA to implement the objective function in Equation 4, where the quality of the learned tabular embedding $\phi({\bm{x}})$ directly influences the classification/regression performance over ${\bm{x}}$.
As mentioned in subsection 3.2, another common strategy for learning $\phi$ involves a two-stage process: first, $\phi$ is optimized to match the comparison relationships between mapped instances, such as triplets, and then simple classifiers like KNN and LR are applied based on $\phi$.
Since a mini-batch $\mathcal{B}$ of examples is sampled at a time during the training process, we investigate the Supervised Contrastive loss [, ], where supervision is generated within a mini-batch. The objective to learn the mapping $\phi$ is:

$$ $\displaystyle\ell(\phi\mid\mathcal{B})=-\sum_{({\bm{x}}_{i},y_{i})\in\mathcal{ B}}\sum_{{\bm{x}}_{j}\in P({\bm{x}}_{i})}\log\frac{\exp\left(\phi({\bm{x}}_{i} )\cdot\phi({\bm{x}}_{j})/\tau\right)}{\sum_{{\bm{x}}_{k}\in\mathcal{D},{\bm{x} }_{k}\neq{\bm{x}}_{i}}\exp\left(\phi({\bm{x}}_{i})\cdot\phi({\bm{x}}_{k})/\tau \right)}\;,$ (6) $\displaystyle P({\bm{x}}_{i})=\{{\bm{x}}_{j}\mid\forall({\bm{x}}_{j},y_{j})\in \mathcal{B}\wedge y_{j}=y_{i}\}\;.$ (7) $$

$P({\bm{x}}_{i})$ is the set of instances in the mini-batch that have the same label as ${\bm{x}}_{i}$. $\tau$ is the temperature parameter used to calibrate the loss function. By minimizing Equation 6, $\phi$ also pulls same-class instances together and pushes dissimilar ones apart. With the learned $\phi$, we apply KNN for classification/regression for inference. We discretize the label values in the regression scenario and denote this baseline method as Tabular Contrastive (TabCon).

## 5 Experiments

We validate the performance of ModernNCA (M-NCA) in standard tabular classification and regression tasks. Then, we analyze the properties of ModernNCA with ablation studies.

### 5.1 Setups

Datasets. We validate our ModernNCA on 12 classification and 10 regression datasets collected from OpenML [] and Kaggle. The detailed statistics of datasets are listed in Table 5 in the appendix. Four out of the 12 classification datasets have more than two classes.

Evaluation. We mainly follow the setups in [, ] to evaluate all methods. We randomly split each dataset into three partitions for training/validation/test with proportions 64%/16%/20%, respectively. For each dataset, we train a given model with 15 random seeds, and the average performance on the test set is reported. We use accuracy for classification (the higher, the better) and Root Mean Square Error (RMSE) for regression (the lower, the better). We also follow [, ] and report the average performance rank among all methods and datasets (the lower, the better).

Comparison Methods. We compare ModernNCA with three different threads of approaches.
First, we compare classical parametric tabular prediction methods, including the tree-based methods XGBoost[] and CatBoost [].
Then, we compare parametric deep tabular models. We follow the architecture in [] to implement MLP, and MLPPLR [] is a variant of MLP that leverages the piecewise linear encoding on numerical features [].
FT-Transformer (FT-T) [] learns tokens for each attribute and then makes predictions with transformer. PTaRL is a prototype-based tabular method based on space calibration [].
For neighborhood-based tabular methods, we consider $K$NN,
metric-learning methods like LMNN [] and NCA [], TabR [], and our tabular contrastive baseline TabCon.(^1^11The code of our ModernNCA is available at [https://github.com/qile2000/LAMDA-TALENT](https://github.com/qile2000/LAMDA-TALENT).)

Implementation Details.
We pre-process all datasets following [].
For all deep methods, we set the batch size as 1024.
The hyper-parameters of all methods are searched based on the training and validation set via Optuna [] following [, ] over 100 trials. The best-performed hyper-parameters are fixed during the final 15 seeds.
Since the sampling rate of SNS effectively enhances the performance and reduces the training overhead, we treat it as a hyper-parameter and search within the range of [0.05, 0.6].
The metric-learning methods like LMNN are implemented based on the metric-learn library [].
We use one-hot encoding for categorical features, and PLR (lite) embedding, a simplified version of PLR embedding(the combination of periodic embeddings, a linear layer, and ReLU) [] proposed in [], for numerical features. For a fair comparison, the same encoding strategy is also applied in MLPPLR and TabR.

**Table 1: The average accuracy (the higher, the better) of all methods on 12 classification datasets. The best results on each dataset are in bold, and the second-best ones are underlined. We also report the average rank of different methods among all datasets (the lower, the better).**
| $\uparrow$ | M-NCA | TabCon | XGBoost | CatBoost | MLP | FT-T | TabR | KNN | MLPPLR | PTaRL | LMNN | NCA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AD | 87.29 | 86.97 | 87.21 | 87.48 | 85.80 | 85.89 | 86.86 | 84.61 | 86.97 | 85.53 | 83.21 | 83.87 |
| CU | 79.99 | 77.72 | 81.73 | 81.03 | 75.54 | 80.65 | 79.61 | 70.08 | 80.75 | 75.89 | 72.12 | 73.78 |
| EL | 96.12 | 87.44 | 92.06 | 91.69 | 85.49 | 87.50 | 96.46 | 84.27 | 86.93 | 85.10 | 83.56 | 85.15 |
| EY | 99.41 | 82.88 | 71.02 | 71.88 | 61.29 | 71.49 | 98.20 | 59.46 | 74.27 | 60.13 | 54.98 | 66.50 |
| HE | 39.84 | 38.32 | 37.86 | 38.26 | 38.34 | 38.49 | 40.80 | 32.91 | 38.95 | 37.64 | 29.36 | 31.30 |
| HI | 73.09 | 72.36 | 72.73 | 72.66 | 72.29 | 72.92 | 72.85 | 66.78 | 72.64 | 72.48 | 61.44 | 66.94 |
| IP | 87.96 | 87.65 | 88.64 | 87.74 | 84.86 | 84.95 | 87.23 | 83.72 | 87.84 | 84.83 | 83.81 | 84.10 |
| JA | 74.08 | 72.17 | 72.55 | 72.39 | 71.97 | 72.38 | 73.44 | 65.67 | 72.10 | 71.52 | 59.12 | 64.64 |
| MA | 87.90 | 87.22 | 87.67 | 87.87 | 87.08 | 87.73 | 87.91 | 84.65 | 87.17 | 86.94 | 83.68 | 84.73 |
| OT | 82.37 | 78.94 | 82.45 | 82.21 | 81.47 | 80.82 | 82.38 | 78.15 | 81.24 | 81.20 | 77.37 | 78.85 |
| PH | 89.29 | 87.88 | 87.66 | 87.97 | 86.88 | 87.86 | 89.10 | 86.86 | 87.13 | 85.54 | 85.75 | 87.05 |
| WI | 74.74 | 73.72 | 74.39 | 74.59 | 72.50 | 72.64 | 74.22 | 77.10 | 72.50 | 72.16 | 73.58 | 72.60 |
| rank | 2.000 | 5.917 | 3.833 | 3.750 | 8.000 | 5.417 | 3.083 | 10.000 | 5.667 | 9.250 | 11.333 | 9.750 |

### 5.2 Main Results

The comparison results between ModernNCA and other methods for classification and regression tasks are reported in Table 1 and Table 2, respectively, where ModernNCA achieves the best average rank in both cases.

For classification results in Table 1, ModernNCA achieves better results than tree-based methods like XGBoost in most cases, which indicates that utilizing deep neural network ModernNCA has a stronger ability to make nonlinear predictions. It is clearly observed that although classical metric learning methods such as LMNN and NCA can improve KNN in some cases, they are difficult to achieve competitive performance as tree-based and deep methods. When compared with deep tabular methods like FT-T and MLPPLR, ModernNCA still keeps its superiority. The most related comparison method is TabR, which extracts features of all neighbors in the training set and makes predictions with a variant of Transformer. Our ModernNCA outperforms TabR with simpler architecture and shorter training time. The results indicate that ModernNCA acts as an effective tabular deep baseline.
Our TabCon baseline also demonstrates the effectiveness of its learned mapping, outperforming classical metric learning methods in all cases. TabCon is competitive with FT-T and even surpasses XGBoost in some cases, such as “HE” and “PH”. The superiority of ModernNCA over TabCon may result from the direct linkage between the learned mapping and prediction performance in ModernNCA.
Similar phenomena exist in regression cases in Table 2. ModernNCA is the only deep model that consistently outperforms tree-based methods on regression datasets overall. Classical metric-learning methods cannot be applied to regression directly. Our ModernNCA shows its superiority over two competitive methods MLPPLR and TabR.
Figure 1 indicates the comparison between representative classification and regression methods w.r.t. the performance, training time, and model size over all datasets. Although some models, such as TabR, obtain strong performance, they require much longer training time. Our ModernNCA makes a good balance among various model evaluation criteria.

**Table 2: The average RMSE (the lower, the better) of all methods on 10 regression datasets. The best results on each dataset are in bold, and the second-best ones are underlined. The scientific notation beside the dataset name indicates the scale of the results, *e.g*., $\times 10$ means the final value of all results should be multiple by $10$. We also report the average rank among all datasets (the lower, the better).**
| $\downarrow$ | M-NCA | TabCon | XGBoost | CatBoost | MLP | FT-T | TabR | KNN | MLPPLR | PTaRL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AI$\times 10^{-3}$ | .1532 | .1522 | .1527 | .1465 | .1558 | .1565 | .1546 | .2431 | .1522 | .1563 |
| BI$\times 10^{2}$ | .7057 | .7352 | .7180 | .7264 | .7773 | .7466 | .6657 | 1.087 | .7155 | .7193 |
| CA | .4212 | .4581 | .4328 | .4360 | .5074 | .4701 | .4157 | .5843 | .4690 | .5086 |
| CM$\times 10^{3}$ | .4618 | .5011 | .4809 | .4929 | .5131 | .5187 | .5091 | .8754 | .4655 | .5211 |
| CP$\times 10$ | .2381 | .2473 | .2404 | .2365 | .2490 | .2310 | .2337 | .8825 | .2468 | .2487 |
| FO$\times 10$ | .7272 | .7436 | .7384 | .7393 | .7900 | .7885 | .7398 | .8140 | .7428 | .7896 |
| HO$\times 10^{5}$ | .3041 | .3156 | .3026 | .3051 | .3161 | .3074 | .3133 | .3663 | .3093 | .3149 |
| HOU$\times 10^{5}$ | .4291 | .4963 | .4797 | .4546 | .5221 | .4791 | .4149 | .6889 | .4921 | .5118 |
| LA$\times 10^{3}$ | .4135 | .4529 | .4448 | .4506 | .4789 | .4609 | .4606 | .4968 | .4753 | .4777 |
| MIA$\times 10^{5}$ | .8284 | .9322 | .8740 | .8034 | .8973 | .8717 | .8585 | 1.508 | .8562 | .9302 |
| rank | 2.200 | 6.100 | 3.500 | 3.100 | 8.300 | 5.900 | 3.600 | 10.000 | 4.500 | 7.800 |

**Table 3: Results of comparison methods with different architectures. NCA cannot be applied for regression tasks.**
|  | TabR | NCA | Linear | LayerNorm | Residual | M-NCA |
| --- | --- | --- | --- | --- | --- | --- |
| CA $\downarrow$ | .4157 | - | .4212 | .4210 | .4237 | .4212 |
| AD $\uparrow$ | 86.86 | 83.87 | 87.18 | 87.20 | 87.25 | 87.29 |
| MIA$\times 10^{5}$$\downarrow$ | .8585 | - | .8325 | .9070 | .8418 | .8284 |
| PH $\uparrow$ | 89.10 | 87.05 | 89.09 | 88.70 | 89.78 | 89.29 |

### 5.3 Ablation Studies

We analyze different properties of the proposed ModernNCA based on three datasets, with CA/MIA for regression and AD/PH for classification.

Linear vs. Deep Architectures.
We first investigate the design of architectures $\phi$ in ModernNCA, where we add one or more layers of blocks $g(\cdot)$ based on a linear projection. The number of layers is set as a hyper-parameter and could be determined based on the validation set. We consider three choices. First, we set $\phi$ as the linear projection, where the dimensionality of the projected space is the hyper-parameter. We also replace batch normalization in the block with layer normalization. Finally, we equip $\phi$ with another residual link from the block’s input to its output.

The results are listed in Table 3, and we also include the results of TabR/NCA for reference. Although NCA uses a linear projection, we find the linear version of ModernNCA (the fourth column) achieves much better performance on two classification datasets. For example, on AD, the linear version of ModernNCA outperforms NCA by around 4%. The comparison indicates that the objective and training strategy of ModernNCA is essential. When compared with the last column where ModernNCA utilizes the deep neural network, we find the usage of nonlinear deep architecture is necessary on datasets such as MIA. The linear version can achieve competitive or even better results since the projected dimension is searched in a smaller hyper-parameter space, and for ModernNCA the number of additional layers is searched based on the validation set from zero to more layers.

The variant of the fifth column utilizes another normalization strategy, layer normalization, in the blocks instead of batch normalization in ModernNCA. We empirically find that batch normalization performs better on average, especially in classification tasks. When comparing residual and MLP blocks in the last two columns, we find using multiple nonlinear layers directly performs a bit better, so we choose the MLP implementation in Equation 5 for ModernNCA.

**Table 4: Results of ModernNCA variants with different distance functions to obtain neighborhood relationship in the learned embedding space.**
|  | Euclid | Euclid^2 | $\ell_{1}$-Norm | Cosine | Inner Product |
| --- | --- | --- | --- | --- | --- |
| CA $\downarrow$ | .4212 | .4271 | .4184 | .4264 | .4528 |
| AD $\uparrow$ | 87.29 | 87.04 | 87.25 | 87.18 | 87.07 |
| MIA$\times 10^{5}$$\downarrow$ | .8284 | .8637 | .8624 | .8627 | .8497 |
| PH $\uparrow$ | 89.29 | 89.38 | 89.08 | 89.10 | 88.23 |

The Influence of Distance Functions. The predicted label of a target instance ${\bm{x}}_{i}$ is determined by the label of its neighbors in the learned embedding space projected by $\phi$. The distance function is the key to determining the pairwise relationship between instances in the embedding space and influences the weights in Equation 4.
In ModernNCA, we choose Euclidean distance ${\rm dist}_{\rm EUC}(\phi({\bm{x}}_{i}),\phi({\bm{x}}_{j}))=\sqrt{(\phi({\bm{x
}}_{i})-\phi({\bm{x}}_{j}))^{\top}(\phi({\bm{x}}_{i})-\phi({\bm{x}}_{j}))}=\|
\phi({\bm{x}}_{i})-\phi({\bm{x}}_{j})\|_{2}$. We also utilize other distance functions, *e.g*., the squared Euclidean distance ${\rm dist}_{\rm EUC}^{2}(\phi({\bm{x}}_{i}),\phi({\bm{x}}_{j}))$, the $\ell_{1}$-norm distance ${\rm dist}(\phi({\bm{x}}_{i}),\phi({\bm{x}}_{j}))=\|\phi({\bm{x}}_{i})-\phi({
\bm{x}}_{j})\|_{1}$, the (negative) cosine similarity ${\rm dist}(\phi({\bm{x}}_{i}),\phi({\bm{x}}_{j}))=-({\bm{x}}_{i}^{\top}{\bm{x}
}_{j})/(\|{\bm{x}}_{i}\|_{2}\|{\bm{x}}_{j}\|_{2})$, and the (negative) inner product ${\rm dist}(\phi({\bm{x}}_{i}),\phi({\bm{x}}_{j}))=-{\bm{x}}_{i}^{\top}{\bm{x}}
_{j}$.
The results using different distance functions are listed in Table 4, where Euclidean distance achieves good results on average. Although $\ell_{1}$-norm also performs well, it will introduce larger computational costs than the Euclidean distance.

Figure: Figure 2: The change of performance with different sampling rates among {10%, 30%, 50%, 80%, 100%} in stochastic neighborhood sampling strategy.
Refer to caption: x3.png

The Influence of Sampling Ratios.
Due to the huge computational cost of calculating distance in the learned embedding space, our ModernNCA utilizes a Stochastic Neighborhood Sampling (SNS) strategy, where only a proportion of training set data are randomly sampled for each mini-batch. The whole training set is utilized in the test phase. Due to the usage of a subset of sampling data, the training time cost is reduced a lot.
We vary the proportion and evaluate the corresponding test performance of the model in Figure 2.
We observe from Figure 2 that 30%-50% ratios of the training set help more for ModernNCA than using the full training set. SNS not only improves the training efficiency but also the model’s generalization ability.
The plots demonstrate that the ratio of sampled training data matters, which is a hyper-parameter tuned for ModernNCA.

The Influence of Sampling Strategy.
As mentioned before, SNS randomly samples a subset of training data for each mini-batch when calculating the loss of Equation 4. We also investigate whether we could further improve the classification/regression ability of the model when we incorporate richer information during the sampling process, *e.g*., the label of the instances.

We consider another two sampling strategies in addition to the fully random one we used before. First is class-wise random sampling, which means that given a proportion, we sample from each class in the training set and combine them together. This strategy takes advantage of the training label information and keeps the instances from all classes that will exist in the sampled subset. Besides, we also consider the sampling strategy based on the pairwise distances between instances. Since the neighbors of an instance may contribute more (with larger weights) in Equation 4, so given a mini-batch, we first calculate the Euclidean distance between instances in the batch and all the training set with the embedding function $\phi$ in the current epoch. Then we sample the training set based on the reciprocal of the pairwise distance value. In detail, given an instance ${\bm{x}}_{i}$, we provide instance-specific neighborhood candidates and ${\bm{x}}_{j}$ in the training set is sampled based on the probability $\sim 1/({\rm dist}(\phi({\bm{x}}_{i}),\phi({\bm{x}}_{j})))^{\tau}$. $\tau$ is a non-negative hyper-parameter to calibrate the distribution.
The distance calculation requires forward passes of the model $\phi$ over all the training instances, and the instance-specific neighborhood makes the loss related to a wide range of the training data. Therefore, the distance-based sampling strategy has a low training speed and high computational burden.

Figure: Figure 3: The influence of different sampling strategies, namely, the random strategy, the label-based strategy, and the distance-based strategy.
Refer to caption: x5.png

The comparison results between different sampling strategies on two classification datasets are listed in Figure 3. We find the label-based sampling strategy cannot provide further improvements. Although the distance-based strategy helps in certain cases, the improvements are limited. Taking a holistic consideration of the performance and efficiency, we choose to use the vanilla random sampling in ModernNCA.

Figure: Figure 4: The change of performance given few-shot training examples among the different percentages of the training set examples. Since the PH dataset has only one sample at 0.05% shot percent, we only show model performance for shot percents from 0.1% to 1%.
Refer to caption: x6.png

Learning with Few-Shot Data.
We investigate the change in performance when the number of training examples is limited. The results are shown in Figure 4.
Given a dataset, we synthesize the few-shot learning scenario by randomly sampling a subset of the training data with the radio changes from $\{0.05\%,0.1\%,0.2\%,0.5\%,1\%\}$. Due to the small-size of the training set, we learn all methods with the default hyper-parameters. The results are repeated 10 times and the average of them are reported.
Based on the few-shot performance shown in Figure 4, we find our ModernNCA is competitive when compared with other tabular prediction methods.

## 6 Conclusion

Making predictions by leveraging relationships within a learned embedding space is a classical metric learning notion in machine learning. While traditional metric learning methods often underperform compared to tree-based approaches on tabular datasets, our study revisits and enhances one of the most representative methods, NCA. The improved ModernNCA establishes itself as a strong baseline for deep tabular prediction tasks, often outperforming other methods in terms of classification and regression capabilities, while also reducing model size and training time. We believe our research will encourage further investigation into classical methods within the field of tabular data. By making minor yet effective modifications to their core concepts, these methods can be revitalized, offering significant improvements and novel solutions.

## Appendix A Datasets

In this section, we present the preprocessing steps applied to the dataset before training, along with a detailed description of the dataset we used.

### A.1 Data Pre-processing

We follow [] and pre-process datasets for all methods.
For example, we use standardization, involving mean subtraction and scaling, to normalize each numerical dataset. We also apply one-hot encoding on all categorical features.

### A.2 Dataset Information

The datasets come from different domains, including [OpenML](https://www.openml.org/) [] and [Kaggle](https://www.kaggle.com/). These datasets are commonly used in the tabular prediction fields [, ].
The descriptions are shown in Table 5.
To reduce randomness, we used datasets with sample sizes greater than 1000.

**Table 5: Descriptions of all datasets. There are 8 binary classification datasets, 4 multi-class classification datasets, and 10 regression datasets.**
|  | Abbreviation | Task Type | Data Size | Num. | Cat. | Class Num |
| --- | --- | --- | --- | --- | --- | --- |
| Adult | AD | classification | 48842 | 6 | 8 | 2 |
| Ailerons | AI | regression | 13750 | 40 | 0 | - |
| California Housing | CA | regression | 20640 | 8 | 0 | - |
| CPMP-2015 | CM | regression | 2108 | 22 | 2 | - |
| cpu_act | CP | regression | 8192 | 21 | 0 | - |
| electricity | EL | classification | 45312 | 7 | 1 | 2 |
| eye_movements | EY | classification | 10936 | 24 | 3 | 3 |
| Food_Delivery_Time | FO | regression | 45593 | 6 | 2 | - |
| Helena | HE | classification | 65196 | 27 | 0 | 100 |
| Higgs Small | HI | classification | 98050 | 28 | 0 | 2 |
| houses | HOU | regression | 20640 | 8 | 0 | - |
| House_16H | HO | regression | 22784 | 16 | 0 | - |
| Jannis | JA | classification | 83733 | 54 | 0 | 4 |
| bike_sharing_demand_challange | BI | regression | 10886 | 3 | 6 | - |
| KDDCup09_upselling | CU | classification | 5128 | 34 | 15 | 2 |
| kdd_ipums_la_97-small | IP | classification | 5188 | 20 | 0 | 2 |
| Laptop_Prices_Dataset | LA | regression | 4441 | 9 | 0 | - |
| MagicTelescope | MA | classification | 19020 | 9 | 0 | 2 |
| MiamiHousing2016 | MIA | regression | 13932 | 16 | 0 | - |
| Otto-Group-Products | OT | classification | 61878 | 93 | 0 | 9 |
| phoneme | PH | classification | 5404 | 5 | 0 | 2 |
| wine | WI | classification | 2554 | 4 | 0 | 2 |

We randomly sample 20% instances to construct the test set. The remaining 80% instances are used for splitting. In the training set, we randomly hold out 20% of instances as the validation set.
The validation set is used for tuning hyper-parameters and carrying out early stopping.
We selected the model that performed best on the validation set with tuned hyper-parameters for evaluation.

### A.3 Hardware

The majority of experiments were conducted on the Tesla V100 GPU, including those related to time and memory overhead calculation.

### A.4 Implementations of TabCon

TabCon is trained using Supervised Contrastive loss in a two-stage process. In the training stage, a mapping $\phi$, defined the same as ModernNCA for simplicity, is trained to project the original features into a latent space as described in Equation 7. Before calculating the Supervised Contrastive loss, each mapping $\phi({\bm{x}})$ is normalized to unit length. In the validation stage, a simple predictor (*e.g*., KNN and LR) is further trained based on the mappings learned in the training stage. Early stopping is applied during the validation stage based on the performance of the simple predictor. We use KNN with Euclidean distance as the default predictor.

Since Supervised Contrastive loss was originally proposed for classification, we discretize the labels for regression tasks before calculating the loss. Specifically, the samples are divided into different bins according to their values, and we use quantiles as the boundaries of the bins to ensure that each bin contains a similar number of samples. The number of bins is set as a hyper-parameter.

## Appendix B Additional Experiments

### B.1 Additional Ablation Studies

The Influence of PLR Embedding.
The results of ModernNCA in Table 1 and Table 2 leverage the PLR embedding for numerical features [] (where all categorical features are processed in the one-hot form). We compare ModernNCA as well as MLP with or without PLR in Table 6.

The results demonstrate that our ModernNCA can also utilize the advantages of PLR to achieve further improvements.
However, different from MLP, the vanilla variant of ModernNCA without PLR is still competitive. For example, in regression dataset CA, ModernNCA without PLR can outperform MLP as well as MLPPLR. The results validate that the ability of ModernNCA comes from its objective, architecture, and training strategy, not mainly from the PLR encoding strategy.

**Table 6: Results of MLP and our ModernNCA with or without the PLR encoding on numerical attributes.**
|  | MLP w/o PLR | MLP w/ PLR | M-NCA w/o PLR | M-NCA w/ PLR |
| --- | --- | --- | --- | --- |
| CA $\downarrow$ | .5074 | .4690 | .4266 | .4212 |
| AD $\uparrow$ | 85.80 | 86.97 | 86.77 | 87.29 |
| MIA$\times 10^{5}$$\downarrow$ | .8973 | .8562 | .8759 | .8284 |
| PH $\uparrow$ | 86.88 | 87.13 | 88.41 | 89.29 |

Other Implementations of TabCon.
As mentioned in subsection A.4, there are different choices for the predictors constructed based on the learned embeddings $\phi({\bm{x}})$ using Supervised Contrastive loss. We consider two variants here: the first is KNN, and the second is Logistic Regression for classification and Linear Regression for regression. We denote these two prediction methods as TabCon and TabCon (LR), respectively. We compare the results of these two variants in Table 7. The results show that while different methods perform variably across datasets, their overall performance is comparable. Given that KNN has a lower training overhead, we use it as the default configuration.

**Table 7: Results of TabCon variants with different ways to make predictions. The default choice of TabCon utilizes KNN based on the learned embeddings.**
|  | CA$\downarrow$ | AD$\uparrow$ | MIA$\times 10^{5}$$\downarrow$ | PH$\uparrow$ |
| --- | --- | --- | --- | --- |
| TabCon | .4581 | 86.97 | .9322 | 87.88 |
| TabCon (LR) | .4724 | 87.09 | .9017 | 86.69 |

Visualization Results.
To better analyze the properties of ModernNCA, we visualize the learned embeddings $\phi({\bm{x}})$ of ModernNCA, TabCon, and TabR using TSNE []. As shown in Figure 5, all deep tabular methods transform the embedding spaces to be more helpful for classification or regression compared to the raw features.
The embedding space learned by TabCon clusters samples of the same class together and separates samples of different classes, often clustering same-class instances into a single cluster. However, it still struggles with some hard-to-distinguish samples.
TabR and ModernNCA, on the other hand, divide samples of the same class into multiple clusters, ensuring that similar samples are positioned closer to each other. This strategy aligns with the prediction mechanism of KNN, where good performance is achieved by clustering instances with similar neighbors together rather than into a single cluster.
The embedding space learned by ModernNCA is more discriminative than that learned by TabR. The main reason is that TabR leverages a transformer-like architecture to modify the embedding for each instance before making predictions, making the learned embedding space less discriminative compared to ModernNCA.

### B.2 Run-time and Memory Usage Estimation

We make a run-time and memory usage comparison in  Figure 1. Here are the steps that we take to perform the estimation. First, we tuned all models on the validation set for 100 iterations, saving the optimal parameters ever found. Next, we ran the models for 15 iterations with the tuned parameters and saved the best checkpoint on the validation set. The run-time for the models was estimated using the average time taken by the tuned model to run one seed in the training and validation stage. The model size was estimated using the size of the saved checkpoint. Here we present the average results of run-time and memory usage estimation across our benchmark datasets in Table 8.

**Table 8: Training time and memory usage estimation for different tuned models on the benchmark datasets. The average rank represents the mean performance ranking of these models based on the performance metrics (RMSE for regression and accuracy for classification).**
| Model | M-NCA | TabCon | MLP | MLPPLR | FT-T | TabR | XGBoost | CatBoost |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Training Time (s) | 87.5 | 53.16 | 30.36 | 38.94 | 111.91 | 173.34 | 4.53 | 20.48 |
| Size (MB) | 3.36 | 8.54 | 2.92 | 9.53 | 3.19 | 9.13 | 7.44 | 6.92 |
| Average Rank | 2.41 | 5.80 | 7.63 | 4.83 | 5.32 | 3.37 | 3.62 | 3.02 |

Figure: Figure 5: Visualization of the embedding space of different methods.
Refer to caption: x10.png

### B.3 Full Results with Standard Deviations

In this section, we present the whole result with standard deviation for the datasets described in Table 5. The results for the classification tasks are reported in Table 9. The results for regression tasks are shown in Table 10.

**Table 9: Test accuracy and standard deviations for 12 classification datasets across 15 seeds. The best results are highlighted in bold.**
| $\uparrow$ | M-NCA | TabCon | XGBoost | CatBoost | MLP | FT-T | TabR | KNN | MLPPLR | PTaRL | LMNN | NCA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AD | 87.29 | 86.97 | 87.21 | 87.48 | 85.80 | 85.89 | 86.86 | 84.61 | 86.97 | 85.53 | 83.21 | 83.87 |
|  | $\pm$ 0.07 | $\pm$ 0.11 | $\pm$ 0.05 | $\pm$ 0.06 | $\pm$ 0.14 | $\pm$ 0.16 | $\pm$ 0.10 | $\pm$ 0.00 | $\pm$ 0.12 | $\pm$ 0.21 | $\pm$ 0.00 | $\pm$ 0.00 |
| CU | 79.99 | 77.72 | 81.73 | 81.03 | 75.54 | 80.65 | 79.61 | 70.08 | 80.75 | 75.89 | 72.12 | 73.78 |
|  | $\pm$ 0.55 | $\pm$ 1.85 | $\pm$ 0.17 | $\pm$ 0.38 | $\pm$ 0.57 | $\pm$ 0.61 | $\pm$ 1.18 | $\pm$ 0.00 | $\pm$ 0.67 | $\pm$ 0.62 | $\pm$ 0.00 | $\pm$ 0.00 |
| EL | 96.12 | 87.44 | 92.06 | 91.69 | 85.49 | 87.50 | 96.46 | 84.27 | 86.93 | 85.10 | 83.56 | 85.15 |
|  | $\pm$ 0.11 | $\pm$ 0.23 | $\pm$ 0.21 | $\pm$ 0.18 | $\pm$ 0.28 | $\pm$ 0.25 | $\pm$ 0.12 | $\pm$ 0.00 | $\pm$ 0.61 | $\pm$ 0.39 | $\pm$ 0.00 | $\pm$ 0.00 |
| EY | 99.41 | 82.88 | 71.02 | 71.88 | 61.29 | 71.49 | 98.20 | 59.46 | 74.27 | 60.13 | 54.98 | 66.50 |
|  | $\pm$ 0.33 | $\pm$ 9.49 | $\pm$ 0.42 | $\pm$ 0.34 | $\pm$ 0.98 | $\pm$ 0.85 | $\pm$ 0.54 | $\pm$ 0.00 | $\pm$ 2.28 | $\pm$ 1.20 | $\pm$ 0.00 | $\pm$ 0.00 |
| HE | 39.84 | 38.32 | 37.86 | 38.26 | 38.34 | 38.49 | 40.80 | 32.91 | 38.95 | 37.64 | 29.36 | 31.30 |
|  | $\pm$ 0.24 | $\pm$ 0.23 | $\pm$ 0.09 | $\pm$ 0.11 | $\pm$ 0.25 | $\pm$ 0.22 | $\pm$ 0.18 | $\pm$ 0.00 | $\pm$ 0.12 | $\pm$ 0.27 | $\pm$ 0.00 | $\pm$ 0.00 |
| HI | 73.09 | 72.36 | 72.73 | 72.66 | 72.29 | 72.92 | 72.85 | 66.78 | 72.64 | 72.48 | 61.44 | 66.94 |
|  | $\pm$ 0.13 | $\pm$ 0.17 | $\pm$ 0.08 | $\pm$ 0.11 | $\pm$ 0.12 | $\pm$ 0.17 | $\pm$ 0.13 | $\pm$ 0.00 | $\pm$ 0.21 | $\pm$ 0.14 | $\pm$ 0.00 | $\pm$ 0.00 |
| IP | 87.96 | 87.65 | 88.64 | 87.74 | 84.86 | 84.95 | 87.23 | 83.72 | 87.84 | 84.83 | 83.81 | 84.10 |
|  | $\pm$ 0.28 | $\pm$ 0.56 | $\pm$ 0.20 | $\pm$ 0.08 | $\pm$ 0.42 | $\pm$ 0.34 | $\pm$ 0.76 | $\pm$ 0.00 | $\pm$ 0.33 | $\pm$ 0.28 | $\pm$ 0.00 | $\pm$ 0.00 |
| JA | 74.08 | 72.17 | 72.55 | 72.39 | 71.97 | 72.38 | 73.44 | 65.67 | 72.10 | 71.52 | 59.12 | 64.64 |
|  | $\pm$ 0.14 | $\pm$ 0.37 | $\pm$ 0.08 | $\pm$ 0.09 | $\pm$ 0.17 | $\pm$ 0.16 | $\pm$ 0.22 | $\pm$ 0.00 | $\pm$ 0.19 | $\pm$ 0.27 | $\pm$ 0.00 | $\pm$ 0.00 |
| MA | 87.90 | 87.22 | 87.67 | 87.87 | 87.08 | 87.73 | 87.91 | 84.65 | 87.17 | 86.94 | 83.68 | 84.73 |
|  | $\pm$ 0.24 | $\pm$ 0.22 | $\pm$ 0.22 | $\pm$ 0.17 | $\pm$ 0.32 | $\pm$ 0.28 | $\pm$ 0.22 | $\pm$ 0.00 | $\pm$ 0.22 | $\pm$ 0.17 | $\pm$ 0.00 | $\pm$ 0.00 |
| OT | 82.37 | 78.94 | 82.45 | 82.21 | 81.47 | 80.82 | 82.38 | 78.15 | 81.24 | 81.20 | 77.37 | 78.85 |
|  | $\pm$ 0.14 | $\pm$ 0.49 | $\pm$ 0.12 | $\pm$ 0.12 | $\pm$ 0.15 | $\pm$ 0.28 | $\pm$ 0.14 | $\pm$ 0.00 | $\pm$ 0.14 | $\pm$ 0.21 | $\pm$ 0.00 | $\pm$ 0.00 |
| PH | 89.29 | 87.88 | 87.66 | 87.97 | 86.88 | 87.86 | 89.10 | 86.86 | 87.13 | 85.54 | 85.75 | 87.05 |
|  | $\pm$ 0.63 | $\pm$ 0.45 | $\pm$ 0.53 | $\pm$ 0.49 | $\pm$ 0.72 | $\pm$ 0.40 | $\pm$ 0.49 | $\pm$ 0.00 | $\pm$ 0.49 | $\pm$ 0.63 | $\pm$ 0.00 | $\pm$ 0.00 |
| WI | 74.74 | 73.72 | 74.39 | 74.59 | 72.50 | 72.64 | 74.22 | 77.10 | 72.50 | 72.16 | 73.58 | 72.60 |
|  | $\pm$ 1.55 | $\pm$ 1.55 | $\pm$ 0.85 | $\pm$ 0.91 | $\pm$ 0.50 | $\pm$ 0.80 | $\pm$ 1.29 | $\pm$ 0.00 | $\pm$ 1.35 | $\pm$ 0.82 | $\pm$ 0.00 | $\pm$ 0.00 |
| rank | 2.000 | 5.917 | 3.833 | 3.750 | 8.000 | 5.417 | 3.083 | 10.000 | 5.667 | 9.250 | 11.333 | 9.750 |

**Table 10: Test RMSE and standard deviations for 10 regression datasets across 15 seeds. The best results are highlighted in bold.**
| $\downarrow$ | M-NCA | TabCon | XGBoost | CatBoost | MLP | FT-T | TabR | KNN | MLPPLR | PTaRL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AI$\times 10^{-3}$ | .1532 | .1522 | .1527 | .1465 | .1558 | .1565 | .1546 | .2431 | .1522 | .1563 |
|  | $\pm$ .001 | $\pm$ .002 | $\pm$ .001 | $\pm$ 0.00 | $\pm$ .002 | $\pm$ .003 | $\pm$ .002 | $\pm$ 0.00 | $\pm$ .001 | $\pm$ .002 |
| BI$\times 10^{2}$ | .7057 | .7352 | .7180 | .7264 | .7773 | .7466 | .6657 | 1.087 | .7155 | .7193 |
|  | $\pm$ .007 | $\pm$ .011 | $\pm$ .006 | $\pm$ .007 | $\pm$ .007 | $\pm$ .011 | $\pm$ .006 | $\pm$ 0.00 | $\pm$ .003 | $\pm$ .005 |
| CA | .4212 | .4581 | .4328 | .4360 | .5074 | .4701 | .4157 | .5843 | .4690 | .5086 |
|  | $\pm$ .005 | $\pm$ .007 | $\pm$ .002 | $\pm$ .005 | $\pm$ .005 | $\pm$ .004 | $\pm$ .005 | $\pm$ 0.00 | $\pm$ .005 | $\pm$ .010 |
| CM$\times 10^{3}$ | .4618 | .5011 | .4809 | .4929 | .5131 | .5187 | .5091 | .8754 | .4655 | .5211 |
|  | $\pm$ .010 | $\pm$ .012 | $\pm$ .004 | $\pm$ .013 | $\pm$ .006 | $\pm$ .015 | $\pm$ .023 | $\pm$ 0.00 | $\pm$ .020 | $\pm$ .004 |
| CP$\times 10$ | .2381 | .2473 | .2404 | .2365 | .2490 | .2310 | .2337 | .8825 | .2468 | .2487 |
|  | $\pm$ .003 | $\pm$ .003 | $\pm$ .007 | $\pm$ .004 | $\pm$ .004 | $\pm$ .003 | $\pm$ .003 | $\pm$ 0.00 | $\pm$ .006 | $\pm$ .003 |
| FO$\times 10$ | .7272 | .7436 | .7384 | .7393 | .7900 | .7885 | .7398 | .8140 | .7428 | .7896 |
|  | $\pm$ .001 | $\pm$ .001 | $\pm$ 0.00 | $\pm$ .001 | $\pm$ .001 | $\pm$ .001 | $\pm$ .013 | $\pm$ 0.00 | $\pm$ .001 | $\pm$ .001 |
| HO$\times 10^{5}$ | .3041 | .3156 | .3026 | .3051 | .3161 | .3074 | .3133 | .3663 | .3093 | .3149 |
|  | $\pm$ .004 | $\pm$ .003 | $\pm$ .001 | $\pm$ .002 | $\pm$ .004 | $\pm$ .005 | $\pm$ .006 | $\pm$ 0.00 | $\pm$ .003 | $\pm$ .003 |
| HOU$\times 10^{5}$ | .4291 | .4963 | .4797 | .4546 | .5221 | .4791 | .4149 | .6889 | .4921 | .5118 |
|  | $\pm$ .002 | $\pm$ .020 | $\pm$ .008 | $\pm$ .002 | $\pm$ .009 | $\pm$ .005 | $\pm$ .005 | $\pm$ 0.00 | $\pm$ .004 | $\pm$ .004 |
| LA$\times 10^{3}$ | .4135 | .4529 | .4448 | .4506 | .4789 | .4609 | .4606 | .4968 | .4753 | .4777 |
|  | $\pm$ .014 | $\pm$ .007 | $\pm$ .003 | $\pm$ .003 | $\pm$ .005 | $\pm$ .005 | $\pm$ .007 | $\pm$ 0.00 | $\pm$ .013 | $\pm$ .005 |
| MIA$\times 10^{5}$ | .8284 | .9322 | .8740 | .8034 | .8973 | .8717 | .8585 | 1.508 | .8562 | .9302 |
|  | $\pm$ .017 | $\pm$ .014 | $\pm$ .016 | $\pm$ .008 | $\pm$ .011 | $\pm$ .023 | $\pm$ .012 | $\pm$ 0.00 | $\pm$ .012 | $\pm$ .017 |
| rank | 2.200 | 6.100 | 3.500 | 3.100 | 8.300 | 5.900 | 3.600 | 10.000 | 4.500 | 7.800 |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: -
soundness: 3
strengths: - The method is simple.
- Generally, I tend to agree that nearest neighbors may be underexplored in the context of tabular data. While TabR seems to close this gap to some extent, ModernNCA looks like a good addition to the field.
- The SNS strategy looks simple and effective, and also differentiates the method from TabR.
- On the considered benchmark, the proposed ModernNCA achieves a better average rank and a better balance of task performance and training time compared to baselines.
- A large number of baselines and datasets.
- An ablation study is provided.
summary: This study considers learning on tabular data, and proposes ModernNCA -- a deep version of the classic Neighborhood Components Analysis algorithm. Contrary to NCA, the transformation in ModernNCA is non-linear, and is powered by a neural network. To make the training of ModernNCA more efficient and more effective, the paper also proposes Stochastic Neighborhood Sampling (SNS). In experiments on 300 datasets, ModernNCA is reported to achieve the best average rank among the considered baselines, including gradient-boosted decision trees (GBDT).
weaknesses: Note: regarding the "datasets" and "metrics" weaknesses, I admit that the field lacks standardized benchmarks and metrics.

**Datasets**

My understanding is that the benchmark consists of many automatically collected datasets. In the light of the recent studies about tabular datasets, it is unclear how representative the benchmark is. Examples of the studies:

- Towards quantifying the effect of datasets for benchmarking: A look at tabular machine learning
- A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data
- TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks

**Metrics**

The metrics such as ranks or wins do not show the scale of performance gaps between methods. It is unclear how significant are the wins and losses of ModernNCA (not from statistical perspective, but from the practical perspective).

**Presentation**

- In my opinion, the presentation could be more efficient. The proposed NCA extensions are not conceptually novel, so I believe that the story on the first six pages could be more compact. Perhaps, some of the details and discussion can be moved to appendix.
- Based on my understanding of the TabR baseline
    - The explanation of TabR on L152 is not correct, since it is not a Transformer variant. Quoting the TabR paper: *"a feed-forward network with a custom k-Nearest-Neighbors-like component in the middle"*.
    - The description of TabR on L216-L226 is: (1) not complete, (2) not correct, and, if I am not mistaken, (3) not used in the story. In the light of (3), I do not go into details about (1) and (2). Perhaps, this description can be simply removed?
- Generally, communicating the empirical nature of a study as in L108-L113 is fine. However, personally, I would change the first sentence on L108 to something more neutral.
- Perhaps, Figure 1 can be placed closer to the related experiments, but this can be subjective.

**Related work.**

There is a missing related work: "Improving Generalization via Scalable Neighborhood Component Analysis" ECCV 2018. That paper also describes how to efficiently train a deep NCA, and I think their method is more advanced than the one proposed in this submission. Though their method can be too complicated for the scope of this paper. In that case, I recommend discussing this related work and explaining why the proposed SNS is a better choice for this work compared to the method from the referenced paper.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Could you provide or point to the raw per-dataset metrics?
- Is it possible to add some other means of comparison along with average ranks (e.g. relative improvement to a baseline)? What does it show? 
- What are the methods limitations?
soundness: 3
strengths: - The proposed method is well motivated. Attention to simple KNN-based methods in deep tabular models was limited, except TabR which is well addressed in the text.
- The method is both conceptually simple and easy to implement, without sacrificing performance
- The experimental results and ablations are extensive and insightful:
  - The step-by-step Linear-NCA ablation (table 2) is a principled and convincing way to explore a model design space
  - Stochastic Neighborhood Sampling ablation shows an interesting result (improved performance from sampling) and provides a practically important outcome for retrieval-based tabular NNs
  - Other important minor details (like the numerical feature embedding ablations), slight improvements in the distance function used, loss functions
- The writing and overall storytelling is engaging and well thought-out
summary: The paper proposes a revised take of the NCA algorithm for supervised learning on tabular data, where the neighborhood aggregation is done in a representaion space of a neural network, the model is optimized via SGD and additional stochasticity is introduced in subsetting the neighbors list. 

The resulting architecture is conceptually simpler than prior state-of-the-art tabular retrieval models, while improving in performance and eficiency as shown via an extensive experimental evaluation.
weaknesses: - All experimental results and ablations rely on the average ranks of a set of methods being tested. This provides some signal for which modificaitons are usefull, but other means of comparison might make this even clearer. For example, additinal relative improvement compared to a strong baseline (e.g. a well-tuned MLP) would provide usefull additional signal besides the average rank metric (e.g. what is the scale of such improvements). 
- Minor additional to the previous point: I find that providing raw unaggregated results for the core (or even for all) the experiments is very usefull for quick sanity-checks and comparisons in future work. So that others could assess results on the individual datasets by consulting the paper text.
- I believe limitation should be discussed somewhere in the main text. E.g. what are the confines of the proposed mehtod. Are there any cases where it may perform poor. 
   - Maybe some post-hock meta-analysis akin to the one in tabzilla paper (https://arxiv.org/abs/2305.02997) of what are the datasets where ModernNCA performs worse. 
   - Some recent benchmarks demonstrate that retrieval-based models might not be universaly superior (https://arxiv.org/abs/2406.19380)

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: If my understanding is incorrect, could you please clarify what is the novel concept introduced in this paper?
soundness: 2
strengths: - The approach effectively leverages modern deep learning techniques to enhance classical NCA, demonstrating strong empirical results across a large number of datasets.

- The paper provides comprehensive benchmarks, including comparisons with state-of-the-art methods in both classification and regression tasks.
summary: The paper revisits the Neighborhood Components Analysis (NCA) and adapts it for tabular data learning, proposing ModernNCA as an enhanced approach. The modifications include (1) calculating distances in a representation space, (2) using stochastic gradient descent (SGD) instead of L-BFGS for optimization, and (3) training in a mini-batch fashion rather than on the entire dataset at once. The authors benchmarked ModernNCA against numerous methods across 300 datasets, finding that it achieved consistently superior performance, often comparable to leading models like CatBoost and outperforming many deep tabular learning methods.
weaknesses: **Lack of Novelty**: While the paper shows strong empirical performance, the core modifications (using a representation space for distance calculations, employing SGD, and mini-batch training) have already been explored in prior research. This raises concerns regarding the originality of the contribution. The changes appear more like tunings of established techniques rather than introducing a fundamentally new method.

[Prior research example] J Kang et al., Deep metric learning based on scalable neighborhood components for remote sensing scene characterization, 2020.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - MODERNNCA relies on Euclidean distance as the default metric. How adaptable is the model to other distance metrics, such as cosine similarity, and do the authors have insights on how these choices might impact performance?

- The paper discusses various deep learning modifications, but how sensitive is MODERNNCA to hyperparameter choices such as learning rate, number of neighbors (K), and embedding dimensions? Are there specific configurations where performance significantly degrades?
soundness: 2
strengths: **Originality**
Revisiting a classic nearest-neighbor approach with contemporary deep learning techniques is a novel approach, particularly since NCA had been previously limited by computational efficiency and scalability. This approach aims to unify insights from both traditional and modern tabular prediction methods.

**Quality**
The authors conducted extensive experiments across 300 datasets, providing thorough evidence of the model’s strengths and weaknesses. 

**Clarity**
The explanation of the modifications, including SGD, nonlinear embeddings, and SNS, is detailed and clear, making the improvements accessible to the reader.
summary: This paper explores the potential of modernizing the classical Nearest Neighbor approach for tabular data by leveraging a differentiable K-nearest neighbors variant, Neighborhood Components Analysis (NCA). The authors introduce MODERNNCA, an improved version of NCA that integrates deep learning techniques such as stochastic gradient descent (SGD), nonlinear embeddings, and a Stochastic Neighborhood Sampling (SNS) strategy to boost computational efficiency and model performance. They demonstrate that MODERNNCA matches or outperforms both tree-based models and current deep tabular models across 300 datasets in classification and regression tasks.
weaknesses: **Marginal Contribution**  
   The paper’s contribution feels incremental rather than pioneering. The improvements in MODERNNCA rely on established techniques (SGD, SNS, and nonlinear embeddings) without introducing a fundamentally new concept or method. This makes the novelty limited, as it essentially optimizes an existing algorithm rather than providing a unique advancement. 

**Limited Novelty in Comparison to KNN Variants**  
   The paper lacks direct comparisons with other KNN-inspired deep learning methods that have similarly benefited from modern optimization strategies. This limited scope of comparison weakens the argument for MODERNNCA’s distinctiveness and impact.

**Lack of Theoretical Analysis**  
   The paper’s focus is primarily empirical, with little theoretical exploration of why the proposed enhancements lead to improved performance. A deeper theoretical perspective on the modifications—such as the effect of stochastic sampling on generalization—would provide valuable insights and strengthen the work's academic contribution.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How does modernnca perform with different distance metrics, such as cosine similarity, especially for datasets with high-dimensional features?


2. Given that modernnca outperformed many models, have you tested its performance on imbalanced datasets?

3. Could you provide more insights into cases where modernnca significantly underperforms compared to catboost or other tree-based methods?
soundness: 3
strengths: 1. the authors structure their explanation to make complex methods like nn and nca accessible and coherent. this clarity is helpful for understanding both the motivation and methodology behind the modifications to nca.

2. the paper includes an extensive evaluation on a broad range of datasets, demonstrating modernnca’s effectiveness with detailed performance metrics and statistical significance tests.


3.  The authors do a great job explaining the steps taken to modernize nca
summary: This paper re-evaluates the nearest neighbor approach for tabular data, proposing a modernized version of neighborhood component analysis (nca) called modernnca. starting from classical nn techniques, the authors incorporate current deep learning methods into nca, such as using stochastic gradient descent and adding stochastic neighborhood sampling. the experiments on 300 tabular datasets demonstrate that modernnca performs comparably to leading models like catboost, outperforming other deep learning approaches for classification and regression tasks. the authors also provide insights into how modern techniques like batch normalization and non-linear architectures improve nn performance.
weaknesses: W1: while the model shows high performance, the paper lacks specific scenarios or guidance on applying modernnca practically, such as when dealing with imbalanced datasets

Meta-review: This paper explores more traditional approaches for supervised tabular data modelling integrated with modern deep learning techniques. It uses a Neighbourhood Components Analysis approach in the latent space of a neural network which can be optimized with SGD.

Reviewers brought up many concerns, but most of them were addressed during the discussion. The primary outstanding concern from several reviewers, but not all, was around novelty, in that this work revives old techniques through a modern lens. In my view this approach is sufficiently novel, at least compared to the typical paper accepted at ICLR, and considering the positive empirical results I am recommending acceptance as a Poster.

Several reviewers asked about the limitations of this method, and cases where it may underperform. The authors provided some useful thoughts, which should be featured in the main text of the paper so that readers are properly informed. I request that the authors make this change for the camera-ready version.

Decision: Accept (Poster)