Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Later
Authors: Han-Jia Ye
Huai-Hong Yin
De-Chuan Zhan
Wei-Lun Chao
Venue: ICLR 2025 Poster (2025)
Keywords: Tabular data
tabular machine learning
deep tabular models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: The widespread enthusiasm for deep learning has recently expanded into the domain of tabular data. Recognizing that the advancement in deep tabular methods is often inspired by classical methods, e.g., integration of nearest neighbors into neural networks, we investigate whether these classical methods can be revitalized with modern techniques.
We revisit a differentiable version of $K$-nearest neighbors (KNN) --- Neighbourhood Components Analysis (NCA) --- originally designed to learn a linear projection to capture semantic similarities between instances, and seek to gradually add modern deep learning techniques on top. Surprisingly, our implementation of NCA using SGD and without dimensionality reduction already achieves decent performance on tabular data, in contrast to the results of using existing toolboxes like scikit-learn. Further equipping NCA with deep representations and additional training stochasticity significantly enhances its capability, being on par with the leading tree-based method CatBoost and outperforming existing deep tabular models in both classification and regression tasks on 300 datasets. We conclude our paper by analyzing the factors behind these improvements, including loss functions, prediction strategies, and deep architectures.
The code is available at https://github.com/LAMDA-Tabular/TALENT.

Published as a conference paper at ICLR 2025
REVISITING NEAREST NEIGHBOR FOR TABULAR DATA: A DEEP TABULAR BASELINE TWo DECADES LATER
Han-Jia Ye Huai-Hong Yin 1, De-Chuan Zhan 1.2 & Wei-Lun Chao
Schooo of Artificial Intelligence, Nanjing University National Key Laboratory for Novel Software Technology, Nanjing University The Ohio State University (yeh), yinhh, zhande) ) @ @lamda.... edu. chao 20w.osu..uuuc.omm edu
ABSTRACT
The widespread enthusiasm for deep learning has recently expanded into the do- main of tabular data. Recognizing that the advancement in deep tabular methods is often inspired by classical methods, e,g,, integration of nearest neighbors into neural networks, we investigate whether these classical methods can be revitalized with modern techniques. We revisit a differentiable version of K -nearestt neighbors (KNN) Neighbourhood Components Analysis (NCA) originally designed to learn a linear projection to capture semantic similarities between instances, and seek to gradually add modern deep learning techniques top. Surprisingly, our implementation of NCA using SGD and without dimensionality reduction already tabular data, in contrast to the results of using ex-
achieves decent performance isting toolboxes like scikit-learn. Further equipping NCA with deep representations and additional training stochasticity significantly enhances its capability, being on par with the leading tree-based method CatBoost and outperforming existing deep tabular models in both classification and regression tasks on 300 datasets. We conclude paper by analyzing the factors behind these improvements, including loss functions, prediction strategies, and deep architectures. The code is available at htttpps///ittt hub. com/ LAMDA Tabular. ar TALENT
INTRODUCTION
Tabular data, characterized by its structured format of rows and columns representing individual examples and features, is prevalent in domains like healtheare (Hassan et al. 2020) and e-commerce (Nederstigt et al. 2014). Motivated by the success of deep neural networks in fields like computer
vision and natural language processing (Simonyan & Zisserman 2015 Vaswani et al. 2017 Devlin
eet al. 2019), numerous deep models have been developed for tabular data to capture complex feature
interactions Cheng et al. 2016: Guo et al. 2017: Popov et al. 2020 Arik & Prister 2021 Gorishniy
ct al. 2021 1
Katzir et al.
2021 Chang et al. 2022 Chen et al. 2022 Hollmann et al. 2023).
Despite all these attempts, deep tabular models still struggle match the accuracy of traditional machine learning methods like Gradient Boosting Decision Trees (GBDT) (Prokhorenkova et al.
2018 Chen & Guestrin (016) on tabular tasks. Such a fact raises our interest: to excel in tabular tasks, perhaps deep methods could draw inspiration from traditional methods. Indeed, several deep tabular methods have demonstrated promising results along this route. Gorishniy et al. (202T): )
Kadra et al.) (2021) consulted classical tabular techniques to design specific MLP architectures and weight regularization strategies, significantly boosting MLPs' accuracy on tabular datasets. Recently, inspired by can-aaramttrc methods (Mohri et al.) 2012), TabR (Gorishniy et al. 2024) retrieves neighbors from the entire training set and constructs instance-specifie scores with a Transformer architecture, leveraging relationships between instances for tabular predictions.
We follow this route but from a different direction. Instead of incorporating classic techniques into the already complex deep models, perform an Occam's ss-azoor-stlle exploration starting from the classic method and gradually increasing its complexity by adding modern deep techniques. We hope such exploration could reveal the key components from both worlds to excel in tabular tasks.
To this end, we build upon TabR (Gorishniy et al., 2024] and choose to start from a classical, differentiable version of K-nearest neighbors (KNN) named Neighbourhood Component Analysis

---

Published as a conference paper at ICLR 2025
200
Tab
150
TabR
160
120
Time
Time
90
80
MOnFRNNCA
(Ours)
FT-T
MLP-PLR
60
MoDeRNNCA
(Ours)
FT-T
MLP-PLR
MLP
L-NCA
20
Optimll
CatBoust
XGBoost
L-NCA (Ours)
of
Optimal
CalBoost
0
(Ours)
O MLP
xGooss
KNN
5
Average rank
Average rank
(a) Classification
(b) Regression
comparison hetween MODERNNCA and existing methods on classification (a) and regression (h) datasets. Representative tabular prediction methods, including the classical methods (in green), the parumetric deep methods (in blue), and the won-aarrmmettcceeeghooooodddd bossedddddddddddddoooooee deep methods (in red), are investigated. based on their records over 300 datasets in Table I] Figure 2] The average rank among these eight methods is used as the performance measure. We calculate the average training time (in seconds) and the memory usage of the model (denoted by the radius of the circles, where the larger the circle, the bigger the model). MODERNNCA achieves high training speed compared to other deep tabular models and has a relatively lower memory usage. L-NCA is our improved linear version of NCA.
Figure 1: Fitermancc--ffiiiieee---e---
(NCA) (Goldberger et al., 2004). NCA optimizes the KNN prediction accuracy of a target instance by learning linear projection, ensuring that semantically similar instances closer than dissimilar ones. Its differentiable nature makes it a suitable backbone for adding deep learning modules.
Our first attempt is to re-implement NCA, using deep learning libraries like PyTorch (Paszke et al.) 2019). Interestingly, by replacing the default L-BFGS optimizer (Liu & Nocedal 1989) in scikit- learn Pedregosa et al. 2011 with stochastic gradient descent (SGD), already witnessed notable accuracy boost on tabular tasks. Further enabling NCA to learn a linear projection into a larger dimensionality (hence not dimensionality reduction) and use soft nearest neighbor inference rule (Salakhutdinov & Hinton 2007 Frosst et al.) (2019) bring another gain, making NCA on par with deep methods like MLP. (See (See|section) |section 6 for detailed ablation studies and discussions.)
Our second attempt is to replace the linear projection with a neural network for nonlinear embeddings. As NCA's objective function involves the relationship of instance to all the other training instances, a naive implementation would incur a huge computational burden. We thus employ a stochastic neighborhood sampling (SNS) strategy, randomly selecting subset of training data as candidate neighbors in each mini-batch. We show that SNS not only improves training efficiency but enhances the model's generalizability, as it introduces additional stochasticity (beyond SGD) in training.
Putting things together, along with the use of pre-defined feature transform on numerical tabular entries (Gorishniy et al., 2022). J our deep NCA implementation, MODERNNCA. achieves remarkably encouraging empirical results. Evaluated on 300 tabular datasets, MODERNNCA is ranked first in classification tasks and just shy of CatBoost (Prokhorenkova et al. 2018) in regression tasks while outperforming other based and deep tabular models. Figure I further shows that MODERNNCA well balances training efficiency (with lower training time compared to other deep tabular models), generalizability (with higher average accuracy), and memory efficiency. We also provide detailed ablation study and discussion on MODERNNCA, comparing different loss functions, training and prediction strategies, and deep architectures, aiming to systematically reveal the impacts of deep learning techniques on NCA, after its release in 2004. In sum, our contributions are two-folded:
We revisit the classical nearest neighbor approach NCA and systematically explore ways to improve it using modern deep learning techniques.
Our proposed MODERNNCA achieves outstanding performance in both classification and regres- sion tasks, essentially serving as strong deep baseline for tabular tasks.
Remark. In conducting this study, we become aware of several prior attempts to integrate neural networks into NCA (Salakhutdinov & Hinton 2007 Min al. 2010). However, their results and
We that the original NCA paper Gollderrger et al. 2004 did specify the optimizer.
2

---

Published as a conference paper at ICLR 2025
applicabillity were downplayed by tree-based methods, and attribute this to the less powerful deep-learning techniques two decades ago (e.g., restricted Boltzmann machine). In other words, our work can be viewed as a revisit of these attempts from the lens of modern deep-learning techniques.
While our study is largely empirical, we believe it offers valuable insights. For years, nearest- neighbor-based methods (though with solid theoretical foundations) have been overlooked in tabular data, primarily due to their low competitiveness with tree-based methods. We hope that our thor- ough exploration of deep learning techniques for nearest neighbors and the outcome tabular baseline on par with the leading CatBoost (Prokhorenkova et al., 2018) would revitalize nearest neighbors and open up new research directions, ideally theoretical foundations behind the improvements.
strong
2
RELATED WORK
Learning with Tabular Data. Tabular data is a common format across various applications such as click-through rate prediction (Richardson al. 2007) and time-series forecasting (Ahmed et al. 2010). Tree-based methods like XGBoost (Chen & Guestrin 2016), LightGBM (Ke et al.) 2017). and CatBoost (Prokhorenkova et al. 2018) have proven effective at capturing feature interactions and are widely used in real-world applications. Recognizing the ability of deep neural networks to learn feature representations from raw data and make nonlinear predictions, recent methods have
applied deep learning techniques to tabular models (Cheng et al. 20166] Guo et al.
2017 Popov et al.
2020 Borisov et al. 2022 Arik & Pfister 2021; Kadra et al.) [021] Katzir et al.
2021 Chen et al.
2022 Zhou et al., 2023). For instance, deep architectures such residual networks and transformers
have been adapted for tabular prediction (Gorishniy et al. 202 Hollmann et al. 2023). Moreover,
data augmentation strategies have been introduced mitigate overlttinng in deep models (Ucar et al.,
2021 Bahri et al. 2022] Rubachev et al. 2022). Deep tabular models have demonstrated competitive performance across a wide range of applications. However, researchers have observed that deep models still face challenges in capturing high-order feature interactions as effectively as tree-based
models (Grinsztajn et al. 2022) McElfresh al. 2023 Ye al. 2024a).
NCA Variants. Nearest Neighbor approaches make predictions based on the relationships between an instance and its neighbors the training set. Instead of identifying neighbors using raw features, NCA employs differentiable Nearest Neighbor loss function (also known as soft-NN loss) to learn a linear projection for better distance measurement (Goldberger et al., 2004). Several works have
extended this idea with alternative loss functions (Globerson & Roweis 2005 Tarlow et al.) (013). while others explore NCA variants for data visualization (Venna et al. (010)). A few nonlinear extensions of NCA, developed over decade ago, demonstrated a bit improved performance on image classification tasks using architecture like restricted Boltzmann machines (Salakhutdinov & Hinton| 2007: Min et al. 2010). For visual tasks, the entanglement effects of soft-NN loss deep learned representations have been analyzed (Frosst et al.] 2019). and variants of this loss have been applied to few-shot learning scenarios (Vinyals al. 2016;] Laenen & Bertinetto| 2021]. The effectiveness of NCA variants in fields like image recognition suggests untapped potential (Wu et al., 2018), motivating our revisit of this method with modern deep learning techniques for tabular data.
Metric Learning. NCA is form of metric learning (Xing et al. 2002). where a projection is learned to pull similar instances closer together and push dissimilar ones farther apart, leading to improved classification and regression performance with KNN (Davis et al. 2007 Weinberger & Saul 2009
Kulis 2013: Bellet et al. 2015) Ye et al.) 2020). Initially applied to tabular data, metric learning has evolved into a valuable tool, particularly when integrated with deep learning techniques, across domains like image recognition (Schroff et al. 2015) Sohn 2016, Song et al. 2016 Khosla et al. 2020), person re-identification el. |014 Yang et al. 20118) 1), and recommendation systems (Hsieh) ct al. 2017 Wei et al. 2023). Recently, LocalPFN (Thomas et al.) 2024) incorporates KNN with TabPFN. TabR (Gorishniy et al. 2024) introduced a feed-forward network with a custom attention- like mechanism to retrieve neighbors for each instance, enhancing tabular prediction tasks. Despite its promising results, the high computational cost of neighborhood selection and the complexity of its architecture limit the practicality of TabR. In contrast, our paper revisits NCA and proposes simpler deep tabular baseline that maintains efficient training speeds without sacrificing performance.
3

---

Published as a conference paper at ICLR 2025
3 PRELIMINARY
In this section, we first introduce the task learning with tabular data. We then provide a brief overview of NCA [Goldberger et al. 2004) and TabR (Gorishniy al. 2024).
3.1 LEARNING WITH TABULAR DATA
A labeled tabular dataset is formatted as N examples (rows in the table) and d features/attributes (columns in the table). An instance I; is depicted by its d feature values. There are two kinds of features: the numerical (continuous) ones and categorical (discrete) ones. Given 2ij the j-th feature of instance Li, we use [. E R and æ to denote numerical (e.g., the height of a person) and categorical (e.g., the gender of person) feature values of an instance, respectively. The categorical features are usually transformed in a one-hot manner, i.e., Zij E {0, 1}K,, where the index of value 1 indicates the category among the Kj options. We assume the instance 2; E R w.l.o.g. and will explore other encoding strategies later. Each instance is associated with a label Vs. where y: E [C] ,,, C} in a multi-class classification task and y: E IR in regression task.
Given a tabular dataset ' = {(a:., yi) } we aim to learn model f on 'D that maps æ; to its label V/- We measure the quality of by the joint likelihood over DD iee, max, [I Pr(y: [xx))).
The objective could be reformulated in the form of negative log- ooglliill of the true labels,
min Σ
log Pr(y: f(x:))
Σ (R
Elyi, Di f(x:))
(I)
(EL
or equivalently, the discrepancy between the predicted label ÿ, and the true label yı measured by the loss ((,-),, e.g., cross-entropy. We expect the learned model is able extend its ability unseen instances sampled from the same distribution as D. f could be implemented with classical methods such SVM and tree-based approaches MLPs.
3.2 NEAREST NEIGHIBOR FOR TABULAR DATA
KNN is one of the most representative con-parametric tabular models for classification and regression making predictions based on the labels of the nearest neighbors (Bishop) 20066 Mohri et al. 2012))
In other words, the prediction [a;;; D) of the model conditions on the whole training set. Given an instance æ,, KNN calculates the distance between and other instances in D. Assume the K
nearest neighbors are N(x;; D) {(x1 (<11y1),.... W1)..... (æ 3K) }. then, the label y: of æ; is predicted based on those labels in the neighbor set (æ;;;;;; For classification task ÿ is the majority voting of labels in (x;;;; D) while is the average of those labels in regression tasks.
The distance dist(x;; æ.) in KNN determines the set of nearest neighbors W(x;; D), which one of its key factors. The Euclidean distance between a pair (z,, (a, / V(x; - 22 (æ: 2,). A distance metric that reveals the characteristics of the dataset will improve KNN and lead more accurate predictions (Xing al. 2002 Davis et al.) 2007 Weinberger & Saul 2009 Bellet et al. 2015).
Neighbourhood Component Analysis (NCA). NCA focuses on the classification task (Goldberger et al., 2004). According to the INN rule, NCA defines the probability that æ; locates in the neighborhood of x; by
exp (-iis 4;; L (Ta))) Σ (m..))DD exp dist x;, Tx)))
Pr(æ,, EN(a;;; D) o; D, D)
(2)
Then, the posterior probability that an instance æ; is classified as the class is:
Pr(O: = yi x,, D, =
Σ
Pr(m, E NN(,;; D) x,, D, L).
(3)
(R. (ED
L E dd is linear projection usually with d' < d, which reduces the dimension of the raw input. Therefore, the posterior that an instance x; belongs to the class yi depends on its similarity (measured by the negative squared Fuclidean distance in the space projected by L) between its neighbors from
4

---

Published as a conference paper at ICLR 2025
class Y/ in D. Equation 3 approximates the expected leave-one-out error for 2;. and the original NCA maximizes the sum of Pr(i; = yi x;, L) over all instances in D. Instead of considering all instances in the neighborhood equally, this objective mimics a soft version of KNN, where all instances in the training set are weighted (nearer neighbors have more weight) for the nearest neighbor decision. In the test stage, KNN is applied to classify an unseen instance in the space projected by L.
TabR is a deep tabular method that retrieves the neighbors of an instance 2; using deep neural networks. Specifically, TabR identifies the K nearest neighbors in the embedding space and defines the contribution of each neighbor (æj, Mj) to æ; as follows: a(E;, 2j m3) W yj + T(L E(æ,) - L F..))). Here, T is a transformation composed of a linear layer without bias, dropout, ReLU activation, and another linear layer. E' represents the encoder module for TabR, while W is linear projection and yj is the encoded label vector of y/. The instance-specific scores are then aggregated as: R( Mj. 24) = [(e,,,,)
Ot s(z,, z. Mj), where the weight a, defined as a, x {- dist(( L E(æ,), L' E(x,))} and normalized using softmax function. Finally, R(æ,, V/+ xz,) is added to E(x:), and the result is processed by a prediction module to obtain ÿt. For further details, including instance-level layer normalization, numerical attribute encoding, and the selection strategy for nearest neighbors in the summation, please refer to Gooiihnny et al. (2024).
MODERNNCA
Given the promising results of TabR tabular data, we take the original NCA starting point and gradually enhance its complexity by incorporating modern deep learning techniques. This Occam's s-razor- exploration may allow us to identify the key components that lead to strong performance in tabular tasks, drawing insights from both classical and deep tabular models. In the following, we introduce our proposed MODERNNCA (abbreviated M-NCA) through key attempts to improve upon the original NCA.
4.1 THE FIRST ATTEMPT
We generalize the projection in Equation by introducing a transformation 6, which maps æ. into space with dimensionality d'. To remain consistent with the original NCA, we initially define &i as a linear layer, i.e.. o(x,) Linear(xx)). consisting of linear projection and bias term.
Learning Objective. Assume the label y; is continuous in regression tasks and in one-hot form for classification tasks. We modify |Equation 3 follows:
exp (- dist (p(xx), ((x,)))
0 Σ
(MJDn)ED E pa exp (- diiistt((( ((xx))) yi
This formulation ensures that similar instances (based on their distance in the embedding space mapped by Φ) yield closer predictions. For classification, [Equation 4 generaalizes [qqaation 3 pre- dicting the label of a target instance by computing a weighted average of its neighbors across the C' classes. Here, i E C is a probability vector representing [Pr(i, = c xx, D, ]ejcc- In regression tasks, the prediction is the weighted sum of scalar labels from the neighborhood.
By combining Equation
combining Equation 3 with Equation 1] we define l [Equatin I as negative likelicld for classification and mean square eтror for regression. This classification loss is also known as the soft Nearest Neighbor (soft-NN) loss (Frosst al. 2019 Khosla al. (020) for visual tasks. Different from Goldberger et al.) (2004); Salakhutdinoo & Hinton (2007) that used sum of probability as in the original NCA's loss, we find sum of log probability provides better performance on tabular data.
Prediction Strategy. For test instance, the original NCA projects all instances using the learned øb and applies KNN to classify the test instance based on its neighbors from the entire training set D. Instead of employing the traditional "hard" KNN approach, we adopt the soft-NN rule (Equation 4) to estimate the label posterior, applicable to both classification and regression. Specifically, in the classification case, Eqqutionn 4 produces C-dimensional vector, with the index of the maximum value indicating the predicted class. For regression, i directly corresponds to the predicted value.
Furthermore, we do not limit the mapping to dimensionality reduction. The linear projection o can transform x; into a higher- dimensional space necessary. We also replace the L-BFGS optimizer (used in scikit-learn) with stochastic gradient descent (SGD) for better scalability and performance.
5

---

Published as a conference paper at ICLR 2025
These modifications result in notable accuracy boost for NCA on tabular tasks, making it competitive with deep models like MLP. We refer to this improved version of (linear) NCA as L-NCA.
4.2 THE SECOND ATTEMPT
We further enhance L-NCA by incorporating modern deep learning techniques, leading to our strong deep tabular baseline, MODERNNCA (M-NCA).
Architectures. To introduce nonlinearity into the model, we first enhance the transformation & in subsection 4.1 with multiple nonlinear layers appended. Specifically, we define a one-layer nonlinear
mapping as sequence of operators following Gorishniy et al. (02II) consisting of one-dimensional batch normalization (loffe & Szegedy 2015). y a linear layer, ReLU activation, dropout (Srivastaval
et al. 2014), and another linear layer. In other words, the input a; will be transformed by
g(x;) Linear (Dropout ((ReLU (Linear (BatchNorm (ax))))))
One or more layers of such block can be appended on top of the original linear layer in sub. section to implement the final nonlinear mapping é, which further incorporates an additional batch normalization at the end to calibrate the output embedding. Empirical results show that batch normalization outperforms other normalization strategies, such as layer normalization (Ba et al. 2016), in learning a robust latent embedding space.
For categorical input features, we use one-hot encoding. and for numerical features, we leverage PLR (lite) encoding, following TabR (Gorishmiy et al. 2024). PLR encoding combines periodic embeddings, a linear layer, and ReLU to project instances into a high-dimensional space, thereby increasing the model's capacity with additional nonlinearity (Gorishniy et al., (022). PLR (lite) restricts the linear layer to be shared across all features, balancing complexity and efficiency.
Stochastic Neighborhood Sampling. SGD commonly applied optimize deep neural networks- a mini-batch of instances is sampled, and the average instance-wise loss in the mini-batch is calculated for back proppapppaaa However, the instance- loss based on the predicted label in Equation 4 involves pairwise distances between an instance in the mini-bated and the entire training set D, imposing a significant computational burden.
To accelerate the training speed of MODERNNCA, we propose a Stochastic Neighborhood Sampling (SNS) strategy. In SNS, a subset D of the training set ' is randomly sampled for each miii--btchh. and only distances between instances the mini-batch and this subset are calculated. In other words, D replaces D in Equation 4. and only the labels in are used to predict the label of a given instance during training. During inference, however, the model resumes the scarches for neighbors using the entire training set DD. Unlike deep metric learning methods that only consider pairs of instances within a sampled mini-batch (Schroff et al. 2015 Song et al. 2016 Sohn 2016). i.e., D is the mini-batch, SNS approach retains both efficiency and diversity in the selection of neighbor candidates.
We empirically observed that SNS not only increases the training efficiency of MODERNNCA, since fewer examples are utilized for pro-pprop,g,tion, but also improves the generalization ability of the learned mapping ø. We attribute the improvement to the fact that á is learned on more difficult, stochastic prediction tasks. The resulting & thus becomes more robust to the potentially noisy and unstable neighborhoods in the test scenario. The influence of sampling ratio and other sampling strategies are investigated detail in the experiments.
Distance Function. Empirically, we find that using the Euclidean distance instead of its squared form in Equation 4 leads to further performance improvements. Therefore, we adopt Euclidean distance as the default. Comparisons of various distance functions are provided in the appendix.
EXPERIMENTS
5.1
SETUPS
We evaluate MODERNNCA 300 datasets from recently released large-scale tabular bench- mark (Ye et al. 2024a), comprising 120 binary classification datasets, 80 multi-class classification datasets, and 100 regression datasets sourced from UCI, OpenMI. (Vanschoren et al. 2014), Kaggle,

---

Published as a conference paper at ICLR 2025
Accuracy
RMSE
10.00 DAAets
DANets 1200
letft
ModaraNGA
ExceFormor
Caliesst
2002
SNNT 12:0
FT-T
DEN6
FT-1
area
MLP-LL/
MUP
(a) Classification
(b) Regression
Figure 2: The critical difference diagrams based on the Wilcoxon- Holm test with significance level of 0.05 to detect pairwise significance for both classification tasks (evaluated using accuracy) and regression tasks (evaluated using RMSE).
Table I: The Win/Tie/Lose ratio between MODERNNCA and 20 comparison methods across the 300 datasets, covering both classification (based on accuracy) and regression tasks (based on RMSE). This ratio is determined using significant (-test at 95% confidence interval.
Method
Win
Tie
Lose
Method
Win
Tie
Lose
SVM
0.78 0.13
0.10
KNN
0.79 0.07
0.14
SwitchTab (Wa et al.
2024
0.88 0.09
0.03
DANets Chen 2022
0.74 0.18
0.08
NODE Popoy
2020
0.70 0.15
0.15
Tangos Jelfares
2023
0.66 0.20
0.14
TabCaps Chen el al.
2023
0.64 0.23
0.13
PTaRL el. 2024b
0.62 0.22
0.16
DCNv2 Wang al.
0.62 0.20
0.18
MLP Gorishniy
0.61 0.23
0.15
ResNet Gorishniy et al.
2021
0.59 0.30
0.11
MLP-PLR (oorishniy et
2022
0.57 0.27
0.16
KandomForess
0.57 0.18 0.26
ExcelForner (Chen
2024
0.56 0.28
0.16
SAINT Somepalli et
2022
0,55 0.28 0.18
FT-T Gorishniy et al.
2021
0.50 0.28
0.23
XGBoost (Chen
2016
0.49
0.19
0.32
LightCiBM Ke 2017
0.45 0.23
0.32
TabR Gorishniy al.
2024
0.41 0.36 0.23
CatBoost (Prokhorenkova al.
2018
0.38 0.27 0.35
and other repositories. The dataset collection in Ye et al. (2024a) was carefully curated, consider- ing factors such as data diversity, representativeness,, and quality mentioned Kohli et (2024);
Tschalzev et al. (2024).
Evaluation. We follow the evaluation protocol from Gorishniy et al. (2021 2024). Each dataset randomly split into training, validation, and test sets in proportions of 64% 66%/20% respectively. For each dataset, we train each model using 15 different random seeds and calculate the average performance on the test set. For classification tasks, consider accuracy (higher is better), and for regression tasks, we use Root Mean Square Error (RMSE) (lower is better). To summarize overall model performance, we report the average performance rank across all methods and datasets (lower ranks are better), following Delgado et al. (2014); McElfresh et al. (2023). Additionally, we conduct statistical l-tests to determine whether the differences between MODERNNCA and other methods statistically significant.
Comparison Methods.
We compare MODERNNCA with 20 approaches among three different categories, including classical parametric methods, parametric deep models, and neighborhood-basee methods. For brevity, only 8 of them shown Figure
Implementation Details. We pre-process all datasets following Gorishniy et al. (2021). For all deep methods, we set the batch size as 1024. The hyper-parameters of all methods are searched based on the training and validation set via Optuna (Akiba et al. 2019)
wwrnsnngGorrshiyy..ooo et al. (202T) 2024) over 100 trials. We set the ranges the hyper-parameters for the compared methods following Gorishniy et al. 2021 2024) and their official codes. The best-performed hyper-parameter are fixed during the final 15 seeds. Since the sampling rate of SNS effectively enhances the performance and reduces the training overhead, we treat it dyper-parameter and search within the range of [0.05, 0.6]. For additional implementation details, please refer to Liu et al. (2024).
5.2
MAIN RESULTS
The comparison results between MODERNNCA, L-NCA, and six representative methods pre- sented in infuieere] I All methods are evaluated across three aspects: performance (average performance rank), average training time, and average memory usage across all datasets. While some models,
7

---

Published as a conference paper at ICLR 2025
such TabR, exhibit strong performance, they require significantly longer training times. In contrast, MODERNNCA strikes an excellent balance across various evaluation criteria.
We also applied the Wilcoxon-Hool Holm test (Demsar 2006 to assess pairwise significance among all methods for both classification and regression tasks, The results are shown in Figure 2 For classification tasks (shown in the left part Figure 2). MODERNNCA consistently outperforms tree- based methods like XGBoost in most cases, demonstrating that its deep neural network architecture more effective capturing nonlinear relationships. Furthermore, compared to deep tabular models such as FT-T and MLP-PLR, MODERNNCA maintains its superiority. When combined with the results in Figure 1, these observations validate the effectiveness of MODERNNCA. It achieves performance on par with the leading tree-based method, CatBoost, while outperforming existing deep tabular models both classification and regression tasks across 300 datasets.
Additionally, we calculated the Win/Tie/Lose ratio between MODERNNCA and other comparison methods across the 300 datasets. If two methods show no significant difference (based on a f-test at 95% confidence interval), they are considered tied. Otherwise, one method is declared the winner based on the comparison of their average performance. Given the no free lunch theorem, is challenging for any single method to statistically outperform others across all cases. Nevertheless, MODERNNCA demonstrates superior performance in most cases. For instance, MODERNNCA outperforms TabR on 123 datasets, ties 108 datasets, and does with simpler architecture and shorter training time. Compared to CatBoost, MODERNNCA wins on 114 datasets and ties on 81 datasets. These results indicate that MODERNNCA serves effective and competitive deep learning baseline for tabular data.
ANALYSES AND ABLATION STUDIES OF MODERNNCA
In this section, we analyze the sources of improvement in MODERNNCA. All experiments are conducted on tiny tabular benchmark comprising 45 datasets, as introduced in (Ye et al. (024a) The benchmark consists of 27 classification datasets and 18 regression datasets. The average rank of various tabular methods on this benchmark closely aligns with the results observed on the larger set of 300 datasets, as detailed in Ye et al. 2024a).
6.1 IMPROVEMENTS FROM NCA TO L-NCA
We begin with the original NCA (Goldberger et al. 2004), using the scikit-learn implementation (Pe dregosa et al.) 2011). We progressively replace key components in NCA and assess the resulting performance improvements. Since the original NCA only targets classification tasks, this subsection focuses on the 27 classification datasets in the tiny benchmark. To ensure fair comparison, re-implement the original NCA using the deep learning framework Py PyTorch (Paszke et al., 2019). denoting this baseline version as "NCAvO".
Does Projection to a Higher Dimension Help? In the scikit-learn implementation, NCA is con- strained to perform dimensionality reduction, i.e., d' < for the projection L. We remove this constraint, allowing NCA to project into higher dimensions, and refer to this version as "NCAvl". Although higher dimensions by linear projections do not inherently enhance the representation ability of the squared Euclidean distance, the improvements in average performance rank from NCAv0 to NCAv1 (shown in Table 2) indicate that projecting to a higher dimension facilitates the optimization of this non-convex problem and improves generalization.
Does Stochastic Gradient Descent Help? Stochastic gradient descent (SGD) is widely used optimizer in deep learning. To explore whether SGD can improve NCA's performance, we replace the default L-BFGS optimizer used scikit-learn with SGD (without momentum) and denote this variant as "NCAv2". The performance improvements from NCAv1 to NCAv2 in Table 2 indicate that SGD makes NCA more effective in tabular data tasks.
The Influence of the Loss Function. The original NCA maximizes the expected leave-one-out accuracy as shown in |Equation 3. In contrast, we minimize the negative log version of this objective as described in Equation 1| Although the log version for classification tasks was mentioned in GGllberger et al. (2004);| SSllahuudiion Hinton (2007). the original NCA preferred the leave-one- out formulation for better performance. We denote the variant with the modified loss function as
8

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of the average rank of (the linear) NCA variants and (the nonlinear) MLP across 27 classification datasets in the tinybbenchmark. The check marks indicate the differences in components among the variants. The average rank represents the overall performance of a method across all datasets, with lower ranks indicating better performance. The final variant, NCAv4, corresponds to the L-NCA version discussed in our paper.
High dimension
SGD optimizer
Log loss
Soft-NN prediction
Average rank
NCAvO
4,400
NCAv1
3.708
NCAv2
3,296
NCAv3
3.192
NCAv4
2.962
MLP
3.000
Table Comparison among various configurations of the deep architectures used to implement ø. where MLP is the default choice in MODERNNCA. We show the change in average performance rank (lower is better) across the four configurations on the 45 datasets in the tiny benchmark.
MLP
Linear
w/ LayerNorm
ResNet
Classification
2.333
2.778
2.537
2.352
Regression 2.333
2,433
2.528
2.806
"NCAv3". As shown in Table 2 (NCAv2 vs. NCAv3), we find that using the log version slightly improves performance, especially when combined with deep architectures used in MODERNNCA. Further comparisons with alternative objectives provided in the appendix.
The Influence of the Prediction Strategy. During testing, rather than applying a "hard" KNN with the learned embeddings in standard metric learning, adopt a soft nearest neighbor (soft-NN) inference rule, consistent with the training phase. This variant, using soft-NN for prediction, is referred as "NCAv4", which is equivalent to the "L-NCA" version defined in ssussccction 4.1 Based on the change of average performance rank in Table 2 this modified prediction strategy further enhances NCA's classification performance, bringing linear NCA surpassing deep models like MLP.
6.2 IMPROVEMENTS FROM L-NCA TO M-NCA
In this subsection, investigate the influence of architectures and encoding strategies systemati- cally reveal the impacts of more deep learning techniques on NCA.
Linear Deep Architectures. We first investigate the architecture design for 6 in MODERNNCA, where one or more layers of blocks g(.) are added on top of a linear projection. We consider three configurations. First, we set as a linear projection, where the dimensionality of the projected space is hyper-yyppaaee pappeerrrrmeter" Then we replace batch normalization with layer normalization in the block. Finally, we add residual link from the block's input to output. Based on classification and regression performance across 45 datasets, we present the average performance rank of the four variants in Table 3) To avoid limiting model capacity, ppraappareeteess such as the number of layers are determined based on the validation set. Further comparisons of fixed architecture configurations are listed in the appendix.
We first compare NCA with MLP vs. with the linear counterpart in Table In classification tasks, MLP achieves lower rank, highlighting the importance of incorporating nonlincarity into the model. However, in regression tasks, the linear version performs well, with MLP showing only small improvements. Although the linear projection is part of MLP's search space, the linear version benefits from a smaller
space, potentially resulting in better generalization.
As described in |sussection 4.2, MLP uses batch normalization instead of layer normalization. Em- pirically, batch normalization performs better average in both classification and regression tasks as shown in Table 3] Additionally, we compare the MI. implementation with and without residual connections. While performing similarly classification, MLP shows superiority, especially in regression. Therefore, we adopt the MLP implementation in Table 3 for MODERNNCA.
"This "linear" version also includes the SNS sampling strategy and the nonlinear PLR encoding.

---

Published as a conference paper at ICLR 2025
Table 4: Comparison among MODERNNCA, MLP (Gorishniy al. 2021 , and TabR (Gorishniy et al. 2024) with or without PLR encoding for numerical features. We show the change in average performance rank across the four configurations on the 45 datasets in the tiny-benchmark.
w/o PLR
w/ PLR
MLP
TabR
MODERNNCA
MLP
TabR
MODERNNCA
Classification Regression
4.556
3.148
3.037
4,480
3.037
2.630
4.444
3.167
3.389
3.333
3.444
3.222
Average Rank
MiseemiiCA
Average Rank
4.5
MedemBGA
4.0-
3.5
3.5
1.0-
1.0
2.5
2:5
2.0
2.0
10
30
100
30
50
100
Sample Rate (%)
Sample Rate (%)
(a) classification
(b) regression
Figure 3: The change of average performance rank with different sampling rates among (10%, 30%, 50%, 80%, 100% in SNS strategy. The dotted line denotes the rank of MODERNNCA.
Influence of the PLR Encoding. PLR encoding transforms numerical features into high-dimensional vectors, enhancing both model capacity and nonlinearity. To assess the impact of PLR encoding, we compare MODERNNCA with MLP and TabR, both with and without PLR encoding. Following similar setup as in Table 3) we present the change in average performance rank across six methods in both classification and regression tasks Table
Without PLR encoding, TabR outperforms MLP, and MODERNNCA shows stronger performance in classification while performing slightly worse regression (although still better than MLP). PLR encoding improves all methods, evidenced by the decrease in average performance rank. In the right section of Table 4 we observe that MODERNNCA performs best both classification and regression tasks, effectively leveraging PLR encoding better than TabR. This may be because the nonlinearity introduced by PLR compensates for the relative simplicity results also validate that the strength of MODERNNCA comes from combination of its objective, architecture, and training strategy, rather than relying solely on the PLR encoding strategy.
MODERNNCA, The
The Influence of Sampling Ratios. Due the huge computational cost of calculating distances in the learned embedding space, MODERNNCA employs Stochastic Neighborhood Sampling (SNS) strategy, where only subset of the training data is randomly sampled for each mini-batch. Therefore, Saii-batch.Thheforre.. the training time and memory cost is significantly reduced. We experiment with varying the pro- portion of sampled training data while keeping other haper-parameters constant, then evaluate the corresponding test performance. As shown in Figure 3 sampling 30%-50% of the training set yields better results for MODERNNCA than using the full set. SNS not only improves training efficiency but also enhances the model's 's generalization ability. The plots also indicate that, with a tuned sampling ratio, MODERNNCA achieves a superior performance rank (dotted lines the figure).
7 CONCLUSION
Leveraging neighborhood relationships for predictions a classical approach machine learning. In this paper, we revisit and enhance one of the most representative neighborhood-basee methods, NCA, by incorporating modern deep learning techniques. The improved MODERNNCA establishes itself as a strong baseline for deep tabular prediction tasks, offering competitive performance while reducing the training time required to access the entire dataset. Extensive results demonstrate that MODERNNCA frequently outperforms both tree-based and deep tabular models in classification and regression tasks. Our detailed analyses shed light on the key factors driving these improvements, including the enhancements introduced to the original NCA.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This research is partially supported by NSFC (62376118) 18). Collaborative Innovation Center of Novel Software Technology and Industrialization, Key Program of Jiangsu Science Foundation (BK20243012). We thank Si-Yang Liu and Hao-Run Cai for helpful discussions.
REFERENCES
Nesreen K Ahmed, Amir F Atiy,, Neamat El Gayar, and Hisham El-Shishiny. An empirical comparison of machine learning models for time series forecasting. Econometric reviews, 29(5-)) 594-621, 2010.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In KDD, 2019.
Sercan Ö. Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In AAAI, 2021.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. 0ass/607.00450, 2016.
CoRR,
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. ICLR, 2022.
Aurélien Bellet, Amaury Habrard, and Mare Sebban. Metric Learning. Morgan & Claypool Publishers, 2015.
Christopher Bishop. Pattern recognition and machine learning. Springer, 2006.
Vadim Borisov, Tobias Leemann, Kathrin SeBler, Johannes Haug. Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions Neural Networks and Learning Systems, abs/21 (0.01889:1211, 2022.
Hao-Rum Cai and Han-Jia Ye. Understanding the limits of deep tabular methods with temporal shift. CoRR, abs/2502.20260. 2025.
Chun-Hao Chang, Rich Caruana, and Anna Goldenberg. NODE-GAM: neural generalized additive model for interpretable deep learning. In ICLR, 2022.
Jintai Chen, Kuanlun Liao, Yao Wan, Danny Z. Chen, and Jian Wu. Danets: Deep abstract networks for tabular data classification and regression. In AAAI, 2022.
Jintai Chen, KuanLun Liao, Yanwen Fang, Danny Chen, and Jian Wu. Tabcaps: A capsule neural network for tabular data classification with bow routing. ICLR, 2023.
Jintai Chen, Jiahuan Yan, Qiyuan Chen, Danny Ziyi Chen, Jian Wu, and Jimeng Sun. Can deep learning model be sure bet for tabular prediction? In KDD, 2024.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. KDD, 2016.
Heng-Tee Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye. Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong. Vihan Jain, Xiaobing Liu, and Hemal Shah. Wide & deep learning for recommender systems. In DLRS, 2016.
Jason V. Davis, Brian Kulis, Prateek Jain, Suvrit Sra, and Inderjit Dhillon. iifommation-hhooreee metric learning. In ICML, 2007.
Manuel Fernández Delgado, Eva Cemadas, Senén Barro, and Dinani Gomes Amorim. Do we need hundreds of classifiers solve real world classification problems? Journal of Machine Learning Research, 15(1):3133-3181 2014.
Janez Demsar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7:1-30, 2006.
11

---

Published as a conference paper at ICLR 2025
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton. Analyzing and improving representations with the soft nearest neighbor loss. In ICML, 2019.
Amir Globerson and Sam T. Roweis. Metric learning by collapsing classes. In NIPS, 2005.
Jacob Goldberger, Sam Roweis, Geoffrey Hinton, and Ruslan Salakhutdinov. Neighbourhood components analysis. In NIPS, 2004.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. NeurIPS, 2021.
Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. In NeurIPS, 2022.
Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. Tabr: Tabular deep learning meets nearest neighbors in 2023. In ICLR, 2024.
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning typical tabular data? In NeurlPS, 2022.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: A factorization- machine based neural network for CTR prediction. In IJCAI, 2017.
Md. Rafuul Hassan, Sadiq AI-Insaif, Muhammad Imtiaz Hossain, and Joarder Kamruzzaman. A machine learning approach for prediction of pregnancy outcome following IVF treatment. Neural Computing and Applications, 32(7):2283-229) 2020.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. In ICLR, 2023.
David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned mlps and boosted trees tabular data. In NeurlPS, 2024.
Cheng-Kan Kang Hsieh, Longqi Yang, Yin Cui, Tsung-Yi Lin, Serge J. Belongie, and Deborah Estrin. Collaborative metric learning. In WWW, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JCML, 2015.
Alan Jeffares, Tennison Liu, Jonathan Crabbé, Fergus Imrie, and Mihaela van der Schaar. Tangos: Regularizing tabular neural networks through gradient orthogonalization and specialization. In ICLR, 2023.
Xiangjian Jiang, Andrei Margeloiu, Nikola Simidjievski, and Mateja Jamnik. Protogate: Prototype- based neural networks with global-to-local feature selection for tabular biomedical data. In ICML, 2024.
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. In NeurIPS, 2021.
Liran Katzir, Gal Elidan, and Ran El-Yaniv. Net-dnf: Effective deep modeling of tabular data. In ICLR, 2021.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: highly efficient gradient boosting decision tree. In NIPS, 2017.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NewrIPS, 2020.
Ravin Kohli, Matthias Feurer, Katharina Eggensperger. Bernd Bischl, and Frank Hutter. Towards quantifying the effect datasets for benchmarking: look at tabular machine learning. In ICLR Workshop, 2024.
12

---

Published as a conference paper at ICLR 2025
Brian Kulis. Metric learning: A survey. Foundations and Trends in Machine Learning, 5(4), 2013. Steinar Laenen and Luca Bertinetto. On episodes, prototypical networks, and few-shot learning. In NeurIPS, 2021.
Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programminng 45(1):503-555 -200008----111 1989.
Kuan Liu, Aurélien Bellet, and Fei Sha. Similarity learning for high-dimensional sparse data. In AISTATS, 2015.
Si- Yang Liu and Han-Jia Ye. Tabpfn unleashed: A scalable and effective solution to tabular classifi- cation problems. CoRR, abs/2502.02527, 2025.
Si-Yang Liu, Hao-Run Cai, Qi-Le Zhou, and Han-Jia Ye. TALENT: A tabular analytics and learning toolbox. CoRR, abs/2440.044057 2024.
Duncan C. McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C., Ganesh Ramakrish- nan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? In NeurIPS, 2023.
Martin Renqiang Min, Laurens van der Maaten, Zineng Yuan, Anthony J. Bonner, and Zhaolei Zhang. Deep supervised t-distributed embedding. In JCML, 2010.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT Press, 2012.
Lennart J Nederstigt, Steven Aanen, Damir Vandic, and Flavius Frasincar. Floppies: a framework for large-scale ontology population of product information from tabular data in e-commerce stores. Decision Support Systems, 59:296-311 311, 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 22:2025-2830. 2011.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In ICLR, 2020.
Liudmila Ostroumova Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In NeurIPS, 2018.
Matthew Richardson, Ewa Dominowska, and Robert Ragno. Predicting clicks: estimating the thick-through rate for new ads. In WWW,2000
Ivan Rubachev, Artem Alekberov, Yury Gorishniy, and Artem Babenko. Revisiting pretraining objectives for tabular deep learning. CoRR, abs/2207.032001 2022.
Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, and Artem Babenko. Tabred: benchmark of tabular machine learning inithe-wild. In ICLR, 2025.
Ruslan Salakhutdinov and Geoffrey E. Hinton. Learning nonlinear embedding by preserving class neighbourhood structure. In AISTATS, volume 2, 2007.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015.
Yuan Shi, Aurélien Bellet, and Fei Sha. Sparse compositional metric learning. In AAAI, 2014.
13

---

Published as a conference paper at ICLR 2025
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2013.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NIPS, 2016.
Gowthami Somepalli, Avi Schwarzschild, Micah Goldblum, C. Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training. In NeurlPS Workshop. 2022.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. CVPR, 2016.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning 2014.
Research, 15(1):1929-11)
Daniel Tarlow, Kevin Swersky, Laurent Charlin, Ilya Sutskever, and Richard S. Zemel. Stochastic k-neighborhood selection for supervised and unsupervised learning. In ICML, 2013.
Valentin Thomas, Junwei Ma, Rasa Hosseinzadeh, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony L. Caterini. Retrieval & fine-tuning for in-context tabular models. In NewrlPS, 2024.
Andrej Tschalzev, Sascha Marton, Stefan Lüdtke, Christian Bartelt, and Heiner Stuckenschmidt. A data-centric perspective on evaluating machine learning models for tabular data. In NeurlPS, 2024.
Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for self-supervised representation learning. NeurIPS, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (11),, 2008.
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49-66) 60, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Jarkko Venna, Jaakko Peltonen, Kristian Nybo, Helena Aidos, and Samuel Kaski. Information retrieval perspective to nonlinear dimensionality reduction for data visualization. Journal of Machine Learning Research, 11:451-490 2010.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, 2016.
Ruoxi Wang, Rakesh Shivanna, Derek Zhiyuan Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Chi. DCN V2: improved deep cross network and practical lessons for web-scale learning to rank systems. In WWW, 2021.
Tianjun Wei, Jianghong Ma, and Tommy W. Chow. Collaborative residual metric learning. In SIGIR, 2023.
Kilian Q Weinberger and Lawrence Saul. Distance metric learning for large margin nearest neighbor classification. Journal of Machine Learning Research, 10:207-244, 2009.
Jing Wu, Suiyao Chen, Qi Zhao, Renat Sergazinov, Chen Li, Shengjie Liu, Chongchao Zhao, Tianpei Xie, Hanqing Guo, Cheng Ji, Daniel Cociorva, and Hakan Brunzell. Switchtab: Switched autoencoders are effective tabular learners. In AAAI, 2024.
Zhirong Wu, Alexei A. Efros, and Stella X. Yu. Improving generalization via scalable neighborhood component analysis. ECCV, 2018.
Eric P. Xing, Andrew Y. Ng, Michael I. Jordan, and Stuart Russell. Distance metric learning with application to clustering with side-information. In NIPS, 2002.
14

---

Published as a conference paper at ICLR 2025
Xun Yang, Meng Wang, and Dacheng Tao. Person re-identification with metric learning using privileged information. IEEE Transactions on Image Processing, (1(2):791-805. 2018.
Han-Jia Ye, De-Chuan Zhan, Nan Li, and Yuan Jiang. Learning multiple local metries: Global consideration helps. IEEE Transactions on pattern analysis and machine intelligence, 42(7): 1698-1712, 2020.
Han-Jia Ye, Yiny Liu, Hao-Run Cai, Qi-Le Zhou, and De-Chuan Zhan. A closer look at deep learning on tabular data. CoRR, hwtp22407.00556. 2024a.
Han-Jia Ye, Si- Yang Liu, and Wei-Lun Chao. A closer look at tabpfn v2: Strength, limitation, and extension. CoRR, abs/2502.17361, 2025.
Hangting Ye, Wei Fan, Xiaozhuang Song, Shun Zheng, He Zhao, Dan dan Guo, and Yi Chang. Ptarl: Prototype-based tabular representation learning via space calibration. In ICLR, 2024b.
Dong Yi, Zhen Lei, Shengeai Liao, and Stan Z. Li. Deep metric learning for person re-identification. In ICME, 2014.
Qi-Le Zhou, Han-Jia Ye, Leye Wang, and De-Chuan Zhan. Unlocking the transferability of tokens in deep models for tabular data. CoRR, abs/2310.15144 2023.
15

---

Published as a conference paper at ICLR 2025
The Appendix consists of two sections:
Appendix A
Datasets and implementation details. Additional experimental results.
Appendix B
APPENDIX A
DATASETS AND IMPLEMENTATION DETAILS
In this section, we outline the preprocessing steps applied to the datasets before training, as well as descriptions of the datasets used.
A.1 DATA PRE-PROCESSING
We follow the data preprocessing pipeline from Gorishiny et al. (2021) for all methods. For numerical features, we apply standardization by subtracting the mean and scaling the values. For categorical features, we use one-hot encoding to convert them for model input.
A.2 DATASET INFORMATION
We the recent large- tabular benchmark from Ye et al. (022a)) which includes 300 datasets covering various domains such as healthcare, biology, finance, education, and physics. The dataset sizes range from 1,000 to I million instances. More detailed information the datasets can be found in Ye et al. (002a)).
For each dataset, we randomly sample 20% of the instances to form the test set. The remaining 80% is split further, with 20% of which held out as a validation set. The validation set is used to tune hyper- prper-parameters and apply early stopping. The hyper-parameten with which the model performs best on the validation set are selected for final evaluation with the test set.
The datasets used in our analyses and ablation studies follow the tiny-benchmark in Ye al. (2024a) which consists of 45 datasets. The performance rankings of methods on this smaller benchmark are consistent with those on the full benchmark, making a useful probe for tabular analysis.
A.3 HARDWARE
The majority of experiments, including those measuring time and memory overhead, conducted on a Tesla V100 GPU.
A.4 POTENTIAL ALTERNATIVE IMPLEMENTATION
We explore an alternative strategy learn the embedding é two steps. First, we apply Supervised Contrastive loss (Sohn) 2016 Khosla et al. 2020), where supervision is generated within a mini-batch. After learning Φ. we use KNN for classification or regression during inference. In the regression scenario, label values are discretized, and we refer to this baseline method as Tabular Contrastive (TabCon). Empirically, we find that certain components of MODERNNCA, such as the Soft-NN loss for prediction, cannot be directly applied to TabCon, even when é is implemented using the same nonlinear MLP as in MODERNNCA. Despite this, the TabCon baseline remains competitive with FI-Transformer (FT-T), achieving average ranks similar to L-NCA in both classification and regression tasks.
A.5 COMPARISON METHODS
We compare MODERNNCA with approaches among three different categories. First, we consider classical parametric methods, including linear SVM and tree-based methods like RandomForest.
XGBoost (Chen & Guestrin 2016). LightGBM Ke et al. (017)). and CatBoost (Prokhorenkova
et al.
2018). Then, we consider parametric deep models. including NODE (Popov et al.) 2020). 2021). ResNet (Gorishniy et al. 2021), SAINT (Somepalli 2022), DCNv2 (Wang et al. 2021 Transffforerr (Gorishniy et al. 2021). DANets (Chen
MLP
Kadra et al. 2021] Gorishniy et al.
et al.
et al.
2022). MLP-PLR (Gorishniy et al. 2022), TabCaps (Chen et al. 2023), Tangos (Jeffarrs et al.
2023
PTARL. (Ye et al. 2024b), SwitchTah (Wu et al. 2024), and ExcelFormer (Chen et al. 2024)
16

---

Published as a conference paper at ICLR 2025
(a) AD Feature
(b) AD 1 TabR
(c) AD MODERNNCA
(c) AD TabCon
(a) PH f Raw Feature
(b) PH 1 TabR
(c) PH MODERNNCA
(c) PH 1 TabCon
(a) CAJ Raw Feature
(b) CA2 TabR
(c) CA MODERNNCA
(c) CA| TabCon
(a) MIA Raw Feature
(b) MIA TabR
(c) MIA, MODERNNCA
(c) MIA TabCon
Figure 4: Visualization of the embedding space of different methods.
For neighborhood-basee methods, consider KNN and TabR (Gorishniy al. 2024). For fair comparison, the same PLR numerical encoding applied MLP-PLR, TabR, and MODERNNCA.
APPENDIX B ADDITIONAL EXPERIMENTS
B. VISUALIZATION RESULTS
To better analyze the properties of MODERNNCA, we visualize the learned embeddings Φ(Φ) of MODERNNCA, TabCon (mentioned in (sussection A.4), and TabR using TSNE (Van der Maaten & Hinton 2008). As shown in Figure 4, all deep tabular methods transform the embedding spaces to be more helpful for classification or regression compared to the raw features. The embedding space learned by TabCon clusters samples of the same class together and separates samples of different classes, often clustering same-class instances into single cluster. However, still struggles with some hard-to-distinguske samples. TabR and MODERNNCA, on the other hand, divide samples of the same class into multiple clusters, ensuring that similar samples are positioned closer to each other. This strategy aligns with the prediction mechanism of KNN, where good performance is achieved by clustering instances with similar neighbors together rather than into a single cluster. The embedding space learned by MODERNNCA is more discriminative than that learned by TabR. The main reason that TabR leverages an additional architecture to modify the prediction score for each instance, making the learned embedding space less discriminative compared MODERNNCA.
17

---

Published as a conference paper at ICLR 2025
Table Comparison among various distances used to implement Equation 4 where Euclid distance is the default choice in MODERNNCA. We show the change in average performance rank (lower is better) across the five configurations on the 45 datasets in the tiny tny-benchharr.
Euclid
Dot Product
Cosine
Squared Euclid L1-Norm
Classification 2.593
3.852 3.222
2.111
2.630
3.769
Regression 2.500
2.529
2.722
3.889
Table 6: Comparison of different loss functions. The log loss used in MODERNNCA, the original NCA's summation loss, the MCML. loss, and the t-distribution loss. The change in average perfor- mance rank (lower is better) is presented across these four configurations on the 45 datasets in the tiny-benchmark.
MODERNNCA
NCA
MCML
t-distribution
Classification
2.074
2.519
3.074
2.333
Regression 1.500
1.540
B.2 ADDITIONAL ABLATION STUDIES
The Influence of Distance Functions. The predicted label target instance æ; determined by the label of its neighbors in the learned embedding space projected by ø. The distance function dist(:.)) is the key to determining the pairwise relationship between instances in the embedding space and influences the weights in Equation 4.
In MODERNNCA, we choose Euclidean distance
distave ()(x;), d(x,)) = v (o(x;) -
- ((x,)) = 11(())))))))))))))))) - -(x;))2 (6)
We also utilize other distance functions, e.g., the squared Euclidean distance, dist) ((x))).
the f fi-momm distance
dost()( v(x;)) = [1<((;;) ) <bxr)))11
the (negative) cosine similarity (ist((( (xx,)) = (2) xz)) (1== (e::1l21e, ||,), and the (negative) inner product (is))))();), ((xr,)) = ((x) (.x,)) The results using different distance functions are listed in Table 5] which contains the average performance rank over 45 datasets among the five variants. On average, Euclidean distance performs well across both classification and regression tasks. While cosine distance yields better results classification datasets (with average performance rank of 4.5939 compared to MODERNNCA and 20 other methods across 300 datasets, please check Figure 2 for details), its advantage diminishes on regression tasks.
Other Possible Loss Functions. NCA GGolderger et al., 2004) originally explored two loss functions: that maximizes the sum of probabilities in Eqqation 3] and another that minimizes the sum of log probabilities as in EEquation| I] The former was selected in the original implementation of NCA due to its better performance. We also investigated several alternative loss functions for NCA. For instance, MCML (Globerson & Roweis, 2005) minimizes the KL-divergence between the learned embedding in Eqqation and constructed ground-truth label distribution for each instance, but only applies to classification tasks. Another variant is the t-distributed NCA (Min et al., 2010), which uses heavy-tailed I-distribution to measure pairwise similarities in the objective function. We tested both MCML and the t-distribution loss functions in MODERNNCA, and the results are summarized in Table 6) showing the average ranks across 45 datasets. The log objective in Equation Pperform best for classification tasks and slightly outperforms the t-distribution variant regression tasks.
The Influence of Sampling Strategy. As mentioned before, SNS randomly samples a subset of training data for each mini-batch when calculating the loss of Equation 4, We also investigate whether we could further improve the classffictionnregressson ability of the model when we incorporate richer information during the sampling process, e.g., the label of the instances.
We consider two other sampling strategies in addition to the fully random one we used before. First Mass-wise random sampling, which means that given proportion, sample from each class
18

---

Published as a conference paper at ICLR 2025
Table Comparison of different sampling strategies: "Random", "Label", and "Distance" represent MODERNNCA 's naive uniform sampling, class-wise random sampling, and distance-based sampling. respectively. The change in average performance rank (lower better) presented across these three configurations on the 45 datasets in the tiny-benchmark.
Random
Label
Distance
Classification 1.869
2.230
1.901
Regression 1.508
1,492
Table 8: Comparison various architecture choices based on a fixed 2-layer MLP. We only tune architecture-indest inniuuedenddde arper-parameters for different variants. The change in average performance rank (lower is better) is shown across three configurations (default, Layer Norm, and Residual) on the 45 datasets in the nay-beecchmar.
MLP
w/ LayerNorm
ResNet
Classification
1.905
2.048
2.048
Regression
1.813
2.313
1.875
in the training set and combine them together. This strategy takes advantage of the training label information and keeps the instances from all classes that will exist in the sampled subset. Besides, we also consider the sampling strategy based on the pairwise distances between instances. Since the neighbors of an instance may contribute more (with larger weights) in EEuutioo 4| so given mini-batch, we first calculate the Euclidean distance between instances in the batch and all the training set with the embedding function é in the current epoch. Then we sample the training set based on the reciprocal of the pairwise distance value. In detail, given an instance 2;, we provide instance- specific neighborhood candidates and æj in the training set is sampled based on the probability 1/(dist(of (o(æ,), (xx)))) T a non-negative hyper-parameter to calibrate the distribution. The distance caleulation requires forward passes of the model & over all the training instances, and the instance-speciffi neighborhood makes the loss related to wide range of the training data. Therefore, the distance-based sampling strategy has low training speed and high computational burden.
The comparison results, i.e., the average performance rank, among different sampling strategies on 45 datasets listed in Table We empirically find the label-based sampling strategy cannot provide further improvements. Although the distance-based strategy helps in certain cases, the improvements limited. Taking holistic consideration of the performance and efficiency, choose the vanilla random sampling in MODERNNCA.
Comparison between Different Deep Architectures. Unlike the ablation studies in subsection 6.2] where we fixed the model family and tuned detailed syper-parametert (such as the number of layers and network width) based the validation set, here we fix the main architecture two-layer MLP and only tune architecture inneetureeddeeeedeeettee hyper- parameters, such as the learning rate.
With this base MLP architecture, we evaluate three variants: the base MLP, one with batch normaliza- tion replaced by layer normalization, and one with an added residual link. The average ranks of the three variants across 45 datasets are presented Table We observe that the basic MI.P remains a better choice compared to the versions with a residual link or layer normalization.
B.3 RUN-TIME AND MEMORY USAGE ESTIMATION
We make a run nun-time and memory usage comparison in Figure 1 Here the steps that we take to perform the estimation. First, we tuned all models on the validation set for 100 iterations, saving the optimal parameters ever found. Next, ran the models for 15 iterations with the tuned parameters and saved the best checkpoint on the validation set. The run-time for the models was estimated using the average time taken by the tuned model to run seed in the training and validation stage.
We present the average results of run-time and memory usage estimation across the full benchmark (300 datasets) in Table 9
19

---

Published as a conference paper at ICLR 2025
Table Training time and memory usage estimation for different tuned models over 300 datasets. The average rank represents the mean performance ranking of these models based on the performance metries (RMSE for regression and accuracy for classification).
Model
M-NCA L-NCA
MLP
MLP-PLR
FT-T
TabR
XGBoost
CatBoost
Training Time (s)
87.5
33.62
30.36
42.87
111.91
173.34
4.53
20.48
Memory Usage (GB)
5.36
1.42
1.15
2.37
4.98
10.13
0.84
1.06
Average Rank
4.56
6.30
7.53
6.94
6.29
5.36
5.62
4.61
B.4 FULL RESULTS ON THE BENCHMARK
Due the extensive size of the results table, we have uploaded the complete performance metries of ModernNCA alongside other comparison methods, including RealMLP (Holzmüller et al. 2024), at
https://qi
B.5 LIMITATIONS
ModernNCA has two possible limitations.
The first limitation pertains to handling tabular data with distribution shifts, as discussed in Rubachev al. (205). Specifically, ModemnNA does not explicitly account for implicit temporal relationships between instances and their neighbors during the neighborhood search. However, a recent study (Cai random splits for
Ye 2025) has shown that adopting alternative data-splitting protocols-sude training and sondaation-iinnfficaall improves ModernNCA's performance, making competitive with other methods. Furthermore, ModemnNCA's performance is further enhanced when incorporating temporal embeddings.
The second limitation lies in handling high-dimensional datasets where d N (Jiang al. 2024). as observed in Ye et al. (2025). This challenge is well-known in classical metric learning (Shi al. 2014 Liu 2015), where distance calculations become less reliable due to the curse of dimensionality. Figgh-dimensiona data can lead to reduced neighborhood retrieval effectiveness, impacting prediction accuracy. Potential mitigations include pre-processing with dimensionality reduction techniques and leveraging ensemble approaches (Liu & Ye 2025), which may help alleviate the adverse effects high dimensionality.
20


Reviews:
[{"review_id": "Akn2AF6dUb", "reviewer": "Reviewer_PJDn", "summary": "This study considers learning on tabular data, and proposes ModernNCA -- a deep version of the classic Neighborhood Components Analysis algorithm. Contrary to NCA, the transformation in ModernNCA is non-linear, and is powered by a neural network. To make the training of ModernNCA more efficient and more effective, the paper also proposes Stochastic Neighborhood Sampling (SNS). In experiments on 300 datasets, ModernNCA is reported to achieve the best average rank among the considered baselines, including gradient-boosted decision trees (GBDT).", "questions": "-", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method is simple.\n- Generally, I tend to agree that nearest neighbors may be underexplored in the context of tabular data. While TabR seems to close this gap to some extent, ModernNCA looks like a good addition to the field.\n- The SNS strategy looks simple and effective, and also differentiates the method from TabR.\n- On the considered benchmark, the proposed ModernNCA achieves a better average rank and a better balance of task performance and training time compared to baselines.\n- A large number of baselines and datasets.\n- An ablation study is provided.", "weaknesses": "Note: regarding the \"datasets\" and \"metrics\" weaknesses, I admit that the field lacks standardized benchmarks and metrics.\n\n**Datasets**\n\nMy understanding is that the benchmark consists of many automatically collected datasets. In the light of the recent studies about tabular datasets, it is unclear how representative the benchmark is. Examples of the studies:\n\n- Towards quantifying the effect of datasets for benchmarking: A look at tabular machine learning\n- A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data\n- TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks\n\n**Metrics**\n\nThe metrics such as ranks or wins do not show the scale of performance gaps between methods. It is unclear how significant are the wins and losses of ModernNCA (not from statistical perspective, but from the practical perspective).\n\n**Presentation**\n\n- In my opinion, the presentation could be more efficient. The proposed NCA extensions are not conceptually novel, so I believe that the story on the first six pages could be more compact. Perhaps, some of the details and discussion can be moved to appendix.\n- Based on my understanding of the TabR baseline\n    - The explanation of TabR on L152 is not correct, since it is not a Transformer variant. Quoting the TabR paper: *\"a feed-forward network with a custom k-Nearest-Neighbors-like component in the middle\"*.\n    - The description of TabR on L216-L226 is: (1) not complete, (2) not correct, and, if I am not mistaken, (3) not used in the story. In the light of (3), I do not go into details about (1) and (2). Perhaps, this description can be simply removed?\n- Generally, communicating the empirical nature of a study as in L108-L113 is fine. However, personally, I would change the first sentence on L108 to something more neutral.\n- Perhaps, Figure 1 can be placed closer to the related experiments, but this can be subjective.\n\n**Related work.**\n\nThere is a missing related work: \"Improving Generalization via Scalable Neighborhood Component Analysis\" ECCV 2018. That paper also describes how to efficiently train a deep NCA, and I think their method is more advanced than the one proposed in this submission. Though their method can be too complicated for the scope of this paper. In that case, I recommend discussing this related work and explaining why the proposed SNS is a better choice for this work compared to the method from the referenced paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Whl7LxqRyu", "reviewer": "Reviewer_Cjck", "summary": "The paper proposes a revised take of the NCA algorithm for supervised learning on tabular data, where the neighborhood aggregation is done in a representaion space of a neural network, the model is optimized via SGD and additional stochasticity is introduced in subsetting the neighbors list. \n\nThe resulting architecture is conceptually simpler than prior state-of-the-art tabular retrieval models, while improving in performance and eficiency as shown via an extensive experimental evaluation.", "questions": "- Could you provide or point to the raw per-dataset metrics?\n- Is it possible to add some other means of comparison along with average ranks (e.g. relative improvement to a baseline)? What does it show? \n- What are the methods limitations?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is well motivated. Attention to simple KNN-based methods in deep tabular models was limited, except TabR which is well addressed in the text.\n- The method is both conceptually simple and easy to implement, without sacrificing performance\n- The experimental results and ablations are extensive and insightful:\n  - The step-by-step Linear-NCA ablation (table 2) is a principled and convincing way to explore a model design space\n  - Stochastic Neighborhood Sampling ablation shows an interesting result (improved performance from sampling) and provides a practically important outcome for retrieval-based tabular NNs\n  - Other important minor details (like the numerical feature embedding ablations), slight improvements in the distance function used, loss functions\n- The writing and overall storytelling is engaging and well thought-out", "weaknesses": "- All experimental results and ablations rely on the average ranks of a set of methods being tested. This provides some signal for which modificaitons are usefull, but other means of comparison might make this even clearer. For example, additinal relative improvement compared to a strong baseline (e.g. a well-tuned MLP) would provide usefull additional signal besides the average rank metric (e.g. what is the scale of such improvements). \n- Minor additional to the previous point: I find that providing raw unaggregated results for the core (or even for all) the experiments is very usefull for quick sanity-checks and comparisons in future work. So that others could assess results on the individual datasets by consulting the paper text.\n- I believe limitation should be discussed somewhere in the main text. E.g. what are the confines of the proposed mehtod. Are there any cases where it may perform poor. \n   - Maybe some post-hock meta-analysis akin to the one in tabzilla paper (https://arxiv.org/abs/2305.02997) of what are the datasets where ModernNCA performs worse. \n   - Some recent benchmarks demonstrate that retrieval-based models might not be universaly superior (https://arxiv.org/abs/2406.19380)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7pw4wDlCAk", "reviewer": "Reviewer_Qykr", "summary": "The paper revisits the Neighborhood Components Analysis (NCA) and adapts it for tabular data learning, proposing \bModernNCA as an enhanced approach. The modifications include (1) calculating distances in a representation space, (2) using stochastic gradient descent (SGD) instead of L-BFGS for optimization, and (3) training in a mini-batch fashion rather than on the entire dataset at once. The authors benchmarked ModernNCA against numerous methods across 300 datasets, finding that it achieved consistently superior performance, often comparable to leading models like CatBoost and outperforming many deep tabular learning methods.", "questions": "If my understanding is incorrect, could you please clarify what is the novel concept introduced in this paper?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The approach effectively leverages modern deep learning techniques to enhance classical NCA, demonstrating strong empirical results across a large number of datasets.\n\n- The paper provides comprehensive benchmarks, including comparisons with state-of-the-art methods in both classification and regression tasks.", "weaknesses": "**Lack of Novelty**: While the paper shows strong empirical performance, the core modifications (using a representation space for distance calculations, employing SGD, and mini-batch training) have already been explored in prior research. This raises concerns regarding the originality of the contribution. The changes appear more like tunings of established techniques rather than introducing a fundamentally new method.\n\n[Prior research example] J Kang et al., Deep metric learning based on scalable neighborhood components for remote sensing scene characterization, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "noLBH3Ujz1", "reviewer": "Reviewer_3moZ", "summary": "This paper explores the potential of modernizing the classical Nearest Neighbor approach for tabular data by leveraging a differentiable K-nearest neighbors variant, Neighborhood Components Analysis (NCA). The authors introduce MODERNNCA, an improved version of NCA that integrates deep learning techniques such as stochastic gradient descent (SGD), nonlinear embeddings, and a Stochastic Neighborhood Sampling (SNS) strategy to boost computational efficiency and model performance. They demonstrate that MODERNNCA matches or outperforms both tree-based models and current deep tabular models across 300 datasets in classification and regression tasks.", "questions": "- MODERNNCA relies on Euclidean distance as the default metric. How adaptable is the model to other distance metrics, such as cosine similarity, and do the authors have insights on how these choices might impact performance?\n\n- The paper discusses various deep learning modifications, but how sensitive is MODERNNCA to hyperparameter choices such as learning rate, number of neighbors (K), and embedding dimensions? Are there specific configurations where performance significantly degrades?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Originality**\nRevisiting a classic nearest-neighbor approach with contemporary deep learning techniques is a novel approach, particularly since NCA had been previously limited by computational efficiency and scalability. This approach aims to unify insights from both traditional and modern tabular prediction methods.\n\n**Quality**\nThe authors conducted extensive experiments across 300 datasets, providing thorough evidence of the model’s strengths and weaknesses. \n\n**Clarity**\nThe explanation of the modifications, including SGD, nonlinear embeddings, and SNS, is detailed and clear, making the improvements accessible to the reader.", "weaknesses": "**Marginal Contribution**  \n   The paper’s contribution feels incremental rather than pioneering. The improvements in MODERNNCA rely on established techniques (SGD, SNS, and nonlinear embeddings) without introducing a fundamentally new concept or method. This makes the novelty limited, as it essentially optimizes an existing algorithm rather than providing a unique advancement. \n\n**Limited Novelty in Comparison to KNN Variants**  \n   The paper lacks direct comparisons with other KNN-inspired deep learning methods that have similarly benefited from modern optimization strategies. This limited scope of comparison weakens the argument for MODERNNCA’s distinctiveness and impact.\n\n**Lack of Theoretical Analysis**  \n   The paper’s focus is primarily empirical, with little theoretical exploration of why the proposed enhancements lead to improved performance. A deeper theoretical perspective on the modifications—such as the effect of stochastic sampling on generalization—would provide valuable insights and strengthen the work's academic contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wI4jErwsQO", "reviewer": "Reviewer_B7pB", "summary": "This paper re-evaluates the nearest neighbor approach for tabular data, proposing a modernized version of neighborhood component analysis (nca) called modernnca. starting from classical nn techniques, the authors incorporate current deep learning methods into nca, such as using stochastic gradient descent and adding stochastic neighborhood sampling. the experiments on 300 tabular datasets demonstrate that modernnca performs comparably to leading models like catboost, outperforming other deep learning approaches for classification and regression tasks. the authors also provide insights into how modern techniques like batch normalization and non-linear architectures improve nn performance.", "questions": "1. How does modernnca perform with different distance metrics, such as cosine similarity, especially for datasets with high-dimensional features?\n\n\n2. Given that modernnca outperformed many models, have you tested its performance on imbalanced datasets?\n\n3. Could you provide more insights into cases where modernnca significantly underperforms compared to catboost or other tree-based methods?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the authors structure their explanation to make complex methods like nn and nca accessible and coherent. this clarity is helpful for understanding both the motivation and methodology behind the modifications to nca.\n\n2. the paper includes an extensive evaluation on a broad range of datasets, demonstrating modernnca’s effectiveness with detailed performance metrics and statistical significance tests.\n\n\n3.  The authors do a great job explaining the steps taken to modernize nca", "weaknesses": "W1: while the model shows high performance, the paper lacks specific scenarios or guidance on applying modernnca practically, such as when dealing with imbalanced datasets", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)