xRFM: Accurate, scalable, and interpretable feature learning models for tabular data
Authors: Daniel Beaglehole
David Holzmüller
Adityanarayanan Radhakrishnan
Mikhail Belkin
Venue: ICLR 2026 Poster (2026)
Keywords: Tabular data
kernel methods
tree-based methods
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We develop a non-neural, non-boosting kernel method that is state-of-the-art for supervised learning of tabular datasets.

Abstract: Inference from tabular data, collections of continuous and categorical variables organized into matrices, is a foundation for modern technology and science. Yet, in contrast to the explosive changes in the rest of AI, the best practice for these predictive tasks has been relatively unchanged and is still primarily based on variations of Gradient Boosted Decision Trees (GBDTs). Very recently, there has been renewed interest in developing state-of-the-art methods for tabular data based on recent developments in neural networks and feature learning methods.  In this work, we introduce xRFM, an algorithm that combines feature learning kernel machines with a tree structure to both adapt to the local structure of the data and scale to essentially unlimited amounts of training data. We show that compared to $31$ other methods, including recently introduced tabular foundation models (TabPFN-v2) and GBDTs,  xRFM achieves best performance across $100$ regression datasets and is competitive to the best methods across $200$ classification datasets outperforming GBDTs. Additionally, xRFM provides  interpretability natively through the Average Gradient Outer Product.

XRFM: ACCURATE, SCALABLE, AND INTERPRETABLE FEATURE LEARNING MODELS FOR TABULAR DATA
Daniel Beaglehole
David Holzmüller
Computer Science and Engineering Halıcioğlu Data Science Institute
INRTA Saclay
david. hww.mmmleelerrrii......... fr
UC San Diego
dbeaglehole@ucsd. edu
Adityanarayanan Radhakrishnan
Mikhail Belkin
MIT Mathematics
Halcroğlu Data Science Institute
Broad Institute of MIT and Harvard a aradha@m edu
UC San Diego mbe kin@ucsd.edd
ABSTRACT
Inference from tabular data, collections of continuous and categorical variables organized into matrices, is foundation for modern technology and science. Yet. in contrast to the dramatic developments in the rest of AI, the best practice for these predictive tasks has been relatively unchanged and still primarily based on variations of Gradient Boosted Decision Trees (GBDTs). Very recently, there has been renewed interest in developing state-of-the- methods for tabular data based on recent developments in neural networks and feature learning methods. In this work, we introduce xRFM, an algorithm that combines feature learning kernel machines with tree structure to both adapt to the local structure of the data and scale to essentially unlimited amounts of training data.
On the TALENT benchmark, we show that compared to 31 other methods, in- cluding recently introduced tabular foundation models (TabPFNv2) and GBDTs, xRFM achieves best performance across 100 regression datasets and is competi- tive to the best methods across 200 classification datasets outperforming GBDTs. Additionally, xRFM provides interpretability natively through the Average Gradi- ent Outer Product.
INTRODUCTION
Tabular data collections of continuous and categorical variables organized into matrices under- lies all aspects of modern commerce and science from airplane engines to biology labs to bagel shops. Yet, while methods for Machine Learning and AI in language and vision have seen un- precedented progress, the primary methodologies of prediction from tabular data have been rela- tively static, dominated by variations of Gradient Boosted Decision Trees (GBDT's), such as XG- Boost (Chen & Guestrin (2016). Nevertheless, hundreds of tabular datasets have been assembled to form extensive regression and classification benchmarks (Bischl et al.) 2025 Grinsztajn et al.) al. 2014 Ye al.) 2024 Erickson al., (025), and, recently, there has been renewed interest in building state-of- -hee--ff-hheeeeeee art predictive models for tabular data (Hollmann al. 2025 Holzmüller al. 2024; Gorishniy al.) (025). Notably, given the remarkable effective- ness of large, "foundation" models for text, there has been much excitement in developing similar models on tabular data, and recent effort has led the development of TabPFN-v2, foundation model for tabular data appearing in Nature (Hollmann et al. 2025). Yet. despite this progress, build- ing scalable, effective, and interpretable machine learning models in this domain remains an open challenge.
2022 Fernández-Delgado
In this work, we introduce xRFM, a tabular predictive model that combines recent advances in fea- ture learning kernel machines with adaptive tree structure, making highly accurate. scalable, and interpretable. xRFM builds upon the Recursive Feature Machine (RFM) algorithm from Radhakr- ishnan al. 2024). which enabled feature learning form of supervised dimensionality reduction)

---

in general machine learning models. xRFM works follows: Given training data, it first builds bi- nary tree structure to split data into subsets based on features relevant for prediction within each split. When splits reach certain size (of S C' data samples), we train leaf RFM (a pepee--arammter and compute optimized version of the original RFM).
There are two important consequences of using a tree structure in conjunction with leaf RFMs. The first is that it enables local feature learning, meaning that xRFM can learn different features for different subsets of the data. This property is crucial for tackling hierarchical structure in tabular data (for example, one set of features may be relevant when a specific feature takes on high value while another set of features relevant when the same feature takes low values). The second conse- quence is that it enables xRFM to scale nearly linearly in the number of samples (O(n log given n samples) during training and logarithmically (O(log ni) during inference. such, the inference time of xRFM is comparable to that of tree-based models.
In practice, we show xRFM has the best performance across 100 tabular regression tasks and is com- petitive with -tte--fft on 200 tabular classification tasks from the TALENT benchmark (Ye et al. (004). In particular, xRFM outperforms 31 other methods on these regression tasks including the GBDT and neural network variations mentioned in the opening paragraph. On the TabArena- Lite benchmark (Erickson et al. 2025). we show xRFM achieves one of the best tradeoff's (is on the empirical Pareto frontier) between performance and inference time among all methods in gression and is again competitive on classification. We show that xRFM achieves similar results on the largest datasets from the meta-test benchmark (Holzmüller al., 2024), where directly solving standard kernel machines becomes intractable on standard GPUs An additional benefit of xRFM that it natively provides interpretability by exposing features learned and used for prediction. In particular, each leaf RFM learns features through a mathematical object known as the Average Gradient Outer Product (AGOP), whose diagonal indicates coordinates relevant for prediction and whose top eigenvectors indicate directions in data most relevant for prediction. Using examples of xRFM trained on synthetic and real data, we show how AGOP matrices at each leaf RFM shed light on features relevant for prediction, including cases when leaf RFMs learn different features for different splits of data.
To summarize, xRFM combines feature learning kernel methods with adaptive data partitioning using the AGOP. It is a fast, effective, and interpretable model on tabular data at all scales.
2
PRELIMINARIES
We begin with a review of kernel machines and kernel-RFM, which we use to build xRFM
2.1 KERNEL MACHINES
A kernel machine is non non-parametric machine learning model (Schölkopf & Smola 2002; Aron- szajn [950).. The idea behind kernel machines is that a nonlinear predictive model can be trained by first transforming input data with a fixed, nonlinear feature map and then performing linear regres- sion on the transformed data. Kernel machines make this procedure computationally tractable even for infinite dimensional feature maps by using explicitly defined kernel functions (inner products of feature mapped data). We describe kernel machines in the context of supervised learning below. Let X E n denote training inputs with (:( denoting the example in the 7th row of X for [n] and y nx denote training labels. Let K : Rddd x Rd + R denote a kernel function (a posi- tive semi-definite, symmetric function). Given a regularization parameter 2 0, a kernel machine trained on the data (X,y) is a predictor, ʃ: Rdd -> TR' of the form:
f(x) K (I. X)a
a = (K(X,))+ y:
where the notation K(x, X) E Rxxn denotes an n-dimensional row vector with K(I, (I,, X); =
K(x,x x())) and XX E Raxn denotes a matrix with K(X,X); = K (xx(t) x 00) Exam- ples of kernel functions used in practice are the Gaussian kernel (K(x, = exp(-I|x- - =
or the Laplace kernel (K(x, 2) = exp(-l|z 101-|===]122//L) L)) The advantage of kernel functions is that the predictor admits a closed form solution, which can be computationally efficient to fit on datasets under 100k samples. The limitations of kernel machines are twofold: (1) the formulation in Eq. (I) does not scale well (it is super-quadratic in the number of samples) and (2) the choice of kernel is fixed independently of the input data. The former limitation is addressed by recent, pre-conditioned
2

---

gradient descent methods for solving for in Eq. ① and methods that construct low-dimensional approximations of the kernel matrix. These include EigenPro (Ma & Belkin 2017: 2019;
Abed-
soltan et al. 2023 Falkon (Rudi et al. (017), and randomly pivoted Cholesky (Chen et al. 2025).
The latter limitation is addressed by procedures for adapting the kernel to the training data. We describe such approach known the kernel Recursive Feature Machine (kemel-RFF) below.
2.2
RECURSIVE FEATURE MACHINES (RFMs)
The ability to learn telk-relevant features from data is key to building effective predictors (Damian ct al. 2022; Ghorbani et al. 2021 Abbe et al. 2022 RFM, introduced in Radhakrishnan et al. 22024), is algorithm that enables feature learning general machine learning models through a mathematical object known as the Average Gradient Outer Product (AGOP). Given a predictive
model f: Rd TR and data S = (xx11, (o(",,..,,(())) C RR, the AGOP is defined as
AGOP(F,S)= 1
va (T) E Rdddd
where V (x((()) denotes the gradient of the point (0)) The AGOP an estimate of the (un- centered) covariance of the gradients of and intuitively captures the subspace along which the predictor highly varies (Trivedi et al. 2014 Kpotufe al.] 2016). The RFM algorithm involves iterating between training a predictive model and using the AGOP of the trained model to select features and linearly transform input data. As such, feature learning through RFM can be viewed as a form of "supervised" PCA.
In the particular case of kernel machines, which have native mechanism for feature learning, RFM enables feature learning by adapting the kernel to the data. We describe the kernel-RFM algorithm below. Following the notation in the previous section, let (X, y) denote training inputs and labels, let K denote the kernel function, and À denote the regularization parameter. Letting M1 / and c > 0, kernel- RFM repeats the following two steps for iterations:
Step 1: F((x) K (Maz XM))a
a (XXX,, '
Step 2: M1+1 |AGOP(
Typical choices of T and practice are T 10 and E () (Radhakrishnan et 2024 Beaglehole et al., 2025; 2024, Mallinar et al. 2024). (In practice, we return the Jt with best valida- tion performance rather than returning fr-) When training labels, y, are multi-dimensional, we use the Jacobian of ff instead the gradient in Step (2) (i.e., averaging the AGOP over output dimen- sions). Kernel-RFM is particularly effective identifying low dimensional subspaces (or subsets of variables) relevant for prediction (Radhakrishnan et al. 2025 Zhu et al. 2025), making it a useful interpretability tool, will show in Section 5
XRFM ALGORITHM OVERVIEW
We now describe algorithm xRFM, which consists of the following two components (Fig.
(1) An improved kernel-Re -FMM termed leaf RFM, that is trained on the subset of data; A binary tree that splits the data into subsets, termed leaves, of maximum size (leaf size) that is independent of the number of training samples. Leaf RFMs are then trained on the subset at each leaf. The splits stratify data based on features relevant for prediction.
Together these components enable xRFM to perform local feature learning (learning different fea- tures on different subpopulations) and achieve (n log training time as well as O(log n) inference time given samples. We outline these two components in detail below.
3.1
LEAF RFM
Here, we describe the changes we made to the original kernel RFM algorithm from Radhakrish- nan al. (2024) to produce leaf RFMs. The kernel RFM model was primarily built using kernlss
3

---

A.
XRFM training
XREM inference
Training data: (x E X]
point:
x K
x' by >
Vi
x P2 P
us oi
U. <
z Pr < Oo
AGDP
Local
LFOF
Leof RRM
Leof RPM
Loot RFM
LOSTRFM
Leaf RFMs trained subsets size S
Prediction: 9
Overview of xRFM training and inference procedures. (A) xRFM is trained by split- ting the data along the median projections (denoted C1, C2) onto computed split directions (denoted v1, v1,02) 02). Data is split repeatedly into leaves, which contain most C' training samples. Leaf RFMs are trained on the data at each leaf. (B) During inference, test data is routed to the appropriate leaf RFM based on split directions. The prediction is generated by the selected leaf RFM.
Figure 1:
that were invariant to orthonormal transformations of data (those where K(x, =) = K(Ux, U2) for any orthogonal matrix U). Examples include the Gaussian kernel of the form K (xx exp(-Lifz 2 or the Laplace kernel of the form (xx z) exp(-LIlzz zl12). Such invariance does not effectively leverage special property of tabular data the fact that each coordinate can be independently meaningful. This property has been hypothesized as an explanation for the su- perior performance tree-based models, which compose functions of individual coordinates. over traditional neural networks on tabular datasets (Grinsztajn et al. (022).. To account for this special structure, we introduce the following modifications to kernel-RFM:
We tune over more general class of kernels Kp.g x²) = exp(-I|x: x a'llg/T.9) that are positive definite for ( < s 2 (Schoenberg) 1942 Theorems 1,5).
(2) We tune over using the full AGOP and just the diagonal the AGOP. The latter is known be theoretically grounded approach for coordinate selection (Zhu et al. (025), and introduces an axis-aligned bias that has been observed to match the structure of tabular data (Grinsztajn al. 2022).
In addition to the above changes, we also implemented opti- mizations to speed up computations for categorical variables and an adaptive approach for tuning bandwidth separately for data each leaf. Additional details regarding these changes and hyperparameter search spaces for leaf RFMs on TALENT, TTabArea--Litte and meta-test are provided in Appendix A
Normali0,
Izol (uy + st),
f(x)
lzo (2t (on am + +ams),
otherwise
AGOP matrices:
AGOP
Loaf2 AGOP
3.2 TREE- BASED DATA PARTITIONING
kB5
We now explain how xRFM builds binary tree structure to partition data into leaves on which leaf RFMs are trained. First, given a dataset S with n samples, we subsample points and train leaf RFM (referred as a split model) for one it- eration on this subsample. The split model serves to learn direction that can be used to partition the data into two sub- sets. To this end, we extract the top eigenvector, v, of the AGOP from the split model and create two subsets of the n datapoints: S {I S TT x Median(vT for S)} and S2 = {i E S v x Median(v") z for 2 E 5)}. We repeat this procedure on S, and S2 and their corresponding children until all leaves are less than maximum leaf size C. We finally train
414
Figure 2: Training xRFM on syn- thetic data where splitting the top AGOP direction enables xRFM to learn locally relevant features.
4

---

leaf RFM each of these leaves. This procedure is illustrated in Fig. TA and detailed in the 'TreePartition' procedure of Algorithm A.2
Note that our splitting procedure results in leaves of equal sizes (because of the split based on median), unlike tree-based predictive models, which can result in uneven splits. We typically split large datasets until leaves contain at most 60,000 samples. Leaf RFM hyperparameters are tuned using only on the validation data that falls into the leaf (as determined by splits down the binary tree). In the case that validation data at given leaf has too few samples, we further hold out subset of the training data at the leaf to use for validation.
Note that it is possible to have split data based on unsupervised splitring procedures (e.g., those based on random projections or using top cigenvectors of PCA) (Dasgupta & Freund 2008). A key advantage of our split approach that it groups together data points based on the features most relevant for prediction, as is captured by the top eigenvector of the AGOP. We found that splitting in this manner outperformed splits based on unsupervised approaches, including those described above. We include discussion and comparison of split methods in Appendix [ and Tables I E.. Prior works have also split data using the random forest procedure. For example, |Hollman et al. (2025) train TabPFN-v2 on the samples routed to a given leaf of tree in random forest.
The procedure in xRFM differs as it stratifies samples by projection onto a direction rather than individual coordinates and produces only one balanced tree instead of forest.
Advantages of tree splits over traditional methods for scaling kernels. Because of tree-based partitioning, xRFM is able to scale log-linearly in the number of sam- ples during training and logarithmically in the number of samples during inference. While this is a computationally appealing aspect practice, we note that there are number of existing methods for scaling kernel machines. Examples include the Nyström approximation (Williams 2000) and fast preconditioner methods for solving kernel regression (Ma & Belkin, 2017, Rudi et al., 2017; Ma & Belkin 2019: Abedsoltan al. 2023).
A.
Lapiace s80 Nom Nomaltzed R.
Mizcor saas pv
KBRt
Seeger
24
0.2
01
6.5
05
06
2'
04
18
8.
XRFM ized
Lo
major advantage of xRFM over these other methods and kernel RFM itself is the ability for tree-based partitioning to learn features local to subsets of data. Here, in Fig. 2 provide example clearly demonstrating this bene-
SEM (Onginal Normalzzdd R
fit. In this example, the function f TRd - R depends
22
RRM bettor
on coordinates 31, F1, 335 if xo > O and depends ordinates I11, if zo S 0. xRFM learns as the initial split direction (through the AGOP of the split model trained on random subset), then learns the rel- evant features for each case in separate leaf RFM mod- els. In contrast, if rumning kernel RFM from Radhakr-) ishnan et al. (2024), the model would simply return that (zo. T1, 43, , 2g, 211 113 are all relevant features with- out decomposing into the two cases.
xRMM
65
18
Figure 4: Comparisons of xRFM
with (A) kernel ridge regression and (B) the original RFM (Radhakrishnan et al. 2025) on TALENT regression datasets (metric is normalized R see Appendix A). Each point is dataset.
4 XREM PERFORMANCE
We now apply xRFM to three tabular data benchmarks: the TALENT benchmark YYe et al.) 2024). the TabArena-Lite benchmark (Erickson et al. 2025). and large datasets from the meta-lest bench- mark (Holmmüller et al. 2024). We use TALENT and TabArena-Lite to evaluate xRFM on datasets of various sizes between 500 and 150, 000 training samples
We use the meta-test benchmark to
1 'hee complexity for the applications of RFM itself scales only linearly, but splitting the data with the median at every tree node increases the complexity to O(n log n). We use the "Lite" version of TabArena, the full version is computationally expensive due to at least 9x
outer folds.
5

---

A.
B.
C.
dieg doodisatiin Lightul
Sinary Segaicaaioon
2.40
RgidomForest
Scalll SrcalFormel Frmmer
0,140
XGBoost BCN2
00:30
0.38
Better
MUP-PLR
50.15
FTT
des
0.125
Better
30
Xcbooott
LightGDM
Tabit
0.130
0.125
Better
MUP-PLL
CatBoog
0.120
XGBoost Lighraan
Cattooet
RealMiLP Jhbb
0.34
CatBeost
TobPFN-22
RANIMLP
0,120
RFM
0.115
&
JabPFN-u2
10
10
12
10
107
Tp
10
TDr
Aug. 1000 samp
Any. nar 2000 sumplee
Aeg. rortim samako
D.
E.
Regression
uaa classification
Biury classification
1992
1993
Deterr
bokkk a0 Pacte hasic:
1200
8
2
1100
1180
1002
1000
16-1
Meddan infernce time per 1093 curples 24
Medion eiemnree tie per loss samp es
Meiónn inferrce time per 1060 sumple
G.
Degresszon
H.
Multi clustifcation
Sirary cbstifiattoon
Derieu Tumed (T)
Detter
Dutter
1
Pedienn LOCS mampleeio
Medien 170 time 2200 sumples
Figure 3: Performance and runtime of xRFM the TALENT (Plots A-C) and TabArena-Lin benchmarks (Plots D-I). The y-axes of plots A-C are the shifted geometric mean of the error across all datasets in that category, while the x-axes inference time per 1000 samples for just the best hyperparameter configuration (meaning if a dataset has samples, we compute the training and inference time on the n samples divide the total time by m/1000). The y-axes in plots D-F are Elo, the main metric used in TabArena, and reflect the relative win-rate of each method, while the x-axes the median inference time per 1000 total samples. Plots G-I show normalized eтror versus median train time per 1000 total samples on TabArena-Lite. The TabArena-Lite plots D-I additionally show the default methods and plots D-F show the Pareto tradeoff curve for inference time versus Elo. The TALENT plots are (A) nRMSE over 100 regression datasets, (B) classification error over 80 multi-class datasets, (C) classification error over 120 binary classification datasets. The TabArena-Lite plots are (D,G) regression, (E,H) multi-class, and (F,I) binary datasets. For multiclass and binary classification plots H and I, the TabArena benchmark computes the error metric as log-loss and 1-AUROC, respectively.
the average over all datasets of the training plus
evaluate xRFM on larger datasets with at least 70, 000 and up to 500, 000 samples (a setting in which direct linear solvers become intractable, taking more than 40GB VRAM).
Performance evaluation measures. When measuring performance on these datasets, we use Root Mean Square Error (RMSE) for regression tasks (as is used in TALENT), and we use classification error (1 classification accuracy) for classification tasks. For TabArena-Lite. binary classification performance for individual datasets are measured by AUROC, while multi-class datasets are eval- uted with log-loss. To measure performance on aggregate over TALENT, we consider the following aggregation metrics individual dataset performance measures: Shifted Geometric Mean, Arith- metic Mean, and Normalized Arithmetic Mean. AarAnn---Ltte uses Elo (Elo 1967). a rating system that calculates the relative skill levels of each method based on the probability of winning when pairs of methods are compared on individual datasets (see Erickson et al. (2025) for how Elo is estimated in this benchmark). All other metrics are defined in Appendix

---

Results on the TALENT and TabArena-Lite benchmarks. The TALENT benchmark consists of 300 total datasets comparing 31 different supervised learning algorithms (Ye al. 2024). Among these 31 algorithms are strong. widely used predictive models including Gradient Boosted Decision Tree (GBDT) variants (like XGBoost, CatBoost, LightGBM), hyper-parameter optimized neural networks (RealMLP), and recent transformer-base foundation models (TabPFN-v2). The TAL- ENT benchmark consists of 100 regression tasks, 120 binary classification tasks, and 80 multiclass classification tasks. The number of training samples per task varies between 500 and 100, 000 and each task contains at least 5 input variables. TabArena-Lite contains 51 total datasets (13 regres- sion and 38 classification) comparing 15 different supervised learning algorithms including many of those in TALENT. The datasets in TabArena-Lan vary in sample size from 700 to 150, 000.
XRFM is the best performing method on TALENT regression tasks according to all aggregation metrics over RMSEs on individual datasets (Fig.3A). It is also the fastest method per configuration, although TabPFN-v2 does not incur a x00x overhead for hyperparameter tuning. xRFM is also com- petitive classification datasets (Fig. BB, C). Namely, xRFM is the third highest ranked method on classification tasks. xRFM also achieves win-rates significantly higher than 50% when compared to sate-oo-thee thethee-t neural networks and GBDT models the TALENT benchmark (Fig. F.T We additionally compare the performance (in Elo) and inference time of xRFM using the TabArena-Lite benchmark (Fig. 3). In particular, when compared to all default and tuned methods for this bench- mark, tuned xRFM is among the top three methods for regression while being orders of magnitude faster for inference. Indeed, xRFM lies along the empirical Pareto frontier, meaning that there is no method that dominates xRFM in both performance and inference time for these tasks (Fig. 3D). For classification tasks, xRFM is near the empirical Pareto frontier (Fig. 3 F). We additionally show that when comparing the mean normalized error metric (rather than ELO, which is based on winrate), tuned xRFM outperforms all other tuned and default methods except for the TabDPT foun- dation model on regression tasks, is among the top three methods on binary classification tasks, and competitive on multi-class classífication tasks (Fig. IG-I). We also compare xRFM performance against that of kernel ridge regression and standard RFM from (Radhakrishnan et al. (022)). xRFM is significantly better than these models (p-value < 10 10-4, Wilcoxon test, Fig.
Fig. 5 shows that the overall training+inference cost of xRFM initially scales super-linearly How- ever, xRFM is still reasonably fast at 60K samples, after which the log-linear scaling thanks to the tree-based data partitioning kicks in (Fig. F.2). Notably, xRFM is extremely fast on small datasets: it is at least two orders of magnitude faster than other methods when the dataset contains fewer than 3000 samples (we omit TabPFN-v2 in our comparison it could not be run all datasets due to restrictions on the allowable sample size, number of features, and output dimensionality).
Results on large datasets from the meta-test benchmark. To analyze xRFM performance on large datasets beyond those in TALENT and TabArena, we consider the 17 largest datasets in the meta-test benchmark from Holzmüller et al. (2024) (Table E.9
We compare the performance of xRFM to other models with results reported in the literature these 17 datasets (Fig, 6). These other models include GBDTs (XGBoost, LightGBM, CatBoost), and neural networks (ResNet, MLP, MLP-PLR (Gorishniy et al. 2022), RealMLP). We report the percentage improvement of each method over MLPs, following the procedure in Gorishniy et al.) (2025); Qu et al. (2025). Similar to the results on TALENT, xRFM is best on regression tasks, and second best on classification tasks. All results are presented in Tables E.9 E. 10 For meta-test, we search over slightly larger regularization parameters and slightly smaller bandwidth parameters to adapt to the much larger sample sizes of these datasets.
5 FEATURES LEARNED BY XRFM ON TABULAR DATA
An advantage of xRFM is that it immediately provides means of identifying features relevant for prediction (without stacking on any additional interpretability methods). Namely, we can extract learned AGOPs of leaf RFMs and visualize the features they select.
We use two approaches for identifying features selected by the AGOP. The first is to identify the elements along the diagonal of the AGOP with highest magnitude. By definition, these entries in- dicate how much a leaf RFM's predictions vary when perturbing a given coordinate. As such, they a natural measure of feature importance. The second approach is to examine the loadings onto
7

---

A.
B.
C.
Regression
Mutti Class Ciassifiation
Sinary Clsssl cation
102
10
10
10
tire
107
10f
Relled
DAR
NJP
10"4
10a
10
7atal number of sampples
Total number of campeeee
Total rumber of sampliss
Figure 5: Total training and inference time for the best hyperparameter configuration as a function across the TALENT benchmark. Curves
of the number of samples trrining-validation-essiin indicate piece-wise linear fit to measures each dataset (shown as points). (A) Results across 100 regression tasks. (B) Results across 80 multi-class classification tasks. (C) Results across 120 binary classification tasks.
A. Improvement over MLP on meta-test regression
B.
Improvement over MLP on meta-test classification
LightGaw
XGltoest
Cathooot
MUPPIR
Reslltt
MLP
8
MLP-PLR
Ren
De
xRFM
xAPM
Improvement
RealMEP
Improvement
-125
-5
125
-125
25
125
Improvement % thigher is better)
Improvement % (higher is better)
Figure 6: Performance comparison across the 17 large datasets from meta-test (70,000-500,000 samples). (A, B) Percentage improvement over MLP error on (A) regression and (B) classification datasets. Average percentage improvement is denoted as diamond. Points denote individual dataset results (points are jittered for visibility).
the top eigenvectors of the AGOP. This approach allows us to identify joint effects of feature pertur- bations on the prediction. Namely, if coordinate of the top eigenvector is positive and coordinate is negative, then increasing one of these features and decreasing the other changes the prediction.
We use both approaches above understand what features learned by xRFM tabular data from scikit-learn and meta-test (Fig. 7). As examples, we study the AGOPs for four such datasets: (1) California housing a regression task for predicting the average price of house, (2) Covertype a multiclass classification task for identifying the dominant tree species in a given location, (3) NYC Taxi Tipping a regression task for predicting the dollar tip amount in a taxi ride, and (4) Breast cancer a classification task for identifying malignancy from features of biopsy images.
As the California housing and breast cancer datasets contain fewer than 50k samples (the parameter we used for leaf size), we visualize a single AGOP. For covertype, we visualize one of the AGOPs from a leaf RFM (for this dataset, the leaf RFM AGOPs generally indicate the same pattern), and for taxi tipping we visualize several leaf RFM AGOPs. Upon visualization, it is apparent that AGOPs indicate low rank structure: either they highlight a subset of coordinates relevant for prediction (Fig. 7A, B) they have decay in the cigenvalue spectrum (Fig. 7C)
In Fig. 7 and B. we list feature names for the features with highest diagonal AGOP entries (darker shades of red indicate higher values). In all cases, we find that AGOP identifies sensible features for prediction. For the California housing dataset, xRFM identifies longitude, or east-west location, the most important feature for predicting the average price of house. Given that beach fronts (and major cities) in California are typically located to the west, the importance of house longitude is consistent with the hypothesis that homes closer the beach more expensive on average. For
8

---

A.
California Housing
Cavertyne
Breast Cancer
2007 Sp55ie
40
675
Legest
1- Hivevttee
deahoaes
Ringera
Top Eigenvector of Breast Cancer AGOP
B.
Texi Tipping (1)
Taxi Tipping (2)
Taxi Tipping (3)
05
Stdd
Coordaaai
Figure 7: Interpreting xRFM through the AGOP of its constituent Leaf RFM models. (A) Exam- ining the most important features for xRFM trained on California Housing (price prediction) and Covertype (dominant tree species prediction) datasets, based on the magnitude of diagonal entries. (B) Examining the features identified across three different Leaf RFM models for the NYC Taxi Tipping dataset. (C) Examining features learned for Breast Cancer detection from processed FNA imaging. The spectrum of this AGOP is plotted and the top eigenvector is shown in a bar plot. The most positive and negative entries of this cigenvector are boxed.
the Covertype prediction dataset, the example AGOP from a leaf RFM shows that elevation, distance roadways, and distance firepoints are the most important features. This finding consistent with the hypothesis that elevations with different climates and the existence of fires roadways can significantly affect viability of different tree species. For the taxi tipping dataset, we observe that leaf RFMs identify varying local features. For example, one leaf RFM (denoted Taxi Tipping 1 in Fig. B) selected pickup location as an important feature, while this feature was less important for different leaf RFM (Taxi Tipping 3). Furthermore, fare code and the MTA tax have varying feature relationships at each leaf: in leaf RFM and 2, the fare code and MTA tax has neutral or synergistic effects the prediction value, while for leaf RFM 3, increasing fare code has the opposite effect prediction as increasing MTA tax (shown as a blue square in Fig. 3).
For the breast cancer dataset, the most important feature found by xRFM is the mean number of concave points in the biopsy image, which has been shown to be significant indicator malignancy (Narasimha al. 2013). When examining the top eigenvector of the AGOP, we find that standard error in the compactness measurements of cell nuclei is the feature with highest importance and has an opposite effect on malignancy concavity (Fig. 7C). Our method suggests that benign cells have less uniform compactness than malignant cells.
DISCUSSION
Tabular data has historically provided a proving ground for the development of novel predictive models. However, even as the last years have seen spectacular improvements in language pro- cessing and computer vision, the progress in tabular data has been far more modest. While this may changing with the developments such as hyperparameterroptiiiiii ReaLMLP (Holzmüller et al.) 2024)), and very recently, pre-trained transformer- tabular net- works, such as TabPFN-v2 (Qu al. 2025), there still large scope for improved tabular data prediction.
neural networks (e.g.,
Kernel machines provide a powerful conceptually and computationally elegant approach for pre- dictive modeling one simply transforms data with nonlinear feature map and then performs lin- ear regression. Yet, historically, these methods have not out-performed Gradient Boosted Decision Trees (GBDTs), while also being significantly more difficult to scale. There two key limitations of standard kernel machines: (1) standard kernel choices lack adaptivity to features relevant for pre- diction, and (2) naive implementations scale quarrailadat in the number of training samples,

---

making them difficult to train on larger datasets (past 70k training samples), Significant progress in overcoming limitation (1) was made in recent work (Radhakrishnan et al. 2024). ) which introduced Recursive Feature Machines (RFM) to enable feature learning in a range of models, including ker- nel machines. In fact, feature learning through RFM can be provably more sample efficient than standard kernels (Zhu et al. 2025). There has also been significant work addressing the limitation (2) including (Williams & Seeger 2000; Ma & Belkin 2019) Rudi et al. (017). to reference a few.
In this work, we have taken different, tree-based approach to overcome these limitations. Our approach enables scalability to large datasets (log-linear training time and logarithmic inference time) while also leveraging the strength of RFM in identifying and exploiting local features. Indeed, the ability of xRFM to learn different features for different subpopulations of data through AGOPs at leaves is useful for characterizing heterogeneity across large datasets. Building on this core novelty and other algorithmic optimizations, we discuss a number of potential improvements and follow ups below.
A direction for future improvement of xRFM is to explore the use of iterative kernel solvers to train leaf RFMs. By using early stopping. these methods could obviate the need for tuning ridge regularization. The use of the adaptive binary tree structure to split data also opens up new avenues for exploration. For instance, it would be interesting to characterize the tradcoff between leaf size and performance and to develop new methods for determining when to stop splitting based statistics of leaf data. It might also be useful to consider alternative bandwidth selection methods for each leaf, for example by optimizing for bandwidths directly as in (He al.) (023). Lastly, it would be important to understand the effectiveness of AGOP as an interpretability mechanism, particularly in comparison other widely used methods such as Shapley values (Lundberg & LLe|[2017] 2017). gradient-
based importance scores (Sel varaja el al., 2017: [Shrikumar et al. 2017). and feature importance scores for tree-based models (Breiman et al. 1984).
Overall, we have shown that xRFM is an effective algorithm for inference from tabular data that scales essentially unlimited data sizes and achieves performance exceeding or comparable to the current state-of-thee-rt. It combines the advantages of tree-based methods with the power and elegance of feature-learning kernel machines. We envision that xRFM will be used for both high- performing predictive modeling and uncovering heterogeneous structure in large-scale tabular data.
ACKNOWLEDGMENT
We would like to thank Jingang Qu, Ingo Steinwart, Tizian Wenzel, and Gaël Varoquaux for rele- vant discussions. We thank Charles Durham for contributing the KerMac library, which led to drastic speed up in kernel evaluations and gradient computations. We acknowledge support from the Na- tional Science Foundation (NSF) and the Simons Foundation for the Collaboration on the Theoreti- cal Foundations of Deep Learning through awards DMS- 20310000300000000 883 and #8 1439 as well as the TILOS institute (NSF CCF-2112666 the Office of Naval Research (ONR N000142412635 (HR001125CD00) 125CD020). This work used the programs (1) XSEDE (Extreme science and engineering discovery environment) which is supported by NSF grant numbers ACI- and (2) ACCESS (Advanced cyberinfrastructure coordination ecosystem: services & support) which is supported by NSF grants numbers #2138259, #21 38286, #21 #2138307, #2137603, and #2138296. Specifically, we used the resources from SDSC Expanse GPU compute nodes, and NCSA Delta system, via alloca- tions TG-CIS220009. This work was performed using HPC resources from GENCI-IDRIS (Grant 2024-AD01101 1014644R1..
and DARPA
REFERENCES
Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-stairease property: necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Po-Ling Loh and Maxim Raginsky (eds.), Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pp. 4782-4887 PMLR, 02-05 Jul 2022. URL https: httpss///oooceedeee http://eeeedddoooo.o.eesses////a 78 abbb222.b html
mlr prwss
Amirhesam Abedsoltan, Mikhail Belkin, and Parthe Pandit. Toward large kernel models. In Pro- ceedings of the 40th International Conference on Machine Learning, ICML/23, JMLR. 2023.
10

---

Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American mathematical sociery, 68(3):337-40) 404, 1950.
Daniel Beaglehole, Peter Súkeník, Marco Mondelli, and Mikhail Belkin. Average gradient outer product as a mechanism for deep neural collapse. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomczak, and C. Zhang (eds.), Advances Neural Information Processing Systems, volume 37, pp.
Curran Associates, Inc., ctw.://ddd.aaseeeurrs.......... http://paeeeeiieee/peee//2/2222 fil Ff12ooocssscaaee3335566622233220
2024. URL
pdf
Daniel Beaglchole, Adityanarayanan Radhakrishnan, Enric Boix-Adserå, and Mikhail Belkin. To- ward universal steering and monitoring ai models, 2025. URL httpp:://axxi.... /
abs/ 20a//2502 03502.03700
Bernd Bischl, Giuseppe Casalicchio, Taniya Das, Matthias Feurer, Sebastian Fischer, Pieter Gijs- bers, Subhaditya Mukherjee, Andreas Müller, László Németh, Luis Oala, Lennart Purucker, Sahithya Ravi, Jan N. van Rijn, Prabhant Singh, Joaquin Vanschoren, Jos van der Velde, and Marcel Wever. Openml: Insights from 10 years and more than thousand papers. Patterns, 6(7), 2025. doi: 1101///..atter..//00 2015,
Lco Breiman, Jerome Fricdman, Richard A. Olshen, and Charles Stone. Classification and Re- gression Trees. Chapman and Hall/CRC, 1st edition, 1984.
Tianqi Chen and Carlos Guestrin. Xghoost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Yifan Chen, Ethan N Epperly. Joel A Tropp, and Robert J Webber. Randomly pivoted cholesky: Practical approximation of kernel matrix with few entry evaluations. Communications on Pure and Applied Mathematics, (845):995-0041,2000
Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. In Po-Ling Loh and Maxim Raginsky (eds.), Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Leurning Research, pp. 5413-5452. PMLR, 02-05 Jul 2022. URL http:::tt///.m httt:tt:::///prreeeee mgtp://///ooeeedn.......... mng.on/o..///s/pooooooo..nn..n.. Samian22a.. html
Sanjoy Dasgupta and Yoav Freund. Random projection trees and low dimensional manifolds. In Proceedings of the fortieth annual ACM symposium on Theory of computing, pp. 537-546, 2008.
Arpad Elo. The proposed uscf rating system, its development, theory, and applications. Chess life, 2(8))222-247, 1967.
Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. Tabarena: living benchmark for machine learning on tab- ular data. arXiv preprint arXiv:2506.1679) 2025.
Manuel Fernández-Delgado, Eva Cernadas, Senén Barro, and Dinani Amorim. Do we need hun- dreds of classifiers solve real world classification problems? The Journal of Machine Learning Research, 15(1)33133-318), 2014.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural net- works outperform kernel methods?* Journal of Statistical Mechanics: Theory and Experiment, 2021012) December 2021. ISSN 1742-5468. doi: 02.1088/1142-54468aaa http://dxx www. ottp////.....ooo/.oooo/o///// 1088 1111211222-22 5468/ac3a5.
URL
Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 55:24991-25000 2022.
Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. Tabm: Advancing tabular deep leamn ing with parameter-efffcient ensembling, 2025. URL httpp::/ htttp://arxivv.ort www.w..og....//......./ 24210

---

Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. still outperform deep learning on typical tabular data? In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.). Advances in Neural Information Processing Systems, volume 35, pp. 507-520. Curran Associates, Inc., 2022. URI. https::/pproceddinn neuri iss cc /acc/ppper__.. lameer_flleee / 7 Sccto Paper Dasteertaattttsss. and Benchmarks.
Why do tree-based models
pdf
Fan He, Mingzhen He, Lei Shi, Xiaolin Huang, and Johan A. K. Suykens. Enhancing kernel flexi- bility via learning asymmetric locally-adaptive kernels. arXiv preprint arXiv:2310.05236, 2023.
Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirreesterr, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319-326, 2025.
David Holzmüller, Léo Grinsztajn, and Ingo Steinwart, Better by default: Strong pre-tuned mips and boosted trees tabular data. Advances in Neural Information Processing Systems, 37:26577- 26658, 2024.
Samory Kpotufe, Abdeslam Boularias, Thomas Schultz, and Kyoungok Kim. Gradients weights improve regression and classification. Journal of Machine Learning Research, 17(22):1-34,
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Siyuan Ma and Mikhail Belkin. Diving into the shallows: a computational perspective large-scale shallow learning. Advances in neural information processing systems, 30, 2017.
Siyuan Ma and Mikhail Belkin. Kernel machines that adapt to gpus for effective large batch training. Proceedings of Machine Learning and Systems, 1:360-373, 2019.
Neil Mallinar, Daniel Beaglehole, Libin Zhu, Adityanarayanan Radhakrishnan, Parthe Pandit, and Mikhail Belkin. Emergence in non-neural models: grokking modular arithmetic via average gradient outer product, 2024. URL
arxiv. wwwpexiw.wogggaaaae org/abs/2401 /0t.////2//200299999//
Aparna Narasimha, Vasavi, and Harendra ML Kumar. Significance of nuclear morphometry in benign and malignant breast aspirates. International Journal of Applied and Basic Medical Research, 3(1) 3(1):22-26, 2013.
Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. Tabicl: A tabular founda- tion model for in-context learning on large data. arXiv preprint arXiv:2502.05564, 2025.
Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Mecha- nism for feature learning in neural networks and backpropagation-ffee machine learning mod- els. Science, 38366690) 2024. doi: 10. 122//sciincce..d55339.. URL https:
www. www.wac.enne... wttp://e..ood////////////// /
/1t// www.ciiecceccic5c3999.. adi5639
Adityanarayanan Radhakrishnan, Mikhail Belkin, and Dmitriy Drusvyatskiy. Linear recursive feature machines provably recover low-rank matrices. Proceedings of the National Academy
of Sciences, (2(13)):241122511 2025. doi: 10. 141112222 URL https:
/ //www. //www.pnas. pnws.
/abs 10. /1/3////a..... 2411 12411334 325122
Alessandro Rudi, Luigi Carratino, and Lorenzo Rosasco. Falkon: An optimal large scale kernel method. Advances in newral information processing systems, 30, 2017.
I. J. Schoenberg. Positive definite functions spheres. Duke Mathematical Journal, 9(1):96 108, 1942. doi: 10. 122-2121500012777944--0000000000 2- 721-2-2-2722---0000
URL httpps///dae www.aii...or.ooooooooooooooooooe ort./1..11/111111331
Bernhard Schölkopf and Alexander J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, 2002.
12

---

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based local- ization. In Proceedings of the IEEE international conference computer vision, pp. 618-626, 2017.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pp. 3145- 3153. PMIR, 2017.
Shubhendu Trivedi, Jialei Wang, Samory Kpotufe, and Gregory Shakhnarovich. A consistent esti- mator of the expected gradient outerproduct. UAA, pp. 819-828, 2014.
Christopher Williams and Matthias Seeger. Using the nyström method speed up kernel machines. Advances in neural information processing systems, 13, 2000.
Han-Jia Ye, Si- Yang Liu, Hao-Run Cai, Qi-Le Zhou, and De-Chuan Zhan. closer look at deep learning tabular data. arXiv preprint wwxiv:240.000956 2024.
Libin Zhu, Damek Davis, Dmitriy Drusvyatskiy, and Maryam Fazel. Iteratively reweighted kernel machines efficiently leam sparse functions, 2025. URL httphhhtt///aaat ULL/atttss//arxx..ooo///b////
08277
A METHODS
A.1 ADDITIONAL LEAF-RFN RFM MODIFICATIONS
Below, we provide further detail the RFM modifications made to implement leaf RFM.
(I) We tune over more general class of kernels K (p x') positive definite for < S I (Schoenberg 1942
exp(-II> >211g// ///1//9)) that are Theorems 1, 5). We typically
search over E (00.7, 1.4) and 2 A..I
(2) We tune over using the full AGOP and just the diagonal the AGOP. The latter is known to be theoretically grounded approach for coordinate selection Zhu et al. (2025), and introduces an axis aligned bias that has been observed to match the structure of tabular data (Grinsztajn al. 2022).
We also make the following changes to improve leaf RFM runtime on tabular data and to allow xRFM to adapt to the variance of input variables at each leaf:
(3) We speed up computation for kernels with & = on categorical variables taking e values by precomputing possible kernel entries follows. We restrict the AGOP matrix to be block diagonal with a block for the c e entries corresponding to the categorical variable. Letting M, E Rexx denote this block and e; {0, 1) denote the one-hot embedding of the categorical variable when it takes on value i, we precompute M.!2 (c: - ee, for all
i,j E [c].
(4) We tune over whether or not to use adaptive bandwidth, which involves separately scaling
T. at each leaf RFM by (Median(]az - x' x2)1 for of a' every leaf independently. Adaptive scaling allows xRFM to adapt to the variance of covariates at different leaves.
A.2 METRICS
Following Holzmüller et al. (2024), we report the shifted geometric mean of the error, which is the geometric mean of the error after shifting by small value to prevent over-sensitivity to datasets with small enrors. This metric is defined as follows.
Definition For a given set of errors on a benchmark EI,...,EN. the shifted geometric mean (SGM)) with parameter takes value:
N log(e + Ei)
SGM. exp
N
13

---

In this work, as in Holzmilleer et al. (202), 0.01.
For regression tasks, normalized Root-mean-ssqurre-rro (nRMSH), which is defined as follows.
Definition 2. The normalized Root-mean-ssquar--ero is defined as,
N
N (3) 8)2
1 N
nRMSE oyy
(v. - 0i) Σ
where By
When refer to other normalized metries, such those used in Table E.4 Table E.5 Table F.6 Table E.7 Table E.8) we are min-max normalizing emoos across methods for each dataset (see be- low).
Definition 3. The normalized error Ej for method on given dalaset, where the un-normalized error is Ej, has the form:
Emin
F,
Eminn min Ek; Fmax max Fot
Fmua
Emnnn
A.3 HYPERPARAMETERS
Hyperparameter
TALENT
TabArena-Lite
Meta-test
Bandwidth
log2/(1, 200) [constant):
(0.5, 200) (constant]
log(((0.4880)
Bandwidth Mode
[constant, adaptive)
Categorical Transformations
(one hot
(one hot)
(ordinal encoding, one hot)
Diagonal
[False. True}
(False. True] 1.1 U(0.7,,.4
[False, True)
Early Stop Multiplier
1.1 1(0.7,,14)
1.1
Exponent !
U((077,
Kernel Type
(80%: Kp.g. 20% K2.4
(80%: Kp.q- 20% K24)
(80%: Ky.q, 20%x
P (when kernel type is K Kp.9)
U(g.9 0.8(2 - Q))
U(g q 0.8(2 g))
U(q, 0.8 9))
Normalization
(standard]
(standard)
{standard]
Regularization
1002((10 " ,,1)
log24(10 10)
log44(00 .0)
Refill size (Nvai)
1500
1500
1500
Min. split subset size (C)
60,000
60,000
60,000
Table A.1: Search spaces for xRFM on the TALENT (Figure 3). TabArena-Lite (Figure 3 and Meta-test benchmarks 6F)
Data pre- rre-processing. For the TALENT benchmark, each method's hyperparameters are tuned af- ter fixing a single data normalization and categorical encoding scheme. For xRFM, kernel ridge regression, and standard kernel RFM, one-hot encode categorical features and z-score input ordinates separately. For meta-test, methods also tune over choice of ordinal or one-hot encoding of categorical variables. On meta-test, also normalize data by z-scoring input coordinates separately prior to tuning xRFM parameters.
Details of various methods. For scaling kernel ridge regression and the standard kernel RFM to large TALENT datasets, we used Eigenpro-2 (EP2) (Ma & Belkin 2019) that was initialized with the coefficients obtained from directly solving RFM on a random sample of 70,000 points. To tune the optimization hyperparameters for EP2, we tuned the model parameters for 100 trials on random 70,000 sample subset, then for the final run, we used the tuned hyperparameters (and kernel RFM the AGOP from the best iteration). For TabPFN-v2, we used the code provided from the benchmark github (https: http://gttttba.u com/ LAMDA LAw.AATaaulasTTA.L......
which subsampled to 10,000 samples to avoid out-of-memory issues. The provided code did not apply TabPFN-v2 to datasets with more than 10 classes. For efficient Lg kernel computations on GPU we used the KerMac library (https:: // twwb.com/Kerre.....eeoooooaaa Kerne Mach nes ke rmac).
14

---

Split method and TabPFN-v2 embeddings experimental details For the experiments evaluating choices of split methods, we tune among hard-routing (0 temperatures) and 15 additional tempera- tures at spacings in log space from 0.025 to 4.5, We use minimum split subset size of 40,000 samples. For the experiments on learning from embeddings of TabPFN-v2, we tune hyperparameterss for xRFM (on embeddings
the tabular data itself) by sampling 50 random configurations of the TabArena search space. For preprocessing, we treat the embeddings from TabPFN-v2 as numeric inputs and apply standard scaling each coordinate.
A.4 ALGORITHMIC DETAILS
Note AGOP computation For computing gradients of the predictor training data (for AGOP computation), we omit the contribution to the gradient from the kernel evaluation between each training point and itself. This is because the kernel function is often not differentiable (e.g. Laplace kernels) when evaluated for two identical points.
Algorithm A.1 Leaf RFM
Require:
(() E Rd MEE E Raxc Train data
Xval E Ruxd 1/val Im
Validation data
K(. p): Kernel parametrized by bandwidth L E RR+ and exponent p E (0,2)
I E Z
Number of iterations
À E Rt: Ridge parameter
e E + Stability parameter
.dsa:: Boolean indicating whether to diagonal of AGOP only
adapt.bandwidth: Boolean indicating whether to adapt bandwidth
Mo Laxd
X (,())...,,_.1)))) E Rx
if adapt. bandd.bandwidth then
++ AdaptBandwiddhh()
Adapt bandwidth if enabled
end
for 1. 0, 0 do
use_ then
XM + X @ diag(M)))/2
Solve a such that (K(M,, XM3 + Al)a = y
(() K(x o diag(AM, XM)aa.
denotes element wesse multiplication
else
NM XM!/2
Solve a such that (K(XM, XM) + Al)a =y ((0 K(M,/² /2 XM)a,
end if
Compute E, +- Error ((t) Xvals 3vval)
Validate model
M111 " n v (z( ())))) (10) E Rdxd
Feature matrix (AGOP)
computation
M M,+1/(E + m Mr (b.
Normalize feature matrix
end for
1" <- arg min, Fx return aur, Mr-
KRR coefficients: ac, feature matrix: Mc- from best iteration val. set
15

---

Algorithm A.2 TreePartition
Require:
D {(x(11) ,y(1), (a y (()) Full dataset with x:(1) IR y IR
K(-, L, p): Kernel parametrized by bandwidth L E RT and exponent p E (0, 2]
N E 2t: Number of sample points for Leaf RFM
E t Maximum leaf size
À E Rt: Ridge parameter
function TREEPARTITION(D)
[D] <
L then
return Leaf node with dataset D
end if
Sample NN points S = {x((1)) y (I(aN), go(l))} from D
X, = [x(()) x(ax) E R Nxd
y. = ())))) ......y/a)) x E Nxx
Solve such that (,, + a1)==
> Fit Leaf RFM on sampled data
Define predictor ((x) K(x, X3)a
Compute AGOP: M + EN
V
Rdxd
Extract top eigenvector of M
b Principal direction
for 1..... (D] Compute median projection: m + Median (p()) ((PP)))
Project all data points: p(*) + vj (1
Split dataset:
Den + {(x((t) (0) E E V ( (0) < m
Dright f- ((x(1)) (10) E ED: 10 > m)
left.child +- TreePartitio
Recursive call Recursive call
sighl TreePartition, D Dmgit)
return Internal node with splitting vector t1, threshold m, and children end function
Algorithm A.3 Route (find the leaf that contains a point)
Require:
T a (possibly trained) tree whose internal nodes store
splitting vector e IRd
threshold m
x E gd query point
function ROUTE(2, 7) ..root
Initialize current node from the tree
while is an internal node do
if o x < r.threshold then r - r.left child
Check projection is less than (median) threshold
else
r.ril
end if
end while
return r end function
D r is now the leaf node that contains x
16

---

Algorithm A.4 xRFM (Iraining)
Require:
Derain Dval training and validation sets
K(:; T,, p) kernel (bandwidth T., exponent
TreeHyp {N, Imaxx Asplity hyyee-paaaaeeeet for TREEPARTITION
LealHyp {T, Ateafo E, .use_diag) hyper-parameters for LEAFRFM
function XRFM-FIT(DLrain Dval)
T +- TREEPARTITION(ILanr K, TreeHyp)
Alg. A.2
for all leaf node é LEAVES(T) do
Dirai <- data stored in &
Diall <- (1r, E Daal ROUTE(I, T) E}
(ag, M2) LEAFRFMOD Dini K, LeafHyp)
Alg. A.1
Define predictor fr(x) according to (a,, M2) and store it
end for
return
b Tree whose leaves now carry trained predictors
end function
Algorithm A.5 xRFM (inference)
Require:
trained returned by XRFM-FIT (Alg, A.4
x E Rdd test point
function PREM-PREDICTTT. 8 + ROUTH(I, x T) - f(( return 7
Alg. A.3
Leaf predictor stored in
end function
17

---

B OPTIMIZING THE XRFM TREE STRUCTURE
Here discuss two avenues for optimizing the tree structure of xRFM: (1) ensembling Leaf RFM models at prediction time, and (2) selecting the method for extracting split directions.
We may expect for some datasets that the target function being estimated exhibits relatively simple parametric form, e.g. a fixed quadratic function of the input features. In these cases, or in cases where the choices of split are poor, splitting the data across leaves may worsen performance. To account for this, we also consider routing test samples at prediction time to multiple leaf RFM models, then forming the prediction by taking a weighted average of the predictions across these models. We implement this weighted averaging and demonstrate its improved performance, at the expense of larger evaluation times and slightly larger training times.
Temperature tuning (TT). Our ensembling procedure works by assigning the weight to each leaf equal to the product of node-wise weights at each node along the path to that leaf. The weight for node i, with a given split threshold e and split direction it. is equal a - where 3 is B. scaling factor chosen for each node. We set the scaling factor 3, B IQRR, where B is temperature parameter and TQR, is the inter-quartile range of value for v. where are all the training samples whose paths to their leaves include that node. As this temperature parameter can be tuned independently of training the leaf RFM models, we simply fit all the leaf RFM models individually, then optimize the temperature parameter by grid-search on the validation set. Note that the "hard-routed" xRFM model, which deterministically routes each test point to a single leaf, is equivalent to this "soft-routed" xRFM model with B O. For additional inference- speed-up, we allow only the top 8 highest weighted leaves to participate in the ensemble, and re- re-normalize the weights so that they sum to across participating leaves for each test point.
Split methods. addition to using the top eigenvector of the AGOP on a split model, we consider two additional splitting choices: (1) Unsupervised splits (PCA) using the top principal component of the training samples at each node, and (2) Random forest importances (RF), where the coordinate with maximal information gain is selected at every node.
We evaluate temperature tuning and split methods on the large meta-test datasets where at least 2- 3 splits are required for every dataset using minimum split size of 40,000 samples. Our results show that supervised splits outperform unsupervised ones, (Tables E. E.2). Further, while AGOP splitting gives the fastest method, using AGOP or RF splitting along with Temperature tuning gave the best test error.
C META-FEATURE ANALYSIS
We examine the effect of dataset selection and metafeatures the relative performance of xRFM First, we examine whether xRFM is better on the TALENT or TabArena datasets. In Fig[E.14 we show that xRFM is the best method on regression using the TALENT baselines and tuning proce- dure, even when restricted to the datasets in TALENT that are also in TabArena. Further, xRFM actually improves the best overall method on binary classification on the TabArena subset, versus the second overall method to TabPFN-v2 on the remaining, non- TabArena subset. Thus, the dif- ference in results between TabArena and TALENT cannot explained by dataset selection alone. In particular, we show xRFM is the best overall method on regression and binary classification on the subset of datasets in TALENT that also in the TabArena benchmark (new Tables E. and E.15). Other factors that could contribute to the explanation Optuna), cross- volssavali,,,on,, and stronger (more expensive) baselines in TabArena. Morcover, for clas- sification, TabArena uses different metrics (AUC and log-loss), where the log-loss could potentially be improved using post-hoe calibration.
the use of random search (instead of
In Fig F3 we report the effect of three metafeatures the normalized errors of xRFM and other top methods. We do not see an obvious trend in the relative performance of xRFM across sample size, feature dimension, or the fraction of features that are categorical.
18

---

D COMBINING XREM WITH OTHER MODELS
We also analyzed to what extent xRFM could be combined with other models to give effective ensembles. In Table E.17 we show that performance can be further improved on the TALENT benchmark by including xRFM in a portfolio with CatBoost and XGBoost. In particular, we select the best of xRFM, CatBoost, and XGBoost on the validation set and evaluate this best model on the test set. Note the validation results for TALENT were only provided for 200 of the 300 datasets for CatBoost.
19

---

E ALL RESULTS
Splitting method AGOP + TT
Dataset
AGOP
PCA + TT
RF+TT TT
Airlines Airlines.DepDelayy 10M
0,9792
0.9782
0,9785
0,9785
Allstate_Claine Claims_Severitg
0.6480
0.6509
0.6526
0.6521
black.friday
0.6879
0.6879
0.6891
0.6877
Buzzinsocialmedia Twitter
0.2300
0.2290
0.2137
0.2138
nyc-taxi-green-dee-222
0.5268
0.5250
0.6283
0,5256
wave _evegen
0.0024
0.0024
0.0024
0.0024
Yolanda
0.7852
0.7790
0.7824
0.7824
Number of wins:
4
2
2
Shifted geometric mean:
0.3446
0.3440
0.3499
0.3411
Arithmetic mean:
0.5514
0,5503
0.5638
0.5489
Average fit time [s]:
7755
8287
7765
7831
Average eval time [s]:
77
187
183
174
Table E.1: Evaluation split methods on large regression datasets from meta-test. We consider three methods for choosing split directions AGOP, Principal Component Analysis (PCA), and Random Forest criterion (RF). We also evaluate ensembling leaf RFM models using the temperature tuning method described in Appendix B
Splitting method AGOP TT
Dataset
AGOP
PCA + TT
RF+ TT
airlines
0.3318
0.3309
0.3307
0.3289
covertype
0.0254
0.0250
0.0262
0,0252
Diabetes 130US
0.3856
0.3854
0.3855
0.3854
dionis
0.0905
0.0905
0.0905
0.0905
Fashion-MNN -NIST
0.0880
0.0880
0.0880
0.0880
Higgs
0.2597
0.2551
0.2548
0.2546
jannis
0,2719
0.2718
0.2725
0.2714
KDDCupp9
0.0003
0.0003
0.0002
0.0002
kick
0.0972
0.0972
0.0972
0.0972
MiniBooNE
0.0539
0.0539
0.0542
0.0545
numerai28.6 6
0.4738
0.4738
0.4738
0.4738
perto-seguro
0.0382
0.0382
0.0382
0.0382
Number of wins:
8
Shifted geometric mean:
0.1159
0.1156
0.1159
0.1156
Arithmetic mean:
0.1764
0.1758
0.1760
0.1757
Average fit time [s]: 6410
6770
6535
6510
Average eval time [s]:
95
136
127
126
Table F.2: Evaluation of split methods large classification datasets from meta-test. We consider three methods for choosing split directions AGOP, Principal Component Analysis (PCA), and Random Forest criterion (RF). We also evaluate ensembling leaf RFM models using the temperature tuning method described in Appendix B
20

---

Dataset
Task
RFM
xRFM
Credit-c
Multiclass
0.1395
0.1072
Rain-in-Austraiia
Multiclass
0.0535
0.0629
SDSS17
Multiclass
0.0108
0.0005
accelerometer
Multiclass
0.0273
0.0190
dessomer-satisfaciion--eeee
Binary Classification
0.0152
0.0146
dabetes-130-ae -hosssosspial walking-activinn
Binary Classification
0.0430
0.0448
Multiclass
0.0627
0.0167
Arithmetic Mean
0.0503
0.0379
Evaluation of xRFM vs. original RFM on the largest datasets from TALENT, where at least one xRFM split required. We show the normalized errors (across all 32 methods). Datasets are ordered by training set size.
Table E.3:
Method
Rank
Score
Norm. Score
Top-1 (%)
Top-3 (%)
Top-5 (%)
Top-8 (%)
SGM,
xRFM
4.70
0.311
0.036
20.0
56.0
69.0
84.0
0.311
TabPFN-v2
6.55
0.323
0.067
20.0
45.0
57.0
72.0
0.323
CatBoost
7.79
0.336
0.053
7.00
24.0
41.0
68.0
0.336
RealMLP
8.20
0.329
0.076
8.00
21.0
37.0
60.0
0.329
ModernNCA
9.39
0.365
0.097
14.0
30.0
38.0
57.0
0.365
LightGBM
9.91
0.353
0.062
6.00
16.0
34.0
52.0
0.353
XGBoost
10.6
0.355
0.071
2.00
12.0
32.0
47.0
0.355
TabR
10.7
0.365
0.133
5.00
20.0
34.0
46.0
0.365
FTT
12.0
0.388
0.149
3.00
11.0
17.0
33.0
0.388
Laplace KRR
12.3
0.373
0.102
3.00
11.0
21.0
35.0
0.373
MLP-PLR
12.7
0.381
0.142
0.0
8.00
15.0
34.0
0.381
Excelformer
12.7
0.399
0.157
0.0
2.00
9.00
28.0
0.399
PTARL
13.1
0.390
0.114
2,00
6.00
9.00
20.0
0.390
RandomForest
13.3
0.389
0.088
5.00
12.0
22.0
35.0
0.389
AutoInt
14.3
0.399
0.158
1.00
2.00
6.00
12.0
0.399
Node
14.4
0.451
0.195
0.0
6.00
18.0
31.0
0.451
MLP
15.2
0.418
0.167
1.00
3.00
4.00
11.0
0.418
DCN2
15.3
0.473
0.227
0.0
2.00
10.0
20.0
0.473
ResNet
15.6
0.426
0.175
0.0
4.00
8.00
15.0
0.426
Tangos
16.3
0.432
0.173
0.0
1.00
4.00
9.00
0.432
SNN
17.2
0.419
0.174
0.0
2.00
4.00
8.00
0.419
KNN
19.1
0.459
0.190
2.00
4.00
4.00
11.0
0.459
TabNet
21.5
0.475
0,227
0.0
0.0
0.0
0.0
0.475
GrowNet
23.0
0.619
0.323
0.0
0.0
0.0
0.0
0.619
SVM
23.2
0.612
0.346
0.0
0.0
1.00
1.00
0.612
Tab TabTransfommer
26.4
1.07
0.750
1.00
1.00
2.00
3.00
1.07
SwitchTab
26.9
1.29
0.745
0.0
0.0
0.0
1.00
1.29
Danets
27.2
1.12
0.830
0.0
1.00
1.00
3.00
1.12
Table E.4: Full TALENT Regression results across 100 datasets. Rank is the average rank among
the ordered methods over all datasets, Score is the metric we use compare methods in Figure 3 in this case SGM,. Normalized score is the arithmetic mean of the normalized nRMSE. Top-X (%) the percentage of datasets for which that method in the top ranks. The final column the shifted geometric mean enror (SGM)).
21

---

Method
Rank
Score
Norm. Score
Top-1 (%)
Top-3 (%)
Top-5 (%)
Top-8 (%)
SGM
TabPFN-v2
7.15
0.823
0.935
25.0
51.5
63.2
70.6
0.108
xRFM
7.60
0.825
0.933
11.8
29.4
48.5
66.2
0.107
RealMLP
7.64
0.823
0.918
10.3
20.6
48.5
66.2
0.107
TabR
7.98
0.828
0.932
8.82
27.9
41.2
60.3
0.106
ModemnNA
8.88
0.825
0.912
10.3
30.9
44.1
61.8
0.106
CatBoost
10.3
0.819
0.905
1.47
19.1
36.8
51.5
0.117
LightGBM
11.7
0.813
0.883
5.88
22.1
33.8
44.1
0.125
XGBoost
12.1
0.815
0.886
4.41
19.1
26.5
41.2
0.124
ResNet
12.3
0.807
0.879
0.0
4.41
16.2
36.8
0.127
FTT
12.8
0.810
0.868
0.0
5.88
10.3
29.4
0.125
MLP-PLR
13.2
0.815
0.881
0.0
7.35
11.8
25.0
0.119
Laplace KRR
13.5
0.804
0.864
5.88
16.2
20.6
33.8
0.133
DCN2
14.0
0.806
0.865
1.47
4.41
10.3
26.5
0.127
MLP
14.2
0.805
0.864
0.0
4.41
8.82
19.1
0.131
SNN
14.9
0.805
0.858
0.0
0.0
4.41
8.82
0.129
AutoInt
15.7
0.804
0.858
0.0
0.0
1.47
7.35
0.130
RandomForest
15.7
0.797
0.831
5.88
8.82
13.2
27.9
0.137
Excelformer
16.1
0.805
0,853
0.0
0.0
4,41
17.6
0.127
Tangos
16.5
0.797
0.844
0.0
1.47
4.41
13.2
0.133
TabCaps
16.5
0.797
0.837
0.0
1.47
5.88
13.2
0.134
Danets
17.7
0.795
0.829
0.0
1,47
2.94
5.88
0.137
PTARL
19.1
0.796
0.808
0.0
1.47
1.47
4.41
0.139
KNN
20.1
0.786
0.772
2.94
7.35
10.3
14.7
0.152
LogReg
20.2
0.767
0.755
1.47
4.41
7.35
13.2
0.162
TabTransformer
21.5
0.771
0.739
0.0
1.47
1.47
2.94
0.155
Node
22.0
0.767
0.742
0.0
4,41
7.35
10.3
0.173
SVM
23.4
0.750
0.697
2.94
5.88
5.88
8.82
0.182
GrowNet
24.4
0.691
0.596
0.0
0.0
2.94
5.88
0.216
SwitchTab
25.0
0.748
0.674
0.0
1.47
1.47
2.94
0.187
TabNet
26.0
0.753
0.615
0.0
0.0
0.0
0.0
0.183
NaiveBayes
29.4
0.609
0.327
0.0
0.0
0.0
0.0
0.321
NCM
30.5
0.627
0.278
0.0
0.0
1.47
1.47
0.326
Table E.5: Full TALENT Multiclass classification (5 10 classes) results. Rank is the average rank among the ordered methods over datasets. (Normalized) Score is the metric we use to compare is the mean classification accuracy (1 minus the error in Figure 3). Top-X (%) is the percentage of datasets for which that method the top ranks. The final column is the shifted geometric mean error (SGM.).
methods in Figure 3 In this case
22

---

Method
Rank
Score
Norm. Score
Top-1 (%)
Top-3 (%)
Top-5 (%)
Top-8 (%)
SGM,
RealMLP
3.75
0.730
0.964
8.33
50.0
91.7
91.7
0.164
xRFM
5,46
0.718
0.949
25.0
50.0
66.7
83.3
0.183
TabR
6.00
0.720
0.948
16.7
50.0
50.0
66.7
0.172
ResNet
6.33
0.720
0.949
0.0
16.7
50.0
83.3
0.178
ModemNCA
6.83
0.713
0.913
33.3
41.7
50.0
83.3
0.168
FTT
8.79
0.709
0.919
0.0
8.33
25.0
50.0
0.187
MLP-PLR
8.79
0.712
0.925
0.0
0.0
25.0
50.0
0.185
MLP
9.58
0.710
0.922
0.0
0.0
8.33
41.7
0.190
Laplace KRR
10.8
0.679
0,889
0.0
16.7
41.7
50.0
0.218
DCN2
11.2
0.707
0.915
0.0
8.33
8.33
8.33
0.193
AutoInt
12.2
0.701
0.899
0.0
0.0
0.0
16.7
0.194
SNN
12.2
0.704
0.906
0.0
0.0
0.0
33.3
0.197
CatBoost
12.8
0.679
0.842
8.33
25.0
33.3
33.3
0.218
Danets
15.9
0.687
0.882
0.0
0.0
0.0
0.215
Tangos
16.1
0.660
0.823
0.0
8.33
8.33
16.7
0.225
XGBoost
16.2
0.678
0,874
8.33
8.33
8.33
25.0
0.236
Excelformer
16.6
0.675
0,851
0.0
0.0
0.0
16.7
0.217
TabCaps
16.8
0.670
0.853
0.0
0.0
0.0
0.0
0.231
LightGBM
16.9
0.668
0.850
0.0
0.0
0.0
16.7
0.261
PTARL
18.7
0.656
0.823
0.0
0.0
0.0
0.0
0.233
kNN
20.2
0.637
0.800
0.0
0.0
0.0
8.33
0.273
RandomForest
20.7
0.620
0.782
0.0
8.33
8.33
8.33
0.311
TabTransformer
20.8
0.582
0.702
0.0
0.0
8.33
8.33
0.307
LogReg
22.0
0.538
0,622
0.0
0.0
0.0
0.0
0.338
SVM
25.1
0.486
0.523
0.0
8.33
8.33
8.33
0.385
SwitchTab
25.2
0.525
0.603
0.0
0.0
0.0
0.0
0.372
Node
26.2
0.536
0.611
0.0
0.0
0.0
0.402
GrowNet
26.2
0.402
0.477
0.0
0.0
0.0
0.0
0.565
TabNet
26.3
0.486
0.516
0.0
0.0
0.0
0.368
NaiveBayes
29.5
0.410
0.380
0.0
0.0
0.0
0.0
0.545
NCM
30.2
0.399
0.339
0.0
0.0
0.0
0.0
0.552
Table E.6: Full TALENT Multiclass classification (> 10 classes) results. Rank is the average rank among the ordered methods over all datasets. (Normalized) Score is the metric we use to compare In this case score is the mean classification accuracy (1 minus the error in Figure 3). Top-X (%) is the percentage of datasets for which that method is in the top X ranks. The final column the shifted geometric mean error (SGM.).
methods in Figure?
23

---

Method
Rank
Score
Norm. Score
Top-1 (%)
Top-3 (%)
Top-5 (%)
Top-8 (%)
SGM
XRFM
5.96
0.845
0.981
29.6
55.6
59.3
74.1
0.119
RealMLP
7.44
0.839
0.951
7.41
14.8
29.6
77.8
0.125
TabR
7.83
0.844
0.970
18.5
33.3
55.6
70.4
0.116
ModernNCA
9.69
0.843
0.963
3.70
29.6
40.7
59.3
0.120
CatBoost
10.2
0.827
0.891
7.41
25.9
33.3
55.6
0.128
LightGBM
10.5
0.826
0.884
7.41
18.5
40.7
55.6
0.128
TabPFN-v2
10.9
0.834
0.923
7.41
22.2
25.9
48.1
0.129
MLP-PLR
11.2
0.827
0.890
0.0
0.0
18.5
40.7
0.131
XGBoost
12.1
0.824
0.875
3.70
18.5
33.3
51.9
0.129
FTT
12.4
0.829
0.900
0.0
11.1
18.5
29.6
0.135
MLP
12.7
0.833
0.915
0.0
7,41
14.8
29.6
0.130
ResNet
13.3
0.834
0.920
0.0
3.70
7.41
29.6
0.130
DCN2
13.4
0.832
0.919
0.0
3.70
3.70
22.2
0.130
PTARL.
13.4
0.831
0.911
0.0
3.70
7.41
14.8
0.131
AutoInt
13.7
0.831
0.911
0.0
0.0
3.70
11.1
0.130
Excelformer
14.7
0.821
0.858
3.70
7.41
7.41
11.1
0.143
SNN
14.8
0.831
0.912
0.0
3.70
7.41
18.5
0.131
Laplace KRR
17.0
0.826
0.879
3.70
11.1
14.8
14.8
0.138
Danets
17.4
0.827
0.887
0.0
0.0
0.0
0.0
0.133
TabCaps
18.4
0.822
0.862
0.0
3.70
7.41
7.41
0.136
TabTransformer
19.2
0.816
0.796
0.0
3.70
7.41
18.5
0.160
RandomForest
19.6
0.806
0.772
0.0
3.70
7.41
18.5
0.144
Node
20.1
0.793
0.724
3.70
3.70
3.70
7.41
0.151
LogReg
20.5
0.807
0.778
0.0
3.70
18.5
25.9
0.159
Tangos
20.7
0.811
0.798
0.0
0.0
3.70
3.70
0.139
SVM
21.8
0.802
0.754
0.0
0.0
3.70
18.5
0.162
GrowNet
22.7
0.804
0.771
0.0
0.0
0.0
3.70
0.160
TabNet
23.2
0.814
0.818
0.0
0.0
0.0
0.0
0.142
KNN
24.3
0.787
0.673
0.0
0.0
0.0
7.41
0.166
SwitchTab
27.3
0.786
0.670
0.0
0.0
0.0
0.0
0.163
NaiveBayes
30.2
0.678
0.154
0.0
0.0
0.0
0.0
0.283
NCM
31.3
0.679
0.174
0.0
0.0
0.0
0.281
Table E.7: Full TALENT Binary classification (> 10, 000 samples) results. Rank is the average rank among the ordered methods over all datasets. (Normalized) Score the metric we use to compare methods in Figure 3 In this score is the mean classification accuracy (1 minus the error in Figure 3). Top-X (%) is the percentage datasets for which that method is the top X ranks. The final column is the shifted geometric error (SGM.).
24

---

Method
Rank
Score
Norm. Score
Top-1 (%)
Top-3 (%)
Top-5 (%)
Top-8 (%)
SGM,
TabPFN-v2
5.02
0.864
0.947
26.9
59.1
72.0
80.6
0.104
xRFM
8.75
0.856
0.893
7.53
29.0
49.5
63.4
0.111
ModemnNCA
10.3
0.856
0.886
11.8
22.6
31.2
50.5
0.111
CatBoost
10.3
0.845
0.867
4.30
23.7
34.4
54.8
0.119
LightGBM
10.4
0.845
0.869
7.53
18.3
35.5
54.8
0.119
TabR
10.5
0.851
0.873
4.30
17.2
30.1
43.0
0.116
XGBoost
11.4
0.844
0,851
5.38
18.3
35.5
49.5
0.120
RealMLP
11.7
0.850
0.864
1.08
7.53
18.3
35.5
0.116
FTT
13.3
0.836
0.813
1.08
6.45
17.2
32.3
0.124
MLP-PLR
14.2
0.838
0.817
0.0
3.23
11.8
29.0
0.124
Random
14.3
0.836
0.824
3.23
10.8
19.4
33.3
0.129
AutoInt
15.2
0.834
0.803
0.0
3,23
5.38
16.1
0.129
Tangos
15.4
0.834
0.800
0.0
3,23
7.53
16.1
0.131
DCN2
15.5
0.839
0.816
1.08
2.15
9.68
21.5
0.127
Laplace KRR
15.7
0.836
0.801
1.08
16.1
20.4
26.9
0.133
MIP
16.0
0.836
0.797
1.08
3.23
8.60
17.2
0.132
ResNet
16.1
0.839
0.811
1,08
3,23
8.60
19.4
0.129
TabCaps
16.2
0.835
0.804
1.08
3,23
5.38
14.0
0.130
Excelformer
16.6
0.830
0.778
0.0
1.08
7.53
16.1
0.131
SNN
16.6
0.836
0.800
1.08
3.23
4.30
7.53
0.129
Node
17.0
0.824
0.757
0.0
5.38
15.1
25.8
0.137
PTARL
17.0
0.830
0.780
0.0
1.08
3.23
8.60
0.134
Danets
18.0
0.829
0.759
0.0
3.23
6,45
14.0
0.137
TabTransformer
19.7
0.820
0.715
1.08
1.08
6.45
12.9
0.148
LogReg
20.1
0.819
0.723
3,23
9.68
12,9
17.2
0.146
kNN
20.8
0.811
0.677
5.38
10.8
14.0
18.3
0.158
SVM
21.6
0.816
0.698
2.15
3.23
6.45
10.8
0.150
GrowNet
22.2
0.817
0.714
0.0
0.0
0.0
2.15
0.151
SwitchTab
24.0
0.809
0.675
0.0
0.0
0.0
0.0
0.156
TabNet
25.8
0.802
0.633
0.0
0.0
0.0
1.08
0.157
NaiveBayes
28.6
0.695
0.281
1.08
2.15
3.23
4.30
0.253
NCM
29.9
0.723
0.230
0.0
0.0
0.0
2.15
0.257
Table E.8: Full TALENT Binary classification (< 000 samples) results. Rank is the average rank among the ordered methods over all datasets. (Normalized) Score the metric we use to compare methods in Figure 3 In this score the mean classification accuracy (1 minus the етrоr in Figure 3). Top-X (%) is the percentage of datasets for which that method is in the top X ranks. The final column is the shifted geometric mean error (SGM)).
Dataset
xRFM
XGB
CatBoost
LGBM
RealMLP
MLP-PLR
MLP-RTDL
ResNet-RTDL.
Airlines_ DepisseeeeeDeDeleeee 1OM
0.9818
0,9813
0.9796
0,9798
0,9786
0.9795
0.9824
0,9818
Allstate, MllttateClaiims. ClaertySeverity
0.6489
0.6547
0.6510
0.6530
0.6495
0.6537
0.6557
0.6537
black friday
0.6881
0.6807
0.6792
0.6787
0.6859
0.6862
0.6929
0.6892
Buzzinsocialmedia Twitter
0.2080
0.2134
0.3147
0.2789
0.2566
0.2553
0.2840
0.2906
nyc-taxi-green green-ee-2206
0.5834
0.6649
0.6567
0.6489
0.6142
0.6523
0.6657
0.6365
wave_energy
0.0020
0.0918
0.0499
0.0821
0.0024
0.0073
0.0254
0.0434
Yolanda
0.7816
0.8012
0.8094
0.7970
0.7869
0.7897
0.7927
0.7856
Table E.9; Meta-test regression datasets with more than 70,000 samples. Error reported is nRMSE averaged over five train/test splits using pyt abkit.
25

---

Dataset
xRFM
XGB
CatBoost
LGBM
RealMLP
MLP-PLR
MLP-RTDL
ResNet-RTDL
airlines
0.3341
0.3292
0.3315
0.3287
0.3344
0.3342
0.3342
0.3339
covertype
0.0257
0.0420
0.0612
0.0333
0.0280
0.0364
0.0404
0.0385
Higgs
0.2640
0.2576
0.2576
0,2549
0,2473
0.2528
0,2515
0.2435
jannis
0.2701
0.2799
0.2816
0.2779
0,2702
0.2764
0.2865
0.2795
MiniBooNE
0.0539
0.0529
0.0538
0.0525
0.0484
0.0504
0,0503
0.0488
numerai28.6
0.4778
0.4812
0.4790
0.4782
0,4800
0.4775
0.4771
0.4800
porto-seguro
0.0380
0.0380
0.0381
0.0381
0.0380
0.0380
0,0380
0.0380
dionis
0.0926
0.1219
0.1041
0.1076
0.0887
0.1257
0.1110
0,0907
MNNII
0,0889
0.0928
0.0969
0.0895
0.0913
0.1064
0.1041
0.1011
kick
0.0972
0.0965
0.0956
0.0964
0.0976
0.0978
0.0979
0.0970
Table E.10: Meta-test classification datasets with greater than 70,000 samples. Error reported is classification error averaged over five train/test splits using pytabkit.
26

---

Model
Elo (*)
Norm. score (7)
Avg. rank (L)
Harm. mean rank (L)
#wins (?)
Improva-
Train time per 1K [s]
Predict time
billt (J)
per 1K [s]
AntoGluon 1.3 (4h)
7,06
ReaLMLP (T+B)
1721
0.660
6.6
5,4
0
3.1%
6860.5 54
7.68
ModemNCA (T+E)
1626
0.622
9.2
3
5.4%
3811443
7.58
LightGBM (T+E)
1602
0,478
10.1
D
6.0%
686.46
5.48
TabDPT (D)
1575
0.515
10.9
2
3.9%
16.97
8.70
XRFM (T+F)
1563
0,492
11.5
5.1%
365.57
0.72
CatBoost (T+E)
1558
0.435
11.8
8.9
D
5.7%
2895.38
1.32
TahM (T=F)
CatBoost (T)
1515
0,406
13.3
6.8
D
5.9%
2895.3 38
0.07
XRFM (T)
1504
0.406
13.8
9.7
D
5.7%
365.57
0.09
LightGBM (T)
1498
0.354
13,9
10.2
6.8%
686.46
0.74
XGBoost (T+R)
1470
0.328
15.2
13.9
D
6.7%
848.99 99
2.38
XGBoost (T)
0.325
15.2
13.3
6.7%
848.99
0.47
ModemNNCA.
1457
0.323
15.6
11.0
D
8.1%
16.07
0.29
TabM (T)
12.6
5.25
0.13
RealMLP (T)
1439
0.306
16.6
13.3
D
5.9%
6860.54
0.32
CatBoost (D)
1419
0.278
17.4
12.0
D
7.9%
8.35
0.09
ModemNCAA
1411
0.347
17.6
7.9%
3811.43
0.45
TahPFNv2 (T+E)
1377
0,410
19.1
2.7
4
4, 4
3805.62
10.41
TabM (D)
1361
0.252
6.9%
13.90
0.12
TabPHNv2 (T)
1314
0.281
22.1
6.9
D
6.4%
3805.62
0.26
TorchMLP (T+E)
1301
22.7
8.1%
4452.11
0.85
ExtraTrees (T+E)
1297
0.169
23.1
16.9
D
11.1%
161.73
0.78
RealMLP
1274
0.085
24.0
21.6
D
8.2%
23.30
1.44
ExtraTrees
1273
0.174
24.0
16.7
11.4%
161.73
0.12
TabPFNv2 (D)
1264
0.262
24.3
7.7
D
7.6%
2.78
0.32
TorchMLP
xRFM (D)
1236
0.149
25.8
8.6
11.7%
1.65
0.08
LightGBM (D)
1231
25.9
D
9.7%
2.03
0.30
XGBoost
1195
0,104
27.4
24.5
D
10.4%
2.15
0.18
RaneonnForstt (T+E)
1193
0.056
27.5
25.5
D
12.1%
526.17
0.77
RandomForest (T)
1142
0.046
29.9
27.5
D
12.8%
$26.17
0.12
EBM (T+E)
1133
0.122
30.0
16.4
D
14.7%
2124.78
0.12
ExtralTrees
FastaiMLP (T+E)
1086
0.005
31.8
30.6
13.1%
527.21
2.83
EBM (T)
0.131
32.2
9.4
I
15.3%
2124.78
0.01
TorchMLP
1075
0.015
32.4
30.3
12.9%
20.50
0.08
MastaiMT.P
1046
0.000
33.3
32.3
D
13.6%
$27.21
0.31
EBM (D)
34.7
30.3
O
16.0%
7.25
0.04
TabICL (D)
1006
0.000
34,8
34.2
D
14.2%
0.63
0.06
RandomForestt
FastaiML.
916
0.000
37.2
36.5
D
17.8%
3.08
0.29
KNN (I+E)
533
0.000
43.8
43.6
0
37.3%
2.25
0.15
Linear (T+E)
485
0,000
44.3
44.2
35.5%
46.50
0.14
KNN (T)
434
0.000
44.9
44.7
0
38.0%
2.25
0.03
Lincar
35.7%
46.50
0.04
Linear (D)
312
0.000
46.0
46.0
0
38.1%
1.16
0.08
(D)
Table E. Full regression results on TabArena-Lite.
27

---

Model
Elo (*)
Norm. score (7)
Avg. rank (L)
Harm. mean rank (L)
#wins (?)
Improva-
Train time per 1K [s]
Predict time per 1K [s]
billty (J)
AntoGluon 1.3 (4h)
CatBoost (T)
1441
0,439
12.5
9.6
D
11.9%
3307 58
0.14
CatBoost (T+E)
1441
0.446
12.4
D
11.1%
3307.58
1.18
TabPENv2 (T+E)
1419
0.518
13.5
4.2
13.8%
2584.13
12.37
LighiGBM (T+E)
1418
0.393
13.2
D
13.6%
1280.01
4.08
LightGBM (T)
1405
0.381
13.9
D
13.6%
1280.01
1.05
TabM (T-E)
1387
0.405
14,8
9.7
D
15.7%
3568. 31
1.78
(T+R)
TabM (T)
1373
0,401
15.5
9.1
D
16.0%
$568.31
0.37
XGBoost (T)
1373
0.317
15.6
13.1
13.7%
2029.77
1.04
Rea|MPP (T+E)
1359
0.337
16.2
16.8%
6866.35
10.40
TahPFNVv2 (D)
1358
0.361
16.4
12.5%
5.48
0.35
XRFM (T+E)
0.369
16.2
14.8%
$15.01
1.68
TabPFNv2 (T)
1346
0.366
17.0
6.3
D
14.9%
2584.13
0.41
ModemMNA (T+E)
16.9
10.4
15.3%
9.59
ModemNCA (T)
1332
0.248
17.8
12.4
D
15.6%
6684.65
0.75
RealMLP (T)
1304
0,253
19.1
15.5
D
18.4%
6866.35
0.92
Calood
1296
0,201
19.2
16.9
16. 1
43,10
0.25
TabICL (D)
1293
0.325
19.9
20.5%
11.51
1.95
TabM
1293
0.284
17.09
0.15
ExtraTrress (T++)
1286
0.289
20.2
11.
17.7%
728.32
2.44
xRFM (T)
20.4
16.4%
515.01
0.20
ExtraTroes (T)
1265
0,262
21.2
11.2
17.3%
728.32
0.36
RandonFForet (T+E)
1260
0.321
20.9
6.1
D
15.9%%
729.17
1.83
TorchMI P (T+F)
1260
0,188
21.2
17.5
19.6%
3646.83
2.16
TorchMLP (T)
1218
0.156
23.9
17.8
D
21.6%
3646.83
0.19
RandomForest (T)
XGBoost (D)
1188
0.102
24.9
21.5
D
18.15
4.93
0.59
TabDPT
1186
0.259
25.2
I
23.3%
33.52
20.75
FastaiMPP (T+E)
1182
0,205
25,2
12.1
D
24.2%
2721.87
12.59
LighiGBM (D)
1180
0.158
25.4
19.6
D
21.3%
5.12
0.44
ERM (T+F)
0.166
26.0
17.1
O
24.6%
1471.12
0.27
EBM (T)
1144
0.160
27.4
19.0
D
24
1471.12
0.03
Modem:NAA (D)
1110
0.051
29.1
26.8
D
25.9%
17.24
0.57
FastaiMLP (T)
1100
0.104
29.6
20.3
D
25.8%
2721.87
1.08
RealMLP (D)
1049
0.071
32,0
24.5
D
26.3%
26.02
4.18
FBM (D)
1046
0.095
32,0
24.3
D
27.7%
6.16
0.08
RondomForest (D)
0.000
33,9
32.0
33.3%
0.15
TorchML (D)
972
0.004
35.0
33.6
D
28.
14.37
0.36
EataiMLP
ExtraTrees (D)
862
0.013
38.6
35.1
D
39.1%
0.76
0.15
Linear (T+E)
752
0.000
41.8
41.5
43.6%
170.51
0.20
Linear
718
0,000
42.3
41.9
45.8%
170.51
0.13
KNN (T+F)
714
0.000
42.4
41.0
52.5%
2.99
0.17
Lincar
42.9
46.9%
3.89
0.16
KNN (D)
544
0.000
45.4
45.1
0
69.5%
0.33
0.05
Table F.12: Full multiclass results on TabArena-Lite.
28

---

Model
Elo (7)
Norm. Score (7)
Avg. rank (L)
Harm. mean rank
twins (1)
Improva- billy (1)
Train time per 1K [s]
Predict time
per 1K [s]
AutoGluon (4h)
-300--32+32
1.74
RealMLP (T+E)
1461 30,+-1
0.516
11.0
5.1
2
8.3%
6177.16
8.66
TabM (T+E)
1416 2900-29.430
0.457
13.0
2
9.1%
2180.12
1.13
LightiBM (T+E)
1398 +333
0.362
13.8
0
11.1%
328. 64
0.77
Tab[CI (D)
1393 14
0.427
14.0
3
8.0%
8.05
2.01
CatBoost (D)
1380 22
0.365
14.8
10.6%
3.84
0.07
XGBoost (T+E)
1372
0.358
15.1
6.7
11.7%
462.92
0.61
(T+F)
33,-
ModemNCA (T)
1361 +3.
0.397
15.7
6.4
2
10.2%
3436.74
0.41
ModemNCA (T+E)
1347 30,
0.414
16.3
2
10.4%
3436,74
8448
CatBoost (T)
1347 +-3
0.331
16.6
11.3%
1043.89
0.04
TabM (T)
1345 40,
0.382
16.6
7.7
10.3%
2180.12
0.12
XGBoost (T)
1328 31.
0.305
17.2
12.1% 1%
462.92
0.11
LightGBM (T)
1328
0.288
17.4
13.1
0
12.3%
328.64
0.09
TabPFNv2 (T+E)
34,35
18.2
11.8%
17.91
XRFM (T+E)
1310
0.283
18.4
12.3
12.0%
247.54
0.82
EBM (T+E)
1296 1296-29.+33 29.4+33
0,250
19.1
10.1
0
14.0%
852.29
0.21
TabM (D)
1282 3-5
0.274
19.7
11.7
13.4%
8.00
0.12
TorchMLP (T+E)
1265 3-
0.219
20.6
13.1
12.3%
2206.58
2.30
TabPFNv2
1264 42.
0.296
13.8%
2914.83
0.22
ЕBM
1254 32,4-32
0.187
21.3
12.4
U
14.7%
852.29
0.02
xRFM (T)
1251 21.+-1
0.209
21,4
12.7
13.7%
247.54
0.05
RealMLP (T)
1248 127
0.215
21.8
14.1
13.4%
6177.16
0.26
RealMLP (D)
1247 33,+34
0.177
21.8
14.4
13.3%
18.75
0.99
FRM
1238 123
0.191
22.3
10.3
15.2%
4,40
0.03
TabPFNv2 (D)
1209
0.241
23,9
7.0
2
15.7%
2.61
0.26
FastaiMLP
1206 29.
$61.29
4,46
XGBoos (D)
1198
0.168
24.5
11.6
15.0%
1.42
0.12
ModemNCA (D)
1185 22
0.157
25.2
11.2
15.8%
11.15
0.31
TorcMLP (T)
1174 +.0
0.156
25.7
19.3
14.8%
2206.58
0.11
ExtraTrees (T+E)
1168 27.
0.115
26.2
19.6
17.3%
122.90
0.58
LightGBM
1134 29. 127
0,103
28.1
23.7
1%
0,93
0.09
RaneomForest (T+E)
1132-33.422
0.078
28.1
21.9
O
18. 18.4%
171.61
0.55
1129 +93
TabDPT (D)
1123 1123-
0.152
28.7
13.2
17.2%
17.79
8.53
ExtraTrres (T)
1122 30, 127
127
0.106
28.7
16.6
0
18.9%
122.90
0.07
RundomFForstt
1098 24
0.029
29,9
27.1
19.15
171.61
0.05
TorchML (D)
1030
28
0.052
33.2
28.2
20.5% 0
4.97
0.09
RaneomForest
1000 0.+00
0.032
34,5
26.6
23.4%
0.03
FastalMLP (D)
998
0.055
34,6
30.2
0
21.1%
2.79
0.26
960
xRFM (D)
943 28
0.053
36.8
30.1
24.9% 0
1.07
0.05
ExtraTrress (D)
926 28
0.023
37.6
33.3
0
25.4%
0.18
0.03
Linear (T)
923
0.025
37.6
27.1
0
29.0%
40.63
0.05
Linear (D)
899 81.1
0.017
38.5
35.3
0
29.8%
0.95
0.08
KNN (T+E)
722 122-98.+33
43,2
36.3
47.7%
0.15
KNN (I)
691 -37.
0.020
43.8
30.5
0
48.8%
2.75
0.03
Table E.13: Full binary classification results on TabArena.
29

---

Method
TabArena
Remaining
XRFM
0.2380
0.3190
TabPFN-v2
0.2444
0.3323
CatBoost
0.2818
0.3424
RealMLP
0.2831
0.3336
LightGBM
0.3065
0.3580
FT-Transformer
0,3092
0.3963
XGBoost
0.3168
0,3596
AutoInt
0.3200
0.4083
TabR
0.3241
0.3692
Random Forest
0.3246
0.3959
Table E.14: Performance of the best ten regression methods on the TALENT benchmark restricted to the datasets also included TabArena (first column). In the second column, performances the same methods are shown for the remaining datasets not included in TabArena. The value reported the shifted geometric mean the error, with the best method of each subset is in bold.
Method
TabArena
Remaining
xRFM
0.1485
0.1022
TabPFN-v2
0.1487
0.1021
CatBoost
0,1502
0.1150
LightGBM
0.1503
0.1154
ModernNCA
0.1510
0.1066
TabR
0.1515
0.1093
RealMLP
0.1522
0.1117
MLP-PLR
0.1524
0.1203
XGBoost
0.1524
0.1156
FT-Transformer
0.1530
0.1211
Table E.15: Performance of the best ten binary classification methods on the TALENT benchmark restricted to the datasets also included in TabArena (first column). In the second column, perfor- mances the same methods are shown for the remaining datasets not included in TabArena. The value reported is the shifted geometric mean of the error, with the best method of each subset is in bold.
Method
TabArena
Remaining
TabPFN-v2
0.0737
0.1
XGBoost
0,0763
0.1408
LightGBM
0.0775
0.1436
xRFM
0.0791
0.1185
CatBoost
0.0803
0.1312
RealMLP
0.0804
0.1160
ModemnNCA
0.0874
0.1097
TabR
0.0875
0.1152
Fr-Trrasformmr
0,0939
0.1356
ExcelFormer 0.0955
0.1407
Table E.16: Performance of the best ten multiclass classification methods the TALENT bench- mark restricted to the datasets also included TabArena (first column). In the second column, performances of the same methods shown for the remaining datasets not included in TabArena. The value reported the shifted geometric mean of the error, with the best method of each subset is in bold.
30

---

Method
Regression (n=70)
Binary (n=78)
Multiclass (n=52)
xRFM
0.3393
0.1138
0.1203
XGBoost
0.3763
0.1246
0.1383
CatBoost
0,3615
0.1240
0,1310
Portfolio (xRFM or XGBoost)
0.3425
0.1144
0.1177
Portfolio (xRFM or CatBoost)
0.3379
0.1137
0.1179
Portfolio (xRFM, XGBoost, or CatBoost)
0.3418
0.1141
0.1173
Table E. 17: Portfolios of xRFM with CatBoost/ / XGBoost on the TALENT benchmark (restricted to datasets where validation results are provided for XGBoost and CatBoost). Portfolio constructed by taking the best of the listed methods on the validation dataset and evaluating on the test set. Reported values are SGM of the errors with the lowest bolded.
31

---

F ADDITIONAL FIGURES
XRFM vs. Baseline Win Rates by Task Type
100
Regression
Binary Classification
Multiclass Classification
80
(r-102)
74.0%
(1s02001
10.0%
X/VPM Wih Rate (k)
18.0%
(1-220)
(200
50 2%
11-
1261
40 3/%
(n-033)
40
20
TabFFN-22
RezIMLP
XGBoast
CatBoost
LighGBM
Figure F.1: Win-rate of xRFM versus other methods on the TALENT benchmark.
4096
CatBoost
Average Training Time (seconds)
2048
XRFM (L1 kernel)
XRFM (L2 kernel)
1024
ReaIMLP
XGBoost
512
256
128
64
32
50
100
150
200
250
300
350
Number of Training Samples (x 1000)
Figure F.2: Runtime comparison as a function of the number of training samples on the covertype dataset. Here, L1 kernel refers the LJ kernel.
32

---

Regression
Binary Classification
Multic ass Cassification
10
Carbic
hao
101
104
0.0
107
101
0.0
10
10"
Number samples
Number of samples
Number samples
Regression
Binery Classification
Multic ass Cassif-cation
10
1.0
Eroor
0.0
0.6
Sorm 22
Festve
10'
10'
dimension
Feature dimansion
Regression
Binary Classification
Multiclass Classification
1.0
1.0
1.0
8.
0.8
0.8
Zome 1200
0.6
0.6
0.6
0.4
Ear
0.4
Feeiiu
Bralie
UgsiCaM
0.00
02
0.
0.0
Categorical
Catogorica
Categorical
Figure F3: Evaluation of the normalized errors across dataset metafeatures for the the top methods. A biece-wise linear fit is shown for (A) number of samples, (B) feature dimension, and (C) the fraction of features that are categorical versus the normalized error. Individual datasets are shown as dots.
33


Reviews:
[{"review_id": "zd4fTVup7z", "reviewer": "Reviewer_XCAi", "summary": "This paper addresses the limitations of traditional tabular data modeling in terms of scalability and feature learning capabilities and proposes a new model, xRFM. This model aims to simultaneously possess (1) local feature learning capabilities, (2) interpretability, and (3) computational efficiency that can be scaled to very large datasets, thereby surpassing existing gradient boosted decision trees (GBDTs) and the recent tabular data base model in both regression and classification tasks.", "questions": "Q1. Is the AGOP splitting criterion stable in the case of high-dimensional sparse features? Has a comparison been conducted with splitting methods based on axis information gain?\n\nQ2. What is the main bottleneck that causes xRFM's performance to lag behind in multi-classification tasks? Is it the tree splitting strategy, the Leaf RFM structure, or insufficient hyperparameter tuning?\n\nQ3. If xRFM is combined with TabPFN-v2 (as a backend or feature extractor), can the gap in classification tasks be narrowed?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. Combining RFM with tree-based data partitioning and using AGOP main directions for supervised splitting, this combination of local feature learning and scalable kernel methods is novel in tabular ML.\n\nS2. The method's detailed derivation is clear, and the theoretical background and splitting criteria for AGOP are clearly referenced and supported. Technical improvements, such as kernel parameter space exploration, categorical variable optimization, and bandwidth adaptation, are detailed.\n\nS3. Experiments covering ultra-large datasets (>500,000 samples) demonstrate computational scalability. Multiple benchmarks validate its advantages in regression tasks.\n\nS4. Feature interpretation is natively supported, eliminating the need for external tools. Locally relevant features can be analyzed directly from the AGOP of each leaf node.", "weaknesses": "W1. Based on experimental tables (such as the TALENT binary and multi-classification results), xRFM is only competitive in most classification tasks, rather than significantly leading. Significant improvements in a single regression domain do not fully demonstrate the model's versatility.\n\nW2. The results lack fine-grained analysis comparing different model families. Although benchmark rankings are reported, there is a lack of grouped performance analysis for model families (Tree-based, Kernel-based, and NN-based), which fails to clearly demonstrate in which structural scenarios xRFM excels. The charts focus on overall rankings and do not show the correlation between different task attributes and performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nCRSHnshGP", "reviewer": "Reviewer_7Jyu", "summary": "This paper introduces **xRFM**, a novel algorithm for learning from tabular data that integrates **feature-learning kernel machines** with **adaptive tree-based data partitioning**. The key idea is to build a binary tree that recursively splits the data along informative directions derived from the **Average Gradient Outer Product (AGOP)**, training a *Recursive Feature Machine (RFM)* at each leaf. xRFM achieves **log-linear training complexity** and **logarithmic inference time**, scaling effectively to very large datasets (up to 500K samples). It is evaluated extensively on **TALENT**, **TabArena-Lite**, and **Meta-Test** benchmarks, outperforming 31 baselines (including GBDTs and TabPFN-v2) on regression tasks and remaining competitive on classification tasks.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The combination of **kernel-based feature learning** (via RFM) with **tree-based local partitioning** and echanism based on **AGOP decomposition** is conceptually novel and elegant.\n\n* The paper demonstrates rigorous experimental validation across **three major benchmarks** and **hundreds of datasets**, establishing strong empirical credibility. The comparisons with baselinse are comprehensive, with fair baselines including recent foundation models like **TabPFN-v2** and **RealMLP**.\n* The **scaling analysis** (O(n log n) training, O(log n) inference) is clearly supported by empirical runtime curves (Fig. 5).\n* The method maintains theoretical ties to kernel learning and AGOP-based supervised PCA, providing a sound foundation for interpretability and feature relevance analysis.\n* The paper is very well written, with clear motivation, detailed algorithmic descriptions (Algorithms A.1–A.5), and informative figures (especially Fig. 1,2).\n* The paper addresses a long-standing gap: scalable, interpretable, and high-performing models for **tabular data**. The reported performance and efficiency make xRFM a viable candidate for large-scale deployment in applied ML contexts (e.g., finance, healthcare, industrial analytics).", "weaknesses": "* While the empirical results are compelling, the paper lacks a **formal analysis** of convergence or generalization bounds for the tree-partitioned RFM structure.\n\n* The paper could better isolate the contributions of individual components:\n    * How much performance gain is from **tree-based scaling**? For example a comparison with normal RFM can show it.\n    * How much performance gain is from **AGOP**? If using normal tree splitting, how much will the performance drop? What if using other splitting method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TGaon6lMVM", "reviewer": "Reviewer_wAt5", "summary": "The paper proposes xRFM, a model that uses the Average Gradient Outer Product (AGOP) to identify directions of maximal predictive variation, recursively splitting the data and fitting local kernel ridge regressors in each leaf. Overall, this is a solid paper that effectively demonstrates the usefulness of AGOP for tabular data.", "questions": "I have a few minor points for the authors to address:\n\n1. In Equation (2), what happens if the gradients point in opposite directions or are orthogonal? Could this cause AGOP to fail or weaken its signal?\n\n2. Please investigate or discuss why the proposed method performs well on TALENT, but not better than fine-tuned CatBoost on TabArena.\n\n3. Could gradient boosting be combined with xRFM to further improve performance? A small-scale study or discussion on this would be helpful.\n\n4. Figure 2 nicely illustrates how xRFM captures feature interactions. For this method, is it possible to derive a feature importance ranking similar to that in XGBoost? Please consider adding a short discussion.\n\n5. Please consider performing a post-hoc meta-analysis to identify which types of datasets favor xRFM and which ones favor existing methods such as PFN or CatBoost.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is a solid paper that effectively demonstrates the usefulness of AGOP for tabular data.", "weaknesses": "See Questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IJFeRjNx5t", "reviewer": "Reviewer_LmzM", "summary": "This paper introduces a new architecture for tabular prediction tasks called xRFM. Specifically, xRFM combines feature learning kernel machines with a tree structure to both adapt to the local structure of the data. Empirically, the authors showed that xRFM outperforms various baselines in tabular regression tasks and shows competitive results in tabular classification tasks.", "questions": "See the above Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is fast, scalable, efficient, and also accurate.\n\n2. The authors have done comparative experiments on a total of 300 datasets. This is a really big contribution to the tabular learning society and ICLR.\n\n3. xRFM is a very efficient solution of tabular prediction tasks, while ensuring competitive performance.\n\n4. The paper is well written.\n\n5. Tabular prediction tasks are challenging and important problems in ML society. Moreover, I highly agree that this field is still dominated by simple GBDTs and some new architectures should be proposed progressively.", "weaknesses": "1. Can the authors provide the win rate of xRFM compared to competitive baselines like TabPFN-v2, RealMLP, etc?\n\n2. Is xRFM also effective for few-shot tabular predictions tasks, where the number of class-per samples is just 1 or 5?\n\n3. Are you going to open-source this project?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)