# xRFM: Accurate, scalable, and interpretable feature learning models for tabular data

**Daniel Beaglehole**

Computer Science and Engineering
Halıcıoğlu Data Science Institute
UC San Diego
dbeaglehole@ucsd.edu

**David Holzmüller**

INRIA Paris
École Normale Supérieure
PSL University
david.holzmuller@inria.fr

**Adityanarayanan Radhakrishnan**

MIT Mathematics
Broad Institute of MIT and Harvard
aradha@mit.edu

**Mikhail Belkin**

Halıcıoğlu Data Science Institute
UC San Diego
mbelkin@ucsd.edu

## Abstract

Inference from tabular data, collections of continuous and categorical variables organized into matrices, is a foundation for modern technology and science. Yet, in contrast to the explosive changes in the rest of AI, the best practice for these predictive tasks has been relatively unchanged and is still primarily based on variations of Gradient Boosted Decision Trees (GBDTs). Very recently, there has been renewed interest in developing state-of-the-art methods for tabular data based on recent developments in feature learning methods. In this work, we introduce xRFM, an algorithm that combines feature learning kernel machines with a tree structure to scale to essentially unlimited amounts of training data. On the TALENT benchmark, we show that xRFM achieves best performance across 100 regression datasets and is competitive across 200 classification datasets, outperforming GBDTs.

## 1 Introduction

Tabular data – collections of continuous and categorical variables organized into matrices – underlies all aspects of modern commerce and science from airplane engines to biology labs to bagel shops. Yet, while Machine Learning and AI for language and vision have seen unprecedented progress, the primary methodologies of prediction from tabular data have been relatively static, dominated by variations of Gradient Boosted Decision Trees (GBDTs), such as XGBoost [6]. Nevertheless, hundreds of tabular datasets have been assembled to form extensive regression and classification benchmarks [5, 10, 24, 9], and, recently, there has been renewed interest in building state-of-the-art predictive models for tabular data [14, 15, 12].

In this work, we introduce xRFM, a tabular predictive model that combines recent advances in feature learning kernel machines with an adaptive tree structure, making it effective, scalable, and interpretable. xRFM builds upon the Recursive Feature Machine (RFM) algorithm from [20], which enabled *feature learning* (a form of supervised dimensionality reduction) in general machine learning models. Given training data, xRFM first builds a binary tree structure to split data into subsets based on features relevant for prediction within each split. When splits reach a certain size, we train a *leaf RFM* (a hyper-parameter and compute optimized version of RFM).

In practice, we show xRFM has the best performance across 100 tabular regression tasks and is competitive with state-of-the-art on 200 tabular classification tasks from the TALENT benchmark [25]. On the TabArena-Lite benchmark [9], we show xRFM empirically achieves one of the best tradeoffs (is on the empirical Pareto frontier) between performance and inference time among all methods in regression and is again competitive on classification. We show that xRFM achieves similar results on the largest datasets from the meta-test benchmark [15], where directly solving standard kernel machines becomes intractable on standard GPUs.
<figure><img src="image_1.png" /><figcaption>**Figure 1:** Overview of xRFM training and inference procedures. (A) xRFM is trained by splitting the data along the median projections (denoted $c_1, c_2$) onto computed split directions (denoted $v_1, v_2$). Data is split repeatedly into leaves, which contain at most *C* training samples. Leaf RFMs are trained on the data at each leaf. (B) During inference, test data is routed to the appropriate leaf RFM based on split directions. The prediction is generated by the selected leaf RFM.</figcaption></figure>

## 2 Preliminaries

We review kernel machines and kernel-RFM, which we use to build xRFM.

**Kernel machines** A kernel machine is a non-parametric machine learning model [23, 2]. The idea behind kernel machines is that one can train a nonlinear predictive model by first transforming input data with a fixed, nonlinear feature map and then performing linear regression on the transformed data. Kernel machines make this procedure computationally tractable even for infinite dimensional feature maps by using kernel functions (inner products of feature mapped data). We describe kernel machines in the context of supervised learning below. Let $X \in \mathbb{R}^{n \times d}$ denote training inputs with $x^{(i)}^T$ denoting the example in the $i^{th}$ row of $X$ for $i \in [n]$ and $y \in \mathbb{R}^{n \times c}$ denote training labels. Let $K: \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}$ denote a kernel function (a positive semi-definite, symmetric function). Given a regularization parameter $\lambda \ge 0$, a kernel machine trained on the data $(X, y)$ is a predictor, $\hat{f}: \mathbb{R}^d \to \mathbb{R}^c$, of the form: $\hat{f}(x) = K(x, X)\alpha$; $\alpha = (K(X, X) + \lambda I)^{-1}y$; where the notation $K(x, X) \in \mathbb{R}^{1 \times n}$ denotes an $n$-dimensional row vector with $K(x, X)_i = K(x, x^{(i)})$ and $K(X, X) \in \mathbb{R}^{n \times n}$ denotes a matrix with $K(X, X)_{ij} = K(x^{(i)}, x^{(j)})$. A typical kernel functions used in practice is, e.g., the Gaussian kernel ($K(x, z) = \exp(-\|x - z\|_2^2)/L^2$).

**Recursive Feature Machines (RFMs)** The ability to learn task-relevant features from data is key to building effective predictors [7, 11, 1]. RFM, introduced in [20], is an algorithm that enables feature learning in general machine learning models through a mathematical object known as the Average Gradient Outer Product (AGOP). Given a predictive model $\hat{f}: \mathbb{R}^d \to \mathbb{R}$ and data $S = \{x^{(1)}, \dots, x^{(n)}\} \subset \mathbb{R}^d$, the AGOP is defined as AGOP($\hat{f}, S) = \frac{1}{n} \sum_{i=1}^n \nabla \hat{f}(x^{(i)}) \nabla \hat{f}(x^{(i)})^T \in \mathbb{R}^{d \times d}$ where $\nabla \hat{f}(x^{(i)})$ denotes the gradient of $\hat{f}$ at the point $x^{(i)}$. The AGOP is an estimate of the (un-centered) covariance of the gradients of $\hat{f}$ and intuitively captures the subspace along which the predictor highly varies [24, 16]. The RFM algorithm involves iterating between training a predictive model and using the AGOP of the trained model to select features and linearly transform input data.

In the case of kernel machines, which have no native mechanism for feature learning, RFM enables feature learning by adapting the kernel to the data. We describe the kernel-RFM algorithm below. Following the notation in the previous section, let $(X, y)$ denote training inputs and labels, let $K$ denote the kernel function, and $\lambda$ denote the regularization parameter. Letting $M_1 = I$ and $c > 0$, kernel-RFM repeats the following two steps for $T$ iterations: (Step 1) Compute $\hat{f}_t(x) = K(M_t x, XM_t)\alpha_t$ and $\alpha_t = [K(XM_t, XM_t) + \lambda I]^{-1}y$, (Step 2) Compute $M_{t+1} = [\text{AGOP}(\hat{f}_t(M_t x), X)]^c$. We provide additional details on the hyperparameters of this algorithm in Appendix B.2.
<figure><img src="image_2.png" /><figcaption>**Figure 2:** Performance and runtime of xRFM on the TALENT (Plots A-C) and TabArena-Lite benchmarks (Plots D-F). The x-axes are the average over all datasets of the training plus inference time per 1000 samples for just the best hyperparameter configuration (meaning if a dataset has *n* samples, we compute the training and inference time on the *n* samples divide the total time by *n*/1000).</figcaption></figure>

## 3 xRFM algorithm overview

We now describe our algorithm xRFM, which consists of the following two components (Fig. 1): (1) An improved kernel-RFM, termed *leaf RFM*, that is trained on the subset of data, and (2) A binary tree that splits the data into subsets, termed *leaves*, of a maximum size (*leaf size*) that is independent of the number of training samples. Leaf RFMs are then trained on the subset at each leaf. The splits stratify data based on features relevant for prediction. We detail these two components below.

**Leaf RFM** Here, we describe the changes we made to the original kernel RFM algorithm from [20] to produce leaf RFMs. The kernel RFM model was primarily built using kernels that were invariant to orthonormal transformations of data. Such invariance does not effectively leverage a special property of tabular data – the fact that each coordinate can be independently meaningful. To account for this, we introduce the following modifications: (1) We tune over a more general class of kernels $K_{p,q}(x, x') = \exp(-\|x - x'\|_p^q/L^q)$ that are positive definite for $0 < q \le p \le 2$ [22, Theorems 1, 5], and (2) we tune over using the full AGOP and just the diagonal of the AGOP.

In addition to the above changes, we also implemented optimizations to speed up computations for categorical variables and an adaptive approach for tuning bandwidth separately for data on each leaf. Additional details and the selected hyperparameter search spaces are provided in Appendix B.

**Tree-based data partitioning** We now explain how xRFM builds a binary tree to partition data. First, given a dataset *S* with *n* samples, we subsample *m* points and train a leaf RFM (called a *split model*) on this subsample. We extract the top eigenvector, *v*, of the AGOP from the split model and create two subsets of the *n* datapoints: $S_1 = \{x \in S ; v^T x > \text{Median}(v^T z \text{ for } z \in S)\}$ and $S_2 = \{x \in S ; v^T x \le \text{Median}(v^T z \text{ for } z \in S)\}$. We repeat this procedure on $S_1$ and $S_2$ and their corresponding children until all leaves are less than a maximum leaf size *C*. We finally train a leaf RFM on each of these leaves. This procedure is illustrated in Fig. 1A and detailed in the 'TreePartition' procedure of Algorithm B.2.

A key advantage of our split approach is that it groups together data points based on the features most relevant for prediction, as is captured by the top eigenvector of the AGOP. Prior works have also split data using the random forest procedure. For example, the authors of [14] train TabPFN-v2 on the samples routed to a given leaf of a tree in a random forest. The procedure in xRFM differs as it stratifies samples by projection onto a direction rather than individual coordinates and produces only one balanced tree instead of a forest.
<figure><img src="image_3.png" /><figcaption>**Figure 3:** Performance comparison across the 17 large datasets from meta-test (70,000-500,000 samples). (A, B) Percentage improvement over MLP error on (A) regression and (B) classification datasets. Average percentage improvement is denoted as a diamond. Points denote individual dataset results (points are jittered for visibility).</figcaption></figure>

# 4 xRFM results

We now apply xRFM to three tabular data benchmarks: the TALENT benchmark [25], the TabArena-Lite benchmark [9], and large datasets from the meta-test benchmark [15]. We use TALENT and TabArena-Lite to evaluate xRFM on datasets of various sizes between 500 and 150,000 training samples. We use the meta-test benchmark to evaluate xRFM on larger datasets with at least 70,000 and up to 500,000 samples (a setting in which direct linear solvers become intractable, taking more than 40GB VRAM). We describe the performance measures used for these benchmarks in Appendix B.

**Results on the TALENT and TabArena-Lite benchmarks.** The TALENT benchmark consists of 300 total datasets comparing 31 different supervised learning algorithms [25]. Among these 31 algorithms are strong, widely used predictive models including Gradient Boosted Decision Tree (GBDT) variants (like XGBoost, CatBoost, LightGBM), hyper-parameter optimized neural networks (RealMLP), and recent transformer-based foundation models (TabPFN-v2). TabArena-Lite contains 51 total datasets comparing 15 different supervised learning algorithms including many of those in TALENT.

xRFM is the best performing method on TALENT regression tasks according to all aggregation metrics over RMSEs on individual datasets (Fig. 2A). It is also the fastest method per configuration, although TabPFN-v2 does not incur a 100× overhead for hyperparameter tuning. xRFM is also competitive on classification datasets (Fig. 2B, C). We also compare the performance (in Elo) and inference time of xRFM using the TabArena-Lite benchmark (Fig. 2). In particular, when compared to all default and tuned methods for this benchmark, tuned xRFM is among the top three methods for regression while being orders of magnitude faster for inference. Indeed, xRFM lies along the empirical Pareto frontier, meaning that there is no method that dominates xRFM in both performance and inference time for these tasks (Fig. 2D). For classification tasks, xRFM is near the empirical Pareto frontier (Fig. 2E, F).

**Results on large datasets from the meta-test benchmark.** To analyze xRFM performance on large datasets beyond those in TALENT and TabArena, we consider the 17 largest datasets in the meta-test benchmark from [15] (Table C.6). We compare the performance of xRFM to other models with results reported in the literature (Fig. 3). On this benchmark, xRFM is best on regression tasks, and second best on classification tasks. All results are presented in Tables C.6, C.7.

**Interpretability with xRFM** An advantage of xRFM is that it immediately provides a means of identifying features relevant for prediction (without stacking on any additional interpretability methods). Namely, we can extract learned AGOPs of leaf RFMs and visualize the features they select. We show examples of this interpretability in Appendix A.

# 5 Discussion

Overall, we have shown that xRFM is an effective algorithm for inference from tabular data that scales to essentially unlimited data sizes and achieves performance exceeding or comparable to the current state-of-the-art. It combines the advantages of tree-based methods with the power and elegance of feature-learning kernel machines. We envision that xRFM will be used for both high-performing predictive modeling and uncovering heterogeneous structure in large-scale tabular data.
## References

[1] E. Abbe, E. B. Adsera, and T. Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In P.-L. Loh and M. Raginsky, editors, *Proceedings of Thirty Fifth Conference on Learning Theory*, volume 178 of *Proceedings of Machine Learning Research*, pages 4782–4887. PMLR, 02–05 Jul 2022.

[2] N. Aronszajn. Theory of reproducing kernels. *Transactions of the American mathematical society*, 68(3):337–404, 1950.

[3] D. Beaglehole, A. Radhakrishnan, E. Boix-Adserà, and M. Belkin. Toward universal steering and monitoring of ai models, 2025.

[4] D. Beaglehole, P. Súkeník, M. Mondelli, and M. Belkin. Average gradient outer product as a mechanism for deep neural collapse. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, *Advances in Neural Information Processing Systems*, volume 37, pages 130764–130796. Curran Associates, Inc., 2024.

[5] B. Bischl, G. Casalicchio, T. Das, M. Feurer, S. Fischer, P. Gijsbers, S. Mukherjee, A. C. Müller, L. Németh, L. Oala, L. Purucker, S. Ravi, J. N. van Rijn, P. Singh, J. Vanschoren, J. van der Velde, and M. Wever. Openml: Insights from 10 years and more than a thousand papers. *Patterns*, 6(7), 2025.

[6] T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In *Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining*, pages 785–794, 2016.

[7] A. Damian, J. Lee, and M. Soltanolkotabi. Neural networks can learn representations with gradient descent. In P.-L. Loh and M. Raginsky, editors, *Proceedings of Thirty Fifth Conference on Learning Theory*, volume 178 of *Proceedings of Machine Learning Research*, pages 5413–5452. PMLR, 02–05 Jul 2022.

[8] A. E. Elo. The proposed uscf rating system, its development, theory, and applications. *Chess life*, 22(8):242–247, 1967.

[9] N. Erickson, L. Purucker, A. Tschalzev, D. Holzmüller, P. M. Desai, D. Salinas, and F. Hutter. Tabarena: A living benchmark for machine learning on tabular data. *arXiv preprint arXiv:2506.16791*, 2025.

[10] M. Fernández-Delgado, E. Cernadas, S. Barro, and D. Amorim. Do we need hundreds of classifiers to solve real world classification problems? *The Journal of Machine Learning Research*, 15(1):3133–3181, 2014.

[11] B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. When do neural networks outperform kernel methods?*. *Journal of Statistical Mechanics: Theory and Experiment*, 2021(12):124009, Dec. 2021.

[12] Y. Gorishniy, A. Kotelnikov, and A. Babenko. Tabm: Advancing tabular deep learning with parameter-efficient ensembling, 2025.

[13] L. Grinsztajn, E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, *Advances in Neural Information Processing Systems*, volume 35, pages 507–520. Curran Associates, Inc., 2022.

[14] N. Hollmann, S. Müller, L. Purucker, A. Krishnakumar, M. Körfer, S. B. Hoo, R. T. Schirrmeister, and F. Hutter. Accurate predictions on small data with a tabular foundation model. *Nature*, 637(8045):319–326, 2025.

[15] D. Holzmüller, L. Grinsztajn, and I. Steinwart. Better by default: Strong pre-tuned mlps and boosted trees on tabular data. *Advances in Neural Information Processing Systems*, 37:26577–26658, 2024.

[16] S. Kpotufe, A. Boularias, T. Schultz, and K. Kim. Gradients weights improve regression and classification. *Journal of Machine Learning Research*, 17(22):1–34, 2016.

[17] S. Ma and M. Belkin. Kernel machines that adapt to gpus for effective large batch training. *Proceedings of Machine Learning and Systems*, 1:360–373, 2019.

[18] N. Mallinar, D. Beaglehole, L. Zhu, A. Radhakrishnan, P. Pandit, and M. Belkin. Emergence in non-neural models: grokking modular arithmetic via average gradient outer product, 2024.
[19] A. Narasimha, B. Vasavi, and H. M. Kumar. Significance of nuclear morphometry in benign and malignant breast aspirates. *International Journal of Applied and Basic Medical Research*, 3(1):22–26, 2013.

[20] A. Radhakrishnan, D. Beaglehole, P. Pandit, and M. Belkin. Mechanism for feature learning in neural networks and backpropagation-free machine learning models. *Science*, 383(6690):1461–1467, 2024.

[21] A. Radhakrishnan, M. Belkin, and D. Drusvyatskiy. Linear recursive feature machines provably recover low-rank matrices. *Proceedings of the National Academy of Sciences*, 122(13):e2411325122, 2025.

[22] I. J. Schoenberg. Positive definite functions on spheres. *Duke Mathematical Journal*, 9(1):96 – 108, 1942.

[23] B. Schölkopf and A. J. Smola. *Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond*. MIT Press, 2002.

[24] S. Trivedi, J. Wang, S. Kpotufe, and G. Shakhnarovich. A consistent estimator of the expected gradient outerproduct. In *UAI*, pages 819–828, 2014.

[25] H.-J. Ye, S.-Y. Liu, H.-R. Cai, Q.-L. Zhou, and D.-C. Zhan. A closer look at deep learning on tabular data. *arXiv preprint arXiv:2407.00956*, 2024.

[26] L. Zhu, D. Davis, D. Drusvyatskiy, and M. Fazel. Iteratively reweighted kernel machines efficiently learn sparse functions, 2025.

## A Features learned by xRFM on tabular data

We use two approaches for identifying features selected by the AGOP. The first is to identify the elements along the diagonal of the AGOP with highest magnitude. By definition, these entries indicate how much a leaf RFM's predictions vary when perturbing a given coordinate. As such, they are a natural measure of feature importance. The second approach is to examine the loadings onto the top eigenvectors of the AGOP. This approach allows us to identify joint effects of feature perturbations on the prediction. Namely, if coordinate *i* of the top eigenvector is positive and coordinate *j* is negative, then increasing one of these features and decreasing the other changes the prediction.

We use both approaches above to understand what features are learned by xRFM on tabular data from scikit-learn and meta-test (Fig. A.1). As examples, we study the AGOPs for four such datasets: (1) California housing - a regression task for predicting the average price of a house, (2) Covertype - a multiclass classification task for identifying the dominant tree species in a given location, (3) NYC Taxi Tipping - a regression task for predicting the dollar tip amount in a taxi ride, and (4) Breast cancer - a classification task for identifying malignancy from features of biopsy images.

As the California housing and breast cancer datasets contain fewer than 50k samples (the parameter we used for leaf size), we visualize a single AGOP. For covertype, we visualize one of the AGOPs from a leaf RFM (for this dataset, the leaf RFM AGOPs generally indicate the same pattern), and for taxi tipping we visualize several leaf RFM AGOPs. Upon visualization, it is apparent that AGOPs indicate low rank structure: either they highlight a subset of coordinates relevant for prediction (Fig. A.1A, B) or they have a decay in the eigenvalue spectrum (Fig. A.1C).

In Fig. A.1A and B, we list feature names for the features with highest diagonal AGOP entries (darker shades of red indicate higher values). In all cases, we find that AGOP identifies sensible features for prediction. For the California housing dataset, xRFM identifies longitude, or east-west location, as the most important feature for predicting the average price of a house. Given that beach fronts (and major cities) in California are typically located to the west, the importance of house longitude is consistent with the hypothesis that homes closer to the beach are more expensive on average. For the Covertype prediction dataset, the example AGOP from a leaf RFM shows that elevation, distance to roadways, and distance to firepoints are the most important features. This finding is consistent with the hypothesis that elevations with different climates and the existence of fires / roadways can significantly affect viability of different tree species. For the taxi tipping dataset, we observe that leaf RFMs identify varying local features. For example, one leaf RFM (denoted Taxi Tipping 1 in Fig. A.1B) selected pickup location as an important feature, while this feature was less important for a different leaf RFM (Taxi Tipping 3). Furthermore, fare code and the MTA tax have varying feature relationships at each leaf: in leaf RFM 1 and 2, the fare code and MTA tax has neutral or synergistic
<figure><img src="image_4.png" /><figcaption>**Figure A.1:** Interpreting xRFM through the AGOP of its constituent Leaf RFM models. (A) Examining the most important features for xRFM trained on California Housing (price prediction) and Covertype (dominant tree species prediction) datasets, based on the magnitude of diagonal entries. (B) Examining the features identified across three different Leaf RFM models for the NYC Taxi Tipping dataset. (C) Examining features learned for Breast Cancer detection from processed FNA imaging. The spectrum of this AGOP is plotted and the top eigenvector is shown in a bar plot. The most positive and negative entries of this eigenvector are boxed.</figcaption></figure>

effects on the prediction value, while for leaf RFM 3, increasing fare code has the opposite effect on prediction as increasing MTA tax (shown as a blue square in Fig. A.1B).

For the breast cancer dataset, the most important feature found by xRFM is the average number of concave points in the biopsy image, which has been shown to be a significant indicator of malignancy [19]. When examining the top eigenvector of the AGOP, we find that standard error in the compactness measurements of cell nuclei is the feature with highest importance that has an opposite effect on the malignancy label as concavity (Fig. A.1C). This finding suggests that benign cells have less uniform compactness than malignant cells.

# B Methods

## B.1 Code availability

Code for xRFM (following a scikit-learn-style API) is available at: https://github.com/dmbeaglehole/xRFM.

## B.2 Additional leaf-RFM modifications

Typical choices of the *T* and *c* parameters for kernel-RFM in practice are *T* ≤ 10 and *c* ∈ {1/4, 1/2} [20, 3, 4, 18]. (In practice, we return the *f*<sub>*t*</sub> with best validation performance rather than returning *f*<sub>*T*</sub>.) When training labels, *y*, are multi-dimensional, we use the Jacobian of *ŷ* instead of the gradient in Step (2) (i.e., averaging the AGOP over output dimensions). Kernel-RFM is particularly effective at identifying low dimensional subspaces (or subsets of variables) relevant for prediction [21, 26], making it a useful interpretability tool.

Below, we provide further detail on the RFM modifications we made to implement leaf RFM.

(1) We tune over a more general class of kernels $K_{p,q}(x, x') = \exp(-\|x - x'\|_p^q / L^q)$ that are positive definite for $0 < q \le p \le 2$ [22, Theorems 1, 5]. We typically search over $p \in U(0.7, 1.4)$ and $p \in \{q, 2\}$ (Table B.1).

(2) We tune over using the full AGOP and just the diagonal of the AGOP. The latter is known to be a theoretically grounded approach for coordinate selection [26], and introduces an axis-aligned bias that has been observed to match the structure of tabular data [13].
We also make the following changes to improve leaf RFM runtime on tabular data and to allow xRFM to adapt to the variance of input variables at each leaf:

(3) We speed up computation for kernels with $q = 1$ on categorical variables taking $c$ values by precomputing possible kernel entries as follows. We restrict the AGOP matrix to be block diagonal with a block for the $c \times c$ entries corresponding to the categorical variable. Letting $M_c \in \mathbb{R}^{c \times c}$ denote this block and $e_i \in \{0, 1\}^c$ denote the one-hot embedding of the categorical variable when it takes on value $i$, we precompute $M_c^{1/2}(e_i - e_j)$ for all $i, j \in [c]$.

(4) We tune over whether or not to use *adaptive bandwidth*, which involves separately scaling $L$ at each leaf RFM by $(\text{Median}(\|x - x'\|_p))^{-1}$ for $x \neq x'$ at every leaf independently. Adaptive scaling allows xRFM to adapt to the variance of covariates at different leaves.

## B.3 Metrics

**Performance evaluation measures.** When measuring performance on these datasets, we use Root Mean Square Error (RMSE) for regression tasks (as is used in TALENT), and we use classification error (1 – classification accuracy) for classification tasks. For TabArena-Lite, binary classification performance for individual datasets are measured by AUROC, while multi-class datasets are evaluated with log-loss. To measure performance on aggregate over TALENT, we consider the following aggregation metrics on individual dataset performance measures: Shifted Geometric Mean, Arithmetic Mean, and Normalized Arithmetic Mean. TabArena-Lite uses Elo [8], a rating system that calculates the relative skill levels of each method based on the probability of winning when pairs of methods are compared on individual datasets (see [9] for how Elo is estimated in this benchmark).

Following [15], we report the shifted geometric mean of the error, which is the geometric mean of the error after shifting by a small value to prevent over-sensitivity to datasets with small errors. This metric is defined as follows.

**Definition 1.** For a given set of errors on a benchmark $\varepsilon_1, \dots, \varepsilon_N$, the shifted geometric mean (SGM$_\varepsilon$) with parameter $\varepsilon$ takes value:

$$ \text{SGM}_{\varepsilon} = \exp \left( \frac{1}{N} \sum_{i=1}^{N} \log(\varepsilon + \varepsilon_i) \right). $$

In this work, as in [15], we use $\varepsilon = 0.01$.

For regression tasks, we use normalized Root-mean-square-error (nRMSE), which is defined as follows.

**Definition 2.** The normalized Root-mean-square-error is defined as,

$$ n\text{RMSE} = \sigma_y^{-1} \sqrt{\frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2}, \quad \text{where } \sigma_y = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (y_i - \bar{y})^2} $$

When we refer to other normalized metrics, such as those used in Table C.1, Table C.2, Table C.3, Table C.4, Table C.5, we are min-max normalizing errors across methods for each dataset (see below).

**Definition 3.** The normalized error $\tilde{E}_j$ for method j on a given dataset, where the un-normalized error is $E_j$, has the form:

$$ \tilde{E}_j = \frac{E_j - E_{\min}}{E_{\max} - E_{\min}}, \quad E_{\min} = \min_k E_k, \quad E_{\max} = \max_k E_k. $$

## B.4 Hyperparameters

**Data pre-processing.** For the TALENT benchmark, each method's hyperparameters are tuned after fixing a single data normalization and categorical encoding scheme. For xRFM, kernel ridge regression, and standard kernel RFM, we one-hot encode categorical features and z-score input coordinates separately. For meta-test, methods also tune over choice of ordinal or one-hot encoding of categorical variables. On meta-test, we also normalize data by z-scoring input coordinates separately prior to tuning xRFM parameters.
<table><caption>**Table B.1:** Search spaces for xRFM on the TALENT (Figure 2), TabArena-Lite (Figure 2), and Meta-test benchmarks (Figure 3).</caption><thead><tr><th>Hyperparameter</th><th>TALENT</th><th>TabArena-Lite</th><th>Meta-test</th></tr></thead><tbody><tr><td>Bandwidth</td><td>log U(1, 200)</td><td>log U(0.5, 200)</td><td>log U(0.4, 80)</td></tr><tr><td>Bandwidth Mode</td><td>{constant}</td><td>{constant}</td><td>{constant, adaptive}</td></tr><tr><td>Categorical Transformations</td><td>{one_hot}</td><td>{one_hot}</td><td>{ordinal_encryption, one_hot}</td></tr><tr><td>Diagonal</td><td>{False, True}</td><td>{False, True}</td><td>{False, True}</td></tr><tr><td>Early Stop Multiplier</td><td>1.1</td><td>1.1</td><td>1.1</td></tr><tr><td>Exponent q</td><td>U(0.7, 1.4)</td><td>U(0.7, 1.4)</td><td>U(0.7, 1.3)</td></tr><tr><td>Kernel Type</td><td>{80%: K<sub>p,q</sub>, 20%K<sub>2,q</sub>}</td><td>{80%: K<sub>p,q</sub>, 20%K<sub>2,q</sub>}</td><td>{80%: K<sub>p,q</sub>, 20%K<sub>2,q</sub>}</td></tr><tr><td>p (when kernel type is K<sub>p,q</sub>)</td><td>U(q, q + 0.8(2 - q))</td><td>U(q, q + 0.8(2 - q))</td><td>U(q, q + 0.8(2 - q))</td></tr><tr><td>Normalization</td><td>{standard}</td><td>{standard}</td><td>{standard}</td></tr><tr><td>Regularization</td><td>log U(10<sup>-6</sup>, 1)</td><td>log U(10<sup>-6</sup>, 10)</td><td>log U(10<sup>-5</sup>, 50)</td></tr><tr><td>Refill size (N<sub>val</sub>)</td><td>1500</td><td>1500</td><td>1500</td></tr></tbody></table>

**Details of various methods.** For scaling kernel ridge regression and the standard kernel RFM to large TALENT datasets, we used Eigenpro-2 (EP2) [17] that was initialized with the coefficients obtained from directly solving RFM on a random sample of 70,000 points. To tune the optimization hyperparameters for EP2, we tuned the model parameters for 100 trials on a random 70,000 sample subset, then for the final run, we used the tuned hyperparameters (and for kernel RFM the AGOP from the best iteration). For TabPFN-v2, we used the code provided from the benchmark github (https://github.com/LAMDA-Tabular/TALENT), which subsampled to 10,000 samples to avoid out-of-memory issues. The provided code did not apply TabPFN-v2 to datasets with more than 10 classes. For efficient $L_p^q$ kernel computations on GPU we used the KerMac library (https://github.com/Kernel-Machines/kermac).

## B.5 Algorithmic details

**Note on AGOP computation** For computing gradients of the predictor on training data (for AGOP computation), we omit the contribution to the gradient from the kernel evaluation between each training point and itself. This is because the kernel function is often not differentiable (e.g. Laplace kernels) when evaluated for two identical points.
<caption>**Algorithm B.1 Leaf RFM**</caption>

**Require:**
* $x^{(1)}, \dots, x^{(n)} \in \mathbb{R}^d, y \in \mathbb{R}^{n \times c}$: Train data
* $X_{\text{val}} \in \mathbb{R}^{m \times d}, y_{\text{val}} \in \mathbb{R}^{m \times c}$: Validation data
* $K(\cdot, \cdot; L, p)$: Kernel parametrized by bandwidth $L \in \mathbb{R}^+$ and exponent $p \in (0, 2]$
* $\tau \in \mathbb{Z}^+$: Number of iterations
* $\lambda \in \mathbb{R}^+$: Ridge parameter
* $\varepsilon \in \mathbb{R}^+$: Stability parameter
* `use_diag`: Boolean indicating whether to use diagonal of AGOP only
* `adapt_bandwidth`: Boolean indicating whether to adapt bandwidth

$M_0 \leftarrow I_{d \times d}$
$X = [x^{(1)}, \dots, x^{(n)}]^\top \in \mathbb{R}^{n \times d}$
**if adapt_bandwidth then**
$L \leftarrow \text{AdaptBandwidth}(L)$
**end if**
**for** $t = 0, \dots, \tau - 1$ **do**
**if** `use_diag` **then**
$X_M \leftarrow X \odot \text{diag}(M_t)^{1/2}$
Solve $\alpha_t$ such that $(K(X_M, X_M) + \lambda I)\alpha_t = y$
$f^{(t)}(x) = K(x \odot \text{diag}(M_t)^{1/2}, X_M)\alpha_t$
**else**
$X_M \leftarrow XM_t^{1/2}$
Solve $\alpha_t$ such that $(K(X_M, X_M) + \lambda I)\alpha_t = y$
$f^{(t)}(x) = K(M_t^{1/2}x, X_M)\alpha_t$
**end if**
Compute $E_t \leftarrow \text{Error}(f^{(t)}, X_{\text{val}}, y_{\text{val}})$
$M_{t+1} \leftarrow \frac{1}{n} \sum_{i=1}^n \nabla_x f^{(t)}(x^{(i)}) \nabla_x f^{(t)}(x^{(i)})^\top \in \mathbb{R}^{d \times d}$
$M_{t+1} \leftarrow \frac{M_{t+1}}{\varepsilon + \max_{i,j} M_{t+1}[i, j]}$
**end for**
$t^* \leftarrow \arg \min_t E_t$
**return** $\alpha_{t^*}, M_{t^*}$
$\triangleright$ KRR coefficients: $\alpha_{t^*}$, feature matrix: $M_{t^*}$ from best iteration on val. set
**Algorithm B.2 TreePartition**

**Require:**

* $\mathcal{D} = \{(x^{(1)}, y^{(1)}), \dots, (x^{(n)}, y^{(n)})\}$: Full dataset with $x^{(i)} \in \mathbb{R}^d, y^{(i)} \in \mathbb{R}^c$

* $K(\cdot, \cdot; L, p)$: Kernel parametrized by bandwidth $L \in \mathbb{R}^+$ and exponent $p \in (0, 2]$

* $N \in \mathbb{Z}^+$: Number of sample points for Leaf RFM

* $L \in \mathbb{Z}^+$: Maximum leaf size

* $\lambda \in \mathbb{R}^+$: Ridge parameter

**function** TREEPARTITION($\mathcal{D}$)

if $|\mathcal{D}| \le L$ then

return Leaf node with dataset $\mathcal{D}$

end if

Sample $N$ points $S = \{(x^{(a_1)}, y^{(a_1)}), \dots, (x^{(a_N)}, y^{(a_N)})\}$ from $\mathcal{D}$

$X_s = [x^{(a_1)}, \dots, x^{(a_N)}]^\top \in \mathbb{R}^{N \times d}$

$y_s = [y^{(a_1)}, \dots, y^{(a_N)}]^\top \in \mathbb{R}^{N \times c}$

Solve $\alpha$ such that $(K(X_s, X_s) + \lambda I)\alpha = y$

▷ Fit Leaf RFM on sampled data

Define predictor $f(x) = K(x, X_s)\alpha$

Compute AGOP: $M \leftarrow \frac{1}{N} \sum_{i=1}^{N} \nabla_x f(x^{(a_i)}) \nabla_x f(x^{(a_i)})^\top \in \mathbb{R}^{d \times d}$

Extract top eigenvector $v_1$ of $M$

▷ Principal direction

Project all data points: $p^{(i)} \leftarrow v_1^\top x^{(i)}$ for $i = 1, \dots, |\mathcal{D}|$

Compute median projection: $m \leftarrow \text{Median}(\{p^{(1)}, \dots, p(|\mathcal{D}|)\})$

Split dataset:

$\mathcal{D}_{\text{left}} \leftarrow \{(x^{(i)}, y^{(i)}) \in \mathcal{D} : v_1^\top x^{(i)} \le m\}$

$\mathcal{D}_{\text{right}} \leftarrow \{(x^{(i)}, y^{(i)}) \in \mathcal{D} : v_1^\top x^{(i)} > m\}$

left\_child \leftarrow \text{TreePartition}(\mathcal{D}_{\text{left}})

▷ Recursive call

right\_child \leftarrow \text{TreePartition}(\mathcal{D}_{\text{right}})

▷ Recursive call

return Internal node with splitting vector $v_1$, threshold $m$, and children

end function

**Algorithm B.3 Route (find the leaf that contains a point)**

**Require:**

* $\mathcal{T}$: a (possibly trained) tree whose internal nodes store

– splitting vector $v_1 \in \mathbb{R}^d$

– threshold $m \in \mathbb{R}$

* $x \in \mathbb{R}^d$: query point

**function** ROUTE($x, \mathcal{T}$)

$r \leftarrow \mathcal{T}.\text{root}$

▷ Initialize current node $r$ from the tree

while $r$ is an internal node do

if $v_1^\top x \le r$.threshold then

▷ Check if projection is less than (median) threshold

$r \leftarrow r$.left\_child

else

$r \leftarrow r$.right\_child

end if

end while

return $r$

▷ $r$ is now the leaf node that contains $x$

end function
**Algorithm B.4** xRFM (*training*)

**Require:**

• $\mathcal{D}_{train}, \mathcal{D}_{val}$: training and validation sets

• $K(\cdot, \cdot; L, p)$: kernel (bandwidth $L$, exponent $p$)

• TreeHyp = {$N, L_{max}, \lambda$split} : hyper-parameters for TREEPARTITION

• LeafHyp = {$\tau, \lambda_{leaf}, \varepsilon, use\_diag$} : hyper-parameters for LEAFRFM

**function** xRFM-FIT($\mathcal{D}_{train}, \mathcal{D}_{val}$)
  $\mathcal{T} \leftarrow \text{TREEPARTITION}(\mathcal{D}_{train}, K, \text{TreeHyp})$ \quad $\triangleright$ Alg. B.2
  **for all** leaf node $\ell \in \text{LEAVES}(\mathcal{T})$ **do**
    $\mathcal{D}_{\ell}^{\text{train}} \leftarrow$ data stored in $\ell$
    $\mathcal{D}_{\ell}^{\text{val}} \leftarrow \{(x, y) \in \mathcal{D}_{\text{val}} : \text{ROUTE}(x, \mathcal{T}) = \ell\}$
    $(\alpha_{\ell}, M_{\ell}) \leftarrow \text{LEAFRFM}(\mathcal{D}_{\ell}^{\text{train}}, \mathcal{D}_{\ell}^{\text{val}}, K, \text{LeafHyp})$ \quad $\triangleright$ Alg. B.1
    Define predictor $f_{\ell}(x)$ according to $(\alpha_{\ell}, M_{\ell})$ and store it in $\ell$
  **end for**
  **return** $\mathcal{T}$
  **end function**
  $\triangleright$ Tree whose leaves now carry trained predictors

**Algorithm B.5** xRFM (*inference*)

**Require:**

• $\mathcal{T}$: trained tree returned by XRFM-FIT (Alg. B.4)

• $x \in \mathbb{R}^d$: test point

**function** XRFM-PREDICT($\mathcal{T}, x$)
  $\ell \leftarrow \text{ROUTE}(x, \mathcal{T})$ \quad $\triangleright$ Alg. B.3
  $\hat{y} \leftarrow f_{\ell}(x)$ \quad $\triangleright$ Leaf predictor stored in $\ell$
  **return** $\hat{y}$
  **end function**
C All results

<table><caption>**Table C.1:** Full TALENT Regression results across 100 datasets. Rank is the average rank among the ordered methods over all datasets. Score is the metric we use to compare methods in Figure 2, in this case $SGM_{ε}$. Normalized score is the arithmetic mean of the normalized nRMSE. Top-X (%) is the percentage of datasets for which that method is in the top X ranks. The final column is the shifted geometric mean error ($SGM_{ε}$).</caption><thead><tr><th>Method</th><th>Rank</th><th>Score</th><th>Norm. Score</th><th>Top-1 (%)</th><th>Top-3 (%)</th><th>Top-5 (%)</th><th>Top-8 (%)</th><th>SGM<sub>ε</sub></th></tr></thead><tbody><tr><td>xRFM</td><td><strong>4.70</strong></td><td><strong>0.311</strong></td><td><strong>0.036</strong></td><td><strong>20.0</strong></td><td><strong>56.0</strong></td><td><strong>69.0</strong></td><td><strong>84.0</strong></td><td><strong>0.311</strong></td></tr><tr><td>PFN-v2</td><td>6.55</td><td>0.323</td><td>0.067</td><td><strong>20.0</strong></td><td>45.0</td><td>57.0</td><td>72.0</td><td>0.323</td></tr><tr><td>CatBoost</td><td>7.79</td><td>0.336</td><td>0.053</td><td>7.00</td><td>24.0</td><td>41.0</td><td>68.0</td><td>0.336</td></tr><tr><td>RealMLP</td><td>8.20</td><td>0.329</td><td>0.076</td><td>8.00</td><td>21.0</td><td>37.0</td><td>60.0</td><td>0.329</td></tr><tr><td>ModernNCA</td><td>9.39</td><td>0.365</td><td>0.097</td><td>14.0</td><td>30.0</td><td>38.0</td><td>57.0</td><td>0.365</td></tr><tr><td>LightGBM</td><td>9.91</td><td>0.353</td><td>0.062</td><td>6.00</td><td>16.0</td><td>34.0</td><td>52.0</td><td>0.353</td></tr><tr><td>XGBoost</td><td>10.6</td><td>0.355</td><td>0.071</td><td>2.00</td><td>12.0</td><td>32.0</td><td>47.0</td><td>0.355</td></tr><tr><td>TabR</td><td>10.7</td><td>0.365</td><td>0.133</td><td>5.00</td><td>20.0</td><td>34.0</td><td>46.0</td><td>0.365</td></tr><tr><td>FTT</td><td>12.0</td><td>0.388</td><td>0.149</td><td>3.00</td><td>11.0</td><td>17.0</td><td>33.0</td><td>0.388</td></tr><tr><td>Laplace KRR</td><td>12.3</td><td>0.373</td><td>0.102</td><td>3.00</td><td>11.0</td><td>21.0</td><td>35.0</td><td>0.373</td></tr><tr><td>MLP-PLR</td><td>12.7</td><td>0.381</td><td>0.142</td><td>0.0</td><td>8.00</td><td>15.0</td><td>34.0</td><td>0.381</td></tr><tr><td>Excelformer</td><td>12.7</td><td>0.399</td><td>0.157</td><td>0.0</td><td>2.00</td><td>9.00</td><td>28.0</td><td>0.399</td></tr><tr><td>PTARL</td><td>13.1</td><td>0.390</td><td>0.114</td><td>2.00</td><td>6.00</td><td>9.00</td><td>20.0</td><td>0.390</td></tr><tr><td>RandomForest</td><td>13.3</td><td>0.389</td><td>0.088</td><td>5.00</td><td>12.0</td><td>22.0</td><td>35.0</td><td>0.389</td></tr><tr><td>AutoInt</td><td>14.3</td><td>0.399</td><td>0.158</td><td>1.00</td><td>2.00</td><td>6.00</td><td>12.0</td><td>0.399</td></tr><tr><td>Node</td><td>14.4</td><td>0.451</td><td>0.195</td><td>0.0</td><td>6.00</td><td>18.0</td><td>31.0</td><td>0.451</td></tr><tr><td>MLP</td><td>15.2</td><td>0.418</td><td>0.167</td><td>1.00</td><td>3.00</td><td>4.00</td><td>11.0</td><td>0.418</td></tr><tr><td>DCN2</td><td>15.3</td><td>0.473</td><td>0.227</td><td>0.0</td><td>2.00</td><td>10.0</td><td>20.0</td><td>0.473</td></tr><tr><td>ResNet</td><td>15.6</td><td>0.426</td><td>0.175</td><td>0.0</td><td>4.00</td><td>8.00</td><td>15.0</td><td>0.426</td></tr><tr><td>Tangos</td><td>16.3</td><td>0.432</td><td>0.173</td><td>0.0</td><td>1.00</td><td>4.00</td><td>9.00</td><td>0.432</td></tr><tr><td>SNN</td><td>17.2</td><td>0.419</td><td>0.174</td><td>0.0</td><td>2.00</td><td>4.00</td><td>8.00</td><td>0.419</td></tr><tr><td>kNN</td><td>19.1</td><td>0.459</td><td>0.190</td><td>2.00</td><td>4.00</td><td>4.00</td><td>11.0</td><td>0.459</td></tr><tr><td>TabNet</td><td>21.5</td><td>0.475</td><td>0.227</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.475</td></tr><tr><td>GrowNet</td><td>23.0</td><td>0.619</td><td>0.323</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.619</td></tr><tr><td>SVM</td><td>23.2</td><td>0.612</td><td>0.346</td><td>0.0</td><td>0.0</td><td>1.00</td><td>1.00</td><td>0.612</td></tr><tr><td>TabTransformer</td><td>26.4</td><td>1.07</td><td>0.750</td><td>1.00</td><td>1.00</td><td>2.00</td><td>3.00</td><td>1.07</td></tr><tr><td>SwitchTab</td><td>26.9</td><td>1.29</td><td>0.745</td><td>0.0</td><td>0.0</td><td>0.0</td><td>1.00</td><td>1.29</td></tr><tr><td>Danets</td><td>27.2</td><td>1.12</td><td>0.830</td><td>0.0</td><td>1.00</td><td>1.00</td><td>3.00</td><td>1.12</td></tr></tbody></table>
<table><caption>**Table C.2:** Full TALENT Multiclass classification (≤ 10 classes) results. Rank is the average rank among the ordered methods over all datasets. (Normalized) Score is the metric we use to compare methods in Figure 2. In this case score is the mean classification accuracy (1 minus the error in Figure 2). Top-X (%) is the percentage of datasets for which that method is in the top X ranks. The final column is the shifted geometric mean error (SGM<sub>e</sub>).</caption><thead><tr><th>Method</th><th>Rank</th><th>Score</th><th>Norm. Score</th><th>Top-1 (%)</th><th>Top-3 (%)</th><th>Top-5 (%)</th><th>Top-8 (%)</th><th>SGM<sub>e</sub></th></tr></thead><tbody><tr><td>PFN-v2</td><td><strong>7.15</strong></td><td>0.823</td><td><strong>0.935</strong></td><td><strong>25.0</strong></td><td><strong>51.5</strong></td><td><strong>63.2</strong></td><td><strong>70.6</strong></td><td>0.108</td></tr><tr><td>xRFM</td><td>7.60</td><td>0.825</td><td>0.933</td><td>11.8</td><td>29.4</td><td>48.5</td><td>66.2</td><td>0.107</td></tr><tr><td>RealMLP</td><td>7.64</td><td>0.823</td><td>0.918</td><td>10.3</td><td>20.6</td><td>48.5</td><td>66.2</td><td>0.107</td></tr><tr><td>TabR</td><td>7.98</td><td><strong>0.828</strong></td><td>0.932</td><td>8.82</td><td>27.9</td><td>41.2</td><td>60.3</td><td><strong>0.106</strong></td></tr><tr><td>ModernNCA</td><td>8.88</td><td>0.825</td><td>0.912</td><td>10.3</td><td>30.9</td><td>44.1</td><td>61.8</td><td>0.106</td></tr><tr><td>CatBoost</td><td>10.3</td><td>0.819</td><td>0.905</td><td>1.47</td><td>19.1</td><td>36.8</td><td>51.5</td><td>0.117</td></tr><tr><td>LightGBM</td><td>11.7</td><td>0.813</td><td>0.883</td><td>5.88</td><td>22.1</td><td>33.8</td><td>44.1</td><td>0.125</td></tr><tr><td>XGBoost</td><td>12.1</td><td>0.815</td><td>0.886</td><td>4.41</td><td>19.1</td><td>26.5</td><td>41.2</td><td>0.124</td></tr><tr><td>ResNet</td><td>12.3</td><td>0.807</td><td>0.879</td><td>0.0</td><td>4.41</td><td>16.2</td><td>36.8</td><td>0.127</td></tr><tr><td>FTT</td><td>12.8</td><td>0.810</td><td>0.868</td><td>0.0</td><td>5.88</td><td>10.3</td><td>29.4</td><td>0.125</td></tr><tr><td>MLP-PLR</td><td>13.2</td><td>0.815</td><td>0.881</td><td>0.0</td><td>7.35</td><td>11.8</td><td>25.0</td><td>0.119</td></tr><tr><td>Laplace KRR</td><td>13.5</td><td>0.804</td><td>0.864</td><td>5.88</td><td>16.2</td><td>20.6</td><td>33.8</td><td>0.133</td></tr><tr><td>DCN2</td><td>14.0</td><td>0.806</td><td>0.865</td><td>1.47</td><td>4.41</td><td>10.3</td><td>26.5</td><td>0.127</td></tr><tr><td>MLP</td><td>14.2</td><td>0.805</td><td>0.864</td><td>0.0</td><td>4.41</td><td>8.82</td><td>19.1</td><td>0.131</td></tr><tr><td>SNN</td><td>14.9</td><td>0.805</td><td>0.858</td><td>0.0</td><td>0.0</td><td>4.41</td><td>8.82</td><td>0.129</td></tr><tr><td>AutoInt</td><td>15.7</td><td>0.804</td><td>0.858</td><td>0.0</td><td>0.0</td><td>1.47</td><td>7.35</td><td>0.130</td></tr><tr><td>RandomForest</td><td>15.7</td><td>0.797</td><td>0.831</td><td>5.88</td><td>8.82</td><td>13.2</td><td>27.9</td><td>0.137</td></tr><tr><td>Excelformer</td><td>16.1</td><td>0.805</td><td>0.853</td><td>0.0</td><td>0.0</td><td>4.41</td><td>17.6</td><td>0.127</td></tr><tr><td>Tangos</td><td>16.5</td><td>0.797</td><td>0.844</td><td>0.0</td><td>1.47</td><td>4.41</td><td>13.2</td><td>0.133</td></tr><tr><td>TabCaps</td><td>16.5</td><td>0.797</td><td>0.837</td><td>0.0</td><td>1.47</td><td>5.88</td><td>13.2</td><td>0.134</td></tr><tr><td>Danets</td><td>17.7</td><td>0.795</td><td>0.829</td><td>0.0</td><td>1.47</td><td>2.94</td><td>5.88</td><td>0.137</td></tr><tr><td>PTARL</td><td>19.1</td><td>0.796</td><td>0.808</td><td>0.0</td><td>1.47</td><td>1.47</td><td>4.41</td><td>0.139</td></tr><tr><td>kNN</td><td>20.1</td><td>0.786</td><td>0.772</td><td>2.94</td><td>7.35</td><td>10.3</td><td>14.7</td><td>0.152</td></tr><tr><td>LogReg</td><td>20.2</td><td>0.767</td><td>0.755</td><td>1.47</td><td>4.41</td><td>7.35</td><td>13.2</td><td>0.162</td></tr><tr><td>TabTransformer</td><td>21.5</td><td>0.771</td><td>0.739</td><td>0.0</td><td>1.47</td><td>1.47</td><td>2.94</td><td>0.155</td></tr><tr><td>Node</td><td>22.0</td><td>0.767</td><td>0.742</td><td>0.0</td><td>4.41</td><td>7.35</td><td>10.3</td><td>0.173</td></tr><tr><td>SVM</td><td>23.4</td><td>0.750</td><td>0.697</td><td>2.94</td><td>5.88</td><td>5.88</td><td>8.82</td><td>0.182</td></tr><tr><td>GrowNet</td><td>24.4</td><td>0.691</td><td>0.596</td><td>0.0</td><td>0.0</td><td>2.94</td><td>5.88</td><td>0.216</td></tr><tr><td>SwitchTab</td><td>25.0</td><td>0.748</td><td>0.674</td><td>0.0</td><td>1.47</td><td>1.47</td><td>2.94</td><td>0.187</td></tr><tr><td>TabNet</td><td>26.0</td><td>0.753</td><td>0.615</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.183</td></tr><tr><td>NaiveBayes</td><td>29.4</td><td>0.609</td><td>0.327</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.321</td></tr><tr><td>NCM</td><td>30.5</td><td>0.627</td><td>0.278</td><td>0.0</td><td>0.0</td><td>1.47</td><td>1.47</td><td>0.326</td></tr></tbody></table>
<table><caption>**Table C.3:** Full TALENT Multiclass classification (> 10 classes) results. Rank is the average rank among the ordered methods over all datasets. (Normalized) Score is the metric we use to compare methods in Figure 2. In this case score is the mean classification accuracy (1 minus the error in Figure 2). Top-X (%) is the percentage of datasets for which that method is in the top X ranks. The final column is the shifted geometric mean error (SGM<sub>ε</sub>).</caption><thead><tr><th>Method</th><th>Rank</th><th>Score</th><th>Norm. Score</th><th>Top-1 (%)</th><th>Top-3 (%)</th><th>Top-5 (%)</th><th>Top-8 (%)</th><th>SGM<sub>ε</sub></th></tr></thead><tbody><tr><td>RealMLP</td><td><strong>3.75</strong></td><td><strong>0.730</strong></td><td><strong>0.964</strong></td><td>8.33</td><td><strong>50.0</strong></td><td><strong>91.7</strong></td><td><strong>91.7</strong></td><td><strong>0.164</strong></td></tr><tr><td>xRFM</td><td>5.46</td><td>0.718</td><td>0.949</td><td>25.0</td><td>50.0</td><td>66.7</td><td>83.3</td><td>0.183</td></tr><tr><td>TabR</td><td>6.00</td><td>0.720</td><td>0.948</td><td>16.7</td><td>50.0</td><td>50.0</td><td>66.7</td><td>0.172</td></tr><tr><td>ResNet</td><td>6.33</td><td>0.720</td><td>0.949</td><td>0.0</td><td>16.7</td><td>50.0</td><td>83.3</td><td>0.178</td></tr><tr><td>ModernNCA</td><td>6.83</td><td>0.713</td><td>0.913</td><td><strong>33.3</strong></td><td>41.7</td><td>50.0</td><td>83.3</td><td>0.168</td></tr><tr><td>FTT</td><td>8.79</td><td>0.709</td><td>0.919</td><td>0.0</td><td>8.33</td><td>25.0</td><td>50.0</td><td>0.187</td></tr><tr><td>MLP-PLR</td><td>8.79</td><td>0.712</td><td>0.925</td><td>0.0</td><td>0.0</td><td>25.0</td><td>50.0</td><td>0.185</td></tr><tr><td>MLP</td><td>9.58</td><td>0.710</td><td>0.922</td><td>0.0</td><td>0.0</td><td>8.33</td><td>41.7</td><td>0.190</td></tr><tr><td>Laplace KRR</td><td>10.8</td><td>0.679</td><td>0.889</td><td>0.0</td><td>16.7</td><td>41.7</td><td>50.0</td><td>0.218</td></tr><tr><td>DCN2</td><td>11.2</td><td>0.707</td><td>0.915</td><td>0.0</td><td>8.33</td><td>8.33</td><td>8.33</td><td>0.193</td></tr><tr><td>AutoInt</td><td>12.2</td><td>0.701</td><td>0.899</td><td>0.0</td><td>0.0</td><td>0.0</td><td>16.7</td><td>0.194</td></tr><tr><td>SNN</td><td>12.2</td><td>0.704</td><td>0.906</td><td>0.0</td><td>0.0</td><td>0.0</td><td>33.3</td><td>0.197</td></tr><tr><td>CatBoost</td><td>12.8</td><td>0.679</td><td>0.842</td><td>8.33</td><td>25.0</td><td>33.3</td><td>33.3</td><td>0.218</td></tr><tr><td>Danets</td><td>15.9</td><td>0.687</td><td>0.882</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.215</td></tr><tr><td>Tangos</td><td>16.1</td><td>0.660</td><td>0.823</td><td>0.0</td><td>8.33</td><td>8.33</td><td>16.7</td><td>0.225</td></tr><tr><td>XGBoost</td><td>16.2</td><td>0.678</td><td>0.874</td><td>8.33</td><td>8.33</td><td>8.33</td><td>25.0</td><td>0.236</td></tr><tr><td>Excelformer</td><td>16.6</td><td>0.675</td><td>0.851</td><td>0.0</td><td>0.0</td><td>0.0</td><td>16.7</td><td>0.217</td></tr><tr><td>TabCaps</td><td>16.8</td><td>0.670</td><td>0.853</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.231</td></tr><tr><td>LightGBM</td><td>16.9</td><td>0.668</td><td>0.850</td><td>0.0</td><td>0.0</td><td>0.0</td><td>16.7</td><td>0.261</td></tr><tr><td>PTARL</td><td>18.7</td><td>0.656</td><td>0.823</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.233</td></tr><tr><td>kNN</td><td>20.2</td><td>0.637</td><td>0.800</td><td>0.0</td><td>0.0</td><td>0.0</td><td>8.33</td><td>0.273</td></tr><tr><td>RandomForest</td><td>20.7</td><td>0.620</td><td>0.782</td><td>0.0</td><td>8.33</td><td>8.33</td><td>8.33</td><td>0.311</td></tr><tr><td>TabTransformer</td><td>20.8</td><td>0.582</td><td>0.702</td><td>0.0</td><td>0.0</td><td>8.33</td><td>8.33</td><td>0.307</td></tr><tr><td>LogReg</td><td>22.0</td><td>0.538</td><td>0.622</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.338</td></tr><tr><td>SVM</td><td>25.1</td><td>0.486</td><td>0.523</td><td>0.0</td><td>8.33</td><td>8.33</td><td>8.33</td><td>0.385</td></tr><tr><td>SwitchTab</td><td>25.2</td><td>0.525</td><td>0.603</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.372</td></tr><tr><td>Node</td><td>26.2</td><td>0.536</td><td>0.611</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.402</td></tr><tr><td>GrowNet</td><td>26.2</td><td>0.402</td><td>0.477</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.565</td></tr><tr><td>TabNet</td><td>26.3</td><td>0.486</td><td>0.516</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.368</td></tr><tr><td>NaiveBayes</td><td>29.5</td><td>0.410</td><td>0.380</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.545</td></tr><tr><td>NCM</td><td>30.2</td><td>0.399</td><td>0.339</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.552</td></tr></tbody></table>
<table><caption>**Table C.4:** Full TALENT Binary classification (> 10,000 samples) results. Rank is the average rank among the ordered methods over all datasets. (Normalized) Score is the metric we use to compare methods in Figure 2. In this case score is the mean classification accuracy (1 minus the error in Figure 2). Top-X (%) is the percentage of datasets for which that method is in the top X ranks. The final column is the shifted geometric mean error (SGM<sub>e</sub>).</caption><thead><tr><th>Method</th><th>Rank</th><th>Score</th><th>Norm. Score</th><th>Top-1 (%)</th><th>Top-3 (%)</th><th>Top-5 (%)</th><th>Top-8 (%)</th><th>SGM<sub>e</sub></th></tr></thead><tbody><tr><td>xRFM</td><td><strong>5.96</strong></td><td><strong>0.845</strong></td><td><strong>0.981</strong></td><td><strong>29.6</strong></td><td><strong>55.6</strong></td><td><strong>59.3</strong></td><td>74.1</td><td>0.119</td></tr><tr><td>RealMLP</td><td>7.44</td><td>0.839</td><td>0.951</td><td>7.41</td><td>14.8</td><td>29.6</td><td><strong>77.8</strong></td><td>0.125</td></tr><tr><td>TabR</td><td>7.83</td><td>0.844</td><td>0.970</td><td>18.5</td><td>33.3</td><td>55.6</td><td>70.4</td><td><strong>0.116</strong></td></tr><tr><td>ModernNCA</td><td>9.69</td><td>0.843</td><td>0.963</td><td>3.70</td><td>29.6</td><td>40.7</td><td>59.3</td><td>0.120</td></tr><tr><td>CatBoost</td><td>10.2</td><td>0.827</td><td>0.891</td><td>7.41</td><td>25.9</td><td>33.3</td><td>55.6</td><td>0.128</td></tr><tr><td>LightGBM</td><td>10.5</td><td>0.826</td><td>0.884</td><td>7.41</td><td>18.5</td><td>40.7</td><td>55.6</td><td>0.128</td></tr><tr><td>PFN-v2</td><td>10.9</td><td>0.834</td><td>0.923</td><td>7.41</td><td>22.2</td><td>25.9</td><td>48.1</td><td>0.129</td></tr><tr><td>MLP-PLR</td><td>11.2</td><td>0.827</td><td>0.890</td><td>0.0</td><td>0.0</td><td>18.5</td><td>40.7</td><td>0.131</td></tr><tr><td>XGBoost</td><td>12.1</td><td>0.824</td><td>0.875</td><td>3.70</td><td>18.5</td><td>33.3</td><td>51.9</td><td>0.129</td></tr><tr><td>FTT</td><td>12.4</td><td>0.829</td><td>0.900</td><td>0.0</td><td>11.1</td><td>18.5</td><td>29.6</td><td>0.135</td></tr><tr><td>MLP</td><td>12.7</td><td>0.833</td><td>0.915</td><td>0.0</td><td>7.41</td><td>14.8</td><td>29.6</td><td>0.130</td></tr><tr><td>ResNet</td><td>13.3</td><td>0.834</td><td>0.920</td><td>0.0</td><td>3.70</td><td>7.41</td><td>29.6</td><td>0.130</td></tr><tr><td>DCN2</td><td>13.4</td><td>0.832</td><td>0.919</td><td>0.0</td><td>3.70</td><td>3.70</td><td>22.2</td><td>0.130</td></tr><tr><td>PTARL</td><td>13.4</td><td>0.831</td><td>0.911</td><td>0.0</td><td>3.70</td><td>7.41</td><td>14.8</td><td>0.131</td></tr><tr><td>AutoInt</td><td>13.7</td><td>0.831</td><td>0.911</td><td>0.0</td><td>0.0</td><td>3.70</td><td>11.1</td><td>0.130</td></tr><tr><td>Excelformer</td><td>14.7</td><td>0.821</td><td>0.858</td><td>3.70</td><td>7.41</td><td>7.41</td><td>11.1</td><td>0.143</td></tr><tr><td>SNN</td><td>14.8</td><td>0.831</td><td>0.912</td><td>0.0</td><td>3.70</td><td>7.41</td><td>18.5</td><td>0.131</td></tr><tr><td>Laplace KRR</td><td>17.0</td><td>0.826</td><td>0.879</td><td>3.70</td><td>11.1</td><td>14.8</td><td>14.8</td><td>0.138</td></tr><tr><td>Danets</td><td>17.4</td><td>0.827</td><td>0.887</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.133</td></tr><tr><td>TabCaps</td><td>18.4</td><td>0.822</td><td>0.862</td><td>0.0</td><td>3.70</td><td>7.41</td><td>7.41</td><td>0.136</td></tr><tr><td>TabTransformer</td><td>19.2</td><td>0.816</td><td>0.796</td><td>0.0</td><td>3.70</td><td>7.41</td><td>18.5</td><td>0.160</td></tr><tr><td>RandomForest</td><td>19.6</td><td>0.806</td><td>0.772</td><td>0.0</td><td>3.70</td><td>7.41</td><td>18.5</td><td>0.144</td></tr><tr><td>Node</td><td>20.1</td><td>0.793</td><td>0.724</td><td>3.70</td><td>3.70</td><td>3.70</td><td>7.41</td><td>0.151</td></tr><tr><td>LogReg</td><td>20.5</td><td>0.807</td><td>0.778</td><td>0.0</td><td>3.70</td><td>18.5</td><td>25.9</td><td>0.159</td></tr><tr><td>Tangos</td><td>20.7</td><td>0.811</td><td>0.798</td><td>0.0</td><td>0.0</td><td>3.70</td><td>3.70</td><td>0.139</td></tr><tr><td>SVM</td><td>21.8</td><td>0.802</td><td>0.754</td><td>0.0</td><td>0.0</td><td>3.70</td><td>18.5</td><td>0.162</td></tr><tr><td>GrowNet</td><td>22.7</td><td>0.804</td><td>0.771</td><td>0.0</td><td>0.0</td><td>0.0</td><td>3.70</td><td>0.160</td></tr><tr><td>TabNet</td><td>23.2</td><td>0.814</td><td>0.818</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.142</td></tr><tr><td>kNN</td><td>24.3</td><td>0.787</td><td>0.673</td><td>0.0</td><td>0.0</td><td>0.0</td><td>7.41</td><td>0.166</td></tr><tr><td>SwitchTab</td><td>27.3</td><td>0.786</td><td>0.670</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.163</td></tr><tr><td>NaiveBayes</td><td>30.2</td><td>0.678</td><td>0.154</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.283</td></tr><tr><td>NCM</td><td>31.3</td><td>0.679</td><td>0.174</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.281</td></tr></tbody></table>
<table><caption>**Table C.5:** Full TALENT Binary classification (≤ 10,000 samples) results. Rank is the average rank among the ordered methods over all datasets. (Normalized) Score is the metric we use to compare methods in Figure 2. In this case score is the mean classification accuracy (1 minus the error in Figure 2). Top-X (%) is the percentage of datasets for which that method is in the top X ranks. The final column is the shifted geometric mean error (SGM<sub>e</sub>).</caption><thead><tr><th>Method</th><th>Rank</th><th>Score</th><th>Norm. Score</th><th>Top-1 (%)</th><th>Top-3 (%)</th><th>Top-5 (%)</th><th>Top-8 (%)</th><th>SGM<sub>e</sub></th></tr></thead><tbody><tr><td>PFN-v2</td><td><strong>5.02</strong></td><td><strong>0.864</strong></td><td><strong>0.947</strong></td><td><strong>26.9</strong></td><td><strong>59.1</strong></td><td><strong>72.0</strong></td><td><strong>80.6</strong></td><td><strong>0.104</strong></td></tr><tr><td>xRFM</td><td>8.75</td><td>0.856</td><td>0.893</td><td>7.53</td><td>29.0</td><td>49.5</td><td>63.4</td><td>0.111</td></tr><tr><td>ModernNCA</td><td>10.3</td><td>0.856</td><td>0.886</td><td>11.8</td><td>22.6</td><td>31.2</td><td>50.5</td><td>0.111</td></tr><tr><td>CatBoost</td><td>10.3</td><td>0.845</td><td>0.867</td><td>4.30</td><td>23.7</td><td>34.4</td><td>54.8</td><td>0.119</td></tr><tr><td>LightGBM</td><td>10.4</td><td>0.845</td><td>0.869</td><td>7.53</td><td>18.3</td><td>35.5</td><td>54.8</td><td>0.119</td></tr><tr><td>TabR</td><td>10.5</td><td>0.851</td><td>0.873</td><td>4.30</td><td>17.2</td><td>30.1</td><td>43.0</td><td>0.116</td></tr><tr><td>XGBoost</td><td>11.4</td><td>0.844</td><td>0.851</td><td>5.38</td><td>18.3</td><td>35.5</td><td>49.5</td><td>0.120</td></tr><tr><td>RealMLP</td><td>11.7</td><td>0.850</td><td>0.864</td><td>1.08</td><td>7.53</td><td>18.3</td><td>35.5</td><td>0.116</td></tr><tr><td>FTT</td><td>13.3</td><td>0.836</td><td>0.813</td><td>1.08</td><td>6.45</td><td>17.2</td><td>32.3</td><td>0.124</td></tr><tr><td>MLP-PLR</td><td>14.2</td><td>0.838</td><td>0.817</td><td>0.0</td><td>3.23</td><td>11.8</td><td>29.0</td><td>0.124</td></tr><tr><td>RandomForest</td><td>14.3</td><td>0.836</td><td>0.824</td><td>3.23</td><td>10.8</td><td>19.4</td><td>33.3</td><td>0.129</td></tr><tr><td>AutoInt</td><td>15.2</td><td>0.834</td><td>0.803</td><td>0.0</td><td>3.23</td><td>5.38</td><td>16.1</td><td>0.129</td></tr><tr><td>Tangos</td><td>15.4</td><td>0.834</td><td>0.800</td><td>0.0</td><td>3.23</td><td>7.53</td><td>16.1</td><td>0.131</td></tr><tr><td>DCN2</td><td>15.5</td><td>0.839</td><td>0.816</td><td>1.08</td><td>2.15</td><td>9.68</td><td>21.5</td><td>0.127</td></tr><tr><td>Laplace KRR</td><td>15.7</td><td>0.836</td><td>0.801</td><td>1.08</td><td>16.1</td><td>20.4</td><td>26.9</td><td>0.133</td></tr><tr><td>MLP</td><td>16.0</td><td>0.836</td><td>0.797</td><td>1.08</td><td>3.23</td><td>8.60</td><td>17.2</td><td>0.132</td></tr><tr><td>ResNet</td><td>16.1</td><td>0.839</td><td>0.811</td><td>1.08</td><td>3.23</td><td>8.60</td><td>19.4</td><td>0.129</td></tr><tr><td>TabCaps</td><td>16.2</td><td>0.835</td><td>0.804</td><td>1.08</td><td>3.23</td><td>5.38</td><td>14.0</td><td>0.130</td></tr><tr><td>Excelformer</td><td>16.6</td><td>0.830</td><td>0.778</td><td>0.0</td><td>1.08</td><td>7.53</td><td>16.1</td><td>0.131</td></tr><tr><td>SNN</td><td>16.6</td><td>0.836</td><td>0.800</td><td>1.08</td><td>3.23</td><td>4.30</td><td>7.53</td><td>0.129</td></tr><tr><td>Node</td><td>17.0</td><td>0.824</td><td>0.757</td><td>0.0</td><td>5.38</td><td>15.1</td><td>25.8</td><td>0.137</td></tr><tr><td>PTARL</td><td>17.0</td><td>0.830</td><td>0.780</td><td>0.0</td><td>1.08</td><td>3.23</td><td>8.60</td><td>0.134</td></tr><tr><td>Danets</td><td>18.0</td><td>0.829</td><td>0.759</td><td>0.0</td><td>3.23</td><td>6.45</td><td>14.0</td><td>0.137</td></tr><tr><td>TabTransformer</td><td>19.7</td><td>0.820</td><td>0.715</td><td>1.08</td><td>1.08</td><td>6.45</td><td>12.9</td><td>0.148</td></tr><tr><td>LogReg</td><td>20.1</td><td>0.819</td><td>0.723</td><td>3.23</td><td>9.68</td><td>12.9</td><td>17.2</td><td>0.146</td></tr><tr><td>kNN</td><td>20.8</td><td>0.811</td><td>0.677</td><td>5.38</td><td>10.8</td><td>14.0</td><td>18.3</td><td>0.158</td></tr><tr><td>SVM</td><td>21.6</td><td>0.816</td><td>0.698</td><td>2.15</td><td>3.23</td><td>6.45</td><td>10.8</td><td>0.150</td></tr><tr><td>GrowNet</td><td>22.2</td><td>0.817</td><td>0.714</td><td>0.0</td><td>0.0</td><td>0.0</td><td>2.15</td><td>0.151</td></tr><tr><td>SwitchTab</td><td>24.0</td><td>0.809</td><td>0.675</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.156</td></tr><tr><td>TabNet</td><td>25.8</td><td>0.802</td><td>0.633</td><td>0.0</td><td>0.0</td><td>0.0</td><td>1.08</td><td>0.157</td></tr><tr><td>NaiveBayes</td><td>28.6</td><td>0.695</td><td>0.281</td><td>1.08</td><td>2.15</td><td>3.23</td><td>4.30</td><td>0.253</td></tr><tr><td>NCM</td><td>29.9</td><td>0.723</td><td>0.230</td><td>0.0</td><td>0.0</td><td>0.0</td><td>2.15</td><td>0.257</td></tr></tbody></table>

<table><caption>**Table C.6:** Meta-test regression datasets with more than 70,000 samples. Error reported is nRMSE averaged over five train/test splits using pytabkit.</caption><thead><tr><th>Dataset</th><th>xRFM</th><th>XGB</th><th>CatBoost</th><th>LGBM</th><th>RealMLP</th><th>MLP-PLR</th><th>MLP-RTDL</th><th>ResNet-RTDL</th></tr></thead><tbody><tr><td>Airlines_DepDelay_10M</td><td>0.9818</td><td>0.9813</td><td>0.9796</td><td>0.9798</td><td><strong>0.9786</strong></td><td>0.9795</td><td>0.9824</td><td>0.9818</td></tr><tr><td>Allstate_Claims_Severity</td><td><strong>0.6489</strong></td><td>0.6547</td><td>0.6510</td><td>0.6530</td><td>0.6495</td><td>0.6537</td><td>0.6557</td><td>0.6537</td></tr><tr><td>black_friday</td><td>0.6881</td><td>0.6807</td><td>0.6792</td><td><strong>0.6787</strong></td><td>0.6859</td><td>0.6862</td><td>0.6929</td><td>0.6892</td></tr><tr><td>Buzzinsocialmedia_Twitter</td><td><strong>0.2080</strong></td><td>0.2134</td><td>0.3147</td><td>0.2789</td><td>0.2566</td><td>0.2553</td><td>0.2840</td><td>0.2906</td></tr><tr><td>nyc-taxi-green-dec-2016</td><td><strong>0.5834</strong></td><td>0.6649</td><td>0.6567</td><td>0.6489</td><td>0.6142</td><td>0.6523</td><td>0.6657</td><td>0.6365</td></tr><tr><td>wave_energy</td><td><strong>0.0020</strong></td><td>0.0918</td><td>0.0499</td><td>0.0821</td><td>0.0024</td><td>0.0073</td><td>0.0254</td><td>0.0434</td></tr><tr><td>Yolanda</td><td><strong>0.7816</strong></td><td>0.8012</td><td>0.8094</td><td>0.7970</td><td>0.7869</td><td>0.7897</td><td>0.7927</td><td>0.7856</td></tr></tbody></table>
<table><caption>**Table C.7:** Meta-test classification datasets with greater than 70,000 samples. Error reported is classification error averaged over five train/test splits using `pytabkit`.</caption><thead><tr><th>Dataset</th><th>xRFM</th><th>XGB</th><th>CatBoost</th><th>LGBM</th><th>RealMLP</th><th>MLP-PLR</th><th>MLP-RTDL</th><th>ResNet-RTDL</th></tr></thead><tbody><tr><td>airlines</td><td>0.3341</td><td>0.3292</td><td>0.3315</td><td><strong>0.3287</strong></td><td>0.3344</td><td>0.3342</td><td>0.3342</td><td>0.3339</td></tr><tr><td>covertype</td><td><strong>0.0257</strong></td><td>0.0420</td><td>0.0612</td><td>0.0333</td><td>0.0280</td><td>0.0364</td><td>0.0404</td><td>0.0385</td></tr><tr><td>Higgs</td><td>0.2640</td><td>0.2576</td><td>0.2576</td><td>0.2549</td><td>0.2473</td><td>0.2528</td><td>0.2515</td><td><strong>0.2435</strong></td></tr><tr><td>jannis</td><td><strong>0.2701</strong></td><td>0.2799</td><td>0.2816</td><td>0.2779</td><td>0.2702</td><td>0.2764</td><td>0.2865</td><td>0.2795</td></tr><tr><td>MiniBooNE</td><td>0.0539</td><td>0.0529</td><td>0.0538</td><td>0.0525</td><td><strong>0.0484</strong></td><td>0.0504</td><td>0.0503</td><td>0.0488</td></tr><tr><td>numerai28.6</td><td>0.4778</td><td>0.4812</td><td>0.4790</td><td>0.4782</td><td>0.4800</td><td>0.4775</td><td><strong>0.4771</strong></td><td>0.4800</td></tr><tr><td>porto-seguro</td><td><strong>0.0380</strong></td><td><strong>0.0380</strong></td><td>0.0381</td><td>0.0381</td><td><strong>0.0380</strong></td><td><strong>0.0380</strong></td><td><strong>0.0380</strong></td><td><strong>0.0380</strong></td></tr><tr><td>dionis</td><td>0.0926</td><td>0.1219</td><td>0.1041</td><td>0.1076</td><td><strong>0.0887</strong></td><td>0.1257</td><td>0.1110</td><td>0.0907</td></tr><tr><td>Fashion-MNIST</td><td>0.0889</td><td>0.0928</td><td>0.0969</td><td><strong>0.0895</strong></td><td>0.0913</td><td>0.1064</td><td>0.1041</td><td>0.1011</td></tr><tr><td>kick</td><td>0.0972</td><td>0.0965</td><td><strong>0.0956</strong></td><td>0.0964</td><td>0.0976</td><td>0.0978</td><td>0.0979</td><td>0.0970</td></tr></tbody></table>
<table><caption>Table C.8: Full regression results on TabArena-Lite.</caption><thead><tr><th>Model</th><th>Elo (↑)</th><th>Norm. score (↑)</th><th>Avg. rank (↓)</th><th>Harm. mean rank (↓)</th><th>#wins (↑)</th><th>Improva- bility (↓)</th><th>Train time per 1K [s]</th><th>Predict time per 1K [s]</th></tr></thead><tbody><tr><td>AutoGluon 1.3 (4h)</td><td>1779</td><td>0.673</td><td>5.2</td><td>3.2</td><td>1</td><td>3.6%</td><td>1734.20</td><td>7.06</td></tr><tr><td>RealMLP (T+E)</td><td>1721</td><td>0.660</td><td>6.6</td><td>5.4</td><td>0</td><td>3.1%</td><td>6860.54</td><td>7.68</td></tr><tr><td>ModernNCA (T+E)</td><td>1626</td><td>0.622</td><td>9.2</td><td>2.6</td><td>3</td><td>5.4%</td><td>3811.43</td><td>7.58</td></tr><tr><td>LightGBM (T+E)</td><td>1602</td><td>0.478</td><td>10.1</td><td>7.5</td><td>0</td><td>6.0%</td><td>686.46</td><td>5.48</td></tr><tr><td>TabDPT (D)</td><td>1575</td><td>0.515</td><td>10.9</td><td>3.8</td><td>2</td><td>3.9%</td><td>16.97</td><td>8.70</td></tr><tr><td>xRFM (T+E)</td><td>1563</td><td>0.492</td><td>11.5</td><td>5.3</td><td>1</td><td>5.1%</td><td>365.57</td><td>0.72</td></tr><tr><td>CatBoost (T+E)</td><td>1558</td><td>0.435</td><td>11.8</td><td>8.9</td><td>0</td><td>5.7%</td><td>2895.38</td><td>1.32</td></tr><tr><td>TabM (T+E)</td><td>1529</td><td>0.439</td><td>12.8</td><td>8.5</td><td>0</td><td>4.3%</td><td>4228.53</td><td>1.19</td></tr><tr><td>CatBoost (T)</td><td>1515</td><td>0.406</td><td>13.3</td><td>6.8</td><td>0</td><td>5.9%</td><td>2895.38</td><td>0.07</td></tr><tr><td>xRFM (T)</td><td>1504</td><td>0.406</td><td>13.8</td><td>9.7</td><td>0</td><td>5.7%</td><td>365.57</td><td>0.09</td></tr><tr><td>LightGBM (T)</td><td>1498</td><td>0.354</td><td>13.9</td><td>10.2</td><td>0</td><td>6.8%</td><td>686.46</td><td>0.74</td></tr><tr><td>XGBoost (T+E)</td><td>1470</td><td>0.328</td><td>15.2</td><td>13.3</td><td>0</td><td>6.7%</td><td>848.99</td><td>2.38</td></tr><tr><td>XGBoost (T)</td><td>1470</td><td>0.325</td><td>15.2</td><td>13.3</td><td>0</td><td>6.7%</td><td>848.99</td><td>0.47</td></tr><tr><td>ModernNCA (D)</td><td>1457</td><td>0.323</td><td>15.6</td><td>11.0</td><td>0</td><td>8.1%</td><td>16.07</td><td>0.29</td></tr><tr><td>TabM (T)</td><td>1453</td><td>0.311</td><td>16.0</td><td>12.6</td><td>0</td><td>5.2%</td><td>4228.53</td><td>0.13</td></tr><tr><td>RealMLP (T)</td><td>1439</td><td>0.306</td><td>16.6</td><td>13.3</td><td>0</td><td>5.9%</td><td>6860.54</td><td>0.32</td></tr><tr><td>CatBoost (D)</td><td>1419</td><td>0.278</td><td>17.4</td><td>12.0</td><td>0</td><td>7.9%</td><td>8.35</td><td>0.09</td></tr><tr><td>ModernNCA (T)</td><td>1411</td><td>0.347</td><td>17.6</td><td>6.7</td><td>0</td><td>7.9%</td><td>3811.43</td><td>0.45</td></tr><tr><td>TabPFNv2 (T+E)</td><td>1377</td><td>0.410</td><td>19.1</td><td>2.7</td><td>4</td><td>4.9%</td><td>3805.62</td><td>10.41</td></tr><tr><td>TabM (D)</td><td>1361</td><td>0.232</td><td>20.1</td><td>16.5</td><td>0</td><td>6.9%</td><td>13.90</td><td>0.12</td></tr><tr><td>TabPFNv2 (T)</td><td>1314</td><td>0.281</td><td>22.1</td><td>6.9</td><td>0</td><td>6.4%</td><td>3805.62</td><td>0.26</td></tr><tr><td>TorchMLP (T+E)</td><td>1301</td><td>0.141</td><td>22.7</td><td>19.1</td><td>0</td><td>8.1%</td><td>4452.11</td><td>0.85</td></tr><tr><td>ExtraTrees (T+E)</td><td>1297</td><td>0.169</td><td>23.1</td><td>16.9</td><td>0</td><td>11.1%</td><td>161.73</td><td>0.78</td></tr><tr><td>RealMLP (D)</td><td>1274</td><td>0.085</td><td>24.0</td><td>21.6</td><td>0</td><td>8.2%</td><td>23.30</td><td>1.44</td></tr><tr><td>ExtraTrees (T)</td><td>1273</td><td>0.174</td><td>24.0</td><td>16.7</td><td>0</td><td>11.4%</td><td>161.73</td><td>0.12</td></tr><tr><td>TabPFNv2 (D)</td><td>1264</td><td>0.262</td><td>24.3</td><td>7.7</td><td>0</td><td>7.6%</td><td>2.78</td><td>0.32</td></tr><tr><td>TorchMLP (T)</td><td>1245</td><td>0.124</td><td>25.2</td><td>20.8</td><td>0</td><td>8.7%</td><td>4452.11</td><td>0.09</td></tr><tr><td>xRFM (D)</td><td>1236</td><td>0.149</td><td>25.8</td><td>8.6</td><td>1</td><td>11.7%</td><td>1.65</td><td>0.08</td></tr><tr><td>LightGBM (D)</td><td>1231</td><td>0.069</td><td>25.9</td><td>24.5</td><td>0</td><td>9.7%</td><td>2.03</td><td>0.30</td></tr><tr><td>XGBoost (D)</td><td>1195</td><td>0.104</td><td>27.4</td><td>24.5</td><td>0</td><td>10.4%</td><td>2.15</td><td>0.18</td></tr><tr><td>RandomForest (T+E)</td><td>1193</td><td>0.056</td><td>27.5</td><td>25.5</td><td>0</td><td>12.1%</td><td>526.17</td><td>0.77</td></tr><tr><td>RandomForest (T)</td><td>1142</td><td>0.046</td><td>29.9</td><td>27.5</td><td>0</td><td>12.8%</td><td>526.17</td><td>0.12</td></tr><tr><td>EBM (T+E)</td><td>1133</td><td>0.122</td><td>30.0</td><td>16.4</td><td>0</td><td>14.7%</td><td>2124.78</td><td>0.12</td></tr><tr><td>ExtraTrees (D)</td><td>1117</td><td>0.056</td><td>30.6</td><td>27.7</td><td>0</td><td>13.0%</td><td>0.42</td><td>0.06</td></tr><tr><td>FastaiMLP (T+E)</td><td>1086</td><td>0.005</td><td>31.8</td><td>30.6</td><td>0</td><td>13.1%</td><td>527.21</td><td>2.83</td></tr><tr><td>EBM (T)</td><td>1076</td><td>0.131</td><td>32.2</td><td>9.4</td><td>1</td><td>15.3%</td><td>2124.78</td><td>0.01</td></tr><tr><td>TorchMLP (D)</td><td>1075</td><td>0.015</td><td>32.4</td><td>30.3</td><td>0</td><td>12.9%</td><td>20.50</td><td>0.08</td></tr><tr><td>FastaiMLP (T)</td><td>1046</td><td>0.000</td><td>33.3</td><td>32.3</td><td>0</td><td>13.6%</td><td>527.21</td><td>0.31</td></tr><tr><td>EBM (D)</td><td>1009</td><td>0.071</td><td>34.7</td><td>30.3</td><td>0</td><td>16.0%</td><td>7.25</td><td>0.04</td></tr><tr><td>TabICL (D)</td><td>1006</td><td>0.000</td><td>34.8</td><td>34.2</td><td>0</td><td>14.2%</td><td>0.63</td><td>0.06</td></tr><tr><td>RandomForest (D)</td><td>1000</td><td>0.000</td><td>34.8</td><td>34.2</td><td>0</td><td>14.2%</td><td>0.63</td><td>0.06</td></tr><tr><td>FastaiMLP (D)</td><td>916</td><td>0.000</td><td>37.2</td><td>36.5</td><td>0</td><td>17.8%</td><td>3.08</td><td>0.29</td></tr><tr><td>KNN (T+E)</td><td>533</td><td>0.000</td><td>43.8</td><td>43.6</td><td>0</td><td>37.3%</td><td>2.25</td><td>0.15</td></tr><tr><td>Linear (T+E)</td><td>485</td><td>0.000</td><td>44.3</td><td>44.4</td><td>0</td><td>35.5%</td><td>46.50</td><td>0.14</td></tr><tr><td>KNN (T)</td><td>434</td><td>0.000</td><td>44.9</td><td>44.7</td><td>0</td><td>38.0%</td><td>2.25</td><td>0.03</td></tr><tr><td>Linear (T)</td><td>426</td><td>0.000</td><td>44.9</td><td>44.8</td><td>0</td><td>35.7%</td><td>46.50</td><td>0.04</td></tr><tr><td>Linear (D)</td><td>312</td><td>0.000</td><td>46.0</td><td>46.0</td><td>0</td><td>38.1%</td><td>1.16</td><td>0.08</td></tr><tr><td>KNN (D)</td><td>263</td><td>0.000</td><td>46.5</td><td>46.2</td><td>0</td><td>41.6%</td><td>0.04</td><td>0.02</td></tr></tbody></table>
<table><caption>Table C.9: Full multiclass results on TabArena-Lite.</caption><thead><tr><th>Model</th><th>Elo (↑)</th><th>Norm. score (↑)</th><th>Avg. rank (↓)</th><th>Harm. mean rank (↓)</th><th>#wins (↑)</th><th>Improva- bility (↓)</th><th>Train time per 1K [s]</th><th>Predict time per 1K [s]</th></tr></thead><tbody><tr><td>AutoGluon 1.3 (4h)</td><td>1521</td><td>0.586</td><td>9.0</td><td>3.0</td><td>2</td><td>9.7%</td><td>4917.81</td><td>4.05</td></tr><tr><td>CatBoost (T)</td><td>1441</td><td>0.439</td><td>12.5</td><td>9.6</td><td>0</td><td>11.9%</td><td>3307.58</td><td>0.14</td></tr><tr><td>CatBoost (T+E)</td><td>1441</td><td>0.446</td><td>12.4</td><td>9.4</td><td>0</td><td>11.1%</td><td>3307.58</td><td>1.18</td></tr><tr><td>TabPFNv2 (T+E)</td><td>1419</td><td>0.518</td><td>13.5</td><td>4.2</td><td>1</td><td>13.8%</td><td>2584.13</td><td>12.37</td></tr><tr><td>LightGBM (T+E)</td><td>1418</td><td>0.393</td><td>13.2</td><td>7.5</td><td>0</td><td>13.6%</td><td>1280.01</td><td>4.08</td></tr><tr><td>LightGBM (T)</td><td>1405</td><td>0.381</td><td>13.9</td><td>6.9</td><td>0</td><td>13.6%</td><td>1280.01</td><td>1.05</td></tr><tr><td>TabM (T+E)</td><td>1387</td><td>0.405</td><td>14.8</td><td>9.7</td><td>0</td><td>15.7%</td><td>5568.31</td><td>1.78</td></tr><tr><td>XGBoost (T+E)</td><td>1377</td><td>0.310</td><td>15.2</td><td>13.3</td><td>0</td><td>14.0%</td><td>2029.77</td><td>4.11</td></tr><tr><td>TabM (T)</td><td>1373</td><td>0.401</td><td>15.5</td><td>9.1</td><td>0</td><td>16.0%</td><td>5568.31</td><td>0.37</td></tr><tr><td>XGBoost (T)</td><td>1373</td><td>0.317</td><td>15.6</td><td>13.1</td><td>0</td><td>13.7%</td><td>2029.77</td><td>1.04</td></tr><tr><td>RealMLP (T+E)</td><td>1359</td><td>0.337</td><td>16.2</td><td>5.3</td><td>1</td><td>16.8%</td><td>6866.35</td><td>10.40</td></tr><tr><td>TabPFNv2 (D)</td><td>1358</td><td>0.361</td><td>16.4</td><td>4.7</td><td>1</td><td>12.5%</td><td>5.48</td><td>0.35</td></tr><tr><td>xRFM (T+E)</td><td>1357</td><td>0.369</td><td>16.2</td><td>7.0</td><td>0</td><td>14.8%</td><td>515.01</td><td>1.68</td></tr><tr><td>TabPFNv2 (T)</td><td>1346</td><td>0.366</td><td>17.0</td><td>6.5</td><td>0</td><td>14.9%</td><td>2584.13</td><td>0.41</td></tr><tr><td>ModernNCA (T+E)</td><td>1345</td><td>0.302</td><td>16.9</td><td>10.4</td><td>0</td><td>15.3%</td><td>6684.65</td><td>9.59</td></tr><tr><td>ModernNCA (T)</td><td>1332</td><td>0.248</td><td>17.8</td><td>12.4</td><td>0</td><td>15.6%</td><td>6684.65</td><td>0.75</td></tr><tr><td>RealMLP (T)</td><td>1304</td><td>0.253</td><td>19.1</td><td>15.5</td><td>0</td><td>18.4%</td><td>6866.35</td><td>0.92</td></tr><tr><td>CatBoost (D)</td><td>1296</td><td>0.201</td><td>19.2</td><td>16.9</td><td>0</td><td>16.1%</td><td>43.10</td><td>0.25</td></tr><tr><td>TabCL (D)</td><td>1293</td><td>0.325</td><td>19.9</td><td>4.5</td><td>1</td><td>20.5%</td><td>11.51</td><td>1.95</td></tr><tr><td>TabM (D)</td><td>1293</td><td>0.284</td><td>19.9</td><td>9.4</td><td>0</td><td>19.7%</td><td>17.09</td><td>0.15</td></tr><tr><td>ExtraTrees (T+E)</td><td>1286</td><td>0.289</td><td>20.2</td><td>11.0</td><td>0</td><td>17.7%</td><td>728.32</td><td>2.44</td></tr><tr><td>xRFM (T)</td><td>1283</td><td>0.273</td><td>20.4</td><td>5.2</td><td>1</td><td>16.4%</td><td>515.01</td><td>0.20</td></tr><tr><td>ExtraTrees (T)</td><td>1265</td><td>0.262</td><td>21.2</td><td>11.2</td><td>0</td><td>17.3%</td><td>728.32</td><td>0.36</td></tr><tr><td>RandomForest (T+E)</td><td>1260</td><td>0.321</td><td>20.9</td><td>6.1</td><td>0</td><td>15.9%</td><td>729.17</td><td>1.83</td></tr><tr><td>TorchMLP (T+E)</td><td>1260</td><td>0.188</td><td>21.2</td><td>17.5</td><td>0</td><td>19.6%</td><td>3646.83</td><td>2.16</td></tr><tr><td>TorchMLP (T)</td><td>1218</td><td>0.156</td><td>23.9</td><td>17.8</td><td>0</td><td>21.6%</td><td>3646.83</td><td>0.19</td></tr><tr><td>RandomForest (T)</td><td>1190</td><td>0.241</td><td>24.9</td><td>12.1</td><td>0</td><td>17.6%</td><td>729.17</td><td>0.33</td></tr><tr><td>XGBoost (D)</td><td>1188</td><td>0.102</td><td>24.9</td><td>21.5</td><td>0</td><td>18.1%</td><td>4.93</td><td>0.59</td></tr><tr><td>TabDPT (D)</td><td>1186</td><td>0.259</td><td>25.2</td><td>4.7</td><td>1</td><td>23.3%</td><td>33.52</td><td>20.75</td></tr><tr><td>FastaiMLP (T+E)</td><td>1182</td><td>0.205</td><td>25.2</td><td>12.1</td><td>0</td><td>24.2%</td><td>2721.87</td><td>12.59</td></tr><tr><td>LightGBM (D)</td><td>1180</td><td>0.158</td><td>25.4</td><td>19.6</td><td>0</td><td>21.3%</td><td>5.12</td><td>0.44</td></tr><tr><td>EBM (T+E)</td><td>1168</td><td>0.166</td><td>26.0</td><td>17.1</td><td>0</td><td>24.6%</td><td>1471.12</td><td>0.27</td></tr><tr><td>EBM (T)</td><td>1144</td><td>0.160</td><td>27.4</td><td>19.0</td><td>0</td><td>24.9%</td><td>1471.12</td><td>0.03</td></tr><tr><td>xRFM (D)</td><td>1129</td><td>0.094</td><td>28.4</td><td>22.1</td><td>0</td><td>23.9%</td><td>2.22</td><td>0.20</td></tr><tr><td>ModernNCA (D)</td><td>1110</td><td>0.051</td><td>29.1</td><td>26.8</td><td>0</td><td>25.9%</td><td>17.24</td><td>0.57</td></tr><tr><td>FastaiMLP (T)</td><td>1100</td><td>0.104</td><td>29.6</td><td>20.3</td><td>0</td><td>25.8%</td><td>2721.87</td><td>1.08</td></tr><tr><td>RealMLP (D)</td><td>1049</td><td>0.071</td><td>32.0</td><td>24.5</td><td>0</td><td>26.3%</td><td>26.02</td><td>4.18</td></tr><tr><td>EBM (D)</td><td>1046</td><td>0.095</td><td>32.0</td><td>24.3</td><td>0</td><td>27.7%</td><td>6.16</td><td>0.08</td></tr><tr><td>RandomForest (D)</td><td>1000</td><td>0.000</td><td>33.9</td><td>32.0</td><td>0</td><td>33.3%</td><td>0.74</td><td>0.15</td></tr><tr><td>TorchMLP (D)</td><td>972</td><td>0.004</td><td>35.0</td><td>33.6</td><td>0</td><td>28.5%</td><td>14.37</td><td>0.36</td></tr><tr><td>FastaiMLP (D)</td><td>952</td><td>0.038</td><td>35.9</td><td>31.9</td><td>0</td><td>33.4%</td><td>8.37</td><td>0.66</td></tr><tr><td>ExtraTrees (D)</td><td>862</td><td>0.013</td><td>38.6</td><td>35.1</td><td>0</td><td>39.1%</td><td>0.76</td><td>0.15</td></tr><tr><td>Linear (T+E)</td><td>752</td><td>0.000</td><td>41.8</td><td>41.5</td><td>0</td><td>45.6%</td><td>170.51</td><td>0.20</td></tr><tr><td>Linear (T)</td><td>718</td><td>0.000</td><td>42.3</td><td>41.9</td><td>0</td><td>45.8%</td><td>170.51</td><td>0.13</td></tr><tr><td>KNN (T+E)</td><td>714</td><td>0.000</td><td>42.4</td><td>41.0</td><td>0</td><td>52.5%</td><td>2.99</td><td>0.17</td></tr><tr><td>Linear (D)</td><td>689</td><td>0.000</td><td>42.9</td><td>42.7</td><td>0</td><td>46.9%</td><td>3.89</td><td>0.16</td></tr><tr><td>KNN (D)</td><td>544</td><td>0.000</td><td>45.4</td><td>45.1</td><td>0</td><td>69.5%</td><td>0.33</td><td>0.05</td></tr><tr><td>KNN (T)</td><td>541</td><td>0.000</td><td>45.6</td><td>45.5</td><td>0</td><td>58.3%</td><td>2.99</td><td>0.06</td></tr></tbody></table>
<table><caption>Table C.10: Full binary classification results on TabArena.</caption><thead><tr><th>Model</th><th>Elo (↑)</th><th>Norm. score (↑)</th><th>Avg. rank (↓)</th><th>Harm. mean rank (↓)</th><th>#wins (↑)</th><th>Improva- bility (↓)</th><th>Train time per 1K [s]</th><th>Predict time per 1K [s]</th></tr></thead><tbody><tr><td>AutoGluon 1.3 (4h)</td><td>1779</td><td>0.673</td><td>5.2</td><td>3.2</td><td>1</td><td>3.6%</td><td>1734.20</td><td>7.06</td></tr><tr><td>RealMLP (T+E)</td><td>1721</td><td>0.660</td><td>6.6</td><td>5.4</td><td>0</td><td>3.1%</td><td>6860.54</td><td>7.68</td></tr><tr><td>ModernNCA (T+E)</td><td>1626</td><td>0.622</td><td>9.2</td><td>2.6</td><td>3</td><td>5.4%</td><td>3811.43</td><td>7.58</td></tr><tr><td>LightGBM (T+E)</td><td>1602</td><td>0.478</td><td>10.1</td><td>7.5</td><td>0</td><td>6.0%</td><td>686.46</td><td>5.48</td></tr><tr><td>TabDPT (D)</td><td>1575</td><td>0.515</td><td>10.9</td><td>3.8</td><td>2</td><td>3.9%</td><td>16.97</td><td>8.70</td></tr><tr><td>xRFM (T+E)</td><td>1563</td><td>0.492</td><td>11.5</td><td>5.3</td><td>1</td><td>5.1%</td><td>365.57</td><td>0.72</td></tr><tr><td>CatBoost (T+E)</td><td>1558</td><td>0.435</td><td>11.8</td><td>8.9</td><td>0</td><td>5.7%</td><td>2895.38</td><td>1.32</td></tr><tr><td>TabM (T+E)</td><td>1529</td><td>0.439</td><td>12.8</td><td>8.5</td><td>0</td><td>4.3%</td><td>4228.53</td><td>1.19</td></tr><tr><td>CatBoost (T)</td><td>1515</td><td>0.406</td><td>13.3</td><td>6.8</td><td>0</td><td>5.9%</td><td>2895.38</td><td>0.07</td></tr><tr><td>xRFM (T)</td><td>1504</td><td>0.406</td><td>13.8</td><td>9.7</td><td>0</td><td>5.7%</td><td>365.57</td><td>0.09</td></tr><tr><td>LightGBM (T)</td><td>1498</td><td>0.354</td><td>13.9</td><td>10.2</td><td>0</td><td>6.8%</td><td>686.46</td><td>0.74</td></tr><tr><td>XGBoost (T+E)</td><td>1470</td><td>0.328</td><td>15.2</td><td>13.3</td><td>0</td><td>6.7%</td><td>848.99</td><td>2.38</td></tr><tr><td>XGBoost (T)</td><td>1470</td><td>0.325</td><td>15.2</td><td>13.3</td><td>0</td><td>6.7%</td><td>848.99</td><td>0.47</td></tr><tr><td>ModernNCA (D)</td><td>1457</td><td>0.323</td><td>15.6</td><td>11.0</td><td>0</td><td>8.1%</td><td>16.07</td><td>0.29</td></tr><tr><td>TabM (T)</td><td>1453</td><td>0.311</td><td>16.0</td><td>12.6</td><td>0</td><td>5.2%</td><td>4228.53</td><td>0.13</td></tr><tr><td>RealMLP (T)</td><td>1439</td><td>0.306</td><td>16.6</td><td>13.3</td><td>0</td><td>5.9%</td><td>6860.54</td><td>0.32</td></tr><tr><td>CatBoost (D)</td><td>1419</td><td>0.278</td><td>17.4</td><td>12.0</td><td>0</td><td>7.9%</td><td>8.35</td><td>0.09</td></tr><tr><td>ModernNCA (T)</td><td>1411</td><td>0.347</td><td>17.6</td><td>6.7</td><td>0</td><td>7.9%</td><td>3811.43</td><td>0.45</td></tr><tr><td>TabPFNv2 (T+E)</td><td>1377</td><td>0.410</td><td>19.1</td><td>2.7</td><td>4</td><td>4.9%</td><td>3805.62</td><td>10.41</td></tr><tr><td>TabM (D)</td><td>1361</td><td>0.232</td><td>20.1</td><td>16.5</td><td>0</td><td>6.9%</td><td>13.90</td><td>0.12</td></tr><tr><td>TabPFNv2 (T)</td><td>1314</td><td>0.281</td><td>22.1</td><td>6.9</td><td>0</td><td>6.4%</td><td>3805.62</td><td>0.26</td></tr><tr><td>TorchMLP (T+E)</td><td>1301</td><td>0.141</td><td>22.7</td><td>19.1</td><td>0</td><td>8.1%</td><td>4452.11</td><td>0.85</td></tr><tr><td>ExtraTrees (T+E)</td><td>1297</td><td>0.169</td><td>23.1</td><td>16.9</td><td>0</td><td>11.1%</td><td>161.73</td><td>0.78</td></tr><tr><td>RealMLP (D)</td><td>1274</td><td>0.085</td><td>24.0</td><td>21.6</td><td>0</td><td>8.2%</td><td>23.30</td><td>1.44</td></tr><tr><td>ExtraTrees (T)</td><td>1273</td><td>0.174</td><td>24.0</td><td>16.7</td><td>0</td><td>11.4%</td><td>161.73</td><td>0.12</td></tr><tr><td>TabPFNv2 (D)</td><td>1264</td><td>0.262</td><td>24.3</td><td>7.7</td><td>0</td><td>7.6%</td><td>2.78</td><td>0.32</td></tr><tr><td>TorchMLP (T)</td><td>1245</td><td>0.124</td><td>25.2</td><td>20.8</td><td>0</td><td>8.7%</td><td>4452.11</td><td>0.09</td></tr><tr><td>xRFM (D)</td><td>1236</td><td>0.149</td><td>25.8</td><td>8.6</td><td>1</td><td>11.7%</td><td>1.65</td><td>0.08</td></tr><tr><td>LightGBM (D)</td><td>1231</td><td>0.069</td><td>25.9</td><td>24.5</td><td>0</td><td>9.7%</td><td>2.03</td><td>0.30</td></tr><tr><td>XGBoost (D)</td><td>1195</td><td>0.104</td><td>27.4</td><td>24.5</td><td>0</td><td>10.4%</td><td>2.15</td><td>0.18</td></tr><tr><td>RandomForest (T+E)</td><td>1193</td><td>0.056</td><td>27.5</td><td>25.5</td><td>0</td><td>12.1%</td><td>526.17</td><td>0.77</td></tr><tr><td>RandomForest (T)</td><td>1142</td><td>0.046</td><td>29.9</td><td>27.5</td><td>0</td><td>12.8%</td><td>526.17</td><td>0.12</td></tr><tr><td>EBM (T+E)</td><td>1133</td><td>0.122</td><td>30.0</td><td>16.4</td><td>0</td><td>14.7%</td><td>2124.78</td><td>0.12</td></tr><tr><td>ExtraTrees (D)</td><td>1117</td><td>0.056</td><td>30.6</td><td>27.7</td><td>0</td><td>13.0%</td><td>0.42</td><td>0.06</td></tr><tr><td>FastaiMLP (T+E)</td><td>1086</td><td>0.005</td><td>31.8</td><td>30.6</td><td>0</td><td>13.1%</td><td>527.21</td><td>2.83</td></tr><tr><td>EBM (T)</td><td>1076</td><td>0.131</td><td>32.2</td><td>9.4</td><td>1</td><td>15.3%</td><td>2124.78</td><td>0.01</td></tr><tr><td>TorchMLP (D)</td><td>1075</td><td>0.015</td><td>32.4</td><td>30.3</td><td>0</td><td>12.9%</td><td>20.50</td><td>0.08</td></tr><tr><td>FastaiMLP (T)</td><td>1046</td><td>0.000</td><td>33.3</td><td>32.3</td><td>0</td><td>13.6%</td><td>527.21</td><td>0.31</td></tr><tr><td>EBM (D)</td><td>1009</td><td>0.071</td><td>34.7</td><td>30.3</td><td>0</td><td>16.0%</td><td>7.25</td><td>0.04</td></tr><tr><td>TabICL (D)</td><td>1006</td><td>0.000</td><td>34.8</td><td>34.2</td><td>0</td><td>14.2%</td><td>0.63</td><td>0.06</td></tr><tr><td>RandomForest (D)</td><td>1000</td><td>0.000</td><td>34.8</td><td>34.2</td><td>0</td><td>14.2%</td><td>0.63</td><td>0.06</td></tr><tr><td>FastaiMLP (D)</td><td>916</td><td>0.000</td><td>37.2</td><td>36.5</td><td>0</td><td>17.8%</td><td>3.08</td><td>0.29</td></tr><tr><td>KNN (T+E)</td><td>533</td><td>0.000</td><td>43.8</td><td>43.6</td><td>0</td><td>37.3%</td><td>2.25</td><td>0.15</td></tr><tr><td>Linear (T+E)</td><td>485</td><td>0.000</td><td>44.3</td><td>44.4</td><td>0</td><td>35.5%</td><td>46.50</td><td>0.14</td></tr><tr><td>KNN (T)</td><td>434</td><td>0.000</td><td>44.9</td><td>44.7</td><td>0</td><td>38.0%</td><td>2.25</td><td>0.03</td></tr><tr><td>Linear (T)</td><td>426</td><td>0.000</td><td>44.9</td><td>44.8</td><td>0</td><td>35.7%</td><td>46.50</td><td>0.04</td></tr><tr><td>Linear (D)</td><td>312</td><td>0.000</td><td>46.0</td><td>46.0</td><td>0</td><td>38.1%</td><td>1.16</td><td>0.08</td></tr><tr><td>KNN (D)</td><td>263</td><td>0.000</td><td>46.5</td><td>46.2</td><td>0</td><td>41.6%</td><td>0.04</td><td>0.02</td></tr></tbody></table>
<figure><img src="image_5.png" /><figcaption>**Figure C.1:** Runtime comparison as a function of the number of training samples on the covertype dataset. Here, L1 kernel refers to the $L_p^p$ kernel.</figcaption></figure>

**Task:** $x \sim \text{Normal}(0, I)$;

$$f(\mathbf{x}) = \begin{cases} |x_0|(x_1 + x_3 + x_5), & x_0 > 0 \\ -|x_0|(x_9 + x_{11} + x_{13}), & \text{otherwise} \end{cases}$$

**AGOP matrices:**

<figure><img src="image_6.png" /><figcaption>**Figure C.2:** Training xRFM on synthetic data where splitting on the top AGOP direction enables xRFM to learn locally relevant features.</figcaption></figure>