MotherNet: Fast Training and Inference via Hyper-Network Transformers
Authors: Andreas C Mueller
Carlo A Curino
Raghu Ramakrishnan
Venue: ICLR 2025 Poster (2025)
Keywords: hypernetwork
tabular data
meta-learning
foundational models
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: MotherNet is a foundational hypernetwork that can produce trained neural networks for small tabular datasets using in-context learning without finetuning.

Abstract: Foundation models are transforming machine learning across many modalities, with in-context learning replacing classical model training. Recent work on tabular data hints at a similar opportunity to build foundation models for classification for numerical data. However, existing meta-learning approaches can not compete with tree-based methods in terms of inference time. In this paper, we propose MotherNet, a hypernetwork architecture trained on synthetic classification tasks that, once prompted with a never-seen-before training set generates the weights of a trained ``child'' neural-network by in-context learning using a single forward pass. In contrast to most existing hypernetworks that are usually trained for relatively constrained multi-task settings, MotherNet can create models for multiclass classification on arbitrary tabular datasets without any dataset specific gradient descent.
The child network generated by MotherNet outperforms neural networks trained using gradient descent on small datasets, and is competitive with predictions by TabPFN and standard ML methods like Gradient Boosting. Unlike a direct application of TabPFN, MotherNet generated networks are highly efficient at inference time.

Published as a conference paper at ICLR 2025
Mo T TERNETT: FAST TRAINING AND INFERENCE VIA HYPER-NETWORK TRANSFORMERS
Andreas C. Müller, Carlo Curino & Raghu Ramakrishnan Gray Systems Lab
Microsoft
(amueller, ccurino,
ABSTRACT
Foundation models are transforming machine learning across many modalities, with in-context learning replacing classical model training. Recent work on tab- ular data hints similar opportunity to build foundation models for classifi- cation for numerical data. However, existing meta-learning approaches can not compete with tree-based methods terms inference time. In this paper, we propose Mot MotherNet. hypernetwork architecture trained synthetic classifica- tion tasks that, once prompted with a never-seen-beiee training set generates the weights of trained "child" neural-network by in-context learning using single forward pass. In contrast to most existing hypernetworks that are usually trained for relatively constrained multi-task settings, MotherNet can create models for multiclass classification on arbitrary tabular datasets without any dataset specific gradient descent. The child network generated by Mot rNet outperforms neu- ral networks trained using gradient descent on small datasets, and is comparable to predictions by TabPFN and standard ML methods like Gradient Boosting. Un- like direct application of TabPFN, MotherNe generated networks are highly efficient at inference time. We also demonstrate that HyperFast unable to per- form effective in-context learning on small datasets, and heavily relies dataset specific fine-tuning and hyper-parameter tuning, while MotherNe requires no fine-tund fine-tuning per-dataset byper-parameee
INTRODUCTION
Foundation models, i.e., large transformer-based (Vaswani et al. 2017) models trained on massive corpora, are disrupting machine learning in many areas such as natural language and reasoning tasks. These domains shifted from small task-specific models to large generic models with task- specific instructions via prompting and in-context learning. However, this shift has not yet reached tabular data, the most common data type in real-world machine learning applications (Chui et al. 2018). which still dominated by traditional machine learning methods and neural networks with in-weight learning. This paper explores Models to tabular classification, demonstrating their potential to replace costly and slow AutoMI. with in-context learning. The existing TabPFN (Hollmann et al. (022)) approach is promising in terms of accuracy and training time, but falls short of state-of-the-at art classical solutions in terms of training scale, being restricted 1000 3000 data points for training, and inference runtime, being approximately ten times slower to predict than a comparable tree- based method.
approach applying transformer-basee Foundational
We introduce a new architecture, called MotherNet, which adapts the TabPFN architecture to a hypemetwork setup to produce model weights for feed forward neural network. Our method performs competitively with baseline methods such as gradient boosting (Friedman) 2001 Chen & Guestrin 2016) and outperforms learning a neural network with gradient descent, being much faster to train, providing higher accuracy and requiring no tuning of hyperparameters on small tabular datasets. Our approach combines the transformer architecture of TabPFN (Hollmann et al.) 2022) with the idea of hypernetworks (Ha et al.) 2017). to produce state-of fate--o-thhe-aee ff-thhe-n art classification models in single forward pass. Unlike original work in hypemnetorks (Ha et al. (017), which used small hyper network to generate large "main" network, we are training large, transformer based hypernetwork generate compact classification network.

---

Published as a conference paper at ICLR 2025
Compared to the approach of Hollmann et al. (2022), , this allows for much shorter prediction time. However, just as TabPFN, MotherNet is restricted by the quadratic memory requirements of the transformer architecture, and does not scale well above upproximately 5,000 data points. Compared to earlier work on hypernetworks, we train a single hypernetwork to address tabular classification on numeric data in general, i.e. in the style of foundational model, instead of task-specific or multi-task hypernetwork.
We demonstrate that is possible generate neural networks directly as the output of a transformer model, without the need to do any dataset-specific learning or gradient descent. Using a fixed model structure, we are able produce neural networks that work well on small numeric tabular datasets from the OpenML CC-18 benchmark suite (Bischl et al.) 2017), and show that our approach also provides a good trade-off of speed and accuracy on the TabZilla dataset collection McElfresh et al. 2024). Training and inference code and pre-trained model weights are made publicly available
2 RELATED WORK
2.1 LLMS FOR TABULAR DATA
There have been several works investigating the use of small, heterogeneous tables for question answering, and extracting tables from data, using large language models (LLMs) and specifically fine-tuned transformers (Yin et al. [2200] lida et al. (02T). These architectures are table-specifis but meant to answer natural language questions. Our work focuses on supervised classification. Separately approaches have been proposed to apply Large Language Models directly to supervised tabular classification tasks (Dinh et al., 2022 Hegselmann et al.) 2023), These works generally require tokenization on the level of each input feature. This allows including world-knowledge about feature names and potentially feature values, but comes at an extreme computational cost, and strong limitations on the number of features and data points, the size of the attention matrix is quadratic in both features and samples, and has a non-trivial constant factor for encoding floating point numbers into separated string values.
2.2 TABPFN
Recently Hollmann et al. (2022). building on the work of Muller al. (00TT) introduced trans- former architecture that is capable of performing supervised classification on tabular numeric data. This work quite distinct from other transformer architectures on tabular data in that is focused on numeric input and numeric output.
TabPFN uses a transformer where each input "token" a row of the tabular dataset. The model is adapted to work with a variable number of features by zero- proo-padding (and scaling) to 100 features. For the training data, linear projections of the input rows are summed with linear projections of integer classification labels. For the test data, the labels are omitted and class-probabilities are produced as output tokens. The model trained to minimize cross-entropy on the test data points. Attention is masked so that all training points attend to all other training points, while test points can only attend to training points. A variable number of classes handled by training for up to ten classes, and when predicting for dataset with & < 10 classes, using only the first k outputs in the softmax layer. The authors design a prior over synthetically generated datasets, based on Structural Causal models and Bayesian Neural Networks. Using draws from this prior, they are able to train model that generalizes to perform supervised classification on real-world tabular datasets. TabPFN FN showed strong predictive performance without any per-dataset tuning, and with extremely fast time to train and predict on small datasets (< 3000 data points) (McElfresh et al. 2024). Because of the quadratic nature of the self- celf-attention matrix, training on larger datasets is impractical with the method proposed in Hollmann et al. (2022). Our work builds on the work of Hollmann et al. (0022). and adds the capability to create speaset- model.
2

---

tuning is necessary Published as a conference paper at ICLR 2025
2.3 LEARNING TO LEARN AND
There has been long history of approaches to "learning to learn" and build neural network that produce other neural networks (Schmidhuber, 1992 Ravi & Larochelle, 2016)] Andrychowicz et al. 2016 Thrun & Prati 1998). Given the long history, we will only review some more recent and closely related approaches. Most approaches solve transfer-learning or task-adoption within a fairly narrow family of tasks, often one-dimensional regression or character recognition, while our work addresses classification on any small tabular dataset. |Ha et al. (2017) introduce the term hypernet- work for networks that produce networks using task-specific embeddings. The hypernetwork and embedding are both learned via gradient descent the same dataset, reducing the approach (in the non-recurrent case) to a standard neural network with a low-rank structure in its weights. [Bertinetoo et al.] (2016) propose a gradient-free approach to produce student networks based single-shot examples in OCR and object tracking. Their objective and formulation closely resembles ours; however, in this work, a single "exemplar" is a tabular training dataset, while in Bertinetto et al. (016), it is a single handwritten digit, or a single object to track. A transformer based approach for a hypernetwork generating convolutional neural networks has been investigated in /Zhmoginov et al. 2022). Conditional Neural Processes (Garnelo et al. (0118) also perform task adaption without gra- dient descent. In contrast to the original work Garnelo et al. (2188) we are using a transformer architecture instead of a feed-forward neural network, and are able to address much wider range of tasks. While later work on Neural Processes used transformers (Nguyen & Grover (022) in an ar- chitecture closely related to TabPFN, are not aware of an implementation of Conditional Neural Processes using transformers, which would yield an architecture more similar to Mot he reet. Most recently,Boonet
|Bonet et al.) (2023) introduced HyperFast, a hypernetork that, similar to Mot rNet, is trained to perform classification on generic tabular datasets. HyperF ast avoids the quadratic complexity of the transformer attention mechanism, and therefore scales to larger datasets. We compare to HyperF ast with and without gradient descent and with hyper-parameter tuning in Section
4 and find that despite the large overlap of the HyperFass Fast training set and our test-set, HyperF ast without gradient descent performs poorly, and for good performance.
3 METHODOLOGY
Hollmann et al. (2022) introduced TabPFN, an adaption of the transformer architecture to solve tabular classification problems. As this work closely builds on TabPFN, we want to briefly re- view its architecture. TabPFN uses transformer where each input "token" is row of the tabular dataset. The model is adapted to work with a variable number of features by zero-padding (and scaling) to 100 features. For the training data, linear projections of the input rows summed with linear projections of integer classification labels. For the test data, the labels are omitted and poas-prrbabiliitiss are produced as output tokens. The model is trained minimize cross-entropy on the test data points. Attention is masked so that all training points can attend to all other training points, while test points can only attend to training points. A variable number of classes is handled by training for up to ten classes, and when predicting for a dataset with & < 10 classes, using only the first outputs the softmax layer. TabPFN showed strong predictive performance without any per-dataset tuning, and with extremely fast time to train and predict on small datasets (S 3000 data points) (MEllfresh et al., 2024). Because of the quadratic nature of the self-attention matrix, training on larger datasets is impractical with the method proposed in Hollmann et al. (2022).
Limitations of TabPFN Comparing speed and computational efficiency between TabPFN and traditional ML and AutoML methods is somewhat complicated, as they have very different charac- teristics. In particular, there no dataset specific training phase after meta-training when applying TabPFN, only near-instantaneous in-context learning. Prediction, on the other hand, is signifi- cantly slower than prediction in standard ML models, as prediction requires computing attention between test and training data. On the other hand, gradient boosted trees (Friedman) 2001 Chen & Guestrin 2016, Ke et al., 2017) have significant training cost, in particular when accounting for papeeeprreeeter tuning, but are extremely fast for prediction. Together with the memory quirements of a large transformer model, this makes TabPFN impractical for settings where fast, on-demand predictions are required. Next we present two approaches, an effective baseline distil-
3

---

Published as a conference paper at ICLR 2025
y
Encoding
ReLU
Transformer Block
Trensformer Black
Translormer Block
redeceed
Rod
Transformer Block
Transformer Block
Transtormer Block
BeLLU
Linwrr Embbeddingg
Linwar Embetdling
Lineer Embelding
fred
3/1
2/2
f/2
Yr
3
Figure MotherNet architecture. Given training data (ø1, vi)..... (Z,, yr), the transformer produces vector (, which is reshaped weight matrices of an MIP with low-rank weight structure. Green blocks are individual data points and their activations, orange layers are activations created during in-context learning or in the forward-pass during meta-learning, and light blue layers learned during meta-training.
lation approach, and MotherNet, that address some of the runtime and scalability limitations of TabPFN.
3.1 MOTHERNET: GENERATING MODEL WEIGHTT
Motivated by the success of T TabPFN, and inspired by previous work on hypernetworks, we propose NotherNet, transformer architecture that is trained to produce machine learning models with trained weights single forward pass. This methodology combines the benefits a Foundation Model that does not require dataset specific training tuning with the high efficiency of a compact model at inference time. The resulting models are small feed-forward neural networks (an MLP with two hidden layers of size 512 in our experiments) that have competitive performance, created without the use of back-propagation or any loss minimization on the training set. The training process of the overall architecture can be described as:
min 0
C(MPP D.),
where o = Motherreee (D),
Where are the parameters of the MotherNer transformer, DŁ and D. are training and prediction portion of dataset i, MLP& is the feed-forward neural network with parameters given by á and C(M, D) is the cross entropy loss of the model AM evaluated datasets D. The model architecture shown in Figure T Training performed by back- back- propaaaaoon through the whole architecture (from the output of the child model, and through the transformer layers) where each training sample corresponds to one dataset. During this meta-training, the parameters 0 are learned using synthetic datasets, using the prior from [ollmann et al. (002)) and then frozen. To apply the model new (real) dataset D consisting of a training portion De and a prediction portion DP, we evaluate Mothe Nitberree(D (DM (D*,)), 8), which produces a vector of parameters 3. This vector then used as the weight and bias vectors of feed-forward neural network (properly reshaped), which can be used to make predictions on DP. We refer to this approach of applying MotherNet create child network as in-context learning. The absence of per-dataset gradient descent in our method not only provides advantage in terms of runtime complexity, also eliminates the need apply and tune regularization, as the model was trained directly for generalization, similar to a Conditional Neural Process (Garelo al. 2018).
Mot he reet maintains the structure of input encoding and twelve transformer layers TabPFN on the training set, which produces embeddings of size m (512 for the experiments) for each pair of

---

Published as a conference paper at ICLR 2025
Hyperfast
Hyperfast defoult-
12
2
ResNet
model
RandomFores
Logistic Regression"
K022 Hapeffe:: Ins Pyperfst Iyperfas Rendemtored ResNet"
ToPPPN XCRooet
Mothertaat FT!
ML MLP-tistil
MotherNet
XGBoost
Rlegessaoon
TaDPFN
0.85 0.87 Average RDC AUC. (normallzed)
Figure 2: Comparison of TabPFN, HyperFast, ML.Pdi dttdisttii ill and MotherNet with tuned
baselines over the test datasets of Hollmann al. (2022), listed Table g t means 1h of HPO on CPU, $ means 1h of HPO on GPU. Left: Comparison of normalized mean ROC AUC. Right: Critical Differences Diagram (Demšar 2006).
training data point and label. We use a one-hot encoding of classes, unlike Hollmann et al. (2022) who use a linear layer. We use the training labels to compute the average embedding per class, re- ducing all activations to single dataset embedding E of size maul (512- 10 in the experiments). This embedding E is decoded into the vector 6 using mne-hddeen-layer The vector á of activations based on the transformer is then reshaped into weights and biases for the "child" doeward network. We evaluated different variants of the architecture on the validation set of Hollmann al. (2022). Initial experiments showed this approach to be very successful, but lead to very large transformer models as a function of the size of the network that was be produced. To reduce model size, we decomposed the weights into two components, WP that is produced as prediction by the transformer, and W. that is learned during the meta-training phase and fixed during in-context learning, similar to Ha et al. (2017).
forw-forward neural network.
The low-rank structure drastically reduces the number of entries in ø for given size of neural network produced. All our experiments use the low-rank version, which yielded slightly better AUC the validation set, much smaller model size. As output architecture, with two hidden layers with 512 hidden units each, and with weight-matrices of rank 32. More specifically, for rank r 32, hidden dimension h 512, d 100 features and N 10 classes, with W 1, WP E Raxrr and W E N produced by the transformer, and W! E Rrxd and W E Rexh learned during meta-learning, predictions are made as
MLP
h = relu (el
h = relu WW
P(O) = softmax (W)h2)
The best-performing version of our architecture 89M parameters, with 63M of these in the decoder attached. Somewhat surprisingly, we found that decoding MLP with hidden layer of size of 4096 works well. This means the whole training dataset first compressed to a vector of length 4096, and then expanded into a vector of size 25, 738 to encode the low-rank components of the weights for the network that produced. We train MotherNet on a single A100 GPU with 80GB of GPU memory, which takes approximately four weeks. We are using increasing batch sizes of 8, 16 and 32 and a learning rate of 0.00003, with cosine annealing (Loshchilov & Hutter 2016).
3.2 IN-CONTEXT LEARNING WITH MOTHERNET
For in in-context learning on a new dataset D' with features and classes, we perform forward pass in the transformer to obtain e MotherNet(D,. 0). We discard all but the first rows of the input layer matrix W and keep only the first columns of the output matrix WP. resulting in network with an input layer of size r, hidden layers of size 512 and an output layer of size e. Because of the quadratic complexity of full attention, the size of training dataset that is feasible to process is limited by available memory. We were able to process up to 30,000 data points on an A100 GPU with 80GB of memory, and 100,000 samples CPU. However, did not evaluate accuracy on datasets of this size. We found that the ensembling strategy of Hollmann et al. (2022) improves predictive performance. We apply similar strategy, using different circular permutations of features and classes, optionally using hote-iinnnn for categorical variables and optionally using quantile encoding for continuous features. For this larger ensembling sample 8 models
5

---

Published as a conference paper at ICLR 2025
Imput data
Nearest Neighbors
Linear SVM
Random Forest
Neural Nett
TabPFN
MotherNet
888
.78
.88
.88
.00
Figure Comparing decision boundaries synthetic toy datasets, adapted from the Mikit-llearn PPedregosa et al., 2011) documentation. MotherNet decision boundaries closely resemble TabPFN and to lesser degree traditional Neural Network boundaries.
from this space for our experiments; for TabPFN we follow the default setting of from (Holl- mann et al. (2022), which seems sufficient for that model. Sampling larger number for either improves performance but slows down both training and predictions with diminishing returns. The predictions that are produced, both by individual networks, and the ensemble, are quite smooth and similar traditionally learned neural networks or TabPFN, see Figure]
3.3 DISTILLATION BASELINE
To get a better understanding of the limitations and trade-offs inherent in the MotherNet architec- ture, we also investigate a baseline approach for creating small neural network based on TabPFN via distillation. Distillation is natural approach to reducing prediction time, while making use of the excellent performance of FN. Distillation is slower and less direct way to extract dataset specific model from the TabPFN FN approach than using Mothe rNet. However, allows us to dis- entangle the contribution of model capacity, the ability of the hypernetwork to create appropriate weights, and the predictive bias of the TabPFN training procedure. We apply the predictions of TabPFN teacher model for small feed-forward neural network, that trained specifically for a dataset, analogous to the methodology Hinton et al. (2013). Note that we are not attempting to distill TabPFN, but instead create datasct-specific distillations for each dataset we want predict on. Furthermore, while TabP FN is acting as teacher model, it never saw the dataset for which a teacher during training time. While this natural way to distill the model for dataset-specific prediction, we are not aware of this being investigated before. Tuning the distilled model architec- ture on validation set of Hollmann et al. (2022) found that relatively small model suffices for good performance, leading to a reduction in size of the model of nearly orders of magnitude: We use an MLP with two hidden layers of size 128, which results in a maximum ~ 30k parameters (with 100 input features and 10 targets, i.c., the limit in the datasets consider most datasets smaller), while the original TabPFN has ~ 26M parameters.
4 EXPERIMENTAL EVALUATION
We evaluate Mo rNet on two tabular benchmarks, the small datasets OpenML. CC18, as used by Hollmann et al. (2022) and version of the TabZilla benchmark (McElfresh et al. 2024). As previous work has shown, selection the benchmark can have a large effect on ranking of algorithms; therefore, any ranking can only be relative to a given benchmark. We use two bench- marks from the literature to show that MotherNet has competitive predictive performance, while maintaining unique combination of no hyper-parameter tuning, extremely efficient (in context) learning, and efficient prediction. All our experiments were done on a A100 GPU with 80GB of RAM on cloud infrastructure.

---

Published as a conference paper at ICLR 2025
normalized AUC
fit
predict time (s)
fit + predict
rank
AUC
time (s)
model
TabPFN
3.733
0.850+0.0026
0.003=0.003
0.008
0.337
0.344
MLP-Distill
4.767
0.802:10.027
0.885=0.004
4.382
0.002
4.384
MotherNet
5.567
0.780:10.037
0.889=0.004
0.136
0.006
0.143
MotherNet (CPU)
7.904
0.107
8.010
XGBoost
5,933
0.891=0.003 0.804
4.912
0.024
4.936
MotherNet NE
6,500
0.706+00.066
0.887 0.003
0.040
0.001
0.040
MIP
7.267
0.744
0.882 0.003
2.257
0.001
2.258
Logistic Regression!
7.300
0.74910.024
0.883 0.002
0.209
0.000
0.209
ResNet
7.867
0.710=:0.025
0.877=0.004
1.668
0.001
1.669
CandomForest
7.900
0.722
0.880=0.003
0,200
0.032
0.232
MotherNet FT
7.900
0.702
0.883=0.044
0.839
0.002
0.841
HyperFast HyperFast default
7.900
0.729
0.007
15,931
0.083
16.014
8.567
0.681100020
0.873
25.960
0.046
26.006
HyperFast no FT
11.033
0.530+00.300
1.859-0000
3.792
0.045
3.837
KNN
12.767
0.453#0.021
0.848=0.003
0,000
0.007
0.008
Table 1: Summary results on small CC-18 datasets. Average rank is based on normalized ROC
AUC. + denotes std over the five paired splits of each dataset. t means of HPO on CPU, means 1h of HPO on GPU. Note that HyperFast and ResNet use 14 of GPU tuning time per dataset, while MotherNet requires single fit that take 0.14 seconds on average.
4.1 OPENML CC18 (SMALL)
We follow the experimental evaluation of Hollmann et al. (2022), and focus our evaluation on the 30 datasets within the CC-18 with less than 2000 samples, listed in Table 8 in the appendix, and compare models using one-vs-rest ROC AUC. When aggregating across datasets, we normalize scores with the minimum and maximum performance achieved on the dataset by any algorithm. As in Hollmann et al. (2022), we split each dataset 50/50 into training (or in-context learning) and test set. and repeat this split five times. We refer to that work for an in-depth comparison of TabPFN with current AutoML methods. We compare predictive performance and runtime of the following models: TabP as provided by the authors Hollmann al. (2022), HyperFas Fast (Bonet al,, 2023). a recent hyper-network architecture for tabular classification, MLP-distill
toe Neet, vanilla MLP, ResNet using the architecture of Gorishniy et al. (202T) and baselines consisting of Histogram Gradient Boosting (Chen & Guestrin 2016 Friedman (001), k-Nearest Neighbors, Logistic Regression and Random Forests [Breiman (000T) In contrast to Hollmann
al. (0022). include datasets which contain categorical features missing values; fill missing values with zeros as in Hollmann et al. (2022). We perform no dataset specific tuming for the transformer architectures, while baseline approaches 60 minutes hour of randomized hyper-parameter tuning.
Quantitative results are shown in Figure 2 and Table 4 where errors are given over the five paired splits of the data. We see that TabPFN outperforms all other methods, though not statisti- cally significantly so, even at 60 minutes of tuning time for reference methods. While the dis- tilled version sat-distill does not achieve the same level of performance, outperforms all the baseline models even without dataset specific tuning. It seems the probabilistic predictions produced by TabPFN provide enough regularization for good generalization. Our MotherNes outperforms all the baseline approaches, and outperforms MLP sole-distill in terms of normalized ROC AUC, but outperformed by MLP-di st ill terms of rank. Results using the validation set Hollmann et al. (2022) can be found in the appendix in Figure 4 Comparing with the recent HyperFast (Bonet et al.) 2023) it should noted that the 30 datasets we use for evaluation, 18 are in the HyperFast training set, giving it direct advantage. We find that without per-dataset finetuning HyperFast (Hypere rF ast no FT) does not provide comparable performance to any of the baseline approaches except KNN. Using per-dataset fine-tuning and the default HyperFas ast (HyperFast default) is also outperformed by the baselines. Using 60 minutes of ran- domized hapeeeparameter tuning on GPU, HyperFast is competitive with XGBoost. i, but outper-
7

---

Published as a conference paper at ICLR 2025
XXN Myperhok Hyperfiit cetutt
mie Recress
Tendomforsal NLF 2uul
Iyperfust Kley! hyperhet cenuut Randoriiions Fowe!
TasPFN
xGDicse!
ModherMer:
Roc ABC (nemallad))
Figure 4: Comparison of TabPFN, HyperFast, MLP-distilI and Mot MotherNet with tuned baselines over the validation datasets of Hollmann et al. (2022). listed in Table means Ih of HPO on CPU, $ means Ih of HPO on GPU. Left: Comparison of normalized mean ROC AUC. Right: Critical Differences Diagram (Demšar 2006). Compare with Figure the test set. We did not include HyperFast MLP and ResNet because of the extreme computational cost of tuning these models for each dataset.
formed by MotherNet, MLP-distill and TabPFN, despite a large fraction of the test datasets being included in the HyperFast training. Also, note that 1h of per-dataset crper-parameter tuning on GPU corresponds to approximately 25,000x more compute than Mothe rNet, which requires no fine-tuning or hyper-parameter tuning and trains within 0.14s on average on the test datasets. Inter- estingly, the MLP which is trained using standard gradient descent is performing much worse than MLP-distiD ill and MotherNet despite extensive hyper-parameter tuning. Clearly MotherNes provides an efficient and easy-to easy-to-use use alternative to training with gradient descent on these datasets, and for the small dataset regime that investigate, syper-parameter tuning can be difficult. It's noteworthy that on the OpenML CC-18 collection, Logistic Regression performs surprisingly well. This is likely consequence of the dataset selection. Compare Figure for for the validation set, which has a more typical ranking of algorithms.
To determine whether fine-tuning is beneficial for models produced by MotherNet, perform an experiment in which we apply gradient descent to the child model on the training dataset that was used for in-context learning. Since fine-tuning for all ensemble members would be costly, we compare MotherNet without ensembling (MotherNet NE) with the fine-tuned MotherNet (MotherNet FT). We tune learning rate, weight decay, use of dropout, number of epochs, and whether or not to use one-hot-encoding (for both in-context learning and gradient descent). We were unable to improve results using careful fine-tuning using 1h of HPO on GPU. Search produced either results that were equivalent to the MLP model, ignoring the MotherNet initialization, or left the initialization unchanged. This is not entirely surprising: in most cases, Mothe rNet improves over the MLP model, and we hypothesize that this improvement stems from learned regularization performed by the hyper-network; in particular for small datasets, overfitting neural architectures, and tuning hyper-parmeter parmeters is difficult since validation set results noisy. Applying gradient descent in this setting nullifies the benefits of the learned regularization. This might longer hold when using larger datasets, which plan investigate in future work.
major issue for
Regarding algorithm speed, we only consider prediction time, TabPFN on GPU is about ten times slower than XGBoost, while MotherNet on GPU about five times faster than XGBoost, or 50 times faster than TabP FN. This is the main advantage we look to gain from using wotherNet over TabPFN. However, MLP disiiiisillll ill even faster, about 3x the speed of Mot hNret, likely due to the ensembling described in Section 3.2 However, if we look at the speed of training (assuming optimum hyper- aaper-parameters are known) together with prediction, measurement particularly critical for model development, we see that Mot Nere on GPU provides a 33x speedup over XGBoost, over 30x slower than MotherNet. See Figure (left) for visualization
while MI.P-d of the speed-AUC trade-off.
In most real-world settings, parameparameters are unknown, results in Figure 2 and Table shows that even with Ih of pypar-parameter tuning, the baseline models underperform the transformer models. Taking this tuning time into consideration, using MotherNet on the GPU results more than 25,000x speedup for model-devedd developmee,,,,, while MotherNet speedup. We want to point out that these speedup figures might depend strongly on the tuning time estimated for other algorithms. This points practical difficulty of tuning parameters: in practice it
the CPU still provides 450x
8

---

Published as a conference paper at ICLR 2025
20191817
Hyperrsst
Hoperaast
107
MLP-DistIl, VLP-DisuIAAGBBor XSBoost
004predet
ResNet MIP!
KNN
CatBoost
DecisionTeee
200
Mother FT!
Tablet
SAINT
ResNet
JEPPFN
LndearModel
RandomFFoestt
RandomForeet
Logistic Regression' Motherket
Motherfett
MLP-ttdl
DANet
0.75
0.50
0.85
SVM
LightGBM
Normalized
FTTanstomer
NODE
Figure 5: Left: Visualizing runtime normalize AUC trade-off, based numbers in Table 4 not considering tuning times for the competing algorithms. Bottom right means fast and accurate algorithms. Note that the y-axis in log-scale. Right: Critical Difference Diagram on the TabZilla benchmark using ROC AUC, corresponding to results in Table 2.
often unelear how much time should be allocated for hyper- ppper-prreettrr tuming. Using Mothe removes this trade-off by providing competitive accuracy near-instantaneouslyy without any tuning. Mothe rNet and TabPFN can be found
more detailed comparison between MLP-distiD MLP-distiLL performs much worse than in Table 3 and T We see that some datasets, MotherNet: likely because of the sensitivity of gradient descent to hyper-parameters. Detailed results can be found in Table 3 in Appendix
4.2 TABZILLA
We the TabZilla (McElfresh et al. 2024) benchmark compare wide varity of deep learning and tree-based approached. Table 4.2 reproduces the results of McElfresh et al. (2024), with our results for MotherNet added. For this evaluation, we follow McElfresh et al. (2024) in their setup for FabPFN, and subsample 3000 data points for MotherNet, as the full datasets are too large for the transformer architectures. This means that both MotherNet and TabPFN have severe disadvantage, this disadvantage, TabPFN still outranks other algorithms. MotherNet is outranked by some of the tree-based learners, as well as SAINT and ResNet in rank, though MotherNet, SAINT and ResNet have equivalent mean normalized AUC. The critical difference diagram using ROC- be found in Table 7 and Table 0 in
they only see a fraction of the data provided to other algorithms. Despite
AUC can be found in Figure 5 (right), and more results the Appendix. The CD diagram shows no significant differences between the top seven algorithms, despite the fact that other algorithms were given up 10h of compute and up 30 hyper-paramete sets, while MotherNet requires no hyper-parameter tuning and finishes within seconds on all datasets. The combined training and prediction time of Mot MotherNet is competitive with those of the tree-based models (though comparing Mot herNet on GPU with tree-based models on CPU) and orders of magnitude faster than other deep learning approaches.
5 LIMITATIONS AND FUTURE WORK
One of many open questions is to understand how the models produced by MotherNet differ from those produced by gradient descent. The nature of the optimization fundamentally differ- ent, and in essence, meta-training, instead of using a hard-coded regularization strategy such as weight decay or early stopping. We were able to achieve good performance with single neural network architecture across all datasets, both for MotherNer and chitectures for the two), which seems hard to achieve with standard gradient descent. The relative performance of MLP-distill TabPFN and KotherNet is somewhat muddled, and inconsis- tent between the test set and validation set, figures Figure 2 and Figure 4 and between mean AUC and rank. We found that using ane-hot-encodeee is critical for the prediction network produced by Mot reet to perform well, an issue that is not present in the TabPFN architecture, and neces-
rNet learns to regularize according to the datasets presented during
(though with slightly different ar-

---

Published as a conference paper at ICLR 2025
rank
normalized AUC
fit time (s)
predict time (8)
fit + predict (s)
model
TabPFN
5.10
0.91=0.17
0.25
16.13
16.38
CatBoost
5.99
0.92=0.18
20.75
0.05
20.80
XGBoost
6.55
0,90=0.19
0.85
0.04
0.89
ResNet
7.65
0,8410.19
15.95
1.61
17.56
SAINT
7.66
0.84=0.19
171.13
2.56
173.69
RandomForest
7.93
0.87=0.19
0.41
0.56
0.97
MotherNet
8.30
0.84
0.34*
0.11*
0.45*
DANet
8.65
0.83+0.19
64.50
1.32
65.82
LightGBM
9.20
0.83+0.22
0.89
0.04
0.93
NODE
9.77
0.81=0.20
161.05
1.81
162.86
FTTransformer
10.00
0.7910.20
27.88
1.85
29.73
SVM
10.52
0.75=0.22
61.21
0.24
61.45
MLP-rtdl
10.61
0.73=0.21
15.18
1.57
16.75
STG
12.01
0.66
18.66
0.03
18.69
Logistic Regression
12.79
0.62-0023
0.04
0.01
0.05
MLP
13.30
0.65+0.23
18.32
1.48
19.80
TabNet
13.50
0.63+0.32
35.19
0.61
35.80
DecisionTree
14.68
0.53+0.30
0.02
0.01
0.03
KNN
15.40
0.52
0.01
0.42
0.43
VIME
15.84
0.49
17.90
1.45
19.35
Table 2: Ranking of algorithms on TabZilla dataset collection, using normalized ROC AUC. As datasets have widely varying sizes, times are per 1000 data points. *Indicated experiments run for this paper, which are run on A100 GPU as opposed V100 GPU as used by McElfresh et al. (204)) to produce the other timing results.
sitates additional bagging for prediction. future work, we hope to address this issue directly in the architecture. There are also certain failure cases that TabPFN and MotherNet share, which are discussed in Appendix B Another area exploration is scaling the MotherNet method to larger training datasets. As mentioned above, the transformer architecture does not scale well in number of datapoints, and we focus our evaluation on training sets with 3000 samples or fewer. There is substantial literature improving the complexity of attention mechanisms (see (Tay et al., 2022) for an overview), ), as well as more recent work into attention free architectures (Fu et al. 2023 Poli al. 2023). Both promising candidates for scaling Mothe rNet larger dataset sizes.
CONCLUSION
We demonstrated that it is possible to achieve competitive results on small numeric tabular datasets by producing neural networks using in-context learning via single forward pass in MotherNes rNet architecture, without using dataset specific gradient descent or hyper-parameter tun- ing. By employing pure meta-learning approach, similar Conditional Neural Processes, remove the need for explicit regularization, and therefore eliminate the hyper-parametert usually associated with learning neural networks. Compared to TabPFN, prediction speed on the set is much faster, and training and prediction speed are both comparable to highly optimized tree- based models. We also find that distilling TabPFN, into small neural network highly effec- tive and doesn't require dataset-specis hyper-parameter tuning as opposed to training a similar neural network from scratch. Our work outperforms the other recent hyper-network architecture HyperFast small datasets, both in accuracy and runtime, while not requiring dataset-specific gradient descent or hyper- hyper-paramenn parameter tuning, and despite our test set having large overlap with the HyperFast training set. The fact that competitive models can be generated with simple forward pass is quite surprising, and opens up a new direction for producing high- gigh-performance models with fast inference using deep learning techniques.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for optimization. Advances in Neural Information Processing Systems, 24, 2011.
Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed- forward one-shot learners. Advances in neural information processing systems, 29, 2016.
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G Mantovani, Jan N Rijn, and Joaquin Vanschoren. Openml benchmarking suites. arXiv preprint arXiv: 03733, 2017.
David Bonet, Daniel Mas Montserrat, Xavier Giró-i Nieto, and Alexander Ioannidis. Hyperfast: Instant classification for tabular data. In NeurlPS 2023 Second Table Representation Learning Workshop, 2023.
Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Michael Chui, James Manyika, Mehdi Miremadi, Nicolaus Henke, Rita Chung, Pieter Nel, and Sankalp Malhotra. Notes from the frontier: Insights from hundreds use cases. McKinsey Global Institute, 2, 2018.
Janez Demšar.
Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1-30, 2006.
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: Language-interface fine-tuning for non- language machine learning tasks. Advances in Neural Information Processing Systems, 35:11763: 1763- 11784, 2022.
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pp. 1189-1232, 2001.
Daniel Y Fu, Simran Arora, Jessica Grogan, Isys Johnson, Sabri Eyuboglu, Armin W Thomas, Benjamin Spector, Michael Poli, Atri Rudra, and Christopher Ré. Monarch mixer: simple sub-quadratic gemm-based architecture. arXiv preprint 23X1:2310.122009 2023.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes, In International conference on machine learning, pp. 1704-1713. PMLR, 2018.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 18:188922-8944 2021.
David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks. 2017. URL / net/ppdFid==kppceell.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 1549-5581. PMLR, 2023. PMLR,
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: /503.025 2015.
11

---

Published as a conference paper at ICLR 2025
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A trans- former that solves small tabular classification problems in a second. NeurIPS 2022 First Table Representation Workshop, 2022. URL hatps::/apenreview... forem/ff
eu9fviVasr4]
Hiroshi lida, Dung Thai, Varun Manjunatha, and Mohit lyyer. Tabbie: Pretrained representations of tabular data, arXiv preprint arXiv:2105.02584 2021.
Guolin Ke, Qi Meng. Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint orXiv:1688.00990 2016.
Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems, 36, 2024.
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Trans- formers can do bayesian inference. arXiv preprint arXiv:2112.10510 2021.
Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty meta learning via sequence modeling. In International Conference on Machine Learning, pp. 16569-16594. PMLR, 2022.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-leam: Machine learning python. the Journal of machine Learning research, 12:2825-2830, 2011.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint 2:Xx::2302.1086 2023.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International conference on learning representations, 2016.
Jürgen Schmidhuber. Learning to control fast weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1): 1992.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Process- ing Systems, 33:7537-7547, 2020.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, ACM Computing Surveys, 55(6):1-28, 2022.
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. Learning to learn, pp. 3-17. Springer, 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding of textual and tabular data, arXiv preprint arXiv:2005.08314, 2020.
Andrey Zhmoginov, Mark Sandler, and Maksym Vladymyrov. Hypertransformer: Model generation for supervised and suni-super sew-shot learning. In International Conference on Machine Learning, pp. 27075-27095 PMLR, 2022.
12

---

Published as a conference paper at ICLR 2025
Hyper Fast
KNNT
Log Regi
MLPF
MN
MN
MN FT
RFF
ResNet TabPEN XGB
dataset
NE
MiceProtein
1.00
1.00
1.00
1.00 1.00 1.00 1.00 1.00
1.00
1.00
1.00
analcatdata
1.00
1.00
1.00
1.00 1.00 1.00 1.00 1.00
1.00
1.00
1.00
authorship
analcatdata .mfft
0.56
0.55
0.57
0.57
0.57 0.56 0.55 0.59
0.54
0.58
0.57
balance-scale
0.99
0.89
0.96
0.99 0.99
0.99 0.99 0.84
0.97
1.00
0.99
banknote
1.00
1.00
1.00
1.00 1.00 1.00 1.00 1.00
1.00
1.00
1.00
authentication
blood-transfuse trasssstnnf
0.73
0.71
0.75
0.73 0.76 0.76 0.76 0.72
0.74
0.76
0.74
service-center
breast-w
0.99
0.98
0.99
0.99 0.99 0.99 0.99 0.99
0.99
0.99
0.99
car
0.99
0.92
0.98
1.00 0.97 0.97 0.98 0.99
1.00
1.00
1.00
climate-mon model
simulation
0.90
0.85
0.93
0.91 0.95 0.94 0.93 0.87
0.92
0.94
0.93
crashes
cmc
0.69
0.63
0.68
0.67 0.73 0.72 0.71 0.73
0.68
0.73
0.73
credit-approval
0.93
0.91
0.92
0.92 0.93 0.93 0.93 0.94
0.91
0.93
0.94
credit-g
0.76
0.73 0.77
0.76 0.79 0.79 0.79 0.79
0.76
0.79
0.79
cylinder-bands
0.81
0.78 0.82
0.84 0.83 0.83 0.80 0.87
0.83
0.83
0.89
diabetes
0.81
0.81 0.84
0.84 0.84 0.84 0.83 0.83
0.83
0.84
0.84
dresses-sales
0.55
0.56 0.57
0.57 0.59 0.61 0.58 0.56
0.56
0.54
0.59
eucalyptus
0.87
0.80 0.90
0.89 0.93 0.92 0.90 0.90
0.89
0.92
0.90
ilpd
0.70
0.65 0.74
0.73 0.73 0.73 0.70 0.71
0.70
0.74
0.71
kc2
0.80
0.78 0.83
0.81 0.83 0.83 0.83 0.83
0.81
0.83
0.82
mfcat-fourier
0.98
0.97 0.98
0.97 0.97 0.97 0.97 0.98
0.98
0.98
0.98
mfeat-karhunen
1.00
0.99 1.00
1.00 0.99 0.97 0.99 1.00
1.00
1.00
1.00
mfeat-morphologiann
0.97
0.95 0.97
0.97 0.96 0.96 0.96 0.96
0.97
0.97
0.96
mfeat-zernike
0.98
0.98 0.98
0.98 0.98 0.97 0.97 0.97
0.98
0.98
0.97
pel
0.82
0.78 0.83
0.80 0.83 0.85 0.84 0.84
0.79
0.87
0.84
pe3
0,82
0.75 0.79
0.80 0.81 0.81 0.80 0.82
0.79
0.84
0.82
pe4
0.90
0.82 0.89
0.90 0.93 0.92 0.91 0.92
0.88
0.94
0.93
gsar-biodeg
0.92
0.89 0.92
0.92 0.92 0.92 0.92 0.92
0.92
0.93
0.92
stecl-plates-fun
0.95
0.92
0.94
0.95 0.94 0.91 0.93 0.96
0.95
0.96
0.96
tic-tac-toe
0.92
0.99
1,00
1.00 0.99 1.00 1.00 0.98
1.00
0.96
1.00
vehicle
0.95
0.88
0.95
0.96 0.95 0.94 0.94 0.92
0.95
0.96
0.93
wdbe
0.99
0.99
0.99
0.99 1.00 1.00 1.00 0.99
0.99
1.00
0.99
Table 3: Per dataset results on the test set for small CC-18 datasets, averaged over splits. means
Ih of HPO on CPU, $ means of HPO on GPU. MN is herNer,,, MN no bag means a single network without bagging, and MN GD means a single network without bagging, with additional per-dataset gradient descent. It's unclear how to combine bagging and gradient descent here, which is why we consider comparison against the unbagged model.
Appendices
APPENDIX A PER-DATASET COMPARISON ON THE TEST SET
We show a per-dataset comparison of average ROC AUC of TabPFN, MotherNet, MLP-distill and XGBoost in Table 3 In contrast to the validation set, there seems to be no clear winner between ULP-distill and MotherNet. Overall, it seems hard to determine overall trends, but it's likely that dataset characteristics play a role, as we can observe similar rela- tive performance in eucalyptus. datasets and Mi ceP rotein show a very different profile. We plan to revisit this comparison after addressing the issues discussed in Section
and cyl cylinder- -badds, while the mfeat
13

---

Published as a conference paper at ICLR 2025
0.9
0.9
0.8
model
0.7
MLP
0.7
MLP
TabPFN
TabPFN
Rand Farest
RandomForest
MetherNet
0.6
MatherNet
tabpfn_ bool pool_prior__g
40
60
100
10
steps
rank
Figure 6: Mean test-set accuracy on synthetic binary classification datasets comparing TabPFN and MotherNet to untuned variants of with variable number of jumps. Right: boolean functions of varying rank.
classifiers. Left: one-dimensional dataset
APPENDIX B VALIDATION SET RESULTS
Maybe somewhat surprisingly. the ranking is quite different than on the test set, with MotherNet outperforming TabPFN both in rank and normalized ROC AUC. Examined datasets with at least 0.05 ROC AUC difference between Mother rNet and XGBoost, which are shown in Table 4 Overall, MotherNet and TabPFN have similar characteristics, might be expected from the shared synthetic training data. It's no- table that both outperform XGBoost on the same dataasets (top rows) and are both outperformed by XcBoost the same datasets. The last three rows of Table show particular stark failure which
Experimental results for the validation set are shown in Figure 4
mode of TabPFN and MotherNet, who perform at chance level on marity5-plun essentially solved by XGBoost, and lag severely behind XGBoost on teachingAssist.. and schizo.
Investing these datasets, we found that there are different failure modes present. The datasets wwwhhhhsssssssse. ant and schizo have single features that are highly informative but con- tain strong discontinuities with respect the target class, see Figure C Both could be consid- ered data leakage via an ID column, but in essence these point to the fact that discontinuous func- tions with many steps, and/or memorization of ID variables are not well captured by TabPFN and Mothe rNet. While in these particular cases, the datasets could be considered faulty, there was information included in data that tree-based model was able to exploit, while TabPFN and Mothe rNet could not; in this discontinuous functions with many jumps in single continu- ous feature.
The parity5 plus illustrated different failure case: this dataset relies matching boolean patterns on subset of the columns. While Hollmann et al. (2022) showed that irrelevant features degrade the performance of TabPFN, removing the irrelevant features did not improve performance on parity5. 1acy55ppuuss-- plwppuus... aluu__ _5; the issue rather seems in the inability of FN and MotherNet to mem- orize binary patterns. Based these two failure cases, generated families of synthetic functions to illustrate the shortcomings. We compare TabPFN and MotherNet to two simple baselines, RandomFord www.ooorrretttasss:s....
and MLPClass fier from scikit-lears Pedregosa et al. (011)) with default parameters without tuming. see Appendix C for details. Figure shows that as complexity of the dataset increases, either terms of jumps in a 1d function, or in terms of complexity of a boolean function, TabPFN and Mot herNtt degrade in performance much more quickly than the Random Forest model. The MLP is able to easily learn the boolean datasets, the discontinuous Id datasets; somewhat suprisingly, given the underperformance of Mothe Net on this task, MotherNet slightly outperforms the MLP. We speculate that these failure cases can be addressed in future work by including similar synthetic datasets in the prior. It might also be necessary to adopt the architecture of Mether for example using Fourier features Tancik et al. 2020) to account for discontinuities.
not
14

---

Published as a conference paper at ICLR 2025
model
MotherNet
TabPFN
XGBoost
MLP-Distill
XGB MN
dataset
KnuggetChase3
0.754
0.770
0.643
0.651
-0.111
analcatdata_apneaz
0.929
0.877
0.840
0.939
-0.089
mc2
0.770
0.767
0.681
0.726
+0.088
conference atteendance
0.572
0,576
0.498
0.538
+0.755
disclosure.z
0.576
0.572
0.509
0.581
-0.068
PieChartl
0.863
0.885
0.808
0.879
-0.555
disclosure x noise
0.531
0.533
0.480
0.540
-0.051
Chscase_fundh
0.695
0.679
0.644
0.698
00.51
meta
0.810
0.769
0.864
0.788
0.054
analcatdata
0.890
0,865
0.947
0.899
0.057
tac
0.650
0.675
0.708
0.699
0.058
triazines
0.760
0.772
0.821
0.731
0.061
frnnsfllass
0.809
0.852
0.889
0.825
0.080
pm10
0.496
0.511
0.591
0.531
0.094
pbeseq
0.783
0.839
0.890
0.832
0.107
schizo
0.616
0.636
0.796
0.627
0.180
teachingAssistant
0.679
0.692
0.940
0.709
0.261
parity5 plus
0.453
0.456
0.992
0.452
0.539
Table 4: Subset of validation data where there is difference of at least 0.05 average ROC-AUC between MotherNet and XGBoost.
APPENDIX C FAILURE CASE DATASET GENERATION
created two families of synthetic datasets. The first is a binary classification task on a single feature, that is distributed uniformly between and 1. For each dataset that generate,
Inspired by the results shown in Table
draw 2000 samples from the uniform distribution, and Bsapps cut-off points, also between and 1. At each cut-off point we flip the class label. resulting dataset for Tesposs is show in Figure 8 where we show only 100 points for illustration purposes. Note that since the split into training and test data is done using an (class-stratified) i.i.d. assumption, this dataset is trivial to learn for any tree-based or neighbors- based learner. While is possible learn such dataset with neural network, this might require tuning the architecture, and using the M MPP with default parameters from scikit-leas fails learn this data.
The other family of synthetic datasets is inspired by the par sarity plus dataset and is random combination of boolean conjunctions of a certain rank. The training samples in all cases are all binary sequences of length 10, where each bit is one feature, hence the feature space is X = {0. 1) 1) The labels for each dataset are constructed iteratively using a logical disjunction conjunctions. In every iteration, term is created by conjoining bits chosen random, with each bit also randomly assigned a negation or not. Terms are continually added to the disjunction until at least one- me-third of the samples satisfy the formula, ensuring relatively balanced dataset. We split the dataset randomly (but class-stratified) into training and test set. This is a relaxation of the classical parity problem; for rank 1, the label would correspond simply easily learnable for any algorithm. For rank 10, the dataset is an arbitrary boolean function, which not learnable (in the sense that seeing only the training set in expectation provides no information on the test set).
of the input features and therefore should be
For the experiments in Figure 6 five-fold stratified cross cross-validand validation for each of them.
we generated 20 datasets for each rank step, and performed
APPENDIX D HYPER PARAMETER SPACES FOR BASELINE METHODS
The hyperparameters used for the baseline models discussed in Section 4 are shown in Table 5 and were tuned with HyperOpt [Brgsttra et al. (201T) following the setup of Hollmann et al. (2022). using random search.
15

---

Published as a conference paper at ICLR 2025
Model
Hyperparameters
Random Forest
n estimators: randint(20, 200), max features: choice([None, 'sqit', 'log2'I),
max depth: randint(1, 45), min samples split: choice([2, 5, 10])
MLP
hize:
size: choice([16, 32, 64, 128, 256, 512], learning rate: loguniform(10 0.01),
n_epochs: choice([10, I0, 100, 1000]), dropout_rate: choice([O, 0.1, 0.3)).
n layers: choice([1, 2, 3]), weight decay: loguniform(10 0.01)
ResNet
hidden_size: choice([ 16, 32, 64, 128, 256, 512]), learning_rates loguniform(10 ,0.11,
n_epochs: choice([10, 10, 100, 1000]), dropout choice([O, 0.1, 0.3)).
n n_layers: choice([I, 2, 3J), weight_decap, loguniform(10 0.01)
hidden_multiplies choice([1, 2,
KNN
n neighbors: randint(1, 16)
XGBoost
learning_rate: loguniform(e" -1 1), max max_hepth: randint( 10),
subsample: uniform(0.22 2. 1), colsample.bytrec: uniform(O.2. 1).
sieamppe_byleved uniform(0.2. 1), min_child weight: loguniform(e-19, e²),
alpha: loguniform(e 16, e²), lambda: loguniform(e e²),
gamma: leguniform(e 16 e²), n_estimators: randint( 100, 4000)
Logistic Regression penalty: choice(('lI', '12", "none") J), max_iter: randint(50, 500).
fit intercept: choice([True, False]), C: loguniform(e log(5))
Table 5: Hyperparameters for each model
3
OCOI
XXCD
X
class
2
CEXCC
CDCO
(0:00
1
COOD
COX
COIO
XXX
0
25
50
75
100
125
150
ID
Figure 7: Class label plotted against column in teachingAssiseee stant dataset shows strong correlation that is likely data leakage.
1
COO
OOOO
class
0
aXOI C
100
0.0
0.2
0.4
0.6
0.8
1.0
feature
Figure 8: Example of a synthetic classification example in 1d.
16

---

Published as a conference paper at ICLR 2025
min rank max rank mean rank median rank mean AUC
algorithm
TabPEN (default)
8.65
6.00
0.91
CatBoost
1
38
10.03
7.50
0.92
CatBoost (default)
37
10.89
9.00
0.92
XGBoost
32
11.35
9.00
0.91
ResNet
38
13.41
12.00
0.85
SAINT
38
13.50
11.00
0.86
RandomForest
37
13,83
13.50
0.89
MotherNet (default)
36
13.85
12.00
0.87
XGBoost (default)
1
37
14.10
13.00
0.89
DANet
34
15.22
14.50
0.85
ResNet (default)
39
15.92
16.50
0.82
LightGBM (default)
36
16.08
15.00
0.86
LightGBM
38
16.15
15.00
0.86
RandomFores (default)
37
16.55
14.00
0.85
NODE
37
16.99
17.00
0.83
SAINT (default)
39
17.05
16.00
0.82
FTTransformer
39
17.64
18.50
0.82
SVM
39
18.52
20.00
0.79
MLP-rtdl
39
18.54
17.50
0.77
NODE (default)
39 18.66
18.00
0.81
FTTranfformer (default)
39
20.93
23.00
0.72
STG
37
21.27
23.00
0.73
DANet (default)
39
21.51
22.00
0.76
MLP-rtdl (default)
39
22.95
24.50
0.66
LinearModel
39
23.22
25.00
0.68
MLP
38
23,42
25.50
0.71
TabNet
39
24.27
27.00
0.71
SVM (default)
39
24.60
28.00
0.63
DecisionTrea
39
26.89
28.00
0.62
TabNet (default)
39
27.13
29.00
0.63
KNN
39
27.82
29.00
0.62
VIME
38
28.31
31.00
0.60
MLP (default)
1
39
28.52
31.00
0.54
DecisionTree (default)
39
28.66
31.00
0.56
STG (default)
39
29.10
33.00
0.52
KNN (default)
29.12
31.00
0.57
VIME (default)
39
31.92
35.50
0.39
Table 6: TabZilla algorithm ranking using normalized ROC AUC, including the default configu- rations of all algorithms. TabPFN and MotherNet have no hyper- syper-parameters and therefore labeled "default".
17

---

Published as a conference paper at ICLR 2025
min rank max rank mean rank median rank mean Accuracy
algorithm
CatBoost
19
5.80
4.00
0.87
TabPFN
20
6.14
5,00
0.83
XGBoost
18
7.30
6.00
0.81
ResNet
20
8.09
8.00
0.75
SAINT
20
8.35
7.00
0.73
NODE
20
8.38
8.00
0.74
FTTransformer
19
8.61
8,00
0.76
RandomForest
20
8.70
8.00
0.76
LightGBM
20
8.95
8.00
0.76
MotherNet
20
9.29
9.00
0.72
SVM
19
9.59
10.50
0.69
DANet
19
10.29
10.00
0.73
MLP-rtdl
20
10.37
11.00
0.66
STG
20
12.38
13.00
0.56
DecisionTree
20
12.43
14.00
0.59
MLP
20
12.65
14.00
0.57
LinearModel
20
12.89
15.00
0.51
TabNet
20
13.36
15.00
0.55
KNN
20
14.43
16.00
0.45
VIME
20
15.80
17.50
0.37
Table 7: TabZilla algorithm ranking using (normalized) Accuracy, compare with Table in McEl-
fresh et al. (2024)
did name
d
k
did
d
11 balance-scale
5 625
1049 pe4
38
1458
14 mfeat-fourier
77 2000 10
1050 pc3
38
1563
15 breast-w
699
1063 kc2
22
522
16
65 2000 10
1068 pel
22
1109
mfeat-mmrppologicall
7 2000
1462 banknote-authentication
1372
22 mfeat-zernike
48 2000 10
1464 blood-transfusion....
5
748
23 eme
1473 3
1480 ilpd
11
583
29 credit-approval
16 690
2
1494 gsar-biodeg
42
1055
31 credit-g
21 1000
1510 wdbc
31
569
37 diabetes
9 768
6332 cylinder-bands
40
540
50 tic-tac-toe
958
23381 dresses-sales
13
500
54 vehicle
19
846
40966 MiceProtein
82
1080
188 eucalyptus
20
736
5
40975 car
7
1728
458 analcatdata authorshipp
71
841
40982 steel-plates -fult
28
1941
469 analcatdata dmft
5
797
40994 climate- modiate-mddll.....
21
540 2
Table 8: Test dataset names and properties, taken from Hollmann et al. (022)). Here did is the OpenML Dataset ID, d the number of features, the number of instances, and k the number of classes each dataset.
18

---

Published as a conference paper at ICLR 2025
APPENDIX E VALIDATION SET
We use the validation set of Hollmann et al. 2022). as listed in Table y
19

---

Published as a conference paper at ICLR 2025
did name
d
did
name
13 breast-cancer
286
765
shalcatdata_apneal
475 2
25 colic
27
368 2
767 analcatdata
475 2
35 dermatology
35
366
774 disclosure x bias
662 2
40 sonar
61
208 2
778 bodyfat
15
252 2
41 glass
10
214 6
786 cleveland
14
303 2
43 haberman
4
306
788 triazines
61
186 2
48 tae
6
151 3
795 disclosure_x tamppeed
662 2
49 heart-c
303
796 cpu
209 2
51 heart-h
14
294 2
798 cholesterol
14
303 2
53 heart-statlog
14
270
801 chscase funds
185 2
55 hepatitis
20
155 2
802 pbeseq
19 1945 2
56 vote
17
435
810 pbc
19 418 2
59 ionosphere
35
351 2
811
264 2
61 iris
5
150
814 aascasevinnes
468 2
187 wine
14
178
820 chatfield_4
13
235 2
329 hayes-roth
5
160
825 boston_corrected
21 506 2
333 monks-problems-1
7
556 2
826 sensory
12 576 2
334 monks-problems-2
7
601
827 disclosure x noise
662 2
335 monks-problems-3
7
554 2
831 autoMpg
8 398 2
336 SPECT
23
267 2
839 kdd nino-smmll
9 782 2
337 SPECTF
45
349
840 autoHorse
26
205 2
338 grub-damage
9
155 4
841 stock
10 950 2
377 synthetic control
61
600
844 breastTumor
10 286 2
446 prnn crabs
8
200 2
852 analcatdata gsssexsurvey
10
159 2
450 analcatdata_lawvuii
5
264
853 boston
14
506 2
451 irish
6
500 2
854 fishcatch
158 2
452 analcatdata braicaddaaabboadwwymmlllll
8 285
860 vinnie
380 2
460 reviewrr
reviewer
8
379 4
880 mu284
11
284 2
463 backache
32
180
886 no2
500 2
464 prmn_synth
3
250 2
895 chscase chscase.geyserl
222 2
466 schizo
15 340 2
900 chscase censusó
400 2
470 profb
10 672
906 chscase censuss
400 2
475 analcatdata.germange
6 400 4
907 chscase.censuss
8 400 2
481 biomed
9
209
908 chscase
400 2
679 rmftsa sleepdata
3 1024 4
909 chscase census2
8 400 2
694 diggle table a2
9
310
915 plasma retinol
14 315 2
717 mffts_ladata
11
508 2
925 visualizing gaalizingg-allxyy
323 2
721 pwLinear
200
930 colleges_usnewe usnews
34
1302 2
724 analcatdata vineyaad
4
468 2
931 disclosure
662 2
733 machine_gp _cpu
7
209
934 socmob
1156 2
738 pharynx
11
195 2
939 chscase whale
228 2
745 auto price
16
159
940 water-treatment
37
527 2
747 servo
5
167 2
941 lowbwt
10
189 2
748 analcatdata wildcat
6
163
949 arsenic-female- bladder
559 2
750 pm10
8
500
966 analcatdata halloffaaee
17
1340 2
753 wisconsin
33
194 2
968 analcatdata birthday
365 2
756 autoPrice
159
984 analcatdata.die draft
366 2
757 meta
22
528
2
987 collins
23
500 2
764
analcatdata apnea3
4
450
996 prnn fglass
10
214
2
Table 9; Validation dataset names and properties, taken from Hollmann al. (2022). Here did is the OpenML Dataset ID, the number of features, n the number of instances, and the number of classes in each dataset.
Table 10: Validation datasets, continued
20

---

Published as a conference paper at ICLR 2025
did name
d
did name
n
1523
vertebra-column
310
1048 jEdit 4.2 4.3
369
4153 Smartphone-Based Re...
68
180
6
1054 mc2
40
161
23499 breast-canee-drroped.....
10
277
1071 mw1
38
403
40496 LED-display-domminn..
8
500 10
1073 JEdit_4.0.4.2
9
274
40646 GAMETES Epistasis
2-...
21
1600
1100 PopularKids
11
478
40663 calendarDOW
399
1115 teachingAssistant
7
151
40669 corral
7 160
1412 lungcancer GSE31210
24
226
40680 mofn-3-7-10
1324
1442 MegaWatt1
38
253
40682 thyroid-new
215 3
1443 PizzaCutterl
38
661
40686 solar-flare
315
1444 PizzaCutter3
38
1043
40690 threeOf9
10 512
2
1446 CostaMadrel
38
296
40693 xd6
10 973
1447 CastMetall
38
327
40705 tokyol
45
959
2
1448 KnuggetChase3
40
194
40706 parity5_pluc 5
11 1124
1451 PieChartl
38
705
40710 cleve
14 303
1453 PieChart3
38
1077
40711 cleveland-nominal
8 303
1488 parkinsons
23
195
40981 Australian
15 690
1490 planning- relax
13
182
41430 DiabeticMellitus
98
281
2
1495 qualitative- bannrupty
7
250
41538 conference attendance
246
1498 sa-heart
10
462
41919 CPMP-2015-runtime.
23
527
4
1499 seeds
8
210
41976 TuningS
156
1506 thoracic-surgery
17
470
42172 regime. aegime_alimenaire
20
202 2
1508 user-knowledge
6
403
42261 iris-example
150
1511 wholesale-customers
9
440
42544 Touch2
11
265
1512
heart-long-beace
14
200
42585 penguins
344
1520 robot-failures-p55
91
164
42638 titanic
891
Table 11: Validation dataset, continued
21


Reviews:
[{"review_id": "qln8G23j4b", "reviewer": "Reviewer_B59Z", "summary": "The authors propose a novel transformer based hypernetwork model that can be 'in-context' prompted with a supervised dataset and it can generate weights for a small neural network that can generalize well on this new task. Their approach borrows ideas from meta-learning, hypernetworks, transformers and distillation, and demonstrates a modern and effective way to achieve large speed ups when a task-specific task model is trained, while also retaining the generalization capability of a full training run. \n\nOverall I find the creativity, elegance and rigor demonstrated in this paper very refreshing and commendable. \n\nI have a bunch of questions, and no paper is without weakness, but I find the work a reasonable and worthwhile contribution to the field.", "questions": "1. **Scalability Constraints**: While MotherNet's performance on small datasets is impressive, it shares the quadratic memory limitations of standard transformers, restricting its use to smaller datasets. Have you considered modifying MotherNet to incorporate larger context or memory-efficient transformer architectures, such as those designed for long-context handling or reduced attention complexity? How might these adaptations impact its scalability and performance?\n\n2. **Limited Domain Exploration**: While the focus on tabular data showcases MotherNet’s strengths, its hypernetwork and in-context learning capabilities seem promising for other types of data, such as text or images. Do you see potential for adapting MotherNet to these modalities? If so, what adjustments would be necessary to leverage its in-context learning for different types of tasks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "**Strengths**:\n- **Creative Approach**: MotherNet is an inventive application of hypernetworks and transformer-based architectures, demonstrating how in-context learning can effectively generate task-specific models without gradient descent.\n- **Efficiency and Speed**: The method achieves significant inference speed improvements over TabPFN, offering practical advantages for use cases requiring fast, on-demand predictions.\n- **Eliminates Hyper-Parameter Tuning**: MotherNet operates without per-dataset hyper-parameter tuning or gradient-based training, simplifying the pipeline for real-world applications.\n- **Strong Empirical Evidence**: The paper provides a thorough comparison with a range of baselines, including TabPFN, HyperFast, and traditional models like XGBoost and Random Forests, illustrating MotherNet’s consistent performance across benchmarks.\n- **Open-Source Contribution**: The work includes an open-source implementation, supporting reproducibility and facilitating future research in the field.\n- **Thoughtful Methodology**: The architecture decisions, such as low-rank decomposition of weights, are well-documented and provide insights into balancing memory efficiency and model capability.\n- **Solid Trade-off Analysis**: The authors give a clear breakdown of the trade-offs in terms of training and inference time, presenting scenarios where MotherNet excels over other approaches.", "weaknesses": "- **Scalability Constraints**: MotherNet, like TabPFN, is bound by the quadratic memory requirements of transformers, limiting its usability for datasets larger than around 5,000 samples. This could be viewed as a major limitation for more extensive applications.\n- **Presentation Gaps**: Some sections, particularly in the methodology and results, could benefit from improved clarity and structure to enhance readability and understanding.\n- **Comparison Depth**: Although the evaluation includes multiple models, comparisons with newer, more diverse architectures would strengthen the paper’s impact and situate MotherNet more firmly in the broader ML landscape.\n\n\nAnd my favourite 'weakness'\n- **Limited Domain Exploration**: The paper focuses solely on tabular data, leaving questions about whether MotherNet’s advantages could extend to other data modalities like text or images. This work could be a neat contender for in-context learning paradigms across the board. Hypernetworks are an intriguing idea.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xyhFaGpHyD", "reviewer": "Reviewer_QK8H", "summary": "This paper shows that in-context tabular models can be trained to generate MLP weights directly. The resulting MLPs generally outperform standard trained MLPs and are competitive with other tabular predictive models. The paper focuses on small datasets due to limitations of the in-context modelling being used.", "questions": "- Since the model could handle 30,000 data points on GPU and 100,000 data points on CPU, why didn't you evaluate on larger datasets (or in the case of Tabzilla, with a larger training set)?\n\n- In general, how do you see this paper contributing to future research or practical applications? I think it's interesting that this sort of model is possible, but I'm having trouble seeing what the broader contributions are, especially because I'm not convinced the experimental results show a practical niche for the model as proposed.\n\n- What are the main contributions of this paper compared to Hyperfast specifically? The performance of the two models is only compared on very small datasets. But beyond that, I'd like to know how you'd compare this work to Hyperfast because a comparison isn't given in Section 2.\n\nMinor issues:  \n- On page 5, how did you get the number 25,738? I calculate the number of low-rank weights as 33,088 ($2hr+Nr$).\n- In Section 3.1, $r$ is used to represent a rank whereas in Section 3.2, it's used for the number of input features, which is a bit confusing.\n- On page 6, the text says that MotherNet outperforms MLP-Distill in terms of normalized AUC, but the results table only shows it outperforming in terms of raw AUC.\n- There are a few references to \"Table 4.1\" and \"Table 4.2\" that are incorrect.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The fact that this approach works well is surprising and compelling. It appears that training on synthetic datasets has a useful regularizing effect in generating MLPs compared to standard MLPs that are just trained on the dataset in question.\n\n- The presentation is generally very clear.", "weaknesses": "- The experimental results aren't very compelling on the whole. The CC-18 evaluation is limited to very small datasets and the actual differences in AUC between the top ten models are very small. Given the scale of the data and relative effectiveness of very simple models like logistic regression, it's tough to see the computational efficiency of the proposed model as actually mattering in practice, especially since it's using GPU hardware. The Tabzilla evaluation provides a wider range of dataset sizes, but in this regime, the proposed method performs about as well as an untuned XGBoost, and significantly worse than tuned GBDT models (in addition to TabPFN).\n\n- I think the use of ensembling also weakens the experimental results. While TabPFN did this in their paper, it's generally the case that ensembling any stochastically trained model can improve performance, and I think that using ensembling on some methods and not others isn't a fair comparison.\n\n- I'm not confident about the contributions of this paper beyond an interesting result - I'd like to hear more from the authors on that (see below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wQhZN6APX0", "reviewer": "Reviewer_DaxP", "summary": "This study proposes a classification hypernetwork for tabular data, called MotherNet. A hypernetwork is a \"parent\" network that, given training data, produces the weights for another \"child\" network that is immediately ready to make predictions on the test data. This allows completely avoiding the traditional gradient-based training and hyperparameter tuning.\n\nTo learn how to produce good child networks, MotherNet itself is trained on synthetic data for a long time (28 days on one GPU A100). The synthetic data generation is inherited from the TabPFN project. The parent network is a 12-layer Transformer, same as TabPFN. The child network is a lightweight 2-layer MLP. Since the actual predictions are made by the child network, it means that the inference efficiency is high.\n\nOn datasets of size <5000 objects, a competitive performance of MotherNet is reported.", "questions": "-", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "I like the overall idea of the project. This paper shows an interesting new path with certain benefits over the original TabPFN, including the high inference efficiency.\n\n*(Assuming that the authors will share the code and the model checkpoints linked in the paper)*\nTo me, a big positive thing is the provided code for training MotherNet and the model weights. Training hypernetworks is non-trivial and costly, and I like that this project gives the community a ready-to-use hypernetwork baseline.\n\nContinuing the previous point, the proposed model seem to outperform HyperFast, one of the previous (and the only?) tabular hypernetwork. So MotherNet advances the niche of tabular hypernetworks, and gives a better starting point for future work in this direction.\n\nI also appreciate that the paper explicitly communicates the scope of dataset sizes (<5000 objects), as well as the analysis of some failure cases in Appendix.", "weaknesses": "**Benchmarks**\n\nI have mixed feelings about the benchmarks. I tend to agree that MotherNet outperforms HyperFast, however, the overall ranking of models is less convincing to me. I can imagine that the relative performance of the methods will change significantly in a different setup, and especially on different datasets.\n\nRegarding datasets, if I understand correctly, there are three parts of results:\n\n- (Figure 2 and Table 1) The performance on datasets with unusual ranking of models, in particular with the high performance of a linear model, as directly noted in the paper on L357. It is unclear how reliable are the conclusions made on these datasets.\n- (Figure 6) The performance on the \"validation\" datasets. My understanding is that these results are somewhat additional, and thus presented in Appendix, not in the main text. I should admit I do not fully understand the role of the validation datasets, so I may be missing something.\n- (Table 2, Figure 3, Table 6) The results on the TabZilla benchmark. There are also certain unusual results, e.g. the linear model outperforming MLP, or SVM outperforming MLP-rtdl.\n\nRegarding hyperparameters:\n- I noticed that the number of training epochs for MLP and ResNet is tuned within the grid (10, 100, 1000). I can imagine that on some datasets all three values are suboptimal, e.g. 10 can lead to underfitting, and 100 and 1000 can lead to overfitting.\n- If the \"choice\" distritbution for HyperOpt is a categorical distribution that is order-unaware, then this can be misleading for the hyperparameter tuning engine (hard to tell to what extent).\n\n**Related work**\n\nSection 2 (related work) does not cover a whole family of methods that, I think, is highly relevant. However, this seems to be easy to fix, plus, some of the methods are already used as baselines. Namely, I imply the traditional machine learning models that should be trained from scratch for each task. That includes classic ML algorithms (linear models, tree-based methods, etc.) and DL architectures (there are too many of them to list; I suggest taking any popular baseline, e.g. FT-Transformer from the same paper as the already used ResNet, and traverse the citation graph backwards and forwards).\n\n**Analysis**\n\nIn my opinion, the analysis of the MotherNet's performance should be extended. I suggest the following experiments:\n\n(1) Finetuning the child network produced by MotherNet. Even if it is not how MotherNet is supposed to be used in practice, it would show the full potential of the MotherNet system. If this experiment is not presented in this paper, then it can become a somewhat must have experiment for future researchers willing to use MotherNet as a baseline. I recommend conducting this experiment directly in this submission.\n\n(2) Extensive hyperparameter tuning (learning rate, weight decay, dropout, etc.) for a plain MLP of the *exactly* same architecture as the child network. Otherwise, it is unclear how optimal are the weights produced by MotherNet. For example, if the traditional tune-and-train approach gives significantly better results for the child architecture, then it can be the preferable approach when the task performance is more important than the (almost) zero cost of the MotherNet's forward pass (plus, training on small datasets is cheap).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YnyuuGLDx0", "reviewer": "Reviewer_DRbs", "summary": "MotherNet performs in-context learning on tabular data via generating a feed-forward network from a pre-trained transformer to perform prediction. The work is a synthesis of ideas from TabPFN and meta-learning/hypernetworks. MotherNet’s efficiency stems from requiring only a forward pass on the pre-trained model and the simplicity of the feed-forward MLP. Experiments are conducted on the OpenML CC-18 and TabZilla benchmarks.", "questions": "I strongly encourage the authors to restructure Section 4 and rearrange the results. Also, a stronger argument is necessary to show why MotherNet is better than tree-based methods and TabPFN.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The pipeline is efficient during training/fitting and inference. \n2. Empirical evaluations include an extensive set of baselines, including distillation methods, and two frequently-used benchmarks for tabular data.", "weaknesses": "1. MotherNet is not a competitive method based on the average rank numbers. It is outperformed on CC-18 by MLP-Distill and TabPFN. Such is the case for TabZilla, but the error intervals are too wide for any conclusions to be drawn. This experiment is not effective. Pre-training time is not rigorously compared as part of the method’s cost.\n2. The paper itself is of subpar quality, especially in the presentation of results. There are too many references to appendix figures and tables, which should be discouraged and used sparingly. Paradoxically, the paper does not even take up the full page limit of 10 pages. There are errors in references (e.g. line 405 - Table 4.2). Understanding the results is very difficult. The critical difference diagrams are not explained. \n3. Novelty is limited as the method is heavily influenced by TabPFN. It could be seen as an extension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)