MotherNet: Fast Training and Inference via Hyper-Network Transformers

Authors: Andreas C Mueller, Carlo A Curino, Raghu Ramakrishnan
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: hypernetwork, tabular data, meta-learning, foundational models

Abstract:
Foundation models are transforming machine learning across many modalities, with in-context learning replacing classical model training. Recent work on tabular data hints at a similar opportunity to build foundation models for classification for numerical data. However, existing meta-learning approaches can not compete with tree-based methods in terms of inference time. In this paper, we propose MotherNet, a hypernetwork architecture trained on synthetic classification tasks that, once prompted with a never-seen-before training set generates the weights of a trained ``child'' neural-network by in-context learning using a single forward pass. In contrast to most existing hypernetworks that are usually trained for relatively constrained multi-task settings, MotherNet can create models for multiclass classification on arbitrary tabular datasets without any dataset specific gradient descent.
The child network generated by MotherNet outperforms neural networks trained using gradient descent on small datasets, and is competitive with predictions by TabPFN and standard ML methods like Gradient Boosting. Unlike a direct application of TabPFN, MotherNet generated networks are highly efficient at inference time.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730412601923

thread_type:
official_review

<thread_object>
creator: reviewer_drbs

<content>
summary:
MotherNet performs in-context learning on tabular data via generating a feed-forward network from a pre-trained transformer to perform prediction. The work is a synthesis of ideas from TabPFN and meta-learning/hypernetworks. MotherNet’s efficiency stems from requiring only a forward pass on the pre-trained model and the simplicity of the feed-forward MLP. Experiments are conducted on the OpenML CC-18 and TabZilla benchmarks.

strengths:
1. The pipeline is efficient during training/fitting and inference. 
2. Empirical evaluations include an extensive set of baselines, including distillation methods, and two frequently-used benchmarks for tabular data.

weaknesses:
1. MotherNet is not a competitive method based on the average rank numbers. It is outperformed on CC-18 by MLP-Distill and TabPFN. Such is the case for TabZilla, but the error intervals are too wide for any conclusions to be drawn. This experiment is not effective. Pre-training time is not rigorously compared as part of the method’s cost.
2. The paper itself is of subpar quality, especially in the presentation of results. There are too many references to appendix figures and tables, which should be discouraged and used sparingly. Paradoxically, the paper does not even take up the full page limit of 10 pages. There are errors in references (e.g. line 405 - Table 4.2). Understanding the results is very difficult. The critical difference diagrams are not explained. 
3. Novelty is limited as the method is heavily influenced by TabPFN. It could be seen as an extension.

questions:
I strongly encourage the authors to restructure Section 4 and rearrange the results. Also, a stronger argument is necessary to show why MotherNet is better than tree-based methods and TabPFN.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730640237536

thread_type:
official_review

<thread_object>
creator: reviewer_daxp

<content>
summary:
This study proposes a classification hypernetwork for tabular data, called MotherNet. A hypernetwork is a "parent" network that, given training data, produces the weights for another "child" network that is immediately ready to make predictions on the test data. This allows completely avoiding the traditional gradient-based training and hyperparameter tuning.

To learn how to produce good child networks, MotherNet itself is trained on synthetic data for a long time (28 days on one GPU A100). The synthetic data generation is inherited from the TabPFN project. The parent network is a 12-layer Transformer, same as TabPFN. The child network is a lightweight 2-layer MLP. Since the actual predictions are made by the child network, it means that the inference efficiency is high.

On datasets of size <5000 objects, a competitive performance of MotherNet is reported.

strengths:
I like the overall idea of the project. This paper shows an interesting new path with certain benefits over the original TabPFN, including the high inference efficiency.

*(Assuming that the authors will share the code and the model checkpoints linked in the paper)*
To me, a big positive thing is the provided code for training MotherNet and the model weights. Training hypernetworks is non-trivial and costly, and I like that this project gives the community a ready-to-use hypernetwork baseline.

Continuing the previous point, the proposed model seem to outperform HyperFast, one of the previous (and the only?) tabular hypernetwork. So MotherNet advances the niche of tabular hypernetworks, and gives a better starting point for future work in this direction.

I also appreciate that the paper explicitly communicates the scope of dataset sizes (<5000 objects), as well as the analysis of some failure cases in Appendix.

weaknesses:
**Benchmarks**

I have mixed feelings about the benchmarks. I tend to agree that MotherNet outperforms HyperFast, however, the overall ranking of models is less convincing to me. I can imagine that the relative performance of the methods will change significantly in a different setup, and especially on different datasets.

Regarding datasets, if I understand correctly, there are three parts of results:

- (Figure 2 and Table 1) The performance on datasets with unusual ranking of models, in particular with the high performance of a linear model, as directly noted in the paper on L357. It is unclear how reliable are the conclusions made on these datasets.
- (Figure 6) The performance on the "validation" datasets. My understanding is that these results are somewhat additional, and thus presented in Appendix, not in the main text. I should admit I do not fully understand the role of the validation datasets, so I may be missing something.
- (Table 2, Figure 3, Table 6) The results on the TabZilla benchmark. There are also certain unusual results, e.g. the linear model outperforming MLP, or SVM outperforming MLP-rtdl.

Regarding hyperparameters:
- I noticed that the number of training epochs for MLP and ResNet is tuned within the grid (10, 100, 1000). I can imagine that on some datasets all three values are suboptimal, e.g. 10 can lead to underfitting, and 100 and 1000 can lead to overfitting.
- If the "choice" distritbution for HyperOpt is a categorical distribution that is order-unaware, then this can be misleading for the hyperparameter tuning engine (hard to tell to what extent).

**Related work**

Section 2 (related work) does not cover a whole family of methods that, I think, is highly relevant. However, this seems to be easy to fix, plus, some of the methods are already used as baselines. Namely, I imply the traditional machine learning models that should be trained from scratch for each task. That includes classic ML algorithms (linear models, tree-based methods, etc.) and DL architectures (there are too many of them to list; I suggest taking any popular baseline, e.g. FT-Transformer from the same paper as the already used ResNet, and traverse the citation graph backwards and forwards).

**Analysis**

In my opinion, the analysis of the MotherNet's performance should be extended. I suggest the following experiments:

(1) Finetuning the child network produced by MotherNet. Even if it is not how MotherNet is supposed to be used in practice, it would show the full potential of the MotherNet system. If this experiment is not presented in this paper, then it can become a somewhat must have experiment for future researchers willing to use MotherNet as a baseline. I recommend conducting this experiment directly in this submission.

(2) Extensive hyperparameter tuning (learning rate, weight decay, dropout, etc.) for a plain MLP of the *exactly* same architecture as the child network. Otherwise, it is unclear how optimal are the weights produced by MotherNet. For example, if the traditional tune-and-train approach gives significantly better results for the child architecture, then it can be the preferable approach when the task performance is more important than the (almost) zero cost of the MotherNet's forward pass (plus, training on small datasets is cheap).

questions:
-
</content>
</thread_object>

</thread>
<thread>
creation date:
1730673133976

thread_type:
official_review

<thread_object>
creator: reviewer_qk8h

<content>
summary:
This paper shows that in-context tabular models can be trained to generate MLP weights directly. The resulting MLPs generally outperform standard trained MLPs and are competitive with other tabular predictive models. The paper focuses on small datasets due to limitations of the in-context modelling being used.

strengths:
- The fact that this approach works well is surprising and compelling. It appears that training on synthetic datasets has a useful regularizing effect in generating MLPs compared to standard MLPs that are just trained on the dataset in question.

- The presentation is generally very clear.

weaknesses:
- The experimental results aren't very compelling on the whole. The CC-18 evaluation is limited to very small datasets and the actual differences in AUC between the top ten models are very small. Given the scale of the data and relative effectiveness of very simple models like logistic regression, it's tough to see the computational efficiency of the proposed model as actually mattering in practice, especially since it's using GPU hardware. The Tabzilla evaluation provides a wider range of dataset sizes, but in this regime, the proposed method performs about as well as an untuned XGBoost, and significantly worse than tuned GBDT models (in addition to TabPFN).

- I think the use of ensembling also weakens the experimental results. While TabPFN did this in their paper, it's generally the case that ensembling any stochastically trained model can improve performance, and I think that using ensembling on some methods and not others isn't a fair comparison.

- I'm not confident about the contributions of this paper beyond an interesting result - I'd like to hear more from the authors on that (see below).

questions:
- Since the model could handle 30,000 data points on GPU and 100,000 data points on CPU, why didn't you evaluate on larger datasets (or in the case of Tabzilla, with a larger training set)?

- In general, how do you see this paper contributing to future research or practical applications? I think it's interesting that this sort of model is possible, but I'm having trouble seeing what the broader contributions are, especially because I'm not convinced the experimental results show a practical niche for the model as proposed.

- What are the main contributions of this paper compared to Hyperfast specifically? The performance of the two models is only compared on very small datasets. But beyond that, I'd like to know how you'd compare this work to Hyperfast because a comparison isn't given in Section 2.

Minor issues:  
- On page 5, how did you get the number 25,738? I calculate the number of low-rank weights as 33,088 ($2hr+Nr$).
- In Section 3.1, $r$ is used to represent a rank whereas in Section 3.2, it's used for the number of input features, which is a bit confusing.
- On page 6, the text says that MotherNet outperforms MLP-Distill in terms of normalized AUC, but the results table only shows it outperforming in terms of raw AUC.
- There are a few references to "Table 4.1" and "Table 4.2" that are incorrect.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674455246

thread_type:
official_review

<thread_object>
creator: reviewer_b59z

<content>
summary:
The authors propose a novel transformer based hypernetwork model that can be 'in-context' prompted with a supervised dataset and it can generate weights for a small neural network that can generalize well on this new task. Their approach borrows ideas from meta-learning, hypernetworks, transformers and distillation, and demonstrates a modern and effective way to achieve large speed ups when a task-specific task model is trained, while also retaining the generalization capability of a full training run. 

Overall I find the creativity, elegance and rigor demonstrated in this paper very refreshing and commendable. 

I have a bunch of questions, and no paper is without weakness, but I find the work a reasonable and worthwhile contribution to the field.

strengths:
**Strengths**:
- **Creative Approach**: MotherNet is an inventive application of hypernetworks and transformer-based architectures, demonstrating how in-context learning can effectively generate task-specific models without gradient descent.
- **Efficiency and Speed**: The method achieves significant inference speed improvements over TabPFN, offering practical advantages for use cases requiring fast, on-demand predictions.
- **Eliminates Hyper-Parameter Tuning**: MotherNet operates without per-dataset hyper-parameter tuning or gradient-based training, simplifying the pipeline for real-world applications.
- **Strong Empirical Evidence**: The paper provides a thorough comparison with a range of baselines, including TabPFN, HyperFast, and traditional models like XGBoost and Random Forests, illustrating MotherNet’s consistent performance across benchmarks.
- **Open-Source Contribution**: The work includes an open-source implementation, supporting reproducibility and facilitating future research in the field.
- **Thoughtful Methodology**: The architecture decisions, such as low-rank decomposition of weights, are well-documented and provide insights into balancing memory efficiency and model capability.
- **Solid Trade-off Analysis**: The authors give a clear breakdown of the trade-offs in terms of training and inference time, presenting scenarios where MotherNet excels over other approaches.

weaknesses:
- **Scalability Constraints**: MotherNet, like TabPFN, is bound by the quadratic memory requirements of transformers, limiting its usability for datasets larger than around 5,000 samples. This could be viewed as a major limitation for more extensive applications.
- **Presentation Gaps**: Some sections, particularly in the methodology and results, could benefit from improved clarity and structure to enhance readability and understanding.
- **Comparison Depth**: Although the evaluation includes multiple models, comparisons with newer, more diverse architectures would strengthen the paper’s impact and situate MotherNet more firmly in the broader ML landscape.


And my favourite 'weakness'
- **Limited Domain Exploration**: The paper focuses solely on tabular data, leaving questions about whether MotherNet’s advantages could extend to other data modalities like text or images. This work could be a neat contender for in-context learning paradigms across the board. Hypernetworks are an intriguing idea.

questions:
1. **Scalability Constraints**: While MotherNet's performance on small datasets is impressive, it shares the quadratic memory limitations of standard transformers, restricting its use to smaller datasets. Have you considered modifying MotherNet to incorporate larger context or memory-efficient transformer architectures, such as those designed for long-context handling or reduced attention complexity? How might these adaptations impact its scalability and performance?

2. **Limited Domain Exploration**: While the focus on tabular data showcases MotherNet’s strengths, its hypernetwork and in-context learning capabilities seem promising for other types of data, such as text or images. Do you see potential for adapting MotherNet to these modalities? If so, what adjustments would be necessary to leverage its in-context learning for different types of tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732165134158

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Response

comment:
We want to thank all the reviewers for their valuable feedback, which will help us improve the submission.
We want to call out that many of the reviewers find the work interesting:
- Overall I find the creativity, elegance and rigor demonstrated in this paper very refreshing and commendable. (B59Z)
- The fact that this approach works well is surprising and compelling. (QK8H)
- This paper shows an interesting new path with certain benefits over the original TabPFN, including the high inference efficiency. (DaxP)

We hope this method can be an interesting contribution, addressing one of the weaknesses of TabPFN, demonstrating how in-context learning can outperform gradient descent training and outperforming the similar HyperNet architecture without necessarily toping all benchmark lists. The goal of this work is not to outperform TabPFN, but to illustrate a possible path to overcoming one of its weaknesses, and proof the feasibility for pure in-context learning for tabular data.

We agree that the empirical results do not show a very clear ranking of models (apart from TabPFN leading). However, we disagree that this is a weakness of our work. We used two established benchmarks from the community, and performed a rigorous meta-validation and meta-test set split that is common in the AutoML community but unfortunately not broadly adopted in the wider ML community. In fact, it is hard to find two published benchmarks that agree on algorithm rankings, and by including two, we are aiming at providing a broader picture. We are happy to consider further benchmarks suits if the reviewers have other suggestions; however, two is double the amount of most papers that we are familiar with. Two reviewers asked for a broader comparison with other deep learning methods; we want to point out that the TabZilla benchmark contains a wide variety of recent deep learning methods, and a comparison is shown in table 2.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a hypernetwork for in-context learning on tabular datasets, inspired by TabPFN and meta-learning paradigms. MotherNet generates task-specific models via a pre-trained transformer, allowing for inference without gradient descent or hyperparameter tuning. The experiments focus on small datasets (<5,000 samples) and demonstrate competitive performance against strong baselines, such as TabPFN and HyperFast, while achieving significant inference speed gains. The accuracy of MotherNet is lower than that of TabPFN and tuned GBDT models. The benchmarks, while rigorous, reveal narrow settings where the model is practical at the moment. There are also some concerns about technical novelty compared to TabPFN. Overall 3 of the reviewers were positive about the method and in particular considered that the direction is promising with a sufficiently strong initial demonstration for a challenging area.