DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weights
Authors: Saumya Gupta
Scott Biggs
Moritz Laber
Zohair Shafi
Robin Walters
Ayan Paul
Venue: ICLR 2026 Poster (2026)
Keywords: metanetworks
deep weight space
parameter space symmetry
flow matching
canonicalization
Primary area: generative models

Abstract: Building efficient and effective generative models for neural network weights has been a research focus of significant interest that faces challenges posed by the high-dimensional weight spaces of modern neural networks and their symmetries. Several prior generative models are limited to generating partial neural network weights, particularly for larger models, such as ResNet and ViT. Those that do generate complete weights struggle with generation speed or require finetuning of the generated models. In this work, we present \ours, a Flow Matching model that operates directly in weight space to generate diverse and high-accuracy neural network weights for a variety of architectures, neural network sizes, and data modalities. The neural networks generated by \ours do not require fine-tuning to perform well and can scale to large networks. We apply Git Re-Basin and TransFusion for neural network canonicalization in the context of generative weight models to account for the impact of neural network permutation symmetries and to improve generation efficiency for larger model sizes. The generated networks excel at transfer learning, and ensembles of hundreds of neural networks can be generated in minutes, far exceeding the efficiency of diffusion-based methods. DeepWeightFlow models pave the way for more efficient and scalable generation of diverse sets of neural networks.

Published as a conference paper at ICLR 2026
DEEPWEIGHTFLOW: RE-BASINED FLOW MATCHING FOR GENERATING NEURAL NETWORK WEIGHTS
Saumya Gupta, Scott Biggs, Maritz Laber, Zohair Shafi, and Ayan Paul
Northeastern University
Boston MA 02119, USA {gupta. saumy, bi biggs. laber.m. sha fi.e,i.2,, r. paalters a.w.aaterraaaaaaaaall paaoosstatta.
ABSTRACT
Building efficient and effective generative models for neural network weights has been a research focus of significant interest that faces challenges posed by the digh-dimensional weight spaces of modern neural networks and their symmetries. Several prior generative models are limited to generating partial neural network weights, particularly for larger models, such as ResNet and ViT. Those that do generate complete weights struggle with generation speed require finetuning of the generated models. In this work, we present Deep WeightFlow, Flow Match- ing model that operates directly in weight space to generate diverse and high- accuracy neural network weights for a variety of architectures, neural network sizes, and data modalities. The neural networks generated by DeepWeightFlow do not require fine-tuning to perform well and can scale to large networks. We apply Git Re-Basin and TransFusion for neural network canonicalization in the context of generative weight models to account for the impact of neural network permutation symmetries and to improve generation efficiency for larger model sizes. The generated networks excel at transfer learning, and ensembles of hun- dreds of neural networks can be generated in minutes, far exceeding the efficiency of diffusion- based methods. DeepWeightFlow models pave the way for more ef- ficient and scalable generation of diverse sets neural networks.
INTRODUCTION
Generating neural network weights is sampling challenge that explores the underlying high- dimensional distribution of weights, where neural networks trained on similar datasets and tasks exhibit statistical regularities. The development of generative models capable of learning the dis- tributional properties of trained weights faces challenges of symmetries and high-dimensionality of the weight spaces. Treating large collections of neural network weights as structured and high-dimensional data modality promises advances in model editing (Mitchell et al., 2022: Meng et al., 2022), accelerating transfer learning (Knyazev et al., 2021; Schürholt et al., 2022), facilitat- ing uncertainty quantification (Lakshninarayanan al., 2017), and advancing neural architecture search (Chen et al., 2019; Chen, 2023). Unlike traditional machine learning tasks that aim to opti- mize weights for specific downstream tasks, this concept advocates sampling from the weight space itself. In this work, we focus on the efficient generation of complete neural network weights that can achieve high performance for given task and excel transfer learning thus addressing funda- mental limitations in current deep learning workflows, such as computational bottlenecks in iterative training. vulnerability to adversarial attacks (Goodfellow al., 2015; Madry al., 2018) and pri- vacy concerns arising from training data reconstructions (Nasr et al., 2019; Tramer et al., 2022).
Generating neural network weights faces three main challenges: Firstly, neural network weights have a rich class of symmetries (Hecht-Nielsen, 1990; Entezari et al., 2022; Navon et al., 2023; Zhao et al., 2025), i.c., transformations of the weights that leave the neural network functionally invariant. Most prominently, joint permutations of hidden neurons in adjacent layers of multi-layer percep- trons (MLP) do not change the encoded function. Other architectural choices, such as incorporating attention heads or the choice of non- non-linear activation, induce additional symmetries. Techniques for dealing with weight space symmetries fall into three main categories: (1) data augmentation, (2) equivariant architectures, and (3) canonicalization. Prior work, such Wortsman et al. (2022);

---

Published as a conference paper at ICLR 2026
DeepWeightFlow Pipeline
a) Training data
b) Canonicalization (oprional)
<) Weight generation
Target networks
[(x W Wx)
W: = P.W.PT
(W 1:
W -
Flow
Matching
Pg
W W; W.
W1
W2
WE
W2
W
Schematic depiction of DeepWeightFlow: a) We construct a training dataset of weights by fully training neural networks with weights WI,..., WL on a given target task. b) Optionally, we use canonical- ization, i.e. choosing a canonical representative W. from the sume urbit as W. to break the permutation symmetry in purumeter space. c) We truin a flow model pa for efficient generation of higg--performunce weights (W1...., W1)= ~ far the target task.
Figure 1:
Wang et al. (2024); Soro et al. (2025); Saragih et al. (2025a), does not actively account for symme- tries in their generative models, while others, such as Saragih et al. (2025b), use equivariant architec- tures. Data augmentation has also been explored in weight representation learning (Schürholt al., 2024; Shamsian et al., 2023; 2024), and to a lesser extent in weight generation (Schürholt et al., 2024; Wang et al., 2025). Finally, canonicalization has recently found application in weight space learning (Schürholt et al., 2024; Wang et al., 2024; 2025), borrowing ideas from model merging and alignment (Ainsworth al., 2023; Rinaldi et al., 2025). Secondly, neural network weights are sigh-dimensional, varying from tens of millions for small ResNet (He et al., 2016) to hundreds of billions for modern large language models (Touvron et al., 2023; Guo et al., 2025). This challenge is often addressed by non-linear, dimensionality reduction techniques, including variational autoen- coders (VAEs) (Soro et al., 2025) and graph autoencoders (Schürholt et al., 2022: Saragih et al., 2025b; Soro al., 2025). Despite increasing efficiency, dimensionality reduction requires training an additional model for dimensionality reduction and can be detrimental to the quality of the gener- ated weights if the compression is lossy. Lastly, generative models proposed recently either generate partial weights for large models, or require finetuning post-generation, or have long generation time per sample, making them impractical.
To address these challenges, we propose DeepWeightFlow, a method for efficient generation of high- performance neural network weights via Flow Matching (FM) and apply it to MLP for vision and labular data, as well as ResNet (He et al., 2016), and ViT (Dosovitskiy et al., 2021) for computer vision tasks, and BERT for natural language processing (NLP) (Devlin al., 2019). We rely on canonicalization techniques, such as Git Re-Basin (Ainsworth et al., 2023) and TransFusion (Ri- naldi et al., 2025), to resolve parameter permutation symmetries, and show that canonicalization aids weight generation for large neural networks but offers limited benefits when the weight space dimension is moderate. We show that neural networks generated by DeepWeightFlow excel at the target task and are competitive with state-of-the-arr weight generation methods such as RPG (Wang et al., 2025), D2NWG (Soro et al., 2025), FLoWN (Saragih et al., 2025b), and P-diff (Wang et al., 2024) while overcoming several of the limitations of these models. A schematic of our methods is shown in Figure 1. While Deigheigh samples directly from weight spaces, we show that the models can scale to generating larger networks using PCA while keeping the training and the generation time low. In summary, the contributions of this work are as follows:
DeepWeightFlow is a new method for complete neural network weight generation based on FM, unconditioned by dataset characteristics, task descriptions, or architectural specifications. DeepWeightFlow does not require additional training of autoencoders for dimensionality re- duction and can scale to high- -igghhiiensional weight spaces using PCA.
We show that our method can generate weights for neural networks with O(100M) parame- ters, and diverse architectures, such as MLP, ResNet, ViT, and BERT that, without fine-tuning, exhibit high performance tasks in the vision, tabular, and natural language domains.
We empirically elucidate the role of parameter symmetry for weight generation, showing that canonicalization of the training data aids the generation of very high-dimensional weights but offers no additional benefit for weights of modest dimension.
ReepWeightFlow with a simple MLP implementation, and without any equivariant architec- ture, is far more efficient in generating diverse samples compared to diffusion-based models.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
HyperNetworks:
Early explorations of neural network generation focus on HyperNetworks, which learn neural network parameters as a relaxed temporal weight sharing process (Ha et al., 2017). 7). Hy- perNetworks have been applied generating weights through density sampling. GAN, and diffusion methods by learning latent representations of neural network weights (Ha et al., 2017; Frankle & Carbin, 2019; Ratzlaff & Fuxin, 2019; Schürholt al., 2022: Kiani al., 2024). They have also been used to build meta-learners augmentations or substitutes for Stochastic Gradient Descent op- timization, which condition generation of new weight checkpoints on prior weights and task losses (Peebles et al., 2022; Zhang et al., 2024; Wang et al., 2025).
Generative Models for Neural Network Weights: certusion-based generative models for weights have been successful at neural network weight generation, but often do not directly resolve weight space symmetries. These approaches either provide no treatment (Wang et al., 2024), or rely on Variational Auto Encoding (VAE) methods to concurrently resolve weight symmetries and reduce the dimensionality of the generative task (Ha et al., 2017; Frankle & Carbin, 2019; Schürholt et al., 2022: Kiani et al., 2024; Soro et al., 2025). In contrast, weight canonicalization is done pre- training step in SANE (Schürholt et al., 2024), which uses kernel density sampling of hypernetwork latents to autoregressively populate models layer-wise, allowing for complete weight generation, but requires fine-tuning, unlike Deep BeepWeightFlow, Diffusion has been applied directly to generating par- tial (Wang al., 2024) complete weights (Soro al., 2025; Wang et al., 2025). RPG (Wang al., 2025) generates complete weights by using a recurrent diffusion model. However, RPG shows long generation times, often taking hours to generate set of networks that DeepWeightFlow takes min- utes to complete. Subsequent Conditional Flow Matching (CFM) methods (Saragih et al., 2025b;a) explore dataset embeddings as conditioning for transfer learning and weight generation. These CFMs also report using VAE methods to reduce the dimensionality of the generative task and to resolve weight symmetries (Saragih et al., 2025b;a). We develop this further with DeepWeight which operates directly in deep weight space to generate complete weight sets, and demonstrates the viability of PCA as a strategy for surpassing ((100M) parameter sets.
Permutation Symmetries in Weight Space: SANE (Schürholt et al., 2024) applies Git Re-Basin as a canonicalization for hypernetwork training (Schürholt et al., 2022; 2024; Ainsworth et al., 2023). Unlike DeepWeightFlow, SANE tokenizes weights layer-wise and autoregressively samples them to populate new neural models. RPG (Wang et al., 2025) uses a different strategy to address per- mutation symmetry by one-hot encoding models to differentiate between potential permutations of similar weights. D2NWG (Soro et al., 2025) and FLoWN (Saragih et al., 2025b) both evalu- VAEs, while FLoWN additionally considers permutation invariant graph autoencoding methods to appeal to the manifold and lottery ticket hypotheses (Ha et al., 2017; Frankle & Carbin, 2019; Schürholt et al., 2022; Kiani al., 2024). DeepWeightFlow extends the canonicalization methods from previous works to transformers through TransFusion, and thoroughly evaluates the impact of canonicalization on generating complete weight sets (Schürholt et al., 2024; Wang al., 2024; 2025; Soro al., 2025).
BACKGROUND
Deep FeepWeigttFlow is an FM model using an MLP architecture trained on canonicalized neural net- works. In this section, give brief overview of the various methods we use to build it.
3.1 FLOW MATCHING
Flow Matching (Lipman al., 2023) is generative technique for learning vector field to trans- port a noise vector to a target distribution. Given an unknown data distribution q(x), we define a probability path pt for & E [0, 1] with Po (N(0, 1) and p1 R q(x). FM learns vector field with parameters 0, ua (z, t), that transports to p1 by minimizing
(i) F2 =x1). [1lvo () ((,,,))1)))
where u(x t) is the true vector field generating (x). and [0, 1] denotes the uniform distribution on the unit interval [0, 1). This loss is minimized if uo matches u, effectively following the probability
3

---

Published as a conference paper at ICLR 2026
path from Po P1. FM offers several advantages over diffusion for neural network weight gener- ation as it enables simpler and faster sampling, relies on direct vector field regression for training, and scales efficiently to high- -ighesiienni spaces, making it particularly well-suited for generating complete neural network weights.
3.2 PERMUTATION SYMMETRIES OF NEURAL NETWORKS AND RE-BASIN
Permutation symmetry common weight space symmetry in neural networks (Hecht-Nielsen, 1990). Consider the activations žt E R& at the th layer of simple MLP, with weights We E R.de+1xde xde, biases by E TR, and activation a, 2441 a (Weze + be), where 2g x is the input data. Applying permutation matrix P E R x of appropriate dimension, yields
2++1 Pz PW by) P'o(PWezz Pbe),
where PTP = I. This shows that a permutation of the output features of the 8th layer, when met with the appropriate permutation of the input features of the next layer - 1, will leave the overall MLP functionally invariant (Ainsworth et al., 2023).
Similar permutation symmetries (Lim et al., 2024) exist for the channels of convolutional neural networks and the attention heads of the transformer architecture (Hecht-Nielsen, 1990; Ainsworth et al., 2023; Rinaldi et al., 2025). These symmetries shape the loss landscape (Pittorino et al,, 2022), impacting optimization(Neessabbn et al., 2015a; Liu, 2023; Zhao al., 2024), generaliza- tion(Neyshabur et al., 2015b; Dinh et al., 2017), and model complexity (Zhao et al., 2025). They also impact the ability of generative models learn distributions over neural network weights. Per- mutation symmetry gives rise to highly multi- modii-modal loss surface that renders the resulting models equivalent in task performance (Hecht-Nielsen 1990; Lim al., 2024).
In model alignment, weights are aligned with respect to reference model to produce unique "canon- ical" representations for each equivalence class of the weight permutation symmetry. The Git Re- Basin (Ainsworth et al., 2023) weight matching approach permutes the hidden units of an MLP such that the inner product between reference and permuted weights is maximized. The resulting optimization problem is sum of bilinear assignment problems (SOBLAP). Git Re-Basin solves this problem approximately, using coordinate descent, reducing each layer's 's subproblem to linear assignment and iterating until convergence. TransFusion (Rinaldi et al., 2025) extends this idea of weight alignment to transformers where permutation symmetries exist both in MLPs and within and between attention heads, applying iterative alignment steps reconcile permutations of heads and hidden units. More details on this can be found in Appendix A and Appendix B.
4 METHODS
We implement simple MLP-based FM model. The explicit encoding of the symmetries of the neu- ral networks is done using TransFusion for transformers and Git Re-Basin for all other architectures.
Flow Matching Architecture and Training: DeepWeightFlow uses time-conditioned neural net- work that predicts a velocity vector along a trajectory between source and target network weights. The source is a distribution of Gaussian noise given by 2o ~ (0, o²1), and the target is a distribu- tion of trained weights (æ1 ~ Ptarge)). The source distribution has the same dimensions as the target. Given sampled time t [0, (uniformly distributed), an interpolated point along the straight-line trajectory is computed as μl = (1 - t)zo + tz To stabilize training, stochastic points are generated by adding Gaussian noise 2; 14
with (O0 I2.). The instantaneous target velocity along this linear trajectory is ut 21 zo (since Aut
dt 21 -zp), which is constant along the straight-line path. The network sees as imput, while u, is derived from the endpoints (æo, LT). The scalar time is embedded into a higher-dimenssonal vector lembed MLP((t) Ru where dime varies depending on the complexity of the model for which we are training DeepWeightFlow We use a shallow MLP with layer normalization, dropout regularization, and GELU activations. This fembed is concatenated with n, and fed into the main network, allowing the network to condition on time in a learnable, flexible manner. The network maps (Iz, fembed) + ug t). where ee is the leamned vector field. The main network consists of fully connected layers with LayerNorm, GELU activations, and Dropout, ending with a linear layer mapping back to the flattened weight dimen- sion. Finally, new weight configurations are generated by integrating the learned vector field from
4

---

Published as a conference paper at ICLR 2026
random Gaussian inputs in the same flattened weight space the source distribution. This inte- gration is performed using a fourth-order Runge- Kutta (RK4) method, which ensures high-aceuracy trajectories in weight space. Concretely, each integration step, the vector field is evaluated at the current point and time, and RK4 increments are computed to update the weights. This procedure allows sampling of realistic neural network weight configurations that smoothly interpolate between source and target distributions.
Canonicalization: We apply canonicalization to align the training set to single reference, as neu- ral network loss landscapes are inherently degenerate due to permutation symmetries in the weight space. This simplifies the learning process without the need for complex equivariant architectures. To implement canonicalization for smaller MLPs and ResNets, we use the weight-matching proce- dure of Git Re-Basin (Ainsworth et al., 2023) for 100 iterations. For ViTs, we use the TransFusion procedure (Rinaldi et al., 2025) for 10 iterations as the latter uses spectral decomposition and is slower than Git Re-Basin. The detailed description of these methods can be found in Appendix A and Appendix B. Subsection D.I provides an estimate of the time required for canonicalization.
Batch Normalization Statistics Based Recalibration: We implement a post-generation recalibra- tion procedure where butch normalization (BN) (loffe & Stegedy, 2015) statistics are recomputed using the training dataset for each set of generated weights. Neural networks with BN pose chal- lenges for weight generation, even perfectly generated weights can underperform if BN statistics are misaligned. WeepWeightFlow addresses this by recalibrating BN statistics after weight gen- eration, ensuring models are accurate. While the FM framework successfully learns BN weight parameters (7 and 3), the running statistics (mean and variance) require more careful processing. These statisties are intrinsically tied the training data distribution and must be precisely calibrated for each generated weight set. Our experiments, summarized in Table 7, reveal that directly trans- ferring running statistics from reference model yields suboptimal performance. We provide our recalibration algorithm in Algorithm 1 (Wortsman et al., 2021; 2022). Layer normalization (Ba et al., 2016) is permutation invariant and does not need recalibration (Ainsworth et al., 2023).
Incremental and Dual PCA for scaling to large neural networks: We use incremental and Dual PCA to scale to larger networks, as training on unprocessed training data for larger neural networks is limited by available GPU memory. We use incremental PCA to preprocess the training data when the weight space dimension of O(10M) and Dual PCA when the dimension of the weight space is O(100M), and inverse PCA during generation. The algorithmic and computational details of the latter can be found in Subsection D. I. We also perform ablation studies to show the improvement in training time by using PCA (Table 8 in Appendix D).
Training Data Generation: All training data used in this work was generated ab initio from a set of randomly initialized neural networks trained separately, thus generating diverse set of neu- ral networks. Details of the training dataset generation can be found in Appendix E. We test DeepWeightFlow on diverse tasks such as the Iris (Fisher, 1936), MNIST (Lecun al., 1998), Fashion-MNIST (Xiao et al., 2017), CIFAR-I0 (Krizhevsky et al., 2009), and Yelp (Xiang Zhang, 2015) datasets for both classification and regression tasks. Recent work by Zeng et al. (2025) has raised concerns about the lack of diversity of weights sampled from generative models trained on checkpoints from training a single neural network (Wang et al., 2024). We generate neural net- work weights independently trained from random initialization and not drawn from a sequence of checkpoints from training a single neural network, thus increasing the diversity of the training set, for training all provide code to generate the training dataset in https: / //github. wwwccoom/nn///c Www.Nuaaaaaaaaaaaaaaaa DeepWe i g and hyperparameters in Table 12 and Table 13 of Appendix E. The datasets will be made available in the future.
models. We provide the hyperparameters in Appendix E. We
5 EXPERIMENTS
We conduct a series of experiments to evaluate the effectiveness of our approach across different architectures, training conditions, and downstream tasks. We show that DeepWeightFlow generates complete weights for MLPs, ResNets, ViTs, and BERTs with high accuracy, and canonicalization improves performance at low FM model capacity. We see that incremental and Dual PCA enables scaling DeepWeightFlow to (100M) parameters. Our approach is robust across diverse initializa- tion schemes, including Kaiming, Xavier, Gaussian, and Uniform. We see that Gaussian source
5

---

Published as a conference paper at ICLR 2026
distributions outperform Kaiming, with variance choice being most critical at low capacity. Gen- erated CIFAR-10 models transfer effectively to STL-10 and SVHN. Lastly, the generated neural networks are diverse while maintaining strong accuracy, and training and sampling are significantly faster than diffusion models such as RPG, D2NWG, and P-diff. Unless explicitly stated, all training sets are 100 terminal neural networks (not checkpoints from a single training round) initialized with unique seeds (Appendix E and Appendix F). All Deep WeightFlow models are architecture-specific except when we probe class-conditioning (Subsection K.2).
5.1 COMPLETE WEIGHT GENERATION ACROSS ARCHITECTURES
Table 1: Comparison of Deep Deightigh with other SOTA neural network weight generating methods for complete generation of weights for MNIST classifiers, without finetuning.
Model
Neural Network
Original
Generated
Reference
DeepWeightfFoow (w/ Re-Basin)
96.17 0.31
DeepWeightFlow (w/o Git Re-Basin)
3-Layer MLP
96.32 + 0.20
96.19 : 0.27
WeightFlow (Geometric, aligned + OT) 3-Layer MLP
93.3
78.6
Erdogan (2025)
FLoWN (Unconditioncd)
modium-CNN
92.76
83.58
Saragih et al. (2025b)
Table 2: Comparison of DeepWeightFlow with other SOTA neural network weight generating models for com- plete ResNet-18 CIFAR- 10 classifier weight generation, without fine tuning.
Model
Original
Generated (Partial)
Generated (Complete)
Reference Reference
DeepWeightFlow (w/ Git Re- (e-Basin)
93.55 1: 0.13 93.47 1 0.20
DeepWeighttlow (w/o Git Re- Basin)
94.45 = 0.14
RPG
95.3
95.1
Wang et al. (2025)
SANE
92.14 L 0.12
68.6 6 T. 1.2
Schürholt et al. (2024)
D2NWG
94.56
94.57 +: 0.0
Sor et al. (2025)
N. (Unconditioned)
94.54
94.36
Saragih al. (2025a)
Wang el al. (2024)
P-diff (best neural network)
94.54
94.36
(Saragih al., 2025b)
FLoWN (best neural network)
94.54
94.36
Saragih al. (2025b)
Models use autoregression to generate complete models over multiple passes.
Table 3: Comparison of Deep with other SOTA neural network weight generating models for
plete ResNet-18 STL-10 classifier weight generation, without fine-tuning.
Model
Original
Generated (Partial)
Generated (Complete)
Reference
DeightightFlow (w/ Re-Basin)
62.46 0.79
Deep WeightFFw (w/o Re- Basin)
62.30 1..00+0.77 0.77
62,50 a 0.66
P-diff'
62.00
62.24
Wang et al. (2024)
FLoWN
62.00
62.00
Saragih et al. (2025b)
NM (Unconditioned))
62.00
62.00
Saragih et al. (2025a)
Table 4: Comparison of Deep WeightFwww with other SOTA neural network weight generating models for VIT family CIFAR- 10 classifiers, without finetuning. We have used ViT- VTT-small- 192, indicating an embedding dimen- sion of 192 Wang et al. (2025); Schürholt et al. (2024); Soro et al. (2025); Dosovitskiy et al. (2021).
Model
Neural Network
Original
Generated
Reference
DeeghtightFlow (w/ TransFusion) DeepWeightFlow (w/o TransFusion)
Vit-Small-192
83.30 1 0.29
83.07 0.42
82.58 + 0.07
P-diff (Best)
ViT-mini
73.0
73.6
Wang el al. (2024)
RPG
VIT-Base
98.7
98.9
Wang et al. (2025)
Deep DeepWeightFlow generates complete neural network weights and the generated networks perform well the training set. In Table 1, Table 2, Table 3, and Table 4, highlight the results of gener- ating MLPs, ResNet- CEANNet-1/2000 18/20s and ViTs from BeepeeighFlow models. We have conducted our exper- iments MNIST, Fashion-MNIST, CIFAR-10, STL-110 (Coates et al., 2011), and SVNH (Goodfel-

---

Published as a conference paper at ICLR 2026
low et al., 2013) datasets. As noted before, we generate the complete weights for all neural networks, including those with batch normalization such as ResNet-18 and ResNet-20. The comprehensive weight generation scope of DeepWeightFlow is unlike existing approaches such as FLoWN (Saragih et al., 2025b) and P-diff (Wang et al., 2024), which primarily generate only partial weight sets (lim- ited to batch normalization parameters due to lack of scalability with neural network parameter size). Moreover, Deep BeepWeightFlow generated networks perform as well as the training set without the requirement of additional conditioning during training or inference. With sufficient flow model capacity, performance converges regardless of canonicalization or noise scheduling strategy, sug- gesting that model capacity can compensate for suboptimal design choices. The choice of source distribution significantly impacts FM performance and generated model diversity (cf. Figure 2).
the careful selection of the standard deviation parameter of the source distribution: optimal results are achieved when the source distribution' standard deviation matches or slightly undershoots that of the target weight distribution. Our empirical analysis demonstrates that Gaussian noise consistently outperforms al- ternative initializations (e.g., Kaiming initialization (He et al., 2015)) as the source distribution (Ta- ble 16 in Appendix H). This sensitivity is particularly pronounced in smaller flow models, where insufficient capacity amplifies the importance of proper initialization (Saragih et al., 2025b).
Effeet of Source Distributions: Critical to the success of
Table 5: Cononicalization is heneficial when DeepWeighrPlog has limited capacity, leading to superior perfor- mance. As model capacity increases, both canonicalized and non-canonicalized models perform comparably. with the best results highlighted in bold.
Original
Generated
Dataset
Architecture
dp
with Re-Basin (metric) mean dev.
without Re-Basin (metric) mean
(metric) mean
dev.
Classification Tasks (Accuracy %)
256
91.43 1. 2.07
91.03 1: 2.20
Iris
MLP
128
90.70 1: 2.02
91.43 + 2.46
90.87 + 3,25
64
91.87 1 2.23
90.80 1 4.86
32
90.80 = 2.54
88.93 + 6.09
512
96.17 + 0.31
96.19 0.27
MNIST
MLP
96.32 - 0.20
96.21 + 0.28
96.20 + 0,23
128
91.74 + 10.37
89.71 + 17.93
57.80 1 9.85
25.54 1 12.90
512
89.10 + 0.29
89.11 + 0.28
Fashion-MMn -NIIT
MLP
89.24 - 0.27
89,06 + 0.29
89.02 + 0.30
128
88.09 + 2.24
85.81 + 11.32
77.76 +: 3.72
53.35 30.49
75.07 1.24
74.92 0.80
CIFAR-10
ResNet-20
256
73.62 + 2.24
75.32 1. 0.83
74.91 1 0.97
128
73.08 + 4.35
72.35 1 8.86
20.16 13.44
20.06 15.76
384
82,99 0.11
82.58 + 0.07
CIFAR-10
Vit-Small- -92
256
83.30 1 0.29
83.07 + 0.42
82,51 1 0.55
128
69,09 1 25.20
41,15 + 25.26
64
43.13 + 30.28
7.11 12.67
1024
93.55 +: 0.13
93.47 + 0.20
CIFAR-10
ResNet-18f
512
94.45 + 0.14
93.49 + 0.19
93.43 0.64
128
57.98 34,02
47.55
37.46
64
29.92 + 19.79
21.93 + 19.86
Regression Task (Spearman Correlation)
Yelp Review
BERT- I 18M
1024
768
0.7902 + 0.061
0.7909 + 0.005 0.7894 I 0.006
0.7892 1 0.015
0.7884 + 0.012
ResNet-118 results use standard incremental PCA-reduced weights.
I BERTT 118M results use dual/Grrm PCA approach.
" "dy; flow hidden dimension
Scaling with PCA: Deep WeiehhegttFlow can seale to large neural networks using PCA (Wold et al., 1987; Hotelling, 1933). For models with tens of millions of parameters, we employ incremental PCA (Ross et al., 2008) to reduce the dimensionality and inverse transformation post-generation. This approach maintains accuracy levels, can be seen from Table in Appendix D, while enabling tractable training of DeepWeightFlow for large-scale architectures. This demonstrates the feasibility of extending our methodology to generate complete weight sets for contemporary large neural networks without the requirement of training additional models for dimensionality reduction, such autoencoders, is often done for latent diffusion-
flattened weight vectors in the training set,
7

---

Published as a conference paper at ICLR 2026
based models ( (Wang et al., 2024). We demonstrate that DeepWeightFlow can be scaled to (1100M) parameters with Dual PCA. Given the reduction of resources and time required with Dual PCA, we estimate that models of O(IB) parameters might be possible to generate using DeepWeightFlow and leave that as future work.
Impact of Canonicalization: We observe a capacity-dependel behavior of DeepWeightFlow mod- els with and without canonicalization. At lower capacity of the FM models, models trained on canonicalized neural network weights generate higher performing ensembles than the FM mod- els trained on non-canonicalized data. However, as the capacity of the FM model increases, the performance of the ensembles of generated neural networks become similar. In general. FM mod- els trained on canonicalized neural network weights approach the performance of the training set ("original" neural networks) with lower capacity. Moreover, when flow model parameters are lim- ited, models trained on canonicalized data generate neural networks with observably lower variance in accuracy compared to non-canonicalized counterparts. Table 5, we show the performance of DeepWeightFlow with and without canonicalization.
Robustness Across Initialization Schemes: To evaluate generalization capability, we conducted extensive robustness testing using MLP models trained on the Iris dataset with diverse initialization strategies (Kaiming (He et al., 2015), Xavier (Glorot & Bengio, 2010). Kaiming weights and zero for biases, normal, and uniform distributions). Training single flow model on this heterogeneous collection (100 models total: 20 seeds x 5 initialization types) successfully generated novel weights achieving high test accuracy, demonstrating the framework's 's ability learn from and generate weights across different initialization regimes. All other experiments maintained consistency by using Kaiming initialization with varied random seeds.
5.2 TRANSFER LEARNING ON UNSEEN DATASETS
Our generated models can be effectively used for transfer learning (Nava el al., 2023; Zhang et al., 2024) across unseen datasets. In our experiments, we trained DeepWeightFlow on ResNet-18 mod- els for the CIFAR- 10 dataset using PCA, generated 5 models, and recalibrated their batch normaliza- tion running mean and variance on small subset of CIFAR- 10 in the same way applied in Table 5 and claborated on in Table 14. These models were then evaluated under zero-shot and finetuning settings on STL-10 and SVHN datasets. The results presented in Table 6. Deep Weightitttl generated models consistently outperformed state-of-the-art FM models such as FloWN (Saragih al., 2025b) in both zero-shot and finetuning evaluations. Furthermore, they significantly outper- formed randomly initialized models, proving the effectiveness of the method. The same comparison done with SANE (Schürholt et al., 2024) and reaches the same conclusion. Results on transfer learning for CIFAR- 100 models fine-tuned on CIFAR- 10 ResNet- 18 backbone can be found in Ap- pendix J.
5.3 DIVERSITY OF GENERATED MODELS
To evaluate the DeepWeightFlow models' generative capabilities, we compute the maximum IoU (mloU) between the generated neural networks and the neural networks in the training (referred to as the "original" neural networks). The mloU is constructed from the intersection over union of the wrong predictions made by the neural networks (Wang et al., 2024). is defined as loU [pwroong pP / / /|Parrog U P pirrong]. where P1 comes from the set being compared (such as from the generated set) and P2 comes from reference set (such the set of original neural networks). We disregard the IoU of a neural network with itself as it is trivially 1. The mIoU measure scales from complete dissimilarity at 0 to complete similarity at 1.
In Figure 2, we compare the original neural networks with the generated ones, with noise added to the weights of the original neural networks, and with neural networks generated with different FM source distributions. The upper row compares the cases for the FM models trained with Re-Basin, and the lower panels, without. In the left-most panels, we see that i) the original networks are quite diverse from each other, as evident from the blue cloud. This is the case as, unlike several previous works, we do not use checkpoints from the training of a single neural network as the training set of the Deep WeepWeigtFFoow model. The training set for BeepWeightFlow consists strictly of terminal models of unique random initializations. Details for dataset generation are outlined in Appendix E. ii) The yellow and green clouds show that adding progressively increasing Gaussian noise to the
8

---

Published as a conference paper at ICLR 2026
Table 6: Transfer learning performance across different architectures For ResNet-18, we compare CIFAR- 10 classifiers generated by DeepWeightFlows FLoWN, and Randomlnit. For SmalICNN, we compare with SANE (Schürholt et al., 2024) trained on CIFAR-IO and transferred STL-10 using the same architecture as mentioned in Schürholt et al. (2024). Randominit refers randomly initialized neural networks with Kaiming- He initialization. Pretrained refers to neural networks from our training dataset, and Generated refers to weights sampled from the respective generalive model.
Architecture
Epoch
Model
Method
STL-10
SVHN
18 Results (Comparison with FLo (Saragih et al, 20256))
FLoWN
Randomlnit
10.00 0.00
10.00
0.00
Generated
35.16 1.24
17.99
+ 0.82
ResNet-18
RandomInit
11.18 1: 1.48
8.01 1 1.41
DeepWeighFFlow
Pretrained
48.31 = 0.17
11.51 + 0.31
Generated
48.32 0.34
11.57
0.49
FLoWN
Randomlnit
18.94 0.09
19.50 0.03
Generated
36.15 1 1.14
68.64 + 7.07
ResNet-18
RandomInit
38.28 + 1.07
84.07 + 1.76
DeepWeightFlow
Pretrained
79.81 1 0,54
91.29 1 0.76
Generated
79.69 + 1.08
91.66 0.79
RandomInit
28.24 1. 0.01
39.59 10.0
FLoWN
Generated
37.43 = 1.19
77.36 + 1.07
ResNet-18
RandomInit
51.35 1. 0.51
93.82 0.16
DeepWeighFFoow
Pretrained
84.61 0.21
95.82 0.16
Generated
84.63 + 0.17
95.85
0.09
SmalICNN Results (Comparison with SANE (Schürholl et al., 2024))
Train fr. scratch
~10 16.2 2.3
SANE
Pretrained SAN EsuB
17.4 1.4
SmallCNN
0
RandomInit
9.47 0.52
DeepWeight
Pretrained
35.18 0.71
Generated
35.29 = 0.48
Train fr. scratch
21.3 1.6
SANE
Pretrained EANEsU
24.8 2 0.8
25.6 1.7
SmallCNN
RandomInit
21.09 + 2.52
DeepWeighttlow
Pretrained
41.66 1.75
Generated
41.03 1.22
Train fr. scratch
44.0 1.0
SANE
Pretrained
49.0 0.9
SAN ESUB
49.8 0.6
SmalICNN
25
Randomlnit
44.33 + 1.54
DeepWeightFlow
Pretrained
62.14 0.84
Generated
62.62 + 0.46
original networks makes them progressively diverse from the original networks as expected (loU < 1). iii) The red cloud representing the generated networks shows diversity from the original set but seems to overlap with the green set, which represents the set created by adding noise sampled from (/000.01) to the original neural network weights.
From the middle panels in Figure 2, we see that the red cloud representing the generated neural net- works is sufficiently diverse from the original ones with added noise sampled from (0,, 0.01). This gives us confidence that the generated neural networks are, indeed, not the same as the original net- works with noise added the weights. Lastly, the right-most panels show how diverse the generated neural networks are when generated with different source distributions. Hence, Deep DeepWeighFFoow capable of generating diverse of neural networks while maintaining the accuracy of the task. In Appendix I, we provide the numerical estimates of mIoU, the Lensen-Shannor Wasserstein, I?2 cosine similarity, and Nearest Neighbors (NN) distances between generated and original neural networks and supplemental mloU analysis of ResNet- 18 weights generated by
5.4 TRAINING AND SAMPLING EFFICIENCY
Deep WlightFlow is significantly faster to train and generate neural network weights when compared to diffusion models in complete neural network weights generation. DeepWeightFlow takes up to O(10) minutes to train for most neural network architectures with up to ((100M) parameters. as compared to the several hours that it takes to train RPG (Wang et al., 2025). Deep FeighttttFFww takes few seconds to generate neural networks compared to the minutes hours it takes to generate

---

Published as a conference paper at ICLR 2026
With Git Re-Basin
Gemid
Deeedlte 2.01
Without Git no-Basin
Origen
Figure 2: Maximum IoU vs test set accuracy for MNIST classifying MLPs. Lower maximum IoU implies greater diversity in the neural network weights. The left panels are generated and original neural networks (from the SreepWeighFFlee training set) with different scales of Gaussian noise odded to the original neural networks. The middle panels show that the generated neural networks and the original neural networks with noise added, which overlap in the left panels, are concretely different. The right panels contain the original and generated neural networks with different source distributions. All panels include 500 generated neural networks.
using RPG, P-Diff, or D2NWG. Yet, DeepWeightFlow generates ensembles of neural networks that have comparable outcomes for ResNet-18s and ViTs. This is primarily because the other models diffusion models, whereas DeepWeight WeightFlow is based on FM using a simple MLP implementation. detailed comparison of training and generation efficiency can be found in Appendix G.
CONCLUSION
In this work, we introduce DeepWeightFlow, generative model for neural network weights that performs FM directly in weight space, unconditioned by dataset characteristics, task descriptions, or architectural specifications, and avoiding nonlinear dimensionality reduction. We show that Deep- WeightFlow generates diverse neural network weights for a variety architectures (MLP, ResNet, VIT, BERT) that show excellent performance on vision, tabular classification, and natural language tasks (regression). We provide empirical evidence that canonicalizing the training data facilitates weights with
the generation of larger networks but is of limited use for increasing FM model capacity. DeepWeightFlow can be combined with simple linear dimension- ality reduction techniques like incremental PCA and Dual PCA alleviate restrictions neural network size and demonstrate scalability to large neural networks of O(100M) parameters with possibilities of scaling even further. The compatibility of DeepWeightFlow with model distilla- tion, low-rank approximations, or sparsity remains as future work. As such, some open questions about the relative merits of canonicalization, equivariant architecture design, and data augmentation for learning in deep weight spaces remain. Lastly, we demonstrate DeepWeightFloow 's ability to generalize to multi-class generation through class conditioning (Appendix K). We extend Deep- WeightFlow to combining multi-class and multi-architecture generation of complete weights. The results do not seem promising and we leave further exploration future work with possibilities of combining DeepWeighuFlow and dataset conditioning similar to FLoWN or D2NWG. Nevertheless, DeepWeightFlos shows promise for extension to real-world applications such as rapid generation of neural networks for vision and NLP tasks in distributed devices for sensing of changing environ- ments and in privacy privacy-protecting model distribution to avoid leakage of training data.
REPRODUCIBILITY STATEMENT
The architectural details along with the hyperparameters used to generate the data have been pro- vided in the main text and Appendix E and Appendix F. The dataset will be made available request and/or uploaded to a data repository. The code necessary to reproduce the results in https: / / thub. com Ntep/aNeuaallaaaaaaa Dynam cs c ittt
10

---

Published as a conference paper at ICLR 2026
AUTHOR CONTRIBUTIONS
S.G. and S.B. contributed all aspects of the work including ideation, implementation, analysis and drafts. M.L. and Z.S. contributed to ideation, analysis, and drafting of the work. R. W. and A.P. contributed to the ideation, methodological guidance, and drafting of the work.
ACKNOWLEDGEMENTS
R. would like to acknowledge support from NSF Grants 2442658 and 2134178. This work is sup- ported by the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http:/ /tt:://iai //waaiiff../////////a.////...aaa org/). M.L. was supported by the Joseph E. Aoun Endowment. S.G. and M.L. would also like to ac- knowledge Derek Lim for fruitful discussion about this work. The authors grateful Voltage Park for providing computational resources that enabled part of the work.
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git be-bsin: ulo permutation symmetries. In The Eleventh International Conference on Learning Representa- fions, 2023. URL httpp::/openneeviiieee
Merging models mod-
net forum? www.ffoodddnnmmnmmaooo 1d==Com//m1P55..
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL /etps:///rxivv wttp://gg///b///.ooooobb
06450.
Hervé Cardot, Peggy Cénac, and Pierre-Arnaud Zitt. Online principal component analysis in high dimension: Which algorithm to choose. International Statistical Review, 86(1):29-50, 2018.
Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. Progressive feature alignment for unsupervised domain adaptation. In 20/9 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 627-636, 2019. URL https:://///mmm weeeeeeeeeelrreeeeeeeeeeeeeeeeee www.ooreeeeeeeeeeeeeeeee.eee.eee org/ document//////oo/oooocooco /8953748.
Xiangning Chen. Advancing Automated Machine Learning: Neural Architectures and Optimiza- fion Algorithms. PhD thesis, University of Californi,, Los Angeles, United States Califor- nia, 2023. URL https: /ww.. proquest......o com/ http//dviwwwweeeeeeee /bb.t//t/////////////////oo BCADEIC2664A444CGO0
Adam Coates, Andrew Ng, and Honglak Lee. An Analysis of Single-Layer Networks in Unsuper- vised Feature Learning. In Proceedings of the Fourleenth International Conference Artificial Intelligence and Statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, June 2011. URL https: ppwppeeocoeeggsssee mlr press / vvlv/v15/////// wtt.//ooaasaaa...cooooo.aaa..ala html.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Com- putational Linguistics. doi: 10. 10.18653/v1/N19 1423. URL https: aaclaacantttt /alaaaaaaannotoyyoyyoooooo org/ N1 N19-1423/.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1029-1028. PMLR, 2017. URL https: / /pproceedings mlr. www..peat...apttmmmmmm.....a..aa
wwwppppessvvv.... /v70 dinh1 7b. html.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: /ppeneeview net. wowt//oormm////mm d=Yi cbF

---

Published as a conference paper at ICLR 2026
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation in- variance in linear mode connectivity of neural networks. In International Conference on Learning Representations, 2022. URL https: : httpp//://///p//////eeeeeeee http://wwww..ee/tttnnneeeeeeeeee
Ege Erdogan. Geometric flow models over neural network weights, 2025. URL https: //arxive www.v.orr//aeee
03710.
R. A. Fisher. The used of multiple measurements in taxonomic problems. Annals of Eugenics, 7(2): 179-188, 1936. URL https: htRLtttts:////inneee wwwooonnneiniaie wiley com/doi/abee /1lps//.... 1111/j. 1469-1809, 1936. tb02137.*
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https:// openreview wttlleewweeeeeeeeeee (or.mmid--r1--beeee
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 249-256, 2010. URL http: http://ppooceedngg mlr www.mmlr.p...... prr.ssvv/ glorot10a. html.
Steven Golovkine, Edward Gunning, Andrew J. Simpkin, and Norma Bargary. On the use of the gram matrix for multivariate functional principal components analysis. Journal of Multi- variate Analysis, 212:105525, 2026. ISSN 0047-259X.
doi: https///oii.rr/10..100
2025.105522 URL https:// Settps::/www... http://wwwiiceedrreeeeee 50047259X25001200
con//aciene//atticle///.
Ian J. Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnudd, and Vinay Shet. Multi-digit number recognition from street view imagery using deep convolutional neural networks. In Proceedings of the 2013 International Conference on Machine Learning (ICML), 2013. URL. http//bs/1333 6082.
https://arrti
Ian Goodfellow. Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv, 2015. URL https::t//// /t///rixiiii////./// www.wiwwwww.aaaaaoooooooooereeae abs////bb//122 6572.
/
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang. Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhi- hong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjic Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Ji- ashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang. Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuip- ing Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xi- aotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaox- iang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan. Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang. Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong. Yuxiang Luo. Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang. Yaohui Li, Yi Zheng. Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie. Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang.
12

---

Published as a conference paper at ICLR 2026
DeepSeek-RI incentivizes reasoning in LLMs through reinforcement learning. Nature, 645
(8081):633-63 638, September 2025, ISSN 1476-4687. doi: 10. 10.1038//4158--110 020--0025--2222-22 URL
https:/ / /wwww... matture.com/ /ttp://aarceeeee1111-0---------1
David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetworks. International Conference on Learn- ing Representations, 2017. URL ht tattpp::. opwnrevieneeeeeeeeeeeee net/ wow.mmiddokppaaii.
N. Halko, P. G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2):217-28) 2011. doi: 10. 10.1137/090771800 URL https: org /1t.11117/00977111CC
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into recti-
fiers: Surpassing human-level performance imagenet classification. In Proceedings of the 2015 IEEE Internationaa Conference Computer Vision (ICCV), Pp. 1026-1034,
2015. URL https::/ //wp::///wwww////// wwwp:wwwvveeeeee co-foondaiion wttp////aacceess..eee//////////e ceceecccessc/oteettte _ccv__ 2015 10155/hmmllm hhtph/leeeeeeeeeeeeeeeeee htttmm//eeeeeellii_eeeeeeeeee oor_rri____ee_ittoooooo_______ /001: 2ar22011pppprrhmmmmm html.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pp. 770-778, 2016. URL https: hpwp//ccpccccsssceeeepeeoce.eese thecvÉ. wwtpccowcccceeee..eeooooooocoo.t content_etttttt Seatttmlleeeeeeeeeeeee htte://aneeeeeeeeeeeeeaaaaaaaeee BewpiiduaaLLLarnnnnn__gCeCC 2016 2aapee__22o__pppe_____1__
Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Rolf Eckmiller (ed.), Advanced Neural Computers, pp. 129-135. North-Holland, Amsterdam, 1990. ISBN 978-0-444-88400-5 URL https: //ww:. www.scc.ee scis eciencedirect. wwm/ sci/sceence/ article//ll//7770044444000038
Harold Hotelling. Analysis of complex of statistical variables into principal components. Journal of Educational Psychology, 24(6):417-441, 1933. doi: 10.1037/h0071325. URL https://
psycnet. wwe.net.ppa.c.m htt.a//..rr.eeeeee///90000666660
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Ma- chine Learning Research, pp. 448 456, Lille, France, 07-09 Jul 2015. PMLR. URL https: / //proceedings. ww..m...ppr.. war..press..... / vv /eiofoffll555 html.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Proceedings of the 34th Conference Uncertainty in Artificial Intelligence (UAI), pp. /attps:/aaxiin org/ a 1803 05407.
AUAI Press, 2018. URL
R. Jonker and A. Volgenant. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing, 3844)325-,440 November 1987. ISSN 0010-485X. URL https: //oii.////ii//// wttpi/orgg/....1...11. /00.0.00///0002771111
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. Repair: Renor- malizing permuted activations for interpolation repair. arXiv preprint Sttv:2211.0800.. 2022. doi: 10. (0.48550/arxiv.2211 2.000403.. URL https:://// /ttp:///aaiiiiaa..a org/ abs 2211 08403.
Bobak Kiani, Jason Wang, and Melanie Weber. Hardness of learning neural networks under the manifold hypothesis. In The Tiihth Annual Conference on Neural Information Processing Systems, 2024. URL httph::///// hwwp//eeeeeeeeeeee/eeeeeeee/eeee https://openrevieww net / forum?? id=dkkggKMMi17..
Boris Knyazev, Michal Drozdzal, Graham W. Taylor, and Adriana Romero. Parameter predic- tion for unseen deep architectures. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: / //op//nevvew.net...... /tt aett///ooorittetvaaaaaaAACC
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL http: /tt:://www www. as toronto. edu/kriz/ct far. html.
13

---

Published as a conference paper at ICLR 2026
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Confer- on Neural Information Processing Systems, NIPS' 17, pp. 6405-6416, Red Hook. NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964. URL https: / /dl. acm. wtt.orrggoooooooo doi/ 10. 025555533252222 . 3295387.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE, 36(11))227--3224 1998. URL https: bttttps://ieexpl..oo ieeexplore. leee. org/ httg//doumeet/./ (ttpuumennn72699...
Derek Lim, Theo Putterman, Robin Walters, Haggai Maron, and Stefanie Jegelka. The empirical im- pact of neural parameter symmetries, or lack thereof. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: http:////e/ppeeee..eoeooe. net forum? id=pCVxYw6FKg.
Yaron Lipman, Ricky Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL https: / http///ppeeeeeeeeeeeeeeeeeeeeeee net. seee:/forummldddP/AAROTTT
Ziyin Liu. Symmetry leads to structured constraint of learning, 2023. URL org/abs/ 2309. 1.....6635 1932.
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. arXiv preprint arXiv: SKtiv:1902.02770 2019. URL https: / /ttp//aaavi////////////////aa
http://////.......a/a/////////// wwtp/bs// 0010/990.00000 02476.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu, To- wards deep learning models resistant to adversarial attacks, In International Conference on Learn- ing Representations, 2018. URL https: www.reviee..eeeeeeeeeeeeee net. forum
Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual asso- ciations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /pwnrevvee.eeeeeee net post/Eorum?ice iid- h6WA
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference Learning Representations, 2022. URL
https: //ppr//ieeeeeeeeeeeeeeee .nt/ fwrumroddoooooooooooo Lit.ummeiddDDDeeeEEPP.....
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and feder- ated learning. In 2019 IEEE symposium on security and privacy (SP), pp. 739-753. IEEE, 2019. URL https:/ /ttp:://wwww /twww//ww../.../////.o wwwptte..orggooooooooeee cstp://occceeiiggs article/ 11122209/666000733////
70.
Elvis Nava, Seijin Kobayashi, Yifei Yin, Robert Katzschmann, and Benjamin Grewe. Meta- learning via classifier(-free) diffusion guidance. Transactions on Machine Learning Research, 4: 1-20, 2023. URL https::tp//.......t. /ttp::///peeeeeeeeeeeeeee net htt///oommmmmdi/////o j17.3...........
Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equiv- ariant architectures for learning in deep weight spaces. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org, 2023. URL https: / //dl./d//////// acm. www.org///oo................oo.r
doi/10. 5555 3618406188888 0111668..6191440011.111111111111 3619481.
Behnam Neyshabur, Ruslan Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. In Advances in Neural Information Process- ing Systems, volume 28, 2015a. URL https: http://appprs....m niw.r....pppccomm httpipp.ccppppp////p oed intettmiiiiiiii on- hww.mratt----nn-----eeee--- -n------eeerrrlleeeeeeee--------
pdf.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neu- ral networks. In Proceedings of the 28th Conference Learning Theory, volume 40 of Proceedings of Machine Learning Research, pp. 1376-1401. PMLR, 2015b. URL https: ipproceedings mlr www.a......a/ppmmmmmmmmmmmmm...e v40 htt..ppreee///////e//abbbba Neyshabur115555 html.
14

---

Published as a conference paper at ICLR 2026
William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A. Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints, 2022. URL https: //arxiv. org/abs. 2209. 12892.
Fabrizio Pittorino, Antonio Ferraro, Gabriele Perugini, Christoph Feinauer, Carlo Baldassi, and Riccardo Zecchina. Deep networks toroids: Removing symmetries reveals the structure of flat regions the landscape geometry. In Proceedings of the 39th International Conference on mmlr.
Machine Learning, pp. 17781. PMLR, 2022. URL http::://ttcceeeeaeeeee press/ vvl///112/////// wwtppppittttaaa....pa html.
Neale Ratzlaff and Li Fuxin. HyperGAN: A generative model for diverse, performant neural net- works. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- ternational Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5361-5361 PMLR, 09-15 Jun 2019. URL https: httpc////occeeeeeeeeeee mllr.
press/v97/ratzlen
html.
Filippo Rinaldi, Giacomo Capitani, Lorenzo Bonicelli, Donato Crisostomi, Federico Bolelli, ELISA FICARRA, Emanueic Rodolà, Simone Calderara, and Angelo Porrello. Update your transformer to the latest release: Re-basin of task vectors. Forty-second International Conference Ma- chine Learning, 2025. URL https: : openreview .net/ forum?id=stie ImzN9pL.
David A. Ross, Jongwoo Lim, Ruei-Sung Lin, and Ming-Hsuan Yang. Incremental learning for robust visual tracking. International Journal of Computer Vision, 77(1):25-141,, 2008. URL https: https://linte www.sgerrrn............
hpppss//iinsssrrnngerr.... com http//rrtcce111111 http:/1...10//111111 000000/01112200700077----555
00000000---00
Daniel Saragih, Deyu Cao, and Tejas Balaji. Flows and diffusions on the neural manifold, 2025a.
URL atttps://azxte org/ abs/220177 10623.
Daniel Saragih, Deyu Cao, Tejas Balaji, and Ashwin Santhosh. Flow learn: Flow matching on neural network parameters. In Workshop on Neural Nenwork Weights as a New Data Modality, 2025b. URL https::t//// /ttptt::://peeeeee
net forum? id=r0ynsstq3c.
Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Nonlinear component analysis kernel eigenvalue problem. In Neural Computation, volume 10, pp. 1299-1319. MIT Press, 1998. URL
ieee. wwwo/////ooooooooo /ttp:///ccmmenn/d/eem 6790375.
Konstantin Schürholt, Boris Knyazev, Xavier Giró Nieto, and Damian Borth. Hyper- representations as generative models: Sampling unseen neural network weights. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural In- formation Processing Systems, 2022. URL uyEXNg2HHFO.
net forumiid=
Konstantin Schürholt, Michael W. Mahoney, and Damian Borth. Towards scalable and versatile weight space learning. In Forty-first International Conference on Machine Learning, 2024. URL https: /ttp::s//ppeeevview net forum? diuruumidduuguuoA23ee
Aviv Shamsian, David Zhang, Aviv Navon, Yan Zhang, Miltiadis Kofinas, Idan Achituve, Ric- cardo Valperga, Gertjan Burghouts, Efstratios Gavves, Cees Snoek, Ethan Fetaya, Gal Chechik, and Haggai Maron. Data Augmentations in Deep Weight Spaces. In NeurlPS 2023 Work- shop on Symmetry and Geometry in Neural Representations, November 2023. URL https: / opwnreview..o. wwttfforummiiee wtwmmmdddddddddddssssssssss
Aviv Shamsian, Aviv Navon, David W. Zhang, Yan Zhang, Ethan Fetaya, Gal Chechik, and Haggai Maron. Improved generalization of weight space networks via augmentations. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of (CML'24, pp. 44378- 44393, Vienna, Austria, July 2024. JMLR.org. URL httpp::/dd/ acm org/ d 10. 5555 0005553369000 33000709920000 3693876.
John Shawe-Taylor Christopher KI Williams, Nello Cristianini, and Jaz Kandola. On the eigen- spectrum of the gram matrix and the generalization error of kernel-pea. IEEE Transactions on Information document/1459055.
25222. 2005. URL https: https://leeexploree iiee...rrreeeee...........eeee.e org/
15

---

Published as a conference paper at ICLR 2026
Gil Shomron and Uri Weiser. Post-training batchnorm recalibration, 2020. URL https:
/ wwwpxii.org/aba/////ae 2010 05625.
Bedionita Soro, Bruno Andreis, Hayeon Lee, Wonyong Jeong, Song Chong, Frank Hutter, and Sung Ju Hwang. Diffusion- neural network weights generation. In The Thirteenth Interna tional Conference on Learning Representations, 2025. URL https: /tops://ooe /ttps/////eeeeeeee....mo het.nneeeeieeeeeeeeeeeeee net/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, February 2023. URL http: aawiiiiivvvo..aa org/abs 2302. 13971.
Florian Tramer, Andreas Terzis, Thomas Steinke, Shuang Song, Matthew Jagielski, and Nicholas Carlini. Debugging differential privacy: A case study for privacy auditing. arXiv, 2022. URL
fttps://arze org/ alss /202 12219.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4068-4078, 2021. URL https:/ /aaxxaaxvvvoog///.e abs/2105 06905.
Kai Wang. Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural Network Diffusion, February 2024. URL htth::///arxee orgg haagg/ab///e abs/2402. 13144.
Kai Wang, Dongwen Tang, Wangbo Zhao, Konstantin Schürholt, Zhangyang Wang. and Yang You. Recurrent diffusion for large-scale parameter generation, 2025. URL httphttts://axiv.ooom /201 11587.
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2(1))37-52, 1987. ISSN 0169-7439. doi: https://do..or 10.1016/0169- 141106600699-44494888880000000 7439487)80084-99 URL https: / /www. sciencedirect. com/ science/ article/pp p i / 01 677439877788888888
Proceedings of the Multivariate Statistical Workshop
for Geologists and Geochemists.
Mitchell Wortsman, Maxwell C Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Raste- gari. Learning neural network subspaces. In Marina Meila and Tong Zhang (eds.), Proceed- ings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11217-11227. PMLR, 18-24 Jul 2021. URL https: / //pppoeeenngss... mlr
mlr ptwps/esssseeee stress/v1390 v13// www. wttp://wtttaaaaa.aa//aa html.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Loper Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Lud- wig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accu- racy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965-23998. PMLR, 17-23 Jul 2022. URL https: /pppceeeingg...pcpp/ppppp./...sp htthttp:://rrreeeddnn mlr. mll..ress/ presss v1 62. 302/wortsman22:: html.
Yann LeCun Xiang Zhang, Junbo Zhao. Character-level convolutional networks for text classifica- tion, 2015. URL https: URR.tttss://ugginnge htthggggggggccceeee/ce ctt//at.aetttteeel///lpeeeeeeeee
full.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms, 2017. URL http:: //tp:////////vva///a///l//at org/ab/gg/.oo.. abs 111111000777 07747.
Boya Zeng, Yida Yin, Zhiqiu Xu, and Zhuang Liu. Generative modeling of weights: Generalization or memorization?, 2025. URL htt.h:tp//// h howgg/bb//5000 07998.
Baoquan Zhang, Chuyao Luo, Demin Yu, Xutao Li, Huiwei Lin, Yunming Ye, and Bowen Zhang. Metadiff: Meta-leaming with conditional diffusion for few-shot learning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(15))16687116695, Mar. 2024. URL https: // //ojs. aaai. org/ hwtp...pp//.AAt//rriiiiiiic
16

---

Published as a conference paper at ICLR 2026
Bo Zhao, Robert M. Gower, Robin Walters, and Rose Yu. Improving convergence and generalization using parameter symmetries. In International Conference on Learning Representations, 2024. URL
net 2tw.:e/iweweeeetoorrreeee..... Forum?1d--OOOGphhi.
Bo Zhao, Robin Walters, and Rose Yu. Symmetry in Neural Network Parameter Spaces, June 2025.
URL http:://axxx...rr/ppp/p/p/ org abs/ hwtp://ppp..........././ 13018.
A
GIT RE-BASIN
Git Re-Basin weight matching, formulated by Ainsworth et al. (2023), is greedy permutation coordinate descent algorithm for moving a model's weights 0a into the same 'basin' in the loss landscape of the model class ʃg as a reference model's weights Og.
This operation is applied here canonicalization step before weight flattening and the subsequent training of the DeepWeightFlow models. The procedure reduces the space of the task from Re to quotient space of R modulo permutation symmetry.
Applying this across the model layers constructs transformed model 0' by
W = PWe 1/ = Pbe, W = -W++1 PT
(3)
The 'distance' between two permutations is therefore a Frobenius inner product of P&W, and W.P. written (A, B) En, Bt, for real-valued matrices A and B. Accounting for the transforms outlined above, the process of matching the permutations across the stack of layers becomes,
arg maxx
E
(W.. P.W.A.AAA PTT with P =
(4)
(PPH #=1
This formulation presents Symmetric Orthogonal Bilinear Assignment Problem (SOBLAP), which NP-hard. However, when relaxed to focus on single permutation P, at a time ceteris paribus, the problem simplifies (Ainsworth et al., 2023; Zhao et al., 2025; Rinaldi et al., 2025). These LAPs can be solved in polynomial time by methods like the Hungarian Algorithm (Jonker & Volgenant, 1987).
series of Linear Assignment Problems (LAPs) of the form below
arg argmax max (WP P.WA (W (411 Pt
(5)
The product of this process is permutation n' of model A's weights into the same basin in fo's loss landscape as model B with exact functional equivalence (fo. = f However, sequences of LAPs are understood to be coarse approximations of SOBLAPs and, as such, strong conclusions cannot be drawn about the optimality of n' (Rinaldi et al., 2025; Ainsworth et al., 2023).
B TRANSFUSION
We canonicalize a collection of Vision Transformers (ViTs) using the method of Rinaldi et al. (2025), which introduces a structured alignment procedure for multi-head attention transformer weights (Rinaldi et al., 2025).
The core difficulty in transformers arises from multi-head attention and residual connections: Naive global permutations either mix information across heads or break functional equivalence in residual branches (Zhao et al., 2025). To address this, the method applies a two-level permutation scheme:
1. Inter-Head Alignment: For each mult attention layer, attention heads from different checkpoints are first matched. This done by comparing the singular value spectra their projection matrices, which are invariant under row and column permutations, and then solving the resulting assignment problem with the Hungarian algorithm. This step ensures that corresponding heads are correctly paired across models.
For a sub matrix representing a single attention head in model A, h = -W1 e g where k is the key value dimension and is the attention embedding dimension, apply
17

---

Published as a conference paper at ICLR 2026
singular value decomposition (A UEVT) access the spectral projection matricies E, which are invariant to row and column permutations. For every head a layer of model A, construct distance, d, [|Σ; - These distances can be constructed for g, k, and for each head and combined linearly Da,- d% + d + with Da.j E IR x (H is the number of heads). Therefore the optimal pairing of heads for model and is (Rinaldi al., 2025),
P Pinter head arg min
D PESH
2. Intra-Head Alignment: Once heads are paired, the method refines the alignment by per- muting rows and columns within each head independently, again solved via assignment on pairwise similarity scores. Restricting permutations within heads preserves head isolation and guarantees that residual connections remain valid after alignment.
After matching the heads of A B the goal aligns closely with Git Re-Basin (Ainsworth et al., 2023) to reorder h such that the Frobenius inner product is maximized between H sub portions (Rinaldi et al., 2025),
Iutra head arg ma Phyya)
By iterating these two stages across all transformer layers, the procedure yields canonicalized parameterization in which weights are aligned up to permutation symmetries. The goal is to permute units in such way that two weight sets A and 83 become functionally comparable, reducing the effective size the weight space that the FM encounters Rinaldi et al. (2025). This similar to the case of Git Re-Basin (Ainsworth al., 2023) for canonicalization.
C RECALIBRATION OF BATCH NORMALIZATION WEIGHTS
Given generated neural network with randomly initialized flow-matched weights, the batch normalization layers contain statistics that may not match the actual data distribution. Naively inter- polating weights of trained networks can lead variance collapse (Jordan al., 2022: Ainsworth et al., 2023), where the per-channel activation variances shrink drastically, breaking normalization and degrading performance. The recalibration process computes proper running statisties using the target dataset(Izmailo) et al., 2018; Maddox et al., 2019; Shomron & Weiser, 2020; Wang et al., 2021).
We include these statistics parameters of batch normalization layers in the PermutationSpec of Git Re-Basin, a config that defines the permutation ordering across layers for weight matching, so that these statistics are also permuted and correctly maintained, ensuring that the permuted networks retain the same weights and accuracy as the original network.
C.I STANDARD BATCH NORMALIZATION
For feature map x E XxCxXXX where is batch size, C' channels, and H, W are spatial dimensions:
N H
Me NHW ΣΣΣ
Inechhw
(8)
h= w=l
N ΣΣX
02 NHW h=1
Σ (Inee.h.w Mc)2
in.c.h.w En,e,h,w Me
(10)
Vo
yn.c.hww Yofn.ehh.uu 3c
(11)
where % and B. are learnalle scale and shift parameters, and is a small constant for numerical stability. During training, BatchNorm (loffe & Szegedy, 2015) maintains running statistics using
18

---

Published as a conference paper at ICLR 2026
exponential moving average:
Pe (1 a) + appl
(12)
2() (1 -a)o (() + ao?
(13)
where a is the momentum parameter, typically 0.1, and denotes the time step.
Algorithm Batch Normalization Recalibration
1: Input: Calibration dataset D (e.g., test dataset), batch size B
2: H and W denote the height and width of feature maps
3: Fi,,,h, denotes the activation of sample channel c, at spatial position (h, w).
4: Initialize f.. 0, a 1, nh 0 for all channels
5: Disable exponential moving average (momentum) updates
6: Partition D into mini-batch sequence {Bi, .,....,,By where U BB = D
7: Define batch statistics for each B₁ and channel c:
H
(R)
Fi,c,h,w
1B HW
iEB
H
2(m)
12
oc
B& HWW
BB A=
w=
8: Compute running statistics where ni
|Bg HW and m
ne
(k-1)
+ Yagi
1)
(1
1)
(k)
11e
+ 11k Me
1ec
(1)
ncc
(k-1)
2(k-1)
2(h)
(1k11) (1())
ne
0c
+ n
de
jic
2())
ne
9:
Final recalibrated statisties: fhc Pic
for all channels Restore exponential moving average updates (set momentum = 0.1)
(K)
o² de
2( K)
10:
C.2 RECALIBRATION PROCESS
For generated networks, recompute running BatchNorm statistics:
Reset: Initialize running mean and variance for all channels, and set total sample count to zero.
2. Disable momentum: Turn off exponential moving average updates.
3. Forward pass and incremental update: For each mini-batch in the calibration dataset:
Compute the mean and variance of the batch for each channel.
Update the running mean as a weighted average of the previous running mean and the batch mean.
Update the rumning variance by combining the previous variance, the batch variance, and correction for the shift in means.
Update the total sample count.
4. Restore momentum: Re-enable exponential moving average updates with the original momentum value.
The algorithm we use for recalibration of the batch normalization running statistics is provided in Algorithm 1. In Table 7 we show the results of recalibration the generated neural networks. This clearly shows the importance batch normalization, running statistics recalibration on the generation of neural networks that have batch normalization in their architecture.
19

---

Published as a conference paper at ICLR 2026
Table T: Comparing the impact of batch norn recalibration on complete ResNer-18 18 and 20s generated by DeepWeightFlow Recalibrating batch normalization statistics on a small subset of target data significantly improves the accuracy of generated models.
Model
Git Re-Basin
Strategy
Mean Std (%)
Min (%)
Max
ResNet-18
Yes
No Calibration
10.00 = 0.00
10.00
10.00
Ref BN
19.06 + 9.68
10.00
94.05
Recalibrated
93.05 4.42
49.12
93.93
ResNet-18
No
No Calibration
10.00 0.00
10,00
10.00
Ref BN
10.28 = 1.24
6,23
15.93
Recalibrated
93.49 1 0.21
92.77
93.96
ResNet-20
Yes
No Calibration
14.36 + 3.10
5.84
19.03
17.88 4.66
9,96
26.54
Recalibrated
74.57 + 0.84
71.47
76.17
ResNet-20
No
No. Calibration
12.64 2.22
8.12
18.19
Ref BN
10.23 + 0.79
8,04
14.92
Recalibrated
75.21 = 0.79
72.06
76.52
Ref BN: Uses batch normalization statistics from reference model (seed 0)
D PCA AS AN EFFECTIVE COMPRESSION STRATEGY
Table 8: Accuracy and efficiency comparison of DeepWeightFlow with and without incremental PCA compres-
sion. Training/generatin times in minutes. Generation time is the total generation+ inference time for 100 models.
Original
Generated (Accuracy)
Time (min)
Model
Method
dp
Mean
With Re-basin
Without Re-basin
Train
Generation
ResNet-220
Without PCA
512
2.24 73.62
75.07 + 1.24 75.96 + 0.89
74.92 = 0.80 75.97 * 0.86
11,25
6.00
ResNet-20
With PCA
512
73.62 2.24
1.23
5.78
Vit-Small-192
Without PCA
384
83.30
0.29
82.99 + 0.1
82.58 = 0.07
21.00
3.60
Vit-Small-192
With PCA
1024
83.30 # 0.29
83.08 + 0.19
83.28 + 0.01
2.90
1.75
In Table 8, we show the effects of using PCA to reduce the dimension of the neural network weight space. This is necessary as DeepWeightFlow cannot be trained on with the full rank of the larger neural networks, such ResNet-18, due memory constraints on single GPU. Hence, we reduce dimensionality using PCA and decompress after generation. To test the validity of PCA, we trained the DeepWeightFlow models on ResNet-20 and ViT with and without using PCA as shown in Ta- ble 8. We observe that the accuracy and diversity of the neural networks (indicated by the standard deviation in the accuracy) are sufficiently representative of the original sample with or without PCA. This gives confidence that much larger neural networks can be generated by DeepWeightFHl ing PCA. We leave the complete implementation this as future work.
Here have performed incremental PCA that lets us perform PCA in chunks without loading all data into memory, but the math and essential foundation for it is exactly the same as stan- dard PCA. Incremental PCA reduces the dimensionality of the generated weight matrices, start with data of shape (Pramples flat_ -dim), incremental PCA projects it into latent space of size (nsamples, latent_dim). where we set latent_dim 99. Since PCA orders components by explained variance and the rank of the data matrix is bounded by nammlles - 1, at most 99 meaningful directions can exist for 100 samples we used. Therefore, using 99 principal components retains essentially all the variance of the dataset, while compressing the original high-dimensional representation into very compact latent space.
D. DUAL PCA
While we have demonstrated results using incremental PCA for models with tens of millions of pa- rameters, scaling to models with up to 100M parameters introduces significant memory constraints. Traditional PCA algorithms require loading all data into memory simultaneously, which becomes infeasible when analyzing thousands of deep neural network models with hundreds of millions to billions of parameters. In such settings, directly constructing the covariance matrix is computation- ally expensive and ceanory-rooibitive To address this, we exploit the dual PCA formulation, in which principal directions recovered from the eigen-decomposittinn of the Gram matrix rather
20

---

Published as a conference paper at ICLR 2026
than the covariance of the features (Schölkopf et al., 1998; Shawe-Taylors al., all,l,,00000 2005). This approach has been extended to functional and multivariate settings, where the dual eigenproblem provides a scalable approximation to the spectra of covariance operators (Golovkine et al., 2026). By project- ing the data into the space spanned by the memdels samples instead of the original plparams features, the dimensionality is reduced from Mqurams X Parrrns to Pemddels Temodekss mathematically, this is equivalent to standard PCA because the nonzero eigenvalues of the covariance matrix X and the Gram matrix X X coincide, and the principal components in the original space can be recon- structed from the sample- space eigenvectors. To further scale PCA to extremely high-dimensional models, we combine this dual formulation with randomized numerical linear algebra. Specifically, the eigendecomposition of the Gram matrix computed using a randomized SVD scheme, which reduces computational cost while preserving spectral accuracy (Halko et al., 2011). Since storing full datasets or full parameter vectors is infeasible, both covariance and Gram matrices are con- structed incrementally. We build on the principles of incremental and streaming PCA algorithms (Ross et al., 2008; Cardot et al., 2018), adapting them to extremely high-dimensional model param- eters with micro-batch accumulation and GPU-accelerated matrix operations. Model parameters are streamed from disk in batches, enabling PCA on datasets that exceed available memory. Our method performs PCA in four stages (D.1): (1) incremental estimation of the empirical mean, (2) streamed construction of the Gram matrix, (3) randomized eigendecomposition, and (4) vectorized recovery of the principal components in the original parameter space. This results in a scalable PCA framework suitable for analyzing collections of models with billions of parameters, even when the complete dataset cannot fit in memory. Table demonstrates that dual PCA can be effectively used to accurately generate weight spaces for models such as ResNet18 and ViT-Small-192. with parameter counts on the order of (10MM), as well as for larger models like BERT-Bass with up to O(100M) parameters.
Table 9: Flow Matching Hyperparameterss and Performance Results For 100 generated samples projected to 98-99 PCA components using dual PCA
Model
Hidden Dim
Time Embed
Org. Scores
Avg Score
ResNer18 (dotaset: CIFAR-10, metric: accuracy %)
ResNet18
1024
128
94,45 * 0.14
93.52 = 0.16
ViT-Small-P9' -nnaall-12aaa (dataset: CIFAR-IO, metrie: accuracy %)
ViT-Small- 192
512
83.30 0.29
83.83 + 0.I
BERT-BE Base (dataset: Yelp, metric: Spearman's correlation)
BERT- Base
1024
64
0.7902 = 0.0061 0.7909 = 0.005
D.2 NOTATION AND ALGORITIM
Let W = [wi...., Wa] E BRxn denote the weight matrix where n is the number of trained models, d is the number of parameters per model, & is the number of principal components to retain, and w, E Rd is the i-th model's flattened weights. Let W = W - -11" E Bdxx denote the centered weight matrix where μ = A >i w; the empirical mean.
The algorithm consists of four sequential passes:
1. Incremental Mean Computation: Compute the empirical mean in batches to avoid loading all models into memory:
W
T
2. Gram Matrix Construction: Build the X n Gram matrix block-wise, exploiting GPU parallelism while keeping only two micro-batches in GPU memory at a time:
Gy (w: TT) (w, n)), i.j 1....,
3. Randomized Figendecomposition: Compute the top k eigenvectors of G using randomized SVD (Halko et al., 2011):
G USUT, U Σ diag(o1,....
21

---

Published as a conference paper at ICLR 2026
where oj singular values. Since G WTW W is symmetric, eigenvalues are X; =
4. Principal Components in Parameter Space: dimensional space via back- projeerno:c
Recover components in the original d-
P WU E Rdx
Components are computed using GPU-accelerated matrix multiplication and normalized to unit length.
D.2.1
COMPLEXITY ANALYSIS
Time complexity per pass:
Incremental Mean Computation: ((nd) single pass through all data
Gram Matrix Construction: O(n²d) compute n² pairwise inner products
Randomized SVD: (nn²k) randomized SVD with 5 iterations
Principal Components in Parameter Space: (Idkk) - back-project to components
Complexity is practically limited by (n?2d) when k < d. dominated by Gram matrix struction.
D.2.2 EMPIRICAL TIMING ANALYSIS
We conducted comprehensive timing study of our pipeline using single NVIDIA A100 40GB GPU to understand the computational costs of each phase. We analyzed the end-to-end timing for three representative architectures R (1 IM parameters), ViT-Small-192 (5.5M parameters), and BERT-Base (1 18M parameters), each trained on 100 models. All experiments were run on a sin- gle NVIDIA A100 GPU with FP16 precision for Dual PCA implementation. The timing estimates can be found in Table 10 and Table 11.
Table 10: Setup Phase Timing Breakdown NVIDIA A100 A100
Model
Canonicalization
PCA Fitting
Flow Training
Total Setup
ResNet18 (11M)
144s
60s
270s
VIT-Small (2.84M)
1,002s
12s
60s
1,074s
BERT-Bas (118M)
6,900s
360s
66s
7,322s
Setup phase is executed once per model collection (100 models) and prepares the system for subsequent model generation.
Table 1I: Generation Phase Timing per Single Model on NVIDIA A100
Model
Latent Flow
Inverse PCA
Inference*
Total
RosNet18 (11M)
0.032s
0.049s
1.68s"
1.76s
VIT-Small (2.84M)
0.031s
0.015s
1.633s
1.67s
BERT-Base (118M)
0.150s
1.60s
20s
21.75s
Inference includes WSO reconstruction, model loading. and evaluation on test
set.
ResNet18 inference time includes BatchNorm recalibration
D.2.3 SCALABILITY DISCUSSION
The dual PCA formulation is particularly advantageous when d >> n, as the Gram matrix E Roxx is much smaller than the d x d covariance matrix required by standard PCA. This reduces both com- putational cost (from O(nd) to On?2d) for covariance construction) and memory requirements (from (dd)) to ))) With modern high-memory GPUs (e.g., NVIDIA H100 with 80GB HBM3) and FP16 precision, the micro- batch size m can be tuned to balance GPU memory constraints and computational efficiency. The FP16 option effectively doubles these capacity limits while introduc- ing negligible numerical error. As GPU memory and compute continue to improve, we expect this approach scale naturally even larger model collections.
22

---

Published as a conference paper at ICLR 2026
E DATASET GENERATION
Table 12 and Table 13 provide the details of the architecture and training hyperparameters used to create the trained neural network datasets that were used to train DeepWeightFlow. The training datasets be made available request.
Table 12: Hyperparameters for training the neural networks that were used the training datasets for Deep- WeightFlow. Final weights for each seed after the epocks listed in table are treated as a single datapoint. We train 100 such models, using early stopping to halt training when validation performance plateaus.
Model
Dataset
Params
LR Schedule
Optimizer
LR
Weight Decay
Batch Size
Epochs
MLP
Iris
131
None
Adam
le-3
0
16
100
MLP
MNIST
26.5K
None
Adam
le-3
MLP
Fashion
118K
None
Adam W
le-3
0
128
25
SmallCNN
CIFAR-I
12.4K
None
AdamW
le-3
le-3
128
ResNet-18
STL-10
11.2M
Warmup+ Warmup+Cosine
SGD
0.1
Se-4
128
ResNet-18
CIFAR-I 10
11.2M
Cosine
SGD
0.1
5c-4
128
100
ResNet-20
CIFAR-10
0.27M
None
Adam
0
128
Vit-Small-192
CIFAR-BB 10
2.8M
Cosine
Adam
3e-4
0.05
128
300
BERT-Base
Yelp Review
118M
None
Adam
le.4
0
32
Table 13: Model architectures for the neural networks used to train Deep DeepWeightFlow. For MLPs, the first
number in the Architecture definition is the input dimension. For the ResNets, "blocks" refer to residual blocks. For training BERT models, use only subset of the YelpReview dataset for training and testing for this experiment.
Model
Architecture
Parameters
Dataset
Input Dim
MLP
[4. 16
131
Iris
4 150
MLP
1784, 32, 32, 10]
26,506
MNIST
28 28
MLP
[784, 128, 10]
117.770
Fashion-MNIST
MNIST
28
SmalICNN
3 conv, 2. FC
12,042
CIFAR-10
x2
ResNet-20
3 x [3, 3] blocks
272,474
CIFAR-10
x x3
ResNet-18
4 2, blocks
11.17M
CIFAR-10
32 32
ResNet-18
x [2, 2.
blocks
11.17M
STL-10
96 96 x
Vit-Small-199
194 embedding dimension, blocks, heads
2.87M
CIFAR-10
32 32
BERT-Base
768 embed dim, 12 blocks, 12 heads
118M
Yelp Review
128 tukens
The ResNet-20 neural networks used have notably lower parameter counts than the ResNet-18 neural networks, the former is narrower while being deeper reduce model complexity in training for smaller datasets. The ResNet- configuration typical (He et al., 2016). The specific block layouts described in Table 13.
F HYPERPARAMETERS OF DEEPWEIGHTFLOW MODELS
In Table 14 we provide the hyperparameters of the DeepWeightFlow models. The FM model architecture varies by the dimensionality of the neural network weights in the training set and their architecture.
G COMPUTATIONAL EFFICIENCY: TRAINING AND GENERATION TIME
WeightigttFlow demonstrates significant computational advantages existing parameter gener- ation methods. We compare our approach with RPG (Wang et al., 2025), the current state-of-the-art recurrent parameter generation, across multiple architectures and configurations.
When incorporating Git Re-basin (Ainsworth et al., 2023) for weight alignment, the additional com- putational overhead is minimal:
ResNet-18 18: minutes for aligning 100 models
Vitt-mall-- 192 (Transfusion): 13 minutes for aligning 100 models
The results Table 15 show that Deep WeightFlow consistently generates high-quality models while having lower training and inference time similar GPUs.
23

---

Published as a conference paper at ICLR 2026
Table 14: DeepWeighiFlow Flow Matching training hyperparameterss
Parameter
Value
Parameter
Value
Architecture
Training
Flow Model Hidden Dims
Idn, d,/2, d 4-128
Optimizer
AdamW
Time Embedding Dim
Learning Rate
5 x 10 1 10 10-8
Activation Function
GELU
Weight Decay
Layer Normalization
Yes
AdamW B
(0.9, 0.95)
Dropout Rate
0.1-0.4
Batch Size
2-8
Flow Matching
Training
Time Distribution
Uniform / Beta' 0.001 (00, T)
Training Iterations Training Data Size
30,000
Noise Scale (r)
100 models CosineAnncaling
Source Distribution
I.R Scheduler
?/min
1 x 10
Generation
Preprocessing
ODE Solver
Runge-Kutta- 4 100
Weight Matching
Git Re- Ce-BBainn/Transuuse Incremental/Dunt PCA' ResNets only
Integration Steps
PCA Method
Generated Samples
25-
BN Recalibration
dn E {32, 64, 128, 256, 384, 512, 1024) depending on architecture complexity
Time embedding: 4 for MLP, 64 for ResNee-0MMNIST/Fsshioot -iNt:///FaSsiii--MIIIVViii..
192/BERT-Base. 128 for ResNet-18
Dropout: 0.4 for Iris MLP, 0.1 for all other architectures
Batch size: 2 for BERT-Base Basc, for Vit-Small- 192, 8 for all others
0, 0.001 for Vit-Small-199 and BERT-Base, d, 0.01 for all other architcctures Vision Transformers
Git Re-Basin ResNets/MLPs, TransFusion
BERT
BatchNorm statistics recalibrated using test data only for ResNet architectures post-generation
Learning rate: 1 10 for BERT-Base, 5 10 for all others
Time distribution: (255(2,5) for SERT-Base, Uniform for all others
PCA: Incremental PCA (scikit-leam) for ResNet- Saai--888////-aaa 192; GPU-accelerated Dual PCA
(Gram matrix, FP16) for BERT-Base
Generated samples: 25 for Vit-Small-192. 100 for all other architectures
H CHOOSING THE RIGHT SOURCE DISTRIBUTION
The choice of source distribution for these generative models has a significant impact on the per- formance of the generated models. Table Table 16 highlights the importance of selecting source distribution that aligns well with the target distributions to ensure reliable and high-quality weight generation.
I DIVERSITY OF THE GENERATED NEURAL NETWORKS
In Table 17, we provide the numerical estimates of mloU, the Jensen-Shannon, Wasserstein, and Nearest Neighbors (NN) distances between generated and original neural networks, highlighting the diversity of the generated neural networks.
J FINETUNING MODELS FOR TRANSFER LEARNING ON UNSEEN DATASETS
We leverage ResNet-18 models trained and generated on the CIFAR-10 dataset adapt other unseen datasets, specifically STL-10 and SVHN (Table 6). We first evaluate the performance of the generated CIFAR-10 models on these datasets without any fine-tuning (Epoch 0). Subsequently, we fine-tune the models using the standard training set of the target dataset and evaluate them on the corresponding test set. Fine-tuning is performed for up to 5 epochs using the AdamW optimizer with learning rate of 10-4, weight decay of 1 x 10-4, and cosine learning rate scheduler with Tmax = epochs for smooth decay. We use a detach ratio 0.4 (same as used by Saragih et al. (2025b)) and the cross- crosspyntropy loss is used as the objective function. We further experiment with SmalICNN models generated for the STL-10 dataset and transfer them to the CIFAR-10 dataset in similar fashion, comparing our results with those reported by Schürholt al. (2024).
24

---

Published as a conference paper at ICLR 2026
Table 15: Performance comparison between DeepWeightFlow, RPG, P-diff. and D2NWG (Wang al., 2025; 2024; Soro al., 2025). RPG generates single neural network per run, while DeepWeighiFlow generates neural networks sequentially in a single workflow. D2NWG and P-diff only generuie 2048 weights within the pretrained ResNet18 backbone (Soro et al., 2025).
Model
Method
Hidden Dim
Training Time
Generation Time model)
GPU
(sequential)
min
RPG (partially parallel)
1.8 min
I1100
ResNet-18
RPG (fully parallel)
min
11000
(11.7M params)
DeepWeight
min 2 min + min
1.38 seconds
A100
DeepWeightFlow + rebasin
1024
1.38 seconds
A100
P-diff
hours
D2NWG
hours
RPG (flatten)*
6.2 hours
9.8 min
H100
ViT-Tiny
RPG (by channel)
14.2 hours
9.8 min
H100
(5M params)
RPG (within layer)*
6.2 hours
9.8 min
H100
RPG (partially parallel)
1.1 min
H100
RPG (fully parallel)
1.1 min
H100
WeegWeightFlow
256
21 min
2.16 seconds
A100
Vit-Small-192 (2.8M params)
DeepWeightFlow
min
1.70 seconds
H100
DeepWeightFlow transfusion
384
13 min min
1.70 seconds
H100
Available RPG inference
Wang et al. (2025).
RPG training sequential inference time from Wang al. (2025) (Table 4 and Table 18); numbers available for single neural network generation.
Deep DeepWeightFlow performs sequential generation of models. Numbers reported here are for ResNet-18 18 generated using standard incremental PCA and ViT-Small-199 for training and generation without PCA.
P-diff and D2NWG perform only partial generation of 2048 weights within pretrained backbone (Soro al., 2025) (Table 11).
P-diff and D2NWG times reported are likely generating 100 models; divide by 100 for approximate per-model (P-diff: 1.8 min/model, D2NWG: min/model).
Table 16: Evaluating the impact of various source distribution choices FM mapping on the performance of complete weights generated by Deep BeepWeightFlow.
Model & Source Distribution
With Rebasin (%)
Without Rebasin (%)
Vit-Small-199 -92 on CIFAR-10
Original Accuracy
83.29 0.29
Gaussian(O. 0.01)
78.31 +: 10.99
76.69 + 14.37
Gaussian(O, 0.001)
82,90 + 0.70
82.40 = 5.29
MI. MNIST
Original Accuracy
96.32 0.20
Kaiming Initialization Gaussian(O, 0.01)
81.33 + 14.10
67.35 + 26.10
96.18 1 0.23
96.22 1 0.22
VIT: Architecture: Vit-Small-192 (2.7M parameters), Dataset: CIFAR-10, Flow
Hidden Dim: 384, Time Embed Dim:
MLP: Architecture: MLP (26.5 26.5K parameters), Dataset: MNIST, Flow Hidden
Dim: 256, Time Embed Dim: 64 Dropout: 0.1
In these experiments, we evaluate three approaches: (1) random initialization (baseline), (2) di- rect transfer from original pretrained models from the source dataset, and (3) transfer from flow- generated models trained on the source weight distribution. All models are fine-tuned on the tar- get dataset using the same protocol described above. Our results demonstrate that www-generated models achieve comparable or occasionally slightly superior performance to the original pretrained models when transferred to the target domain. This validates that our flow matching approach suc- cessfully captures the essential characteristics of the learned weight distributions, producing high- quality models that preserve transferable features from the source task. The competitive performance of generated models relative to their pretrained counterparts confirms that the flow- based generative process maintains the representational quality necessary for effective transfer learning.
25

---

Published as a conference paper at ICLR 2026
Table 17: Comparison of 100 complete neural network weights generated by DeepWeightFlow with and without Git Re- Re-Basin through maximum Intersection over Union (IoU), Shasnnnnnnnoon, Wasserstein. and Nearest Neighbors (NN) distances. For MNIST, we use MLP with d2 = 512 and 10% dropout. For CIFAR-10. we use ResNet-18 with dr = 1024, Lower scores indicate closer relationships (Org. original, Gen. generated)
Dataset/A Arehtterrche
Metric
Org. to Org.
Org. to Gen.
Gen. to Org.
Gen. to Gen.
MNIST MLP
IoU Wassesstin Jensen- ShasennShamnon
0.8187 0,0385
13.4122
11.6721
Weighteigt w/ Re-Basin
0.7146
0.8326
23.0393 0.2214
9.7232 10.4398
1.7526
1676
11.7407
10.5471
Cosine Sim. L
0.1962
0,2093
0.2093
0.2157
25.5268
25.2278
25.2278
25,1367
ToUl
0. 0.8256 +: 0.0748
Wasscrstein
15.1185
25 6979
17. 17.6939
Weightiigw w/o Re-Basin
Jensen Shannon
0.8181
0.8 0.8322
0. 0.7293
NN Cnsine Sim.
27. 77.4895 + 0.2007 12.3710 + 12 4410
1916 + 0.3753
9. 9.7956
:: 11.2484
0.0088
0.0187
0.0187
0.0189
28.3513
28.1081
1681
28.2423
CIFAR-10 ResNet-18
IoU
0.6289 + 0.0160
Wasserstein Jensen-Shanner
15. 1236
27 27.5994
20.3590
Flightiwt Re-Basin
0,82
8382
0.8 8242
NN Cosine Sim. I.²
27.9643 + 0.0841
13.3136 14. 0490
O. 0.3649 + 0. 0,0836
7. 7.9625 = 12.6314
U.2497
0.2542
0,2542
0.2570
28.9520
28.8494
28.8494
28.8105
IoU
0.6314 0.0198
7654
20,9545
Weightight w/o Re-Basin Jensen-Shamon
0,5018
0.8326
30.2421 0.0766
13.4767 14.8165
0.3667 0.0590
9.
13.7908
Cosine Sim.
0.1754
0.1818
0.1818
0.1832
L
30.3523
30.2332
30.2332
30.
J.1 TRANSFER LEARNING FOR DATASETS WITH DIFFERENT NUMBERS OF CLASSES
We evaluate the transferability of flo-ggenerated neural network weights by leveraging ResNet- 18 models trained and generated on the CIFAR-10 dataset to adapt to the CIFAR- 100 dataset in Ta- ble 18, which presents a significantly more challenging task with 100 classes compared to CIFAR- 10's 10 classes. We compare three approaches: (1) random initialization (baseline), (2) direct trans- fer from original CIFAR-10 10 pretrained models, and (3) transfer from flow-generater models as de- scribed above. For all pretrained approaches, we replace the final fully-connected layer to accom- modate the 100-class output and reinitialize it using Kaiming initialization. We first assess zero-shot performance (Epoch 0), where models are evaluated on CIFAR-100 without any fine-tuning beyond the FC layer adaptation. Subsequently, we fine-tune the models for 1, 5, and 10 epochs using few- shot learning with 50 samples per class from CIFAR-100 100 dataset. Fine-tuning is performed using the AdamW optimizer with a learning rate of 1 10-3, weight decay of 10-4, and cosine annealing learning rate scheduler with Tmmx set to the number of epochs. The cross-entropy loss is used the objective function. This experimental setup allows to assess whether How-generated models preserve transferable representations learned from CIFAR-10 10 and can effectively adapt to the more challenging CIFAR-100 classification task, demonstrating the quality and utility of our generative weight modeling approach.
K
CONDITIONAL GENERATION WITH MODIFIED DEEPWEIGHTFLOW
K.1 MULTI-CLASS GENERATION WITH DEEPWEIGHTFLOW
To demonstrate the ability of Deep BeighhightFFow generalize across tasks, show conditional gen- eration across datasets by operating directly in weight space with simple time and class embeddings at the flow model input (Lipman al., 2023). The models displayed in Table 19 are different from the MLPs described in Appendix E in that they have equal weight space sizes and an identical architecture.
K.2 MULTI-CLASS AND MULTI-ARCHITECTURE CONDITIONAL GENERATION
To adapt DeepWeightFlog for multi-class and multi archte-ach conditional generation, we incor- porated class embedding MLP to produce dense class embeddings, which are concatenated with
26

---

Published as a conference paper at ICLR 2026
Zero-shot performance epoch 0 and fine-tuning results for complete ResNet-18 parameters trained on CIFAR-10 and transferred the CIEAR-100 dataset. The parameters come from Deep Weight-ht- Flow, SANE (Schärholt et al., 2024), Randominit, and a Pretrained Transfer baseline. Randominit denotes a fresh Kaiming-Ho He initialization. Pretrained denotes models first trained on CIFAR- 10 and then transferred to CIFAR- 100. Generated denotes parameters sampled from the respective generative model. Models pretrained on CIFAR-1O (10 classes) have their classification head replaced to accommodate CIFAR-100's 100 classes during transfer learning, while retaining the learned convolutional features. Best scores for each fine-tuming setting are shown in bold.
Table 18:
Epoch
Model
Method
CIFAR-100
scratch
1.00 + 0.00
SANE
Finctuned
1.0 + 0.3
SAN B
1.1 1 0.2
RandomInit
0.98 + 0.06
DeepWeightFlow
Pretrained
1.01 0.17
Generated
1.06 #: 0.26
fr. scratch
1.7
SANE
Finctuned
25.7 - 1.3
5AN Esu g
26.9 1.4
RandomInit
23.36 1.05
DeepWeightFlow
Pretrained
37.03 +: 1.34
Generated
38.37 1.15
scratch
36.5
SANE
Finetuned
45.7 1.0
Esu
45.6 1.2
Randominit
56.79 = 0.69
DeepWeightFlow
Pretrained
67.39 0.38
Generated
67.37 = 0.53
Table 19: Multiclass DeepWeightFlow generation results without PCA compression and with Git Re-Basin
Dataset
Original
Generated
MNIST
96.74 0.25
96.61 0.22
Pashion-MNISS
86.80 = 0.31
86.45 = 0.28
the input and time embeddings. These combined vectors are then fed into the flow model. We be- gan by training single flow matching model generate weights for MNIST and FasSion-MNIST datasets using an MLP architecture that is identical across both datasets. By conditioning on these class embeddings, the single flow model successfully generated weights that achieved good perfor- mance for both datasets.
Next, we attempted train DeepWeightFlow learn multiple classes in the full-rank weight space, which requires that the models have identical parameter counts. While full-rank learning across multiple classes proved difficult, using PCA-reduced weight space allowed the model handle multiple classes and architectures simultaneously. However, the generated models did not achieve extremely high accuracy, as seen in Table 20. A key reason is that FM models perform best when the weight space distribution is smooth and consistent. Introducing multiple architectures or datasets fragments this space, making it challenging for a single learned flow to interpolate or extrapolate correctly. This remains work in progress.
Table 20: Conditional Multiclass Cruss-Architecce Generation with PCA Compression. Shows 4 classes distinct architectures. DeepWeightfloog trained with all classes canonicalized. All values standard deviation. Models were generated with PCA compression.
Class (Dataset)
Original
Generated
Class (MNIST)
96.78 0.23
$4,11 23.88
Class (Pashion-MNISTT
86.82 0.33
43.21 19.65
Class 2 (Tris)
70.23 9.29
53.03 17.37
Class 3 (ReSNe20-CIIFARI0)
73.62 2.24
50.90 31.24
27


Reviews:
[{"review_id": "P43gS53XAo", "reviewer": "Reviewer_8Vjc", "summary": "This submission presents a novel approach for weight generation capable of assembling full neural network models from a diverse set of neural network model checkpoints. This is accomplished using a flow matching approach, which is coined \"DeepWeightFlow\" by the authors. This approach works directly in the weight space and employs model alignment techniques such as Git Re-basin or TransFusion.\n\nExperimental results are reported on models trained on tabular and computer vision datasets. In-distribution evaluations are done against SOTA approaches, where each SOTA approach is trained on ResNet-18 models trained on CIFAR-10, aiming to sample complete ResNet-18 models for CIFAR-10. The same is done for ViT models trained on CIFAR-10 dataset. Out-of-distribution evaluations are done on models trained on multiple datasets, but only for the presented approach and not compared against SOTA approaches except FLoWN. Multiple ablation studies are reported to understand what impact different components of the presented approach have on sampling performance.\n\nI very much like the presented work and think that the in-distribution results are quite impressive, since it evaluates the generated model's performance without fine-tuning. This is very impressive given the diverse model collection used for training of the presented approach. This is particularly important since some SOTA approaches like RGP do generate weights that are similar to the very homogeneous model collection they have been trained on, and break when randomly initialised models are used for training. I truly appreciate the author's sensitivity to this detail in their experimental setup (line 200-202, line 248-251). \n\nRegardless of this, for this submission, I have some concerns given the experimental setup, which are (i) the limitation to only one computer vision dataset in the comparison against other SOTA approaches (Table 2) and missing broad evaluation of other SOTA approaches in their out-of-distribution evaluation (Table 5). I will outline all details in the strengths, weaknesses, and questions section below.\n\nOverall, I think this work provides a sufficient level of novelty and contribution to be accepted, but I would like to wait to have my questions and concerns addressed, i.e., I would be willing to increase my rating during the discussion with the authors.", "questions": "I have some questions:\n\n- **(Q1)**: This is a short one. Why does the paper mention that the ResNet-18 transfer learning experiments are done using PCA as preprocessing (Section 5.2), and for the in-domain experiments, this is not mentioned (Section 5.1)?\n\n- **(Q2)**: The proposed approach works directly in weight space or in PCA-reduced weight space (as far as I understood it). The paper also mentioned that sampling can be done \"without the requirement of additional conditioning during training or inference\" (line 308-309). How do you make sure the method does not overfit to the given training data? Or said the other way around: how homogeneous does your model dataset need to be to perform well?\n\n- **(Q3)**: In a similar spirit, I see that your model dataset is trained only with one specific set of hyperparameters, which is different to, e.g., the model zoos [1] used in some other work. Don't understand me wrong, I appreciate the focus of this work on diversity (wrt. seeds) of your model dataset, but I would be interested in your thoughts on choosing it this way. [1] Model Zoos: A Dataset of Diverse Populations of Neural Network Models, NeurIPS, 2022\n\n- **(Q4)**: What exactly is used for training the underlying FM backbone? The model dataset for training is outlined nicely in the appendix, but how many checkpoints and which checkpoints are you using for the training of your approach (maybe I missed it)\n\n- **(Q5)**: Table 9 in the appendix shows the number of parameters of the model datasets. Why is the ResNet-20 (0.27M parameter) smaller than the ResNet-18 ()11.2M parameter? I guess this is a typo, right?\n\n- **(Q6)**: I would be wondering if the proposed method is able to generalize beyond one image dataset, i.e, what would happen if the proposed method were trained on model datasets trained on CIFAR-10 and FashionMNIST and MNIST. Would this help in transfer learning scenarios?\n\n- **(Q7)**: Same question wrt.. the architecture. Is the proposed method able to generalize to multiple architectures? I mean not to train one FM model per neural network architecture, but to train the proposed methods with multiple architectures. Similarly, is the proposed method able to generate neural networks of different architectures with the same learned FM model?\n\n- **(Q8)**: I think that the result reported for Soro et al (2025) should be moved to partially generate ResNet model weights and not fully generated ResNet model weights. Please correct me if I am wrong.\n\n- **(Q9)**: Finally, you provide efficiency results for RPG but not for D2NWG. Both provide code for their method. Is there a reason why you do not provide numbers for D2NWG?\n\nAs I already said, I think this is fascinating, and I would be happy to increase my rating once I fully understand the generalizability capabilities of this approach.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **(S1)**: The proposed method is able to generate neural networks that do not require any further fine-tuning. I think this is impressive, given the reported performance being close to the original performance of the model dataset. I am speaking specifically about the results in Table 2 and Table 3. I would have some questions regarding generalizability, though. These are outlined in the questions section.\n\n- **(S2)**: I very much appreciate the emphasis of this work on diversity in the model dataset being used for training. I think this is important for the entire model weight generation community. Speaking about this, I think that the results in Table 2 are a little bit misleading as the numbers reported by Wang et al 2025 (=95.1) have been generated by finetuned trajectories of the same seeded neural network model (this is already mentioned multiple times by the authors in the submission). Having said this, I think that the results of this work (=93.47) are more impressive since they are generated using a much more difficult setup, i.e., a more diverse model dataset for training, as the results from Wang et all 2025 (=95.1). I am not sure about the results of Soro et al (2025), to the best of my knowledg,e they only partially generate ResNet weights and do not fully generate a ResNet model. Finally, the results of Schuerholt et al (2024) also have some leakage of information as they use samples from the target model dataset to fit their KDE for sampling.\n\n- **(S3)**: I also appreciate additional results reported on the diversity of generated models and sampling efficiency. This provides some additional information contextualizing the proposed method.\n\n- **(S4)**: It is great to see a reproducibility statement in this work. The weight generation community needs more of this.\n\n- **(S5)**: This paper is a nice read since it is easy to read and follow.", "weaknesses": "- **(W1)**: As mentioned above, in Table 2, I would like to see more comparisons beyond CIFAR-10 results. I think that might provide a more general impression of the proposed methods' performance.\n\n- **(W2)**: In the same sense, for Table 5, I would like to see comparisons against other SOTA methods. This would help to understand the proposed method's capabilities in the context of related work.\n\n- **(W3)**: Since the authors additionally claim that the proposed method is able to scale, I would love to see some scalability ablation results going beyond ViT-Small-192.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6WNgGD9oBp", "reviewer": "Reviewer_ZyLw", "summary": "This work proposes an approach to generating complete sets of neural network parameters using an MLP generator trained via a flow matching objective. Training data was obtained purely by training many separate models on datasets like MNIST, CIFAR10, SVHN, etc. The proposed method is trained to generate weights for different architectures including MLPs, ResNets, and ViTs. \n\nSpecifically, this method uses incremental PCA to reduce the effective dimensionality of the target weight set, then performs flow matching in that space, followed by an inverse transformation (of the pca reduction) to go back to the original weight space. Permutation symmetry is addressed via cannonicalization using appraoches such as git re-basin and transfusion. \n\nThe approach is evaluated on the above datasets for in-distribution performance, as well as oof performance on CIFAR10 -> SVHN and STL-10. The diversity of generated samples is evaluated in terms of their predictive behavior.", "questions": "1) How does this method scale in terms of absolute VRAM numbers? This is usually the limiting factor with these weight generation methods. At some point we need to do a matrix multiplication where one dimension is the full parameter vector. But with this incremental PCA approach scaling might be better?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* I appreciate that the approach is a sensible way of handling the high dimensional weight space i.e. iterative pca -> flow matching. \n\n* I found the paper well-written and easy to follow\n\n* I am impressed that this work is able to handle complete weight sets even with more complicated architectures like ViTs, and those that include batchnorm. \n\n* The results of this work are compelling compared to other previous work like p-diff and flown.", "weaknesses": "* I'm confused about the effect of cannonicalization. Ostensibly it makes training easier but the results shown in figure 2 don't seem to reveal any difference in final test accuracy of generated samples. What is the effect of not using the (computationally expensive) process)?\n\n* I think diversity could be addressed in more than just behavioral statistics. We might also want to know about the average L2 distance between generated and training networks. We might also want to see if for any (or the best) generated network, is there an exact or near-exact copy of it in the training corpus. These datasets are often quite small and memorization is common here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e7mtPgLItu", "reviewer": "Reviewer_mm9F", "summary": "This paper introduces DeepWeightFlow, a flow-matching approach that generates complete neural-network weights. Using optional Incremental PCA for tractability, it scales from small models to architectures with roughly 10M parameters, delivering performance that matches or exceeds prior methods on the same benchmarks.", "questions": "For more question see weaknesses\n\n1.  **Transfer Learning Baseline:** Why does Table 5 not include the transfer performance of the original models from your training set as a baseline? This would clarify if DeepWeightFlow generates models with inherently better transferability.\n\n2.  **Scaling to Billion-Parameter Models:** How do you foresee the IPCA-based approach scaling to billion-parameter models? Is a single linear projection via PCA sufficient and computationally tractable for such a high-dimensional and complex weight space?\n\n3.  **Rationale for Unconditional Generation:** What was the motivation for focusing on unconditional generation? What are the main challenges you see in extending DeepWeightFlow to a conditional framework for sampling models with specific attributes?\n\n4.  **Generalization Beyond Classification:** Could you comment on the expected challenges of applying DeepWeightFlow to tasks beyond classification (e.g., in NLP or regression)? Would the current canonicalization methods be sufficient for the different symmetries in those domains?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "**Originality**\n\n* Direct Flow Matching in weight space (faster than diffusion).\n* Symmetry-aware via Git Re-Basin / TransFusion.\n* Lean scaling with (Incremental) PCA no learned autoencoder.\n\n**Quality**\n\n* Solid results across MLP/ResNet/ViT and datasets.\n* Clear ablations (capacity vs. canonicalization; source distribution).\n* Practical touches (BN stats recalibration) that boost reliability.\n\n**Clarity**\n\n* Well-structured, accessible explanations\n\n# Significance\n\n* Large efficiency gains: Cuts weight generation from hours to minutes, making high-quality ensemble creation computationally practical.\n* Better starting points: Provides strong initializations that accelerate fine-tuning and enhance transfer learning.\n* Scalable foundation:  A robust, scalable framework for generative modeling in weight space that can serve as a strong baseline for future,extensions.", "weaknesses": "* **Unconditional generation limits controllability:**\n  The method’s unconditional design restricts its ability to generalize or transfer across datasets and architectures. While presented as “data-free,” this property also constrains the model’s flexibility\n\n* **Inability to jointly encode across datasets:**\n  Because the generator is unconditional, it cannot represent or sample from mixtures of weights trained on diverse datasets. Each dataset requires its own independently trained flow.\n\n* **Overstated scalability claims:**\n  The paper refers to networks of roughly *10M parameters* as “large-scale.” While this is nontrivial, it falls short of the scales (hundreds of millions to billions of parameters) typically considered large in contemporary neural network research, leaving true scalability uncertain.\n\n* **Unclear handling of label or output-size mismatches:**\n  When transferring to datasets with a different number of output classes, it is not specified how the method manages mismatched output dimensions or classifier heads. This omission raises questions about generalization to tasks with differing label structures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3C95xfoVZI", "reviewer": "Reviewer_xfHk", "summary": "The submission presents a flow-based method for neural network weight generation. The authors identify generation quality, speed and diversity as key metrics to improve. Their proposed method extends existing work with a flow-based generation method, that demonstrates high quality of generated weights, on par with the original weights without the need for fine-tuning at considerably faster training and generation times. The authors also successfully make use of model canonicalization and PCA to reduce the complexity and size of the domain.", "questions": "See weaknesses above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The topic of the submission is a highly relevant topic, fast and high fidelity generation remains unsolved. The application of flows on weights is well motivated as a match for high fidelity in high dimensions at relatively high speed. \n- The experimental evaluation show that the proposed method does indeed generate weights that perform very well, to the level of the original models. At the same time, training the flow models appears to be significantly faster than previous work. While these numbers are hidden in the appendix, they present a significant improvement in training speed (and potentially generation speed) over previous work. The experiment setup here is not entirely clear to me, but if this holds up in practice it seems a big deal for weight generation. For that reason I'm somewhat baffled the authors chose not to highlight it.\n- I particularly appreciated the evaluation on canonicalization, in which regimes it has an effect and where it seems to diminish. \n- The authors make an effort to review the related work and contextualize their method well.", "weaknesses": "While I overall like the proposed method, and it seems to address limitations of previous work, mostly in quality and speed of generation, there are weaknesses that prevent me from scoring the submission higher at this point. I encourage the authors to address them, and am willing to improve my score. \n\n- Quality, model size and generation speed are relevant problems, and need to be considered in new weight generation methods. The experiments focus on evaluating original vs generated model performance as a proxy for quality, where previous works like D2NWG or RPG [2, 3] appear to perform similarly. The authors further argue with differences in the experimental setup or generation speed, but I could only find generation times for RPG and DeepFlowNets in the Appendix. This comparison appears to take RPG generation times and extrapolate by 100. I can't find any reference numbers to match the author's, so I would ask them to make their calculations transparent. I understand there is an argument to be made for parallelization, but I in general don't see the value of generating 100 models for the same task. If parallelization is the strength here, then I'd ask the authors to add the 1 model vs 100 model generation to extract that signal cleanly.  \n- As the authors note, experiment details matter for parameter synthesis and have made comparisons in previous work challenging. Therefore I would ask them to make their experimental setup clearer. On what and how many model checkpoints is DeepWeightFlow trained? What varies between the checkpoints in the dataset per experiment? Is there one global flow model, or one model per experiment? \n- Where and how is PCA used exactly? If using PCA is the main driver for training and generation speeds, I'd encourage the authors to run ablations so that readers understand where the speed improvements come from. If - as claimed - PCA provides a general compression of significant quality, then improvements should carry over to other methods. That said, previous work using auto encoders generally only achieved high compression ratios for small datasets or datasets with relatively low amount of diversity (shared seeds, etc), so I'm somewhat skeptical that linear compression can achieve as much. \n- The authors use the IoU to show diversity. However, it seems from their experiments that the generated models are substantially less diverse than the original models. While it's in the same diversity range as previous parameter generation methods [1], the experiments aren't directly comparable. Notably, the diversity trends from original to generated are reversed. This does not need to be an issue, but if the authors aim at diversity, this requires some discussion. Also, the authors chose to show diversity on small MLPs and MNIST. Since scalability is one of the main arguments, I encourage the authors to evaluate diversity on larger models and harder tasks.\n- Ultimately, the goal for weight generation has to be to go beyond the original models. This can mean generalization beyond the original data, but might also be a recombination of dataset/architecture/size/knowledge. Here, the contribution is rather narrow. Other work on weight generation has trained on diverse model datasets to cover a broader distribution or introduced generation conditioned on data [3, 4, 5]. I would encourage the authors to discuss how their model helps make progress towards real-world applications.\n- Minor issues in the related work: Permutation symmetries have been used as data augmentations for general in [6, 7]. Graph-based models are used successfully in the INR community, but [2,8] aren't graph auto encoders as far as I can tell. Neither seem to use any connectivity information.  \n- NIT: Commonly, hypernetworks mean generator-only models that are trained to synthesize the weights of a target model using target model signal. That seems different from the representation learning methods that operate on weights and only use weight signals. \n\n[1] Wang et al, 2024: Neural Network Diffusion.\n[2] Wang et al., 2025: Scaling Up Parameter Generation: A Recurrent Diffusion Approach.\n[3] Soro et al, 2024: Diffusion-Based Neural Network Weights Generation.  \n[4] Falk et al., 2025: The Impact of Model Zoo Size and Composition on Weight Space Learning.\n[5] Falk et al, 2025: Learning Model Representations Using Publicly Available Model Hubs.  \n[6] Schuerholt et al, 2021: Self-Supervised Representation Learning on Neural Network Weights. \n[7] Peebles et al, 2022: Learning to Learn with Generative Models of Neural Network Checkpoints. \n[8] Schuerholt et al, 2022: Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)