Diffusion-based Neural Network Weights Generation
Authors: Bedionita Soro
Bruno Andreis
Hayeon Lee
Wonyong Jeong
Song Chong
Frank Hutter
Sung Ju Hwang
Venue: ICLR 2025 Poster (2025)
Keywords: generative hyper-representation learning
diffusion model
neural network weights generation
parameters generation
hypernetworks
Primary area: generative models
TL;DR: D2NWG

Abstract: Transfer learning is a cornerstone of modern deep learning, yet it remains constrained by challenges in model selection and the overhead of extensive model storage. In this work, we present Diffusion-based Neural Network Weights Generation, D2NWG, a novel framework that leverages diffusion processes to synthesize task-specific network weights. By modeling the distribution of weights from a diverse ensemble of pretrained models and conditioning the generation process on dataset characteristics, task descriptions, and architectural specifications, D2NWG circumvents the need for storing and searching through massive model repositories. We evaluate D2NWG across multiple experimental settings. On in-distribution tasks, our framework achieves performance that is on par with or superior to conventional pretrained models, while also serving as an effective initialization strategy for novel domains, resulting in faster convergence and a 6\% improvement in few-shot learning scenarios. Extensive ablation studies further indicate that our approach scales robustly with increased diversity and volume of pretrained models. Moreover, D2NWG demonstrates significant promise for large language model applications. In evaluations on the OpenLM leaderboard, our method improved LLaMA-3-2-1B-Instruct performance by 3\% on challenging mathematical reasoning tasks, with a consistent gain of 0.36\% across a range of benchmarks. These findings establish D2NWG as a versatile and powerful framework for neural network weight generation, offering a scalable solution to the limitations of traditional transfer learning.

Published as a conference paper at ICLR 2025
DIFSUSION-BASED NEURAL NETWORK WEIGHTS GENERATION
Soro Bedionita Bruno Andreis Hayeon Lee!
Wonyong Jeong?
Song Chong'
Frank Hutter² Sung Ju Hwang
KAIST University of Freiburg ADeepAuto.ai
ABSTRACT
Transfer learning is a cornerstone of modern deep learning, yet it remains con- strained by challenges in model selection and the overhead of extensive model storage. In this work, we present Diffusion-basee Neural Network Weights Genera- tion, D2NWG, novel framework that leverages diffusion processes synthesize task-specific network weights. By modeling the distribution of weights from a diverse ensemble of pretrained models and conditioning the generation process on dataset characteristics, task descriptions, and architectural specifications, D2NWG circumvents the need for storing and searching through massive model repositories. We evaluate D2NWG across multiple experimental settings. On in-distribution tasks, our framework achieves performance that is on par with or superior to conventional pretrained models, while also serving strategy for novel domains, resulting in faster convergence and a 6% improvement in few-shot learning scenarios. Extensive ablation studies further indicate that our approach scales robustly with increased diversity and volume of pretrained models. Moreover, D2NWG demonstrates significant promise for large language model applications. In evaluations on the OpenLM leaderboard, our method improved LLaMA-3-2-1B 1BB II-trruttct performance by 3% on challenging mathematical reason- ing tasks, with a consistent gain of 0.36% across range of benchmarks. These findings establish D2NWG as versatile and powerful framework for neural ner- work weight generation, offering a scalable solution to the limitations of traditional transfer learning.
effective initialization
INTRODUCTION
Diffusion-based generative models have emerged as a breakthrough technology in artificial intelli- gence, achieving state-of-the-ar performance in generating complex, high-dimensional data across domains including natural language, audio, images, and video (Gozalo- -Bizuull & GarridoMerchhae 2023). The success of these models stems from their principled approach to data generation through iterative denoising (Ho et al., 2020b; Rombach et al., 2022; Peebles & Xie, 2023; Gao et al., 2023), which has proven remarkably effective for modeling complex probability distributions and generating high-quality samples (Yang et al., 2024). Despite significant advancements generative modeling, a fundamental challenge remains largely unexplored: can diffusion models be leveraged directly generate neural network weights from pretrained models? Successfully addressing this question could reshape core machine learning paradigms, particularly transfer learning and AutoML (Hutter et al., 2019; Doke & Gaikwad, 2021). By synthesizing task-specific network parameters on demand, we could circumvent the computational inefficiencies of traditional fine-tuning while enhancing adaptability to novel tasks.
Recent efforts in weight generation, such as generative expeerrepresennttiion learning (Schürholt et al., 2022a), have only begun to tackle this challenge. Current approaches, including Neural Network Diffusion (Wang et al., 2024) and kernel density estimation based methods uurhol et al., 2024),
'Equal Contribution. Correspondence to: Soro Bedionita <sorobedio @ kaist Bruno Andreis <andries @ kr. Hayeon Lee <hayeon9266 werooonnn@@aiiitccc.c.oaa krs>, Wonyong Jeong <wyjeong krr>, Song Chong <songchong @ kar>t,ackrs>, Frank Hutter cch@cs.unffrihurg.coo Sung Ju Hwang <sjh- wang82 @ kris

---

Published as a conference paper at ICLR 2025
Training Stage Ereecing
Sampling Stage
Weige Sooding
Ereader Ascsiiicee Gassderr Weight Dacoder
I
D
Vesteriaed Weighes
Sunglee Weigres
Figure 1: Stage 1: VAE Encoder and Decoder training process. Stage 2: dataset encoder training
stage. Stage 3: Dataset conditioned diffusion process.
exhibit significant limitations in both scalability and generalization. These methods are primarily constrained to small architectures and focus on unconditional weight generation within predefined distributions, neglecting the essential problem of generating task-specific weights for novel scenarios from diverse pretrained model distributions. While these techniques yield performance gains on familiar tasks, their inability to generalize effectively to unseen domains diminishes their practical applicability in model selection and transfer learning. Meta-learning approaches (Nava al., 2023; Zhang et al., 2024) have made strides in addressing weight generation for visual and few-shot learning tasks. However, these methods remain limited in their ability produce dataset-conditioned solutions, as they often rely on learned priors that do not fully exploit the richness of pretrained model distributions. Consequently, the development of principled approach synthesizing neural network weights in a dataset-aware, scalable manner remains an open research question with profound implications for efficient model adaptation and deployment.
To address these challenges, in this work, we introduce Diffusion-based Neural Network Weight Generation (D2NWG). ). novel approach that leverages latent diffusion to generate neural network parameters by learning from diverse pretrained model weights. Our method re- fe-oomullaees the latent diffusion paradigm for weight generation by incorporating robust dataset and task conditioning capa- bilities. D2NWG learns the distribution of weights from diverse architectures and pretraining datasets conditioned on dataset or task description enabling tataset/taas--seecffff weights generation during infeence while the generated weights maintain performance comparable to individual pretrained mod- els on in-distribution tasks and fast converge when fine-tuned on unseen datasct/task. Additionally, analysis reveals that the diversity and size of the pretrained model training set strongly correlates with improved generalization to unseen datasets and tasks. Our empirical evaluation validates the contribution of D2NWG as follows:
The generated weights match or outperform traditional pretrained models on seen tasks while enabling faster, better learning on new tasks through superior weight initialization.
D2NWG outperforms recent meta-learning Zhang et al. (2024) approach on few-shot setting as well as recent weights generation methods Schürholt al. (2022b); Sch"urholt al. (2024).
enabless learning from a distribution of diverse pre-trained models, each trained on different datasets while matching individual pretrained model performance.
D2NWG scales to small and large datasets, generating weights for architectures with over 400 million parameters including GPT2-Small.
We demonstrate its effectiveness in improving LLM performance by generating task-specific weights from a single pretrained model and sampled weights based LLAMA3-11- 8B and LLAMA3-22 IB models ranked among the top performing models on the open Im-leaderboard
httpss://hhggggg/eeeeeeee co/ hpac/spacs///////aasooocc coen- n-1 1 leadebborrd/open_] leaderboard
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Neural Network Parameters Prediction: As neural networks expand across domains, transfer learning through pretrained weights has become crucial. While hypernetworks have emerged as a promising approach for weight prediction (Chauhan et al., 2023; Ratzlaff & Fuxin, 2020; Denil al., 2013; Ha et al., 2016), subsequent Graph Hypernetworks (GHN) methods leverage model architecture graphs to generate weights (Zhang et al., 2019; Knyazev et al., 2021; Zhmoginov et al., 2022; Knyazev et al., 2023). Though recent transformer- saassrreer-baeedd approaches treat weight generation as autoregressive process (Zhmoginov et al., 2022), these methods remain constrained by their single-task focus, limiting their transfer learning capabilities. Similar to GHNs diffusion models has been used to generate weights in meta learning setting Nava et al. (2023); Zhang et al. (2024). However, the generated parameters are not task-specific and the generator is limited to the classier head.
Parameters Generation from Pretrained Distribution: Parameter generation from pretrained distributions has emerged as promising research direction due to its practical applications. However, existing approaches (Schürholt et al., 2021; Schürholt et al., 2022a; Peebles et al., 2022; Sch urholt al., 2024) remain constrained by their focus on single-dataset parameter learning, leaving the broader potential of cross-domain applications largely unexplored.
Applications of Parameter Generation in LLMs: Despite minimal exploration around learning from pretrained weight distributions, we show in this work that our approach generates diverse task-specific weights for LLMs (Minaee et al., 2024; Zhao et al., 2023; Dubey al., 2024). By generating specialized LoRA modules (Tang et al., 2024; Zhao et al., 2024; Gong et al., 2024), we can enhance model flexibility and transfer learning while reducing computational costs.
APPROACH
3.1 PRELIMINARY
Let's consider a collection of neural network models {A,] each pretrained on one of M distinct datasets (D1, 2...... Dar).
Our primary objective to characterize and learn the underlying distribution p(W) of the pretrained model weights W across this ensemble. Ultimately, we aim to develop a method for conditional sampling of weights p(Wr(DT) that are optimized for any target dataset or task Dr( y), regardless of whether it appeared in the training distribution. These sampled weights should either achieve strong performance on Dr immediately( or require minimal fine-tuning compared to random initialization). The intuition is that there is direct relationship between a pretrained network weights and the dataset it was trained on (see Appendix A. for a formal argument). We argue that this relationship constrains the high-dimensional weight space W Th" to a lower-dimensional manifold M C W with dimension & < n. This hypothesis is supported by the Lottery Ticket literature (Frankle & Carbin, 2019; Liu et al., 2024), which shows that sparse subnetworks can match full network performance: 2) 2 C(0 @ m; D), where m E {0, 13"h is sparse mask. By Whitney's 's Embedding Theorem (Whimey, 1936), M can be smoothly embedded in 2 via a diffeomorphism Φ M + 2, where Z represents a latent space. We approximate this embedding using a variational autoencoder((AAl (VAE). Given the differentiability of 2, we can employ latent diffusion to model the distribution of pretrained weights. This enables our proposed D2WNG framework to not only preserve individual model performance but also generalize to unseen datasets incorporate more pretrained models, leveraging the smoothness and interpolation properties of the latent space. Later, we investigate some possible way to improve LLMs without fine-tuning through through sampling in latent space with D2NWG. In this paper, seen dataset/task and unseen dataset/task to refer to datasets or tasks that are present in or absent from the training set, respectively and Zero-shot means no finetuning is performed on sampled weights and are directly evaluated.
the terms
3.2 WEIGHT ENCODING
Let {A)) be a set of pretrained models. For each A,, we flatten its weights to W; Rd where d; is its parameter count and we define dmax = max; d, We zero-pad each W, to obtain W, E Rdes, giving uniform-length representations (Figure 1) to which we refere to model-wise vectorization.
3

---

Published as a conference paper at ICLR 2025
This setting is suitable for small models and classifier layer adaptation. On the other hand, Layer-wise vectorization keeps each layer's weights separate rather than concatenating them. Each flattened weight vector R is aero-padded to match chosen chunk size multiple, then split into & equal-length subvectors W; E TI' where = [mn/k]] /k]. This enables independent layer-wise sampling during inference, where each vectorized layer serves separate input for subsequent stages. This setting is suitable for large models.
Parametrs Encoding: We then train Variational Autoencoder (VAE) to encode these vectors and minimizing the following objective function:
C = - [log (o(w]=)] + 3KL (=) [|](2)]
(I)
where is the vectorized weight, is the latent representation, Po and qo the reconstruction and approximate posterior terms respectively, p(=) the prior distribution (a Gaussian prior) and is a fixed hyper parameters that regulates the stochasticity of the VAE. Model-wise and layer-wise vectorized parameters are encoded using the same VAE structure, with the only difference being in the input dimensions. In chunk-wise encoding, the original flattened vector is recovered by reassembling the decoded latent chunks through concatenation. The reconstructed chunks w from each layer are concatenated to recover v 11 @ úy 4 + dxg. where @ denotes concatenation. And reshaping v back into the original form W yields close approximation of the original weight W. The quality of reconstruction is assessed by evaluating the reconstructed weights on a designated evaluation dataset or task.
3.3 DATASET ENCODING
Image Dataset Encoding: We adopt Set Transformer- based encoder (Lee et al., 2019a) T to encode the pretraining datasets. This approach effectively handles large, multi-class datasets and has been validated in prior dataset-adaptive methods (Jeong et al., 2021; Lee et al., 2021). Figure appendix provides
the architectural overview of the dataset encoder. Given a dataset with C' classes denoted by 2 = {x,, 31) where 2,, and y/ denote inputs and labels, we use a pretrained clip image encoder to extract the images features and group the data into subsets s; by class, forming Here K; is the number of images belonging to class i, and feat the features dimension. Each subset transformed into embeddings Ze, E 1xd using a transformation T, and these embeddings are aggregated into & RCdd Another transformation T produces the final dataset encoding DD R represented as: 25 T o T(S) This encoding is invariant to the number of classes and dataset size, and it operates without utilizing labels. We train the dataset encoder 'T using a contrastive loss to align dataset embeddings 2D, with pretrained weight embeddings 2; following the CLIP-style approach introduced in HyperCLIP (Nava et al., 2023). This alignment ensures training stability and computational efficiency during diffusion optimization. Specifically, we optimize the following objective:
S S= { with 8; E CRCK
exp(2: . 2D./T)
LCUP - log En exp(2: 2D.//)D//)
where ED, is the dataset embedding for Di, and zi is the corresponding VAE-encoded weight embedding (Section 3.2). This alignment enables efficient probing and integration into downstream tasks.
Language Task Encoding: To enable lesk-description-aate based parameter generation for NLP tasks, we first encode each task description using Maamaa3--B--nsttiee The output from the last hidden layer is used as the task's dataset embedding. These embeddings are then directly incorporated into the diffusion process during both training and inference.
3.4 DATASET-CONDITIONED PARAMETERS GENERATION
At this stage, we have access to a pretrained VAE for encoding neural network weights and pretrained Set Transformer module to encode entire datasets. The next stage involves defining a model to generate latent representations of weights conditioned on the dataset embeddings. We achieve this by using diffusion probabilistic models (DDPM) (Ho et al., 2020a; Rombach et al., 2021) trained the latent representation of the pretrained weights..
4

---

Published as a conference paper at ICLR 2025
Forward Process: Given weight embedding 2, obtained from the encoder of the pretrained VAE, the forward diffusion process involves successive Gaussian noise perturbations of over time steps. At time step t,
p(z (2 / 1, 1,1)
where B. E (0,1) is the noise variance and p(21:1 (20) = [I p(2e|ze-1).
Reverse Process: As in most DDPM approaches the reverse process is approximated by neural network such that:
po(2)112)) = (2(2 Eo (2),
where ps and Eg are neural networks.
The diffusion model is trained on the VAE embeddings 2, condi- tioned the dataset embeddings concatenated with the latent representations of the weights. To leverage existing architectures, we designed the VAE to generate latent representations that are compatible with standard latent diffusion models with minimal adjustments, optimizing the latent diffusion objective defined in Eq. 5.
Conditiooon Training:
LLDM = Eo,. [[]= ( 20. t)]
(5)
where E (21, 2D, t) is implemented as a UNet.
Sampling: New weights are sampled conditionally through the reverse diffusion process follows:
Be
21
(2t
VT Ev t..tt) + of.
(6)
where & ~ (O, 1) and, or a chosen value. After sampling a latent representation (a for given dataset D:). The pretrained VAE decoder is used transform these latents into weight vector V = D22), which is then used to initialize the target network as shown in Figure 1.
3.5 EXPLORING THE OPTIMAL PARAMETERS SPACE OF LLMS
In this section, we extend our method to enhance pretrained LLM performance without fine-tuning by recasting D2NWG as layer-conditioned parameter generation approach. The key challenge is managing the vast parameter space of LLMs. Drawing from (Hartford et al., 2024), we use the Marchenko- Pasttr distribution to identify crucial layers for improving the performance based weights spectrum. We calculate a signal-to-noise ratio (SNR) to distinguish significant weights from
Ed1leg12 noise as: SNR Σ. Le.lee "..' where eigenvalues d, above threshold represent meaningful signals, while those below are considered noise. For this task, we employ layer-wise chunking to manage large layers. We provide more detailed in Appendix A.3 and A. Additionally, we present sequential optimal space exploration algorithm, detailed in Algorithm
EXPERIMENTS
We evaluate our method both with and without finetuning on Few-Shot Learning, Zero-Shot Learning (no fine-tuning), and Model Retrieval tasks, All experiments use single Titan RTX GPU except experiment with LLMs which used single A.000 GPU. Detailed ablation studies are provided the Appendix C.
4.1
WEIGHT GENERATION WITHOUT FINETUNING ON UNSEEN TASK
We present a set of results where the generated weights are evaluated directly without finetuning for few-shot learning and transferring unseen Tasks.
4.1.1 WFIGHTS GENERATION FOR FEW-SHOT LEARNING
Task: We aim to show that learning the distribution of models pretrained independently large set of dataset can enable sampling weights that compete with meta-learning techniques in multi-task few-shot learning, without requiring fine-tuning.
5

---

Published as a conference paper at ICLR 2025
Table 1: Few-Shot Learning. ALL implies generation of the entire parameters and CH denotes
generation of classification head only.
Method
Adapiation
Backhome
Nete-ImageeNe
Nendd Fered-ImaeeNet
5-way
5-way 5-shol
5-way
T-shox
5-way
IMAML (Kajeswaran ot 2019)
ALL
Cone4
49.30 = 88%
59.77 0.0. 737%
38.64 = 1.37%
00.24 = 0.76%
ALFA (Baik ct al., 2020)
ALL
Conv4
50.58 = 0.51%
69.12
11122:000 0.41%
53.16 4: 0.49%
70.54
0. 0.16%
COMLN (Deleu et ol., 2022>
CH
Conv4
63.01 0.62%
70.54
0.5444
54.30 + 0.09%
71.35 0.57%
MctQDA (Zhag et al., 2021)
CH
Conv4
56.41 = 0.80%
72.61
58.11 - 0.48%
74.28 28
0. 0.73%
MetaDiff (Zhang et al,, 2024)
CH
Conv4
55.06 0.81%
73.18 + 0.64%
57.77 + 0.90%
75.46 0.69%
(2us)
Conv4
61.13 8.50%
6.04% 76.94
65.33
6.50%
8.25%
ALFA (Baik 2020)
ALL
ResNet12
59.74
0.49%
77.96
11.96-10..400 0.41%
64.62 1 0.49%
82.48
0.38%
NetsOpiNet (Lee et al. 2019b)
CH
ResNet12
62.64 =: 0.61%
0.46% 78.63
65.99 1: 0.72%
81.56 56
0.53%
LEO (Rusu 2019)
CII
WRN 28. 10
61.76
0.08%
0.12% 77.59
66.31 0.05%
81,44
0.08%
Classifier (Chen et al., 2021)
CH
ResNet12
61.22 = 0.84%
78.72
0.60%6
69.71 1 0.88%
83.87
0.64%
MetaQDA (Zhang et 2021)
CHI
ResNet18
65.12 0.66%
80,96
0.75%
69.97 0.52%
85.51
0.58%
MetaDill (Zhang et al., 2024)
CH
ResNet12
64.99 = 0.77%
0.56% 81.21
72.33 1: 0.92%
86.31
0.62%
D2NWG(Ours)
CH
ResNet12
69.55 3.77%
83,51
6.21%
81,15 9.70%
90.04
6.10%
Table 2: Zero-Shot Transfer Learning. We evalutate on two backbones: Tiny Swin Transformer and ResNet 88
Model
CITAR 10
STL-10
Aircraft
Pets
CITAR-100
Swin
7.38
5.01
1.35
GHN2 (Knyazev et ol., 2021)
48.20
12.7
GIIN3 (Knyazev 2023)
11.9
D2NWGi(Urs)
53.12 0.25
60.42 = 0.14
24.57 2: 3,16
26.47 + 1.90
30.44 = 0.15
10.88
3.75
1.38
GHN2 (Knyazev al,, 2021)
19:52
13.04
D2NWG
2.13
17.29+
0.13
D2NWG_CLPPP CLi)(OUr))
60.42.1 0.75
82.42 1. 0.04
27.70
13.24
32,17
16.30
51.50
10.25
Dataset: We utilize the mini-ImageNet and fiered-ImageNar datasets for this task. For the architec- four-layer ConvNet and ResNet12 backbone provided by Chen al. (2021). We generate the pretrained weights by linear probing a classifier head on each of the 50,000 subsets for 10 epochs and evaluate the performance on 600 subsets from the unseen test split for 1-shot and 5-shot. Analogously to few shot learning, we choose the number of images per class for conditioning be the same as the support set, while the number of images per class in the query set is fixed to 15 for all methods and 600 tasks are used for testing.
tures,
Baselines: We benchmark against iMAML (Rajeswaran et al., 2019). ALFA (Baik et al., 2020), COMNL (Deleu et al., 2022), MetaQDA (Zhang et al., 2021), MetaDiff (Zhang et al., 2024), MetaOptNet (Lee et al., 20) and a classifier baseline introduced in Chen et al. (2021).
Results: Table shows that our approach consistently improves performance on all tasks while utilizing the same backbone as other methods. With the Conv4 backbone, we achieve approximately 6% performance improvement in I-shot learning and 3 to 4% on 5-shot learning on mini- mini-ImageNNe. On Tiered- Imrred-,, we achieve more than 8% performance improvement on 1-shot and 5 to 6% average improvement 5-shots. For the ResNet12 backbone achieve 4 to 9% performance improvement. These results demonstrate the effectiveness of our method against the existing meta- learning methods.
For evaluation, we perform 50 weight sampling iterations per subset and report the average of the top 3 accuracies. We explore both 1-shot and 5-shot settings, ing one and five images per class respectively for conditioning from support set. Our dataset-conditioned weight generation enables effi- cient task adaptation by producing weights specialized to each dataset's characteristics, achieving superior general- ization compared to meta-learning baselines.
Table 3: Model Retrieval via Generative Augmented Weight Sampling
Domain
Pretrained
(2NsG(Ous))
Large Animals
71.11 :: 11.45
70.33 1: 12:42
4.1.2 ZERO-SHOP CLASSIFIER HEAD ADAPTATION
Small Animals
54.04
13.56
$4.70 $ 13.83
Planss
9405
Diseases
81.69
19
Task: We evaluate the performance of the proposed method in adapting the classifier head to unseen datasets, In this experiment, we assess whether our method can conditionally generate the classifier weights, potentially eliminating or significantly speeding up the finetuning process.
Microscopy
Remote Sensing
82,20
7.49
82.68 K.05
Vehicles
57,07
19.57
58.09 :: 18:30
Manufacturing
84.34 01 21.00
$4.32 J: 20.96
Homon Actioni
68.63
12:45
60.00 + 12.73
OCR
68.60
Average
68.32
Runtime
6horrs
seconds

---

Published as a conference paper at ICLR 2025
Dataset: We partitioned ImageNet- Ik into 20k subsets of 10 to 50 classes each with 50 images per class per subset and linear probe a classifier head for 10 epochs using Tiny Swin Transformer (denoted Swin in Table 2), and ResNet18 all pretrained on ImageNet-1k k. For dataset conditioning, we use 5 images per class per subset. The unseen target datasets are CIFAR-10, STL-10, Aireraft, Pets, and CIFAR-100 The baseline methods in these experiments are ResNet18 and Tiny Swin Transformer pretrained on ImageNet-Ik.
Baselines: We benchmark against the pretrained backbones, and two GHN models (Knyazev et al., 2021; 2023). Additonally, we provide a powerful variant of our model D2NWG CLIP where the dataset encoder encodes the CLIP embedding for each sample in the datasets.
Results: Table 2 presents the performance of the sampled weights where it can be seen that the proposed method achieves better performance compared the ImageNet pretrained weights and the GHN family of models. Additionally, the variant of our model that utilizes the CLIP embedding for dataset encoding significantly improves the performance suggesting that better dataset representation learning can boost the performance of the generated weights.
4.1.3 IN DISTRIBUTION FULL MODELS WEIGHTS GENERATION: MODEL RETRIEVAL
Task:
We assess the Generative Augmented Retrieval capability of D2NWG, aiming to show that it can learn the distribution of models pretrained on diverse real-world datasets. This task requires generation of dataset-conditioned weights that achieve performance comparable to the original pretrained models and hence provide access to wide range of pretrained models through efficient sampling.
Dataset: We collected 30 real-worke datasets(Ullahh et al., 2022), spanning 19 to 706 classes and organised into 10 domains with 3 datasets per domain, and fine-tuned a MobileNetV3 subnet sampled from OFA (Cai et al., 2020) for 100 epochs each dataset. We then learned the distribution of the combined pretrained models from the last 20 epochs across all datasets.
Baselines: For this task, compare with the original pretrained weights which are finetuned on each individual dataset. For each dataset, we sample and report the average accuracy of 5 set of weights sampled with D2NWG.
Results: From Table we see that D2NWG conditionally generates high-performing parameters while enhancing the pretrained model, achieving the best average results across all datasets. This demonstrates the strong retrieval capability of our method, suggesting it can be used as neural network weight retriever approaches like (Zhao al., 2024), eliminating the need for pretrained database. Detailed dataset information is provided in Table 12 and further experiments in Ap- pendix C010. Additionally, is much more efficient to generate weights with our model compared to pretraining as shown by the runtime in Table 3.
4.1.4 TRANSFERRING TO UNSEEN ARCHITECTURE
We investigate weight transferability across ResNet architectures by mod- eling the distribution of pretrained weights from ResNet32 (trained on CIFAR-10 and 100).. We propose a weight initialization method that leverages pretrained weight distributions from ResNet32 improve performance across different ResNet architectures. Our approach sam- ples and concatenates weights from the source model while preserving layer-type correspondence, effectively handling varying network dimen- sions. Experiments on ResNNe20//4456/32 demonstrate consistent im- provements over random initialization, even without fine-tuning, partic- ularly on CIFAR-IO classification tasks shown in Figure 2.
Figure 2: Performance evaluation with unseen ar- chitectures CIFAR-10.
4.2 WEIGHTS GENERATION WITH FINE-TUNING
In this section, we evaluate the quality of the generated weights in fine-tuning scenarios to assess their suitability for transfer learning.
7

---

Published as a conference paper at ICLR 2025
90
70
50
80
Accuracy
C7
70
Accura
40
Accuracy
60
60
40
50
80
40
Pretrained
20
Pretrained
20
Pretralnndd Randinit Ours
Pretrained
Randinit
10
Randlnit
10
Randlnit Ours
20
Ours
Ours
10
30 Epoch
20
40
50
10
20
30
40
50
10
20 Epoch
30
40
50
10
20 Epoch
30
46
50
Epoch
(a) Dessert Food
(b) Gemstones
(c) Japanese
(d) Colorectal
Dataset
Dataset
Characters
Histology
Figure 3: Average accuracy evolution fine-tuning for epochs with sampled weights for unseen datasets.
4.2.1 WEIGHT GENERATION WITH FINE-TUNING ON SEEN TASKS
The goal is to assess the behavior of the sampled weights when finetuned the same dataset and compare convergence speed. This experiment fo- cuses on evaluating whether the sampled weights can be effectively fine-tuned to achieved superior final performance, rather than simply aiming for weights producing high initial accuracy and may not lead to superior performance while tinettuning
Task:
Table 4: Finetuning of Generated Weights using the Modelzoo of Schürholt et al. (2022c).
MNIST -07.5
SVIN
CIFAR-1
-1075
68.626.2
$4.525.0
0b
nt
SaN
$68.003
207:14
563166
W.
SAMEan
367:20
72,3:16
579402
435=110
96522003
655.077
SU80:001
44.010.11
S3711-3
600116
00
SANEXOEM
35.500
71.5s14
5822602
435500
SAN
S75304
713214
191163
443110
Datasets: We used the modelzoo of Schürholt et al. (2022c) consisting of ConvNet trained on MNIST, SVHN, CIFAR-10 and STL-100 Our model was trained on the combined pretrained weights from epochs 21 to 25 of all models, consistent with the baseline settings.
D2NWG
878:00
756:13
5921433
44N-02
45.5
316842
$1.500
72:1112
S88104
452198
SANE
390220
736615
453:D9
D2NWG
925119
748:001
ML3181
43.41.11
Ravdemlnit
572163
92,01033
25
SAME
92.310.4
75.1210
S1216.1
480204
DENWNG
962::03
757:405
641111
487:05
We compare against the kernel density estimator approaches from Sch" urroott et al. (2024); Schürholt et al. (2022b), evaluated on the same datasets. Unlike these unconditional methods, we build model specifically for MNIST and SVHN, and another for CIFAR-10 10 and STL--10. For each dataset, five sets of weights were sampled to initialize the models, which were fine-tuned for number of epochs from 0 to 25. We also add RandomInit model trained for 50 epochs and show that our sampled weight finetuned for 25 epochs outperforms this model.
Baselines:
50
Racdontat
91.1226
707288
$1,5607
47,609
Sch"urholt
Results: As shown in Table 4, D2NWG consistently accelerates convergence across related tasks, surpassing the pretrained model and outperforming both basclines Schürholt et al. (2022a); Sch"urhol "urholt et al. (2024). This finding suggests that D2NWG accelerates convergence and improves performance compared to existing methods. This highlights its potential for faster and more efficient model initialization, making it valuable for transfer learning and real-world applications. Interestingly, on MNIST and SVHN, weights with higher initial performance tend to degrade during fine-tuning.
4.2.2 FINE- TUNING ON UNSEEN TASKS: MLP CLASSIFIER
Task: The objective remains the same as Section 4.2.1, but here we evaluate the proposed method solely unseen datasets.
Datasets: We assess D2NWG on real-world dataset of 140 subsets with class counts ranging from 2 to 20, and 10 test sets with up 1,566 classes. We use two-layer MLP top of a CLIP image encoder and tne-tune it on training datasets to collect the pretrained 2se. appendix A.5).
Baselines: The baseline methods are random initialization and pretrained MLP previously trained on ImageNet.
Figure shows performance four unseen datasets, where D2NWG achieves 99.04% initial accuracy on the dessert dataset, outperforming the randomly initialized model even after 50 epochs. D2NWG consistently accelerates convergence across all tasks, surpassing both random and pretrained initialization despite no class overlap between training and test datasets, demonstrating strong transferability. Additional results are provided in Table 22 of the Appendix.
Results:
8

---

Published as a conference paper at ICLR 2025
Table Task Conditioned LoRA parameters Generation. Adaptations are performed on a Roberta- Base model denoted Rob-B.
Methed
Parameters
SST-2 (Acc)
(Ac.)
CoLA MCC.)
QNLI (Aec.]
RTE (Acc.)
STS-B (PCC.)
Avg.
Rob-B
125M
94.8
90.2
63.6
92.8
78.7
91.2
LoRA
45.12:022
78.4:08
91.52:0.2
AdaLoRA
N8.7+055
91.
81.7+06
90.5+002
DyLoRA
0.M
94.3+ 0.5
89,5+0055
61.1+00.6
92 02.2+40.1
78.7=0.7
91.1++0.2
84.5
FounerFT
0&M
94,2+0.3
90.0 0.8
63.82116
92 92.24:0.1
1:0.5
90.80 +: 0.2
85.0
DZNWG
94,3+0.1
64.34112.
92.6+0.20(1055)
79.6-0A00055)
91.000.3(1100.))
85.3(10.3)
4.2.3 FULL MODEL FINE-TUNING ON UNSEEN TASKS
Task: We evaluate each method's generalization CIFAR-I0, STL-10, Pets and Aircrafts, focusing on perfor- mance gains in domain-specifie tasks. The goal is to iden- tify the best initialization strategy for improving model adaptability across diverse data distributions.
DOWNC
Protroined
Accurmy
The baseline in this experiment are the Pre- trained model, which uses weights from model pre- trained on ImageNet and RandomInit, a randomly ini- tialized model.
Baseline:
2,0 124
Datanet
Figure 4: Fine-tuning Unseen Tasks. Datasets: In this experiment we evaluate the transferability to unseen dataset of D2NWG trained in Section 4.1.3 on unseen datasets CIFAR- STL-10, Aireraft100, Aircraft30, and Pets.
We evaluated D2NWG by comparing it against pretrained and 5 randomly initialized models, each fine-tuned for epoch across CIFAR-10, STL-10, Aircraft100, Aircraft30, and Pets datasets. As shown in Figure 4, D2NWG consistently outperforms the baselines. Notably, on AIRCRAFT-100 D2NWG achieved 1. 1.43% accuracy, surpassing both randomly initialized (1.0%) and ImageNet- pretrained (1 (1.24%) models. These results demonstrate D2NWG's generalization and fine-tuning capabilities, even on specialized datasets.
Results:
-etateeet-pretraineed
4.3 TASK CONDITIONED LoRA WEIGHTS GENERATION
Task: In this section, we demonstrate that our method can be applied to LLMs by learning the distribution of LoRA matrices conditioned on task-specific textual descriptions.
Table 6: Exploration of optimal weight space of some instruct LLMS using a diffusion model sam- pled weights. indicates the performance gain
Datasets: We use six tasks from the GLUE benchmark and generate task descriptions using GPT-4 (see Table 14). LoRA weights were gen- erated following the fine-tuning process of Gao al. (2024). We collected LoRA and classifier head checkpoints from the last 5 epochs, combined the pretrained vectors, and conditionally learned their distribution.
Melaak
Winegrono 13 skon
Challeeee 125 das Rellowwgg swill
ELAMNNS IRNNWI
3.11
64.9
67 (ust))ou41
45 P0:1BC1/111
T8.5: 11
DINWI
(092/10.97)
$2.08:
LLAMA
59.29 2001
67.7
Baselines: We compare with base Roberta-base LoRA (Hu et al., 2021), AdaLoRA (Zhang et al., 2023), DyLoRA ( (Valipour et al., 2022) and FourierFT (Gao et al., 2024) which are all LoRA-based RoBERTa- models. We sampled and compared the average accuracy of the top 5 performing sets of weights per dataset.
Results: As shown in Table 5, D2fectffectivee generates weights that match surpass the performance of pretrained models. These results align with our findings from the augmented weight retrieval experiments.
4.4 ENHANCING LLM PERFORMANCE WITH WEIGHT SAMPLING
Task: We aim to demonstrate that D2NWG can enhance existing LLMs by learning the distribution of their pretrained weights, enabling the generation of parameters that improve performance on specific tasks while generalizing unseen tasks.

---

Published as a conference paper at ICLR 2025
Table 7: Performance evaluation on unseen open Ilms leaderboard benchmark. These results are produced by Huggingface after submission to open LLM leaderdoards. f indicate performance improvement while + indicate performance decrease
Method
Meval (8)
Gpgaa
MATII-hard
Musr
MMLU-Pro
Ang
Base Model
Fine tuned
1 11ma-1122118 1.2-nn.
56 76
8.74
3.36
2,96
2.97
7.58
13.76
Vex
D2NWG
58,4411 66)
8.82)10.06)
1.66)1.68)
6.04(5506)
0.66(_2.31)
9.9111.51)
14.12/0036)
LJumse33 2-3B-Iastrud
No
SauckkrunL.M MB lnst.
N0.17
31.00
5.37
11.18
11.52
52,13
25,56
Llama 3.1 XB Last
Yes
D2NWG
90.33 v0.16
31.10110.00
) 5.20010.113
11.56/10.35)
11.52
32.07(0.055
28.64 (10.05)
SaverkruuL.M BB East.
No
Loui Lncensoeed
77,92
29.69
4,56
16.92
7.77
30,90
27.93
Liuna 3.1 1.BB last. Liama 3.1 53
Yes
Liuma 3.1 SB Inst.
78.56
29.89
2.35
17,60
8,41
30,68
27.91
Yes
D2NWG
77.85(10771)
(0.39(10.5)
4.47(12.12)
17.52100.05)
9.64(11.33)
31,02(10.44)
28.50110.59)
Liama 3.1 8B Inst.
No
Datasets: We evaluate on several benchmarks et al., 2023): A12 Reasoning Challenge for grade-school science questions, HellaSwag for commonsense inference, Winogrande for common- sense reasoning.
Baseline: We evaluate our method against various version of LLAMA3 and Mistral-7B.
For each model, We extract the weights of the top 25% of layer excluding embedding and output layer, learn their distribution using chunk based encoding, We then steer through the optimal space to generate spec-fee parameters as shown in Table 6.
Results: The results in Table 6 demonstrates that our approach consistently improve the performance of each models demonstrating new application avenues of our proposed method.
4.5 EVALUATION ON OPEN LM BENCHMARK
We combine the models frome the previous section following Wortsman et al. (2022) and evaluate them on the OpenI.M leaderboard (Fourrier et al., 2024).
Task: We evaluate the robustnets ours best models on the open-Im leaderboard.
Datasets: We evaluate models on 6 key benchmarks datasets: IFEval for instruction adherence, BBH (Big Bench Hard, with 23 challenging tasks (arithmetic, reasoning, language understanding), MATH focusing Level 5 high-school math problems, GPQA with graduate-level Q&A across various fields, MuSR testing complex reasoning with long-range context, and MMLU-Pro for advanced multitask knowledge assessment. These benchmarks assess diverse reasoning and knowledge capabilities in and few-shot settings.
Baselines: We compare method against 1LLAA.11-B-IInttuue and its fine-tuned variant, with evaluations conducted on the leaderboard server.
Results: As shown in Table 7, our method surpasses baseline models the leaderboard and performs comparably to models pretrained on task-specific datasets. Despite not being directly calibrated for leaderboard tasks, D2NWG achieves up to 3% improvement in certain cases. This demonstrates the potential of guided parameter space exploration for task specialization. The consistent gains across benchmarks highlight D2NWG's effectiveness enhancing model robustness and transferability, with our LLaMA-3.2-1D model ranking among the top LLaMA-3.2- entries on the public leaderboard.
Quality Check: Our method enhances text generation quality, as shown in Table 13.
5 CONCLUSION
In this work, we recast latent diffusion for dataset-conditionel neural network weight generation, enabling quick adaptation to novel datasets and efficient fine-tuning and transfer learning without training, Through extensive experiments weights for novel tasks and improves generalization. We extend parameter generation to large language models, demonstrating the scalability and versatility of our approach. Our method effectively encodes architectures with up to 1 billion parameters using a single GPU with less than 80GB, including task- or dataset-conditioned generation.
diverse datasets, our method generates high-quality
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by Institute for Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea (MVI) (RS-2019-1190075,, Artificial Intelligence Graduate School Program(KAIST) and 200.RS-2022-122071,, Meta-learning Applica- ble to Real-world Problems), by Samsung Research Funding Center of Samsung Electronics (No. 10201210-08806-001 Institute Information communications Technology Planning & Evaluation (ITPP) under Open RAN Education and Training Program (ITTP-2024-RS- ----2222-RR-2222-002908888 grant funded by the Korea government(MSII (SSTT) and, by National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00256255) and DeepAuto.ai.

---

Published as a conference paper at ICLR 2025
REFERENCES
Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lee. Meta-learning with adaptive hyperparameters. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 20755-20765. Curran Associates, Inc., 2020.
Edward Beeching, Clémentine Fourrier, Nathan Habib. Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Il leaderboard, 2023.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize for efficient deployment. In International Conference on Learning Representations, 2020.
Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A. Clifton. A brief review of hypemetworks in deep learning. ArXiv, 0bss2200.006555 2023. URL https://api. Semanticschen www.aaa...rr...oeeeeeeeeeeeoeeee Cttpg//orpssDDiD- 259138728.
Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline: Exploring simple meta-learning for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9062-9071, 2021.
Tristan Deleu, David Kanaa, Leo Feng, Giancarlo Kerg, Yoshua Bengio, Guillaume Lajoie, and Pierre-Luc Bacon. Continuous-Time Meta-Learning with Forward Mode Differentiation. In Tenth International Conference on Learning Representations, 2022.
Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando de Freitas. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
Ashwini Doke and Madhava Gaikwad. Survey on automated machine learning (automl) and meta learning. In 2021 12th International Conference Computing Communicattion and Networking Technologies (ICCCNT), pp. 1-5, 2021. doi: 10. 00.1099CCCNTT513552222..88
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal. Anthony Hartshom, Aoho Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodrigue7, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arricta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet. Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupulcti. Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre. Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunoy, Shaoliang Nie,
12

---

Published as a conference paper at ICLR 2025
Sharan Narang, Sharath Raparthy, Sheng Shen. Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudent, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg. Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annic Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf' Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li. Danny Wyatt, David Adkins. David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss. Dingkang Wang. Duc Le. Dustin Holland, Edward Dowling. Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan. Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha. Haroun Habeeb, Harrison Rudolph, Helen Suk. Henry Aspegren, Hunter Goldman, Ibrahim Damlaj. Igor Molybog, Igor Tufanov, Minna-leea Veliche. Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Jessica
Zhen, Jeremy Reizenstein, Jeremy Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhic, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U. Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedr, Madian Khabsa, Manay Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L.. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang. Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanic Max, Stephen Chen, Steve Kehoe, Steve Satterfield. Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho. Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihaileseu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu. Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi. Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vitooo, Zef Rosnbrick, Zhaoduo Wen,
13

---

Published as a conference paper at ICLR 2025
Zhenyu Yang, and Zhiwei Zhao. The Ilama 3 herd of models, 2024.
Clementine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open 1lm leaderboard v2, 2024.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019.
Hanan Gani, Muzammal Naseer, and Mohammad Yaqub. How to train vision transformer on small- scale datasets? In 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022. BMVA Press, 2022.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer. In 2023 JEEE/CVF International Conference on Computer Vision (ICCV), pp. 23107-23116, 2023. doi: 10.1 00.10091CCV510070022..0111.
Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter- efficient fine-tuning with discrete fourier transform, 2024.
Yifan Gong, Zheng Zhan, Yanyu Li, Yerlan Idelbayev, Andrey Zharkov, Kfir Aberman, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficient training with denoised neural weights, 2024. URL https: org/ abs 2407 11966.
Roberto Gozalo-Brizuela and Eduardo C. Garrido-Merchán. survey of generative ai applications, 2023.
David Ha, Andrew Dai, and Quoe V. Le. Hypernetworks, 2016.
Eric Hartford, Lucas Atkins, Fernando Fernandes Neto, and David Golchinfar. Spectrum: Targeted training on signal to noise ratio, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840 68511.
Curran Associates, Inc., 2020a.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arxiv:2006. 11239, 2020b.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1rXiv:2106.09685. 2021.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automated Machine Learning - Methods, Systems, Challenges. Springer, 2019.
Wonyong Jeong, Hayeon Lee, Gun Hong Park, Eunyoung Hyung, Jinheon Back, and Sung Ju Hwang. Task-adaptive neural network search with meta-contrastive learning. In Neural Information Processing Systems, 2021.
Boris Knyazev, Michal Drozdzal, Graham W Taylor, and Adriana Komero-Soriang Parameter prediction for unseen deep architectures. In Advances in Neural Information Processing Systems, 2021.
Boris Knyazev, Doha Hwang, and Simon Lacoste-Julien. Can we scale transformers to predict parameters of diverse imagenet models? International Conference on Machine Learning, 2023.
Hayeon Lee, Eunyoung Hyung, and Sung Ju Hwang. Rapid neural architecture search by learning to generate graphs from datasets. In International Conference on Learning Representations, 2021.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: framework for attention based permutation-invant neural networks. In Proceedings of the 36th International Conference Machine Learning, pp. 3744-3753, 2019a.
14

---

Published as a conference paper at ICLR 2025
K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Meta-learning with differentiable convex optimiza- tion. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 10649-10657, 2019b.
Bohan Liu, Zijie Zhang, Peixiong He, Zhensen Wang, Yang Xiao, Ruimeng Ye, Yang Zhou, Wei- Shinn Ku, and Bo Hui. A survey of lottery ticket hypothesis, 2024.
Shervin Minace, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2024. URL https: /arxiv.. org/abs htr//ab/// 2402. 06196.
Elvis Nava, Seijin Kobayashi, Yifei Yin, Robert K. Katzschmann, and Benjamin F Grewe. Meta- learning via classifier(-fiee (-free) diffusion guidance. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4172-4188 2023. doi: 10. 1009 ICCV51000.2023300333
William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A. Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints, 2022.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché- Buc, E. Fox, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Neale Ratzlaff and Li Fuxin. Hypergam: A generative model for diverse, performant neural networks, 2020.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674 10645, jun 2022.
Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674-10685, 2021.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference Learming Representations, 2019.
Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper-representations as generative models: Sampling unseen neural network weights. In Thirty-Sixth Conference Neural Information Processing Systems (NeurIPS), September 2022a
Konstantin Schürholt, Boris Knyazev, Xavier Giró Nieto, and Damian Borth. Hyper-repreeenntations as generative models: Sampling unseen neural network weights. In Advances in Neural Information Processing Systems, 2022b.
Konstantin Schürholt, Diyar Taskiran, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Model zoos: A dataset of diverse populations of neural network models. In Thirty- Sixth Conference Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, September 2022c.
Konstantin uchhur Michael W. Mahoney, and Damian Borth. Towards scalable and versatile weight space learning. In Proceedings of the 41st International Conference Machine Learning (ICML). PMLR, 2024.
Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-supervised representation learning on neural network weights for model characteristic prediction. In Advances in Neural Information Processing Systems (NeurIPS 202/), Sydney, Australia, 2021.
15

---

Published as a conference paper at ICLR 2025
Zihao Tang, Zheqi Lv, Shengyu Zhang, Fei Wu, and Kun Kuang. Modelgpt: capabilities for tailored model generation, 2024.
Unleashing Ilm's
Ihsan Ullah, Dustin Carrion, Sergio Escalera, Isabelle M Guyon, Mike Huisman, Felix Mohr, Jan N van Rijn, Haozhe Sun, Joaquin Vanschoren, and Phan Anh Vu. Meta-album: Multi-domain meta-dataset for few-shot image classification. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint wwr.v:22200075555 2022.
Kai Wang, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion, 2024.
Hassler Whitney. Differentiable manifolds. Annals of Mathematics, 37(3):645-680, 1936. doi: 10.2307/1968482.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rehecca Roelofs, Raphael Gontijo-Lopes, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965-23998. PMLR, 17-23 Jul 2022.
Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming- Yang. Diffusion models: A comprehensive survey of methods and applications, 2024. URL https: / wtw.ivivvv.oo..a// org/ abs/ 2209222222222 00796.
Baoquan Zhang, Chuyao Luo, Demin Yu, Xutao Li, Huiwei Lin, Yunming Ye, and Bowen Zhang Metadiff: Meta-learning with conditional diffusion for few-shot learning. Proceedings of the AAAI Conference on Artificial Intelligence,
Mar. 2024.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. In International Conference on Learning Representations, 2019.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-effiieet fine- tuning. arXiv preprint arXiv:2303. 105/2, 2023.
Xueting Zhang, Debin Meng, Henry Gouk, and Timothy Hospedales. Shallow bayesian meta learning for real-world few-shot recognition. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 631-640, 2021. doi: 10. 1119/ICCC449922.222..0000 .00069.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang. Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models, 2023. URL https: httpss///axiivv org/ abs/ 2303 18223.
Ziyu Zhao, Leilei Gan, Guoyin Wang, Yuwei Hu, Tao Shen, Hongxia Yang, Kun Kuang, and Fei Wu. Retrieval-augmented mixture lora experts for uploadable machine learning, 2024.
Andrey Zhmoginov, Mark Sandler, and Maksym Vladymyrov. HyperTransforme Model gener- ation for supervised and semi-supervised few-shot learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 27075-27099 PMLR, 17-23 Jul 2022.
16

---

Published as a conference paper at ICLR 2025
LIMITATION AND ETHICAL STATEMENT
Limitations: Our method relies on large collections of pretrained weight tensors and datasets, which require substantial storage and computational resources. However, such pretrained models are becoming more readily available due to the efforts made by open-source communities.
A APPROACH
Broader Impact D2NWG addresses the resource-intensive nature of deep learning by proposing a method for efficient transfer learning. This has the potential reduce the computational resources required for training neural networks, making it more accessible to wider range of researchers and organizations.
Limitation In this work, we focus mainly on generalization across datasets. Additionally, while the diffusion model achieves impressive performance on image generation, there still some challenges to efficiently recast it for weights generation including memory constraint, convergence challenges and considerations of symmetries the weight spaces of different neural network architectures.
A.1 RELATIONSHIP BETWEEN DATASETS AND TRAINED WEIGHTS
Gradient descent based optimization is the commonly used technique to generate optimal neural network weights through training by minimizing loss function, ie. cross-entropy for classifiation tasks. The weights optimized with gradient descent thus contains some information about the training data. Therefore, understanding the correlation between the training dataset and the optimal weights is important for the generation of weights. During the optimization process with gradient descent the weights of each layer i updated as w; W V w(:r, W2,-... W,),, where V L(w,, u wn) ) is input dependent. As an example, let's 's consider two-layer feedforward neural network:
x: inpuls
4 Wix + bj
h Re)U(L1) Wzh b2
(e) sol soflmax((2)
J = CE(y, 0)
Analyzing the weights' update below, of the inputs feature maps and all together they contain information about the training either related the raw input or the feature map at given stage.
observe that the optimal weights noisy perturbation
8, ŌJ
(y 0)
Ol2
OJ
82 0h
81 W W2osgn(h)
W
(6+1)
((() V/V w² f(wi: w2; bi, b2)
= ((O) no² x
(((+1)
(( 17 w2 L(w1, W2 by b2)
W. (8)
not ht
A.2 WEIGHTS VECTORIZATION
For neural network with L layers, the process of vectorizing the weights and biases for both fully connected and convolutional layers is as follows:
For the f'th fully connected layer: W(O) Rd: 1xd; dR the length of the vectorized weights for this layer, including the bias it is not null, is
(W()) R4 1-4 and (()
given by 14 di-
17

---

Published as a conference paper at ICLR 2025
For the é'th convolutional layer: W E Be and 8(0) Reu the length of the vectorized weights for this layer, including the bias if is not null, is ky kw Cin Cout + Cout-
We then concatenate all the flattened weight and bias vectors resulting in vector 8: 0 + T (vec(W(() + b()) where vec denotes the vectorization operation and @ denotes concate- nation. The concatenation operation keeps the ordering of weights in the network.
A.3 LAYER SELECTION STRATEGY
To manage the large number of parameters in LLM architectures, where not all layers are required to be tuned to improve the performance, we propose focusing on the most important layers. These layers are identified using the Marchenko-Pastur (MP) distribution, which serves as a filter to highlight relevant weights while discarding those resembling random noise. The MP law provides a benehmark for distinguishing structured weights from noise by comparing the empirical eigenvalue spectrum of weight matrices to the MP distribution. D2NWG uses this spectrum method (Hartford et al., 2024) to learn the distribution of the most informative weights- those corresponding to eigenvalues that significantly exceed the MP upper bound. By focusing on these critical weights, D2NWG captures meaningful patterns in LLMs, leading to enhanced performance in transfer learning.
The spectrum method, grounded in random matrix theory, applies the Marchenko-Passe (MP) distribution to different types of layers, treating them rectangular random matrices. In transformer networks, functionally similar layers are grouped, such as set for all query layers in multi-hear attention. The method begins by computing the covariance matrix of each layer's 's weight matrix, W Rmxx as D w followed by eigenvalue extraction. Singular value decomposition (SVD), W = USVT, is used to efficiently compute these eigenvalues from the diagonal matrix S, which contains the singular values. The resulting eigenvalues describe the variance captured by each principal component of the squared weight matrix and form what known as the empirical spectrum. To analyze this spectrum, we compare it to the theoretical distribution of eigenvalues predicted by the Marchenko- Pastur (MP) distribution. This distribution p(A), in equation 7, characterizes the eigenvalue behavior of random covariance matrices as m, n oc, with a fixed aspect ratio q = and variance 2².
1
p(A)
2TO2qa V(++ A)(A- -_),
where X e 11,, o1+ 2 and X_ = 0 - From 7, the correspoding bounds for eigen values of W are vX/vn E [E,,E_),E4 = o1 + vT), and e_ o VT).
Interpretation: The Marchenko-Pastte (MP) distribution provides insight into the underlying structure of data layer in case:
Likely represent noise, with their corresponding principal components carrying little meaningful information, indicating the layer's 's lower importance.
Eigenvalues within MP bounds:
Eigenvalues larger than the upper MP bound X4: Capture more variance than noise, suggesting the presence of true signals or patterns in the data.
Eigenvalues smaller than the lower MP bound A_ May indicate compression or degenera- tion in the data structure.
Significant deviations, particularly large eigenvalues, indicate meaningful components that capture more variance than random noise, aiding in the identification of important features or signals. This insight is used to compute the signal-to-noise ratio (SNR), where eigenvalues below the upper bound are considered noise. The SNR is calculated as follows:
Exller
SN R
(8)
Em On
A.4 LEARNING THE DISTRIBUTION OF LLM WEIGHTS
Our method for LLM weight generation employs a layer-wise chunking mechanism that facilitates both layer-wise and chunk-wise sampling. Each layer is divided into independent chunks form the
18

---

Published as a conference paper at ICLR 2025
training data, and then encoded with the VAE. During the diffusion process, index is assigned to each chunk, and the model is trained using class condsicondiio diffusion, where chunk indices serve as class labels. At sampling time, the chunk indices corresponding each layer are grouped into clusters associated with that layer. These clusters are then used to sample new sets of chunks, which concatenated to reconstruct the sampled weights for each layer.
After selecting the top 25% of the layers, we applied chunking with a size of 2,097,15 152 for LLaMA 3.2-IB and 4,194,304 for other models. We then performed sequential refinement using Algorithm 1. Unlike in vision tasks, LLM models are conditioned on chunk indices. Here, we refer to neural network operations such as dense layers and layer normalization layers. The spectrum method provides an ordered set of these layers (q, k, mlp up, dlwp, inl__gatel For architectures like Llama 3-B and Mistral, we only learn the distribution of the top each of these layers, excluding layer normalization. These layers are further divided into two groups: the top 4 and the second top 4, for which we build separate models learn their distributions. As for the normalization layers, we learn the distribution across all of them. The maximum generated parameters is Rs 872M.
Algorithm 1 Sequential Weight Model Improvement
1: Input: Initial weights int ( ,1),, Hypernetwork H4 for each layer i, Validation dataset Dvat, K candidates per layer
2: Output: Final weights O* ..., 02}
3: Initialize O* = jnit
4: Compute initial validation accuracy: cuurent_accuaaee A(Oinits Peal)
5: for each layer = 1 to L do
6: Generate K candidates (0( ...... (0( using Hi
7: for each candidate k = 1 to K do
8: Replace 0g with 0!()) in O* to form ((k)
9: Compute validation accuracy: ((O(k), vaa)
10: end for
11:
Choose 0- = arg max; ((O(k), Daal)
if A(O())) Daa) > current accuuray then
13:
Update O' (k)
14:
Update aurrent_accurage = A(e", D
15: else
16:
Retain 0, in O*
17: end if
18:
end for
19:
20:
return e*
A.5 MODELZOO AND PRETRAINED DATASETS
Model zoo We use the pretrained datasets from Schürholt al. (2022c) structured in Schürholt et al. (2022a). This dataset consists of 4 different datasets with 5000 pretrained weights per architectures and datasets. The details of the architecture used generate the pretrained weights are available in Schürholt et al. (2022c).
KaggleZoo This modelzoo is generated using the dataset provided by Jeong al. (2021). To efficiently generate the pretrained weights, we first compute the features of each image then use MIP with two layers with input size 512, hidden size 256 and leaky ReLUU activation functions. We train the MLP on clip features as it allows to quickly generate high performing weights. For each datasets we used the last 10 checkpoints which results in 1400 pretrained weights for training.
IeageNet zoo To generate the pretrained modelzoo ImageNet, we sample 1000, 5000, 10000 and 20000 subsets with 10 classes each with 100 images per class in the training set and 50 per class in the test set. For the 1000 and 5000 subsets used the same MLP architecture the KaggleZoo. For the 10000 subset, we reduce the hidden dimension to 128 and, for the 20000 subset we use a single linear probing layer. On the other datasets linear probing shows similar generalization performance
19

---

Published as a conference paper at ICLR 2025
as the two-layer MLP. We use Adam optimizer with learning rate of 1e 3 and all models are trained for 30 epochs.
Zoo for Few-shot learning: The few-shot learning pretrained zoo is generated by fine-tuning the classifier head for 10 epochs on each of the 50,000 subsets.
LLMs zoo: We collected the pretrained LLM model from their original HugginFace repositories with no further pertaining on specific tasks or datasets.
Meta-album datasets: We split the meta-album dataset into training set (70%) and test set (30%). Next, we trained the MobileNetV3 OFA subnet with parameters d = 2, k = 3, and e = 3 for 100 epochs. Checkpoints from the last 20 epochs were collected as training data. A detailed breakdown of the dataset can be found in Table 12.
A.6 DETAILS OF THE PROPOSED MODEL
We build our dataset conditioned weight generation model using latent diffusion (Rombach et al., 2021).
We use the same VAE modules of latent diffusion and use the same architecture for all experiments except adaptation of the inputs and output dimensions. We insert a linear layer before the first layer of the encoder such that we can reshape layers. Similarly, linear layer is placed at the last layer of the decoder adapting the output to the vectorized weights representations. For the VAE loss function we removed the discriminator in the original latent diffusion VAE loss function.
AutoEncoder:
output to a representation for the convolution
Diffusion Model:
We utilize same UNet architecture as in latent diffusion with the same training procedure.
Table 8: Models seting, n and in the dataset configuration represent respectively the number of samples per class n=5 for training and с the total number of classes per dataset. The VAE and the diffusion models share similar configuration and ar- chitectures as (Rombach et al., 2021)
Dataset Encoding Mechanisms We investigated three different mechanisms of dataset encoding. Firstly, we use Set Transformer (Lee et al., 2019a) which can be difficult to train when optimized together with the dif- fusion using the weights encoder from the VAE and the Set Transformer.
In addition to the Set Transformer, we explored a two- layer MLP model as the dataset encoder. The first layer is a dynamic linear layer with a maximum in- put feature size set to Timax Cmax. where Temax is the maximum number of images per class and Cmax is the maximum number of classes among all subsets of the pretrained datasets. The shape of the image features in each dataset obtained with the CLIP image encoder is x E Rxxaxd where d is the feature dimension for each corresponding pretrained weight vector. While the Set Transformerrbased encoder uses these inputs directly, the MLP encoder reshapes each input from E Rxxax to x E R dx (n-d) and then applies the dynamic linear layer. If a dataset has more classes or samples than Cmax and mmmax respectively, we only con- sider the first Cmux classes and R'max samples per class. If the dataset has fewer classes or samples, we adjust the dynamic linear layer dimensions accordingly. The output of the dynamic linear layer is E R dxh where h is arbitrarily chosen number greater than zero. We then reshape z from R. ddx to R (fed) (with h d fixed) and apply the final linear layer to obtain the desired output. This model can be jointly optimized with the diffusion model while achieving good performance.
Parameters
Values
Epochs
[50, 2000]
VAE
Optimizer
Adam
Learning Rate Latent Dimiension
1c-3
1024
-L--Dvergence Weight
le-6
Dataset Encoder
Architccture Input Dimension
Set Transformer
x n 512(min)
Output Dimeusion
1024 (min)
Depth of Set Transformer
Diffusion
Optimizer Learning Rate Scheduler
AdamW le4 Linear 1024
Time step
Network
Unet
UNet Input Size
(c x 32 x 32)
Dataset Encoding with Set Transformer We use the Set Transformer for dataset encoding, pretrained as described in Lee al. (2021). The approach involves using the frozen Set Transformer and adding
20

---

Published as a conference paper at ICLR 2025
Algorithm 2 Datasets Encoder Training
Input: pretrained weights image features batch num m Instanciate = Set Transformer Load pretrained Encoder (8).
repeat
Initialize loss = 0.0 for i = 1 to
do
3t ~ n, Di ~ D 2; EnCodervAE
PD. - (
loss loss LCLIP zD.) (Equation ??)
end for
Update weights of until convergence
a single linear layer to adapt its output to our specific problem, utilizing it as the dataset encoder. This method reduces the computational cost of training the Set Transformer and enables joint optimization of the dataset encoder and the diffusion model. The results of these data set encoding schemes are presented in Table 21 for the Hyperzoo dataset.
B TRAINING DETAILS
this section, we describe the training steps used to train our method.
Pretrained Zoo Generation: For classifier head adaptation, we first compute the features for all datasets. Then, we train the classifier head to generate the pretrained zoo.
VAE Training: We train the VAE to encode the pretrained weights following Equation 1. Additionally, pretrained performance predictor can be used to predict the performance of the reconstructed weights and guide the VAE training as described in Equation 9.
Dataset Alignment: If using dataset alignment, we pretrain the Set Transformer to align the pretrained weights' latent representations. This done using the frozen encoder of the VAE and the dataset embeddings. The inputs to the Set Transformer are image features, with five image features per class.
Diffusion Process Training: We train the diffusion model while keeping the Set Transformer and the VAE models frozen. If an MLP is used for dataset encoding, we jointly optimize the diffusion process with the MLP dataset encoder.
Although the dataset encoder can be optimized together with diffusion model, we train them separately to speed up the training process and reduce memory requirements. The VAE and the dataset encoder are trained using the Adam optimizer with learning rate of 1e each experiment is trained with a linear scheduler, a base learning rate of Ic-4, and the AdamW optimizer (Rombach et al., 2021). 1). During the training process of the diffusion model, the output of the dataset encoder is concatenated with the latent representation of the input weights, forming the input to the UNet model. Additionally, we investigate joint training of the diffusion process in the ablation study and Appendix C.5 and A.6. Further details can be found in Table 8.
The diffusion model in
B.1 PREDICTOR TRAINING
To improve the reconstruction and sampling efficiency, we trained an accuracy predictor 9 from pretrained weights w then use the frozen predictor during the training of the VAE as regularizer as shown below:
W fo(w)
min 8,0
+ log o² + [lg(w) ((fo(w)))1
where g(w) is the embedding of the original input and gf w)) is the predictor embedding of the reconstructed weights. The predictor can be either cinasst-connitionne or unconditioned. In general found that dataset-conditioned predictor works only well for large number of samples per dataset.
21

---

Published as a conference paper at ICLR 2025
Set of datasets backbone network
Per dataset embeddings
Set Transformer
Set Transformer
E IR Ixxxxxx
Zi R 1xxxdd
Zo E R1xdo
Per dataset features
Figure 5: Overview structure of the set-transformer-basse dataset encoder. For each pretrained dataset we use n 5 images per class and the embedding dimension dg 1024.
Algorithm 3 Predictor-Guided VAE
Input: Pretrained weights z, accuracy y, batch_nnt num m Instantiate f = Set Transformer, and load pretrained predictor g)
repeat
Initialize loss 0.0
for = 1 to m do
② fo(x), y = g(Ø) 9 g(z)
L 0² + log 8² + 1l0 5
end for Update weights of until Convergence
After the AutoEncoder is trained, we train the dataset-conditioned module which requires a dataset encoder.
C ABLATION STUDY
C.1 CAN THE PROPOSED METHOD HANDLE MULTIPLE ARCHITECTURES?
This section provides a simple way to handle the case where the pretrained zoo contains multiple architectures per task or dataset. Since number of architecture and dataset are predefined, is possible to build set of unique index for each combination of alternative will be to encode the graph representation of the architectures then used that as conditioning. In this ablation study our method. We use CIFARIO and CIFAR 100 as the dataset and as target architectures we utilze a ResNet44 trained on CIFAR- 100 with 667, 188 parameters and ResNet44 trained CIFAR-10 with 661,338 parameters and finally, a MobileNetV2 trained on CIFAR-10 with 700,490 parameters. All models were zero-padded 700,490 parameters, combined into unified dataset, and trained without chunking. The results in Table 9 demonstrate that the proposed method is capable of simultaneously leamning the distributions of diverse architectures trained on diverse datasets.
pairs. An
the simple class indexing approach to demonstrate the versatility of
Model
ResNer44 (CIFAR-10)
ResNer44 (CIFAR-100)
MobileNerV2 (CIFAR-10)
Pretrained 94.01
71.63
92.88
D2NWG
94.10 +0.09
71.64:0.02
93.11+0.20
Table 9: Performance evaluation mixed architectures.
22

---

STL10 Published as a conference paper at ICLR 2025
C.2 TRANSFERABILITY
As demonstrated in Table ??, our approach achieves performance comparable to existing meth- ods while relying on single generative model instead of 38 task-specific pretrained models. Notably, the pretrained model architecture and parameter counts used in this study are pub- licly available on non-affiliated GitHub repository: https://githin com/ chenyaofo/ pytorch-ci far-models..
EVALUATING SAMPLING FOR TRANSFER LEARNING
We compared sampling from a distribution of diverse pretrained models against traditional single- model transfer learning, using ResNet-56 and our generative model trained on weights from 19 diverse architectures pretrained on CIFAR- 10 and CIFAR-100. We tested three experimental setups:
1. Direct evaluation of the pretrained models.
2. Sampling conditioned on training sets (e.g.. STL-10, CIFAR-10).
Results show that our approach consistently outperforms single-model transfer learning. Notably, there is no significant difference between training- and test-conditioned sampling when drawn from the same distribution, demonstrating the robustness of our method. This highlights the practicality of leveraging diverse pretrained model distributions for improved generalization.
Table 10: Performance on CIFARIO.1 and STL10 of D2NWG trained on diverse architectures
Model
CIFAR10.1
Pret-cifar10
75.20
32.37
Pret-cifar100
0.25
0.12
Ours
83.10 + 0.06
35.41 + 0.13 35.47 + 0.12
Ours(test)
83.04 + 0.06
C.3 EFFECT OF MODELZOO SIZE GENERALIZATION
Here we investigates the impact of increasing the number of pretrained datasets on performance with experiments that use model zoos of sizes 5000, 10,000, and 20,000, derived from ImageNet subsets. Unseen target datasets CIFAR- and STL-10 are used. Sampling weights, the average performance of the top performing weights is shown in Figure 6a.
Results: On CIFAR-10 and STL-10, we obtain accuracies of 39.60 1.31% and 44.66 + 0.55% for 5000 subsets, 42.15 + 2.12 and 64.83 + 2.83% for 10000 subsets, and 52.64 + 3.12% and, 80.49 1.77% for 20000 subsets. The maximum accuracies with random initialization are 12. 11% and 17. 12% on CIFAR-10 and STL-110 without fine-tuning. This experiment demonstrated that increasing the number of datasets enhances the generalizability of the proposed method.
C.4 SAMPLING WITHOUT LATENT REPRESENTATION
This section explores model variant that directly learns the diffusion model weights, bypassing the AutoEncoder stage, and compares it to the standard approach. Both variants are trained on 1000 subsets of ImageNet, and evaluated in in-distribution sampling setting three randomly selected subsets from the 1000 subsets. The results, presented in Figure 6b, indicate that learning the distribution of pretrained weights in the latent space is notably successful in generating high- performing weights. The failure of the DDPM process on raw pretrained weights may stem from their higher model capacity requirement.
C.5 CLIP-BASED DATASET ENCODING
In this section, the comparison between the CLIP-based dataset encoding scheme trained at intermediate stage and the Set Transformer encoder jointly trained with the diffusion process is explored. Experiments conducted on 140 Kaggle datasets and their respective model zoos. The
23

---

Published as a conference paper at ICLR 2025
50
80
CIFAR10
VAE+DDPM
STL10
DDPM
70
Pretrained
40
60
Accuracy
Accuracy
50
30
40
20
30
20
10
10
0
0
5K
10K
20K
2
Dataset Size
Dataset Subsets
(a)
(b)
Figure 6: (a) Effect of number of pretrained datasets on sampling weights performance on unseen datasets. (b) Performance comparison on in in-distrbuttio sampling of methods with VAE+DDPM vs DDPM
100
With CLIP
Without CLIP
Rand Init
80
Accuracy
60
40
20
Dessert
Alien vs Preditor Dataset
Covid-19
HoneyBee Pollen
Figure 7:
Performance comparison at initialization of method with jointly trained set se-ttransformrr (Without CLIP) and method clip-based dataset encoder.
results depicted in Figure 7 indicate that both methods achieve similar results for small numbers of datasets during the in-distribution sampling. However, as the number of datasets increases, the Set Transformer jointly trained with the diffusion approach faces challenges in convergence and requires more computational resources, as demonstrated in Figure 7.
C.6 UNCONDITIONAL. SAMPLING
We conduct the experiment using ResNet18 pretrained on CIFA-100 and CIFAR-10. For datasets, the weight vector length is 2048 and we compare with pdiff (Wang et al., 2024). While pdiff requires a separate model for each dataset, our method combines the pretrained weights into single dataset and conditionally learns their distribution. The sample size for each dataset in our method is 200, with a combined total of 400 parameters. The results are provided in Table 11 for 100 sampled weights. Two separate models for are trained for pdiff, CIFA10-pdite and CIFAR 100-pdifff while our method consists of a single model trained once for both datasets. can be seen that our method outperformance the baseline (Wang et al., 2024) in Table 11.
24

---

Published as a conference paper at ICLR 2025
Table 11: Unconditional Sampling Evaluation against Wang et al. (2024) on ResNet18.
Dataset
CIFAR-10
CIFAR-100
Runtime
#Epochs for VAE,DDPM
#Epochs VAE,DDPM
Method
Avg
Median
Max
Avg
Median
Max
pdiff
94,46
94.46 94.47
94.52 94.50
8999,47999 100,200
76.1028 76.1796
76.13
76,21
32999,38999 100,200
3h
D2NWG
94.46
76.18
76.24
1430
C.7 COUPLING WITH AN ACCURACY PREDICTOR
This section reports the extended results of Table 19 in which we compared our method in-distribution and out-of distribution with and without accuracy predictor.
The full results of Table 19 are reported in Table 20. Using an accuracy predictor enable easily selecting highly performing when sampling in-distribution. However, in our case the accuracy predictor struggles to generalize well for unseen dataset as shown in Table 20
Results.:
C.8 SAMPLED WEIGHTS ANALYSIS
In this section, we analyze the characteristics of the sampled weights and compare them to the pre-trained ones based experiments with the model and model pre-trained on a subset of ImageNet. The proposed method samples weights with a large variance, as shown in Figure 10, providing broad range of initialization choices, from weights with low initial performance those with higher initial performance.
Table 12: Details description of aeta-aabuum datasets
Domain
Original Dataset
Classes
Large Animals
Animals with Attributes, Dogs, Birds
50, 120, 315
Small Animals
Insects, Insects 2, Plankton
117, 102, 102
Plants
Fungi, PlantNet, Flowers
25, 25, 102
Plant Diseases
PlantDoc, Medicinal Leaf,
27, 26, 38
Plant Village
Microscopy
Subcel. Human Protein, Pan-
21, 19, 33
Nuke, Bacteria
Remote Sensing
RSD, RSICB, RESISC
43, 45, 45
Vehicles
Boats, Airplanes, Cars
26, 21, 196
Manufacturing
Textures ALOT, Textures
250, 47, 64
DTD, Textures
Human Actions
MPII Human Pose, Stanford
29, 40, 73
Actions, 73 Sports
OCR
Omniprint-MD-6 Omniprint-
703, 706, 706
MD-5-bis, Omniprint-MD-
mix
C.9 EVALUTAION ON LARGE DATASETS
We investigate how our method perform for combined large and small dataset as well for mixed architectures. For this experiment collect the pretrained weights from PyTorch hub with one checkpoints per datasets 10, CIFAR-100, and ImageNet- - After conditionally learning the combined weights distribution, sampled 10 weights for each datasets and report the average accuracy of the top-3 datasets in Table 15. As shown in Table 15, D2NWG consistently produced high performing weights for each dataset from single pretrained checkpoint.
25

---

Published as a conference paper at ICLR 2025
Table 13: Comparison between Base Model and Improved Model Responses
Instruction
Base Model Response
Improved Model Response
What is AT? Artificial Intelligence (AI) refers to the development of computer systems that can perform tasks that would typically require buman intelligence, such as:
What is Artificial Intelligence (AI)? Artifi- cial Intelligence (AI) is a branch of computer science that locuses creating intelligent ma- chines that can think, lear, and hehave like bumans. involves the development algo- rithms, statistical models, and computer pro- grams that enable machines to perform lasks that typically require human intelligence. such as:
1. Learning Al systems can learn from data, experiences, and environments to improve their performance.
Reasoning: Al systems can reason and make decisions based on the data they have learned.
1. Reasoning and problen--solving Al systems can analyze data, identify pat- and make decisions based that data.
3. Problem-soovingg Al systems can solve complex problems and adapt to new situations.
2. Learning: Al systems learn from experience, adapt to new situations, and improve their performance time.
Perception: systems can perceive and understand the world around them through sensors, cameras, and other devices.
3. Perception: AT systems inter- pret and understand data from sensors, such images, speech, and text.
systems use various techniques, such as ma- chine learning, deep learning, natural language processing, and computer vision, to achieve these capabilities.
Prompt: I I"mole" "system", "con- tent": "You are an Al chatbor"), ) ("vole": "user", "content": "Explain what Al is and its importance") ).
4. Natural Language Processing (NLP): Al systems understand and generate human language, ea- abling them to communicate with humans.
Types of Al::
1. Narrow or Weak AI: Designed to perform a specific task, such as facial recognition language translation.
become increasingly important in various ispects of our lives, in- cluding:
Importance
General or Strong AI: Designed to perform any intellectual task that a hu- man cam, such as reasoning, problem- solving, and learning.
1. Automation: Al can autornate repeti- and mundane tasks, freeing up hu- man time for more creative and strate- gic work.
3. Superintelligence
Significanlly
more intelligent tham the best hu- man minds, potentially leading to exponential growth technological advancements.
2. Improving decision-making: Al can analyze large datasets, identify pat- tems, and provide insights that inform business decisions.
Importance of
1. Automation: Al can automate repet- itive and mundane tasks, freeing up human resources for more strategic work.
3. Enhancing customer experience: Al- powered chatbots and vittual assis- tants provide 24/7 support and per- sonalized services.
In a coffee shop, the person who serves the coffee to customers typically referred to a: Barista: This the most common title, and refers skilled person who prepares and serves coffee drinks, as well as other heverages and food items. Coffee Server: This title also used to describe someone who serves cof- fee and other beverages to customers. Coffee Barista Coffee Specialist: These titles of S
At our coffee shop, we call the person who serves the coffee to customers a 'Barista'. . How- ever, some of our team members are also referred to 'Cafe Attendants' or 'Coffee Servers'. ' depending on their specific role and responsibilitiss But generally, the title 'Barista' is the most commonly used and well-known term in the industry.
prompt: ("mle": "system", "con- tent"; "You work at coffee shop") "). ("role": "user", "content": "what sitle do you give to someone who serves the coffee to customers") ").
ten used in specialty coffee shops or high-end establishments to emphasize the barista's ex- pertise and knowledge of coffee. In our coffee shop. I'm barista, and I'm happy serve you your favorite coffee drink.
1. GENERATING THE FULL WEIGHTS FOR RESNET18
We investigate how our method performs when used to generate the full parameters for a ResNet1 8 model pretrained on MNIST, CIFAR-10, and CIFAR-100. In total, we use 100 pretrained weights per dataset and conditionally learn their distribution. The modelzoo generation follows the same setting as Wang et al. (2024). Table 16 demonstrates the effectiveness of our method for generating the entire weights of network.
C.11 GENERATING WEIGHTS FOR MOBILENETV3
So far, our focus has been model zoos populated by relatively simple classifier heads. In this section, we evaluate our method using MobileNetV3,, a subnetwork sampled from OFA (Cai et al., 2020), consisting of 2.8 million parameters fine-tuned on CIFAR-10, STL. SVHN and MNIST for 15
26

---

Published as a conference paper at ICLR 2025
Table 14: Glue benchmark tasks descriptor used in the experiment on glue datasets.
Task Name
Description
SSTB
Predict the similarity score between two sen- tences. Rate their similarity on scale from to 5, where indicates no meaning overlap, I indicates very little overlap, and 5 indicates complete overlap meaning.
MRCP
Determine the semantic equivalence of two given sentences (Sentence and Sentence 2). If the sentences are semantically equivalent, re- turn If they are not, return O.
SST2
Determine the sentiment of a given sentence. Respond with if the sentiment negative and 1 if the sentiment is positive.
COLA
Evaluate whether the given sentence is both syntactically and semantically correct. If it is, respond with "I"; otherwise, respond with "0".
QNLI
Evaluate whether the given response properly answers the provided question. If the response answers the question correctly, return 0; other- wise, return 1.
RTE
Determine if given hypothesis is true (entail- ment), false (contradiction), or undetermined (neutral) based on a provided premise.
Table 15: Evaluation on Large Datasets
CIFARIO (ShuffleNet)
CIFAR (ShullleNet)
ImageNet-I1 Ik (SqueereNet)
Datasets
Methods
Top1
Top5
ToP1
Top5
Topl
Top5
Pretrained
92.98
99.73
72.39
91.46
58.178
80.624
Ours(sampling)
93.14 0.25
99.76 0.22
72.60 + 0.15
91.29 0.13
58.257
1.022
81.01
1.251
epochs. We collect the last 10 checkpoints per dataset and utilize our method to learn the distribution of pretrained weights. Furthermore, we combine the pretrained weights of MNIST and CIFAR-IO, learn their distribution, and then evaluate our method on SVHN and STL-10. Subsequently, we reverse this process by combining the pretrained weights SVHN and STL-10, and evaluate our method on MNIST and CIFAR-10.
As shown in Table 23 our method enhances the performance of the pretrained model. Furthermore, we note that learning the full model weights does not compromise performance. Although learning the distribution of the classifier head is computationally efficient, can result in lower performance.
C.12 GENERATING WRIGHTS FOR VISTON TRANSFORMERS
Our method shows the ability to learn the distribution of all parameters within a vision transformer, including convolutional and linear layers. We present in in-distribution evaluation results in plot Figure highlighting the learning of combined weight distributions conditioned on individual datasets. The model 700 for ViT's is collected based models proposed by Gani et al. (2022).
27

---

Published as a conference paper at ICLR 2025
Table 16: Zero-Shot Transfer Learning This Table represent results of zero-shot evaluation against the pretrained model on Resnet18 full model architecture.
Model
MNIST
CIFAR-IB
CIFAR-100
Pretrained
99.61 99.62 0.07
94.56 94.57 + 0.00
75.86 75.83 = 0.02
D2NWG(ourS)
Acouraty
Accuracy
Accuracy
Acouracy
(a) MNIST
(b) SVHN
(c) CIFAR-10
(d) STL-10
Figure Convergence Plots on Finetuning Generated Weigths: Weights generated by the
competing methods are finetuned for 25 epochs on the training set. We utilize the modelzoos of Schürholt et al. (2022c).
D APPLICATION TO LARGE LANGUAGE MODEL (LLM) OUTPUT LAYER GENERATION
---MMMINI--KKInstrcutt conduct experiments on the Microsoft Phi-3-MINI-4KKInttrae model demonstrate the scalability of our method for generating output layers in large language models (LLMs). The model's 98.5 million-pare parameter output layer was split into 96 chunks, each of size 1,026,048, and used as training data for Variational Autoencoder (VAE) with an embedding size of 1,024. Lacking access to original training data, we used class-conditee conditsonni diffusion process, with chunk embeddings as conditioning data. Post-training, conditioned chunks were sampled and concatenated to reconstruct the original output vector. We evaluate our method using the Open-LLM LeadearBoard-L. 1. As shown in Table 17, our approach effectively scales to the LLMs head generation demonstrating adaptability across diverse domains with minimal adjustments to conditioning data.
Table 17: Generating weights for the Microsoft Phi-3 language model output head.
Challenge (25-shats)
Fasy (25-shots)
HellaSwag (I0-shots)
Winogrande (5-shots)
Pretmined
87 + 0.00
63.23 + 0.01
73.65 = 0.01
76.64 0.01
73.65
In this experiment, we show that our method can learn the distribution of any layer in an LLM by modeling the full distribution of GPT-2 small (164M parameters). We use chunk size of 1,523,712 and, unlike Llama architectures, concatenated all vectorized layer weights before chunking them uniformly. Table 18 highlights the method's 's effectiveness the Open LM-Leaderboard benchmark. While it did not outperform the base model overall, significantly improved performance on certain tasks and maintained average accuracy comparable to the pretrained model.
GPT2:
D.1 FAST CONVERGENCE PERFORMANCE EVALUATION
In this section we report supplementary results for experiment on tiny model zoo dataset. The pretrained weights used here are from epochs 21 to 25 for each dataset where 70% of the resulting modelzoo used for training and 15% for validation and testing respectively. The number of pretrained weights in the modelzoos SVHN. The flattened network weights' length is 2864 for CIFAR-10 10 and STL-10 and, 2464 for MNIST and SVHN. We pad all the weights with zero 2864.
3500 for MNIST, CIFAR-10, and STL-10, and 2864 for
28

---

Published as a conference paper at ICLR 2025
Table 18: Performance evaluation on unseen open 1lms leaderboard v2 benchmark base on full gpt2- 164M small. These results are produced by Huggingface after submission to open LLM leaderdoards. 1 indicate performance improvement while indicate a performance decrease
Method
lieaal
Bhh
MATII-hard (4)
Musr
MMLLU-Pro
Ang
Base Model
Fine tuned
apenii commii-mmmmnity-gge
17.8
2,83
1.12
0.3
13.91
1.84
633
Vox
D2NWG
19.16/11 36y 1.80
1,0110,11)
0.8811006)
12.69(11.29
1,6000 16)
6.39((001)
upenii-commurity
No
Table 19: No Fine-tuning Initialization on Unseen Datasets We transfer from one dataset, or combinations of datasets, to unseen datasets at test time.
Source
Target
Accuracy
Methods
MNIST
SVHN
13.25
SVHN
MNIST
29.30
Sxoe33
CIFAR-I0
STL-I0
15.20
STL-10
CIFAR-10
10
15.40
Sampling from Combined Weights Distribution
MIIST+CIFAR-10
SVHN
18,80
MNIst+CIFAR-10
STL-10
16.21
Ours
SVHN + STL-10
MNIST
36.64
SVHN STL-10
CIFAR-10
18.00
D.2 SAMPLING WEIGHTS FOR UNSEEN DATASETS
Task:
We evaluate the transferability of the models on unseen datasets. We create disjoint modelzoos by combining MNIST and CIFAR-10 10 into single modelzoo and combining the SVHN and STL-10 modelzoos. When we train on the MNIST plus CIFAR-10 modelzoos, we test on the SVHN and STL-10 modelzoos and vers-.
As shown in Table 19, D2NWG able to sample weights with higher accuracy on unseen datasets well for in distribution. Through these experiments method does not only outperform the baseline it also demonstrates promising results for dataset-conditionel sampling for unseen datasets.
Results:
E MISCELLANEA
In Table 24 present the parameter count for the model used to learn the distribution of the 25% of Ilama-3.2-1B transformer blocks. In Table 25 we showcase the set of experiments and the corresponding number of parameters generated by D2NWG. Although D2NWG capable of generating up to billion parameters, all our experiments were limited to a maximum of 872 million, achieved using the Llama 3.1-8B model with 4 transformer layers, excluding layer normalization, for which we constructed a separate model. This parameter count makes D2NWG the only method, to the best of our knowledge. capable of generating nearly a billion parameters, significantly enabling large architecture weights generation including GPT-2 and most existing image classification models terms of parameter scale. For non-L..M models, we utilize joint distribution learning, enabling task
Model
MNIST
SVHN
CIFAR-10 10
STL-10
100 07.7978
ML7857
Protrained
Pretrained
99.12 0.05
91.62 0.18
93.51 1 0.16
96.01 1 0.10
11AS5
prob
57.23
D2NWG(Uu D2NWG(mol)
99.55
0.02
95.13 0.10
94.23 1 0.27
94.02 1 0.10
SLANER
97.56
0.36
57.41 :: 0.17
83.64 +: 0:47
95.74 :: 6.74
Cross datasets transfer leamming
OFA (PretrainedCCii al.
D2NW Giprob)
42.56
0.62
20.971 0.78
26.56 1.22
47.33 1 0.32
Table 23: MobileNet Weight Generation.
Figure 9; Experiment with ViT
29

---

Published as a conference paper at ICLR 2025
Table 20: Performance evaluation at initialization without fine-tuning. For the baseline we use weights of SVHN for MNIST and vice versa similarly for CIFAR-10 and STL-10
Datasets
MNIST
SVHN
CIFAR10
STL10
Random
10,23=0.56
12.21=3.76
9.98=11447
9.56=1.02
Pretruined models
82.824: 1.38
67.571 0.59
44.68:1 3,15
35,991 1.15
Schürhott al. (2022a)
69.73 5.12
50.25-t: 6.12
26.06: 3.01
17.20 3.43
seen (D2NWG)
83.92=1.92
61.81 = 3.13
43.08410.55
31,45:0.35
(e)))))NGG)(we with Pred)
84.85-0.83
66.03 +: 1,36
03.89
34,29
Spasoochhrroott et al. (2022aXcross)
29.30:1: 3.46
13.25 1.12
15 15,40:: 0.51
15,20:11.24
(D2NGG)
36.64-4.99
18.50+0.58
18.00+0.222
16.21++0.52
Seen(D2N Pred)
30.15=5.09
15.76
17.10:11.12
15.37:10.52
Table 21: In-distribution performance comparison of different image dataset encoding schemes model zoo dataset
Datascts
MNIST
SVHN
CIFARIO
STL10
Pretrained models
82.82: 1,38
67,571 0.59
44,68 3.15
35,9911 1.15
Schürrott et al. (2022a)
69.73= 5.12
50.25 6.12
26.06 3.01
17.20 3.43
MLP_Encoder
MLP_ Encoder
67.04
35.65 +: 13.03
17.41
20.3611..38
Set transf(pret)
78.21+-1.66
60.90 + 1.08
28.68+1.84
34.75+000.3 38
seen (D2NWG)
83,92.11.92
61.81 1 3.13
43.08.1..S
31.45.0.355
(Dtn Pred)
84.85++0.83
66.03 1,36
43.89-+0.15
34,29+0.13
Tis arreld patruieed
Dass
-10
-10
(a) TSNE
(b) Density
2.573
0.850
6:755
6.75
6.799
Preraised
Oura
(c) Accuracy
Figure 10: Analysis of relationship between the prefrained weights and the sampled weights for MNIST dataset
datusee-comditooned sampling. For example, CIFAR-10 10 and ImageNet are considered two separate datasets, while SST-2 and CoLA in the GLUE benchmark are treated as two distinct tasks, regardless of differences in the number of classes subtasks within each dataset or task. Table 25 highlights
30

---

Published as a conference paper at ICLR 2025
Table 22: Performance of the datasets conditional sampling on 10 unseen real-world datasets. We report the averaged accuracy on ten unseen test datasets over 3 different runs fine-tuned for 50 epochs. pret(imnet): pretrained on imagenetIk
No--iin-tuning pret(imnet)
epachs Tuning pret(imnet)
Datasets
Random init.
(
Random init.
D2NWG(ourS)
of classes
Gemstones
1.13 0.52
0.62
1.86
70.59= 0.91
67.49= 0.43
76.06
Dog Breeds
0.55 + 0.22
0.69 + 0.00
1.87 + 0.39
80.78+ 0.28
13+ 0.49
80.88 + 0.88
133
Colorectal Histology
11.77 12.88
11.00 1: 0.00
18.12 1: 0.25
90.34 = 0.33
89.752 0.19
93.65 1 0.10
8
Drawing
+0,93
90.20
90.00= 0.16
Alien vs Prodator
51.48 +2.09
28. 88 +: 0.00
78.15 +0052
98.52- 0.52
98.89- 1.42
91.77 + 0.00
COVID-19
20.13 18.66
46.53 0.00
47.22 +00.00
93.86-0.16
93,40- 0.49
94.56 0.71
honey bee pollen
49.54 11.30
50.00 0.00
56,94 1.4.53
93.05 0.00
88.891 0.00
93.35
4,53
Speed Limit Signs
30.55 +-2.27
25.00 0.00
31.48 10.23
83,334
86.11+ 0.00
90.74
1,31
Jupanese Characters
0.03.10.00
0.08 1. 0.00
0.50.10.22
53.17 0.15
62.33 0.16
62.16 1.0.7 0.45
1566
Table 24: Model components and their configuration modes for Ilma3.2.1B
ID
Name
Type
Params
Mode
0
Model
Drappe
102 M
Train
Model Ema
LitEma
Train
2 First stage Model
VAENoDiscModel
553 M
Eval
Cond Stage Model
IdentityCnndStage
Eval
that the proposed method supports text and image conditioning, as well layer- or chunk-wise conditional sampling. D2NWG is one the first weight generation methods to produce over 800 million parameters in single instance without tiling. Additionally, it is among the first to effectively explore weight generation across various domains, learning the distribution of combined models pretrained on diverse tasks or datasets.
31

---

Published as a conference paper at ICLR 2025
Table 25: Summary of Experiments for Figures and Tables presented. Min #els and Max #els
correspond to the minimum and maximum number of classes respectively.
Object
# Datasets
Min #cls
Max #els
#Params
Trainset Size
Conditioning
Table 1
10
5
2565/8005
50k
Dataset
Table 2
5
50
25600
20k
Dataset
Table 3
30
19
706
M
30
Dataset
Table 4
4
10
10
10853
Dataset
Table 5
6
2
3
0.6M
Text Description
Table 6
NA
NA
NA
872M
NA
Chunk Indices
Table 7
NA
NA
NA
872M
NA
Chunk Indices
Table 9
2
10
100
0.7M
2
Dataset
Table 11
2
100
2048
Dataset
Table 15
3
10
1000
1.4M
Dataset
Table 16
3
100
11M
Dataset
Table 16
4
10
10
2.8M
4
Dataset
Table 17
NA
NA
NA
96M
NA
Chunk Indices
Table 18
NA
NA
NA
164M
NA
Chunk Indices
Figure
10
1566
136468
140
Dataset
Figure
2
10
100
0.47M
Dataset
Figure 6a
2
10
10
5310
2
Dataset
Figure
2
10
10
5310
Dataset
Figure
5
10
200
2.8M
5
Dataset
32


Reviews:
[{"review_id": "191XhwRQSo", "reviewer": "Reviewer_V98o", "summary": "Authors propose a diffusion-based approach to generate parameters of neural networks for transfer learning conditioned on the target dataset. Prior work often relies on the diversity of pretrained models or datasets used to train these parameter prediction models but it limits their ability to generalize to new datasets. Encoding both model parameters and architecture jointly enhances the generality of the parameter prediction approach to unseen tasks. Several experiments are performed to test the efficacy of the approach for domains such as vision and large language models, demonstrating that it outperforms several meta-learning and parameter prediction approaches.", "questions": "See weakness section for my concerns and questions.\n\n\nWriting suggestions:\n- What do the task descriptions look like for NLP tasks which are used for encoding them? Adding pointers in section 3.3 (line 234-237) will be helpful for readers.\n- Section 3.3 is a bit confusing to read due to paragraph titles. It looks like authors use 4 types of encoding in their work corresponding to 4 paragraphs but it’s actually just 2 (image and language dataset encoding). The middle two paragraphs (set-based dataset encoding and contrastive dataset encoding) are essentially describing image dataset encoding in detail. It may be a good idea to re-organize this section to be clearer.\n\nNit:\n- Double dot at the end of line 116.\n- Line 1147: “ios” -> “is”\n- Line 184: “Parametrs” -> “Parameters”\n- Line 1323: “metho” -> “method”\n- Line 512: “robustnets” -> “robustness”\n- Line 1331: remove “par”", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach is conceptually simpler to train and test than previous meta-learning approaches which often involve solving a bi-level optimization problem.\n- The dataset conditioning in the approach increases the generality of the approach to new unseen datasets.\n- Extensive sets of experiments are presented for different settings such as few-shot learning, zero-shot learning, model retrieval, classifier head adaption, LoRA weight generation, and adapting LLM weights for specific tasks.", "weaknesses": "- Although the authors argue that previous approaches rely on diversity of training architectures and dataset for generalization, the proposed approach also has limitations along the same axis: it may only generalize to new datasets which are in-distribution for the dataset encoder otherwise the dataset encodings can be noisy. For instance, if the dataset encoder is only trained on imagenet-like natural images, it may not generalize to unseen dataset distributions like medical images. Table 17 seems to suggest the same since accuracies for unseen datasets are poorer in absolute numbers, so it’s unclear if dataset encoding is really needed otherwise one could simply train a parameter prediction model on combined datasets which could generalize to new ones.\n- Missing baselines/ablations: Building on above, there are no ablations or baselines testing how effective are each component in the proposed approach: dataset encoding, layer-vectorization, spectrum ranking. It’s unclear how much importance all these components play in the final performance of the proposed approach. Moreover, some crucial baselines are missing -- for instance, training a simple param prediction diffusion model (e.g. Peebles et al. (2022)) without dataset encoding vs training it with dataset encoding to see how much generalization performance is improved with dataset encoding (which is one of the main contributions of this work).\n- It seems that image dataset encoding uses the dataset samples (images) with a set transformer model whereas the language dataset encoding is done by simply picking LLM embeddings of the task description which seems weaker. If it’s not, is it possible to also encode image datasets in the same task-description-based way using LLM or VLM?\n- For LLM param generation, authors use spectrum technique from prior work in order to deal with a large number of parameters in LLMs but it’s unclear how effective it is. It may be a good idea to use this technique on small-to-medium scale models where encoding all parameters is feasible to gauge the loss of performance and potential scope for improvement in this stage.\n- It seems that authors train separate D2NWG models for param prediction in each setting. Is it possible to train a single param prediction model? Are there any technical hurdles in the approach which could prevent doing this (assuming access to large-scale compute and data)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xdHDgq0dv5", "reviewer": "Reviewer_ws3W", "summary": "This paper proposes a diffusion-based neural network weight generation technique named D2NWG, which directly learns the weight distributions of models pre-trained on various datasets. Specifically, the authors have developed a complete framework for latent diffusion models to learn from pairs of weights and datasets. The paper evaluates D2NWG across various tasks, both for direct generation and additional fine-tuning, outperforming state-of-the-art meta-learning methods and pre-trained models. The results demonstrate a capability for scalable transfer learning.", "questions": "Please refer to the weakness for my questions. If it is solved, I am willing to raise my score.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper investigates an interesting problem, the methodology is promising, and the paper is well-written and easy to read.\n2. The experimental results are very solid, covering diverse tasks across a variety of datasets. These results outperform state-of-the-art meta-learning methods and pre-trained models.", "weaknesses": "1. Although the idea is interesting and the methodology looks promising, my main concern is about the practical significance of the proposed method, considering that training an effective diffusion model requires substantial computational resources. I will expand on this concern below:\n2. Tables 1 and 2 show that D2NWG outperforms several baselines. Notably, D2NWG is tagged with 'CH', indicating that it only modifies the last classifier layer, similar to linear probing. This suggests that it may still be challenging for D2NWG to generate complete parameters with robust performance.\n3. The concept of 'Zero-shot adaptation' does not seem applicable as the weights generated by D2NWG are conditioned on the target dataset (possibly the test set). This could be considered a form of 'test data leaking', which limits the practical utility of D2NWG.\n4. Section 4.2 discusses the need for further fine-tuning of the generated models, suggesting that D2NWG's role is similar to that of pre-trained models. However, there is no evidence to suggest that training a usable D2NWG model is more cost-effective than using pre-trained models.\n5. The generalization capability remains the most challenging issue for D2NWG. Although the authors attempt to demonstrate its adequacy across a range of datasets, there is no evidence that a well-trained D2NWG can handle diverse tasks and different model architectures simultaneously. Currently, D2NWG seems like a suboptimal alternative to a well-developed optimizer. Additionally, the supporting components such as the autoencoder and the dataset encoder, which are trained separately, also suffer from generalization issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uXJM0vied8", "reviewer": "Reviewer_oEWb", "summary": "This paper presents a method called D2NWG, a diffusion-based neural network weight generation technique that enhances transfer learning by generating task-specific neural network weights based on target datasets. Unlike traditional transfer learning approaches that require fine-tuning or access to large collections of pretrained models, D2NWG uses a latent diffusion process to model and sample weight distributions, allowing efficient generation of adaptable weights for both seen and unseen tasks. This approach is scalable to large architectures, including large language models, and shows improved performance across various domains, outperforming existing meta-learning and pretrained models. The technique has been tested extensively, demonstrating superior adaptability and convergence speed, making it a promising tool for scalable transfer learning and model customization across diverse tasks.", "questions": "1. **Quantifying and Mitigating Bias in Public Checkpoint Weight Distributions**: The paper leverages publicly available pretrained models for weight distribution modeling, but it doesn’t address how potential biases in these weights—arising from the original datasets or training methods—might affect the generated weights. How can biases be quantified in these weight distributions, and what mechanisms could be implemented to ensure the generated weights are robust to such biases? Addressing this might require specific adjustments to the diffusion model or additional constraints on the weight distributions.\n\n2. **Criteria for Selecting Additional Datasets for Training Target Architectures**: The method involves further training on a curated selection of datasets to improve model adaptability. However, there is little guidance on how to select these datasets to optimize the target architecture effectively. What are the criteria or metrics that could guide dataset selection to ensure it sufficiently represents the intended tasks and enhances generalization without introducing redundancies or biases?\n\n3. **Evaluating the Impact of Generating All Parameters Versus Only the Classification Head**: In Table 1, the results compare the performance of models with generated classification heads to baseline models. However, it would be informative to understand the effect of generating all parameters instead of just the classification head. Would this provide additional performance gains, or might it introduce complications in terms of stability or adaptability to new tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The authors considered a wide range of evaluation settings (zero/few-shot learning, model retrieval, fine-tuning, domain transfer, conditioned weight generation, etc.) and provided comprehensive experiments.\n2. The authors considered multiple ways to vectorized the model weights and encode the datasets.", "weaknesses": "1. **High Workload and Tuning Challenges Due to Additional VAE Training**: The additional VAE for parameter encoding introduces significant computational workload and requires hyperparameter tuning, making it less scalable and potentially costly. What steps could be taken to simplify this process, and are there alternative methods for parameter encoding that maintain performance but reduce overhead?\n\n2. **Ambiguity in Dataset Encoding Choices in Section 3.3**: The paper mentions several methods for encoding datasets, but it’s unclear how to choose the best encoding approach for a new dataset. Could the authors clarify the conditions under which each encoding method would be preferable, and provide a decision-making framework or guidelines to help practitioners adapt this approach to their datasets?\n\n3. **Minor Typos and Formatting Issues**: There are minor typos and formatting inconsistencies in the paper that could be addressed to improve clarity and professionalism. For example, a colon is missing on line 177, and \"while\" on line 184 should begin with a capital \"W.\" A thorough proofreading could enhance readability and reduce the risk of misinterpretation due to formatting errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JPOLNMda25", "reviewer": "Reviewer_4nQa", "summary": "This paper proposes to generate parameters using the latent diffusion model, which is a very new direction. Compared to traditional meta-learning, the authors want to show that D2NWG can generate parameters for both seen and unseen dataset, be applied to either classification task or NLP task, and achieve better accuracy with faster convergence. The most important contribution comes from the dataset encoding and use it as the condition when generating parameters.", "questions": "I will list my questions according to the order appearing in the paper. You may want to combine some of them when rebuttal if necessary.\n- Introduction:\n    - You mentioned recent methods in latent diffusion-based weight generation in line 47 and said their limitations are limited to small models and a subset of parameters. However, in this paper, you try to generate parameters for classification heads (in Sect. 4.1) and LoRA (in Sect. 4.4). It seems that you are just generating a small number of parameters. So is this paper different from existing methods under this context? You also mentioned \"generating weights for architectures with over 200 million parameters\" in the fourth contribution, but I cannot find the supporting experiments for that.\n    - You also mentioned some meta-learning few-shot methods in line 51 and said their generated parameters still need fine-tuning. Does this paper overcome the requirement of fine-tuning? I asked this question because in Table 2 of Sect. 4.1.2, the accuracy of D2NWG on those simple classification tasks are still not satisfying, which indicates the requirement of further fine-tuning. For example, for CIFAR-10 and Aircraft, I'm sure that after a few epochs of fine-tuning, the accuracy will reach above 90%. Then the current accuracy of D2NWG is not acceptable.\n    - In Figure 1, in the sampling stage, the unseen architecture is also used as the condition to generate parameters. However, there is no explanation about unseen architecture in the main paper. I do see the results in the Appendix, but the unseen architecture is not really used as the condition. The method actually generates parameters for the seen architecture and then wraps it for the unseen architecture with some \"careful concatenation\" as mentioned in line 1171. So what is the \"careful concatenation\" here? I believe this part should be put in the main paper to fit Figure 1.\n\n- Approach:\n    - Typo in line 171: \"The extraction process follows one one the main preprocessing methods:\"\n    - When will the model-wise vectorization be used and when will the layer-wise one be used?\n    - Can you give an intuitive explanation about set-based dataset encoding in line 208, like encoding images from each class separately, then convert to the dataset encoding? It is currently hard to understand $x_p^{y_i}$.\n    - In dataset-conditioned training, why the weights latent is used as the condition as well? It's like in the text-to-image diffusion model, the model will be conditioned on the text only, and the image is the distribution to learn. Why it is used as a condition as well? If it is conditioned on the weight, how can you do the inference without weight latent?\n    - In Sect. 3.5, you mentioned \"ranks LLM layers based on their importance\", however, the experiments in Sect. 4.4 is about LoRA. If you choose to claim that in the main paper, you should put relevant experiments in the main paper.\n\n- Experiments:\n    - In Table 1, can you explain what is 5-way 1-shot a little bit?\n    - In Table 2, for D2NWG_CLIP, I believe it is using CLIP. Then how about the two rows of D2NWG? What does it use?\n    - In Sect. 4.1.1, you mentioned \"evaluate the performance on 600 subsets from the UNSEEN test split\". What do you mean by \"unseen\" here? It's like in general model training, we have train set and test set, and we test the trained model on the test set. Now you have multiple trained models, and you use that to train the diffusion model to generate weights, which is expected to work on the test set just like what we did in the normal training. Why it can be called \"unseen\" task?\n    - In line 348, you mentioned \"potentially eliminating or significantly speeding up the fine-tuning process\". As I mentioned in the last point, it's normal to NOT fine-tune if it is a SEEN task/dataset. While for real unseen tasks as you did in Sect. 4.3, the generated parameters do need fine-tuning.\n    - Starting from Sect 4.1.3, the experiments become hard to follow since you change the settings in each section frequently. It's better to organize your experiments better. For example, for the experiments in Sect. 4.2, we can still use the dataset in Table 2. Otherwise, it's hard to understand (1) What are the train and test datasets? Are the test ones seen or unseen? How do you define unseen? (2) How do you train the diffusion model? What models' parameters do you use to train the diffusion model? \n    - In Figure 2, why dessert food is so good from the beginning? Do you have any clue? Whether because the class/dataset is put in the trainset occasionally?\n    - Sect. 4.4. is still seen task case right?\n\n- Suggestion:\n    - So generally you'd better make it clear about what you want to claim. For example, you may want to say, for the seen task, the generated parameters can be used directly with high accuracy. For the unseen task, the generated parameters may need fine-tune but with better final accuracy and faster convergence. \n    - Then for each claim, you will have an experiment section to support. Each section may include both results for CNN classification and NLP task.\n    - The settings should be uniform throughout the paper. If some experiments do need different setting, give reason.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- This paper is interesting in terms of the overall topic - generating the model parameters using the latent diffusion model. To my knowledge, there are only two other papers [1,2] in this direction and they have not been published yet. \n- The idea is easy to understand.\n\n[1] Wang K, Xu Z, Zhou Y, et al. Neural network diffusion. arXiv 2024.    \n[2] Jin X, Wang K, Tang D, et al. Conditional lora parameter generation. arXiv 2024", "weaknesses": "- The presentation is poor, especially the experimental parts. It's like people writing each subsection separately and putting them together directly without good organization. \n- There are gaps between the claimed contribution and the real work they did. \n\nPlease refer to the questions part. Thanks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vhBLOFI18f", "reviewer": "Reviewer_ZVaq", "summary": "This paper introduces D2NWG, a diffusion-based method for generating neural network weights conditioned on target datasets. The key idea is to learn the distribution of pretrained model weights across different datasets and use latent diffusion to generate new weights that are optimized for specific tasks. The method can generate both classifier heads and full model weights, and scales to large architectures including LLMs. Extensive experiments demonstrate strong performance on few-shot learning, zero-shot transfer, and model retrieval tasks.", "questions": "**[Q1]:** Could the authors provide a detailed analysis of the computational resources required to maintain and utilize the extensive collections of pretrained weights in D2NWG? Specifically, how does the storage and processing overhead of D2NWG compare to traditional pretraining and fine-tuning methods, particularly when scaling to very large models such as LLMs?\n\n**[Q2]:** In terms of generalization, how does D2NWG ensure that the generated weights do not overfit to specific characteristics of the training datasets? Are there mechanisms in place to promote diversity and prevent homogenization of the generated weights?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Paper introduces a novel approach by combining latent diffusion models with neural network weight generation\n  \n- The authors conduct thorough experiments across various tasks, providing strong empirical evidence of the method’s effectiveness.\n  \n- The method exhibits robust performance both within the distribution of training datasets and on unseen datasets", "weaknesses": "1. **Limited Theoretical Analysis:**\n   \n   While the paper presents strong empirical results, it lacks a comprehensive theoretical framework that explains why diffusion-based weight generation effectively enhances transfer learning and generalization.\n\n2. **Computational Costs:**\n   \n   The D2NWG method necessitates maintaining extensive collections of pretrained weights across a multitude of datasets. This requirement can lead to significant storage and computational overhead, potentially surpassing the costs associated with traditional pre-training approaches. Additionally, details on the training time and resource consumption compared to existing methods are limited.\n\n3. **Reliance on Dataset-Specific Encoding:**\n   \n   The effectiveness of D2NWG heavily depends on the dataset encoding mechanisms, such as the Set Transformer and CLIP-based encoders. This reliance may limit the method's adaptability to datasets with unique characteristics or those that significantly differ from the training datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)