Diffusion-based Neural Network Weights Generation

## Abstract

Abstract Transfer learning has gained significant attention in recent deep learning research due to its ability to accelerate convergence and enhance performance on new tasks. However, its success is often contingent on the similarity between source and target data, and training on numerous datasets can be costly, leading to blind selection of pretrained models with limited insight into their effectiveness. To address these challenges, we introduce D2NWG, a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning, conditioned on the target dataset. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation, learning the weight distributions of models pretrained on various datasets. This allows for automatic generation of weights that generalize well across both seen and unseen tasks, outperforming state-of-the-art meta-learning methods and pretrained models.
Moreover, our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques that rely on task-specific model collections or access to original training data. By modeling the parameter distribution of LLMs, D2NWG enables task-specific parameter generation without requiring additional fine-tuning or large collections of model variants. Extensive experiments show that our method consistently enhances the performance of diverse base models, regardless of their size or complexity, positioning it as a robust solution for scalable transfer learning.

## 1 Introduction

In recent years, generative AI models have transformed artificial intelligence, with impactful applications in natural language processing (NLP), audio generation, image creation, and video synthesis (Gozalo-Brizuela & Garrido-Merchán, ). Among these, diffusion-based models have emerged as a leading approach for generating real-valued data through a denoising process (Ho et al., ; Rombach et al., ; Peebles & Xie, ; Gao et al., ). These models have achieved remarkable results across diverse applications, including task-conditioned signal generation and image and video synthesis, showcasing their versatility and effectiveness in various domains  (Yang et al., ).

Extending Denoising Diffusion Probabilistic Models (DDPM) to neural network weight generation will offer a powerful solution for improving transfer learning and model adaptation. By dynamically generating task-specific weights, such an approach enables more efficient training, better generalization, and faster adaptation, unlocking new potential for automatic machine learning (AutoML) (Hutter et al., ; Doke & Gaikwad, ).

Generative hyper-representation learning (Schürholt et al., ), which generates neural network weights from pretrained distributions, is gaining traction. While recent methods like latent diffusion-based weight generation (Wang et al., ; Sch"urholt et al., ) show promise, they are limited to small architectures or arbitrary parameter subsets without clear justification. Furthermore, they focus solely on in-distribution sampling, overlooking the critical need for task- and dataset-conditioned weight generation for unseen tasks.

To address multi-task learning, Nava et al. () and Zhang et al. () proposed meta-learning methods for weight generation in visual and few-shot tasks. However, these approaches produce suboptimal parameters that still require fine-tuning for both in-distribution and out-of-distribution tasks, limiting their efficiency and adaptability.

Addressing these limitations could unlock more robust, scalable solutions for adaptive learning across diverse tasks. Additionally, most of these methods are focused on vision tasks, particularly image classification, overlooking their potential in large language models (LLMs). Expanding these approaches to LLMs could unlock significant advancements in task-specific weight generation and model adaptation as well as pretrained model optimal parameters space exploration.

In this work, we introduce a generative hyper-representation approach and reffered to as D2NWG , that leverages latent diffusion to harness knowledge from a vast collection of pretrained neural network weights across diverse datasets. Our method generates dataset- or task-conditioned parameters tailored to specific tasks, serving as a generative retriever for in-distribution tasks without fine-tuning and as a task-adaptive parameter generator for unseen tasks.

We extend this approach to large language models (LLMs) by using diffusion-based weight generation to explore optimal weight spaces. This preserves the integrity of pretrained weights while generating task-specific parameters. We hypothesize that both original and task-specific fine-tuned weights lie within the same manifold, enabling our method to produce diverse, optimized weights without the need for fine-tuning. An overview of our method is depicted in Figure 1.

Figure: Figure 1: Stage 1: VAE Encoder and Decoder training process. Stage 2: dataset encoder training stage . Stage 3: Dataset conditioned diffusion process.
Refer to caption: x1.png

Our contributions are as follows:

- •
We introduce a novel neural network weight generation framework using latent diffusion models, generating task-adaptive weights conditioned on datasets or task descriptions.
- •
D2NWG outperforms meta-learning approaches on unseen datasets by generating superior weights.
- •
Our method surpasses the performance of pretrained weights and enables rapid adaptation for both classifier heads and full models.
- •
It scales to small and large datasets, generating weights for architectures with over 400 million parameters including GPT2-Small.
- •
We demonstrate its effectiveness in improving LLM performance by generating task-specific weights from a single pretrained model and our sampled weights based LLAMA3-.1-8B and LLAMA3-.2-1B models ranked among top 2 performing models on the open lm-leaderboard(^1^11[https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard))

## 2 Related work

As neural networks find broader applications, leveraging pretrained weights has become essential for accelerating transfer learning. In this context, generating network weights directly from model zoos, without relying on traditional training pipelines, offers a promising path to improve model efficiency. Several hypernetwork approaches for weight prediction have been proposed to address this challenge (Chauhan et al., ; Ratzlaff & Fuxin, ; Denil et al., ; Ha et al., ).

Neural Network Parameters Prediction:
Recently, Zhang et al. () introduced Graph Hypernetwork (GHN) to generate weights using a model’s directed graph representation. This was enhanced by Knyazev et al. () with GHN2, which focused on generating weights across architectures for the same datasets. Similarly, Zhmoginov et al. () treated weight generation as an autoregressive process, using a transformer to generate weights layer by layer, though this approach is less scalable due to the need for a transformer per layer. Building on this, Knyazev et al. () combined transformer-based techniques with GHN2 to create GHN3, improving generalization across architectures and datasets. However, GHN3’s deterministic sampling limits its generalization, relying heavily on the diversity of training architectures and the pretrained dataset..

Meta Pretrained Weight Generators: Nava et al. () proposed HyperLDM, a generative model for weight generation in visual question answering (VQA) tasks. This model leverages the distribution of weights pretrained in a meta-learning setting and uses latent diffusion for sampling. Similarly, Zhang et al. () integrated diffusion-based meta-weight generation to enhance adaptation for few-shot learning. While generating pretrained weights through meta-training shows promising results, the meta-learning process can be computationally expensive. Additionally, the meta-pretrained weights are not optimal even for in-distribution evaluation they always require some optimization steps.

AutoEncoder-based Weight Generators: Schürholt et al. () proposed learning the distribution of weights by reconstructing them using autoencoder-style architectures. In a follow-up work, Schürholt et al. () introduced a method for learning the distribution of pretrained weights, allowing for unconditional sampling of diverse weights through kernel density estimation. A related approach by Peebles et al. () involves conditioning weight generation on the target loss using a diffusion transformer framework.

Diffusion Models: Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., ; Rombach et al., ; Blattmann et al., ; Croitoru et al., ) enable mapping data representations to a Gaussian distribution and vice versa achieving state-of-the art performance in generative modeling. Latent diffusion models (Dhariwal & Nichol, ; Nichol & Dhariwal, ; Ho & Salimans, ; Zhu et al., ; Rombach et al., ) allow for manipulation of the learned latent space. We utilize such models for our dataset-conditioned weight sampling.

Large Language Models:
Large language models (LLMs)(Minaee et al., ; Zhao et al., ) like GPT-4, LLaMA(Dubey et al., ), and Mistral have transformed natural language processing, demonstrating strong performance across diverse tasks such as summarization, translation, and text generation. While fine-tuning these models for domain-specific tasks enhances their capabilities, their large parameter count poses challenges for efficient optimization without degrading core functionality. Although weight generation offers potential for exploring optimal LLM configurations, research in this area remains limited.

Applications of Generative Hyper-Representation Learning: Despite skepticism around learning from pretrained weight distributions, generating diverse weights for the same model is key to improving model flexibility, initialization, and transfer learning. Task-conditioned parameter generation builds on this by encoding weight distributions from models trained on various datasets, allowing for more robust and adaptive weight retrieval, which can significantly enhance performance across tasks. While previous works, like Tang et al. () and Zhao et al. (), focus on task-conditioned LoRA weight retrieval, and Gong et al. () explores LoRA weights for image generation, our approach extends these efforts by generating neural network weights across a wider range of applications.

## 3 Approach

### 3.1 Dataset conditioned Neural Networks Generation

In this work, we first focus on the following problem: Given a set of neural network models pretrained on a collection of $M$ datasets $\{\mathfrak{D}_{1},\mathfrak{D}_{2},\ldots,\mathfrak{D}_{M}\}$, we aim to learn the distribution $p(W)$ of the pretrained weights ($W$) of these models. Our goal is to enable conditional sampling of a set of weights $p(W_{new}|\mathfrak{D}_{new})$ for a new dataset or tasks $\mathfrak{D}_{new}(x,y)$, such that these weights can achieve good performance on the new dataset either without further training or with only a few optimization steps compared to random initialization. The intuition is that there is a direct relationship between a pretrained network weights and the dataset it was trained on (see Appendix A.1 for a formal argument). By understanding the distribution of pretrained weights and their alignment with the source dataset or tasks, we can generate high-performing weights for a target dataset with minimal or no optimization.

### 3.2 Weight Encoding

Dataset collection: We utilize publicly available pretrained models from model zoos and additionally train target architectures on selected datasets. For each model, we extract the pretrained weights, which are then aggregated to construct our training data. The extraction process follows one one the main preprocessing methods:

Model-wise vectorization: For each model $\mathcal{M}_{i}$, we flatten the weights of each layer into a vector denoted by $w$. These vectors are concatenated to form a single vector $W_{i}\in\mathcal{R}^{1\times d_{i}}$, where $d_{i}$ represents the total number of trainable parameters in the model. To ensure uniformity, all vectors are zero-padded to match the maximum length $d_{\text{max}}$ across models.

Layer-wise vectorization Here, instead of concatenating the weights, each layer’s flattened weight vector is kept separate. This allows for layer-wise sampling during inference, treating each vectorized layer as an independent input for later stages. Since the network layers have varying dimensions, we first zero-pad each flattened parameter vector $\bm{w}\in\mathbb{R}^{mn}$ to a length that is a multiple of a chosen chunk size. We then split the padded vector into smaller chunks of subvectors $\bm{\bar{w}}_{i}\in\mathbb{R}^{l}$, where $i\in\{1,\dots,k\}$ and $l=\bm{\lceil mn/k\rceil}$, ensuring that each chunk is of equal length. $mn$ represents the length of the flattened weight vectors

Parametrs Encoding: We then train a Variational Autoencoder (VAE) to encode these vectors. while minimizing the objective function defined in 1:

$$ $\mathcal{L}=-\mathbb{E}_{q_{\phi}(z|w)}\left[\log p_{\theta}(w|z)\right]+\beta \text{KL}\left[q_{\phi}(z|x)\,||\,p(z)\right]$ (1) $$

where $w$ is the vectorized weight, $z$ is the latent representation, $p_{\theta}$ and $q_{\phi}$ the reconstruction and approximate posterior terms respectively, and $p(z)$ the prior distribution. For the prior, we used a Gaussian. $\beta$ is a fixed hyper parameters that regulate the stochasticity of the VAE. Higher value increase the randomness while lower value increases the reconstruction precision with less randomness. Model-wise and layer-wise vectorized parameters are encoded using the same VAE structure, with the only difference being in the input dimensions.
In chunk-wise encoding, the original flattened vector $w$ is recovered by reassembling the decoded latent chunks through concatenation. The reconstructed chunks $\hat{w}_{i}$ from each layer are concatenated to ensure $\hat{w}=\hat{w}_{1}\oplus\hat{w}_{2}\oplus\dots\oplus\hat{w}_{k}$, where $\oplus$ denotes concatenation. And reshaping $\hat{w}$ back into the original form $\hat{W}$ yields a close approximation of the original weight $W$. The quality of reconstruction is assessed by evaluating the reconstructed weights on a designated evaluation dataset or task.

### 3.3 Dataset Encoding

Image Dataset Encoding: To sample weights conditioned on a dataset, it is crucial to establish an efficient mapping between the latent representations of pretrained weights and the datasets used for pretraining. However, encoding an entire image dataset with multiple classes and millions of samples is challenging. To overcome this, we employ the Set-based dataset encoding with the Set Transformer (Lee et al., ), following the approach used in dataset-adaptive methods (Jeong et al., ; Lee et al., ), which has proven to be effective for dataset encoding. Figure 4 in the appendix provides an overview of the dataset encoder architecture.

Set-based Dataset Encoding: Given a dataset $\mathfrak{D}=\{(x_{i},y_{i})\}_{i=1}^{N}$, where $x_{i}$ and $y_{i}$ are input-output pairs, we form subsets $s_{i}=(x_{p}^{y_{i}})_{p=1}^{K_{i}}$ with $K_{i}=|s_{i}|$ and $s_{i}\in\mathbb{R}^{K_{i}\times c\times h\times w}$, where $c$, $h$, and $w$ represent the image channel, height, and width. The dataset is reorganized into a collection of subsets $\mathcal{S}=\{s_{i}\}_{i=1}^{C}$, where $C$ is the number of distinct classes, grouping images by class. To encode the dataset, we define a transformation $\mathcal{T}$ over the subsets, mapping each set $s_{i}$ to an embedding $z_{s_{i}}\in\mathbb{R}^{1\times d}$. Note that we reuse the notation $z$ from Section 3.2 for weight embeddings. These embeddings are aggregated into a new set $\tilde{s}_{i}\in\mathbb{R}^{C\times d}$, and another transformation $\mathcal{T}$ is applied to produce the final encoded dataset $z_{\mathfrak{D}}\in\mathbb{R}^{d}$. This process is summarized as the composition of Set Transformer blocks: $z_{\mathfrak{D}}=\mathcal{T}\circ\mathcal{T}(\mathcal{S})$. In summary, we refer to this dataset encoding operation as $\mathcal{T}$. The output of the dataset encoder is independent of the number of classes and samples per class, and it does not use the labels. Training the Set Transformer modules alongside the diffusion process can be computationally expensive. To mitigate this, we propose a CLIP-style objective that aligns the dataset embeddings with the corresponding encoded pretrained weights. This alignment can be performed once and then efficiently probed linearly during the diffusion process optimization.

Contrastive Dataset Encoding: The dataset encoder is trained independently using a CLIP-style contrastive loss to align dataset embeddings with weight embeddings. This idea is similar to HyperCLIP (Nava et al., ) for class-conditioned VQA, but instead of a text encoder, we use a dataset encoder, and the VAE encoder from Section 3.2 is used to encode the corresponding weights. Specifically, we use contrastive pairs $\text{Embed}((\mathfrak{D}_{i}(x_{j},y_{j})_{j}^{c_{i}},W_{i}))=(z_{\mathfrak{
D}_{i}},z_{i})$, where $z_{\mathfrak{D}_{i}}$ is the dataset embedding and $z_{i}$ is the VAE-encoded weight embedding. The Set Transformer-based dataset encoder $\mathcal{T}$ is trained to align $z_{\mathfrak{D}_{i}}=\mathcal{T}(\mathfrak{D}_{i}(x_{j},y_{j})_{j}^{c_{i}})$, where $z_{\mathfrak{D}_{i}}\in\mathbb{R}^{1\times d_{z}}$, with the weight embeddings $z_{i}$ using the training objective in Eq. 2.

$$ $\mathcal{L}_{CLIP}=-log\frac{exp(z_{i}.z_{\mathfrak{D}_{i}/\tau)}}{\sum_{k=0}^ {N}exp(z_{i}.z_{\mathfrak{D}_{k}}/\tau)}$ (2) $$

In this step only the frozen VAE encoder is alongside the dataset encoder.

Language Task Encoding: To enable task-description-based parameter generation for NLP tasks, we first encode each task description using Llama-3-8B-Instruct. The output from the last hidden layer is used as the task’s dataset embedding. These embeddings are then directly incorporated into the diffusion process during both training and inference.

### 3.4 Dataset-Conditioned Parameters Generation

At this stage, we have access to a pretrained VAE for encoding neural network weights and a pretrained Set Transformer module to encode entire datasets. The next stage involves defining a model to generate latent representations of weights conditioned on the dataset embeddings. We achieve this by using diffusion probabilistic models (DDPM) (Ho et al., ; Rombach et al., ) trained on the latent representation of the pretrained weights..

Forward Process: Given a weight embedding $z$, obtained from the encoder of the pretrained VAE, the forward diffusion process involves successive Gaussian noise perturbations of $z$ over $T$ time steps. At time step $t$,

$$ $p(z_{t}|z_{t-1})=\mathcal{N}(z_{t};\mu_{t}=\sqrt{1-\beta_{t}}z_{t-1},\beta_{t}I)$ (3) $$

where $\beta_{t}\in(0,1)$ is the noise variance and $p(z_{1:T}|z_{0})=\prod_{i=1}^{T}p(z_{t}|z_{t-1})$.

Reverse Process: As in most DDPM approaches the reverse process is approximated by a neural network such that:

$$ $p_{\theta}(z_{t-1}|z_{t})=\mathcal{N}(z_{t-1};\mu_{\theta}(z_{t},t),\Sigma_{ \theta}(z_{t},t)),$ (4) $$

where $\mu_{\theta}$ and $\Sigma_{\theta}$ are neural networks.

Dataset-Conditioned Training: The diffusion model is trained on the VAE embeddings $z$, conditioned on the dataset embeddings concatenated with the latent representations of the weights. To leverage existing architectures, we designed the VAE to generate latent representations that are compatible with standard latent diffusion models with minimal adjustments, optimizing the latent diffusion objective defined in Eq. 5.

$$ $\mathcal{L}_{LDM}=\mathbb{E}_{z,\varepsilon\sim\mathcal{N}(0,1),Z_{\mathfrak{D }},t}\left[||\varepsilon-\varepsilon_{\psi}(z_{t},z_{\mathfrak{D}},t)||^{2}_{2 }\right],$ (5) $$

where $\varepsilon_{\psi}(z_{t},z_{\mathfrak{D}},t)$ is implemented as a UNet.

Sampling: New weights are sampled conditionally through the reverse diffusion process as follows:

$$ $z_{t}=\frac{1}{\sqrt{a_{t}}}(z_{t}-\frac{\beta_{t}}{\sqrt{1-\tilde{a}_{t}}} \varepsilon_{\psi}(z_{t},z_{\mathfrak{D}},t,))+\sigma\xi,$ (6) $$

where $\xi\sim\mathcal{N}(0,I)$ and, $\sigma_{t}$ a chosen value. After sampling a latent representation $\bar{(}z$ for a given dataset $\mathfrak{D}_{i})$. The pretrained VAE decoder is used to transform these latents into a weight vector $\bar{w}=\mathcal{D}(\bar{z})$, which is then used to initialize the target network as shown in Figure 1.

### 3.5 Exploring the Optimal Parameters Space of LLMs

Our goal is to enhance the performance of pre-trained LLMs on a target dataset without direct fine-tuning. To accomplish this, we introduce layer-conditioned D2NWG , which explores optimal model parameters in latent space. The idea is to sample multiple sets of weights and identify those that improve performance on the target dataset, starting from a single pre-trained model, with no additional training. One of the key challenges in generating LLM parameters is learning the distribution of their vast number of parameters, which is computationally expensive and impractical. Therefore, it is crucial to identify a subset of parameters that, when optimized, leads to significant task performance improvements. To tackle this, we adopt the spectrum technique (Hartford et al., ), which ranks LLM layers based on their importance, offering an efficient approach to achieving substantial performance gains without updating all parameters. We provide more detailed in the appendix in A.3 and A

**Table 1: Few-Shot Learning. ALL implies generation of the entire parameters and CH denotes generation of classification head only.**
| Method | Adaptation | Backbone | mini-ImageNet | tiered-ImageNet |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| 5-way 1-shot | 5-way 5-shot | 5-way 1-shot | 5-way 5-shot |  |  |  |
| iMAML (Rajeswaran et al., ) | ALL | Conv4 | $49.30\pm 1.88\%$ | $59.77\pm 0.73\%$ | $38.54\pm 1.37\%$ | $60.24\pm 0.76\%$ |
| ALFA  (Baik et al., ) | ALL | Conv4 | $50.58\pm 0.51\%$ | $69.12\pm 0.47\%$ | $53.16\pm 0.49\%$ | $70.54\pm 0.46\%$ |
| COMLN  (Deleu et al., ) | CH | Conv4 | $53.01\pm 0.62\%$ | $70.54\pm 0.54\%$ | $54.30\pm 0.69\%$ | $71.35\pm 0.57\%$ |
| MetaQDA  (Zhang et al., ) | CH | Conv4 | $56.41\pm 0.80\%$ | $72.64\pm 0.62\%$ | $58.11\pm 0.48\%$ | $74.28\pm 0.73\%$ |
| MetaDiff  (Zhang et al., ) | CH | Conv4 | 55.06 $\pm$ 0.81$\%$ | 73.18 $\pm$ 0.64$\%$ | 57.77 $\pm$ 0.90$\%$ | 75.46 $\pm$ 0.69$\%$ |
| D2NWG(Ours) | CH | Conv4 | 61.13 $\pm$ 8.50$\%$ | 76.94 $\pm$ 6.04$\%$ | 65.33 $\pm$ 6.50$\%$ | 80.05 $\pm$ 8.25$\%$ |
| ALFA  (Baik et al., ) | ALL | ResNet12 | $59.74\pm 0.49\%$ | $77.96\pm 0.41\%$ | $64.62\pm 0.49\%$ | $82.48\pm 0.38\%$ |
| MetaOptNet (Lee et al., ) | CH | ResNet12 | $62.64\pm 0.61\%$ | $78.63\pm 0.46\%$ | $65.99\pm 0.72\%$ | $81.56\pm 0.53\%$ |
| LEO  (Rusu et al., ) | CH | WRN-28-10 | $61.76\pm 0.08\%$ | $77.59\pm 0.12\%$ | $66.33\pm 0.05\%$ | $81.44\pm 0.09\%$ |
| Classifier  (Chen et al., ) | CH | ResNet12 | 61.22 $\pm$ 0.84$\%$ | 78.72 $\pm$ 0.60$\%$ | 69.71 $\pm$ 0.88$\%$ | 83.87 $\pm$ 0.64$\%$ |
| MetaQDA  (Zhang et al., ) | CH | ResNet18 | 65.12 $\pm$ 0.66$\%$ | 80.98 $\pm$ 0.75$\%$ | 69.97 $\pm$ 0.52$\%$ | 85.51 $\pm$ 0.58$\%$ |
| MetaDiff  (Zhang et al., ) | CH | ResNet12 | 64.99 $\pm$ 0.77$\%$ | 81.21 $\pm$ 0.56$\%$ | 72.33 $\pm$ 0.92$\%$ | 86.31 $\pm$ 0.62$\%$ |
| D2NWG(Ours) | CH | ResNet12 | 69.55$\pm$ 3.77$\%$ | 83.51 $\pm$ 6.21$\%$ | 81.15 $\pm$ 9.70$\%$ | 90.04 $\pm$ 6.10$\%$ |

**Table 2: Zero-Shot Transfer Learning. We evalutate on two backbones: Tiny Swin Transformer and ResNet18.**
| Model | CIFAR-10 | STL-10 | Aircraft | Pets | CIFAR-100 |
| --- | --- | --- | --- | --- | --- |
| Swin | 7.38 | 8.43 | 5.01 | 2.63 | 1.35 |
| GHN2 (Knyazev et al., ) | 48.20 | – | – | – | 12.7 |
| GHN3 (Knyazev et al., ) | 51.8 | – | – | – | 11.9 |
| D2NWG(Ours) | 53.12 $\pm$ 0.25 | 60.42 $\pm$ 0.14 | 24.57 $\pm$ 3.16 | 26.47 $\pm$ 1.90 | 30.44 $\pm$ 0.15 |
| ResNet18 | 10.88 | 6.78 | 3.75 | 2.39 | 1.38 |
| GHN2 (Knyazev et al., ) | 19.52 | 13.04 | – | – | – |
| D2NWG | 33.03 $\pm$ 0.04 | 50.42 $\pm$ 0.13 | 17.60 $\pm$ 2.13 | 17.29$\pm$ 0.13 | 13.71 $\pm$ 0.63 |
| D2NWG_CLIP(Ours) | 60.42$\pm$ 0.75 | 82.42 $\pm$ 0.04 | 27.70 $\pm$3.24 | 32.17 $\pm$6.30 | 51.50 $\pm$0.25 |

## 4 Experiments

We present two sets of results: weight generation with and without finetuning. For weight generation without finetuning, we present results on Few-Shot Learning, Zero-Shot Learning and Model Retrieval in Section 4.1. In Section 4.2, we present a set of results where the generated weights are further finetuned. For all experiments, we utilize a single Titan RTX GPU with 24GB of memory. An extensive set of ablation studies on the proposed method is provided in Appendix C and E

### 4.1 Weight Generation without Finetuning

We present a set of results where the generated weights are evaluated directly without finetuning for few-shot learning, zero-shot learning and model retrieval.

#### 4.1.1 Weights Generation for Few-Shot Learning

Task: We aim to show that learning the distributions of model pretrained independently on a large set of dataset can enable sampling weights that compete with meta-learning techniques in multi-task few-shot learning, without requiring fine-tuning.

Dataset: We utilize the mini-ImageNet and tiered-ImageNet datasets for this task. For the architectures, we use a four-layer ConvNet and a ResNet12 backbone provided by Chen et al. (). We generate the pretrained weights by linear probing a classifier head on each of the 50,000 subsets for 10 epochs and evaluate the performance on 600 subsets from the unseen test split for 1-shot and 5-shot. Analogously to few shot learning, we choose the number of images per class for conditioning to be the same as the support set, while the number of images per class in the query set is fixed to 15 for all methods and 600 tasks are used for testing.

Baselines: We benchmark against iMAML (Rajeswaran et al., ), ALFA (Baik et al., ), COMNL (Deleu et al., ), MetaQDA (Zhang et al., ), MetaDiff (Zhang et al., ), MetaOptNet (Lee et al., ) and a classifier baseline introduced in Chen et al. ().

Results: Table 1 shows that our approach consistently improves performance on all tasks while utilizing the same backbone as other methods. With the Conv4 backbone, we achieve approximately $6\%$ performance improvement in 1-shot learning and 3 to 4% on 5-shot learning on mini-ImageNet. On Tiered-ImageNet, we achieve more than 8% performance improvement on 1-shot and 5 to 6% average improvement on 5-shots. For the ResNet12 backbone we achieve 4 to 9% performance improvement. These results demonstrate the effectiveness of our method against the existing meta-learning methods.

We follow the standard image classification setup with and sample 50 weights for each subset, averaging the top 3 accuracies of all subsets as outlined in Table 1.
In 1-shot learning, conditioning is based on a single image per class per dataset, whereas in 5-shot learning, conditioning relies on five images per class per dataset. Dataset-conditioned weight generation aligns with meta-learning by enabling rapid model adaptation to new tasks. The generated weights are tailored to specific dataset features, allowing the model to leverage prior knowledge and achieve improved generalization surpassing meta-learning approaches.

#### 4.1.2 Zero-Shot Classifier Head Adaptation

Task: We evaluate the performance of the proposed method in adapting the classifier head to unseen datasets.
In this experiment, we assess whether our method can conditionally generate the classifier weights, potentially eliminating or significantly speeding up the finetuning process.

Dataset: We partitioned ImageNet-1k into 20k subsets
of 50 classes each with 50 images per class per subset and linear probe a classifier head for 10 epochs using Tiny Swin Transformer (denoted Swin in Table 1), and
ResNet18 all pretrained on ImageNet-1k. For dataset conditioning, we use 5 images per class per subset. The unseen target datasets are CIFAR-10, STL-10, Aircraft, Pets, and CIFAR-100 . The baseline methods in these experiments are ResNet18 and Tiny Swin Transformer pretrained on ImageNet-1k.

Baselines: We benchmark against the pretrained backbones, and two GHN models (Knyazev et al., ; ). Additonally, we provide a powerful variant of our model D2NWG_CLIP where the dataset encoder encodes the CLIP embedding for each sample in the datasets.

Results: Table 2 presents the performance of the sampled weights where it can be seen that the proposed method achieves better performance compared to the ImageNet pretrained weights and the GHN family of models. Additionally, the variant of our model that utilizes the CLIP embedding for dataset encoding significantly improves the performance suggesting that better dataset representation learning can boost the performance of the generated weights.

Figure: Table 3: Model Retrieval via Generative Augmented Weight Sampling

#### 4.1.3 Model Retrieval

Task: We assess the Generative Augmented Retrieval capability of D2NWG , aiming to show that it can learn the distribution of models pretrained on diverse real-world datasets. This task requires generation of dataset-conditioned weights that achieve performance comparable to the original pretrained models and hence provide access to a wide range of pretrained models through efficient sampling.

Dataset: We collected 30 real-world datasets(Ullah et al., ), spanning 19 to 706 classes and organised into 10 domains with 3 datasets per domain, and fine-tuned a MobileNetV3 subnet(^2^22[https://pytorch.org/hub/pytorch_vision_once_for_all/](https://pytorch.org/hub/pytorch_vision_once_for_all/))
sampled from OFA (Cai et al., ) for 100 epochs on each dataset. We then learned the distribution of the combined pretrained models from the last 20 epochs across all datasets.

Baselines: For this task, we compare with the original pretrained weights which are finetuned on each individual dataset. For each dataset, we sample and report the average accuracy of 5 set of weights sampled with D2NWG .

Results: From Table 4.1.2 we see that D2NWG conditionally generates high-performing parameters while enhancing the pretrained model, achieving the best average results across all datasets. This demonstrates the strong retrieval capability of our method, suggesting it can be used as a neural network weight retriever in approaches like (Zhao et al., ), eliminating the need for pretrained database. Detailed dataset information is provided in Table 11 and more experiments in the Appendix C.10. Additionally, it is much more efficient to generate weights with our model compared to pretraining as shown by the runtime in Table 4.1.2.

Figure: Table 4: Finetuning of Generated Weights using the Modelzoo of Schürholt et al. ().

### 4.2 Weight Generation with Fine-tuning

In this section, we evaluate the quality of the generated weights in fine-tuning scenarios to assess their suitability for transfer learning.

Task: The goal is to assess the behavior of the sampled weights when finetuned on the same dataset and compare convergence speed. This experiment focuses on evaluating whether the sampled weights can be effectively fine-tuned to achieved superior final performance, rather than simply aiming for weights producing high initial accuracy and may not lead to superior performance while fine-tuning.

Datasets: We used the modelzoo of Schürholt et al. () consisting of a ConvNet trained on MNIST, SVHN, CIFAR-10 and STL-10. Our model was trained using the combined pretrained weights from epochs 21 to 25 of all models, consistent with the baseline settings.

Baselines: We compare against the kernel density estimator approaches from Sch"urholt et al. (); Schürholt et al. (), evaluated on the same datasets. Unlike these unconditional methods, we build a model specifically for MNIST and SVHN, and another for CIFAR-10 and STL-10. For each dataset, five sets of weights were sampled to initialize the models, which were fine-tuned for a number of epochs from 0 to 25. We also add RandomInit model trained for 50 epochs and show that our sampled weight finetuned for 25 epochs outperforms this model.

Results: As shown in Table 4.1.3, D2NWG consistently accelerates convergence across related tasks, surpassing the pretrained model and outperforming both baselines  Schürholt et al. (); Sch"urholt et al. (). The finding implies that D2NWG accelerates convergence and improves performance compared to existing methods. This highlights its potential for faster and more efficient model initialization, making it valuable for transfer learning and real-world applications. Interestingly, on MNIST and SVHN, weights with higher initial performance tend to degrade during fine-tuning.

### 4.3 Performance Evaluation on Unseen Datasets

Task: The objective remains the same as in Section 4.2, but here we evaluate the proposed method solely on unseen datasets.

Datasets: We assess D2NWG on a real-world dataset of 140 subsets with class counts ranging from 2 to 20, and 10 test sets with up to 1,566 classes. We use a two-layer MLP on top of a CLIP image encoder and fine-tune it on training datasets to collect the pretrained zoo.(see appendix A.6). The image datasets contains food, dataset, drawing, x-ray and others.

Baselines: The baseline methods are random initialization and a pretrained MLP previously trained on ImageNet.

Results: Figure 2 shows performance on four unseen datasets, where D2NWG achieves 99.04% initial accuracy on the dessert dataset, outperforming the randomly initialized model even after 50 epochs. D2NWG consistently accelerates convergence across all tasks, surpassing both random and pretrained initialization. Despite no class overlap between training and test datasets, it demonstrates strong transferability. Detailed results are available in Table 21 of the Appendix.

Figure: (a) Dessert Food Dataset
Refer to caption: x2.png

### 4.4 Transfer Learning: MobileNet Full Weight Generation

Task: We evaluate each method’s generalization on CIFAR-10, STL-10, Pets and Aircrafts, focusing on performance gains in domain-specific tasks. The goal is to identify the best initialization strategy for improving model adaptability across diverse data distributions.

Figure: Figure 3: Comparison of accuracy for Pretrained, D2NWG , and RandInit methods across CIFAR-10, STL-10, Aircraft100, Aircraft30, and Pets after 1 epoch of fine-tuning.
Refer to caption: x6.png

Baseline: The baseline in this experiment are the Pretrained model, which uses weights from a model pretrained on ImageNet and RandomInit, a randomly initialized model.

Datasets: In this experiment we evaluate the transferability to unseen dataset of D2NWG trained in Section 4.1.3 on unseen datasets CIFAR-10, STL-10, Aircraft100, Aircraft30, and Pets.

Results: We initialized the target model with 5 sampled weights and fine-tuned it for 1 epoch on each dataset, along with 5 pretrained and randomly initialized models. Results are provided in Figure 3 where we transfer across CIFAR-10, STL-10, Aircraft100, Aircraft30, and Pets and show that D2NWG outperforming the baselines. On the AIRCRAFT-100 dataset, our method achieved an accuracy of $1.43\%$, outperforming both the randomly initialized model, which reached 1.0%, and the ImageNet-pretrained model, which achieved $1.24\%$. D2NWG demonstrates superior generalization and adaptability, making it a more effective initialization strategy even on specialized datasets such as Aircrafts and Pets.

**Table 5: Task Conditioned LoRA parameters Generation. Adaptations are performed on a Roberta-Base model denoted Rob-B.**
| Method | Parameters | SST-2 (Acc) | MRPC (Acc.) | CoLA MCC.) | QNLI (Acc.) | RTE (Acc.) | STS-B (PCC.) | Avg. |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Rob-B | 125M | 94.8 | 90.2 | 63.6 | 92.8 | 78.7 | 91.2 | 85.2 |  |
| LoRA | 0.9M | 95.1$\pm$0.2 | 89.7$\pm$0.7 | 63.4$\pm$1.2 | 93.3$\pm$0.3 | 78.4$\pm$0.8 | 91.5$\pm$0.2 | 85.2 |  |
| AdaLoRA | 0.9M | 94.5$\pm$0.2 | 88.7$\pm$0.5 | 62.0$\pm$0.6 | 93.1$\pm$0.2 | 81.7$\pm$0.6 | 90.5$\pm$0.2 | 85.0 |  |
| DyLoRA | 0.9M | 94.3$\pm$ 0.5 | 89.5$\pm$0.5 | 61.1$\pm$0.6 | 92.2$\pm$0.1 | 78.7$\pm$0.7 | 91.1$\pm$0.2 | 84.5 |  |
| FourierFT | 0.6M | 94.2$\pm$0.3 | 90.0 $\pm$ 0.8 | 63.8$\pm$1.6 | 92.2$\pm$0.1 | 79.1$\pm$0.5 | 90.80 $\pm$ 0.2 | 85.0 |  |
| D2NWG | 0.6M | 94.3$\pm$0.1 +0.2 | 90.3$\pm$0.5($\uparrow$0.3) | 64.3$\pm$1.2 ($\uparrow$0.5) | 92.6$\pm$0.2($\uparrow$0.5) | 79.6$\pm$0.4($\uparrow$0.5 ) | 91.0$\pm$0.3($\uparrow$0.0.2) | 85.3($\uparrow$0.3) |  |

### 4.5 Task Conditioned LoRA weights Generation

Task: In this section, we demonstrate that our method can be applied to LLMs by learning the distribution of LoRA matrices conditioned on task-specific textual descriptions.

Datasets: We use six tasks from the GLUE benchmark and generate task descriptions using GPT-4, as shown in Table 13. LoRA weights were generated following the fine-tuning process of Gao et al. (). We collected LoRA and classifier head checkpoints from the last 5 epochs, combined the pretrained vectors, and conditionally learned their distribution.

Baselines: We compare with base Roberta-base, LoRA (Hu et al., ), AdaLoRA (Zhang et al., ), DyLoRA (Valipour et al., ) and FourierFT (Gao et al., ) which are all LoRA-based RoBERTa-base models. We sampled and compared the average accuracy of the top 5 performing sets of weights per dataset.

Results: As shown in Table 5, D2NWG effectively generates weights that match or surpass the performance of pretrained models. These results align with our findings from the augmented weight retrieval experiments. Additional details regarding the task descriptor are provided in Table 13.

**Table 6: Exploration of optimal weight space of some instruct LLMs using a diffusion model. These are the average raw obtained with the lm-harness tool without the flash-attention. $\uparrow$ indicates the performance gain**
| Methods | Winogrande (5 shot) | Arc-Challenge (25 shot) | Hellaswag (25 shot) |
| --- | --- | --- | --- |
| LLAMA-3.1-8B-Instruct | 67.17 $\pm$ 0.01 | 64.93 $\pm$0.01 | 78.58 $\pm$ 0.00 |
| D2NWG | 67.61$\pm$ 0.02($\uparrow$0.44) | 65.74$\pm$0.01($\uparrow$0.81) | 78.86$\pm$ 0.02($\uparrow$0.28) |
| Mistral-7b-Instruct | 69.93 $\pm$ 0.01 | 59.22 $\pm$0.01 | 81.97 $\pm$ 0.00 |
| D2NWG | 70.80$\pm$ 0.02($\uparrow$0.80) | 59.80$\pm$0.01($\uparrow$0.58) | 82.04$\pm$ 0.00($\uparrow$0.07) |
| LLAMA-3.2-1B-Instruct | 56.75$\pm$ 0.01 | 40.96 $\pm$0.01 | 61.67 $\pm$ 0.00 |
| D2NWG | 57.17 $\pm$ 0.01($\uparrow$0.42) | 41.55 $\pm$ 0.01($\uparrow$0.59) | 61.70$\pm$ 0.01($\uparrow$0.03) |

### 4.6 Enhancing LLM Performance with Weight Sampling

Task: We aim to demonstrate that D2NWG can enhance existing LLMs by learning the distribution of their pretrained weights, enabling the generation of parameters that improve performance on specific tasks while generalizing to unseen tasks.

Datasets: We evaluate on several benchmarks(Beeching et al., ): AI2 Reasoning Challenge (25-shot) for grade-school science questions, HellaSwag (25-shot) for commonsense inference, Winogrande (5-shot) for commonsense reasoning.

Baseline: We evaluate our method against various version of LLAMA3 and Mistral-7B.

For each model, We extract the weights of the top 25% of layer excluding embedding and output layer, learn their distribution using chunk based encoding, We then steer through the optimal space to generate task-specific parameters as shown in Table 6.

Results: The results in Table 6 demonstrates that our approach consistently improve the performance of each models demonstrating new application avenues of our proposed method.

### 4.7 Evaluation on Open LM Benchmark

We merge these models following Wortsman et al. () and evaluate them on the OpenLM leaderboard (Fourrier et al., ) as shown in Table 12

Task: We evaluate the robustnets of ours best models on the open-lm leaderboard.

Datasets: We evaluate models on 6 key benchmarks using the Eleuther AI Language Model Evaluation Harness: IFEval (0-shot) for instruction adherence, BBH (Big Bench Hard, 0-shot) with 23 challenging tasks (arithmetic, reasoning, language understanding), MATH (0-shot) focusing on Level 5 high-school math problems, GPQA (0-shot) with graduate-level Q&A across various fields, MuSR (0-shot) testing complex reasoning with long-range context, and MMLU-Pro (0-shot) for advanced multitask knowledge assessment. These benchmarks assess diverse reasoning and knowledge capabilities in 0-shot and few-shot settings.

Baselines: We compare our method against LLMA3.1-8B-Instruct and its fine-tuned variant, with evaluations conducted on the leaderboard server.

Results: As shown in Table7 , our method outperforms the base models on the leaderboard and is on par with models pretrained on task-specific datasets. Although the weights were not directly calibrated using the tasks from the leaderboard, D2NWG achieves up to a 3% improvement on some tasks. This confirms that it is possible to guide a model towards specialization in certain tasks by effectively exploring the optimal parameter space through sampling. The consistent performance gains across diverse benchmarks underscore D2NWG’s effectiveness in improving model robustness and transferability. Ours llama-3.2-1B based model(^3^33[https://huggingface.co/DeepAutoAI/Explore_Llama-3.2-1B-Inst_v1.1](https://huggingface.co/DeepAutoAI/Explore_Llama-3.2-1B-Inst_v1.1)) ranked among the top llama-3.2-1B models on the public leaderboard.

Quality Check: Additionally, our method improves text generation quality, as demonstrated by the experimental results in Table 12. We demonstrate that the generated weights are capable of generating text on par with the base pretrained model. The model used for quality check ranked 4^th on open lm-leaderboard(^4^44 [https://huggingface.co/DeepAutoAI/Explore_Llama-3.1-8B-Inst](https://huggingface.co/DeepAutoAI/Explore_Llama-3.1-8B-Inst)). Further results on LLMs are provided in Appendix D, where we also demonstrate that the proposed method successfully learns the full parameters of GPT-2 small while maintaining performance comparable to the pretrained model.

**Table 7: Performance evaluation on unseen open llms leaderboard v2 benchmark. These results are produced by Huggingface after submission to open LLM leaderdoards. $\uparrow$ indicate performance improvement while $\downarrow$ indicate a performance decrease**
| Method | ifeval (0) | Bbh (3) | Gpqa (0) | MATH-hard (4) | Musr (0) | MMLU-Pro (5) | Avg | Base Model | Fine-tuned |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Meta-Llama-3.2-1B-Instruct | 56.78 | 8.74 | 3.36 | 2.96 | 2.97 | 7.58 | 13.76 | Meta-Llama-3.2-1B | Yes |
| D2NWG | 58.44($\uparrow$1.66) | 8.82($\uparrow$0.08) | 1.68($\downarrow$1.68) | 6.04($\uparrow$3.08) | 0.66($\downarrow$2.31) | 9.09($\uparrow$1.51) | 14.12($\uparrow$0.36) | Meta-Llama-3.2-8B-Instruct | No |
| SauerkrautLM-8B-Instruct | 80.17 | 31.00 | 5.37 | 11.18 | 11.52 | 32.12 | 28.56 | Meta-Llama-3.1-8B-Instruct | Yes |
| D2NWG | 80.33 +0.16 | 31.10($\uparrow$0.10 ) | 5.26($\downarrow$0.11) | 11.56($\uparrow$0.38) | 11.52 | 32.07($\downarrow$0.05) | 28.64 ($\uparrow$0.08) | SauerkrautLM-8B-Instruct | No |
| Lexi-Uncensored-V2 | 77.92 | 29.69 | 4.36 | 16.92 | 7.77 | 30.90 | 27.93 | Meta-Llama-3.1-8B-Instruct | Yes |
| Meta-Llama-3.1-8B-Instruct | 78.56 | 29.89 | 2.35 | 17.60 | 8.41 | 30.68 | 27.91 | Meta-Llama-3.1-8B | Yes |
| D2NWG | 77.85($\downarrow$0.71) | 30.39($\uparrow$0.5) | 4.47($\uparrow$2.12) | 17.52($\downarrow$0.08) | 9.64($\uparrow$1.23) | 31.02($\uparrow$0.34) | 28.50($\uparrow$0.59) | Meta-Llama-3.1-8B-Instruct | No |

Limitations: Our method relies on large collections of pretrained weight tensors and datasets, which require substantial storage and computational resources. However, such pretrained models are becoming more readily available due to the efforts made by open-source communities.

## 5 Conclusion

In this work, we recast latent diffusion for dataset-conditioned neural network weight generation, enabling quick adaptation to novel datasets and efficient fine-tuning and transfer learning without training. Through extensive experiments on diverse datasets, our method generates high-quality weights for novel tasks and improves generalization. We extend parameter generation to large language models, demonstrating the scalability and versatility of our approach. Our method effectively encodes architectures with up to 1 billion parameters using a single GPU with less than 80GB, including task- or dataset-conditioned generation.

## Appendix A Approach

Broader Impact D2NWG addresses the resource-intensive nature of deep learning by proposing a method for efficient transfer learning. This has the potential to reduce the computational resources required for training neural networks, making it more accessible to a wider range of researchers and organizations.

Limitation
In this work, we focus mainly on generalization across datasets. Additionally, while the diffusion model achieves impressive performance on image generation, there are still some challenges to efficiently recast it for weights generation including memory constraint, convergence challenges and considerations of symmetries in the weight spaces of different neural network architectures.

### A.1 Relationship Between Datasets and Trained Weights

Gradient descent based optimization is the commonly used technique to generate optimal neural network weights through training by minimizing a loss function, ie. cross-entropy for classifiation tasks. The weights optimized with gradient descent thus contains some information about the training data. Therefore, understanding the correlation between the training dataset and the optimal weights is important for the generation of weights.
During the optimization process with gradient descent the weights of each layer $i$ are updated as $w_{i}=w_{i-1}-\eta\nabla_{w_{i}}\mathcal{L}(w_{1},w_{2},\ldots,w_{n})$, where $\nabla_{w_{i}}\mathcal{L}(w_{1},w_{2},\ldots,w_{n})$ is input dependent. As an example, let’s consider a two-layer feedforward neural network:

$$ $\displaystyle x$ $\displaystyle:inputs$ $\displaystyle l_{1}$ $\displaystyle=W_{1}x+b_{1}$ $\displaystyle h$ $\displaystyle=ReLU(l_{1})$ $\displaystyle h$ $\displaystyle=ReLU(l_{1})$ $\displaystyle l_{2}$ $\displaystyle=W_{2}h+b_{2}$ $\displaystyle\hat{y}$ $\displaystyle=softmax(l_{2})$ $\displaystyle J$ $\displaystyle=CE(y,\hat{y})$ $$

Analyzing the weights’ update below, we can observe that the optimal weights are noisy perturbation of the inputs feature maps and all together they contain information about the training either related to the raw input or the feature map at a given stage.

$$ $\displaystyle\delta_{1}$ $\displaystyle=\frac{\partial J}{\partial l_{2}}=(y-\hat{y})^{T}$ $\displaystyle\delta_{2}$ $\displaystyle=\frac{\partial J}{\partial l_{1}}=\delta_{1}W_{2}osgn(h)$ $\displaystyle W^{(i+1)}_{1}$ $\displaystyle=W^{(i)}_{1}-\eta\nabla_{w_{1}}\mathcal{L}(w_{1},w_{2},b_{1},b_{2})$ $\displaystyle=W^{(i)}_{1}-\eta\delta_{2}^{T}x$ $\displaystyle W^{(i+1)}_{2}$ $\displaystyle=W^{(i)}_{2}-\eta\nabla_{w_{2}}\mathcal{L}(w_{1},w_{2},b_{1},b_{2})$ $\displaystyle=W^{(i)}_{2}-\eta\delta_{1}^{T}h^{T}$ $$

### A.2 Weights Vectorization

For a neural network with $L$ layers, the process of vectorizing the weights and biases for both fully connected and convolutional layers is as follows:

- •
For the $\ell$’th fully connected layer:
$W^{(l)}\in\mathbb{R}^{d_{l-1}\times d_{l}}\rightarrow\text{vec}(W^{(l)})\in
\mathbb{R}^{d_{l-1}.d_{l}}$ and $b^{(l)}\in\mathbb{R}^{d_{l}}$, the length of the vectorized weights for this layer, including the bias if it is not null, is given by $d_{l-1}d_{l}+d_{l}$.
- •
For the $\ell$’th convolutional layer:
$W^{(l)}\in\mathbb{R}^{k_{h}.k_{w}.c_{in}.c_{out}}$ and $b^{(l)}\in\mathbb{R}^{c_{out}}$, the length of the vectorized weights for this layer, including the bias if it is not null, is $k_{h}\cdot k_{w}\cdot c_{in}\cdot c_{out}+c_{out}$.

We then concatenate all the flattened weight and bias vectors resulting in a vector $\theta$: $\theta=\bigoplus_{l=1}^{L}\left(\text{vec}(W^{(l)})\oplus b^{(l)}\right)$ where vec denotes the vectorization operation and $\oplus$ denotes concatenation.The concatenation operation keeps the ordering of weights in the network.

### A.3 Layer Selection Strategy

To manage the large number of parameters in LLM architectures, where not all layers are required to be tuned to improve the performance, we propose focusing on the most important layers. These layers are identified using the Marchenko-Pastur (MP) distribution, which serves as a filter to highlight relevant weights while discarding those resembling random noise. The MP law provides a benchmark for distinguishing structured weights from noise by comparing the empirical eigenvalue spectrum of weight matrices to the MP distribution.
D2NWG uses this spectrum method (Hartford et al., ) to learn the distribution of the most informative weights—those corresponding to eigenvalues that significantly exceed the MP upper bound. By focusing on these critical weights, D2NWG captures meaningful patterns in LLMs, leading to enhanced performance in transfer learning.

The spectrum method, grounded in random matrix theory, applies the Marchenko-Pastur (MP) distribution to different types of layers, treating them as rectangular random matrices. In transformer networks, functionally similar layers are grouped, such as a set for all query layers in multi-head attention.
The method begins by computing the covariance matrix of each layer’s weight matrix, $W\in\mathbb{R}^{m\times n}$, as $\Sigma=\frac{W^{T}W}{n}$, followed by eigenvalue extraction. Singular value decomposition (SVD), $W=USV^{T}$, is used to efficiently compute these eigenvalues from the diagonal matrix $S$, which contains the singular values. The resulting eigenvalues describe the variance captured by each principal component of the squared weight matrix and form what is known as the empirical spectrum.
To analyze this spectrum, we compare it to the theoretical distribution of eigenvalues predicted by the Marchenko-Pastur (MP) distribution. This distribution $p(\lambda)$, in equation 7, characterizes the eigenvalue behavior of random covariance matrices as $m,n\to\infty$, with a fixed aspect ratio $q=\frac{m}{n}$ and variance $\sigma^{2}$.

$$ $p(\lambda)=\frac{1}{2\pi\sigma^{2}q\lambda}\sqrt{(\lambda_{+}-\lambda)(\lambda -\lambda_{-})},$ (7) $$

where $\lambda\in\left[\lambda_{+},\lambda_{-}\right]$, $\lambda_{+}=\sigma^{2}(1+\sqrt{q})^{2}$, and $\lambda_{-}=\sigma^{2}(1-\sqrt{q})^{2}$. From 7, the correspoding bounds for eigen values of $W$ are $\sqrt{\lambda}/\sqrt{n}\in\left[\varepsilon_{+},\varepsilon_{-}\right]$, $\varepsilon_{+}=\frac{1}{\sqrt{n}}\sigma(1+\sqrt{q})$, and $\varepsilon_{-}=\frac{1}{\sqrt{n}}\sigma(1-\sqrt{q})$.

Interpretation: The Marchenko-Pastur (MP) distribution provides insight into the underlying structure of data or layer in our case:

- •
Eigenvalues within MP bounds: Likely represent noise, with their corresponding principal components carrying little meaningful information, indicating the layer’s lower importance.
- •
Eigenvalues larger than the upper MP bound $\lambda_{+}$: Capture more variance than noise, suggesting the presence of true signals or patterns in the data.
- •
Eigenvalues smaller than the lower MP bound $\lambda_{-}$: May indicate compression or degeneration in the data structure.

Significant deviations, particularly large eigenvalues, indicate meaningful components that capture more variance than random noise, aiding in the identification of important features or signals. This insight is used to compute the signal-to-noise ratio (SNR), where eigenvalues below the upper bound are considered noise. The SNR is calculated as follows:

$$ $SNR=\frac{\sum_{k\,|\,|\sigma_{k}|\geq\varepsilon}\sigma_{k}}{\sum_{n\,|\,| \sigma_{n}|<\varepsilon}\sigma_{n}}.$ (8) $$

### A.4 Learning the Distribution of LLM Weights

Our method for LLM weight generation employs a layer-wise chunking mechanism that facilitates both layer-wise and chunk-wise sampling. Each layer is divided into independent chunks to form the training data, and are then encoded with the VAE. During the diffusion process, an index is assigned to each chunk, and the model is trained using class-conditioned diffusion, where chunk indices serve as class labels. At sampling time, the chunk indices corresponding to each layer are grouped into clusters associated with that layer. These clusters are then used to sample new sets of chunks, which are concatenated to reconstruct the sampled weights for each layer.

After selecting the top 25% of the layers, we applied chunking with a size of 2,097,152 for LLaMA 3.2-1B and 4,194,304 for other models. We then performed sequential refinement using Algorithm 1. Unlike in vision tasks, LLM models are conditioned on chunk indices. Here, we refer to neural network operations such as dense layers and layer normalization as layers. The spectrum method provides an ordered set of these layers (q, k, v, o, mlp_up, mlp_down, mlp_gate). For architectures like Llama 3.1-8B and Mistral, we only learn the distribution of the top 8 each of these layers, excluding layer normalization. These layers are further divided into two groups: the top 4 and the second top 4, for which we build separate models to learn their distributions. As for the normalization layers, we learn the distribution across all of them. The maximum generated parameters is $\approx$ 872M.

Figure: Algorithm 1 Sequential Weight Model Improvement

### A.5 Details of Modelzoo Generation

### A.6 Modelzoo and Pretrained Datasets

Model zoo We use the pretrained datasets from Schürholt et al. () as structured in Schürholt et al. (). This dataset consists of 4 different datasets with 5000 pretrained weights per architectures and datasets. The details of the architecture used to generate the pretrained weights are available in Schürholt et al. ().

KaggleZoo
This modelzoo is generated using the dataset provided by Jeong et al. ().
To efficiently generate the pretrained weights, we first compute the features of each image then use a MLP with two layers with input size 512, hidden size 256 and leaky ReLU activation functions. We train the MLP on clip features as it allows us to quickly generate high performing weights. For each datasets we used the last 10 checkpoints which results in 1400 pretrained weights for training.

ImageNet zoo
To generate the pretrained modelzoo on ImageNet, we sample 1000, 5000, 10000 and 20000 subsets with 10 classes each with 100 images per class in the training set and 50 per class in the test set. For the 1000 and 5000 subsets we used the same MLP architecture as the KaggleZoo. For the 10000 subset, we reduce the hidden dimension to 128 and, for the 20000 subset we use a single linear probing layer. On the other datasets linear probing shows similar generalization performance as the two-layer MLP. We use Adam optimizer with a learning rate of $1e-3$ and all models are trained for 30 epochs.

Zoo for Few-shot learning: The few-shot learning pretrained zoo is generated by fine-tuning the classifier head for 10 epochs on each of the 50,000 subsets.

LLMs zoo: We collected the pretrained LLM model from their original HugginFace repositories with no further pertaining on specific tasks or datasets.

Meta-album datasets: We split the meta-album dataset into a training set (70%) and a test set (30%). Next, we trained the MobileNetV3 OFA subnet with parameters $d=2$, $k=3$, and $e=3$ for 100 epochs. Checkpoints from the last 20 epochs were collected as training data. A detailed breakdown of the dataset can be found in Table 11.

### A.7 Details of the Proposed Model

We build our dataset conditioned weight generation model using latent diffusion (Rombach et al., ).

AutoEncoder: We use the same VAE modules of latent diffusion and use the same architecture for all experiments except adaptation of the inputs and output dimensions. We insert a linear layer before the first layer of the encoder such that we can reshape its output to a representation for the convolution layers. Similarly, a linear layer is placed at the last layer of the decoder adapting the output to the vectorized weights representations. For the VAE loss function we removed the discriminator in the original latent diffusion VAE loss function.

**Table 8: Models seting, $n$ and $c$ in the dataset configuration represent respectively the number of samples per class n=5 for training and c the total number of classes per dataset. The VAE and the diffusion models share similar configuration and architectures as (Rombach et al., )**
| Parameters | Values |
| --- | --- |
| Epochs | [50, 2000] |
| VAE |  |
| Optimizer | Adam |
| Learning Rate | 1e-3 |
| Latent Dimiension | 1024 |
| KL-Divergence Weight | 1e-6 |
| Dataset Encoder |  |
| Architecture | Set Transformer |
| Input Dimension | $c\times n\times 512$(min) |
| Output Dimension | 1024 (min) |
| Depth of Set Transformer | 2 |
| Diffusion |  |
| Optimizer | AdamW |
| Learning Rate | 1e-4 |
| Scheduler | Linear |
| Time step | 1024 |
| Network | Unet |
| UNet Input Size | $(c\times 32\times 32)$ |

Diffusion Model: We utilize same UNet architecture as in latent diffusion with the same training procedure.

Dataset Encoding Mechanisms
We investigated three different mechanisms of dataset encoding. Firstly, we use Set Transformer (Lee et al., ) which can be difficult to train when optimized together with the diffusion using the weights encoder from the VAE and the Set Transformer.

In addition to the Set Transformer, we explored a two-layer MLP model as the dataset encoder. The first layer is a dynamic linear layer with a maximum input feature size set to $n_{\text{max}}\cdot c_{\text{max}}$, where $n_{\text{max}}$ is the maximum number of images per class and $c_{\text{max}}$ is the maximum number of classes among all subsets of the pretrained datasets. The shape of the image features in each dataset obtained with the CLIP image encoder is $x\in\mathbf{R}^{c\times n\times d}$, where $d$ is the feature dimension for each corresponding pretrained weight vector. While the Set Transformer-based encoder uses these inputs directly, the MLP encoder reshapes each input from $x\in\mathbf{R}^{c\times n\times d}$ to $x\in\mathbf{R}^{d\times(n\cdot d)}$ and then applies the dynamic linear layer. If a dataset has more classes or samples than $c_{\text{max}}$ and $n_{\text{max}}$ respectively, we only consider the first $c_{\text{max}}$ classes and $n_{\text{max}}$ samples per class. If the dataset has fewer classes or samples, we adjust the dynamic linear layer dimensions accordingly. The output of the dynamic linear layer is $z\in\mathbf{R}^{d\times h}$, where $h$ is an arbitrarily chosen number greater than zero. We then reshape $z$ from $\mathbf{R}^{d\times h}$ to $\mathbf{R}^{1\times(h\cdot d)}$ (with $h\cdot d$ fixed) and apply the final linear layer to obtain the desired output. This model can be jointly optimized with the diffusion model while achieving good performance.

Dataset Encoding with Set Transformer We use the Set Transformer for dataset encoding, pretrained as described in Lee et al. (). The approach involves using the frozen Set Transformer and adding a single linear layer to adapt its output to our specific problem, utilizing it as the dataset encoder. This method reduces the computational cost of training the Set Transformer and enables joint optimization of the dataset encoder and the diffusion model. The results of these data set encoding schemes are presented in Table 20 for the Hyperzoo dataset.

Figure: Algorithm 2 Datasets Encoder Training

## Appendix B Training Details

In this section, we describe the training steps used to train our method.

- •
Pretrained Zoo Generation: For classifier head adaptation, we first compute the features for all datasets. Then, we train the classifier head to generate the pretrained zoo.
- •
VAE Training: We train the VAE to encode the pretrained weights following Equation 1. Additionally, a pretrained performance predictor can be used to predict the performance of the reconstructed weights and guide the VAE training as described in Equation 9.
- •
Dataset Alignment: If using dataset alignment, we pretrain the Set Transformer to align the pretrained weights’ latent representations. This is done using the frozen encoder of the VAE and the dataset embeddings. The inputs to the Set Transformer are image features, with five image features per class.
- •
Diffusion Process Training: We train the diffusion model while keeping the Set Transformer and the VAE models frozen. If an MLP is used for dataset encoding, we jointly optimize the diffusion process with the MLP dataset encoder.

Although the dataset encoder can be optimized together with diffusion model, we train them separately to speed up the training process and reduce memory requirements. The VAE and the dataset encoder are trained using the Adam optimizer with a learning rate of $1e-4$. The diffusion model in each experiment is trained with a linear scheduler, a base learning rate of 1e-4, and the AdamW optimizer (Rombach et al., ). During the training process of the diffusion model, the output of the dataset encoder is concatenated with the latent representation of the input weights, forming the input to the UNet model. Additionally, we investigate joint training of the diffusion process in the ablation study and Appendix C.5 and  A.7. Further details can be found in Table 8.

Figure: Figure 4: Overview structure of the set-transformer-based dataset encoder. For each pretrained dataset we use $n=5$ images per class and the embedding dimension $d_{0}=1024$.
Refer to caption: extracted/5953445/images/datasetencoderlast.png

### B.1 Predictor Training

To improve the reconstruction and sampling efficiency, we trained an accuracy predictor $g$ from pretrained weights $w$ then use the frozen predictor during the training of the VAE as a regularizer as shown below:

$$ $\underset{\theta,\sigma}{\min}\frac{w-f_{\theta}(w)}{\sigma^{2}}+\log\sigma^{2 }+||g(w)-g(f_{\theta}(w))||^{2},$ (9) $$

where $g(w)$ is the embedding of the original input and $g(f_{\theta}(w))$ is the predictor embedding of the reconstructed weights. The predictor can be either dataset-conditioned or unconditioned. In general we found that dataset-conditioned predictor works only well for large number of samples per dataset. After the AutoEncoder is trained, we train the dataset-conditioned module which requires a dataset encoder.

Figure: Algorithm 3 Predictor-Guided VAE

## Appendix C Ablation Study

### C.1 Can the proposed method handle multiple architectures?

This section provides a simple way to handle the case where the pretrained zoo contains multiple architectures per task or dataset.
Since the number of architecture and dataset are predefined, it is possible to build a set of unique index for each combination of dataset-architecture pairs. An alternative will be to encode the graph representation of the architectures then used that as conditioning. In this ablation study we use the simple class indexing approach to demonstrate the versatility of our method.
We use CIFAR10 and CIFAR100 as the dataset and as target architectures we utilze a ResNet44 trained on CIFAR-100 with 667,188 parameters and a
ResNet44 trained on CIFAR-10 with 661,338 parameters and finally, a MobileNetV2 trained on CIFAR-10 with 700,490 parameters. All models were zero-padded to 700,490 parameters, combined into a unified dataset, and trained without chunking. The results in Table 9 demonstrate that the proposed method is capable of simultaneously learning the distributions of diverse architectures trained on diverse datasets.

**Table 9: Performance evaluation on mixed architectures.**
| Model | ResNet44 (CIFAR-10) | ResNet44 (CIFAR-100) | MobileNetV2 (CIFAR-10) |
| --- | --- | --- | --- |
| Pretrained | 94.01 | 71.63 | 92.88 |
| D2NWG | 94.10 $\pm$0.09 | 71.64$\pm$0.02 | 93.11$\pm$0.20 |

### C.2 Transferability

We conducted a set of experiments using ResNet32 pretrained on CIFAR-10 and CIFAR-100, as the base model to explore the transferability of learned weights. Our objective is to model the distribution of the combined pretrained weights conditioned on their respective datasets.

We then evaluated these conditionally sampled weights on CIFAR-10 across different architectures, including ResNet20, ResNet44, ResNet56, and ResNet32. Inspired by GHN3’s weight tiling approach for larger networks, we adopted the following strategy: instead of tiling, we sampled multiple weights from the base model and concatenated them to match the dimensionality of the target architectures.

This concatenation was carefully structured to align each layer type between the base and target networks. The resulting vector is then used as initialization for the unseen architectures. Figure 5 summarizes our results where "RandInit" refers to random initialization, and "Unseen" refers to architectures where our method was applied to pretrained weights without fine-tuning. Pre-trained models are referenced for comparison, and all models were sourced from publicly available models. From Figure 5, it can be seen that our method achieved higher performance improvement at initialization on the unseen ResNet architectures.

Figure: Figure 5: Performance evaluation with unseen architectures on CIFAR-10.
Refer to caption: x7.png

### C.3 Effect of Modelzoo Size Generalization

Here we investigates the impact of increasing the number of pretrained datasets on performance with experiments that use model zoos of sizes 5000, 10,000, and 20,000, derived from ImageNet subsets. Unseen target datasets CIFAR-10 and STL-10 are used. Sampling 50 weights, the average performance of the top 5 performing weights is shown in Figure 6(a).

Results: On CIFAR-10 and STL-10, we obtain accuracies of $39.60\pm 1.31\%$ and $44.66\pm 0.55\%$ for 5000 subsets, $42.15\pm 2.12$ and $64.83\pm 2.83\%$ for 10000 subsets, and $52.64\pm 3.12\%$ and, $80.49\pm 1.77\%$ for 20000 subsets.
The maximum accuracies with random initialization are $12.11\%$ and $17.12\%$ on CIFAR-10 and STL-10 without fine-tuning. This experiment demonstrated that increasing the number of datasets enhances the generalizability of the proposed method.

Figure: (a)
Refer to caption: x8.png

### C.4 Sampling without Latent Representation

Figure: Figure 7: Performance comparison at initialization of method with jointly trained set-transformer (Without CLIP) and method clip-based dataset encoder.
Refer to caption: x10.png

This section explores a model variant that directly learns the diffusion model on weights, bypassing the AutoEncoder stage, and compares it to the standard approach. Both variants are trained on 1000 subsets of ImageNet, and evaluated in in-distribution sampling setting on three randomly selected subsets from the 1000 subsets. The results, presented in Figure 6(b), indicate that learning the distribution of pretrained weights in the latent space is notably successful in generating high-performing weights. The failure of the DDPM process on raw pretrained weights may stem from their higher model capacity requirement.

### C.5 CLIP-based Dataset Encoding

In this section, the comparison between the CLIP-based dataset encoding scheme trained at an intermediate stage and the Set Transformer encoder jointly trained with the diffusion process is explored. Experiments are conducted on 140 Kaggle datasets and their respective model zoos.
The results depicted in Figure 7 indicate that both methods achieve similar results for small numbers of datasets during the in-distribution sampling. However, as the number of datasets increases, the Set Transformer jointly trained with the diffusion approach faces challenges in convergence and requires more computational resources, as demonstrated in Figure 7.

### C.6 Unconditional Sampling

We conduct the experiment using ResNet18 pretrained on CIFA-100 and CIFAR-10.
For all datasets, the weight vector length is 2048 and we compare with pdiff (Wang et al., ). While pdiff requires a separate model for each dataset, our method combines the pretrained weights into a single dataset and conditionally learns their distribution. The sample size for each dataset in our method is 200, with a combined total of 400 parameters. The results are provided in Table 10 for 100 sampled weights. Two separate models for are trained for pdiff, CIFA10-pdiff and CIFAR100-pdiff while our method consists of a single model trained once for both datasets. It can be seen that our method outperformance the baseline (Wang et al., ) in Table 10.

**Table 10: Unconditional Sampling Evaluation against Wang et al. () on ResNet18.**
| Dataset | CIFAR-10 | CIFAR-100 | Runtime |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Avg | Median | Max | #Epochs for<br>VAE,DDPM | Avg | Median | Max | #Epochs for<br>VAE,DDPM |  |
| pdiff | 94.46 | 94.46 | 94.52 | 8999,47999 | 76.1028 | 76.13 | 76.21 | 32999,38999 | $\approx 3h$ |
| D2NWG | 94.46 | 94.47 | 94.50 | 100,200 | 76.1796 | 76.18 | 76.24 | 100,200 | $\approx 1h30$ |

### C.7 Coupling with an Accuracy Predictor

This section reports the extended results of Table 18 in which we compared our method in-distribution and out-of distribution with and without accuracy predictor.

Results.: The full results of Table 18 are reported in Table 19. Using an accuracy predictor enable easily selecting highly performing when sampling in-distribution. However, in our case the accuracy predictor struggles to generalize well for unseen dataset as shown in Table 19

### C.8 Sampled Weights Analysis

In this section, we analyze the characteristics of the sampled weights and compare them to the pre-trained ones based on experiments with the model zoo and a model pre-trained on a subset of ImageNet. The proposed method samples weights with a large variance, as shown in Figure 10, providing a broad range of initialization choices, from weights with low initial performance to those with higher initial performance.

**Table 11: Details description of meta-album datasets**
| Domain | Original Dataset | # Classes |
| --- | --- | --- |
| Large Animals | Animals with Attributes, Dogs, Birds | 50, 120, 315 |
| Small Animals | Insects, Insects 2, Plankton | 117, 102, 102 |
| Plants | Fungi, PlantNet, Flowers | 25, 25, 102 |
| Plant Diseases | PlantDoc, Medicinal Leaf, Plant Village | 27, 26, 38 |
| Microscopy | Subcel. Human Protein, PanNuke, Bacteria | 21, 19, 33 |
| Remote Sensing | RSD, RSICB, RESISC | 43, 45, 45 |
| Vehicles | Boats, Airplanes, Cars | 26, 21, 196 |
| Manufacturing | Textures ALOT, Textures DTD, Textures | 250, 47, 64 |
| Human Actions | MPII Human Pose, Stanford 40 Actions, 73 Sports | 29, 40, 73 |
| OCR | Omniprint-MD-6, Omniprint-MD-5-bis, Omniprint-MD-mix | 703, 706, 706 |

**Table 12: Comparison between Base Model and Improved Model Responses**
| Instruction | Base Model Response | Improved Model Response |
| --- | --- | --- |
| Prompt: {"role": "system", "content": "You are an AI chatbot"},<br>{"role": "user", "content": "Explain what AI is and its importance"}, | What is AI? Artificial Intelligence (AI) refers to the development of computer systems that can perform tasks that would typically require human intelligence, such as:<br>1.<br>Learning: AI systems can learn from data, experiences, and environments to improve their performance.<br>2.<br>Reasoning: AI systems can reason and make decisions based on the data they have learned.<br>3.<br>Problem-solving: AI systems can solve complex problems and adapt to new situations.<br>4.<br>Perception: AI systems can perceive and understand the world around them through sensors, cameras, and other devices.<br>AI systems use various techniques, such as machine learning, deep learning, natural language processing, and computer vision, to achieve these capabilities.<br>Types of AI:<br>1.<br>Narrow or Weak AI: Designed to perform a specific task, such as facial recognition or language translation.<br>2.<br>General or Strong AI: Designed to perform any intellectual task that a human can, such as reasoning, problem-solving, and learning.<br>3.<br>Superintelligence: Significantly more intelligent than the best human minds, potentially leading to exponential growth in technological advancements.<br>Importance of AI:<br>1.<br>Automation: AI can automate repetitive and mundane tasks, freeing up human resources for more strategic work. | What is Artificial Intelligence (AI)? Artificial Intelligence (AI) is a branch of computer science that focuses on creating intelligent machines that can think, learn, and behave like humans. AI involves the development of algorithms, statistical models, and computer programs that enable machines to perform tasks that typically require human intelligence, such as:<br>1.<br>Reasoning and problem-solving: AI systems can analyze data, identify patterns, and make decisions based on that data.<br>2.<br>Learning: AI systems can learn from experience, adapt to new situations, and improve their performance over time.<br>3.<br>Perception: AI systems can interpret and understand data from sensors, such as images, speech, and text.<br>4.<br>Natural Language Processing (NLP): AI systems can understand and generate human language, enabling them to communicate with humans.<br>Importance of AI AI has become increasingly important in various aspects of our lives, including:<br>1.<br>Automation: AI can automate repetitive and mundane tasks, freeing up human time for more creative and strategic work.<br>2.<br>Improving decision-making: AI can analyze large datasets, identify patterns, and provide insights that inform business decisions.<br>3.<br>Enhancing customer experience: AI-powered chatbots and virtual assistants can provide 24/7 support and personalized services. |
| prompt;<br>{"role": "system", "content": "You work at a coffee shop"},<br>{"role": "user", "content": "what title do you give to someone who serves the coffee to customers"}, | At our coffee shop, we call the person who serves the coffee to customers a ’Barista’. However, some of our<br>team members are also referred to as ’Cafe Attendants’ or ’Coffee Servers’, depending on their specific<br>role and responsibilities. But generally, the title ’Barista’ is the most commonly used and well-known term<br>in the industry. | In a coffee shop, the person who serves the coffee to customers is typically referred to as a:<br>Barista: This is the most common title, and it refers to a skilled person who prepares and serves coffee drinks,<br>as well as other beverages and food items.<br>Coffee Server: This title is also used to describe someone who serves coffee and other beverages to customers.<br>Coffee Barista or Coffee Specialist: These titles are often used in specialty coffee shops or high-end<br>establishments to emphasize the barista’s expertise and knowledge of coffee.<br>In our coffee shop, I’m a barista, and I’m happy to serve you your favorite coffee drink. |

**Table 13: Glue benchmark tasks descriptor used in the experiment on glue datasets.**
| Task Name | Description |
| --- | --- |
| SSTB | Predict the similarity score between two sentences. Rate their similarity on a scale from 0 to 5, where 0 indicates no meaning overlap, 1 indicates very little overlap, and 5 indicates complete overlap in meaning. |
| MRCP | Determine the semantic equivalence of two given sentences (Sentence 1 and Sentence 2). If the sentences are semantically equivalent, return 1. If they are not, return 0. |
| SST2 | Determine the sentiment of a given sentence. Respond with 0 if the sentiment is negative and 1 if the sentiment is positive. |
| COLA | Evaluate whether the given sentence is both syntactically and semantically correct. If it is, respond with "1"; otherwise, respond with "0". |
| QNLI | Evaluate whether the given response properly answers the provided question. If the response answers the question correctly, return 0; otherwise, return 1. |
| RTE | Determine if a given hypothesis is true (entailment), false (contradiction), or undetermined (neutral) based on a provided premise. |

### C.9 Evalutaion on Large Datasets

We investigate how our method perform for combined large and small dataset as well for mixed architectures. For this experiment we collect the pretrained weights from PyTorch hub with one checkpoints per datasets(CIFAR-10, CIFAR-100, and ImageNet-1k). After conditionally learning the combined weights distribution, we sampled 10 weights for each datasets and report the average accuracy of the top-3 datasets in Table 14. As shown in Table 14, D2NWG consistently produced high performing weights for each dataset from a single pretrained checkpoint.

**Table 14: Evaluation on Large Datasets**
| Datasets | CIFAR10<br>(ShuffleNet) | CIFAR100<br>(ShuffleNet) | ImageNet-1k<br>(SqueezeNet) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Methods | Top1 | Top5 | ToP1 | Top5 | Top1 | Top5 |
| Pretrained | 92.98 | 99.73 | 72.39 | 91.46 | 58.178 | 80.624 |
| Ours(sampling) | 93.14 $\pm$ 0.25 | 99.76$\pm$ 0.22 | 72.60 $\pm$ 0.15 | 91.29 $\pm$ 0.13 | 58.257 $\pm$ 1.022 | 81.01$\pm$ 1.251 |

### C.10 Generating the Full Weights for ResNet18

We investigate how our method performs when used to generate the full parameters for a ResNet18 model pretrained on MNIST, CIFAR-10, and CIFAR-100. In total, we use 100 pretrained weights per dataset and conditionally learn their distribution. The modelzoo generation follows the same setting as Wang et al. (). Table 15 demonstrates the effectiveness of our method for generating the entire weights of a network.

**Table 15: Zero-Shot Transfer Learning This Table represent results of zero-shot evaluation against the pretrained model on Resnet18 full model architecture.**
| Model | MNIST | CIFAR-10 | CIFAR-100 |
| --- | --- | --- | --- |
| Pretrained | 99.61 | 94.56 | 75.86 |
| D2NWG(ours) | 99.62 $\pm$ 0.07 | 94.57 $\pm$ 0.00 | 75.83 $\pm$ 0.02 |

### C.11 Generating Weights for MobileNetV3

So far, our focus has been on model zoos populated by relatively simple classifier heads. In this section, we evaluate our method using MobileNetV3, a subnetwork sampled from OFA (Cai et al., ), consisting of 2.8 million parameters fine-tuned on CIFAR-10, STL-10, SVHN and MNIST for 15 epochs. We collect the last 10 checkpoints per dataset and utilize our method to learn the distribution of pretrained weights.
Furthermore, we combine the pretrained weights of MNIST and CIFAR-10, learn their distribution, and then evaluate our method on SVHN and STL-10. Subsequently, we reverse this process by combining the pretrained weights of SVHN and STL-10, and evaluate our method on MNIST and CIFAR-10.

As shown in Table 22 our method enhances the performance of the pretrained model. Furthermore, we note that learning the full model weights does not compromise performance. Although learning the distribution of the classifier head is computationally efficient, it can result in lower performance.

### C.12 Generating Weights for Vision Transformers

Our method shows the ability to learn the distribution of all parameters within a vision transformer, including convolutional and linear layers. We present in-distribution evaluation results in plot Figure 9, highlighting the learning of combined weight distributions conditioned on individual datasets. The model zoo for ViTs is collected based on models proposed by Gani et al. ().

## Appendix D Application to Large Language Model (LLM) Output Layer Generation

Phi-3-MINI-4K-Instruct:We conduct experiments on the Microsoft Phi-3-MINI-4K-Instruct model to demonstrate the scalability of our method for generating output layers in large language models (LLMs). The model’s 98.5 million-parameter output layer was split into 96 chunks, each of size 1,026,048, and used as training data for a Variational Autoencoder (VAE) with an embedding size of 1,024. Lacking access to original training data, we used a class-conditional diffusion process, with chunk embeddings as conditioning data. Post-training, conditioned chunks were sampled and concatenated to reconstruct the original output vector. We evaluate our method using the Open-LLM LeadearBoard-1.
As shown in Table 16, our approach effectively scales to the LLMs head generation demonstrating adaptability across diverse domains with minimal adjustments to conditioning data.

**Table 16: Generating weights for the Microsoft Phi-3 language model output head.**
| Methods | ARC Challenge (25-shots) | ARC Easy (25-shots) | HellaSwag (10-shots) | Winogrande (5-shots) |
| --- | --- | --- | --- | --- |
| Pretrained | 87.16 $\pm$ 0.00 | 63.23 $\pm$ 0.01 | 73.65 $\pm$ 0.01 | 76.64 $\pm$ 0.01 |
| D2NWG | 87.36 $\pm$ 0.01 | 63.74 $\pm$ 0.01 | 73.65 $\pm$ 0.00 | 76.72 $\pm$ 0.01 |

GPT2: In this experiment, we show that our method can learn the distribution of any layer in an LLM by modeling the full distribution of GPT-2 small (164M parameters). We use a chunk size of 1,523,712 and, unlike Llama architectures, concatenated all vectorized layer weights before chunking them uniformly. Table 17 highlights the method’s effectiveness on the Open LM-Leaderboard benchmark. While it did not outperform the base model overall, it significantly improved performance on certain tasks and maintained average accuracy comparable to the pretrained model.

**Table 17: Performance evaluation on unseen open llms leaderboard v2 benchmark base on full gpt2-164M small. These results are produced by Huggingface after submission to open LLM leaderdoards. $\uparrow$ indicate performance improvement while $\downarrow$ indicate a performance decrease**
| Method | ifeval (0) | Bbh (3) | Gpqa (0) | MATH-hard (4) | Musr (0) | MMLU-Pro (5) | Avg | Base Model | Fine-tuned |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| openai-community-gpt2 | 17.8 | 2.83 | 1.12 | 0.3 | 13.91 | 1.84 | 6.3 | na | Yes |
| D2NWG | 19.16($\uparrow$1.36) | 2.85($\uparrow$0.02) | 1.01($\downarrow$0.11) | 0.38($\uparrow$0.08) | 12.68($\downarrow$1.23) | 1.68($\downarrow$0.16) | 6.29($\downarrow$0.01) | openai-community-gpt2 | No |

### D.1 Fast Convergence Performance Evaluation

In this section we report supplementary results for experiment on tiny model zoo dataset. The pretrained weights used here are from epochs 21 to 25 for each dataset where 70% of the resulting modelzoo is used for training and 15% for validation and testing respectively. The number of pretrained weights in the modelzoos are 3500 for MNIST, CIFAR-10, and STL-10, and 2864 for SVHN. The flattened network weights’ length is 2864 for CIFAR-10 and STL-10 and, 2464 for MNIST and SVHN. We pad all the weights with zero to 2864.

Figure: (a) MNIST
Refer to caption: x11.png

**Table 18: No Fine-tuning Initialization on Unseen Datasets We transfer from one dataset, or combinations of datasets, to unseen datasets at test time.**
| Source | Target | Accuracy | Methods |
| --- | --- | --- | --- |
| MNIST | SVHN | 13.25 | SKDE30 |
| SVHN | MNIST | 29.30 |  |
| CIFAR-10 | STL-10 | 15.20 |  |
| STL-10 | CIFAR-10 | 15.40 |  |
| Sampling from Combined Weights Distribution |  |  |  |
| MNIST+CIFAR-10 | SVHN | 18.80 | Ours |
| MNist+CIFAR-10 | STL-10 | 16.21 |  |
| SVHN + STL-10 | MNIST | 36.64 |  |
| SVHN + STL-10 | CIFAR-10 | 18.00 |  |

### D.2 Sampling Weights for Unseen Datasets

Task: We evaluate the transferability of the models on unseen datasets. We create disjoint modelzoos by combining MNIST and CIFAR-10 into a single modelzoo and combining the SVHN and STL-10 modelzoos. When we train on the MNIST plus CIFAR-10 modelzoos, we test on the SVHN and STL-10 modelzoos and vice-versa.

Results: As shown in Table 18, D2NWG is able to sample weights with higher accuracy on unseen datasets as well as for in distribution. Through these experiments our method does not only outperform the baseline it also demonstrates promising results for dataset-conditioned sampling for unseen datasets.

**Table 19: Performance evaluation at initialization without fine-tuning. For the baseline we use weights of SVHN for MNIST and vice versa similarly for CIFAR-10 and STL-10**
| Datasets | MNIST | SVHN | CIFAR10 | STL10 |
| --- | --- | --- | --- | --- |
| Random | 10.23$\pm$0.56 | 12.21$\pm$3.76 | 9.98$\pm$1.47 | 9.56$\pm$1.02 |
| Pretrained models | 82.82$\pm$ 1.38 | 67.57$\pm$ 0.59 | 44.68$\pm$ 3.15 | 35.99$\pm$ 1.15 |
| Skde30Schürholt et al. () | 69.73$\pm$ 5.12 | 50.25$\pm$ 6.12 | 26.06$\pm$ 3.01 | 17.20$\pm$ 3.43 |
| seen (D2NWG) | 83.92$\pm$1.92 | 61.81 $\pm$ 3.13 | 43.08$\pm$0.55 | 31.45$\pm$0.35 |
| seen(D2NWG)(with Pred) | 84.85$\pm$0.83 | 66.03 $\pm$ 1.36 | 43.89$\pm$0.15 | 34.29$\pm$0.13 |
| Skde30Schürholt et al. ()(cross) | 29.30$\pm$ 3.46 | 13.25$\pm$ 1.12 | 15.40$\pm$ 0.51 | 15.20$\pm$1.24 |
| not seen(D2NWG) | 36.64$\pm$4.69 | 18.80$\pm$0.58 | 18.00$\pm$0.22 | 16.21$\pm$0.52 |
| not seen(D2NWG)(with Pred) | 30.15$\pm$5.09 | 15.76$\pm$1.43 | 17.10$\pm$1.12 | 15.37$\pm$0.52 |

**Table 20: In-distribution performance comparison of different image dataset encoding schemes on model zoo dataset**
| Datasets | MNIST | SVHN | CIFAR10 | STL10 |
| --- | --- | --- | --- | --- |
| Pretrained models | 82.82$\pm$ 1.38 | 67.57$\pm$ 0.59 | 44.68$\pm$ 3.15 | 35.99$\pm$ 1.15 |
| Skde30Schürholt et al. () | 69.73$\pm$ 5.12 | 50.25$\pm$ 6.12 | 26.06$\pm$ 3.01 | 17.20$\pm$ 3.43 |
| MLP_Encoder | 67.04$\pm$17.73 | 35.65 $\pm$ 13.03 | 17.41$\pm$3.02 | 20.36$\pm$7.38 |
| Set_transf(pret) | 78.21$\pm$1.76 | 60.90 $\pm$ 1.08 | 28.68$\pm$1.84 | 34.75$\pm$00.38 |
| seen (D2NWG) | 83.92$\pm$1.92 | 61.81 $\pm$ 3.13 | 43.08$\pm$0.55 | 31.45$\pm$0.35 |
| seen(D2NWG)(with Pred) | 84.85$\pm$0.83 | 66.03 $\pm$ 1.36 | 43.89$\pm$0.15 | 34.29$\pm$0.13 |

**Table 21: Performance of the datasets conditional sampling on 10 unseen real-world datasets. We report the averaged accuracy on ten unseen test datasets over 3 different runs fine-tuned for 50 epochs. pret(imnet): pretrained on imagenet1k**
| Datasets | No-fine-tuning | 50 epochs Fine-Tuning | # of classes |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Random init. | pret(imnet) | D2NWG(ours) | Random init. | pret(imnet) | D2NWG(ours) |  |  |
| Gemstones | 1.13 $\pm$ 0.52 | 0.62 $\pm$ 0.00 | 1.86 $\pm$ 0.25 | 70.59$\pm$ 0.91 | 67.49$\pm$ 0.43 | 76.06 $\pm$ 0.88 | 87 |
| Dog Breeds | 0.55 $\pm$ 0.22 | 0.69 $\pm$ 0.00 | 1.87 $\pm$ 0.39 | 80.78$\pm$ 0.28 | 78.13$\pm$ 0.49 | 80.88 $\pm$ 0.88 | 133 |
| Dessert | 21.03 $\pm$ 2.44 | 12.50 $\pm$ 0.00 | 99.40 $\pm$0.02 | 95.83$\pm$0.34 | 94.64$\pm$ 0.00 | 99.40 $\pm$ 0.02 | 5 |
| Colorectal Histology | 11.77 $\pm$2.88 | 11.00 $\pm$ 0.00 | 18.12 $\pm$ 0.25 | 90.34 $\pm$ 0.33 | 89.75$\pm$ 0.19 | 93.65 $\pm$ 0.10 | 8 |
| Drawing | 10.86 $\pm$ 1.22 | 11.00 $\pm$ 0.00 | 11.87 $\pm$0.93 | 90.20 $\pm$ 0.16 | 90.00$\pm$ 0.16 | 89.00 $\pm$ 0.16 | 10 |
| Alien vs Predator | 51.48 $\pm$2.09 | 28.88 $\pm$ 0.00 | 78.15 $\pm$0.52 | 98.52$\pm$ 0.52 | 98.89$\pm$ 1.42 | 97.77 $\pm$ 0.00 | 2 |
| COVID-19 | 20.13 $\pm$18.66 | 46.53 $\pm$ 0.00 | 47.22 $\pm$0.00 | 93.86$\pm$0.16 | 93.40$\pm$ 0.49 | 94.56 $\pm$ 0.71 | 3 |
| honey-bee-pollen | 49.54 $\pm$1.30 | 50.00 $\pm$ 0.00 | 56.94 $\pm$4.53 | 93.05 $\pm$ 0.00 | 88.89$\pm$ 0.00 | 93.55 $\pm$ 4.53 | 2 |
| Speed Limit Signs | 30.55 $\pm$2.27 | 25.00 $\pm$ 0.00 | 31.48 $\pm$10.23 | 83.33$\pm$ 0.00 | 86.11$\pm$ 0.00 | 90.74 $\pm$ 1.31 | 4 |
| Japanese Characters | 0.03$\pm$0.00 | 0.08 $\pm$ 0.00 | 0.50$\pm$0.22 | 53.17 $\pm$ 0.15 | 62.33 $\pm$ 0.16 | 62.16 $\pm$0.47 0.45 | 1566 |

Figure: Table 22: MobileNet Weight Generation.
Refer to caption: x16.png

Figure: (a) TSNE
Refer to caption: x17.png

## Appendix E Miscellanea

**Table 23: Model components and their configuration modes for llma3.2.1B**
| ID | Name | Type | Params | Mode |
| --- | --- | --- | --- | --- |
| 0 | Model | DiffusionWrapper | 102 M | Train |
| 1 | Model Ema | LitEma | 0 | Train |
| 2 | First stage Model | VAENoDiscModel | 553 M | Eval |
| 3 | Cond Stage Model | IdentityCondStage | 0 | Eval |

In Table 23 we present the parameter count for the model used to learn the distribution of the 25% of llama-3.2-1B transformer blocks. In Table 24 we showcase the set of experiments and the corresponding number of parameters generated by D2NWG . Although D2NWG is capable of generating up to 1 billion parameters, all our experiments were limited to a maximum of 872 million, achieved using the Llama 3.1-8B model with 4 transformer layers, excluding layer normalization, for which we constructed a separate model. This parameter count makes D2NWG the only method, to the best of our knowledge, capable of generating nearly a billion parameters, significantly enabling large architecture weights generation including GPT-2 and most existing image classification models in terms of parameter scale.
For non-LLM models, we utilize joint distribution learning, enabling task or dataset-conditioned sampling. For example, CIFAR-10 and ImageNet are considered two separate datasets, while SST-2 and CoLA in the GLUE benchmark are treated as two distinct tasks, regardless of differences in the number of classes or subtasks within each dataset or task.
Table 24 highlights that the proposed method supports text and image conditioning, as well as layer- or chunk-wise conditional sampling. D2NWG is one of the first weight generation methods to produce over 800 million parameters in a single instance without tiling. Additionally, it is among the first to effectively explore weight generation across various domains, learning the distribution of combined models pretrained on diverse tasks or datasets.

**Table 24: Summary of Experiments for Figures and Tables presented. Min #cls and Max #cls correspond to the minimum and maximum number of classes respectively.**
| Object | # Datasets | Min #cls | Max #cls | #Params | Trainset Size | Conditioning |
| --- | --- | --- | --- | --- | --- | --- |
| Table 1 | 10 | 1 | 5 | 2565/8005 | 50k | Dataset |
| Table 2 | 5 | 10 | 100 | 128100 | 20k | Dataset |
| Table 4.1.2 | 30 | 19 | 706 | 3 M | 30 | Dataset |
| Table 4.1.3 | 4 | 10 | 10 | 10853 | 4 | Dataset |
| Table 5 | 6 | 2 | 3 | 0.6M | 6 | Text Description |
| Table 6 | NA | NA | NA | 872M | NA | Chunk Indices |
| Table 7 | NA | NA | NA | 872M | NA | Chunk Indices |
| Table 9 | 2 | 10 | 100 | 0.7M | 2 | Dataset |
| Table 10 | 2 | 10 | 100 | 2048 | 2 | Dataset |
| Table 14 | 3 | 10 | 1000 | 1.4M | 3 | Dataset |
| Table 15 | 3 | 10 | 100 | 11M | 2 | Dataset |
| Table 15 | 4 | 10 | 10 | 2.8M | 4 | Dataset |
| Table 16 | NA | NA | NA | 96M | NA | Chunk Indices |
| Table 17 | NA | NA | NA | 164M | NA | Chunk Indices |
| Figure 2 | 10 | 2 | 1566 | 136468 | 140 | Dataset |
| Figure 5 | 2 | 10 | 100 | 0.47M | 2 | Dataset |
| Figure 6(a) | 2 | 10 | 10 | 5310 | 2 | Dataset |
| Figure 7 | 2 | 10 | 10 | 5310 | 2 | Dataset |
| Figure 9 | 5 | 10 | 200 | 2.8M | 5 | Dataset |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness section for my concerns and questions.


Writing suggestions:
- What do the task descriptions look like for NLP tasks which are used for encoding them? Adding pointers in section 3.3 (line 234-237) will be helpful for readers.
- Section 3.3 is a bit confusing to read due to paragraph titles. It looks like authors use 4 types of encoding in their work corresponding to 4 paragraphs but it’s actually just 2 (image and language dataset encoding). The middle two paragraphs (set-based dataset encoding and contrastive dataset encoding) are essentially describing image dataset encoding in detail. It may be a good idea to re-organize this section to be clearer.

Nit:
- Double dot at the end of line 116.
- Line 1147: “ios” -> “is”
- Line 184: “Parametrs” -> “Parameters”
- Line 1323: “metho” -> “method”
- Line 512: “robustnets” -> “robustness”
- Line 1331: remove “par”
soundness: 2
strengths: - The proposed approach is conceptually simpler to train and test than previous meta-learning approaches which often involve solving a bi-level optimization problem.
- The dataset conditioning in the approach increases the generality of the approach to new unseen datasets.
- Extensive sets of experiments are presented for different settings such as few-shot learning, zero-shot learning, model retrieval, classifier head adaption, LoRA weight generation, and adapting LLM weights for specific tasks.
summary: Authors propose a diffusion-based approach to generate parameters of neural networks for transfer learning conditioned on the target dataset. Prior work often relies on the diversity of pretrained models or datasets used to train these parameter prediction models but it limits their ability to generalize to new datasets. Encoding both model parameters and architecture jointly enhances the generality of the parameter prediction approach to unseen tasks. Several experiments are performed to test the efficacy of the approach for domains such as vision and large language models, demonstrating that it outperforms several meta-learning and parameter prediction approaches.
weaknesses: - Although the authors argue that previous approaches rely on diversity of training architectures and dataset for generalization, the proposed approach also has limitations along the same axis: it may only generalize to new datasets which are in-distribution for the dataset encoder otherwise the dataset encodings can be noisy. For instance, if the dataset encoder is only trained on imagenet-like natural images, it may not generalize to unseen dataset distributions like medical images. Table 17 seems to suggest the same since accuracies for unseen datasets are poorer in absolute numbers, so it’s unclear if dataset encoding is really needed otherwise one could simply train a parameter prediction model on combined datasets which could generalize to new ones.
- Missing baselines/ablations: Building on above, there are no ablations or baselines testing how effective are each component in the proposed approach: dataset encoding, layer-vectorization, spectrum ranking. It’s unclear how much importance all these components play in the final performance of the proposed approach. Moreover, some crucial baselines are missing -- for instance, training a simple param prediction diffusion model (e.g. Peebles et al. (2022)) without dataset encoding vs training it with dataset encoding to see how much generalization performance is improved with dataset encoding (which is one of the main contributions of this work).
- It seems that image dataset encoding uses the dataset samples (images) with a set transformer model whereas the language dataset encoding is done by simply picking LLM embeddings of the task description which seems weaker. If it’s not, is it possible to also encode image datasets in the same task-description-based way using LLM or VLM?
- For LLM param generation, authors use spectrum technique from prior work in order to deal with a large number of parameters in LLMs but it’s unclear how effective it is. It may be a good idea to use this technique on small-to-medium scale models where encoding all parameters is feasible to gauge the loss of performance and potential scope for improvement in this stage.
- It seems that authors train separate D2NWG models for param prediction in each setting. Is it possible to train a single param prediction model? Are there any technical hurdles in the approach which could prevent doing this (assuming access to large-scale compute and data)?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weakness for my questions. If it is solved, I am willing to raise my score.
soundness: 3
strengths: 1. The paper investigates an interesting problem, the methodology is promising, and the paper is well-written and easy to read.
2. The experimental results are very solid, covering diverse tasks across a variety of datasets. These results outperform state-of-the-art meta-learning methods and pre-trained models.
summary: This paper proposes a diffusion-based neural network weight generation technique named D2NWG, which directly learns the weight distributions of models pre-trained on various datasets. Specifically, the authors have developed a complete framework for latent diffusion models to learn from pairs of weights and datasets. The paper evaluates D2NWG across various tasks, both for direct generation and additional fine-tuning, outperforming state-of-the-art meta-learning methods and pre-trained models. The results demonstrate a capability for scalable transfer learning.
weaknesses: 1. Although the idea is interesting and the methodology looks promising, my main concern is about the practical significance of the proposed method, considering that training an effective diffusion model requires substantial computational resources. I will expand on this concern below:
2. Tables 1 and 2 show that D2NWG outperforms several baselines. Notably, D2NWG is tagged with 'CH', indicating that it only modifies the last classifier layer, similar to linear probing. This suggests that it may still be challenging for D2NWG to generate complete parameters with robust performance.
3. The concept of 'Zero-shot adaptation' does not seem applicable as the weights generated by D2NWG are conditioned on the target dataset (possibly the test set). This could be considered a form of 'test data leaking', which limits the practical utility of D2NWG.
4. Section 4.2 discusses the need for further fine-tuning of the generated models, suggesting that D2NWG's role is similar to that of pre-trained models. However, there is no evidence to suggest that training a usable D2NWG model is more cost-effective than using pre-trained models.
5. The generalization capability remains the most challenging issue for D2NWG. Although the authors attempt to demonstrate its adequacy across a range of datasets, there is no evidence that a well-trained D2NWG can handle diverse tasks and different model architectures simultaneously. Currently, D2NWG seems like a suboptimal alternative to a well-developed optimizer. Additionally, the supporting components such as the autoencoder and the dataset encoder, which are trained separately, also suffer from generalization issues.

---

confidence: 3
contribution: 1
initial_rating: 6
presentation: 2
questions: 1. **Quantifying and Mitigating Bias in Public Checkpoint Weight Distributions**: The paper leverages publicly available pretrained models for weight distribution modeling, but it doesn’t address how potential biases in these weights—arising from the original datasets or training methods—might affect the generated weights. How can biases be quantified in these weight distributions, and what mechanisms could be implemented to ensure the generated weights are robust to such biases? Addressing this might require specific adjustments to the diffusion model or additional constraints on the weight distributions.

2. **Criteria for Selecting Additional Datasets for Training Target Architectures**: The method involves further training on a curated selection of datasets to improve model adaptability. However, there is little guidance on how to select these datasets to optimize the target architecture effectively. What are the criteria or metrics that could guide dataset selection to ensure it sufficiently represents the intended tasks and enhances generalization without introducing redundancies or biases?

3. **Evaluating the Impact of Generating All Parameters Versus Only the Classification Head**: In Table 1, the results compare the performance of models with generated classification heads to baseline models. However, it would be informative to understand the effect of generating all parameters instead of just the classification head. Would this provide additional performance gains, or might it introduce complications in terms of stability or adaptability to new tasks?
soundness: 2
strengths: 1. The authors considered a wide range of evaluation settings (zero/few-shot learning, model retrieval, fine-tuning, domain transfer, conditioned weight generation, etc.) and provided comprehensive experiments.
2. The authors considered multiple ways to vectorized the model weights and encode the datasets.
summary: This paper presents a method called D2NWG, a diffusion-based neural network weight generation technique that enhances transfer learning by generating task-specific neural network weights based on target datasets. Unlike traditional transfer learning approaches that require fine-tuning or access to large collections of pretrained models, D2NWG uses a latent diffusion process to model and sample weight distributions, allowing efficient generation of adaptable weights for both seen and unseen tasks. This approach is scalable to large architectures, including large language models, and shows improved performance across various domains, outperforming existing meta-learning and pretrained models. The technique has been tested extensively, demonstrating superior adaptability and convergence speed, making it a promising tool for scalable transfer learning and model customization across diverse tasks.
weaknesses: 1. **High Workload and Tuning Challenges Due to Additional VAE Training**: The additional VAE for parameter encoding introduces significant computational workload and requires hyperparameter tuning, making it less scalable and potentially costly. What steps could be taken to simplify this process, and are there alternative methods for parameter encoding that maintain performance but reduce overhead?

2. **Ambiguity in Dataset Encoding Choices in Section 3.3**: The paper mentions several methods for encoding datasets, but it’s unclear how to choose the best encoding approach for a new dataset. Could the authors clarify the conditions under which each encoding method would be preferable, and provide a decision-making framework or guidelines to help practitioners adapt this approach to their datasets?

3. **Minor Typos and Formatting Issues**: There are minor typos and formatting inconsistencies in the paper that could be addressed to improve clarity and professionalism. For example, a colon is missing on line 177, and "while" on line 184 should begin with a capital "W." A thorough proofreading could enhance readability and reduce the risk of misinterpretation due to formatting errors.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 1
questions: I will list my questions according to the order appearing in the paper. You may want to combine some of them when rebuttal if necessary.
- Introduction:
    - You mentioned recent methods in latent diffusion-based weight generation in line 47 and said their limitations are limited to small models and a subset of parameters. However, in this paper, you try to generate parameters for classification heads (in Sect. 4.1) and LoRA (in Sect. 4.4). It seems that you are just generating a small number of parameters. So is this paper different from existing methods under this context? You also mentioned "generating weights for architectures with over 200 million parameters" in the fourth contribution, but I cannot find the supporting experiments for that.
    - You also mentioned some meta-learning few-shot methods in line 51 and said their generated parameters still need fine-tuning. Does this paper overcome the requirement of fine-tuning? I asked this question because in Table 2 of Sect. 4.1.2, the accuracy of D2NWG on those simple classification tasks are still not satisfying, which indicates the requirement of further fine-tuning. For example, for CIFAR-10 and Aircraft, I'm sure that after a few epochs of fine-tuning, the accuracy will reach above 90%. Then the current accuracy of D2NWG is not acceptable.
    - In Figure 1, in the sampling stage, the unseen architecture is also used as the condition to generate parameters. However, there is no explanation about unseen architecture in the main paper. I do see the results in the Appendix, but the unseen architecture is not really used as the condition. The method actually generates parameters for the seen architecture and then wraps it for the unseen architecture with some "careful concatenation" as mentioned in line 1171. So what is the "careful concatenation" here? I believe this part should be put in the main paper to fit Figure 1.

- Approach:
    - Typo in line 171: "The extraction process follows one one the main preprocessing methods:"
    - When will the model-wise vectorization be used and when will the layer-wise one be used?
    - Can you give an intuitive explanation about set-based dataset encoding in line 208, like encoding images from each class separately, then convert to the dataset encoding? It is currently hard to understand $x_p^{y_i}$.
    - In dataset-conditioned training, why the weights latent is used as the condition as well? It's like in the text-to-image diffusion model, the model will be conditioned on the text only, and the image is the distribution to learn. Why it is used as a condition as well? If it is conditioned on the weight, how can you do the inference without weight latent?
    - In Sect. 3.5, you mentioned "ranks LLM layers based on their importance", however, the experiments in Sect. 4.4 is about LoRA. If you choose to claim that in the main paper, you should put relevant experiments in the main paper.

- Experiments:
    - In Table 1, can you explain what is 5-way 1-shot a little bit?
    - In Table 2, for D2NWG_CLIP, I believe it is using CLIP. Then how about the two rows of D2NWG? What does it use?
    - In Sect. 4.1.1, you mentioned "evaluate the performance on 600 subsets from the UNSEEN test split". What do you mean by "unseen" here? It's like in general model training, we have train set and test set, and we test the trained model on the test set. Now you have multiple trained models, and you use that to train the diffusion model to generate weights, which is expected to work on the test set just like what we did in the normal training. Why it can be called "unseen" task?
    - In line 348, you mentioned "potentially eliminating or significantly speeding up the fine-tuning process". As I mentioned in the last point, it's normal to NOT fine-tune if it is a SEEN task/dataset. While for real unseen tasks as you did in Sect. 4.3, the generated parameters do need fine-tuning.
    - Starting from Sect 4.1.3, the experiments become hard to follow since you change the settings in each section frequently. It's better to organize your experiments better. For example, for the experiments in Sect. 4.2, we can still use the dataset in Table 2. Otherwise, it's hard to understand (1) What are the train and test datasets? Are the test ones seen or unseen? How do you define unseen? (2) How do you train the diffusion model? What models' parameters do you use to train the diffusion model? 
    - In Figure 2, why dessert food is so good from the beginning? Do you have any clue? Whether because the class/dataset is put in the trainset occasionally?
    - Sect. 4.4. is still seen task case right?

- Suggestion:
    - So generally you'd better make it clear about what you want to claim. For example, you may want to say, for the seen task, the generated parameters can be used directly with high accuracy. For the unseen task, the generated parameters may need fine-tune but with better final accuracy and faster convergence. 
    - Then for each claim, you will have an experiment section to support. Each section may include both results for CNN classification and NLP task.
    - The settings should be uniform throughout the paper. If some experiments do need different setting, give reason.
soundness: 3
strengths: - This paper is interesting in terms of the overall topic - generating the model parameters using the latent diffusion model. To my knowledge, there are only two other papers [1,2] in this direction and they have not been published yet. 
- The idea is easy to understand.

[1] Wang K, Xu Z, Zhou Y, et al. Neural network diffusion. arXiv 2024.    
[2] Jin X, Wang K, Tang D, et al. Conditional lora parameter generation. arXiv 2024
summary: This paper proposes to generate parameters using the latent diffusion model, which is a very new direction. Compared to traditional meta-learning, the authors want to show that D2NWG can generate parameters for both seen and unseen dataset, be applied to either classification task or NLP task, and achieve better accuracy with faster convergence. The most important contribution comes from the dataset encoding and use it as the condition when generating parameters.
weaknesses: - The presentation is poor, especially the experimental parts. It's like people writing each subsection separately and putting them together directly without good organization. 
- There are gaps between the claimed contribution and the real work they did. 

Please refer to the questions part. Thanks.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: **[Q1]:** Could the authors provide a detailed analysis of the computational resources required to maintain and utilize the extensive collections of pretrained weights in D2NWG? Specifically, how does the storage and processing overhead of D2NWG compare to traditional pretraining and fine-tuning methods, particularly when scaling to very large models such as LLMs?

**[Q2]:** In terms of generalization, how does D2NWG ensure that the generated weights do not overfit to specific characteristics of the training datasets? Are there mechanisms in place to promote diversity and prevent homogenization of the generated weights?
soundness: 3
strengths: - Paper introduces a novel approach by combining latent diffusion models with neural network weight generation
  
- The authors conduct thorough experiments across various tasks, providing strong empirical evidence of the method’s effectiveness.
  
- The method exhibits robust performance both within the distribution of training datasets and on unseen datasets
summary: This paper introduces D2NWG, a diffusion-based method for generating neural network weights conditioned on target datasets. The key idea is to learn the distribution of pretrained model weights across different datasets and use latent diffusion to generate new weights that are optimized for specific tasks. The method can generate both classifier heads and full model weights, and scales to large architectures including LLMs. Extensive experiments demonstrate strong performance on few-shot learning, zero-shot transfer, and model retrieval tasks.
weaknesses: 1. **Limited Theoretical Analysis:**
   
   While the paper presents strong empirical results, it lacks a comprehensive theoretical framework that explains why diffusion-based weight generation effectively enhances transfer learning and generalization.

2. **Computational Costs:**
   
   The D2NWG method necessitates maintaining extensive collections of pretrained weights across a multitude of datasets. This requirement can lead to significant storage and computational overhead, potentially surpassing the costs associated with traditional pre-training approaches. Additionally, details on the training time and resource consumption compared to existing methods are limited.

3. **Reliance on Dataset-Specific Encoding:**
   
   The effectiveness of D2NWG heavily depends on the dataset encoding mechanisms, such as the Set Transformer and CLIP-based encoders. This reliance may limit the method's adaptability to datasets with unique characteristics or those that significantly differ from the training datasets.

Meta-review: This paper presents a diffusion-based approach for generating neural network parameters conditioned on dataset representations. The key claim is that by learning a latent distribution of pretrained weights from multiple datasets, the method can efficiently generate task-specific or domain-specific weights for both seen and unseen tasks. The approach is tested on various domains, including vision and large language models, and is shown to outperform several  baselines.
Weaknesses involve the complexity and cost of training the VAE and diffusion components, the reliance on dataset encoders (which may limit generality), unclear theoretical results. Some reviewers also questioned the practical benefits compared to simpler baselines and the scalability to large model zoos. Overall, the paper proposes a novel method, opens a promising direction , and has an extensive set of experiments thus the meta-reviewer recommends accept

Decision: Accept (Poster)