Conditional LoRA Parameter Generation

## Abstract

Abstract Generative models have achieved remarkable success in image, video, and text domains.
Inspired by this, researchers have explored utilizing generative models to generate neural network parameters.
However, these efforts have been limited by the parameter size and the practicality of generating high-performance parameters.
In this paper, we propose Cond P-Diff , a novel approach that demonstrates the feasibility of controllable high-performance parameter generation, particularly for LoRA (Low-Rank Adaptation) weights, during the fine-tuning process.
Specifically, we employ an autoencoder to extract efficient latent representations for parameters.
We then train a conditional latent diffusion model to synthesize high-performing model parameters from random noise based on specific task conditions.
Experimental results in both computer vision and natural language processing domains consistently demonstrate that Cond P-Diff can generate high-performance parameters conditioned on the given task.
Moreover, we observe that the parameter distribution generated by Cond P-Diff exhibits differences compared to the distribution obtained through normal optimization methods, indicating a certain level of generalization capability.
Our work paves the way for further exploration of condition-driven parameter generation, offering a promising direction for task-specific adaptation of neural networks.

## 1 Introduction

Recent advancements in generative models  [, , , ] have marked substantial progress across several domains of artificial intelligence.
In the computer vision domain, generative adversarial networks  [], diffusion models  [], and other approaches  [, ] have shown impressive results in image synthesis and manipulation.
Notably, models such as Stable Diffusion  [], DALL-E 2  [], and Imagen  [] have set new benchmarks in the quality and resolution of generated images.
Moreover, video generation models like Sora  [] have shown promising results in producing coherent and high-quality video sequences, opening new avenues for applications in entertainment and media.
In the natural language processing domain [, , ], autoregressive models like GPT [] and Llama [] have demonstrated promising generation capabilities and alignment with human preference [, , , ], which underscore the potential of generative models.

Inspired by these achievements, recent studies [, ] have begun to explore the application of generative models in novel areas, generating high-performing model parameters.
These studies focus on directly generating novel model parameters to accelerate the training process, uncovering parameters that achieve comparable performance with those obtained through conventional optimization methods.

By harnessing the power of generative models, it is possible to substantially reduce the computational cost and time required for model optimization [, , ].
Besides, examining the latent relationships between model parameters and performance provides valuable insights into the behavior and characteristics of neural networks [].

However, previous works on parameter generation [, , , , ] face several limitations.
On the one hand, the scale of parameters generated by prior methods [, , ] is insufficient for practical applications. For example, G.pt [] has been evaluated only on relatively simple datasets such as MNIST and CIFAR-10, which may not sufficiently demonstrate its generalization ability when applied to more complex tasks, and p-diff [] can generate small-scale high-performance parameters for simple architectures. Besides,  [] learn a hyper-representation on model zoos for generative use to sample new small-scale model weights.
On the other hand, previous methods do not support conditional high-performance parameter generation.
P-diff[] lacks support for conditional parameter generation, a crucial feature for real-world applications.
Although G.pt  [] enables controllable parameter generation as an optimizer, it can hardly exhibit comparable performance compared to networks trained by conventional optimization methods.

Therefore, despite their promising potential, these methods grapple with constraints about parameter size, practicality, and overall performance, which yield the primary question to be explored in this paper:
(Q) Can we synthesize high-performance parameters conditioned on the given task practically?

Figure: Figure 1: High-performance LoRA parameters generation process by Cond P-Diff in vision and language domains.
Refer to caption: x1.png

To enhance the practicality of parameter generation, two main challenges exist.
First, parameter generation for complex models entails significant data preparation costs. For example, G.pt [] requires training 23 million models, which is infeasible for large models.
Second, controllable parameter generation is challenging due to the difficulty in modeling the distribution of parameters, making full parameter generation highly complex.
Consequently, we focus on the conditional generation of fine-tuned LoRA (Low-Rank Adaptation) parameters in various domains as LoRA improves downstream task performance with few parameters and a relatively more stable distribution.
Specifically, LoRA  [] is a parameter-efficient fine-tuning technique that adapts pre-trained models to specific tasks by learning low-rank matrices that modify the model’s weights.

To achieve high-performance controllable conditional parameter generation, we propose Conditional Parameter Diffusion, named Cond P-Diff, which utilizes a standard latent diffusion model to perform conditional generation, synthesizing a new set of parameters tailored to specific conditions.
Specifically, we use an autoencoder and a conditional latent diffusion model to capture the distribution of network weights.
First, the autoencoder is trained on a selected set of parameters from models optimized with normal optimization methods, e.g., SGD [], on different datasets, creating latent representations of these parameters.
Second, we utilize a domain-specific condition, e.g., text, style image, projector to encode the condition information and train a conditional diffusion model to reconstruct latent representations.
Finally, as shown in Figure 1, the trained conditional latent diffusion model Cond P-Diff generates latent representations from random noise in the inference process based on specific task conditions.
Then, the decoder of the trained autoencoder processes these generated representations to produce new, high-performing model parameters.

Our method has the following characteristics:
i) It demonstrates comparable or superior performance relative to models trained with conventional methods, spanning various datasets and architectures.
ii) The parameters generated by our approach significantly differ from the parameters obtained during normal training, highlighting its capability to synthesize novel parameters rather than merely replicating the training examples.
iii) Extensive evaluations demonstarte the robustness of our approach.
Our method Cond P-Diff also shows generalizability in generated high-performance model weights space.
We hope that our findings will provide new insights into the potential of applying conditional diffusion models to parameter generation and highlight a promising direction for task-specific parameter generation of neural networks.

## 2 Preliminary

### 2.1 Preliminaries of LoRA

Low-Rank Adaptation (LoRA) [] enhances the efficiency of fine-tuning large pre-trained language models by minimizing the computational demands usually required for full model retraining.
LoRA introduces two trainable matrices, $B\in\mathbb{R}^{d\times r}$ and $A\in\mathbb{R}^{r\times k}$, to each transformer layer. These matrices, where $r$ is much smaller than hidden layer dimension $d$ and task-specific dimension $k$, perform a low-rank approximation of the typical updates made during fine-tuning.
The core idea is that the necessary adjustments for task-specific adaptation have a low "intrinsic dimension," allowing significant reductions in trainable parameters while maintaining performance.
The pretrained weight matrix $W_{\text{0}}$ remains unchanged, with only $B$ and $A$ being optimized, thus speeding up training and decreasing memory and computational needs.
The modified forward pass in LoRA is represented as:

$$ $W_{\text{0}}x+\Delta Wx=W_{\text{0}}x+B(Ax)$ (1) $$

where $\Delta W=BA$ is the update.
Initially, $B$ is zero, ensuring no changes to $W_{\text{0}}$, and $A$ starts with a small random Gaussian distribution.
In deployment, the learned low-rank matrices $B$ and $A$ can be integrated into $W_{\text{0}}$.
In this work, we aim to synthesize LoRA parameters because of the practicality and effective LoRA fusion that show the continuous distribution in LoRA parameter space.

### 2.2 Preliminaries of Conditional Diffusion Models

Conditional diffusion models [, , ] extend the standard diffusion model by incorporating conditions into both the forward and reverse processes.
This conditional information defined by $c$ allows the model to generate data tailored to specific attributes or requirements.

Conditional forward process: The forward process in conditional models involves adding noise to an initial sample while conditioning on $c$.
The probability of transitioning from $x_{t-1}$ to $x_{t}$ under condition $c$ is modeled as a Gaussian distribution:

$$ $q(x_{t}|x_{t-1},c)=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t} \mathbf{I})$ (2) $$

where $\beta_{t}$ are the timestep-dependent noise levels, and $\mathbf{I}$ represents the identity matrix.
The complete forward process conditioned on $c$ is given by:

$$ $q(x_{1:T}|x_{0},c)=\prod_{t=1}^{T}q(x_{t}|x_{t-1},c)$ (3) $$

Conditional Reverse Process:
The reverse process aims to reconstruct the original sample from its noisiest state $x_{T}$ conditioned on $c$. It is formulated by:

$$ $p_{\theta}(x_{t-1}|x_{t},c)=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},t,c),\Sigma _{\theta}(x_{t},t,c))$ (4) $$

In this process, $\mu_{\theta}$ and $\Sigma_{\theta}$ are functions estimated by a neural network, which also processes the condition $c$, ensuring that the recovery of data respects the conditional constraints.

Optimization and Inference with Conditions:
The training procedure involves minimizing the Kullback-Leibler(KL) divergence between the forward and reverse conditional distributions, specifically:

$$ $L_{dm}=\mathbb{E}_{q(x_{0},c)}\left[D_{KL}(q(x_{t-1}|x_{t},x_{0},c)\|p_{\theta }(x_{t-1}|x_{t},c))\right]$ (5) $$

During inference, the model generates new samples by conditioning on $c$ and sequentially applying the learned reverse transitions from a noise distribution, enabling the generation of data that closely adheres to the specified conditions.

## 3 Methodology

### 3.1 Overview

We propose conditional parameter generation to synthesize new parameters tailored to specific task conditions.
Fig 2 illustrates our proposed Cond P-Diff framework.
First, given a training dataset of model parameters, we use an autoencoder [] to extract latent representations of the parameters and reconstruct the latent vectors by decoder.
Then, inspired by [], we train a conditional latent diffusion model to generate high-performance parameters conditioned on specific task information.
Finally, after training, we employ Cond P-Diff by feeding random noise and task-specific conditions into a conditional parameter diffusion model to generate the desired parameters.

Figure: Figure 2: The framework of Cond P-Diff. The autoencoder is employed to extract the latent representation of LoRA parameters and reduce memory consumption. The conditional parameter diffusion model aims to synthesize high-performance parameters based on specific task conditions.
Refer to caption: x2.png

### 3.2 Parameter autoencoder

Dataset preparation.
In this work, we focus on synthesizing LoRA learnable matrix parameters of fine-tuned models by default.
To obtain the training dataset for the parameter autoencoder, we fine-tune the pre-trained model using LoRA on the dataset for task $q$ and collect $N$ different checkpoints in the last $N$ steps.
We denote the training dataset as $\Theta=[\theta_{1},\dots,\theta_{n},\dots,\theta_{N}]$, where $\theta_{k}$ represents the weights of LoRA for the model at a specific fine-tuning stage.
Because the training dataset for Cond P-Diff contains model parameters rather than conventional image or language datasets, we propose task normalization. Specifically, we employ Z-Score normalization on the parameters of each task individually [].

Training procedure. Given a training sample $\theta_{n}$, we flatten parameter matrix $\theta_{n}$ to a one-dimensional vector $w_{n}\in\mathbb{R}^{K\times 1}$, which $K$ is the total number of parameter weights of $w_{n}$.
Then, we utilize an auto-encoder to obtain meaningful and robust latent representations.
Specifically, we formulate the process as Equation 6b, where $\mathcal{E}$ and $\mathfrak{D}$ represent the encoder and decoder functions, respectively.
$z_{n}$ is the latent representation of the parameter matrix. $\hat{w_{n}}$ is the reconstruction of parameter $w_{n}$.
To enhance the generalization and robustness of the autoencoder, we introduce Gaussian noise $\mathbf{\xi_{z}}$ to the latent vector. The final auto-encoder process is formulated as follows:

$$ $z_{n}=\mathcal{E}(w_{n})=\text{Encoder}(w_{n})$ (6a) $\hat{w_{n}}=\mathfrak{D}(z_{n})=\text{Decoder}(z_{n}+\mathbf{\xi_{z}})$ (6b) $$

We train the autoencoder function by minimizing loss function below.

$$ $\mathcal{L}=\frac{1}{N}\sum_{n=1}^{N}\left\|w_{n}-\hat{w_{n}}\right\|^{2}$ (7) $$

### 3.3 Conditional parameter generation

We utilize a conditional latent diffusion model to synthesize high-performance parameters based on conditions $y$ such as text and image.
To handle different tasks and modalities, we adopt the domain-specific encoder, which is denoted as $\tau_{\text{domain}}(y;\rho)$, where $y$ represents the input condition and $\rho$ denotes the encoder parameters.
For example, in the NLP experiments of this work, we employ the text decoder in CLIP[]. Inspired by in-context learning, the input condition $y$ consists of a task description and two-shot examples to capture the task information.
Besides, we utilize stylized images as conditions in style transfer tasks and adopt ResNet [] to extract style latent representations as the condition vector. More details about the condition are shown in Appendix Conditional LoRA Parameter Generation.
Regarding the U-Net architecture, we apply one-dimensional convolutions in denoising autoencoders because the weight matrix parameters do not show strong positional relationships different from images where pixels have two-dimensional spatial relationships.

Therefore, given the condition and training parameters samples, we train the conditional latent diffusion model through

$$ $L_{LDM}:=\mathbb{E}_{\epsilon\sim\mathcal{N}(0,1),t}\left[\|\epsilon-\epsilon_ {\theta}(p_{t},t,\tau_{domain,\rho}(y))\|^{2}\right],\quad$ (8) $$

where $\epsilon_{\theta}$ is learned via Eq. 8.
Finally, after conditional diffusion model training, we feed specific conditions corresponding to tasks and random noise to reverse the inference process to obtain high-performing weights for specific tasks.

## 4 Experiment

In this section, we first show the experiment setup. Then, we present the evaluation results, ablation studies, and analysis of Cond P-Diff.

### 4.1 Experiment setup

Datasets and metrics.
We evaluate our method across various domains.
Specifically, in NLP experiments, we test on the language understanding GLUE benchmark [].
In CV experiments, we focus on the style-transfer tasks.
We use the SemArt and WikiArt datasets [, ], which contain diverse artistic images, and evaluate them using the Fréchet Inception Distance (FID, [], as employed by StyleGAN [], with lower scores indicating better performance.

Dataset collecting and training procedures.
In NLP experiments, we collect 150 training samples for models, including BERT, Roberta, GPT-2 by default.
For instance, in the case of BERT, we fixed pre-trained parameters and fine-tuned the network using LoRA.
Specifically, we conduct the hyperparameter search for fixed values of $r$ and $\alpha$ and select the fine-tuning hyperparameters that yield the best average performance.
During the fine-tuning process, we save the checkpoints of the last 150 steps as the training dataset, which includes the LoRA learnable matrix weights.
In the framework of Cond P-Diff, the autoencoder includes 1D CNN-based encoders and decoders.
We utilize the text encoder from CLIP as the condition text encoder.
In image style transfer tasks, we fine-tune attention modules of a popular text-to-image model, PixArt-$\alpha$ model [] using LoRA and collected the last 64 LoRA checkpoints of the training process once in 10 steps.
In the framework of Cond P-Diff, we used pre-trained ResNet18 to extract style latent as the condition vector.
All experiments were conducted on the Linux server with four NVIDIA A100 GPUs. The noise $\xi_{z}$ is Gaussian noise with an amplitude of 0.001 by default.
Detailed training hyperparameters for LoRA fine-tuning and Cond P-Diff framework are provided in Appendix B.

Inference procedures.
In NLP tasks, we generate 20 LoRA parameters for each task using a conditional diffusion model through random noise and merge these generated parameters into the pre-trained model.
We select the model that exhibits the best performance on the training dataset and report its performance on the validation dataset.
In style-transfer tasks, we synthesize LoRA parameters of the corresponding styles by feeding the conditional diffusion model with images in various styles as conditions.
We then merge parameters with PixArt-$\alpha$’s and utilize them to generate images using a set of prompts.
Finally, we compute the FID score of the generated images.

Baselines.
1) original: The best validation performance among the originally trained models.
2) model soup: The validation performance of the model whose weight is the average of the training dataset. Because Mitchell et al. [] shows averaging the weights of fine-tuned models with different hyperparameter configurations often improves accuracy and robustness.
In style-transfer experiments, we introduce an additional baseline no-lora: we directly employ the predefined PixArt-$\alpha$ model to demonstrate the effectiveness of LoRA fine-tuning in style-transfer tasks.

### 4.2 Experiment results

Cond P-Diff can generate high-performance parameters based on task conditions.
Table 1 presents comparison results of Cond P-Diff and baseline methods across language understanding GLUE benchmark for three models with different LoRA configurations.
We observe that Cond P-Diff consistently yields comparable performance in most scenarios, demonstrating it learns conditional parameter distributions effectively and stably.
Besides, we note that the baseline average’s performance in some cases surpasses the baseline, validating the potential of model averaging to enhance performance [].

Table 2 illustrates the results of Cond P-Diff and the baseline in the image style transfer task for different styles.
We employ the FID [] to quantitatively assess the quality of style-conditioned image generation. Lower FID represents better image generation quality.
Based on our findings, Cond P-Diff efficiently synthesizes specific style-adapted LoRA parameters to generate high-quality images.
Additional visual results are shown in Figure 3(d).
This demonstrates that Cond P-Diff can practically generate high-performance model parameters based on specific conditions.

**Table 1: Results of Cond P-Diff on GLUE. We present results in the format of ’Cond P-Diff/ orginal / model soup’. Cond P-Diff obtains comparable or even better performance than baselines. ’Size’ is the parameter size of LoRA. ’Rank’ is the parameter $r$ in LoRA. Full’ represents fully fine-tuning results.**
| Model | Rank | Size | SST2 | RTE | MRPC | COLA | QNLI | STSB | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BERT | 1 | 73728 | 91.6 / 91.6 / 90.8 | 57.4 / 58.9 / 57.9 | 87.2 / 83.4 / 83.9 | 52.4 / 52.6 / 52.1 | 88.7 / 88.7 / 88.1 | 81.8 / 81.4 / 81.7 | 76.5 / 76.1 / 75.8 |
| 2 | 147456 | 91.4 / 91.4 / 91.5 | 57.5 / 59.9 / 60.1 | 87.3 / 85.1 / 85.5 | 51.4 / 51.3 / 50.7 | 88.6 / 88.1 / 87.4 | 82.6 / 81.6 / 81.7 | 76.5 / 76.2 / 76.2 |  |
| 4 | 294912 | 91.6 / 91.9 / 92.0 | 62.7 / 63.2 / 62.8 | 85.4 / 85.4 / 85.5 | 53.7 / 53.4 / 52.5 | 89.8 / 89.6 / 88.9 | 80.6 / 80.9 / 80.7 | 77.3 / 77.4 / 77.1 |  |
| 16 | 1179648 | 92.1 / 91.6 / 91.5 | 64.2 / 64.3 / 64.5 | 87.4 / 87.0 / 86.8 | 56.9 / 57.0 / 57.5 | 89.8 / 90.1 / 90.2 | 83.8 / 83.3 / 82.3 | 79.0 / 78.9 / 78.8 |  |
| Full | 109482240 | 93.5 | 66.4 | 88.9 | 52.1 | 90.5 | 85.8 | 79.5 |  |
| RoBERTa | 1 | 73728 | 93.3 / 93.7 / 94.1 | 65.6 / 68.6 / 68.0 | 86.9 / 84.7 / 85.0 | 49.8 / 50.2 / 50.5 | 92.4 / 92.0 / 91.4 | 87.3 / 87.5 / 86.9 | 79.2 / 79.4 / 79.3 |
| 2 | 147456 | 93.5 / 93.7 / 93.8 | 63.2 / 68.2 / 68.3 | 87.7 / 85.0 / 84.6 | 50.3 / 50.7 / 50.6 | 92.8 / 92.5 / 92.2 | 86.8 / 87.3 / 87.6 | 79.0 / 79.6 / 79.5 |  |
| 4 | 294912 | 93.8 / 93.5 / 93.1 | 69.8 / 69.7 / 69.5 | 87.9 / 88.3 / 87.9 | 54.1 / 54.0 / 54.1 | 92.0 / 92.4 / 92.9 | 88.3 / 88.2 / 88.6 | 81.0 / 81.0 / 81.0 |  |
| Full | 124645632 | 94.8 | 78.7 | 90.2 | 63.6 | 92.8 | 91.2 | 85.2 |  |
| DeBERTa | 1 | 92160 | 94.4 / 94.4 / 94.7 | 61.4 / 61.0 / 61.5 | 84.0 / 84.0 / 83.2 | 56.8 / 57.0 / 56.1 | 92.4 / 92.8 / 92.1 | 87.4 / 87.8 / 87.0 | 79.4 / 79.5 / 79.1 |
| 2 | 184320 | 94.9 / 94.8 / 94.0 | 62.2 / 62.1 / 62.0 | 86.2 / 85.8 / 86.2 | 58.6 / 58.3 / 57.4 | 92.1 / 92.0 / 92.1 | 85.2 / 85.2 / 84.5 | 79.9 / 79.4 / 79.4 |  |
| 4 | 368640 | 94.6 / 94.5 / 94.7 | 63.2 / 62.8 / 61.9 | 87.1 / 86.9 / 86.2 | 60.3 / 60.3 / 59.9 | 93.4 / 93.5 / 93.1 | 88.7 / 88.7 / 88.7 | 81.2 / 81.1 / 80.7 |  |

**Table 2: FID results of image-transfer tasks. Lower FID is better. Best results are bolded.**
| Style | original | model soup | no-Lora | Cond P-Diff |
| --- | --- | --- | --- | --- |
| Van Gogh | 27.92 | 28.08 | 102.95 | 28.03 |
| Edvard | 27.10 | 27.13 | 96.18 | 26.98 |
| Chalk | 36.22 | 36.00 | 171.82 | 36.18 |
| Charcoal | 40.80 | 40.19 | 132.76 | 40.60 |
| Average | 33.01 | 32.86 | 125.93 | 32.94 |

**Table 4: Ablation studies of Cond P-Diff. We ablate the normalization methods in the training process, the condition representation, and the location of employing Cond P-Diff. The Default settings in Cond P-Diff are marked in gray. Bold entries are best results.**
| Norm. | SST2 | STSB | MRPC |
| --- | --- | --- | --- |
| no norm. | 55.67 | 49.07 | 47.01 |
| batch norm. | 90.60 | 80.90 | 82.50 |
| task norm. | 91.63 | 81.81 | 87.24 |

### 4.3 Ablation study

In this section, we conduct multiple ablation studies to report the characteristics of Cond P-Diff. We focus on the performance of generated LoRA parameters(rank $r=1$) of BERT on SST2, RTE, and MRPC datasets. The training setting is the same as experiments Table 1.

Size of the training dataset
As described in Section 3.2, we collect $N$ different checkpoints in the last $N$ steps as a training dataset for task $q$ using LoRA.
We explore the relationship between dataset size $N$ and performance in Table 3.
We observe that the performance improves as the size of the training dataset increases.
Specifically, a larger training dataset can provide a broader exploration space, thereby enabling Cond P-Diff to generate higher performance parameters.
For instance, performance on the MRPC task improved by 4.53%.

Normalization approach
As described in Section 3.2, we use task normalization method.
Table 3(a) shows the impacts of different normalization strategies on performance, including no norm., batch norm., and task norm..
Specifically, task norm. refers to normalizing the parameters corresponding to each task individually.
batch norm. represents batch normalization.
The experimental setup in Table 3(a) is consistent with that of the experiment in Table 1.
We find that task norm. consistently yields the best average performance.
no norm. leads to the worst performance because the wide variance in weight distributions across different tasks and outliers
hinders the convergence of the autoencoder.
Besides, batch norm. performed inferior to task norm., as it introduces spurious correlations among parameters across different tasks.

Condition information
The representation of the condition critically affects generation results.
We explore how to represent the task condition effectively to guide conditional parameter generation, as detailed in Table 3(b).
Our approach categorizes representations into four types: using one-shot vectors, using only the task description, using only two-shot examples, and using both the task description and two-shot examples.
Table 3(b) shows that combining the task description with examples yields better outcomes, suggesting that in-context learning can provide more information to establish relationships with the weight parameters.

Which part of parameters to synthesis
We generate LoRA parameters for all blocks by default in Table 1.
To explore the effectiveness of Cond P-Diff on different blocks, we present the performance when generating LoRA parameters for only certain blocks.
The experiments in Table 3(c) illustrate that the method is more effective when generating parameters for all blocks.
We hypothesize that as the number of synthesized parameters increases, the model has a larger exploration space, thereby boosting performance.
Conversely, performance is constrained by the exploration space and original parameters when focusing on only a subset of parameters.

Figure: (d) Visualization of images generated by Cond P-Diff parameters in style transfer tasks
Refer to caption: x3.png

Figure: (a) Visualization of the interpolation of two generated parameters in different styles.
Refer to caption: x6.png

### 4.4 Analysis

In this section, we conduct a detailed analysis of Cond P-Diff.
Specifically, we explore two critical questions: First, does Cond P-Diff merely replicate training data, or can it generate high-performance model parameters that are distinct from the originals?
Second, does the generated parameter space of Cond P-Diff have generalizability?

Cond P-Diff is not merely cloning model parameters.

Similarity vs. Performance
First, we calculate the $L_{2}$ distance between the generated and original parameters. Figure 3(f) illustrates the relationship between the similarity of the generated parameters and performance.
We observe that Cond P-Diff attains various similarities and achieves better performance compared to original fine-tuned weights across various datasets.

Parameter distribution
We employ t-SNE [] to analyze the distributions of generated parameters and original weights of fine-tuned models on datasets COLA, QNLI, and STSB, as shown in Figures 3(e).
We observe that the distribution of generated parameters by Cond P-Diff significantly differs from the original parameters.
The distribution of the original parameters can be viewed as following the trajectory of the optimization process.
In contrast, Cond P-Diff generates novel high-performance parameters by learning the distribution of parameters.
Besides, the high-performance parameters generated by Cond P-Diff are dispersed more broadly, underscoring the generative model’s potential to identify novel high-performance parameters beyond traditional optimization pathways.
Interestingly, the high-performance parameter distributions generated by Cond P-Diff for the three datasets are very similar, demonstrating the necessity of exploring the high-performance parameter space.

Trajectories of Cond P-Diff process.
Figure 4(b) visualizes the generated parameters at different time steps during the inference stage using t-SNE [] to explore the generation process in the image style-transfer tasks.
We display five trajectories initialized from five different random noises and present the model soup and the original model parameters.
The parameters derived from the model soup are located near the original parameters.
We observe that the generated parameters gradually approach the original parameters but ultimately maintain some distance from them, indicating that Cond P-Diff generates high-performance parameters that are distributed differently from the original parameters rather than directly replicating them.
The variations in the trajectories also demonstrate the robustness of Cond P-Diff.

Generalizability
We examine the generalization of the generated parameter space in the task of image style transfer.
We select parameters, $\theta_{\text{style1}}$ and $\theta_{\text{style2}}$, generated by Cond P-Diff conditioned two distinct styles, style1 and style2.
To interpolate between these styles, we compute a new set of parameters $\theta_{\text{interp}}$ as $\theta_{\text{interp}}=(1-\lambda)\theta_{\text{style1}}+\lambda\theta_{\text{
style2}}$, where $\lambda\in[0,1]$ is the interpolation factor.
Subsequently, we evaluate the effectiveness of $\theta_{\text{interp}}$ in style transfer.
Figure 4(b) illustrates the visualization of images generated by interpolated parameters between Style-1 and Style-2.
As $\lambda$ increases from left to right, the style gradually shifts towards Style-2.
The continuous style change demonstrates the generalization of the generated parameter space. We also explore the generalization of the condition space in the Appendix C

## 5 Related work

Diffusion models
Diffusion models [, , ] have recently emerged as a powerful class of generative models, enabling high-fidelity synthesis of complex data distributions.
The research on the diffusion model can be generally classified into four categories.
The first category aims to enhance image synthesis quality [, , ]
Second, researchers focus on accelerating the sampling process [, ].
Third, recent research has also focused on reevaluating diffusion models through the lens of continuous analysis like score-based generative modeling [].
Fourth, the success of diffusion models has sparked their application in various domains, [, , ]. In this work, we explore the conditional diffusion model in the parameter generation domain.

Conditional generation
Conditional generation has gained significant attention in computer vision and natural language processing.
Three prominent frameworks have emerged: conditional GANs [, , ], conditional VAEs [, ], and conditional diffusion models xw[, ], which incorporate conditions to guide the generation process, enabling the creation of visually coherent and semantically meaningful data samples.
Conditional GANs incorporate condition information into GAN to generate images conditioned on specific attributes or labels.
Conditional diffusion models take this further by generating visually coherent and semantically meaningful images from the textual description, demonstrating superior image synthesis quality compared to GANs.
Building upon the success of conditional diffusion models, we propose to extend this approach to generating neural network parameters based on specific conditions.

Parameter generation
The field of parameter generation has seen significant progress in recent years, with HyperNetworks ([] and generative models of neural network checkpoints [] emerging as promising approaches.
[] introduced HyperNetworks, which uses a hypernetwork to learn the parameters for another neural network.
[] proposes Model-Agnostic Meta-Learning, which learns an initialization for efficient fine-tuning.
[] introduce the model G.pt to predict the distribution over parameter updates given an initial input parameter vector and a prompted loss or error.
[] trained autoencoder on a model zoo to learn a hyper-representation for generative use to sample new model weights
[] use a GNN-based model to sample network parameters.
[] directly leverages MLP weights and generates neural implicit fields encoded by synthesized MLP weights.
[] uses a diffusion model to generate high-performing neural network parameters across various architectures and datasets.
Different from the previous works, we focus on conditional parameter generation to generate high-performing weights based on specific task conditions practically.

## 6 Conclusion

In this work, we proposed an approach Cond P-Diff for high-performance controllable parameter generation, specially for LoRA parameters.
We utilize an autoencoder and a conditional latent diffusion model to capture the distribution of high-performing parameters and perform conditional generation, synthesizing a new set of parameters tailored to specific conditions.
We show that our method can efficiently synthesize novel and high-quality model parameters.
The parameter distribution generated by Cond P-Diff exhibits differences compared to the distribution obtained through conventional optimization methods, indicating a certain level of generalization capability.

### 6.1 Limitation and future work

Nonetheless, it is essential to recognize that diffusion in parameter generation is still largely unexplored despite the significant advances in the realm of image and video synthesis.
In this work, we present a preliminary methodology for conditional parameter diffusion.
However, several challenges remain unresolved, including reducing memory demands for large model architectures, enhancing the generalizability of generation techniques, and improving the representation of dataset conditions.
Furthermore, integrating knowledge graphs with conditional diffusion offers promising directions for controlling conditional generation.

## Appendix A Detailed related work

Diffusion models
Diffusion models have emerged as a powerful class of generative models, enabling high-fidelity synthesis of complex data distributions.
Diffusion models are based on non-equilibrium thermodynamics, which gradually add noise to data and learn to reverse the diffusion process to generate samples. [, , ]
The research on the diffusion model can be generally classified into four categories.
The first category aims to enhance image synthesis quality, as demonstrated by notable models such as Stable Diffusion [], DALL·E 2 [], and Imagen [] by leveraging techniques like CLIP-based text encoders, latent space diffusion, and hierarchical architectures.
Second, researchers focus on accelerating the sampling process, with key developments including Denoising Diffusion Implicit Models [] and DPM-Solver []. These approaches aim to improve the computational efficiency of diffusion models through deterministic sampling, closed-form expressions, and numerical ODE solvers.
Third, recent research has also focused on reevaluating diffusion models through the lens of continuous analysis like score-based generative modeling [] in continuous-time settings.
Fourth, the success of diffusion models has sparked their application in various domains, including text-to-speech synthesis [], 3D shape generation [], and anomaly detection in medical images [], demonstrating the potential of diffusion models beyond image synthesis. In this work, we explore the conditional diffusion model in the parameter generation domain.

Conditional generation
Conditional generation has gained significant attention in machine learning, particularly in computer vision and natural language processing.
Three prominent frameworks have emerged: conditional GANs [, , ], conditional VAEs [, ], and conditional diffusion models [, ], which incorporate conditions to guide the generation process, enabling the creation of visually coherent and semantically meaningful data samples.
Conditional GANs incorporate condition information into GAN to generate images conditioned on specific attributes or labels.
Conditional diffusion models take this further by generating visually coherent and semantically meaningful images from the textual description, demonstrating superior image synthesis quality compared to GANs.
Building upon the success of conditional diffusion models, we propose to extend this approach to generating neural network parameters based on specific conditions.

Parameter generation
The field of parameter generation has seen significant progress in recent years, with HyperNetworks ([] and generative models of neural network checkpoints [] emerging as promising approaches.
[] introduced HyperNetworks, which uses a hypernetwork to learn the parameters for another neural network.
[] proposes Model-Agnostic Meta-Learning, which learns an initialization for efficient fine-tuning.
[] introduce the model G.pt to predict the distribution over parameter updates given an initial input parameter vector and a prompted loss or error.
[] trained autoencoder on a model zoo to learn a hyper-representation for generative use to sample new model weights
[] use a GNN-based model to sample network parameters.
[] directly leverages MLP weights and generates neural implicit fields encoded by synthesized MLP weights.
[] uses a diffusion model to generate high-performing neural network parameters across various architectures and datasets.
Different from the previous works, we focus on conditional parameter generation to generate high-performing weights based on specific task conditions practically.

## Appendix B Experiment setup

In this section, we show detailed experiment setups, including dataset information and training configuration.

### B.1 Style transfer experiments

In this section, we provide detailed information about the training configurations used for both the autoencoder and the diffusion model in the style transfer task.

Autoencoder configuration:
The encoder is a 1D CNN-based model where the channel of each layer is $(16,32,64,128,256,384,512,768,1024,64)$.
At the bottom layer, we flatten the parameters and map them to a latent dimension of $256$ with a linear layer.
In the decoder part, we use transposed convolutions with the same number of channels and layers to upsample back to the original shape.

The training details of hyperparameters are as follows: total number of parameters $516,096$, kernel size for CNN model $9$, learning rate $2\times 10^{-4}$ with cosine annealing, total training steps $12,000$, batch size $64$. In addition, to reduce memory usage and accelerate computations, mixed-precision is enabled with $bfloat16$ for the first $75\%$ of the training process.

Diffusion Model configuration:
The architecture of the DDPM comprises a 1D CNN-based U-Net [] with channels $(64,128,256,512,768,1024,1024,32)$. A fully connected layer is applied at the bottom of the U-Net after flattening.
In addition to the U-Net, we employ a style feature extraction network as the condition projector, consisting of two convolutional layers, an average pooling layer, and a fully connected layer.
The extracted features are added as embeddings to the bottom layer of the U-Net.
The training details of hyperparameters are as follows: kernel size for CNN model $3$, learning rate $5\times 10^{-4}$ with cosine annealing, total training steps $50,000$, batch size $128$, number of diffusion steps $1,000$, $\beta$ in the diffusion model shifted linearly from $0.0001$ to $0.02$ in diffusion models. And the same as AE training, mixed-precision is enabled with $bfloat16$ for the first $75\%$ of the training process.

Figure: Figure 5: Cond P-Diff framework in style-transfer tasks.
Refer to caption: x8.png

Framework:
This section describes the framework and workflow of the style transfer task with our conditional parameter generation in detail, as illustrated in Figure 5.

Data Preparation: The first step is selecting appropriate data, including style image and parameter data.
For style image data, we select a total of 16 groups of data with different styles.

7 groups, such as Van Gogh, Edvard, and Jacoulet, are manually selected from SemArt and WikiArt [, ] datasets, which totally includes more than 250,000 works by 3,000 artists.
The other 9 groups, such as Chalk and Charcoal, are generated by a traditional image style transfer algorithm [] to make sure the styles of images in a particular group are highly consistent.
For parameter data, we use the PixArt-$\alpha$ [] as the base model, which is a transformer-based text-to-image diffusion model with smaller parameter sizes and competitive quality. We finetuned it with the style image data.
Each set of LoRA parameters holds 64 checkpoints from the last 64 steps of one training.
Thus, we obtained 16 sets of parameter data, with 64 LoRA parameters in each set.

Training of Autoencoder and Conditional Parameter Diffusion:
We introduce details of the training process of the autoencoder and the diffusion models.
For the autoencoder, we use the parameter data to train the autoencoder to encode the LoRA parameters into a 256-dimensional latent space. Note that we did not use the style image data in this process.
For conditional diffusion model, we use style condition extractor to extract the style features of the style image data, and merge the features into the diffusion model as condition information.

Generation Process:
The generation process is divided into two steps.
First, the LoRA parameters are obtained by the conditional parameter diffusion model, and then they are merged into PixArt-$\alpha$ to obtain the style image.

Parameter Generation:
In the inference process, the diffusion model is fed with noise and an image in a particular style as conditions, and the generated latent is fed into the decoder to get completed LoRA parameters.

Image Generation: Next, merge the generated LoRA parameters to PixArt-$\alpha$.
Then, we get the PixArt-$\alpha$ finetuned with a particular style.
Then, we can feed it with a prompt to get an image whose style corresponds to our input condition.

### B.2 Language experiments

#### B.2.1 Datasets

In NLP tasks, we use GLUE benchmark [], a benchmark for evaluating natural language understanding capabilities. SST2 []: A sentiment analysis benchmark using movie review excerpts, labeled as positive or negative, to aid in sentiment understanding. RTE: A dataset for evaluating if one sentence logically entails another, testing models’ understanding of textual entailment. MRPC []: Contains sentence pairs to benchmark models’ paraphrasing and semantic equivalence capabilities. CoLA: Tests language models’ grasp of English grammar, with sentences labeled as grammatically acceptable or not. QNLI: Converts question-answer pairs into inference tasks, assessing if sentences are correct responses to questions. STSB []: A benchmark for measuring semantic similarity between sentences, rated on a scale from 0 to 5 for nuanced meaning comprehension.

#### B.2.2 LoRA configurations

In this section, we introduce the configuration of LoRA fine-tuning as presented in Table 1.
All models are fine-tuned with 20 epochs and a dropout rate of 0.1.
Mixed-precision training is enabled with FP16 to accelerate computation and reduce memory usage.
The learning rate is set to 0.0001, and a warmup ratio of 0.1 is used to gradually increase it at the beginning of the training.
Additionally, a weight decay of 0.1 is applied to regularize the model and prevent overfitting.

**Table 5: Add caption**
| Model | BERT | RoBERTa | DeBERTa |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Rank | 1 | 2 | 4 | 16 | 1 | 2 | 4 | 16 | 1 | 2 | 4 |
| alpha | 8 | 8 | 16 | 32 | 8 | 8 | 16 | 32 | 8 | 8 | 16 |

#### B.2.3 Condition

Figure: Figure 6: Visualization of the image generated by LoRA parameters, which is generated by Cond P-Diff on the test set with conditions that the model has never seen.
Refer to caption: x9.png

## Appendix C Explorations of Cond P-Diff generalizability

We consider that the generalizability of Cond P-Diff is limited by the current amount of data. If we want the model to gain generalizability, we need to sample enough LoRA parameters in the parameter space, which is difficult to achieve. Therefore, in this experiment, we first make a style-continuous dataset, which can be equivalent to sampling enough data points in a subspace to provide enough data for our model. We then trained our model on the style-continuous dataset we created to verify its generalizability.

Make a style-continuous dataset:

Since it is difficult to find style-continuous data, we use some AI-generated images to make a style-continuous parameter-image pair dataset, to verify the continuity of the parameter space and the model’s generalization ability. Here are the detailed steps:

Firstly, we train the LoRA parameters relevant to style-1 using style-1 images; train the parameters relevant to style-2 using style-2 images. Next, we use formula $\theta_{\text{interp}}=(1-\lambda)\theta_{\text{style1}}+\lambda\theta_{\text{
style2}}$ to combine LoRA parameters in different proportions to obtain 1000 LoRA parameters between style-1 and style-2 ($\lambda$ is from $\{0.000,0.001,0.002,\cdots,0.999\}$). Then we merge the 1000 LoRA parameters to PixArt-$\alpha$ in turn and randomly select some prompts to generate images in relevant style. Thus, we obtain a dataset of 1000 parameter-image pairs.

Train on the style-continuous dataset:

With the above style-continuous parameter-image pair data, we can verify continuity of the parameter space and the generalization ability of our model. The detailed training process is as follows:

First, we split the dataset into a train set and a test set. We select 500 parameter-image pairs out of the 1000 pairs as the training set, in which $\lambda$ is from $\left[0.1,0.2\right)\cup\left[0.3,0.4\right)\cup\left[0.5,0.6\right)\cup\left[
0.7,0.8\right)\cup\left[0.9,1.0\right)$, and the rest are as the test set. Next, we train Cond P-Diff on the train set according to the normal method described in Section 3, and evaluate our model on the test set.

The results are shown in Figure 6, where the input conditions are images chosen from the test set, which our model has never seen before. We find that the model can still generate images in the relevant style, which shows our model’s generalizability. In addition, we visualized the training set parameters and the parameters generated by our model in the latent space by PCA [] in Figure 7. The blue dots represent the data used for training, and the place where the blue line is disconnected is left for testing. The orange dots represent the parameters generated by Cond P-Diff, and we find that our model can fit the entire distribution instead of only parts of the train set, illustrating the generalizability of the model.

Figure: Figure 7: PCA in the latent space of the LoRA parameters of train set and generated by Cond P-Diff
Refer to caption: x10.png

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 1
presentation: 1
questions: 1. Why is conditional parameter generation important? The authors state this as a fact but the explanation (even a short one) is missing.
2. Could you explain what is the conditioning for both NLP and image tasks?
3. The related work section should be placed after introduction.
soundness: 2
strengths: 1. The presented qualitative results are on par or slightly better than baselines.
summary: The authors propose a LoRA parameter diffusion models conditioned on specific image and NLP tasks.
weaknesses: 1. The paper seems very rushed and incomplete. Figures are missing even though there are references in the text. As a consequence, no visual results are provided, nor other analysis claimed in the paper.
2. The novelty is very limited. The authors build on top of P-Diff by adding only a task conditioning mechanism.
3. There are numerous editorial errors, e.g. lack of spaces (lines 061, 222, 292), wrong formatting (eq. 6b, 7), grammatical errors (270-271), Fig vs Figure, etc.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1.	Can the authors clarify how different dataset characteristics affect the generated parameters? Including more diverse datasets and citation information would provide context and allow readers to appreciate the model’s flexibility across domains.
2.	Missing visual comparisons (such as t-SNE plots) of generated vs. real parameters hinder the interpretation of the generalization claims. Adding these would substantiate the discussion on the generative capacity of COND P-DIFF.
3.	Could the authors discuss how the condition would affect the performance? Compared with traditional training, generating LoRA weights can use only two examples as the condition to reach a similar or even stronger performance than training, which is not that easy to understand.
4.	Could the authors provide more details on the training pipeline of the model and the role of the autoencoder in terms of efficiency and performance? Whether the training of the model need multiple steps or end-to-end?
soundness: 2
strengths: 1.	Originality: The idea of using conditional diffusion models to synthesize LoRA parameters for fine-tuning is novel, providing an innovative method for model adaptation. To the best of my knowledge, this is the first paper that bridges parameter-efficient tuning methods and conditional diffusion parameter generation.
2.	Quality: The method is well-defined, with experiments that confirm the approach’s effectiveness. Comparisons with traditional fine-tuning and model-averaging techniques are included to highlight the method’s advantages. Additionally, an ablation study explores the impact of training dataset size, normalization techniques, and LoRA layers on performance.
3.	Clarity: Overall, the paper is well-structured, with clear definitions of methods and explanations of experiments. However, certain sections could benefit from clarification, such as the autoencoder design, which is not fully elaborated.
4.	Significance: The approach offers a meaningful advance in parameter generation for LoRA, which is highly valuable for real-world applications requiring adaptive, low-resource methods.
summary: This paper proposes COND P-DIFF, a new method for generating LoRA parameters for model fine-tuning. This approach employs a conditional latent diffusion model to generate LoRA parameters based on specific task conditions, using an autoencoder for efficient latent representation. COND P-DIFF is evaluated in both computer vision and natural language processing tasks, demonstrating that it can produce task-adapted parameters that achieve performance comparable to traditional optimization methods.
weaknesses: 1.	Some figures (e.g., result comparisons and t-SNE analyses of parameter distributions) are missing, limiting the reader’s ability to evaluate the quality and diversity of generated parameters visually. The appendix is also missing.
2.	While the paper mentions using several datasets (e.g., GLUE, SemArt, WikiArt), specific citations and descriptions are absent. This omission makes it challenging for readers to understand the dataset characteristics and reproducibility. Additionally, results on these datasets are only summarized without detailed visualizations, which would improve interpretability.
3.	Although an autoencoder is used to reduce the dimensionality of LoRA parameters, training it adds complexity to the pipeline compared to traditional optimization. More information on the efficiency gains it provides or alternative dimensionality reduction techniques would strengthen this approach’s feasibility.
4.	Some details in the paper need to be revised. For example, in Table 1 the quotation marks are not in the correct format. In Table 2 the method with lowest average FID is not marked correctly.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. The authors should discuss and elaborate on the cost of training the diffusion model and the autoencoder. How long does it take to train both these models?
2. In L249-250, it is mentioned that “we collect 150 training samples for models, including BERT, Roberta, GPT-2 by default”. I assume that this is 150 models for each task. Since a single conditional diffusion model is trained across tasks, there are a total of 900 checkpoints in the dataset (150 * 6 tasks) for training the diffusion model. If this is true, have the authors evaluated the performance on a OOD task (a task that was not present in the training dataset) ?
soundness: 2
strengths: The idea of generating LoRA parameters with a diffusion model is quite interesting. Overall, the paper is well written and clearly explained. The experiments and results section is comprehensive with results on various tasks.
summary: The paper introduces "COND P-DIFF," a novel framework for generating task-specific model parameters using parameter-space conditional latent diffusion models, particularly the LoRA (Low-Rank Adaptation) parameters. The authors first train an autoencoder to encode LoRA parameters into a latent space. The diffusion model is conditioned based on a domain specific encoder. For instance, CLIP Text Decoder with description of task and examples in case of NLP tasks and ResNet features of an image in case of visual domain. The authors show results on a wide range of NLP tasks and image stylization task where COND P-DIFF outperforms the original model.
weaknesses: 1. One major concern is the absence of the corresponding figures in Section 4.4 (Analysis). It is not possible to make any inferences about the experiments without the referenced figures. And I think Section 4.4 is important to understand whether the diffusion model is learning anything new compared to the original LoRA weights.
2. One important baseline that would be useful in this setup is to use randomly select one of the training checkpoint and analyse it for a new test image (in case of style transfer). Similar thing can be done in the case of NLP tasks, where LoRA checkpoint for one task can be used for the other task. Ideally, one should observe a drop in performance when this happens. 
3. One of the biggest motivations for learning a generative model in the parameter space was ensembles (as discussed in [1]). In this case of LoRA parameter, the authors generate 20 LoRA parameters from the diffusion model and then select the best one. The authors should explore other ways of using the parameters potentially. 
4. The writing of this paper needs to improve massively. 

- L269-270 sentence is unclear. I guess the authors mean to take the average of weights of various finetuned models (LoRA).
- In Section 4.4: Analysis, none of the referenced figures are present in the paper.
- L420 has `xw`  for some reason.

[1] Wang, Kai, et al. "Neural network diffusion." arXiv preprint arXiv:2402.13144 (2024).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please see the weaknesses.
soundness: 2
strengths: 1. The authors propose a new approach using a latent diffusion model to synthesize LoRA weights for pretrained models.

2. The authors gather a dataset of LoRA weights tailored to specific pretrained models, using it to train the latent diffusion model.

3. Experiments on both language and image tasks show that this method can synthesizes LoRA weights for pretrained models, achieving good performance.
summary: To generate task-specific parameters for pretrained models, this paper proposes using a latent diffusion model to synthesize LoRA weights. Specifically, the authors collect LoRA weights for adapting pretrained models to various tasks and introduce a task-norm process to standardize the data. Following stable diffusion training strategies, they then train the model to synthesize task-specific parameters based on text or image inputs. Experiments demonstrate that the method can generates task-specific data and achieves good performance.
weaknesses: 1. The dataset size is small, which risks the model memorizing the data. While the authors show that the synthesized parameters differ from the training data, the diffusion process may only add Gaussian noise to the parameters training dataset. Additionally, the authors' use of averaging synthesized weights for specific tasks may eliminate Gaussian noise effects in diffusion process. They should present performance results without averaging.

2. The claim that the proposed method generalizes well is overstated. The authors demonstrate generalizability using images used for collecting the parameter training dataset, which is unconvincing. Testing with new images not used for collecting the training set would strengthen the claim.

3. Related work is covered in both the main paper and the supplementary material, which could be combined. Additionally, some figure references are missing in the main paper.

4. The style transfer results are weak.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: * The abstract could include how the base network is employed during the entire process, as the LoRA params are supposed to be tied to the base network.

* L021 - L023: It could be specified more directly that the differences between the params (COND P-DIFF vs. regular) are tied to generalizability. Or maybe the sentence could be rewritten a bit to denote the kind of differences.

* L054 - L055: Could it be that the scalability approach wasn't studied in those respective works because it was relatively new back then? Or is it that the methods themselves aren't particularly suitable for scaling? 

* L058: "Simple architectures" such as? 

* L060 - L061: The sentence could be expanded a little bit to specify "... do not support task-conditioned parameter generation"? Also, there seems to be a minor formatting issue in the citation (Wang et al. (2024) should go into brackets?) This applies elsewhere, too, throughout the text.

* L088 - L092: Figure 2 could be cited here, stating that a pictorial representation of the methodology is available.

* L092: The authors mention that the diffusion model reconstructs the latent representations. But shouldn't the representations then go to the decoder of the previously trained Autoencoder to obtain the final parameter-space representations? I see that Figure 2 already specifies it and the associated text. So, perhaps the text in this section could include it, too?

* L096 - L097: One/two examples of tasks/datasets could be provided beside the point where the authors claim that COND P-DIFF achieves superior performance. 

* L100: I think the third point is a combination of the earlier two points and could be avoided. Instead, the authors could consider including examples of tasks/datasets they considered in the work and the methods used for comparing the parameter representations. 

* Equation 5 could include the approximation typically followed in the diffusion world, i.e., the "epsilon-prediction" objective [1]. 

* L162: Are these base model parameters or existing LoRA parameters? The Figure 2 caption states that it's the LoRA parameters, but I think the main text also should. 

* In the second block of Figure 2, $z_T$ could directly go to the "Denoising Net" block? Otherwise, the diffusion process seems a little disconnected from the denoising net. 

* I understand that the dataset preparation methodology allows for more controlled experiments, but I think a good value addition of COND P-DIFF could be leveraging existing task-specific LoRA parameters made available by the community (available through the Hugging Face Hub). I can also imagine that some LoRA parameters might be corrupted. So, to mitigate the risk, only popular ones could be considered. I encourage the authors to explore this avenue.

* L199 - L200: If possible, the Z-score normalization could be expanded in the context COND P-DIFF. 

* L196: Could the dimensions of each $\theta_n$ be specified too? 

* For the autoencoder:
  * How much does the inclusion of Gaussian noise help?
  * Did the authors try out a VAE in this setup, as that is what's more commonly used in the latent diffusion literature? 

* L222: There is no decoder involved in CLIP. 

* L223: How much does the choice of the number of examples in the in-context learning matter for the NLP tasks? 

* The CLIP image encoder could be leveraged to extract the style representations or general image-level representations. 

* I request the authors to consider more practical tasks for computer vision, such as image classification, which allows 
us to study how many images we need to achieve a reasonable performance with COND P-DIFF. For many image-level tasks (such as classification), we need more training images than we need for image stylization.


* L262: Why generate 20 LoRA parameters? Is this number sufficiently ablated? Does it incur significant latency? 

* L264: The text says that the parameters are merged into the base model. So, I think what the process follows is that for each 20 generated LoRA parameters, it's first merged into the base model and, the final model is then evaluated. 

* L289: Figure citation is wrong. This applies elsewhere too. 

* L361: "all blocks" should be specified. Did the authors target only the KV layers in the attention modules (as done in the original LoRA paper) or something else?

* For comparing the LoRA parameter representations, a framework like CKA could be used as it is more robust and resilient compared to L2. 

* Do the LoRA parameters obtained through COND P-DIFF show similar behavior during inference when their scales vary (the LoRA scale helps to control how much the LoRA params should contribute)?

**References** 

[1] Denoising Diffusion Probabilistic Models; Ho et al.; 2020.
soundness: 3
strengths: * The paper goes beyond shallow architectures, unlike existing works in the area, and presents results with network architectures that are widely used. Moreover, it uses pre-trained networks and studies parameter generation for LoRAs, which makes it practically compelling. 
* The overall methodology is simple, and the components involved have been studied well in the paper and in the literature. 
* Studies two different modalities (language and images) that make it appealing.
summary: The paper proposes a method to generate LoRA parameters using a conditional latent-diffusion process. It studies the setup for language understanding and image stylization tasks. COND P-DIFF shows performance that is either comparable to the performance of the LoRA params obtained through conventional optimization process or sometimes outperforms them.
weaknesses: * Even though the paper studies two modalities, the number of tasks studied seems limited as far as practicality is concerned, which is a central theme of this work. For the image domain, it studies only image stylization but misses out on other potentially more practical tasks that may require more samples as conditions (image classification, for example). 
* Comparisons between LoRA fine-tuning time and LoRA parameter generation time (including the COND P-DIFF training time) seem to be missing. So, perhaps a time vs. performance trade-off would help make the argument of caring more about approaches like COND P-DIFF stronger. 
* Authors flatten the low-rank weight matrices to construct their datasets. For very large parameter spaces, this can quickly become memory-expensive.