Scaling Up Parameter Generation: A Recurrent Diffusion Approach
Authors: Kai Wang
Dongwen Tang
Wangbo Zhao
Konstantin Schürholt
Zhangyang Wang
Yang You
Venue: NeurIPS 2025 poster ()
Keywords: Parameter Generation
Primary area: general_machine_learning

Abstract: Parameter generation has long struggled to match the scale of today's large vision and language models, curbing its broader utility. In this paper, we introduce Recurrent Diffusion for Large-Scale Parameter Generation (RPG), a novel framework that generates full neural network parameters—up to hundreds of millions—on a single GPU. Our approach first partitions a network's parameters into non-overlapping 'tokens', each corresponding to a distinct portion of the model. A recurrent mechanism then learns the inter-token relationships, producing 'prototypes' which serve as conditions for a diffusion process that ultimately synthesizes the full parameters. Across a spectrum of architectures and tasks—including ResNets, ConvNeXts and ViTs on ImageNet-1K and COCO, and even LoRA-based LLMs—RPG achieves performance on par with fully trained networks while avoiding excessive memory overhead. Notably, it generalizes beyond its training set to generate valid parameters for previously unseen tasks, highlighting its flexibility in dynamic and open-ended scenarios. By overcoming the longstanding memory and scalability barriers, RPG serves as a critical advance in 'AI generating AI', potentially enabling efficient weight generation at scales previously deemed infeasible.

Scaling Up Parameter Generation: A Recurrent Diffusion Approach
Kai Wang"
Dongwen Tang* National University of Singapore mtdovent@gmail. com
National University of Singapore kai www.gann1n2ggmaaaaaaaaaa com
Wangbo Zhao National University of Singapore wangbo. zhwo968gmai..commmmmmmmmmm com
Konstantin Schürholt University of St. Gallen
konstantin. Donsttanti..schuehoolteee ch
Zhangyang Wang University of Texas Austin atlasvangdutexas. edu
Yang You National University of Singapore chinayouyang19911@maa com
Abstract
Parameter generation has long struggled to match the scale of today's large vi- sion and language models. curbing its broader utility. In this paper, we intro- duce Recurrent Diffusion for Large-Scale Parameter Generation (RPG), a novel framework that generates full neural network parameters- up to hundreds of millions-on on a single GPU. Our approach first partitions network's 's parameters into non-overlapping 'tokens', each corresponding to distinct portion of the model. A recurrent mechanism then learns the inter-token relationships, producing pprototypes' which serve as conditions for diffusion process that ultimately syn- thesizes the parameters. Across spectrum of architectures and tasks-including ResNets, ConvNeXts and VITs on ImageNet- K and COCO, and even LoR A-bsed L..M--RPG achieves performance on par with fully trained networks while avoid- ing excessive memory overhead. Notably, it generalizes beyond its training set generate valid parameters for previously unseen tasks, highlighting its flexibility in open-ended scenarios. By overcoming the longstanding memory and scalabil- ity barriers, RPG serves as critical advance in 'Al generating AT', potentially enabling efficient weight generation scales previously deemed infeasible.
Introduction
Scaling up neural networks has been one of the most crucial factors driving the progress of deep learning, enabling remarkable performance across wide range of tasks [34, 25, 45]. However, neural network parameter generation-frot
early HyperNetworks [24] to more recent diffusion- methods [50, 68, 63]-has not kept pace with the expansion of mainstream vision and language models. Indeed, as illustrated in Fig. 1, there is an astonishing scale gap of at least 10 between the parameter counts of widely used models (e.g., ViT. ConvNeXt. LLaMA) and the largest parameter generators available today. This discrepancy not only underscores the challenge of scalability but also confines existing generators academic demonstrations rather than real-world applicability.
"equal contribution, ccorresponding author Code: http://ghhtbb.mmAASHHHCAAAAAAAL. Sat.reeuuurrneeeaaraaeerrraaaaaa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Traditional approaches such as Hyper- Networks struggle handle even mod- erately sized models, primarily due to
#param gonarated #param (M)
10"
language
GPT-3
LLaM43
vision models
parameter generation models
GPT-2
VTT-22B
memory overheads and optimization complexity. While recent diffusion- based works attempt to break the mem- ory bottleneck via partiae-parameter or two-stage generation [68, 63], such methods may overlook key correlations among different parameter subsets.
GPT-1
BERT-L
10"
ELMo
ResNet- ResNeXI-101
viT-B
ConvNeXi-L
SANE
GoogleNat
P-diff
P-diff
HyperNet
SEDE
G.pt
10
partial roadmap
Figure 1: Roadmap of vision, language, and parameter generators. Parameter number in vision or language models is 10 times larger than that of generated parameters.
To illustrate the importance of captur- ing inter-part correlations, we conduct simple experiment: we train two ViT- Tiny [155] models (A and B) on CIFAR-10 [33], then exchange half of their layer parameters. Despite both models performing well originally (i.e., 90.4% for model A and 89.6% for model B), the hybrid model's s accuracy drops dramatically to 45.8% (as seen below).
model
ViT-Tiny A partial A + partial
ViT-Tiny
accuracy (%)
90.4
45.8
89.6
This striking performance gap highlights that adequately modeling parameter correlations is not trivial detail: robust generator must therefore explicitly model such relationships while also surmounting the memory pitfalls of large-scale processing.
Our contributions. We present Recurrent Diffusion for Large-Scale Parameter Generation (RPG), novel, end-to-end framework capable of generating full neural network parameters with up to hundreds of millions of weights using just a single commodity GPU. To the best of our knowledge, RPG is the first method to efficiently synthesize the parameters of models like ConvNeXt-L and LLaMA LLaMA-LORA--- LoRA-with up to ~200 weights-ii in a single pass. This significant leap is achieved by two core technical innovations:
1. Parameter processing for large-scale tokenization. We devise highly customized parameter tokenization strategy designed to preserve both the layer-wise distribution and the cross-layer correlations. Specifically, we (i) split weights according their layer indices, (ii) apply layer-wise normalization to mitigate distribution shifts, (iii) slice parameters into non-overlapping tokens with uniform size, and (iv) introduce a lightweight permutation state to alleviate symmetry issues when collecting multiple checkpoints. Additionally, we employ 2D position embeddings (layer index token index) to ensure the network retains positional awareness of each token within the entire set. This tailored pre- pre-processing pipeline is crucial for learning stable and high-quality parameter representations.
2. Recurrent diffusion modeling. Unlike prior diffusion- generators that flatten or divide all parameters into single sequence chunks, we propose recurrent mechanism to learn the relationships among tokens. Concretely, we map each token into hidden space via recurrent model, whose outputs serve prototypes. These prototypes then condition a ID diffusion process that progressively denoises the parameter tokens. By recurrently modeling global inter-token interactions and leveraging diffusion sampler for synthesis, RPG aligns well with the dual goals of capturing token-to-toon to-kken dependencies and remaining memory-effccient.
Key results and impact. Extensive evaluation Image- Net classification, ADE20K segmentation, COCO detection, and commonsense reasoning show that RPG consistently produces weights on par with trained models. Notably:
Single-GPU feasibility. Our recurrent design and parameter-token strategy enable inference single commodity GPU beyond 100M weights.
Generalization to unseen tasks. RPG can generate nigh-perfomming neural network parameters for unseen binary classification tasks on CIFAR- 10, providing evidence for its broader generalization capabilities.
Architectural versatility. Our method supports diverse families such as ResNet, Vision Transformer, ConvNeXt, and LoRA-based language models, making it unified framework for parameter generation.

---

parameter processing
recurrent diffusion
parameter layers
predicted token
predicted token
layert
layer2
layer3
diffusion process
diffusion process
parameter tokens
c~N((.,)) r)
c (0.
1
2
5
8
9
recurrent model
recurrent model
padding
position embeddings
2
5
1
S
S
permutation state
Illustration of parameter processing (left) and inference recurrent diffusion (right). The recurrent model integrates permutation states and position embeddings, generating prototypes that condition the diffusion model to synthesize the full parameters.
Figure 2:
These results place us significantly closer to the vision of 'Al creating Al', where a single generative model can synthesize entire networks tailored for different tasks. We believe RPG's strong scalability, lightweight memory footprint, and ability to accurately capture global parameter relationships may open new frontiers fields like rapid architecture search, efficient model adaptation, and beyond.
2 Method: Large-Scale Parameter Generation
In this section, we describe our proposed Recurrent Diffusion for Large-Scale Parameter Generation (RPG), which synthesizes full neural network parameters under strict memory constraints. As illustrated Fig. 2, our approach comprises two primary components. parameter processing and recurrent diffusion.
Parameter processing (Sec. 2.2) equips the model with specialized tokenization pipeline, layer- wise normalization, permutation states, and position embeddings- all meticulously designed to mitigate issues like distribution shifts, neural symmetry, and the sheer size of modern network weights. Recurrent diff'usion (Sec. 2.3) then learns these 'tokens' via recurrent model whose outputs condition a ID diffusion process, enabling us to capture global parameter correlations while maintaining feasibility single GPU.
2.1 Overview
Our pipeline for large-scale parameter generation is shown conceptually the right of Fig. Given a set of trained parameters {W} (e.g., from various checkpoints or model variants), we first transform them into parameter tokens, augment them with permutation state, and apply position embeddings. This processed sequence is fed into a recurrent model, which outputs per-token 'prototypes' Finally, diffusion model uses these prototypes, along with random noise, synthesize the entire parameter vector. In practice, we can sample from this generative model to produce diverse family of high-performing weights.
2.2 Parameter Processing
Parameter tokenization. Drawing inspiration from the success of language and vision transform- ers [67, 15], we divide a network's parameters into a sequence of non-overlapping tokens. However, direct flattening or naive partitioning often fails preserve crucial layer-wise statisties and dermines global consistency. To circumvent this, we first segregate parameters by layer and then normalize each layer to reduce distribution shifts:
W
divide by layer
[w,, [I], wn[[]]]
[vón [1],..... ". [1]]
N
M
mean mean(w..1)) (w.li)) 0z
std(w,
Where wx, [X] and wn denote the original and normalized parameter blocks of the i-th layer of the n-th checkpoint in the training set, respectively. The normalization is performed using layer-wise

---

statistics A; and o; which represent the mean and standard deviation of the weights in layer i, averaged across the entire training set. These statistics are previously computed and stored and later employed during the denormalization process to recover the generated parameters.
Next, each normalized layer vi(i] is tokenized into contiguous chunks of uniform size (plus padding if necessary):
tokenize
[[]] K........... ..... padding(k:))
Importantly, the padded regions are excluded from the loss and do not affect gradient updates. By enforcing a consistent token size across layers, we facilitate batch processing of parameter tokens critical step for large-scale training.
Permutation state. When aggregating checkpoints from various training runs, neural symmetry [35] can become a major bottleneck. Essentially, distinct permutations of layer weights may yield identical final performance but appear different generative model. To resolve this ambiguity, we introduce a distinct permutation state Seencdde as a one-hot vector-for each checkpoint W. This extra condition guides the generative model treat otherwise symmetrical weight configurations as unique exemplars, significantly stabilizing distribution learning.
Specifically, in the training phase, the permutation state (i.e., one-hot vector) is first processed by an MLP and then added to the model input, thereby explicitly representing the checkpoint's symmetric state. In the inference phase, we typically only care about the model's performance rather than the model's 's symmetric state. In this case, we can simply sample permutation state at random. Alternatively, we specify particular permutation state to generate models under specific symmetric state.
Position embedding. To preserve token locality and inter-layer relationships, we augment each token with 2D sinusoidal position embedding [15]. Specifically, for the j-th token of layer i, we assign
K[1] pos. embedding
cld| = [c!
The first dimension encodes the layer index, while the second dimension captures the token's 's in-layer position. As result, the generative model knows not only how tokens are grouped by layer, but also each token's 's placement within that layer.
2.3 Recurrent Diffusion
Recurrent model. After obtaining the parameter tokens [[i], permutation states S, and position embeddings e(i), we feed them into a recurrent network f(() that learns token-wise representations while capturing cross-token dependencies. We denote the output as a prototype P.:
P/ - ed.
(5)
where Hy is the internal hidden state for the j-th token in layer In practice we use Mamba [23] (a memory-efficient transformer alternative) followed by an MLP that projects features to the required dimension for our diffusion model. We find that a securrece-bbsee design elegantly handles long sequences of tokens, offering strong balance between model capacity and memory usage. Later in experiments, we compare different recurrent architectures such as LSTT [29] and causal-decoder transformers [67].
Parameter diffusion. Finally, we leverage diffusion model to generate the actual parameter values. Drawing on p-diff [68] and MAR [36], we construct a ID convolutional network that accepts random noise together with the prototypes P. Formally, at diffusion time step t, we aim to predict the noise e in denoising objective:
Lan Fr, K. co (Ke, 1)
where cp(-) is our denoising network with learnable parameters 6, and K₁ is the noisy version of K at step t. Critically, the recurrent prototypes P act as conditional inputs, enforcing global coherence among tokens. Gradients from this diffusion loss flow back into the recurrent model, implicitly learning token correlations as well.
Inference. Once training complete, the pipeline supports two inference modes. Basically, in conditional generation, we feed the permutation state S of an existing checkpoint to replicate or

---

perturb known parameter configurations. Further more, we can sample entirely unseen task states to produce novel parameter draws (Sec. 4). In both cases, the recurrent model outputs prototypes, and the diffusion model denoises random noise into a final parameter set that can scale up to hundreds of millions of weights.
Overall, recurrent diffusion elegantly decouples the complex problem of global parameter cor- relation into (i) recurrent pass that focuses on cross-token interactions, and (ii) diffusion pass that refines each token. This two-step design is precisely what allows RPG to handle large-scale architectures on standard hardware.
Main Experiments
3.1 Setup
Datasets and architectures. We evaluate our method across various tasks, including ImageNet- IK [13] for theclassification, ADE20K [77] for the semantic segmentation, COCO [40] for the object detection, and BoolQ [10]. PIQA [5]. SIQA [54], HellaSwag [75], and ARC [11] for the commonsense reasoning tasks. To verify the scalability, we conduct experiments on various architectures with parameter counts ranging from several to hundred million.
Trained parameters collection. We take parameters collection on the ImageNet- 1K as an example. To save the cost, we finetune the full parameters of the models in timm and save 50 checkpoints as the training data. (More details are Appendix C.6) For each checkpoint, we assign a unique permutation state to guide the generated parameters.
Preprocessing and training details. The length of parameter tokens, permutation states, position embeddings, and prototypes is set 8192. worth noting that the permutation states and position embeddings are fixed during the training. We also study the influence of the token length, varying it from 1024 16384. The parameter diffusion consists of ID convolutional layers. We default using Mamba [23] as the architecture of the recurrent model. More details can be found in Appendix B. Inference details. We input permutation states (relevant experimental results are in Appendix C.I) and position embeddings into the recurrent model to generate the prototypes. Then, the diffusion model utilizes the prototypes conditions, along with random noises, to synthesize the entire network parameters. We repeat the above process 10 times and report the best, average, and minimum.
3.2 Results of Large-Scale Parameter Generation
In this section, we present the results of our approach across range of tasks including classification, semantic segmentation, object detection & instance segmentation, and language tasks. As most previous works encounter the out-of-memoo issue at million-scale parameter generation, we mainly compare with the results from the trained networks, which we denote as 'original'.
On ImageNet-1K. Tab. presents performance comparisons across seven architectures on ImageNet- IK. The parameter number of these architectures ranges from to 197 million. Several observations can be made as follows: i) Our approach successfully generates model parameters at hundred-million scales, overcoming the out-of issues faced by previous works [50, 68]. ii) The performances of the generated models are comparable with the original ones.
arch. \ acc. (%)
ResNet-18
50
ViT-Tiny ViT-Small ViT-Base
ConvNeXt-A ConvNeXt-L
parameters (M)
11.7
25.6
5.7
22.1
86.6
3.7
197.8
original
70.0
79.8
74.9
81.4
84.4
75.2
85.8
best
69.9
79.6
75.4
80.6
84.6
74.6
85.8
average
69.5
79.5
75.3
80.5
84.4
74.4
85.5
minimum
69.0
79.4
75.2
80.1
84.2
74.2
85.2
Table We compare with the results of original models across seven architectures on the ImageNet- IK. Our approach successfully generates the entire model parameters that perform comparable results with the original models.
On ADE20K and COCO. For semantic segmentation, following [76], we adopt UperNet [71] as the segmentation model and train it on ADE20K [77] to prepare checkpoints. For object detection and instance segmentation, we finetune ViTDet [37] on COCO [40] to collect checkpoints and report
5

---

the results of mAP Bbox and mAP Seg.
ADE20K
COCO
method
respectively. All experiments here are conducted based on ViT-Base [15]. Tab. 2 presents the strong generalization of RPG to these two tasks. Specifically, compared to the original models, we achieve com- parable or even slightly better results over all the above metrics.
mloU(%)
ma) mAP Bbox (%)
mAP Seg (%)
original
47.6
58.3
43.6
39.0
ours
47.1
57.5
44.5
39.6
Table 2: Accuracy comparison of original and generated parameters on ADE20K (176.5M) and COCO (110.9M). All models are built based on ViT-Base.
On commonsense reasoning. We employ DoRA [.1], an upgraded version of LoRA [30], to fine- tune LLaMA-7B [65] for commonsense reasoning tasks and save the checkpoints as the training data. We report the results across 7 sub-tasks with rank = 4 and 64 in Tab. 3. The generated models
consistently yield results comparable to those of the original ones.
BoolQ
PIQA
SIQA
HellaSwag
ARC-e
ARC-c
OBQA
rank params. (M)
org
RPG
org
RPG
org
RPG
org RPG
org
RPG
org
RPG
org
RPG
7.8
64.3
63.1
71.3
72.0
66.0
67.5
53.7
56.7
64.4
65.3
49.5
49.7
63.1
66.0
64
113.1
69.3
69.1
78.9
79.4
72.9
73.9
81.1
81.1
72.9
73.1
58.1
58.3
71.2 72.1
Table 3: Accuracy comparisons of original and generated DoRA with varying ranks for LLaMA-7B on the commonsense reasoning tasks. Our approach can generate comparable or even better results than original models. Bold entries are best results.
3.3 Ablation Studies, Analysis, and Comparison
We present the results of the generated ViT-Tiny on the ImageNet-1K, unless stated otherwise. The effect of recurrent model. We employ the recurrent model to learn the relationship among parameter tokens. To keep other factors consistent, we simply remove the state transition function from the recurrent model for comparison, denoted as "- recurrent model'. The experimental results from Tab. 4a confirm that the recurrent model plays crucial role in parameter generation. Without the state transition function, our approach learns each parameter token individually, overlooking the relationships among these tokens. As a result, the generated parameters perform extremely poorly. The manner of position embeddings. In ViT, the position embeddings are learnable by default. Here, we mainly conduct the experiments with three different position embedding manners and show the details as follows:
learnable: Initializing with 2D sinusoidal positional encoding and set be learnable. encoded by index:
Using 1D sinusoidal positional encoding, irrespective of the original network structure, with indices assigned from front to back.
encoded by layer: Using 2D sinusoidal positional encoding to represent layer and token indices.
As shown in Tab. 4b, the learnable embeddings perform slightly better than the other two manners. However, we still recommend using fixed position embeddings, as they offer comparable performance while significantly reducing storage requirements compared to the learnable one.
The manner of tokenization. Considering the differences among various layers, we divide the parameters into tokens within each layer. P-diff [68] flattens the parameters into ID vectors, while SANE [57] divides the parameters by channel within each layer. We compare the results of these 3 strategies in Tab. 4c. Our default strategy achieves better results than the others. Flattening results in single token containing parameters from different layers, which poses challenges for optimization. Tokenizing by the channel may result in excessive padding values for each token, as the channel number is usually much smaller than the token size.
The impact of token size. Token size directly affects model capacity, as larger tokens can encode richer information. In Tab. 5, we compare the performance of models generated by RPG with token sizes ranging from 1024 to 16384. Model performance generally improves with larger token sizes, since smaller tokens provide insufficient information for effective learning; however, excessively large tokens lead to significant memory overhead, detailed in Fig. 8a in the Appendix.
Why not auto-regression? We adopt standard recurrent paradigm with Mamba, rather than an auto- regressive approach. This means the generated parameter tokens are not used as inputs for subsequent token predictions. Our method takes the position embedding and permutation state inputs and synthesizes neural network parameters as outputs, forming a standard recurrent neural network. To investigate the impact of different sequential modeling approaches, we compare auto-regressive and

---

best
avg. min.
pos. emb.
best
avg.
min.
tknz.
best
avg.
min.
time (h)
original
75.2 74.9 74.7
learnable
75.5
75.4
75.3
flatten
75.3 75.2 74.8
6.2
recurrent model fail
fail fail
by index
75.4
75.3
75.0
channel
75.3
75.1
75.0
14.2
recurrent model 75.4 75.3 75.2
by layer
75.4
75.3
75.2
layer
75.4
75.3
75.2
6.2
(a) Recurrent model can largely im- prove the performance and stability.
(b) Learnable performs better, but saving cost needs to be considered.
(c) Our tokenization achieves the best trade-off between acc. and time.
Table 4: Ablation experiments of the recurrent model, position embeddings, and tokenization with ViT-Tiny on ImageNet-1K. Defaults are marked in gray. 'fail' indicates models performing vandom-chanee level. Bold entries are best results.
token size
arch. Vace. (%)
1024
2048
4096
8192
16384
method
best avg. min.
ViT-Tiny
0.3
70.8
75.2
75.3
69.3
original
75.2 74.9 74.7
ViT-Small
0.1
0.7
80.5
80.5
80.4
AR (transformer decoder-only) fail fail fail
ViT-Base
0.1
0.1
0.2
45.3
84.4
recurrent (transformer encoder-only) 75.0 74.8 74.6
Table 5: Accuracy of generated models with the different toke sizes. Large tokens perform better on large models. Bold entries are best results.
Table 6: Auto-regressive (AR) fails to generate meaningful results due to accumulated errors. More details are provided in Appendix C.9.
structure
best avg. min.
time (h) memory (GB)
scales params.
RPG-T RPG-S
RPG-B RPG-L
L.TTM
75.5 75.2 74.4
16.1
38.1
generated model 0005
~0
~00
200
Trans.
75.0 74.8
74.6
4.2
29.1
recurentt part
1.3
256
1018
3076
Mamba
75.4 75.3
75.2
4.1
27.8
denoising part
0.3
17
69
273
Table 7: We study the characteristics of three recurrent structures. Defaults are marked in gray. Bold entries are best results.
Table RPG components and generated param- eter counts across scales. '~'' denotes 'approxi- mate'. All values are in millions.
recurrent formats. For fair comparison, format (using an encoder-only transformer) and an auto-regressive format (AR, using a decoder-only transformer). As shown in Tab. 6, the AR approach performs poorly due error accumulation during the inference stage. More detailed analysis can be found in Appendix C.9.
transformer architectures in both cases: recurrent
The structure of recurrent model. We mainly explore three structures of the recurrent model, including LSTM [29]. Transformer [67], and Mamba [23]. In Tab. 7, we report the performances, training time, and memory cost of generating ViT-Tiny parameters on ImageNet- All three structures achieve good results. However, considering the training time and memory cost, our default Mamba is the best structure of the recurrent model.
RPG parameters vs. generated parameters. We analyze the number of parameters of RPG's recurrent and denoising components, and the number of parameters they can generate. We design four RPG variants to generate models different sizes, as summarized in Tab. (See Tab. 16 the Appendix for architectural details.) Overall, the recurrent model contains most of the parameters, while the denoising network is kept small to speed up generation, as it is applied repeatedly. This design enables RPG to achieve high efficiency while generating large-scale parameters. Moreover, once trained, RPG can produce a virtually unlimited number of distinct models, offering substantial long-term efficiency benefits.
Efficiency of generating large-scale parameters. Rapid synthesis of large- scale parameters is crucial for evalu- ating the practicality of our approach. As illustrated in Tab. 9, we present the time cost for generating models of ViT-Base and ConvNeettL across various DDIM [61] sampling steps. All results are obtained with a sin- gle NVIDIA H100 80G GPU. Our ap- proach shows the capability to generate models within minutes. Notably, even for Con (197.7 M parameters), we can synthesize the entire parameter within 1.3 minutes. Even with only 20 sampling steps, we can achieve promising results. Meanwhile, the inference memory requirement approximately 20GB, RPG can be deployed on NVIDIA RTX 3090 similar-level GPUs.
model / memory
ViT-Base / 20.7GB
ConvNeXt-L.
/2,6GG
diffusion steps
20
60
100
200
20
60
100
200
time (minute)
0.6
0.8
1.1
1.8
0.7
1.3
2.0
3.5
accuracy (%)
83.3
84.4 84.4 84.3
85.0
85.5
85.3
85.3
Table 9: GPU memory and inference time comparisons among diffusion steps. RPG can efficiently generate the entire Le (197.8M parameters) in minutes with only ~20GB of GPU memory.
7

---

method
CNN (S)
CNN (m)
R-18
ViT-B
75
params, (M)
0.003
0.011
11.7
86.6
eccureay (%)
SKDED
26.9 46.1
OOM
OOM
74
p-diff'
48.8 49.0
61.9 62.1
OOM
OOM
nobe-0.01
ro.se-0.05
SANE
57.9 57.2
68.6 85.5
noisen0.cg
original
nosse-0.00 .3
generaled
D2NWG
38.2 44.7
58.8 57.2
94.6 94.6
72
noisem0.04
interpolated
RPG
49.0 49.0 62.0 62.1 95.1 95.3 98.9 98.7
0.750 0.775 0.800 0.825 0 850 0.675 0.600 0.925 0.950
similrity
Table 10: Our approach obtains the best results across all architectures and largely outperforms most previous works. The subscripts denote the average accuracy of corresponding original mod- els. Underline denotes the reproduced results. And OOM denotes out-of-memory issue. The detailed structures of CNN (s) and CNN (m) can found in Model Zoos [58].
Figure 3: The figure shows the trade-off be- tween accuracy and similarity with ViT-Tiny on ImageNet-IK, The shaded area includes the approximate range of noise-added checkpoints. This plot demonstrates the strong trade-off be- tween accuracy and similarity and highlights our advantages over trivial interpolation.
Similarity analysis of generated models. We compare RPG with adding noise and model interpola- tion. Following p-diff [68], we use IoU to measure model similarity by comparing outputs across samples, selecting the maximum IoU with original models as our metric. Fig. 3, as the noise level increases, both the similarity and accuracy of the models decrease. The points representing our generated models are distributed in the upper left region relative to the other conditions, indicating that our models can enhance diversity while maintaining accuracy. Furthermore, our method demon- strates greater diversity in comparison to trivial interpolation methods. Furthermore, we examine how similarity varies with the number of training checkpoints and analyze the diversity of symmetric states using linear mode connectivity [19, 74]; see Appendix C. and C.6 for details.
Comparisons with previous methods. We compare RPG with four previous works, i.e., SKDE30 [55], p-diff [68], D2NWG [63]. and SANE [57]. We report the original and generated performances for comparison. As shown Tab. 10, among all compared methods, only RPG consistently achieves the highest results and comparable results to original models across various architectures. Another key issue is that the previous works usually fail to generate large-scale neural network parameters. (More discussion is in Appendix A.)
4 RPG Generalizes to Unseen Tasks
Until now, experimental results have demonstrated that our approach can efficiently generate large- scale neural network parameters if these models included in the training In this section, mainly investigate whether our approach has the ability to generate models to tackle unseen tasks. This generalization capability is crucial it enables RPG to handle novel tasks without requiring additional training, making it particularly valuable for real-world applications. Such capabilities are essential towards generating high quality models on the fly, and could be used, e.g., for model personalization or to adjust for distribution shifts, new concepts, or new tasks by generating targeted weights.
4.1 Experiment Designs.
Build seen and unseen tasks. To assess RPG's capability for unseen tasks, we construct various binary classification tasks on CIFAR-IO. As shown in Fig. embedding, where each bit corresponds to category. 1 indicates that the corresponding category should be classified as positive (class 1), while indicates negative (class 0). For example, in the third embedding shown, 'ship' and 'truck' are assigned to class 1, while all other categories belong to class Given this encoding strategy, we can create 210 binary embeddings. After removing the two trivial cases (all Os and 1s), we obtain 1022 valid embeddings. For each one, we collect its corresponding parameters, forming embedding parrmmeer pairs. These pairs are then split into non-overl oon-overlapping sets for training and validation, allowing us to evaluate RPG's generality Collection of the checkpoints. We use ViT-Tiny to train 1022 binary classifiers on CIFAR- 10 with different binary embeddings and save 3 models for each classifier. These binary embeddings serve as conditioning inputs for the subsequent RPG training process. Of these tasks, 1002 randomly selected embedding parameter parrameen-- pairs serve as the training set (seen tasks), while the remaining pairs reserved as unseen tasks for evaluation.
encode each task 10-bit binary
unseen tasks.
8

---

Figure 4: Left: binary embeddings encode different CIFAR-10 classi- fication tasks, where Is indicate classes to be classified together (e.g., 'ship' and 'truck' in the
CIAR-10 10 calegores es
emberdiing prrme
For enbbed model whould daeedy "atlpp" and "truck" to clesa 1, others
36000000011
Denrry emoodd 11302 mann
0
0
0
O
D
0
0
0
0
3000000010
0
0
0
0
0
D00000000111
first example). Right: parameter- encoding pairs, formed by network
For this edbindiing model "automobile" "dog" "horse" "uhtpp, and "truck"
0
0
O
1
D
conespond ng
0000000100
parameters with their correspond- ing binary embeddings.
1131011110
Training and evaluation of RPG. We only use the 1002 checkpoints from seen tasks to train RPG. Meanwhile, these embeddings are also fed into RPG as conditional inputs of the recurrent model. During the training stage, the checkpoints trained by the unseen binary embeddings are not accessible. When evaluating, we imput the unseen binary embeddings to the trained RPG to generate the parameters for unseen tasks. The results of the original and generated unseen models are reported for comparison.
4.2 Results for Unseen Tasks
unseen tasks (embeddings)
original
RPG
Performance comparisons. We compare the results of RPG and original models on unseen binary embeddings in Tab. 11. We randomly select ten unseen binary embeddings for compar- ison. Notably, RPG yields commendable perfor- mance in these unseen tasks, even without being trained on the specific unseen embeddings. That demonstrates the strong practicality and poten- tial of our approach in generating models under unseen tasks. The results of the remaining un- seen tasks and other datasets (ImageNet-IK and PASCAL VOC) are shown in Appendix C.2.
1 0
0
0
0
1
97.3
94.4
1
O
0
98.1
96.6
0
1
1
0
97.4
95.0
1
1
98.4
96.1
0
0
0
0
98.9
96.6
0
96.7
92.9
1
01
97.6
94.8
0
1 0
98.1
95.7
1
97.1
93.6
97.0
94.0
Table 11: Result comparisons between original and generated models on unseen embeddings (tasks). Even without seeing the trained parame- ters of unseen tasks, RPG directly generates pa- rameters that achieve competitive performance.
Perception of shifts in parameter distribu- tions. In addition to comparing results, we further investigate RPG's ability perceive shifts in parameter distributions. We select two tasks whose binary conditions differ in each el- umentteefectively representing opposite distributional configurations-aand report the results in Tab. 12. Our approach demonstrates a remarkable capacity to accurately detect such distribution changes and generate corresponding model parameters. It is worth noting that the accuracy would hover around 50% if RPG were insensitive to these distributional shifts.
binary embedding (from seen set)
1
0
1
1
1
0
1
0
0
accuracy (%)
98.0
99.1
98.5
94.4
98.1
92.2
99.2
97.0
97.1
99.1
flipped embedding (from unseen set)
0
1
0
0
0
1
0
accuracy (%)
92.3
98.9
94.0
85.4
92.2
90.8
99.3
95.3
97.6
98.1
Table 12: Comparisons of parameter distribution shifts. The accuracy is reported for each individual class. RPG can be aware of such distribution shifts accurately.
seen binary embedding: 1010011000
unseen binary embedding: 0001100101
org-1.
org
40008
gen.
org.
4008
org-3
unseen binary embedding: 0001100101
binary embedding: 1010011000
0.00
org. gen.
0.00
gen-1-
unseen binary embedding: 0001100101
gen-2.
org gen.
(a) Original and generated models with identical seen binary embeddings are compared. The three original models exhibit homogeneity, while the generated mod- els display diversity and maintain high accuracy across all three models.
(b) Original and generated models with three unseen binary embeddings are compared. The results confirm that our approach can learn high-performing parameter patterns (similar distribution to original models) even when they are not included in training set.
Figure 5: Illustration of the parameters of original and generated models in seen and unseen embed- dings. We select 100 parameters of the classification head and visualize its normalized values.

---

Visualizations of model parameters. We visualize the original and generated models for both seen and unseen tasks in Fig. 5. For seen tasks, our approach generates diverse models compared to the original ones. Surprisingly, as shown in Fig. 5b, we find that our approach can learn unseen parameter patterns. This demonstrates the potential generalization ability of our method.
Efficiency comparison. To evaluate efficiency on unseen tasks, we compare three approaches: (i) training from scratch, (ii) fine-tuning ViT-Tiny 1 model pretrained on ImageNet- 1 K. and (iii) fine-
epnch from scratch finetune RPG init finetune
0
50.0
52.9
94.4
61.4
90.3
96.8
5
69.1
96.3
97.3
tuning a model initialized with RPG. As shown Table 13, RPG initialization dramatically acceler- ates convergence on the unseen task. These results demonstrate that RPG provides a highly effective initialization strategy, substantially reducing train- ing time and computational costs.
10
74.9
97.1
97.5
50
86.7
97.7
97.8
Table 13: Accuracy (%) comparison for differ-
ent training strategies on an unseen task. RPG initialization demonstrates superior performance and faster convergence.
5 Related Works
In this section, we mainly introduce recurrent models and previous works of parameter generation.
Recurrent models. Recurrent neural networks (RNNs) were first proposed to process sequential data. To tackle the vanishing gradient problem in early RNNs, long short-term memory (LSTM) [28. 27] was introduced. Recently, transformer- based models [67] exhibits excellent potential in sequential data processing, due to their parallelized training and scalability, such as linear attentions [69, 8),
RWKV [52], Mamba [22, 12], and xLSTM [4].
Parameter generation. The core idea of parameter generation is to learn the distribution of trained parameters. Stochastic neural networks [60, 6, 21] and Bayesian neural networks [47, 32, 20] model the priors or probability distributions over the parameters. However, these methods are limited by their generality to large-scale parameter generation. HyperNetworks [24]. i.e. is proposed to generate various architectures' parameters for a larger network. Smash [7] extends the range of architectures via memory read-writes scheme. With the development of diffusion models, many works [50, 9, 16, 68, 63, 39, 38, 31,50] adopt diffusion models to generate neural network parameters. Repee-Representatiions [55, 56, 57] use autoencoder to capture the latent distribution of trained models. COND P-DIFF [31] and Tina [38] introduce text-controlled parameter generation method. Unfortunately, the above methods have common drawback: can not generate large-scale parameters, such as whole parameters of ResNel, VIT, ConvNeXt, or LoRA. Therefore, our approach brings new inspiration the field of parameter generation.
Discussion and Conclusion
Our approach demonstrates promising results in large-scale parameter generation across various vision, language and unseen tasks. However, we acknowledge that achieving true 'Al generating Al' remains distant goal. Firstly, while method shows potential in generating models for unseen tasks, it currently faces limitations in generating parameters for novel model architectures. Secondly, our approach is constrained by modeling parameter relationships within a single task, potentially limiting its practical applicability. More importantly, future work should focus on simultaneously modeling parameter relationships across diverse architectures and tasks. Such an approach could yield more powerful and versatile parameter generator, potentially advancing us closer to the 'AI generating AI' era. We hope our approach will inspire and encourage future research in neural network parameter generation.
Acknowledgments and Disclosure of Funding
We sincerely thank Zhiyuan Liang, Zhuang Liu, Gongfan Fang, Xuanlei Zhao, Zangwei Zheng, Ziheng Qin, and Tianlong Chen for valuable discussions and feedbacks. This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: ASS2-PPDD2202--22
10

---

References
[1] Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2009.48836, 2022.
[2] Sudarshan Babu, Richard Liu, Avery Zhou, Michael Maire, Greg Shakhnarovich, and Rana Hanocka. Hyperfields: Towards zero-shot generation of nerfs from text. arXiv preprint WiXiv22310.17073 2023.
[3] Sudarshan Babu, Pedro Savarese, and Michael Maire. Hypernetwork designs for improved classification and robust meta-learning.
[4] Maximilian Beck, Korbinian Poppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm; Extended long short-term memory. arXiv preprint arXiv:2405.04551 2024.
[5] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In AAAI, volume 34, pages 7432-7439, 2020.
[6] Léon Bottou et al. Stochastic gradient learning in neural networks. Proceedings of Neuro-Nmes, 91(8), 1991.
[7] Andrew Brock, Theo Lim, J.M. Ritchie, and Nick Weston. SMASH: One-shot model architec- ture search through hypernetworks. In ICLR, 2018.
[8] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
[9] Gene Chou, Yuval Bahat, and Felix Heide. Diffusion-sdd Conditional generative modeling of signed distance functions. In ICCV, pages
2023.
[10] Christopher Clark, Kenton Lee, Ming-Wee Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 1905: 10044, 2019.
[11) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.054555 2018
[12] Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.2 2024.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, Icee, 2009,
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In NeurlPS, volume 34, pages 8780-8794, 2021.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[16] Ziya Erkoç, Fangchang Ma. Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. In ICCV, 2023.
[17] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
[18) Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. In NeurIPS, 2023.

---

[19] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. International Conference on Machine Learning. pages 2259-3269. PMLR, 2020.
[20] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML. PMLR, 2016.
[21] Alex Graves. Practical variational inference for neural networks. In NeurlPS, 2011.
[22] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
[23] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23 2024.
[24] David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetworks. ICLR, 2017.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[26] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, volume 33, pages 6840-6851, 2020.
[27] Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997.
[28) Sepp Hochreiter. Universität München, 91(1):31, 1991.
Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische
[29] Sepp Hochreiter, Sepp Schmidhuber Jürgen, Hochreiter, and Schmidhuber. Long short-term memory. Neural Comput., 983: nov 1997.
[30] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
[31] Xiaolong Jin, Kai Wang, Dongwen Tang, Wangbo Zhao, Yukun Zhou, Junshu Tang, and Yang You. Conditional lora parameter generation. arXiv preprint arXiv:2408.01415. 2024.
[32] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.6114, 2013.
[33] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[34] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, volume 25, 2012.
[35] Daniel Kunin, Javier Sagastuy-Brent Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In ICLR, 2021.
[36] Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.1 11838, 2024.
[37] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV, pages 280-296. Springer, 2022.
[38] Zexi Li, Lingzhi Gao, and Chao Wu. fext-to-model: Text-conditioned neural network diffusion for rain-once-for-all personalization. arXiv preprint 0rXiv:2405.14132. 2024.
[39] Lequan Lin, Dai Shi, Andi Han, Zhiyong Wang, and Junbin Gao. Unleash graph neural networks from heavy tuning, arXiv preprint arXiv:2405.1252. 2024.
12

---

[40] Tsung- Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollär, and C Lawrencce Zitnick. Microsoft coco: Common objects in context. In Computer Eaiion-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 6-12, 2014, Proceedings, Part 13, pages 740-755. Springer, 2014.
[41] Shih-Yare Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min- Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint wrXiv22402.093355 2024.
[42] Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung- Yi Lin, Ming- Yu Liu, Sanja Fidler, and James Lucas. Att3d: Amortized text-to-3d object synthesis. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 17946-17956, 2023.
[43] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS, volume 35, pages 5775-5787, 2022.
[44] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In CVPR, pages 15762-15772, 2024.
[45] AI Meta. Introducing meta llama 3: The most capable openly available Ilm to date. Meta AI, 2024.
[46] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
[47] Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
[48] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, pages 8162-8171. PMLR, 2021.
[49] Zizheng Pan, Bohan Zhuang. De-An Huang, Weili Nie, Zhiding Yu, Chaowei Xiao, Jianfei Cai, and Anima Anandkumar. T-stitch: Accelerating sampling in pre-trained diffusion models with trajectory stitching. arXiv preprint arXiv:2402.14167 2024.
[50] William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models neural network checkpoints. arXiv preprint arXiv:2209.1285 2022.
[51] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4195-4200 2023.
[52] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing mnn for the transformer era. arXiv preprint arXiv:2305.1 2023.
[53] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages (0684-10695, 2022.
[54] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv: /904. 09728, 2019.
[55] Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper- representations as generative models: Sampling unseen neural network weights. In NeurIPS, volume 35, pages 27906-27920, 2022.
[56] Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper- representations for pre-training and transfer learning. arXiv preprint 2rXiv:2007.1099 2022.
[57] Konstantin Schürholt, Michael W Mahoney, and Damian Borth. Towards scalable and versatile weight space learning. arXiv preprint 20XV:92400.099977.2000 2024.
13

---

[58] Konstantin Schürholt, Diyar Taskiran, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Model zoos: dataset of diverse populations of neural network models. In NeurIPS, 2022.
[59] Junhyuk So, Jungwon Lee, Daehyun Ahn, Hyungjun Kim, and Eunhyeok Park. Temporal dynamic quantization for diffusion models. NewrIPS, volume 36, 2024.
[60] Haim Sompolinsky, Andrea Crisanti, and Hans-Jurgen Sommers. Chaos in random neural networks. Physical review letters, 61(3). 1988.
[61] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[62] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arX1v:2303.01446, 2023.
[63] Bedionita Soro, Bruno Andreis, Hayeon Lee, Song Chong, Frank Hutter, and Sung Ju Hwang. Diffusion-based neural network weights generation. arXiv preprint arXiv:2402. 18153, 2024.
[64] Alberto Tamajo. I
https://github. con/aalbetottamjoo imagenet wwwrss--classeeee 2023. This repository organizes the ImageNet dataset into 10 coarse classes.
[65] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397,, 2023.
[66] Rejin Varghese and Sambath M. Yolov8: A novel object detection algorithm with enhanced performance and robustness. pages 1-6, 2024.
[67] Vaswani. Attention all you need. In NeurlPS, 2017.
[68] Kai Wang, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion. arXiv preprint arXiv:2402. /3144, 2024.
[69] Sinong Wang, Belinda 7 Li, Madian Khabsa, Han Fang. and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
[70] Ross Wightman. GitHub repository: Pytorch image models. GitHub repository, 2019.
[71] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, pages 418-434, 2018.
[72] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei. Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
[73] Xingyi Yang, Daquan Zhou, Jiashi Feng. and Xinchao Wang. Diffusion probabilistic model made slim. In CVPR, pages 22552-22566 2023.
[74] David Yunis, Kumar Kshitij Patel, Pedro Henrique Pamplona Savarese, Gal Vardi, Jonathan Frankle, Matthew Walter, Karen Livescu, and Michael Maire. On convexity and linear mode connectivity in neural networks. OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022.
[75] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint 1905.0783000000 2019.
14

---

[76] Wangbo Zhao, Jiasheng Tang, Yizeng Han, Yibing Song, Kai Wang, Gao Huang, Fan Wang, and Yang You. Dynamic tuning towards parameter and inference efficiency for vit adaptation. arXiv preprini arXiv:2403. 1808, 2024.
[77] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
15

---

We organize our appendix as follows. Discussion with More Related Works
Section A.1: Overview of comparisons.
Section A.2: Discussion with Ryprer-epprssnntationns
Section A.3; Discussion with HyperNetworks.
Section A.4: Details and limitations of G.pt.
Section A.5: Details and limitations of p-diff.
Section A.6: More related works.
Section A.7: Practical advantages of RPG in real-world applications.
Experimental Setting and Other Details
Section B.1: Training recipe.
Section B.2: Detailed structure recurrent diffusion.
Section B.3: Description of datasets.
Additional Experimental Results
Section C.I: Effectiveness of permutation state.
Section C.2: More results of Section 4.
Section C.3: Training memory cost analysis.
Section C.4: Inference memory cost & sampling time.
Section C.5: Parameter sensitivity vs. performance.
Section C.6: Details of trained checkpoints collection.
Section C.7: Impact of the diffussin process.
Section C.8: Computational cost for training.
Section C.9; Why not auto-regression?
Section 10: Generating from noise vs. mapping from embedding.
A Discussion with More Related Works
A.1 Overview of comparisons
We compare RPG and four other methods from three aspects: scalability, performance, and general- ization. Only our RPG excels in all three aspects simultancously.
method
scalability
performance
generalization
SKde [55]
X
X
p-diff [68]
X
SANE [57]
D2NWG [63]
HyperNetwork [24]
X
HyperFields [2]
x
ATT3D [42]
x
HyperNetwork (MAML) [3]
RPG (ours)
Table 14: Comparison of RPG and existing methods on key aspects: scalability, performance, and generalization.
A.2 Discussion with Rxyeer-Repessnttttinn
We mainly compare with three HyperReppessentatoon methods [55, 57, 56]. These methods use an autoencoder to learn the latent features of trained models, so they call the latent feature Hyper- Representation. This HyperRopresentation then used for analyzing the model's performance or characteristics, or for sampling generate new models or pre-traines pre-trained parameters.
[55] utilizes kernel density estimation (KDE) to sample model parameters on the learned HyperRepresenntttonn space. They also emphasize the importance of layer-wise loss nor- malization in the learning process of Hyperrepprsennation. This work achieves parameter generation in small CNNs from Model Zoos [58] with 2864 parameters.
16

---

[56] focuses on using HyperRepresentation to sample the pre-trained model parameters. They also evaluate the ability of transfer learning by using a trained parameter autoencoder to initialize on unseen dataset. This work can be regarded as cheap parameter initialization method.
[57] divides the neural network weights into subsets and utilizes a sequential autoencoder for neural embeddings (SANE) on there subsets with a sliding window. This work can generate the entire parameter of ResNet-18. However, its generation process does not directly derive parameters from noise. Instead, it relies on half-trained model for Kernel Density Estimation (KDE) sampling.
We summarize the main differences as follows:
Hyper- reppeetepressntttioons as generative models is hard to achieve comparable results as their original models that are used for training, but our approach obtains comparable results.
Hyper-representaaiiee for pre-training and transfer learning focuses on parameter initializa- tion while our approach targets to learn the distribution of serforerforming neural network parameters.
SANE is the latest method among these three HyperRepresentation methods. However, SANE uses a sliding window to model the relationship of small part of trained parameters. Our approach uses recurrent model among parameters.
Our approach can synthesize many popular vision and language parameters, such as ConvNeXt-L and LoRA parameters of LLaMA-7B, with maximum parameter count of approximately 200M, which is much larger than previous works.
Additionally, the parameters generated by our model can directly achieve almost peak performance without any finetuning. And the generation process is entirely synthesized from noise, eliminating the need for few prompt examples that are trained for few epochs, as required by the SKDE [57] sampling.
The capability of large-scale and high-accuracy generation makes our method more appli- cable in practical scenarios, significantly bridging the gap between theoretical parameter generation and practical application.
A.3 Discussion with HyperNetworks
The early HyperNettorks [24] can only generate a small number of parameters and cannot scale up, which limits their applications in many domains. However, in certain areas, researchers improve HyperNetworks to develop some interesting works.
HyperFields [2] introduces dynamic HyperNetwork that maps text directly to NeRF parameters, enabling zero-shot or few-shot 3D scene generation. By combining NeRF distillation with this architecture, it achieves faster training and broader generalization than traditional optimization- based methods.
ATT3D [42] trains a single model across many text prompts, avoiding per-prompt optimiza- tion. This makes text-to-3D generation faster, more generalizable, and capable of smooth interpolations for new assets and animations.
HyperNetwork with MAML [3] improves HyperNetworks' design by fixing gradient scaling, regularization, and momentum issues. The resulting Hyperrettorks achieve higher accuracy and stronger robustness in meta-learning than standard models.
We summarize the main differences follows:
HyperFields and ATT3D aim to generate small, domain-specific networks tailored to partic- ular functions. In contrast, our work explores the unified generation of large models-sudd such as vision transformers and LoRA adapters for large language models has emerged in recent years.
HyperNetwork with MAML focuses on meta-learning and tends to produce better ini- tialization rather than directly generating good parameters. In contrast, our work aims to directly generate high-quality parameters.
17

---

A.4 Details and limitations of G.pt
primary limitation of G.pt [50] is the training data collection cost. By default, they collect 23 million checkpoints to train the parameter generator. Besides, they only evaluate the effectiveness of G.pt on small architectures, such as low-dimensional MLP layer or Convolutional layer with limited channels. The maximum number of generated parameters does not exceed 10,000.
A.5 Details and limitations of p-diff
P-diff [68] directly flattens all parameters into a ID vector, disregarding the inter-layer parameter relationships. Furthermore, p-diff faces challenges in scaling up large-scale parameter generation.
A.6 More related works
Diffusion models. Diffusion models [26, 48, 14] gain increasing popularity in recent years, due to
their superiority in image generation. Ever since its advent, many works have been done focusing on improving the generation quality and efficiency diffusion models. For generation quality, [53]
propose to conduct diffusion in the latent space, enabling high-resolution image synthesis. [51]
leverage the transformer [67] to explore scalability of diffusion models, proving the possibility of generating higher quality images by increasing model size. As for efficiency problem, efficient samplers [61, 43, 62], efficiency models [18, 59, 73], and global acceleration approaches [44, 49] are proposed to increase diffusion models' efficiency. These methods facilitate generating high quality images with less computational and/or memory cost. Although diffusion models for image generation have achieved great success, improving quality and efficiency large-scale parameter generation remains to be explored.
A.7 Practical advantages of RPG in real-world applications
RPG offers several practical advantages real-world scenarios. We list some of them below:
Efficient initialization: RPG provides superior parameter initialization compared random
weights, significantly reducing training time and computational costs.
Few-shot learning: For tasks with limited training data, RPG-generated parameters serve as informed priors that enable better performance than training from scratch.
Rapid model deployment: RPG enables near-instantaneous model adaptation for new tasks without requiring extensive training, making it valuable for real-time applications where quick deployment is critical.
B
Experimental Setting and Other Details
B.1 Training recipe
In this section, provide detailed training recipes and supplementary information. The number of parameters generated by our approach ranges from approximately 3K to 200M. The significant disparity necessitates different training settings. Generally, the number of parameters increases, the learning process becomes more challenging, requiring higher training costs, particularly for generating parameters beyond 50 million. Therefore, training settings divided into two categories: the default setting and the setting for parameters over 50 million, shown in Tab. 15.
When the number of parameters is less than 50 million, adopt single GPU to run the training process. For larger number of parameters, we employ distributed data parallelism to facilitate the training.
Data parallelism:
Diffusion batch size: our approach, the diffusion model is shared across all tokens. Typically, all tokens can be fed as single batch into the diffusion model for training. However, in practice, we randomly select subset of tokens from a long sequence for training, rather than feeding parts at once. This approach significantly reduces memory usage without compromising performance. The 'diffusion batch size' in Tab. refers to the number of tokens fed into the diffusion model during a single training iteration.
18

---

training setting
number parameters < 50M
number parameters 50M
batch size
16
8
optimizer learning rate training steps
Adam W
AdamW
3c-5
lc-5
80,000
120,000
weight decay
le-5
le-5
mixed precision diffusino hatch size
blloat16
bfloat16
1024
512
Table 15: Training recipe in detail.
B.2 Detailed structure of recurrent diffusion
In this section, we provide specific details about the proposed recurrent model and diffusion model in RPG, More detailed configurations be found in Tab. 16.
module
setting
RPG-Tiny
RPG-Small
RPG-Base
RPG-Large
adequate number parameters
<50K
50K~10M
5M~50M
>50M
d model of 1st layer
256
4096
8192
12288
model of 2nd layer
256
4096
8192
16384
recurrent (Mamba)
d_state
128
128
128
d_conv
4
4
4
expand
2
parameter counts
1,3M
256M
1018M
3076M
encoder channels
(1, 32, 64, 128)
(1, 32, 64, 128)
(1, 32, 64, 128)
(1, 64, 96)
decoder channels
(128, 64, 32, 1)
(128, 64, 32, 1)
(128, 64, 32, 1)
(96, 64, 1)
token size
256
4096
8192
16384
kemnel size
7
diffusion (ID CNN)
default solver
DDPM
DDPM
DDPM
DDIM
sampling steps
1000
1000
1000
60
3-start & B-end
(0.0001, 0.02)
(0.0001, 0.02)
(0.0001, 0.02)
(0.0001, 0.02)
betas schedule
linear
linear
linear
linear
number time steps
1000
1000
1000
1000
parameter counts
0.3M
17M
69M
273M
Table 16: Detailed information about four different sizes recurrent diffusion. The adequate number of parameters implies that our model is usually adequate to generate parameters in that scale, which empirical results instead of an exact rule. also necessitates considering other factors such as parameter sensitivity.
Details of recurrent model. By default, the recurrent model consists of two Mamba layers [22]. As the increasing of parameters to generate, we need larger recurrent model capture the information these parameters. The size the recurrent model is mainly determined by the token size, which varies according to the number of parameters to be generated. Based on the token size, we categorize our model into four versions: Tiny, Small, Base, and Large.
Details of diffusion model. Following p-diff [68], our diffusion model adopts a 1D convolutional architecture. The parameters of the diffusion model are significantly fewer than those of the recurrent model. We feed the prototypes from the recurrent model as conditions into the diffusion model by directly adding them to the feature map.
The setting of our main experiments. In our main experiments, ViT-Base, ConyNeXt-Large, ADE20K, COCO, and DoRA rank 64 used the setting of RPG-Large. CNN (s) and CNN (m) used the setting of RPG-Tiny, And all other experiments used the setting of RPG-Base.
19

---

B.3 Description of datasets
In this section, we introduce the datasets used in the paper, including those for classification, semantic segmentation, object detection&instance segmentation, and commonsense reasoning.
Classification: ImageNet-1k [13] is large-scale visual database for visual object recognition research. contains over million images across 1000 categories and is widely used for training
and benchmarking deep learning models. CIFAR-10 [33] dataset consists of 60,000 x2 colorful images in 10 different classes. is commonly used for training machine learning and computer vision algorithms, providing standard benchmark for image classification task.
Semantic segmentation: ADE20K [77] a dataset for semantic segmentation and scene parsing, containing over 20,000 images annotated with pixel-level labels for 150 object categories. It is used to train models to understand and segment various objects and scenes in an image, making it valuable for applications in autonomous driving. robotics, and image editing.
Instance segmentation & object detection: COCO [40] dataset is large-scale object detection, segmentation, and captioning dataset. It contains over 330,000 images in various resolutions, with more than 200,000 labeled instances across 80 object categories. COCO widely used for training and evaluating models in object detection, segmentation, and image captioning tasks.
Commonsense reasoning: BoolQ [10]: Yes/No questions based on natural passages. PIQA [5]: Questions about physical tasks and actions. SIQA [54]: Questions about social interactions. Hel- laSwag [75]: Choosing the correct ending for stories. ARC [11]: Multiple-choiee science questions. OBQA [46]: Questions requires multi-step reasoning, commonsense knowledge, and rich text comprehension.
C Additional Experimental Results
C.1 Effectiveness of permutation state
The effect of permutation state. RPG incorporates permutation state operation to address parameter symmetries, which become particularly pronounced when collecting checkpoints from multiple training runs. To evaluate this, we collect checkpoints from different numbers of training runs (1, 3, and 10) and compare the performance with and without permutation state. These results demonstrate that permutation state operation effectively addresses parameter symmetries and enables stable results even when incorporating checkpoints from multiple training runs.
collected runs
original
w/o permutation state
w/ permutation state
88.2
88.0 fail
88.1
3
88.3
88.1
10
88.5
fail
88.2
Table 17: Permutation state effectively mitigates parameter symmetries when collecting checkpoints from different training runs.
Analysis of LMC for evaluating model similarity. Linear mode connectivity [19, 40] (LMC) is technique used to assess the similarity between deep neural network models by examining the performance of linear interpolations between their weight vectors. This metric would actually suggest diversity the sense of models belonging to different basins [1] which is more meaningful.
In this experiment, we use ResNet-18 on CIFAR- 10 with different training and sampling schemes, and test the performance of linear model weights interpolation. The results in Tab. 18 suggest that LMC can be broken in sampled models when multiple permutation states are applied. This further implies that RPG learn the distribution of parameters, and its generating process linear interpolation of the training data.
merely
method accuracy (%)
A
0.5A4 + 0.5B
trained on checkpoints from same hasin
95.1
95.0
95.0
trained checkpoints from 2 hasins and sampled with permutation state
95.0
94.8
94.9
trained on checkpoints from basins and sampled with permutation states
94.9
18.4
94.9
Table 18: LMC of two RPG-generated models under different training and sampling settings.
20

---

C.2 More results of Section 4
Results of generating models for unseen tasks. In Section 4, we show the potential of our approach generating models for unseen tasks. this part, we provide more results. First, we compare the performance of original and generated models using all unseen embeddings in Tab. 19. Results demonstrate that our approach consistently achieves good results in unseen tasks.
unseen binary embeddings
original
best accuracy
average accuracy
standard deviation
0
0
0
97.3
94.4
93.9
0.6
0
98.1
96.6
94.9
2.1
0
0
1
97.4
95.0
94.2
1.1
0
0
1
98.4
96.1
95.8
0.3
0
0
0
0
98.9
96.6
95.2
2.3
0
96.7
92.9
91.6
1.1
I
97.6
94.8
94.1
0.7
I
0
98.1
95.7
91.8
3.7
O
97.1
93.6
90.7
4.3
97.0
94.0
90.1
3,6
97.3
91.3
90.7
0.8
96.3
95.4
89.4
6.3
0
97.6
92.6
90.5
3.2
0
96.3
90.8
89.1
1.9
0
0
96.3
91.9
88.4
4.4
0
0
97.5
93.7
88.0
5.6
0
0
96.5
90.8
85.5
7.0
1 0
0
0
96.4
86.7
83.7
3.6
1 0
97.7
85.6
83.2
2.0
96.3
90.2
79.2
9.6
Table 19: Performance comparisons between original and generated models on unseen tasks. Specif- ically, generated 10 models for each unseen task, with binary embeddings in the unseen as condition. The results consistently show that our generated models perform comparably to the original models.
More experiments for ViT-Small on Coarse ImageNet-IK [64]. To verify the generalization potential of RPG larger-scale datasets and models, evaluated RPG the ImageNet- K dataset (coarse-grained 10 classes) [64] using the ViT-Small architecture. In Tab. 20, experimental results show that, despite not using any training data when generating parameters for new tasks, RPG achieves performance comparable to that of traditionally trained models. Moreover, RPG demonstrates strong generalization ability and adaptability across increasingly large datasets and models (from CIFARIO 10 to ImageNet, and from ViT-Tiny to ViT-Small), further validating its potential for practical applications.
unseen tasks (embeddings)
original
RPG
0
o
0
91.5
89.7
0
0
0
88.5
87.0
0
0
0
89.1
88.1
1
o
0
89.3
87.3
O
0
91.4
90.4
1
0
88.6
87.2
0
92.5
90.1
1
89.5
87.5
0
0
90.1
87.9
90.2
89.2
Table Result comparisons between original and generated models for generated VIT-Small on Coarse ImageNet- [64].
21

---

More experiments for YOLOv8n [66] on PASCAL VOC [17]. For more complex scenarios, we employ YOLOv8n to demonstrate RPG's ability to generate models for unseen tasks. We use the PASCAL VOC dataset, which contains 20 classes, and design 20-bit bit positional embeddings to specify the target detection requirements (*1' indicates that a class must detected, and "0" otherwise). We then train 500 YOLOv8n models on 500 randomly sampled tasks and collect their checkpoints. Performance is evaluated using mAP50-95, comparing models fine-tuned from pretrained weights against those initialized with generated parameters. The results in Tab. 21 show that RPG provides strong initializations for new tasks, enabling faster adaptation and reducing the need for extensive training. This confirms our method's ' scalability and effectiveness on more complex tasks.
unseen tasks (embeddings)
epoch-O
epoch-1
epoch-2
epoch-5
0
0.29
0.01 0.30
0.06 0.06/ 0.33
0.18 0.39
0
0.01 0.25 0.02
0.30
0.05
0.21 /039
0
0
0
0.01 0.28
0.09 0.33
0.12/0.35
0.32 / 0.38
0
0
0
0
0
0
0
0
0.00 / 0.24
0.04
0.04/
0.29
0.13 / 00.35
0
0
0
0
0
0.00 0.03 /.22 0.11 1 0.26 0.21 / /0.3
Table 21: Result comparisons between original and generated models for generated YOLOv8n [66]
on PASCAL VOC [17].
PCA visualization of classification head parameters. We also provide visualization of the param- eters of the classification head (a two-layer fully connected structure with total 38,976 parameters) for 1022 tasks described in Section 4 using Principal Component Analysis (PCA), which presents the structure of the parameter space in Fig. 6a. Our generated model achieves an average accuracy of 91.2% across all binary classification tasks, which indicates that our method has effectively learned this structure. Furthermore, we evaluate the parameters corresponding to unseen tasks and compared their positions in Fig. 6b between the original and generated parameters. It is noteworthy that, even though the original parameters of these tasks are not included in the training data, the generated parameters consistently appeared in close proximity to the original ones. This observation further highlights the capability of our method to model the structure of the parameter space, even for tasks not previously encountered.
all tasks parameter space
unseen tasks parameter space
original generated
(a) Visualization of the classification head of all 1022 tasks. This reveals that there is an inherent structure among the high-dimensional parameters, which can be captured by deep learning models.
(b) Visualization of the classification head in some unseen tasks. Parameters associated with the same task are indicated by a consistent color. Our method can capture the structure of the parameter space.
Figure 6: Principal Component Analysis (PCA) visualization of the classification head. The figures demonstrate the presence of an inherent structure in the parameter space and highlight our method's effectiveness in capturing this structure for unseen tasks.
22

---

Conditional generation guided by LLM. To demonstrate the application scenarios of our model, we utilize large language model to generate binary embeddings to guide RPG in generating corresponding classification models. For the first example in Fig 7, we give the L.M prompt: 'Give me model to select all living things.' With the binary embedding provided by the L.M,, our RPG then generates a ViT-Tiny classifier. After that, We use images in CIFAR-16 to evaluate the model's accuracy. The model should classify 'bird', 'cat', 'deer', 'dog'. 'frog'. and 'horse' to the positive class, which we used as the ground truth. The result is 97.1%. Some other examples are in Tab 22. This experiment demonstrates our method's capability to generate neural network parameters based on natural language guidance, highlighting the potential applications of our method.
Give me model select living things.
LLM
RPG
ViT-Tiny Classifier
x
x x
overall accuracy: 97.1%
Give me madel to select animals that be regarded as pets.
LLM
RPG
ViT-Tiny Classifier
X
overall accuracy: 96.9%
Figure 7: Illustration of RPG- Per-ggeeeatedd models guided by binary embeddings from a large language model (Qwen2.5-3B (72]), demonstrating parameter generation conditioned by natural language.
prompt
expected embedding
acc. (%)
Give me a model select all living things.
O.O.I.IILIIII..OOO
98.7
Find all vehicles that operate roads.
0,1.0.0,000OO000O00.11
95.9
Classify mammals.
0,0.0.I.II...11.0000
97.6
Select all man-made objects. Find all things that are both man-man made and can operate on water. Select all animals that can he regarded as pets. Select all things that can fly. Find all animals with fur.
1,1,0.0,0000,,..IIIIIII
97.7
0,0.0.0.0OOO,,OOI11.00
98.4
0,0.0.1.01110..0000000
96.8
1.0.1,0,0,000,000000
87.7
0,0.I.IILL.O11.000
70.4
Select all pets commonly found in households. Identify all cold-blooded animals.
0,0,1,1.0.110,,.00000
83.3
0,0.0.0.0.O.1...00000
98.9
Table 22: The table demonstrates examples of generating from abstract prompts using LLM and
RPG. We are able to achieve high accuracy on abstract prompts. The expected embedding is used for evaluating the accuracy.
C.3 Training memory cost analysis
In this section, we analyze the GPU memory utilization during training. GPU memory consumption usually highly correlated with two factors: i) the size of the generative model and ii) the size of generated parameters. We analyzed the impact of these two factors on the GPU memory utilization during the training of our approach.
GPU Memory vs. Token Size. In Tab. 5, we present the relationship between token size and RPG's learning capability. Despite the fact that larger token sizes (i.e., larger models) carry a greater volume of information, they also lead substantial GPU memory costs, as illustrated in Fig. 8a. This implies that, when the performance of the generated models is comparable, we prefer to use models with smaller token sizes.
GPU memory v.s. parameter counts. We conduct experiments show the relationship between GPU memory and generated parameter counts in Fig. 8b. In previous methods, the relationship between GPU memory consumption and the number of parameters in the generated model was quadratic [55] or directly proportional [68]. This limits their practicality and application range. In contrast, our approach demonstrates remarkable efficiency: with equivalent GPU memory usage, generate models with 34 to 960 times more parameters compared to previous methods.
23

---

GPU memory v.S. token size
GPU memory v.s, parameter counts
80
ViT-Tiny
VIT-Small
GPU memory usage (GB)
VIT-Base
GPU memory usage (GB)
34x larger
960x larger
40
40
30
30
20
20
SXDEaI
10
10
ours
0
2048
4096
8192
16384
0.01
0.1
1.0
10.0
100.0
token
params. (M)
(a) Visualization GPU memory v.s. token size. GPU memory usage increases proportionally to the token size. Thus, the token size cannot get larger infinitely; we need to choose proper token size.
(b) Visualization GPU memory v.s. parameter counts. Our method can generate much more pa- rameters than existing approaches e.g. SKde30 [55] using single NVIDIA H100 80G GPU.
Figure 8: Training memory cost analysis. Left: GPU memory v.s. token size. Rihgt: GPU memory parameter counts.
C.4 Inference memory cost & sampling time
In this section, present more information about the sampling, including memory usage, inference time, and the balance between sequential and parallel inference. In Tab. 9, we show the sampling time and memory usage for ViT-Base and ConvNeXt-L. Here, we present the sampling time and memory usage for other models. In Tab. 23, we adopt DDPM as the solver and conduct 1000-step sampling. Since the diffusion model in RPG is shared among all the parameter tokens, can adopt different inference modes to find a balance between memory usage and inference speed:
fully parallel: All tokens are fed into the diffusion model simultaneously. This approach results high memory usage but achieves high generation speed.
sequential: Tokens are fed into the diffusion model one by one. This approach significantly reduces memory usage, as the model only occupies memory for inferring single token at time. This enable us to generate parameters of models listed on a GPU with less than 8GB memory.
partially parallel (default): In partial parallel mode, we set 256 tokens batch for the diffusion model inference. This approach significantly boosts speed with slight increase in GPU memory usage, reaching an optimal trade-off between memory and speed. We adopt this as the default setting.
Based on the results Tab. 23, our approach can be flexibly applied to many other GPUs as it can achieve good trade-off between memory and time.
metrics
inference mode
ResNet-18
ResNet-50
ViT-Tiny
ViT-Small
ConvNeXt-A
sequential
18.6
38.0
9.8
33.8
6.8
time (minute)
partially parallel
3.3
2.9
fully parallel
3.3
2.9
sequential
6.3
6.4
6.2
6.4
6.2
memory cost (GB)
partially parallel
10.3
10.5
10.3
10.5
10.3
fully parallel
30.8
50.5
19.4
45.9
15.2
Table 23: We show the inference time and memory cost under different inference modes. All information this table is collected from a single NVIDIA H100 80G GPU. We report the time and memory required to generate single model.

---

C.5 Parameter sensitivity vs. performance
According to conventional understanding, larger parameter quantities are generally more challenging to learn. However, our experiments reveal that this rule is not absolute and demonstrates instability in learning some small model parameters.
This motivates us to investigate the relationship between parameter sensitivity and generation quality. Specifically, we add Gaussian noise with weights of 0.01, 0.10, and 1.00 the original parameters to measure model sensitivity, as shown in Tab. 24. We observe that as noise weight increases, performance decreases for all models, with smaller models being more affected than larger ones. This indicates that smaller models are relatively more sensitive. Additionally, we notice that the performance gap between the original and generated models widens as sensitivity of the model increases. This demonstrates a strong correlation between model's sensitivity and the difficulty of generating parameters.
accuracy decline
model
params, (M)
sensitivity
ours
noise (0.01)
noise (0.10)
noise (1.00)
ConvNeXI-A
3.7
+++
0.85
62.83
0.60
0.03
ResNet-18
11.7
++
0.39
53.56
0.46
0.00
ViT-Base
86.6
+
0.09
5.39
0.02
0.00
Table 24:
The accuracy decline reflects the accuracy gap between the original model and the generated model or the model after adding noise. We add Gaussian noise with weights of 0.01, 10,, and 1.00 to the parameters to measure model sensitivity. Results demonstrate that smaller models are relatively more sensitive than larger ones. The more plus signs (+), the higher the sensitivity.
C.6 Details of trained checkpoint collection
This section mainly supplements the collection of checkpoints in Section 3. First, obtain the pre-trained models, which either come from model libraries (such as timm [70]) or are trained by ourselves. Then, fine-tune the full model for epoch, and save 50 checkpoints during this epoch uniformly. In Tab 17, the term collected runs refers to the number of times the entire process, from pre-training to fine-tuning and saving, is repeated. This done without fixing the seed, resulting in checkpoints from entirely different permutations.
In addition, we also compared the impact of the number of collected checkpoints on the similarity between models. Based on the results in Tab. 25, the analysis reveals that using too few checkpoints lead to low diversity. However, the number of checkpoints increases, the similarity initially decreases and then stabilizes around 0.84 when the count exceeds 50.
number of checkpoints
10
50
200
400
similarity
0.93
0.89
0.84
0.83
0.84
Table 25: Comparison between the number of checkpoints and similarity.
C.7 Impact of the diffusion process
We conduct experiments to analyze the effect of the diffusion process. We compare the performance of our method with and without the diffusion process. The results are obtained by training with ViT-Tiny. The findings are as follows: (1) using the diffusion process allows for generating an infinite number of models, whereas the version without the diffusion process can produce only one model; (2) incorporating the diffusion process increases the diversity between the original and generated models (0.84 vs. 0.91); (3) the model generated without the diffusion process tends to memorize the ensemble of original models more strongly, as indicated by a high similarity score of 0.93.
number of generated models infinite
similarity with ensembled original models
diffusion process
similarity
accuracy (%)
0.84
0.85
75.3
X
only one model
0.91
0.93
75.3
Table 26: Comparison of with and without diffusion processes.
25

---

C.8
Computational cost for training
Training RPG requires computational resources. However, all experiments can be completed within 144 GPU hours (8 GPUs x 18 hours), as shown in Tab. 27.
model
number of parameters
training cost (GPU hours)
ViT-Tiny
6.M
4
DoRA (rank4)
8M
4
ViT-Small
22M
10
VIT-Base
86M
54
DoRA (rank64)
113M
73
ADE20K
177M
132
ConvNeXt-L
198M
144
ViT-Tiny (Section-4)
6M
40
Table 27: Training cost of RPG for various model architectures.
C.9 Why not auto-regression?
It is worth noting that our approach does not employ an auto-regressive method, i.e., we do not feed the output back as input again. Our method takes position embedding and permutation state inputs and synthesizes neural network parameters as outputs, forming a standard recurrent neural network. We have attempted to train our model using an auto-regressive approach such as decoder-only transformer architecture, whose results are shown in Tab 6. To ensure fair comparison, we also applied causal mask to the transformer encoder-only structure, ensuring that information can only be passed in one direction. Due to the accumulation of errors during the auto-regressive process in inference, the parameters generated at the end of sequence become nearly indistinguishable from noise, leading to poor performance. In contrast, in RPG, noise is only introduced in the diffusion model and does not accumulate in the recurrent process, ensuring stable parameter generation.
C.10 Generating from noise vs. mapping from embedding
When generating neural network parameters, methods that synthesize weights from noise-such as RPG-offe significant advantages in memory efficiency and scalability over approaches that map learnable embeddings directly to full parameter tensors.
Lower memory footprint: Embedding- bansedding-based decoders must produce the entire parameter tensor in one go, causing memory usage to scale linearly with model size, In contrast, RPG generates weights sequentially, drastically reducing peak memory consumption.
Superior scalability: RPG's autoregressive generation process naturally accommodates large models without requiring storage or computation of massive, dense parameter tensors once.
Favorable compute- memory trade-off: While RPG incurs a modest increase in compute due sequential generation, it achieves orders-of soteersof--magninn magintue memory savings-making it far more practical for deployment in resource-constrringg or large-scale settings.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All the claims are supported by the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss in the conclusion and discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification:
We provide detailed recipes in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification:
We release all the code to the public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececccccccccct /ttpa//gggiieee/eeeuuuuiiiiiiiii
/CodeSubmic ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include all necessary details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the best, min, and average results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
29

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We introduce this in main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https / http///eerpp//eeeee cc httppppulcc/tciccuiiiaieo
Answer:
[Yes]
Justification:
We strictly follow NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss it in the conclusion and discussion section, and appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
30

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All involved papers are cited properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
31

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
32

---

Justification:
This paper's development does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https:/ for what should or should not be described.
: /ttprr/ps..c///offeeeeeae///////
33


Reviews:
[{"review_id": "YV9uMnmh9B", "reviewer": "Reviewer_D95z", "summary": "- Parameter-generating models have struggled to scale to large numbers of generated parameters.\n- This work presents a novel setup that proposes parameter tokenization combined a recurrent state-space network to model inter-layer dependencies, and a diffusion model for parameter prediction, these techniques enable scaling the number of predicted parameters.\n- The paper evaluates the method on a wide variety of tasks and networks across vision and language, showing that the approach works in practice.\n- Many ablations are provided to justify the modeling choices such as recurrent architecture, type of positional embeddings or tokenization stretegy.", "questions": "1. In Equation 1 weights are normalized during training. How are the corresponding shift and scale values produced at inference?  \n2. If permutation states are one-hot during training, what representation is used at inference?  \n3. Can you report standard deviations for Tables 1, 2 and 7? Additionally, why do the blue figures in Table 7 vary across entries in the same column?  \n4. For the presented tasks/architecture, what is the minimum number of checkpoints required to train a useful RPG model?  \n5. Conversely, for the presented networks what is the smallest RPG parameter count that can match at least 95 percent of baseline accuracy ?\n\n*I would consider raising my score if these questions are answered satisfactorily.*", "limitations": "- Generated models are still below one billion parameters, limiting immediate applicability to large language models.  \n- RPG is not permutation invariant; it learns explicit neuron permutations, which increases model size and training time.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- *Quality* - Wide coverage of architectures (ResNet, ViT, ConvNeXt, LoRA-augmented LLMs) and both vision and language tasks.\n- *Analysis* - Ablation study is thorough, dissecting many modeling choices in order to pinpoint what matters for accuracy and efficiency.  \n- *Significance* - Results show single GPU generation of ~200M parameter networks, pushing parameter generation methods further than prior work.  \n- *Originality* - Combines parameter tokenization with a recurrent decoder and parameter diffusion. I believe this is novel work in this area of research. \n\n**Weaknesses**\n\n- *Clarity* - After reading the paper, I find that training versus inference details are underspecified. E.g. the procedure for predicting weight unnormalization shifts and scales; as well as how to choose permutation states at inference time remains unclear.  \n- *Significance* - Paper never analyzes how many RPG parameters are required relative to each target model, making cost-benefit trade-offs hard to judge. \n- *Significance* - There is no quantification of how many checkpoints are needed to train RPG well. An ablation here would be really informative.\n- *Presentation/Quality* - Section 4 feels like a distraction and my opinion overstates the results. \"Unseen tasks\" have the same training data as seen tasks, and the \"task\" amounts to adapting the weights of the last layer, so it is not a useful result in practice. Furthermore, the relative sizes of training and test sets (1002 and 20 tasks) are not realistic.\n- *Presentation* - In contrast, some key results (Tables 19 and 20) are buried in the appendix but I find them more relevant to the presented work and should be in the main body of the paper.", "was_revised": true, "final_justification": "See my reply to the authors, they addressed some of my key concerns. While the presentation has room for improvement, I believe the work is technically sound.", "rebuttal": "**Q1:** Training vs inference details are underspecified. How to choose permutation states at inference time remains unclear.\n\n**A1:** Thanks for the comment.\n\n- **How is unnormalization implemented?**\n    - The **weights (scales)** and **bias (shifts)** we used for normalization and unnormalization are **NOT calculated independently** for each model, but are based on the **average over the entire training set**.\n    - The **layer-wise mean** and **std** are calculated by: $\\mu_{l}=\\frac{1}{N}\\sum_{n=1}^{N}mean(W_{n}^{l}), \\sigma_{l}=\\frac{1}{N}\\sum_{n=1}^{N}std(W_{n}^{l})$, where $W_{n}^{l}$ denotes layer $l$ of the $n$-th model, and $\\mu_{l}$, $\\sigma_{l}$ denotes its mean, variance, respectively.\n    - These statics are **pre-computed and stored**, and utilized in unnormalization process **for generated parameters**.\n\n- **How to choose permutation state during inference?** \n    - Due to space limitations, please refer to **A4 of DHqN**.\n\n- **Action:**\n    - We put these results and analysis in **Section 3.3**.\n    - We add more **motivation** of permutation state **line 117**, **Section 2.2**.\n\n\n**Q2:** how many RPG parameters are required relative to each target model.\n\n**A2:** Thanks for the comment.\n- We present the details of **structures of RPG** and **number of parameters it can generate** in **Table 12**, **Appendix B.3**. We put the data here:\n\n    |model scales|RPG-T|RPG-S|RPG-B|RPG-L|\n    |-|-|-|-|-|\n    |num. can generate|up to 50K|up to 10M|up to 50M|up to 200M|\n    |recurrent|1.3M|256M|1018M|3076M|\n    |diffusion|0.3M|17M|69M|273M|\n    |total RPG|1.6M|273M|1087M|3349M|\n\n- **Findings:**\n    - The **denoising network**'s parameter scale is **close** to the number of parameters that RPG can generate.\n    - **Most** of RPG's parameters come from **recurrent backbone**.\n- **Conclusions:**\n    - RPG hass a **light-weight** denoising network.\n    - With **lighter backbone**, RPG is possible to generate **billion parameters**.\n    - RPG can generate **almost infinite** number of models once trained, so the efficiency trade-off is **worwhile in the long run**.\n- **Action:**\n    - We add these analysis to **Section 3.1**.\n    - We will explore lighter backbones.\n\n\n\n**Q3:**  How many checkpoints are needed to train RPG well.\n\n\n**A3:** Thanks for the comment.\n- We evaluate RPG's performance on 2 scenarios: **close-set (generate models seen in training)** and **open-set (generate unseen models)** generation.\n\n- **Close-Set:** In **Table 21**, **Appendix C.5**, we conduct ablation studies of **checkpoints number for training RPG**. We use ViT-Tiny on ImageNet.\n\n    - **Setting:**  \n\n    |number of checkpoints|5|25|100|300|1002|\n    |-|-|-|-|-|-|\n    |acc.|75.1|75.2|75.2|75.2|75.3|\n\n    - **Findings and Conclusions:**\n        - **Only 1 checkpoint** can train RPG well, showing its strong close-set geneation ability with **very limited data**.\n\n- **Open-set:** Thanks for the suggestion, we conduct the following ablation:\n    - **Setting:**\n        - We randomly select **5, 25, 100, 300, and 1002 (all)** checkpoints claimed in **Section 4** as training set. All configurations are consistent with **Section 4**.\n\n    |number of checkpoints|5|25|100|300|1002|\n    |-|-|-|-|-|-|\n    |acc. |54.1|84.7|86.9|87.8|90.0|\n\n    - **Findings and conclusions:**\n        - RPG can **generalize** to unseen tasks even trained on a **smaller number of checkpoints**, i.e., **100**.\n        - More data can improve performance, showing that generalization needs **basic amount of training data**.\n \n    \n\n**Q4:** \"Unseen tasks\" have the same training data as seen tasks, and the \"task\" amounts to adapting the weights of the last layer, so it is not a useful result in practice. Furthermore, the relative sizes of training and test sets (1002 and 20 tasks) are not realistic.\n\n\n**A4:** Thanks for the comments. \n- **1. \"Unseen tasks\" have the same training data as seen tasks**.\n- **2. The \"task\" amounts to adapting the weights of the last layer, so it is not a useful result in practice**.\n- **3. The relative sizes of training and test sets (1002 and 20 tasks) are not realistic**.\n- We refer our answers to them as **A 4.1**, **A 4.2**, and **A 4.3**, respectively.\n\n- **A 4.1:** Thanks for the comment.\n    - Parameters for **unseen tasks** and **seen tasks** are of **same network structures**, but they are trained on **different binary embedding classification** tasks.\n    - **ALL** parameters in unseen tasks are not involved in RPG's training, so the evaluation process is **zero-shot**.\n\n- **A 4.2:** Thanks for the comment. \n    - We first conduct **full-parameter fine-tuning**.\n    - Therefore, the backbone parameters for the 1022 training samples are not identical. \n\n- **A 4.3:** Thanks for the comment.\n    - As ablation study in **A3** suggests, RPG can effectively learn from **limited training checkpoints**, i.e., achieving **84.7%** accuracy with only **25 checkpoints**, **86.2** with **100 checkpoints**\n    - Additionally, our experiments on **object detection** and **larger dataset** in **Appendix C.2** provide more generalization results: (Note that we change embedding to hex to save words)\n\n    - **object detection:**\n  \n        - **Setting**: YOLOv8n on PASCAL VOC, using 20-bit 0-1 embeddings. \n          \n        - **Results:**\n     \n            |unseen tasks|epoch-0|epoch-1|epoch-2|epoch-5|\n            |-|-|-|-|-|\n            |2010c|0.00 / 0.29|0.01 / 0.30|0.06 / 0.33|0.18 / 0.39|\n            |08026|0.01 / 0.25|0.02 / 0.30|0.05 / 0.32|0.21 / 0.39|\n            |11048|0.01 / 0.28|0.09 / 0.33|0.12 / 0.35|0.32 / 0.38|\n            |20061|0.00 / 0.24|0.04 / 0.25|0.04 / 0.29|0.13 / 0.35|\n            |0a060|0.00 / 0.19|0.03 / 0.22|0.11 / 0.26|0.21 / 0.33|\n    \n        - **Conclusion:**\n            - RPG offers good initializations, faster adaptation and reduces training costs.\n            - RPG can generalize to complex tasks.\n\n    - **larger dataset:**\n\n        - **Setting:** ViT-Small on ImageNet-1K (coarse-grained 10 classes).\n        \n        - **Results:**\n\n            |unseen tasks|original|RPG|\n            |-|-|-|\n            |015|91.5|89.7|\n            |1a1|88.5|87.0|\n            |3f2|89.1|88.1|\n            |30b|89.3|87.3|\n            |2ad|91.4|90.4|\n            |370|88.6|87.2|\n            |2be|92.5|90.1|\n            |1ea|89.5|87.5|\n            |2d1|90.1|87.9|\n            |360|90.2|89.2|\n    \n        - **Conclusion:**\n            - RPG can also generalize at ImageNet, achieving comparable performance to  trained models.\n\n\n\n**Q5:** Some key results are buried in the appendix.\n\n**A5:** We make the following modifications.\n- Move **Tables 19 and 20** , and relevant analysis to the main text in the revision.\n- Examine other details of our work and refine it accordingly.\n\n\n\n**Q6:** Can you report standard deviations for Tables 1, 2 and 7? \n\n**A6:** We provide the std below and include them in the revision:\n- **Note:**\n    - We present the data in **accuracy (std)** format.\n    - Due to word limit, we only provide **Table 1** and will update the rest results once entered discussion phase.\n- **Table 1:**\n    |arch. \\ acc.(%)|ResNet-18|ResNet-50|ViT-Tiny|ViT-Small|ViT-Base|ConvNeXt-A|ConvNeXt-L|\n    |-|-|-|-|-|-|-|-|\n    |original|70.0(0.28)|79.8(0.05)|74.9(0.13)|81.4(0.08)|84.4(0.08)|75.2(0.07)|85.8(0.10)|\n    |generated|69.5(0.23)|79.5(0.07)|75.3 (0.07)|80.5 (0.14)|84.4 (0.12)|74.4 (0.11)|85.5 (0.19)|\n\n\n**Q7:** Why do blue subscript of Table 7 vary across entries in the same column?\n\n**A7:** Some works did not have updated the code with the latest paper, making them difficult to reproduce. Therefore, all the data in Table 7 are taken from the original papers and are not our reproduced results. Since the checkpoints used in the original papers vary across studies, we have added blue subscript markers to indicate this.\n\n\n\n**Q8:** What is the smallest RPG parameter count that can match at least 95 percent of baseline accuracy?\n\n**A8:** Thanks for the comment.\n- We conduct ablation studies **Appendix C.3**, **Table 19** to explore this, and we put the results here for convenience.\n- **Settings:** We generate **ViT-Tiny** with RPG of various parameter counts and report the accuracy.\n- **Results:**\n    |# RPG parameter|acc.|\n    |-|-|\n    |20M|0.3|\n    |81M|70.8|\n    |273M|75.2|\n    |1087M|75.3|\n    \n- **Findings and Conclusions:**\n    - At parameter count of 81M, RPG generates model with accuracy of 70.8% \n    - As in **Section 3.1**, **Table 1**, baseline accuracy is 74.9%.\n    - The generated model achieves **70.8% ÷ 74.9% = 94.5% (roughly 95%)** of the original model's performance.\n\n\n\n**Q9:** Generated models are still below one billion parameters, limiting immediate applicability to large language models.\n\n**A9:** Thanks for the comment.\n- We admit that RPG currently don't support LLM **full model** generation.\n- As in **A2**, RPG's scaling up **bottleneck** is its heavy **backbone**. We are currently working on finding **lighter alternatives**.\n- Moreover, we can generate **LoRA parameters**, which offers efficient **task adaptation** of LLMs.\n- **Solutions for scaling up**:\n    - Finding **lighter recurrent backbones**.\n    - Improving the **Efficiency of tokenization**.\n\n\n**Q10:** RPG is not permutation invariant; it learns explicit neuron permutations, which increases model size and training time.\n\n**A10:** Thanks for the comment.\n\n- We admit that **permutation invariant** is a crucial question in parameter generation, which enables training on checkpoints **collected from various sources**.\n- We introduce **permutation state** to mitigate this problem, as stated in **A1**. Results show that without permutation state can **cause failure** in training on models with different permutations.\n- Our method can **alleviate** the problem, but achieving permutation invariant remains challenging, and we will continue to explore in the future."}, {"review_id": "97VktbMbOS", "reviewer": "Reviewer_Q57G", "summary": "The paper proposes a network design that generates parameters of another neural network in a scalable and efficient manner. Their proposed network takes a form of a recurrent network so generated layers are generated in a manner that is aware of the inter-part correlations.  Their model is trained with a diffusion objective to enable sampling of network parameters. They show experimentally that the so generated networks are comparable with network trained via backpropagation.", "questions": "1) Is the permutation state essentially a one hot encoding on the trained networks used to train the hypernetwork?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) Proposing recurrent hypernetworks is an interesting and innovative idea.\n2) Sufficient ablation experiments are done to justify the various design choices.\n3) Out-of-distribution experiments are quite interesting and show capabilities of hypernetworks to generalize to new tasks\n\nWeaknesses:\n1) Mis-stated lines:   \n       (a) line 20: However, neural network parameter generation—from early HyperNetworks [20] to more recent diffusion-based methods [45, 63, 58]—has not kept pace with the expansion of main stream vision and language models\"-- I disagree with this part of the paper's motivation. There are works such as [1,2] that have scalable hypernetworks. [1] replaces the MLP hypernetwork in the original HyperNetworks paper,  with a convolutional layer which makes the original hypernetwork (with a few other modifications) quite scalable. [1]  uses hypernetworks to generate models that are around 20M parameters and I dont see why it should not be capable of scaling to ~100M parameter size. For [2] they use it to generate NeRFs which are just a few MLP layers, but these models are also capable of scaling to models of size ~100M. So I think, line 20 is not true, unless the authors can show some experiments where hypernetworks such as the ones in [1,2] fail when required to generate weights for large networks. One distinction I would like to make here is that [1,2] have low dimensional embeddings (that are trainable) that get mapped to weights of the generated network, whereas in the paper, they take a generative approach, which maps noise to weights perhaps line 20 can be rephrased as \"hypernetworks that generate parameters from noise dont scale well\".  \n\n\n\n       (b) line 24: \"This discrepancy not only underscores the challenge of scalability but also confines existing generators to academic demonstrations rather than real-world applicability.\"-- I disagree with this line the way it is stated. [2,3,4] demonstrate that  hypernetworks can be used in real-world applications, they require generating only NeRFs which are typically smaller models.  So imo hypernetworks are not just confined to academic demonstrations due to scalability challenges.  \n\n\n       (c) line 29: \"Traditional approaches such as HyperNetworks struggle to handle even moderately sized models, primarily due to memory overheads and optimization complexity\"--Simple modifications to the orginal HyperNetworks paper as demonstrated in [1] make it very easy to scale. I personally have trained hypernetworks to generate parameters for ResNets of size 100M using the recipe given by [1] for tasks like CIFAR and ImageNet classification. It could be helpful if the authors can show [1,2] not scaling well for some applications.\n\n2) Lacks main motivation: The main motivation of the paper as I see it,\"current parameter generation methods do not scale well when asked to generate weights of large models\" is in my opinion not true. That said, the paper does IMO explore an interesting class of hypernetworks that are capable of generation of parameters from noise as opposed to [1,2] which map learnable embeddings to network weights. I think it would be useful if the authors can motivate why such a class of hypernetworks is more appealing than the class of hypernetworks that map embeddings to network weights as in [1,2]. If the authors can provide a suitable answer to this, I am happy to increase my rating. \n\n3) Measuring diversity of generated networks: While the authors show diversity of models (following prior work) using a reasonable metric of selecting maximum IoU with original models. IMO a better metric would be to see if linear mode connectivity[5,6] breaks, so typically if models end up in the same basin, it is shown that interpolation of those models leads to a model which is also valid model with the same performance level as the models that were interpolated, however if these model are in different basins then interpolation between these models would lead to a model that performs significantly worse than the models used for interpolation. So I am suggesting taking the generated models and seeing with what fraction of the model weights used in training does convex mode connectivity break. This metric would actually suggest diversity in the sense of models belonging to different basins which imo is more meaningful. I understand it might be hard to implement such a metric within the rebuttal period, but just a suggestion to try out.\n\n4) Missing related work: A discussion on [1,2,3,4], especially given [3] uses a recurrent hypernetwork would help in contextualizing the current work.\n\n[1] Babu, Sudarshan, Pedro Savarese, and Michael Maire. \"HyperNetwork Designs for Improved Classification and Robust Meta-Learning.\"\n\n[2] Chen, Yinbo, and Xiaolong Wang. \"Transformers as meta-learners for implicit neural representations.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[3] Babu, Sudarshan, et al. \"Hyperfields: Towards zero-shot generation of nerfs from text.\" arXiv preprint arXiv:2310.17075 (2023).\n\n[4] Lorraine, Jonathan, et al. \"Att3d: Amortized text-to-3d object synthesis.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[5] Frankle, Jonathan, et al. \"Linear mode connectivity and the lottery ticket hypothesis.\" International Conference on Machine Learning. PMLR, 2020.\n\n[6] Yunis, David, et al. \"On convexity and linear mode connectivity in neural networks.\" OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop). 2022.", "was_revised": true, "final_justification": "I recommend this paper for weak accept", "rebuttal": "**Q1:** There are works such as [1,2] that have scalable hypernetworks.\n\n**A1:** Thanks for the constructive suggestions. We answer the question as follows:\n\n- The papers pose are interesting, [1] achieves large scale generation for **meta-learning**, [2] cusmotizes **NeRFs** via transformer hypernetworks.\n- We **change the statement** in **line 20** to \"hypernetworks that generate parameters from noise dont scale well\".\n- We apologize for overlooking these literatures at first, and we will add them to **Section 5** in the revision.\n\n\n**Q2:** There are works such as [2,3,4] that demonstrate hypernetworks can be used in real-world applications.\n\n**A2:** Thanks the reviewer for the comment and we answer as follows.\n- We appreciate reviewer Q57G for pointing out these relevant works. NeRFs are practical models and this serves as a effective indicator for hypernetworks' utility.\n- Our work, on the other hand, are trying to explore the generation of **large models in recent years**, such as **vision transformers** and **LoRA adapter for LLMs**.\n-  We conduct experiment on models with **3 to 200M parameters across image classification, object detection, small language models, and LoRA adapters for text and image generation.** Through these explorations, our scope is **broadening the application scenarios** of parameter generation.\n- We highly apprecitate your interest in hypernetworks and the helpful suggestions posed, and we **rephrase** \"This discrepancy not only underscores the challenge of scalability but also confines existing generators to academic demonstrations rather\n than real-world applicability.\" $\\rightarrow$ \"This discrepancy underscores the challenge of scalability and **if solved, can strengthen parameter generation's real-world applicability.**\"\n\nWe hope these answer can address the concerns, and we appreciate the reviewer Q57G's interest in hypernetworks. We hope to hear more in discussion phase.\n\n\n**Q3:** Why is (a) generating parameters from noise more appealing than (b) mapping learnable embeddings to network weights?\n\n**A3:** Thank you for the question. We answer as below.\n\n- **Better scalability and memory use.**\n- Embedding decoders must output the full parameter tensor in **one pass**, so memory scales with model size.\n- RPG generates weights **sequentially**: each token is conditioned on previous predictions. This design choice trades a **small compute overhead** for **orders-of-magnitude lower memory**.\n\n- **Action:** This discussion is added to the revision.\n\n\n\n**Q4:** Using linear mode connectivity (LMC) to evaluate model similarity.\n\n**A4:** Thanks for the comment. We conduct experiment as follows: \n- **Setting:** We use ResNet18 on CIFAR-10 with different training + sampling schemes, and test the performance of **linear model weights interpolation**:   \n    \n    |Method \\ Accuracy (%)|$A$|$0.5A+0.5B$|$B$|\n    |---|---|---|---|\n    |LMC of 2 models sampled by RPG trained on checkpoints from the same basin|95.1|95.0|95.0|\n    |LMC of 2 models sampled by RPG trained on checkpoints from 2 basins and sampled with 1 permutation state|95.0|94.8|94.9|\n    |LMC of 2 models sampled by RPG trained on checkpoints from 2 basins and sampled with 2 permutation states|94.9|18.4|94.9|\n\n\n**Q5:** Missing related work: A discussion on [1,2,3,4].\n\n**A5:** Thanks for the comment. We discuss papers [1,2,3,4] in our **A1** and **A2**, and we put the results here for convenience.\n\n|method|scalability|performance|generalization|\n|---|---|---|---|\n|$S_{KDE30}$|✖|✖|✖|\n|p-diff|✖|✔|✖|\n|SANE|✔|✖|✖|\n|D2NWG|✔|✔|✖|\n|[1]|✔|✖|✔|\n|[2]|✖|✔|✔|\n|[3]|✖|✔|✔|\n|[4]|✖|✔|✔|\n|RPG|✔|✔|✔|\n\n**Observations**:\n\n- Existing approaches tend to **specialize**: some stress **high performance**, others emphasize **generalization**, and few focus on **scalability**.\n\n- In contrast, our work can deliver strong performance, broad generalization, and efficient scalability **simultaneously**.\n\nWe add this discussion and the table above in the revision.\n\n\n[1] Babu, Sudarshan, Pedro Savarese, and Michael Maire. \"HyperNetwork Designs for Improved Classification and Robust Meta-Learning.\"  \n[2] Chen, Yinbo, and Xiaolong Wang. \"Transformers as meta-learners for implicit neural representations.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.  \n[3] Babu, Sudarshan, et al. \"Hyperfields: Towards zero-shot generation of nerfs from text.\" arXiv preprint arXiv:2310.17075 (2023).  \n[4] Lorraine, Jonathan, et al. \"Att3d: Amortized text-to-3d object synthesis.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n\n\n**Q6:** Is the permutation state essentially a one hot encoding on the trained networks used to train the hypernetwork?\n\n**A6:** Thanks for the comment. \n- We chose one-hot encoding because it offers a **straightforward** way to represent parameter symmetry.\n- However, the method does not depend on this particular choice: any encoding scheme that **distinguishes** different permuatation states is adaptable to our framework.\n- We add this clarification in **Section 2.2**, **line 117** in the revision."}, {"review_id": "CJSqfVMUZM", "reviewer": "Reviewer_DHqN", "summary": "The authors present a new method that tackles the problem of parameter generation, this is the problem of generating well performing network parameters that solve a certain task with no previous information on how to solve the task, that can scale and is more efficient that previous approaches at generating a large ammount of parameters, while being well performant.", "questions": "What is the impact of Nope or a random fixed positional encoding in section 3.3? If I understood correctly,  the 1D sinusoidal is independent of the network structure, and it does not have terrible performance; thus, I wonder what the impact of the positional encoding is in the learning process.\n\nCould you also provide more details on how permutation states are generated and how they work? When you train with permutation states, do you also permute the parameters used for training? Working like a data augmentation?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths:\n+ The paper is overall well written\n+ The method appears to significantly improve over previous works\n\t+ with a good comparison with previous works, and different models and problems\n\nweaknesses:\n+ lack of statistical significance over the results, showing only best, avg, min for the distribution of parameters generated by a single model\n+ no code release, and provided details are not enough to fully reproduce the experiments", "was_revised": true, "final_justification": "The authors addressed all my questions, specially about the significance of the results, as such I maintain the positive rating, as I consider the work offers an improvement from previous works.", "rebuttal": "**Q1:** Lack of statistical significance over the results, showing only the distribution of parameters generated by a single model.\n\n**A1:** We have conducted two additional repeated experiments (*model2* and *model3*) for Table 1. The original content in Table 1 corresponds to *model1* in the updated table. We now present the distribution of parameters generated by 3 independent models as well as the aggregated results across all 3 models (*all*). When time permits, we will also conduct repeated experiments for other key results such as Tables 2, 7, and 8.\n\n|arch. \\\\ acc.(%)|ResNet-18|ResNet-50|ViT-Tiny|ViT-Small|ViT-Base|ConvNeXt-A|ConvNeXt-L|\n|---|---|---|---|---|---|---|---|\n|parameters (M)|11.7|25.6|5.7|22.1|86.6|3.7|197.8|\n|original|70.0|79.8|74.9|81.4|84.4|75.2|85.8|\n|1-maximum|69.85|79.57|75.43|80.62|84.59|74.57|85.78|\n|1-average|69.49|79.51|75.29|80.47|84.39|74.39|85.49|\n|1-minimum|68.96|79.37|75.20|80.12|84.19|74.16|85.16|\n|2-maximum|69.70|79.68|75.38|80.79|84.62|74.60|85.82|\n|2-average|69.40|79.54|75.31|80.51|84.37|74.49|85.46|\n|2-minimum|69.04|79.44|75.20|80.39|84.17|74.38|85.17|\n|3-maximum|69.78|79.53|75.41|80.60|84.55|74.61|85.70|\n|3-average|69.43|79.48|75.30|80.46|84.38|74.48|85.43|\n|3-minimum|69.02|79.42|75.21|80.23|84.24|74.27|85.19|\n|all-maximum|69.85|79.68|75.43|80.79|84.62|74.61|85.82|\n|all-average|69.44|79.51|75.30|80.48|84.38|74.45|85.46|\n|all-minimum|68.96|79.37|75.20|80.12|84.17|74.16|85.16|\n\n\n\n**Q2:** No code release.\n\n**A2:** We will release all experimental codes in the revision. We guarantee that all experiments in the paper are reproducible.\n\n\n\n**Q3:** What is the impact of the positional encoding?\n\n**A3:** We address this question from two perspectives: *(1) the necessity of positional encoding*, and *(2) the choice between 1D and 2D positional encoding*.\n\n- **1. Necessity of positional encoding:** For Mamba-based recurrent models, positional encoding may be less critical since the model can inherently understand token order through its recurrent nature. However, for transformer-encoder-based models, positional encoding is essential to indicate which parameter position is being generated. For consistency across different architectures and framework flexibility, we uniformly apply positional encoding to all models.\n\n- **2. 1D vs. 2D positional encoding:** Under the current paradigm, there is no significant performance difference between 1D and 2D positional encoding. However, we plan to extend our framework to support variable-structure parameter generation in future work. The 2D encoding provides richer structural information about the model architecture, which will be crucial for these planned extensions.\n\n\n\n**Q4:** How are permutation states generated and how do they work?\n\n**A4:** Thanks for your comment. We explain in detail how permutation states are generated and how they work:\n\n- **How permutation states are generated:** We assign different permutation states to checkpoints with different symmetric states. Typically, multiple models saved in one training run have similar symmetric states (assigned the same permutation state), and vice versa. However, in this work, we assume that all checkpoints are in different symmetric states, which is actually a more conservative assumption, so we assign different permutation states to each checkpoint.\n\n- **How to choose permutation state during inference?** \n    - First, neural networks with **different initialization** can have various **symmetric structures** that **perform similarly**[1].\n    - This **symmetry phenomenon** doesn't affect performance, but hinders hyper networks' training. It happens when training with **different random seeds**.\n    - Based on this, we devise the **permutation state** to encode this symmetry and facilitate RPG's learning. That is, **assigning different permutation states** to **checkpoints trained with different random seeds**.\n    - Previous works such as P-diff [2] collect checkpoints from a **single training trajectory**, i.e., **one random seed**. Permutation state may not be **vital** in this setting.\n    - In inference, we only need to **randomly assign** permutation state.\n    - To further explore **permutation state's influence**, we conduct experiment in **Appendix C.1** and put it here for convenience:\n        - **Setting:** we use the ViT-T tuned on CIFAR-10, and test RPG with and without permutation state.\n           \n        | number of random seeds | original | without permutation state | with permutation state |\n        |-|-|-|-|\n        |1|88.2|88.0|88.1|\n        |3|88.3|fail|88.1|\n        |10|88.5|fail|88.2|\n      \n        - **Findings:**\n            - RPG w/, w/o permutation state performs **similar** when trained only on **one type of checkpoints**.\n            - When the number of **random seeds increases**, RPG without permution **soon fails**.\n            - RPG with permutation **works well** with **large number** of different random seeds.\n        - **Conclusions:**\n            - Permutation state encodes the **parameter symmetries** as conditions and works well when trained on checkpoints from **multiple training runs**.\n\n- **Overview of how permutation states work:** In the training phase, the permutation state (i.e., a one-hot encoding) passes through an MLP and is added to the positional encoding of the model input to represent the current model's symmetric state. In the inference phase, we typically only care about the model's performance rather than the model's symmetric state. Therefore, we only need to randomly select a permutation state (default is state #1, i.e., 000...001). Alternatively, we can specify a particular state to generate models in a specific basin.\n\n\n[1] B. Zhao et al. Symmetry in Neural Network Parameter Spaces  \n[2] K. Wang et al. Neural Network Diffusion"}, {"review_id": "ZOgDvRu9Ht", "reviewer": "Reviewer_yKm7", "summary": "The paper introduces Recurrent Diffusion for Large-Scale Parameter Generation (RPG), a novel framework for generating full neural network parameters (up to hundreds of millions) on a single GPU. By partitioning parameters into tokens, modeling inter-token relationships via a recurrent mechanism, and synthesizing parameters through diffusion, RPG achieves performance comparable to trained models across diverse architectures (ResNets, ViTs, ConvNeXts, LoRA-based LLMs).", "questions": "1. How does the method perform when generating parameters for neural network architectures not seen during training?\n2. What is the total computational overhead—including checkpoint collection and diffusion model training—compared to directly training a model on the target task, and under what conditions does this approach offer efficiency gains?\n3. Why does the generated parameter performance degrade significantly on unseen tasks (as shown in Table 8)?\n4. In what real-world scenarios would recurrent diffusion-based parameter generation be more advantageous than direct training, and are there measurable benefits (e.g., cross-architecture transfer, resource savings) that justify its use?", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper presents a well-structured framework with a clearly articulated methodology, including parameter tokenization and recurrent diffusion processes.\n2. The experimental validation is rigorous, encompassing diverse architectures (ResNets, ViTs, ConvNeXts, LLMs) and tasks (ImageNet, COCO, ADE20K, commonsense reasoning). Notably, the work bridges a critical scale gap by generating parameters for models with up to 200M weights.\n\nWeaknesses\n1. Limited architectural generalization: The method lacks validation on generating parameters for unseen neural network architectures, raising questions about its broader applicability.\n2. Computational cost transparency: The proposed approach relies on multiple pre-trained checkpoints as training data (e.g., 50 checkpoints for ImageNet-1K, lines 162–166). While the inference-time efficiency of parameter generation is highlighted, the total computational overhead—including (a) checkpoint collection and (b) diffusion model training—is not compared to directly training a model on the target task. This omission makes the method’s practical efficiency unclear.\n3. Performance gap on unseen tasks: As shown in Table 8, the generated parameters for novel tasks significantly underperform compared to standard training baselines.\n4. Unclear practical utility: The paper does not sufficiently justify why recurrent diffusion-based parameter generation is preferable to direct training in real-world scenarios. Concrete use cases or advantages (e.g., cross-architecture transfer, resource savings) remain unspecified.", "was_revised": true, "final_justification": "Please see my latest comment.", "rebuttal": "**Q1:** How does RPG perform on architectures not observed during training?\n\n**A1:** Thank you for the insightful comment. RPG targets functional generalization: **producing weights for unseen tasks while the network topology remains fixed.** Extending RPG to handle architecture changes belongs to **structural generalization**, a distinct research direction tied to design and hardware considerations. We therefore confine this study to a fixed architecture, where the challenges are already substantial and the practical value clear. We will clarify this limitation and future extension in the revision.\n\n\n**Q2:** The total computational overhead is not compared to directly training a model on the target task.\n\n**A2:** We acknowledge that collecting checkpoints and training RPG requires several hours or days, **but once trained, it can generate models for target tasks indefinitely.** The advantage of direct training on these tasks decreases as the number of tasks increases. \n\nThis is analogous to an image generator: once trained, it can create unlimited images cheaply, and we do not count manually drawing every potential image as part of its training cost.\n\n\n**Q3:** Why does the generated parameter performance degrade significantly on unseen tasks?\n\n**A3:** Thank you for the question. We clarify the details below.\n\n- **Unseen tasks are evaluated in a zero-shot setting**， so some decline relative to full-shot training is expected.\n \n- **The drop is modest**: RPG still outperforms random initialization both at the outset and after finetuning.\n\nConsequently, **RPG provides faster convergence and higher final accuracy**, as shown in Table 18. For your convenience, we reproduce the key numbers below.\n\n- **Setting:** We compare three approaches: (i) training ViT-Tiny from scratch, (ii) fine-tuning an ImageNet-pretrained ViT-Tiny, and (iii) fine-tuning an RPG-initialized model.\n\n    |epoch|from scratch|fine-tune|RPG init + fine-tune|\n    |---|---|---|---|\n    |0|50.0|52.9|94.4|\n    |1|61.4|90.3|96.8|\n    |5|69.1|96.3|97.3|\n    |10|74.9|97.1|97.5|\n    |50|86.7|97.7|97.8|\n\n- **Conclusion:** RPG significantly accelerates training for unseen tasks. **These results establish RPG initialization as an effective approach that substantially reduces training costs while maintaining competitive performance.**\n\n\n\n**Q4:** In what real-world scenarios would recurrent diffusion-based parameter generation be more advantageous than direct training, and are there measurable benefits that justify its use?\n\n**A4:** Thanks for the comment. RPG offers several practical advantages in real-world scenarios:\n\n- **Efficient initialization:** RPG provides superior parameter initialization compared to random weights, significantly reducing training time and computational costs as demonstrated in Table 18.\n- **Few-shot learning:** For tasks with limited training data, RPG-generated parameters serve as informed priors that enable better performance than training from scratch.\n- **Rapid model deployment:** RPG enables near-instantaneous model adaptation for new tasks without requiring extensive training, making it valuable for real-time applications where quick deployment is critical."}]

Author rebuttal:

Meta-review: This paper presents Recurrent Diffusion for Large-Scale Parameter Generation (RPG), a framework that combines parameter tokenization, a recurrent state-space model to capture inter-layer dependencies, and a diffusion process for sampling network weights. RPG achieves performance competitive with backpropagation-trained networks across a wide range of vision and language architectures (ResNets, ViTs, ConvNeXts, LoRA-augmented LLMs). Extensive experiments and ablations validate its design choices and highlight its scalability.

Reviewers appreciated the originality and breadth of the proposed approach, especially the novel combination of recurrence and diffusion for parameter synthesis. They praised the strong empirical results and coverage across multiple domains, along with well-motivated ablations and comparisons. Several reviewers emphasized that the paper pushes the boundary of what parameter-generating networks can achieve, noting successful single-GPU generation of ~200M-parameter models and compelling experiments on out-of-distribution tasks. The method was also described as well-structured and clearly presented, with some reviewers highlighting the potential for further research on generative hypernetworks.

However, reviewers also noted important weaknesses. Some challenged the paper’s core motivation, arguing that existing hypernetworks have already scaled to large models, and the claims that previous methods are limited may be overstated without direct comparisons. Others pointed out the lack of clarity in training vs. inference details, as well as the absence of key evaluations such as architectural generalization to unseen model types, or stronger statistical analyses of model diversity. The method’s computational footprint, including the use of many pretrained checkpoints for training, is underexplored, leaving the practical cost-benefit unclear. Additionally, some reviewers found that performance on unseen tasks (e.g., with new output heads) lagged behind standard training, and suggested the paper would benefit from clearer real-world use cases or scenarios where RPG offers a distinct advantage over traditional training.

The authors' extensive rebuttals resolve all major concerns, with multiple reviewers raising their scores post-rebuttal.   The paper received 1x accept, 2x borderline accepts, and 1x borderline reject.  For the lingering concern of the negative review, the authors clarified that RPG specifically targets functional generalization (producing parameters for unseen tasks while holding the architecture fixed), while structural generalization to unseen architectures is a separate research direction beyond the scope of this paper.  This distinction addresses one reviewer’s concern about unseen architectures and will be made explicit in the revision. The paper also details practical scenarios where RPG is beneficial: 1. efficient initialization, providing stronger starting points than random weights and reducing training costs, 2. few-shot learning, where RPG-generated parameters act as priors for data-scarce tasks, and 3. rapid model deployment, enabling near-instant adaptation to new tasks without retraining. These use cases strengthen the real-world relevance of RPG.

While there are reasonable critiques regarding cost-benefit transparency and reproducibility details (e.g., checkpoint usage, statistical reporting), reviewers widely recognized the originality and significance of the work, noting that it advances parameter-generating models well beyond prior methods and demonstrates strong functional generalization to new tasks within fixed architectures.  The consensus leans toward acceptance.
Decision: Accept (poster)