Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights
Authors: Konstantin Schürholt
Boris Knyazev
Xavier Giró-i-Nieto
Damian Borth
Venue: NeurIPS 2022 Accept (2022)
Keywords: Weight Generation
Representation Learning
Model Zoo
Hyper-Representations
Ensembling

Abstract: Learning representations of neural network weights given a model zoo is an emerg- ing and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation. Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation, which captures intrinsic and extrinsic properties of the models in the zoo. In this work, we ex- tend hyper-representations for generative use to sample new model weights. We propose layer-wise loss normalization which we demonstrate is key to generate high-performing models and several sampling methods based on the topology of hyper-representations. The models generated using our methods are diverse, per- formant and capable to outperform strong baselines as evaluated on several down- stream tasks: initialization, ensemble sampling and transfer learning. Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.

Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights
Konstantin Schürholt
Boris Knyazev b. k.yyazevssammsung com Samsung SAIT AI Lab, Montreal
konstantin. cecaatin.schhit Secuurroot@uuntt ch AIML Lab, School of Computer Science University of St.Gallen
Xavier Giró-i-Nieto
Damian Borth
xavier giroSupc...... edu Institut de Robòtica i Informàtica Industrial Universitat Politècnica de Catalunya
damian. borth borthuunisg. ch
AIML Lab, School of Computer Science University of StlGel
Abstract
Learning representations neural network weights given a model zoo is an emerg- ing and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation. Recently, an autoencoder trained model zoo able learn hypeerreerreenntatinn, which captures intrinsic and extrinsic properties of the models in the zoo. In this work, we ex- tend hyper-repressenttiions for generative use to sample new model weights. We propose layer-wis loss normalization which we demonstrate is key to generate perfforrfoning models and several sampling methods based on the topology of ryperrrepresennatioon.. The models generated using our methods are diverse, per- formant and capable to outperform strong baselines as evaluated on several down- stream tasks: initialization, ensemble sampling and transfer learning. Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-repressetattins thereby paving the avenue for novel research directions.
Introduction
Over the last decade, countless neural network models have been trained and uploaded to different model hubs. Many factors such random initialization and no global optimum ensure that the trained models are different from one another. What could we learn from such a population of neural network models? Since the parameter space of neural networks complex and high-dimensional. representation learning from such populations (often referred to as model zoos) has become an emerging and challenging area.
Recent work along that direction has demonstrated the ability of such learned representations to capture intrinsic and extrinsic properties of the models in 440 3r 27 According to 377 NNs populate a low dimensional manifold, which can be learned with an autoencoder via self-supervised learning directly from the model paramters (weights and biases) without access the original image data and labels. This so called hypee--rereseentalin has been demonstrated to be useful to predict several model properties such accuracy, hyperparameters architecture configurations.
focused on discriminative downstream tasks by exploiting the encoder only. We take However, 37
one step further and extend their work towards the generative downstream tasks by sampling model weights directly from the task-agnostic hypperrepreseentttoon To that end, we introduce layer-wise normalization that improves the quality of decoded neural network weights significantly. Based on
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Zoos
Hypar Representation
Sample Hyper- Reperstopresnntttion
Evaluate Generated Models
W
MINEST
0
ge
Po
dizn;
hpp
Compare performance
CIFANO10
w
diriia
21
dinta
wol
Layer-ou normalization
Drew samples in
Generaae Weights
Finetuning and transfer lear learning
Figure 1: Outline of our approach: Model zoos trained on image classification tasks. Hyper-representations trained with self supe-super learning the weights of the model zoos using layer-wise loss normalization in the reconstruction loss. We sample new embedkdings in syper-representation space and decode weights. Gen- erated models perform significantly better than random initialization or models sampled from baseline hyper- representations. Sampled models achieve high performance fine-tun fine-tuned and transfer learned on new datasets.
careful analysis of the geometry, smoothness and robustness of this space, we also propose several sampling methods to generate weights single forward pass from the repee-repressntation. We evaluate our approach on four image datasets and three generative downstream tasks of (i) model initialization, (ii) ensemble sampling, and (ii) transfer learning. Our results demonstrate its capability to out-perform previous sypee-representatioo learning and conventional baselines.
Previous work on generating model weights proposed (Graph) HyperNetworks 14 45 Bayesian HyperNetworks M HyperGANs [55] and Hyper Transforrass 46 for neural architecture search, model compression, ensembling. transfer- or meta meta-learning. These methods learn representations by using images and labels of the target domain. In contrast, our approach only uses model weights and does not need access to underlying data samples and labels an emergent use case, e.g. of deep learn- ing monitoring services model hubs. In addition to the ability to generate novel and diverse model weights, compared to previous works our approach (a) can generate novel weights conditionally on model zoos from unseen tasks and (b) can be conditioned on the latent factors of the underlying hyper- representation. Notably, both (a) and (b) can be done without the need to retrain hyper-representations..
The results suggest our approach (Figure M to be a promising step towards a general purpose iyper--epresentation encapsulating knowledge of model zoos to advance different down- stream tasks. The hyper-represennaaions and code to reproduce our results are available at https://gittae http/gtttuuu...// HSm/HSG- HHIHHHSGAAAMMMMIIIIEEII htt.://se..ppp.eeeeeaaii...pppep
2 Background: Training Hyper-Representation
We summarize the first stage of our method that corresponds to learning a hyper-representation of population of neural networks, called model z0o 37. In tyi and this paper, model zoo consists [23]. Specifically, a of M models {w.)M.
of models trained on the same task such as CIFAR- 10 image classification
ryperrepprsenntation is learned using autoencoder w h(g(w,)) where w, is the flattened vector of dimension N of all the weights of the i-th model. The encoder compresses vector w, to fixed-size hyper-representation z; (w.) of lower dimension. The decoder decompresses the hyper-representation to the reconstructed vector w. Both encoder and decoder are built on a self-attention block [I] The samples from model zoos are understood as sequences of convolutional or fully connected neurons. Each of the neurons is encoded as token embedding and concatenated to form a sequence. The sequence is passed through several layers of multi-head self-attention. Afterwards, a special compression token summarizing the entire sequence is linearly compressed the bottleneck. The output is fed through a tanh- activation to achieve a bounded latent space z; for the dyper-representation The decoder is symmetric to the encoder, the embeddings are linearly decompressed from hyper-representation z, and position encodings are added.
Training is done in a multi-oojecctve fashion, minimizing the composite loss L 3S..s (1- Las where Le is contrastive loss and L SaSE is weight reconstruction loss (see details in [37 We
can write the latter a layer-wise way to facilitate our discussion in $ 3.T
LMSE = MN
--1 E'
Σ M
|| (1)
(
where (1) w! are reconstructed and original weights for the 1-th layer of the i-th model in the zoo.
The contrastive loss L. leverages two types of data augmentation at train time to impose structure on the latent space: permutation exploiting inherent symmetries of the weight space and random erasing.

---

Methods
In the following, we present layer-wise loss normalization ensure that decoded models are performant, and (ii) sampling methods to generate diverse populations of models.
3.1 Layer- Wise Loss Normalization
We observed that hyper-representttioon as proposed by 67 decode to dysfunctional models, with per- formance around random guessing. To alleviate that, we propose novel layer-wise loss normalization (LWLN), which we motivate and detail in the following.
Weights Weights
Layer
Layer
Layer
Layer 5
0.2
0.4
0.6
Accuracy
Comparison the distributions SVHN 2oo weights w (blue) reconstructed weights (orange) as well as their test accuracy on the SVHN test set. Top: Baseline hyper-representation as proposed by [37], the weights of layers 3. 4 collapse to the mean. These layers form weak link in reconstructed models. The accuracy of reconstructed models drops random guessing. Bottom: repper-eprrsenntatinn trained with layer-wise loss normalization (LWLN). The normalized distributions are balanced, layers are evenly reconstructed, and the accuracy of reconstructed models significantly improved.
Figure 2:
Due to the MSE training loss in (I. the reconstruction error can generally be expected to be uniformly distributed over all weights and layers of the weight vector w. However, the weight magnitudes of many of our 2oos are unevenly distributed across different layers. In these zoos, the even distribution of reconstruction errors lead undesired effects. Layers with broader distributions and large-mage maggnitde weights are reconstructed well, while layers with narrow distributions and maal--aannitue weights are disregarded. The latter layers can become a weak link in the reconstructed models, causing performance to drop significantly down to random guessing. The top row of Figure 2 example of a baseline hyper-representation learned on the zoo of SVHN models initialization schemes [15 [1] produce distributions with different scaling factors per layer, so the issue is not an artifact of the zoos, but can exist in real world model populations. Similarly, recent work on generating models normalizes weights to boost performance 21I. In order to achieve equally accurate reconstruction across the layers, we introduce a layer-wise loss normalization (LWLN) with the mean ir and standard deviation o; of all weights in layer estimated over the train split of the zoo:
shows an
32 Common
M
(1)
M
14
MABE = MN
MN
i=1
3.2
Sampling from Ixyper-Representations
We introduce methods draw diverse and high-quality samples x² p(z) from the learned hyper- representation space to generate model weights w l((z).. Such sampling is facilitated if there knowledge the topology of the space spanned by z. One way achieve that is train variational autoencoder (VAE) with a predefined prior 20 instead the autoencoder of 37 While training VAEs on common domains such as images has become well-understood and feasible, in our relatively novel weight domain, we found it problematic (see details in Appendix D). Other generative methods avoid a predefined prior of VABs, either by analyzing the topology of the space learned by the autoencoder or fitting separate density estimation model on top of the learned representation 2 [3]
These methods assume the representation space to have strong regularities. The hyper- syper-representation space learned by the autoencoder of 3V is already regularized by dropout regularization applied to encoder and decoder as I0 The contrastive loss component requiring similar models to be embedded close to each other may also improve the regularity of the representation space. Empirically, we found our layer-wise normalization (LWLN) to further regularize the representation space by
ensuring robustness and smoothness (see Figure in

---

Given the smoothness and robustness of the learned ryper-representation space, we follow [26 13 TO in estimating the density and topology draw samples from regularized autoencoder. To that end. we introduce three strategies to sample from that space: SKDE. SNigg, SGAN- To model the density and topology in representation space, we use the embeddings of the train set as anchor samples {z:}, We observe that many anchor samples from (2,) correspond to the models with relatively poor accuracy (Figure 2 so to improve the quality of sampled weights, we consider the variants of these methods us- ing only those embeddings of training samples corresponding to the top 30% performing models. We denote these sampling methods as SKDE30, SNeigh30, SGAN30 respectively. These methods can poten- tially decrease sample diversity, however, we found that the generated weights are still diverse enough (e.g. to construct sigh-performant ensembles, Figure 5). Finally, as baseline and sanity check we explore sampling uniformly in representation space Su and sampling in low-pre probability regions Sc-
3.2.1 Uniform Sy
As a naive baseline, we draw samples uniformly in byper-representation space (bounded by tanh, $ and denote it as Sy. This is naive, because we found that the embeddings populate only sections of shell of a nigh-dimensional sphere (see Figures and 12 in Appendix D) So most of the uniform sam- ples lie in the low cow-rrobability regions of the space and not expected to be decoded to useful models.
3.2.2 Density estimation SKDE and counterfactual sampling Sc
The dimensionality D of hyper-reprresentations z in 37 as well as in our work, is relatively high due to the challenge of compressing weights w. Fitting probability density model to such a high-dimensional distribution feasible by making a conditional independence assump- tion: (x2)) w) = p(z(5) ww) where z(5) is the j-th dimensionality of the embedding z. To model the distribution of each j-th dimensionality, we choose kernel density estimation (KDE), as it is a powerful yet simple, non- can-aaaamttri and deterministic method with a single hyperparameter. (i) M
We fit KDE to the M anchor samples {z, Ti=l of each dimension j, and draw samples 2 203 from
that distribution: 201) p(z())) Mh Σ Ei=1 K ()) ())
where K(x>) (2m)-1/2 exp (- is the Gaussian kernel and is a bandwidth hyperparameter. The samples of each dimension =00) are concatenated form samples z [z 222) 2')) This method is denoted as SKDE-
As a sanity check, we invert the SKDE method and explicitly draw samples from regions not populated by anchor samples, i.e. with low probability according to the KDE. This method, denoted Sc. essentially samples counterfactual embeddings and similarly to Sy is expected to perform poorly.
3.2.3 Neighbor sampling SNeigh
Sampling neighbors of anchor samples {z.} could be simple and effective sampling strategy. but due to high sparsity of the hyper-representation space this strategy results in poor-quality samples. We therefore propose to use neighborhood-bane based dimensionality reduction function k RD - Rd that maps z; to low-dimensional embeddings n, E Rd where sampling is facilitated. The assumption is that due to the low dimensionality of R. (we choose d = 3) there will be fewer low-probability regions, so that uniform sampling in RRd can be effective. Specifically, given low-dimensional embeddings n; k(z.), we sample n² uniformly from the cube: n' (miin(n), mar(n)). Samples n' are then mapped back to hyper-representations z' neighborhood topology of RDD in Rd and enable mapping back to RD, we choose & to be an approximate inverse neighhorhood-basee dimensionality reduction function based UMAP 28
(n*). To preserve the
3.2.4 Latent space GAN SGAN
A common choice for generative representation learning is generative adversarial networks (GANs) (12]. While training a GAN directly to generate weights is a promising yet challenging av- enue for future research G51, we found the GAN framework to work reasonably well when trained on
the hyper-representations. This idea follows 26 [
that showed improved training stability and effi- ciency compared to training GANs on inputs directly. We train generator G Rdd - RDD with x = G(n") to generate samples in hyper-representation space from the Gaussian noise n We choose d 16 compromise between size and capacity. See detailed architecture of our GAN in Appendix

---

Experiments
4.1 Experimental Setup
We train and evaluate our approaches on four image classification datasets: MNIST 24 SVHN 32 For each dataset, there is model zoo that we use to train an autoencoder
CIFAR-10 23], STL-10 [ following
Model zoos: In practice, there are already many available model zoos, e.g., on Hugging Face or GitHub, that can be used for hyper-representation learning and sampling. Unfortunately, these zoos are not systematically constructed and require further effort to mine and evaluate. Therefore. in order control the experiment design, ensure feasibility and reproducibility, we generate novel or use the model zoos of B7 GN created in a systematic way. With controlled experiments, we aim to develop and evaluate inductive biases and methods to train and utilize typer-representation, which can be scaled up efficiently to large-scale and -ystemtttaaly constructed zoos later. For each image dataset, a zoo contains M 1000 convolutional networks of the same architecture with three convolutional layers and two fully-connected layers. Varying only in the random seeds, all models of the zoo are trained for 50 epochs with the same hyperparameters following [37] To integrate higher diversity in the 200, initial weights are uniformly sampled from wider range of values rather than using well-tumed initializations of 11 Each zoo is split in the train (70%), ) validation (15%) and test (15%) splits. To incorporate the learning dynamies, we train autoencoders trained for 21-25 epochs following [37]. Here the models have already achieved high performance, but have not fully converged. The development in the remaining epochs of each model are treated as hold-out data to compare against. We use the MNIST and S SVHN zoos from (37 and based on them create the CIFAR-10 10 and STL-10 zoos. Details on the zoos can be found in Appendix A
the models
Experimental details: We train separate hyper-represennttioon on each of the model zoos. Images and labels are not used to train the hypper-reprssennttioon (see $ ( Using the proposed sampling methods ($ 3.2). we generate new embeddings and decode them to weights. We evaluate sampled populations initializations (epoch 0) and by fine-tuning for up to 25 epochs. We distinguish between in-dataset and transfer-lcarning For in-dataset, the same image dataset is used for training and evaluating our lyper-representations and baselines. For transfer-learning, ryperrrepressntatiios (and pre-trained models in baselines) are trained on a source dataset, then all populations are evaluated and fine-tuned and compute is detailed in the Appendix
different target dataset. Full details on training, including infrastructure
Baselines: As the first baseline, we consider the autoencoder of 37 which is same ours but without the proposed layer-wise loss-normalization (LWLN, $ 3.I. We combine this autoencoder with the SKDV3 sampling method and, hence, denote it as BRDEAO We consider two other baselines based on training models with stochastic gradient descent (SGD): training from scratch on the target classification task By, and training latter remains one of the strongest transfer learning baselines
source followed by fine-tuming the target task Bp. The
22].
Reproducibility, reliability and comparability: We compare populations of at least 50 models to evaluate each method reliably. We report standard deviation Tables 22 and statistical significance, effect size and 95% confidence interval in Appendix [F] To ensure fairness and comparability, all methods share training hyperparameters. Fine-tuning uses the hyperparameters of the target domain.
4.2 Results
In the following, we first analyze the learned repeetrrpresentationss further justifying our sampling methods and assumptions made $ 3.2 We then confirm the effectiveness of our approach for model initialization without and with fine-tuning in the in-dataset and transfer learning settings.
4.2.1 Hyper-Representations are Robust and Smooth
We evaluate the robustness and smoothness of the on the SVHN zoo. First, to evaluate robustness, we add different levels of noise to the embeddings of the test set to create 2, decode them to model weights w and compute models' accuracies on the SVHN classification task. We found that both the baseline as well as our hyper-representations are robust to noise as large levels of relative noise >10% are required to affect performance (Figure S a.c). Second, to probe for smoothness, we linearly interpolate between the test set embeddings (i) along
space with two experiments
5

---

Baseline hyper-reprresentatinn
Our hyper-representtaion
Accuracy
Accuracy 02
0.0
05
10
lhe all meant]eel
0.
1.0
11
0.12
84
-03
-05
10-2 [[-21) mesntllagi
10-1
Accuracy Buus
[22 meanti]elll
Accuracy drus -212..m
(b)
(d)
Figure 3: (a,c): Robustness of hyper-represeentation.. For both baseline and our hyper-representation, relatively large levels of relative noise >10% are necessary to degrade the test accuracy (orange) or reconstruction (blue); see the text for further discussion. (b,d): Interpolations along model trajectories (orange) and between z of different models (blue) show the smoothness of our hyper-representation.
the trajectory the same model at different epochs (2, and ep25)) and (ii) between 250 random pairs of embeddings the trajectories of different models (z, and 2,). We decode the interpolated embeddings and compute models' accuracies on the classification task. For our model, we found remarkably smooth development of accuracy along the interpolation in both schemes (Figure 3 d).
The lack of fluctuations along and between trajectories support both local and global notions of smoothness in rypee-representatiin space.
For the baseline autoencoder (without LWLN) decoded models perform close 10% accuracy, so these representations do not support similar notions of smoothness (Figure b). while robustness can be misleading, since accuracy even without adding noise is already low (Figure a). Therefore, LWLN together with regularizations added to the autoencoder allow for learning robust and smooth nyper-representation. This property makes sampling from that representation more meaningful as show next.
4.2.2 Sampling for In-dataset Initialization
Comparison between sampling
methods: We evaluate the performance of different sampled populations (ob- tained with LWLN) without fine-tuning generated weights. On MNIST, all sampled models except those obtained using Sy and Se perform better than random initialization (10% accuracy), worse than models trained from scratch By for 25 epochs (Figure 4b. Distribution-bus based samples (SKDE and SGAN) perform better than neighborhood based samples (SNeigh). The populations based on the top 30% perform better than their 100% counterparts with SKDE30 the strongest sampling method overall. This demonstrates that the learned typer-representation and sampling methods are able to capture complex subtleties in weight space differentiating high and low performing models.
0.1
Accuracy
0.6
0.4
0.0
Sc
Su
Swegs Sveign0
SeaN
SSAV20
SEDE
SIDE30
Br
Figure 4: MNIST results of sampled weights (no fine- tuning) compared to training from scratch with SGD (Br).
Comparison to the baseline hy peerrrrpreeeenttionn:: We also compare SKDE30 that is based autoencoder with layer-wise loss normalization (LWLN) to the baseline autocncoder using the same sampling method (ВкDEзо) without fine-tuning, On all datasets except for MNIST, SKdE30 consid- erably outperform BRDES0 with the latter performing just above 10% (random guessing), see Table (rows with epoch 0). We attribute the success of LWLN two main factors. First, LWLN prevents the collapse of reconstruction to the mean (compare Figure top to bottom). Second, by fixing the weak links, the reconstructed models perform significantly better (see Appendix Sendiixxfor for more results).
In-dataset fine-tuning: When fine-tuning, our SKDE30 and baseline BXDE30 appear to gradually converge to similar performance (Table M. While unfortunate, this result aligns well with previ- ous findings that longer training and enough data make initialization less important 60
34

---

We also compare SKDE30 and BRDESO to training models from scratch (Br). On all four datasets, both ours and the base- line hyper-represenntaioos outperform Br when generated weights are fine-tuned for the same number of epochs as Br. Notably, on MNIST and SVHN generated weights fine-tuned for 25 epochs are even better than By run for 50 epochs. Comparison to 50 epochs is more fair though, since the repeerrepresentationns were trained on model weights trained for up to 25 epochs. These findings show that the models ini- tialized with generated weights learn faster achieving better results in 25 epochs than Br in 50 epochs.
Table 1 Mean and std of test accuracy (%) of sampled populations with LWLN (SXDE30) and without (BxDE30) compared to models trained from scratch Br. Best results for each epoch and dataset are bolded.
Method
Ep.
MNIST
SVHN
CIEAR-10
STL-10
Br
2:10% (randomn guessing)
PRDESO
0
63,2 7.2
10.1 3,2
15.5
1.77+
Skeeeo
68.6
599
26.9 4,9
19.7 2.1
By
20.6 1.6
27.5 1.66
15.4 1.8
Buneso
83.2
SKDE3D
83.7 1.3
69.9 1.6
44.0 0.5
25.9 1.6
Br
25
83.3 2.6
6..7
46.1
3 1.3
Brcess
25
93.2 0.6
75.4 0.9
48.1 + 0.6
38.4 0.9
Skereo
25
0.7
74.2
Br
50
91.1 = 2.6
70.7 a 8.8
48.7 + 1.4
39.0 1.0
Sampling ensembles: We found that potentially useful by- product learning hyper-representations high-performant ensembles at almost no extra computational cost,
the ability to generate
0.8
Accuracy
0.6
By
since both sampling and generation are computationally cheap. To demonstrate this effect, compare ensembles formed using the baseline autoencoder (ВxDE30) and ours (SKDE30) to the ensembles composed of models trained from scratch for 25 epochs (BT) on SVHN. Ensembles generated using the baseline BRDESO stagnate below 20% (Figure 5). In contrast, ensembles generated using SKDES0 gracefully improve with the ensemble size outperforming single Br models and almost matching By ensembles with enough models in the ensembles. Remarkably, the average test accuracy of generated ensembles of 15 models is 77.6%, which considerably higher than 70.7% of models trained on SVHN for 50 epochs. We conclude that hyper-reprresentations learned with LWLN generate models that are not only performant, but also diverse. Although generating ensembles requires learning typer-representation and model zoo first, we assume that in future such a hyper--epresenttttion can be trained once and reused in unseen scenarios as we tentatively explore below (see results in Table B and the discussion therein).
SKDE30
0.4
B BKDE30
0.2
10
20
No. of Models
Figure 5: Generated ensem- bles evaluated on SVHN. Test accuracy is averaged over 15 ensembles of randomly cho- sen models.
Do reconstructed models become similar to the original during fine-tuning? Sampled representations often learn faster and to higher performance than the pop- ulation of models they were trained on (Table D. We therefore explore the question, if reconstructed models develop in weight space in the same direction their original, or find different solution. On SVHN, we found that the reconstructed models (W) after one epoch of fine-tuning per- form similar to their originals (w) and slightly outperform from there on (Figure 6 left). At the same time, pairs of original and reconstructed models move further apart and become less aligned in weight space (Figure g right). It appears that reconstructed models perform better and explore different solutions in weight space to do so. This confirms the intuition that riyreerrprsssnttttion impress use- ful structure on decoded weights. A pass through encoder and decoder thus results not just a noisy reconstruction of the original sample. Instead, it maps to different region on the loss surface, which leads to faster learning and better solutions. Combining this with the ensembling results in Figure 5 hyper-representations do not collapse to single solution, but decode to diverse and useful weights.
0.6
Accuracy
Distance
hyper-
0.6
0.4
|| w////|
acc(w)
cos(w,
acc(w)
0.2
0.4
10
20
10
20
Epochs
Epochs
Figure 6: Progression of test accuracy (left) and distance (right) between weights during fine-tuning SVHN; w initialization with the weights trained using SGD for 25 epochs; w initializaaion with reconstructed weights.
7

---

4.2.3 Sampling Initializations for Transfer Learning
Sctup: We investigate the effectiveness of our method in transfer-learning setup across image datasets. In particular, we report transfer learning results from SVHN to MNIST and from STL-10 to CIFAR-10 10 as two representative scenarios. Results on the other pairs of datasets can be found in Appendix F In these experiments, pre- pre-trained models BF and the hyper-representation model are trained on source domain. Subsequently, the pre-trained models Br and the samples SKDE- SNeigh and SGAN are fine-tuned on the target domain. The baseline approach (Br) is based on training models from scratch on the target domain.
Table 2: Transfer-leaming results (mean and standard deviation of the test accuracy in %). Note that for STL-10 to CIFAR-10 10 the performance of all methods saturate quickly due to the limited capacity of models in the zoo making further improvements challenging as we discuss in
Method
SVHN to MNIST
STL-10 to CIEAR-10
Ep. 0
Ep.
Ep. 50
Ep. 0
Ep. 1
Ep. 50
Br Br
10.0 0.6
20.6 1.6 8.44
91.1 =
10.1 + 1.3
27.5 + 2.1
48.7 + 1.4
5.4
95.0 0.8
15.3 2.3
29.4 + 1.9
49.2 0.7
SKDESO
5.6
86,9 1.4
95.5 + 0.4
14.5
29.6
48.8 + 0.9
SNeigha
10.7 2.7
3.3
95.5 + 0.7
10.1 2.1
29.2 + 1.9
48.9 + 0.7
SCANSO
1.4
6.3
94.9 0.7
10.22255
28.6 + 1.8
48.8 0.8
Results: When transfer learning is performed from SVHN to MNIST, the sampled populations on average learn faster and achieve significantly higher performance than the By baseline and generally compares favorably to By (Figure 7 Table 2 In the STL-10 10 to CIFAR-10 experiment, all populations appear to saturate with only small differences their performances (Table D.. Different sampling methods perform differently the beginning versus the end of transfer learning. Generally. SKDEM performs better in the first epochs, while all methods perform comparably at the end of transfer-learning. These discrepancies underline the difficulty of developing single strong sampling method, which is an interesting area of future research. We further found that all datasets are useful sources for all targets (see Appendix F). Interestingly and other than in related work 29 even transfer from the simpler to harder datasets (c.g., MNIST to SVHN) improves performance. This capture generie inductive prior useful
might be explained by the ability of hyper-represeetations across different domains, which we further investigate next.
Conditioning on unseen Zoos: We explore if the hyper-repressentaiin trained on the models of one zoo (e.g. MNIST) can reconstruct the weights of another unseen zoo (e.g. SVHN). This can be useful to enable generation of weights for novel tasks without the need to retrain a hyper-repressenttion. This is analo- gous to cestance-conditionee GANs that recently were able to generate images from unseen do- mains without retraining GANs [2]. Our results
Table 3: Test accuracy (%) of models generated conditioned on the models of unseen zoos.
Truining
Conditioning
(bokled) accuracy
Z00
(unseen)
One model
Ensemble
MNIST
SVHN
12.7 19.8
13.4 18.7
SVHN
MNIST
16.2 26.0
22.1 / 29.8
CIFAR-10
STL-10
18.0 24.4
23.8 26.7
STL-10
CIFAR-10
16.3 21.2
20.0 23.0
in Table g show that while the performance on the unseen zoos reduced, it is still well above random guessing (10%), especially when multiple model weights are sampled and ensembled. This promising, as the hyper-representations were trained on single-dataset zoos.
4.2.4 Sampling Initializations for Unseen Architectures
Generalization to unseen large architectures with complex connectivity (ResNet, MobileNet, and EfficientNet) is a very interesting and ambitious research problem. As a step towards that goal, we perform experiments in which we attempted to use our hyper-represennation beyond the same simple architecture. Surprisingly, our results indicate the promise of leveraging the hyper-repressenttion for more diverse architectures and settings. Further experiments investigating the cross-architecture generalization capabilities of hyper-representations can be found in Appendix D

---

1.0
0.8
BT
0.6
Steigh30
0.4
SXOC30
0.2
SGAN30
0.8
10
20
30
40
50
Epochs
Epochs
Figure 7: SVHN to MNIST transfer learning experiment: test accuracy epochs. Our sampling methods outperform the baselines after the first epoch. Left: epochs from 0 to 50, Right: epochs from 3 to 9, where By is significantly lower than 80% and thus is not visible.
With this experiment, we aim to verify if it is possible to adapt our approach to architectures seen during training, e.g., with skip connections and/or with more layers. We follow the transfer- learning setup of $ 4.2.3 and use an existing MNIST represetpptnnnnnioonn to sample weights as initializiation for training on SVHN. However, now also vary the architecture. While the decoder outputs fixed-sized vector of weights, we can assign these weights to new architectures by either making sure that the new architecture still has the same number of parameters or by initializing randomly the extra parameters introduced. Specifically, we create three cases: (1) we add ResNet-style skip connections 16 (1x1 conv) the convolutional layers (3-conv + res-skip), (2) re-distribute the weights to smaller four convolutional layers (4-conv), (3) re-distribute smaller four convolutional layers and add identity skip connections (4-conv id.-skip).
Setup:
Results: Surprisingly, despite train- ing our hyper-representation on the models of the same architecture, generated weights for all three cases outperform random initialization and converge significantly faster across all the variations (Table H. In all the variations even just after epochs the models with generated weights are better than training 50 epochs. In the 3-conv + res-skip experiments, some models in both populations did not learn, which leads to high standard deviation. Further analysis is required to explain the gains of our approach in this challenging setup. To extend and scale up our method further, future work could combined it with the methods of growing networks 3 [2. so that some layers are generated while some are initialized a sophisticated way to preserve the functional form the network.
Table 4: Test accuracy (%) on SVHN of populations with generated weights compared to models trained from scratch By- Best results for each epoch and dataset are bolded. r.
i. indicates random initialization, gen. denotes weights generated with our (SKDE30).
Initialization
Epoch
Epoch
Epoch 50
3-conv (r. i.) res-skip (t. i.) 18.9 1.6
31.4 17
50.6 + 28
3-conv (gen.) res-skip (r. i.) 34.5 # 14
60.5 21
68.0 + 21
baseline for
4-conv (r. i.)
19.2 = 1.0
19.2
55.2 = 11
4-conv (gen.)
44.0 + 4.5
57.8
67.6 + 199
4-conv + id.-skip (r. i.)
18.9 1.0
19.6
56.9 66.4 1.7
4-conv + id.-skip (gen.) 48.0 + 4.0
59.9 2.5
4.3 Limitations of Zoos with Small Models
To thoroughly investigate different methods and make experiments feasible, chose the model zoos of the same small scale as in (37. While on MNIST and SVHN, the architectures such achieve high performance,
model zoos allowed all populations is limited by the low capacity of the models zoo's 's architecture. The models saturate around 50% and 40% accuracy, respectively. The sampled populations reach the saturation point and fluctuate, but cannot outperform the baselines, see Appendix [ for details. We hypothesize that due the high remaining loss, the weight updates are correspondingly large without converging or improving performance. This may cause the weights to contain relatively little signal and high noise. Larger model architectures might mitigate this behaviour. Corresponding model zoos have recently been made available in 38 to tackle this issue
CIFAR-10 10 and STL-10,, the performance of
www.modelzoos.co

---

Related Work
HyperNetworks: Recently, representation learning on neural networks is typically based on HyperNetworks that learn low-dimensional structure of model weights to generate weights in deterministic fashion [4 2 45 HyperNetworks have also been extended to meta-learning by conditioning weight generation on data A. Go. Closely related to our work, HyperGANs [355] can sample model weights by combining the hypernetworks and the GAN framework. Similarly, allow for sampling model weights by conditioning the hypernetwork training hypernetwork-basee
noise vector. However, methods require input data (e.g- images) to feed to the neural networks. In practice, there may already be large collections of trained models, while their training data may not always be accessible. Learning representations of model weights without data, called hyper-representation.. has been recently introduced in tr Our methods build that work to allow for better reconstruction and sampling. 7 showed that given a few parameters of a network, the remaining values of single model can be accurately reconstructed. However, in our work leverage the autoencoder to train a representation of the entire model zoo. Very recently, [33] use diffusion population of models to generate model weights for the original task via prompting.
Transfer learning via fine-tuning aims at re-using models and their learned knowledge from a source to target task [44] 4 9 29 [22]. Transfer learning models makes training less expensive, boosts performance, or allows to train on datasets with very few samples and has been applied on wide range of domains [S The common transfer learning methods however only consider transferring from a single model, and so disregard the large variety of pre-trained models and potential benefit of combining them.
Transfer Learning:
that allow to distill knowledge from a model zoo into a single network. Knowledge distillation overcomes the inherent limitation of transfer learning by transferring the knowledge from many large teacher models a relatively small student Knowledge distillation however requires the source models at training as in [25] and at inference as in 39 thus increasing memory cost. Further, the learned knowledge cannot be shared between different target models. Learnable initialization 47 provide methods to improve initialization by leveraging the meta-learning and gradient-flo ideas. In contrast to knowledge distillation and learnable initialization, space, which a more general and powerful approach that can enable sampling an ensemble, property estimation, improved initialization and implicit knowledge distillation across datasets.
Knowledge distillation:
Our work related to 4 25
model 25 39
train reperrrrpreeentttion of model zoo in latent
6 Conclusion
In this paper, we propose a new method to sample from hyper-representations to generate neural network weights in one forward pass. We extend the training objective of hyper-representations by novel layer-wise loss normalization which is key to the capability of generating functional models. Our method allows us to generate diverse populations of model weights, which show high performance as ensembles. We evaluate sampled models both in-dataset as well as in transfer learning and find them capable to outperform both models trained from scratch, well as pre-trained and fine- tuned models. Populations of sampled models, even for some unseen architectures, generally learn faster and achieve statistically significantly higher performance. This demonstrates that such hyper- representation can be used generative model for neural network weights and therefore might serve as building block for transfer learning from different domains, meta learning or continual learning.
Acknowledgments
This work was partially funded by Google Research Scholar Award, the University of St.Gallen Basic Research Fund, and project PID2020-117142GBB funded by MCIN/ AEI 130.13039/501 10001 1033. We are thankful to Michael Mommert for editorial support.
10

---

References
[1] Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed-forward one-shot learners. Advances in Neural Information Processing Systems, 29, 2016.
100
[2] Arantxa Casanova, Marlène Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero So-
riano. Instance-conditioned gan. Advances in Neural Information Processing Systems, 34, 2021.
[3] Tianqi Chen, lan Goodfellow, and Jonathon Shlens. Net2Net: Accelerating Learning via Knowledge Transfer. In International Conference on Learning Representations (ICLR), April
2016. doi: 10.48550/arXiv. 15t 15111111155550 1.05641. g
[4] Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. International Conference Learning Representations (ICLR),
2019.
5 I0
[5] Adam Coates, Honglak Lee, and Andrew Ng. An Analysis of Single-Layer Networks in Unsupervised Feature Learning. In Proceedings of the 14th International Con- Ference on Artificial Intelligence and Statisties (AISTATS), page 9, 2011.
[6] Yann N Dauphin and Samuel Schoenholz. Metalnit: Initializing learning by learning to initialize.
In Neural Information Processing Systems, page 13, 2019. TO
[7] Misha Denil, Babak Shakibi, Laurent Dinh, and Mare' Aurelio Ranzato. Predicting Parameters
in Deep Learning. In Neural Information Processing Systems (NeurlPS), page 9, 2013. TO
[8] Lior Deutsch. Generating Neural Networks with Neural Networks. arXiv:180) 01952 [cs, stat],
April 2018.
2
LC
[9] Guneet $ Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. International Conference on Learning Representations (ICLR),
2019.
IC
[10] Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From Variational to Deterministic Autoencoders. In arXiv: /2436 [Cs, Stat], May 2020.
21
[11] Xavier Glorot and Yoshua Bengio. Understanding the diffieulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statisties, PMLR, page 8, 2010.
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Orair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Conference on Neural Information Processing Systems (NeurIPS), page 9, 2014.
[13] Yong Guo, Qi Chen, Jian Chen, Qingyao Wu, Qinfeng Shi, and Mingkui Tan. Auto-embedding generative adversarial networks for high resolution image synthesis. IEEE Transactions on Multimedia, 21(11):272--273 2019.
[14] David Ha, Andrew Dai, and Quoe V. Le. HyperNetworks. In 1rX1v:1609.09106 [Cs], 2016.
10
[15] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers:
Surpassing Human-Leve Performance ImageNet Classification. In arXiv:1502.01852 [Cs],
2015.
S
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition 2016
[17] Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4918-4927, 2019. 3
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 21
[19] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding Dimensional Collapse in Contrastive Self-supervised Learning. In International Conference on Learning Representa- tions, September 2021.
19

---

[20] Diederik P. Kingma and Max Welling, Auto- Eatooinn Variational Bayes. In International Conference on Learning Representations (ICLR). 2013. 2T
[21] Boris Knyazev, Michal Drozdzal, Graham W. Taylor, and Adriana Prediction for Unseen Deep Architectures. In Conference on Neural Information Processing
Parameter
Systems (NeurIPS), 2021.
TO
[22] Alexander Kolesnikov, Lucas Beyer. Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European
conference on computer vision, pages 491-507. Springer, 2020. 5 TO
[23] Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. page 60, 2009.
[24] Y. LeCun, I.. Bottou, Y. Bengio, and P. Haffner. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(1 (0(11):2278222 2324, November 1998. 5
[25] Iou-Jen Liu, Jian Peng, and Alexander G. Schwing. Knowledge Flow: Improve Upon Your Teachers, In International Conference on Learning Representations (ICLR), April 2019. TC
[26] Jinlin Liu, Yuan Yao, and Jianqiang Ren. An acceleration framework for high resolution image synthesis. arXiv preprint arXiv: 1909003699,,,,, 2019. 3
[27] Charles Martin, Tongsu Serena Peng, and Michael W Mahoney. Predicting trends in the quality of state-of-the-arr neural networks without access to training or testing data. Nature
Communications, (211):1-13, 2021. [
[28] Leland Mclnnes, John Healy, and Nathaniel Saul. UMAP: Uniform Manifold Approximation and Projection. 2018.
[29] Thomas Mensink, Jasper Uijlings, Alina Kuznetsova, Michael Gygli, and Vittorio Ferrari. Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Types.
1333 [cs], November 2021. g 10
[30] Dmytro Mishkin and Jiri Matas. All you need is good init. In International Conference on Learning Representations (ICLR). arXiv, 2016. doi: www.55//ar.i.1111...0. 1.6422. C
[31] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normaliza- tion for Generative Adversarial Networks. In International Conference on Learning Represen- tations (ICLR), 2018. 21
[32] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, page 9, 2011.
[33] William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A. Efros, and Jitendra Malik. Learning
to Learn with Generative Models of Neural Network Checkpoints, September 2022. TT
[34] Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. Advances in neural information processing systems,
28, 2015. 3
[35] Neale Ratzlaff and Li Fuxin. HyperGAN: A Generative Model for Diverse, Performant Neural Networks. In Proceedings of the 36th International Conference on Machine Learning, pages 5361-5369. PMLR, May 2019.
10
[36] James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems, 32, 2019. I0
[37] Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction. In Conference Neural Information Processing Systems (NeurIPS), volume 35, 2021.
[38] Konstantin Schürholt, Diyar Taskiran, Boris Knyazev, Xavier Giró-i-Nieto, and Damian Borth. Model Zoos: A Dataset of Diverse Populations of Neural Network Models. In Thirty-Sixth Conference on Neural Information Processing Systems (NeurIPS) Dutasets and Benchmarks Track, September 2022.
16
[39] Yang Shu, Zhi Kou, Zhangjie Cao, Jianmin Wang, and Mingsheng Long. Zoo-Tuning: Adaptive Transfer from a Zoo of Models. In International Conference on Machine Learning (ICML.,
page 12, 2021.
IC
12

---

[40] Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting Neural Network Accuracy from Weights, arXiv:2002.11448 [cs, stat], February
2020. I
[41]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In arxiv:7706003762 [Cs], December 2017.
[42] Jiayun Wang. Yubei Chen, Stella X. Yu, Brian Cheung, and Yann LeCun. Recurrent Parameter Generators, July 2021.
[43] Kuan-Chieh Wang, Paul Vicol, James I Lucas, Li Gu, Roger Grosse, and Richard Zemel. Adver- sarial distillation of bayesian neural network posteriors. In International conference on machine learning, pages 5190-5199. PMLR, 2018. TO
[44]
Jason Yosinski. Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Neural Information Processing Systems (NeurIPS), November 2014.
10
[45]
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph HyperNetworks for Neural Architecture Search. In International Conference Learning Representations (ICLR), 2019. 2 TO
[46]
Andrey Zhmoginov, Mark Sandler, and Max Vladymyrov. HyperTransffrmer: Model Genera- tion for Supervised and Semi-Supervised Few-Shot Learning. International Conference on Machine Learning (ICML), January 2022.
TC
[47] Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W Ronny Huang, and Tom Goldstein. Gradinit: Learning initialize neural networks for stable and efficient training. Advances in Neural Information Processing Systems, 34, 2021. TC
[48]
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong. and Qing He. A Comprehensive Survey Transfer Learning. In Proceedings of IEEE, 2020.
10
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/amswers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4.3 and
(c) Did you discuss any potential negative societal impacts your work? [No] The method we propose is general machine learning method. It comes with all the risks involved, but our opinion does not pose any particular risk.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URI.)? [Yes] Code, hyper-representatioos of selected model zoos are included in the supplemental material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All the details are specified in the Appendix A and B
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We report means and std for all experiments in the main paper and provide additional details about median, C195 on the median, p-value and effect size the Appendix j
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix B
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Section
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We have trained new model 7oos and will make them publicly available via an URL.
(d) Did you discuss whether and how consent obtained from people whose data you're rre're using/curating? [N/A] Our experiments use publicly available image datasets, which cite. The model zoos and code for hyper-representttion learning was published on github and zenodo by the authors of the original paper under Creative Commons Attribution 4.0.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] Our experiments rely common computer vision datasets and do not contain personally identifiable information nor offensive content.
14

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "Tviry-8UQsa", "reviewer": "Reviewer_cFnj", "summary": "The core idea of this paper is using a collection of pre-trained neural networks (here classifier \"zoos\") to learn a generative model capturing the structure of the \"weight manifold\" of these networks. This generative model can later be used to generate good weight initializations, or model ensembles. Several practical techniques for capturing these weight distributions are discussed and compared to each other (as well as to some natural baselines).\n", "questions": "1. The proposed approach allows the generative model to capture information about numerous pre-trained classifiers, essentially learning some information about the \"weight manifold\" of networks with reasonable performance. Yet in empirical comparisons (see, for example, Section 4.2.3), baseline models are individual pre-trained networks ($B_F$) that seemingly lack any information about the local structure in the weight space. An alternative baseline that could capture some information about the weight manifold, could, for example, be based on an ensemble of pre-trained models that are later fine-tuned on a new task and finally averaged (either in the weight space, which would require similar or identical initialization and would be more fair, or even averaged predictions like in a conventional ensemble). Fine-tuning a collection of models is, of course, more expensive, but does not require an initial investment in building a model zoo. I am curious if the authors agree that it would be an interesting baseline that could perhaps be studied later?\n2. If I am not mistaken, there is an interesting empirical observation that the loss function is typically smooth and well-behaved on a linear segment (in the weight space) connecting two different models trained with identical, or just similar initializations (also for model checkpoints at different training steps). This fact could have implications for some observations in Section 4.2.1 (the authors observed that the loss changes smoothly along linear segments in the weight embedding space). I am curious if the authors think that these two observations could be connected and could point at approximate global linearity of the \"embedding to weight\" map?\n3. It appears that contrastive loss would typically favor the situation where equivalent network weights (up to a channel permutation) are mapped to the same embedding. This in turns means that corresponding generative model would typically be unable to generate different equivalent weight configurations. I wonder if authors find this to be a limiting or advantageous property for generating ensembles of models?\n", "limitations": "The proposed technique shows promising results, but it's current practical applications could be limited by a very significant computational investment necessary to train a model zoo.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The overall idea of learning the structure of the model weight manifold and training generative models to sample from this manifold appears very interesting and intriguing, at least conceptually.\n2. The paper outlines an effective layer-wise loss normalization technique and several different approaches for learning such generative models (using various limiting simplifications, but overall very practical). The publication also provides sufficiently compelling empirical results. It is inspiring to see that trained generative models can capture the structure of the weight manifold and learn the set of sufficiently accurate neural networks, even capable of producing competitive model ensembles.\n\n### Weaknesses\n\n1. The proposed technique could be an invaluable tool for understanding the topology and properties of the weight manifold and it even shows some practical promise, but in its current form it appears to require a very significant computational investment for pre-training an ensemble zoo. This large computational cost could make it somewhat impractical for real-life applications.\n2. Section 3.2.3 proposes to use \"an invertible function\" from $\\mathbb{R}^D$ to $\\mathbb{R}^d$. I do not think that for $D>d$ such smooth invertible functions exist. In my opinion, the explanation in Section 3.2.3 could be clarified and made more rigorous. It is not entirely clear to me how $d$-dimensional samples $n_i$ are mapped back to the $D$-dimensional space.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I1jidie4sqR", "reviewer": "Reviewer_nctY", "summary": "This paper proposes different methods to sample model weights from a VAE trained on network parameters given by a model zoo. To improve sample quality and smoothness of the hyper-representation space, the reconstruction loss is scaled differently for each layer of the architecture (coined layer-wise loss normalization). Experiments confirm that the hyper-representation space is smoother with this loss normalization. In a wide range of experiments, the paper shows that by initializing network parameters with this VAE networks are trained faster with improved performance over training from scratch.", "questions": "Typos:\n- Line 56: \"a self attention blocks\"\n- Line 57: \"Each of the these\"", "limitations": "The paper discusses the main limitation of the paper ({W1}): The limited capacity of the models in the zoo.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- {S1} The paper is well-structured, well-written, and easy to follow.\n- {S2} The approach tackles the significant problem of learning representations of network parameters and shows that initializing parameters with the learned model indeed is advantageous in terms of training time and final network performance.\n- {S3} The paper shows lots of experiments that investigate the applicability of the learned hyper-representation space.\n\nWeaknesses:\n- {W1} The main weakness of the paper is that the learned hyper-representation space is only usable for the same single architecture that the model zoo is comprised of. This substantially reduces the practicability of the method and reduces the challenge of learning the space.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2d3WS_vkm6_", "reviewer": "Reviewer_afuw", "summary": "This work proposes to encode the network weights of a set of models into “hyper-representations”, which can be decoded to generate network weights (i.e., new models). The authors show that weights generated by the proposed hyper-representations are better than random initialization and baseline hyper-representations.\n\nThis work proposes layer-wise loss normalization to learn better hyper-representations and several sampling methods to decode weights from the learned hyper-representation.\n\nExperimental results on CIFAR, MNIST, SVHN, and STL-10 are provided to demonstrate the usefulness of the proposed method on three tasks: (1) network weights initialization, (2) model ensembles, and (3) transfer learning.\n", "questions": "See above for details.\n", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Encoding a set of trained neural networks into compact representations and applying such representations to other tasks is an interesting topic. Networks trained from large-scale datasets clearly contain meaningful knowledge of the data. Training such representations is a possible way for us to extract the knowledge from trained networks and transfer the knowledge to other tasks. From this perspective, this work is of interest to the community.\n\nHowever, I have concerns about the usefulness of the proposed method and the rigorousness of the empirical evaluation.\n\n1. This work only considers a simple architecture - 3 Conv + 2 Fully-Connected Layers. It’s unclear whether the proposed idea is useful on modern network architectures with complex skip connections or branches, e.g., ResNet, MobileNet, and EfficientNet.\n\n2. Also, all networks in the model zoo are restricted to having the same architectures. I feel this is a big limitation. What if we want to generate the weights of a different architecture? It seems there are no trivial solutions that can fix this.\n\n3. The hyper-representations need to be trained on a set of trained models (e.g., 1000), which can be expensive if the dataset is large or the network is large. This work only has results on small datasets. For image classification, results on ImageNet are highly preferred. While I understand that collecting 1000 networks on ImageNet is indeed expensive, this implies the intrinsic limitation of the proposed idea.\n\n4. For model ensembles, we usually directly train multiple models with different random seeds and compute the ensemble. This is much cheaper and easier than collecting a large set of trained networks. Same for transfer learning, we can pre-train the model on dataset A and then finetune on dataset B. Why do we need to use such hyper-representations?\n\nOverall, I am not convinced by the usefulness of the proposed idea. Therefore, I recommend rejection.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept