Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work proposes to encode the network weights of a set of models into “hyper-representations”, which can be decoded to generate network weights (i.e., new models). The authors show that weights generated by the proposed hyper-representations are better than random initialization and baseline hyper-representations.

This work proposes layer-wise loss normalization to learn better hyper-representations and several sampling methods to decode weights from the learned hyper-representation.

Experimental results on CIFAR, MNIST, SVHN, and STL-10 are provided to demonstrate the usefulness of the proposed method on three tasks: (1) network weights initialization, (2) model ensembles, and (3) transfer learning.

strengths_and_weaknesses: Encoding a set of trained neural networks into compact representations and applying such representations to other tasks is an interesting topic. Networks trained from large-scale datasets clearly contain meaningful knowledge of the data. Training such representations is a possible way for us to extract the knowledge from trained networks and transfer the knowledge to other tasks. From this perspective, this work is of interest to the community.

However, I have concerns about the usefulness of the proposed method and the rigorousness of the empirical evaluation.

1. This work only considers a simple architecture - 3 Conv + 2 Fully-Connected Layers. It’s unclear whether the proposed idea is useful on modern network architectures with complex skip connections or branches, e.g., ResNet, MobileNet, and EfficientNet.

2. Also, all networks in the model zoo are restricted to having the same architectures. I feel this is a big limitation. What if we want to generate the weights of a different architecture? It seems there are no trivial solutions that can fix this.

3. The hyper-representations need to be trained on a set of trained models (e.g., 1000), which can be expensive if the dataset is large or the network is large. This work only has results on small datasets. For image classification, results on ImageNet are highly preferred. While I understand that collecting 1000 networks on ImageNet is indeed expensive, this implies the intrinsic limitation of the proposed idea.

4. For model ensembles, we usually directly train multiple models with different random seeds and compute the ensemble. This is much cheaper and easier than collecting a large set of trained networks. Same for transfer learning, we can pre-train the model on dataset A and then finetune on dataset B. Why do we need to use such hyper-representations?

Overall, I am not convinced by the usefulness of the proposed idea. Therefore, I recommend rejection.

questions: See above for details.

limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes different methods to sample model weights from a VAE trained on network parameters given by a model zoo. To improve sample quality and smoothness of the hyper-representation space, the reconstruction loss is scaled differently for each layer of the architecture (coined layer-wise loss normalization). Experiments confirm that the hyper-representation space is smoother with this loss normalization. In a wide range of experiments, the paper shows that by initializing network parameters with this VAE networks are trained faster with improved performance over training from scratch.
strengths_and_weaknesses: Strengths:
- {S1} The paper is well-structured, well-written, and easy to follow.
- {S2} The approach tackles the significant problem of learning representations of network parameters and shows that initializing parameters with the learned model indeed is advantageous in terms of training time and final network performance.
- {S3} The paper shows lots of experiments that investigate the applicability of the learned hyper-representation space.

Weaknesses:
- {W1} The main weakness of the paper is that the learned hyper-representation space is only usable for the same single architecture that the model zoo is comprised of. This substantially reduces the practicability of the method and reduces the challenge of learning the space.

questions: Typos:
- Line 56: "a self attention blocks"
- Line 57: "Each of the these"
limitations: The paper discusses the main limitation of the paper ({W1}): The limited capacity of the models in the zoo.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The core idea of this paper is using a collection of pre-trained neural networks (here classifier "zoos") to learn a generative model capturing the structure of the "weight manifold" of these networks. This generative model can later be used to generate good weight initializations, or model ensembles. Several practical techniques for capturing these weight distributions are discussed and compared to each other (as well as to some natural baselines).

strengths_and_weaknesses: ### Strengths

1. The overall idea of learning the structure of the model weight manifold and training generative models to sample from this manifold appears very interesting and intriguing, at least conceptually.
2. The paper outlines an effective layer-wise loss normalization technique and several different approaches for learning such generative models (using various limiting simplifications, but overall very practical). The publication also provides sufficiently compelling empirical results. It is inspiring to see that trained generative models can capture the structure of the weight manifold and learn the set of sufficiently accurate neural networks, even capable of producing competitive model ensembles.

### Weaknesses

1. The proposed technique could be an invaluable tool for understanding the topology and properties of the weight manifold and it even shows some practical promise, but in its current form it appears to require a very significant computational investment for pre-training an ensemble zoo. This large computational cost could make it somewhat impractical for real-life applications.
2. Section 3.2.3 proposes to use "an invertible function" from $\mathbb{R}^D$ to $\mathbb{R}^d$. I do not think that for $D>d$ such smooth invertible functions exist. In my opinion, the explanation in Section 3.2.3 could be clarified and made more rigorous. It is not entirely clear to me how $d$-dimensional samples $n_i$ are mapped back to the $D$-dimensional space.

questions: 1. The proposed approach allows the generative model to capture information about numerous pre-trained classifiers, essentially learning some information about the "weight manifold" of networks with reasonable performance. Yet in empirical comparisons (see, for example, Section 4.2.3), baseline models are individual pre-trained networks ($B_F$) that seemingly lack any information about the local structure in the weight space. An alternative baseline that could capture some information about the weight manifold, could, for example, be based on an ensemble of pre-trained models that are later fine-tuned on a new task and finally averaged (either in the weight space, which would require similar or identical initialization and would be more fair, or even averaged predictions like in a conventional ensemble). Fine-tuning a collection of models is, of course, more expensive, but does not require an initial investment in building a model zoo. I am curious if the authors agree that it would be an interesting baseline that could perhaps be studied later?
2. If I am not mistaken, there is an interesting empirical observation that the loss function is typically smooth and well-behaved on a linear segment (in the weight space) connecting two different models trained with identical, or just similar initializations (also for model checkpoints at different training steps). This fact could have implications for some observations in Section 4.2.1 (the authors observed that the loss changes smoothly along linear segments in the weight embedding space). I am curious if the authors think that these two observations could be connected and could point at approximate global linearity of the "embedding to weight" map?
3. It appears that contrastive loss would typically favor the situation where equivalent network weights (up to a channel permutation) are mapped to the same embedding. This in turns means that corresponding generative model would typically be unable to generate different equivalent weight configurations. I wonder if authors find this to be a limiting or advantageous property for generating ensembles of models?

limitations: The proposed technique shows promising results, but it's current practical applications could be limited by a very significant computational investment necessary to train a model zoo.
review_title: null

Meta-review: metareview: This paper learns hyper-representations to generate parameters of neural networks. The authors propose layer-wise loss normalization in the generation process. The proposed method is demonstrated on several tasks. 

The paper received two positive reviews and one negative review. 

Reviewer afuw was not convinced by the usefulness of the proposed method, and initially recommended rejection. The authors did address some key concerns of this reviewer in the rebuttal with added experiments. This reviewer raised the rating to borderline reject after reading the rebuttal, but remained concerned with relatively simple architectures and relatively small datasets. The authors further explained that this is currently the norm in this line of research. 

The authors addressed most of the concerns of the other two reviewers in their rebuttals. 

Overall, I feel this is an interesting exploration. Meanwhile I do share the concern of Reviewer afuw. 
confidence: Less certain
Decision: Accept