Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to predict the parameter of a neural network via a hyper-network.
It mainly consists of two components, a compression method to capture the feature of a dataset, and a GRU-based network to generate the per-layer parameter.
Some empirical results are very interesting and surprising.
strength_and_weaknesses: ### Strengths
- I'm not familiar with the area of generating network weight for unseen datasets. But to me, this paper is well-motivated and novel.
- The organization of this paper is pretty good, first using an experiment to demonstrate the correlation between the dataset and the trained weight, then three main components of the method, it's easy to follow and interesting to read.
- Empirical evaluation is thorough and have some interesting results. Although the performance still lay far behind a fully-trained network, it can achieve reasonable results.

### Weaknesses
- The core of the proposed approach is that it can generate parameters for unseen datasets. The paper only shows CIFAR-100 -> Animals-10, I would expect some other evidence. For example, CIFAR-10 -> ImageNet.
- One interesting I notice in the method is that it trains a weight generator, which allows generating parameters of different sizes. However, I don't see any evidence in the experiment part. For instance, can you use the network to generate a larger ResNet-50 and show the accuracy?
- The ablation study reveals the effectiveness of different parts. Despite this, it could be better if the authors can provide more insights and intuitions in the method part. Some part is a bit ad-hoc to me. For instance, adding an additional classification head and trying to match the two predictions.

clarity,_quality,_novelty_and_reproducibility: ### Clarity
The paper is well-written and clear.

### Novelty
Although the employed techniques like the dataset compression are not new, the whole framework is novel to me.

### Reproducibility
The dataset split, training details are provided, but I don't see any code.

summary_of_the_review: The paper is novel and interesting to me in general. However, there are some missing empirical evaluations and intuitions, which I would like to hear from the authors in the feedback phases.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: To overcome heavy burden of computational cost and time to learn network parameters, this work propose a method that predicts parameters of the network on the given unseen dataset. For this, they allow the new hypernetwork to directly predict the network parameters with forward process, by learning mapping between datasets and their corresponding network parameters. With only 0.5 GPU seconds, this method can predict the network parameters of ResNet-18 which is competitive performance compared with the parameters trained on the dataset from scratch.
strength_and_weaknesses: - Strengths
    - Writing is clear and easy to follow.
    - Novel concept idea. This work reveals that there is correlation between datasets and the network parameters.
    - With a single hypernetwork, this work can predict the parameters of multiple datasets. I think this is quite new.
- Weaknesses
    - The main concern is that the experiments to validate the proposed method are limited and weak.
        - While experimental setting is similar with the existing meta-learning methods for the classification task, the most recent or powerful meta-learning methods are not compared.
        - In addition, I think that such similarity with the existing few-shot classificaiton tasks for the meta-learning are not attractive and not challenging. The most architectures are small (3 CNN layer ~ ResNet-18) and the small number of classes in each task, most unseen tasks are sampled from the in-distribution dataset with the seen tasks. I think this work need to more focus on the cross-domain setting or different architectures.
        - For the fair comparison, I think Meta-dataset [1] can be used.
     - There is a related work [2] that a single hypernetwork can predict parameters for unseen architectures. I recommend to add discussion with this related work.

[1] Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples, ICLR 2020
[2] Parameter Prediction for Unseen Deep Architectures, NeurIPS 2021
clarity,_quality,_novelty_and_reproducibility: Regarding writing, the clarity and quality is good. Hypernetwork supporting multiple datasets is novel.
summary_of_the_review: I think the experimental setting is rather limited to support or validate the proposed method. As I mentioned in the strengths and weaknesses section, I hope the authors address my concern by showing the solid and extensive experimental results on more realistic settings.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach to predict parameters of the model for unseen datasets given a fixed model. To do so, the paper proposes to leverage hypernetworks to predict model parameters for unseen datasets. The proposed pipeline is validated on three sets of datasets simulated from standard ML datasets (CIFAR-100, Fashion-MNIST, and ImageNet) and three simple CNN architectures (3-layer CNN, ConvNet-3, and a Resnet18). The reported results highlight the benefits of the proposed pipeline. Moreover, the authors stress-test the pipeline on one cross domain setup by training the model on CIFAR100 and testing it om Animals 10.
strength_and_weaknesses: Strengths:
The topic studied in the paper is of interest to the ML community.
The paper is well written and easy to follow

Weaknesses (details in next section):
The validation of the paper could be strengthened by considering more challenging setups and models.
Positioning the paper w.r.t to transfer learning, few-shot and zero-shot literature is missing

clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is well written and easy to follow. The reviewer has only some minor comments suggestions re clarity:

- Any reason why the authors refer to dataset embeddings as sketches and not as embeddings?

- It would be nice to present the details of the dataset used to validate the ideas in the introduction. 

Quality:
The quality of the paper could be improved along two axes: (1) Improved positioning w.r.t. prior art, and (2) stronger validation.

- The core of the idea is to predict parameters for unseen datasets while training the model on seen datasets. This idea is at core of other well studied topics in ML that go beyond hypernetworks and efficient optimization. The authors should discuss how the introduced ideas relate to topics such as generalization, transfer learning, few-shot learning and zero-shot learning. 

- Introduction. The paragraph starting with: “In this paper, we investigate a new training paradigm for deep neural networks” should contrast what is new in current submissions with the line of works of GHN for network parameter prediction (Ha et al., 2017; Knyazev et al., 2021).

- A relevant paper to consider could be: https://arxiv.org/pdf/1606.02185.pdf

- The paper is validated on rather simple scenarios where the training and the testing datasets are derived from a single dataset e.g. ImageNet. Only one relatively simple cross domain setup is considered (CIFAR-100 -> Animals-10). Adding additional more challenging scenarios would benefit the paper, e.g. training on ImageNet and validating on places datasets and vice-versa.

- Similarly, adding more challenging architectures would also strengthen the validation of the paper, e.g. ResNet50, ResNet101, ViT, ResNext, etc.

- The importance of dataset embedding is not ablated. Adding ablation of a variety of dataset embeddings would strengthen the paper. 

- In the current dataset embedding function, how important is the choice of feature extractor? Which extractors have been considered? Is the feature extractor pretrained?

Novelty:
The paper could have some edge of novelty however the positioning of the idea should be strengthened by differentiating the current pipeline w.r.t. transfer learning, few-shot and zero-shot learning ideas.

Reproducibility:
There is no mentioning about code release. However, based on a reading of the paper, there seem to be enough details to enable the reproduction of the reported results.

summary_of_the_review: Overall, the paper is interesting, well written and easy to follow. However, prior to recommending the paper for acceptance the reviewer would like to see improved validation and better positioning of the introduced idea/pipeline.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new training paradigm to learn a hyper-mapping between the datasets and the optimal neural network parameters. The reviewers agree that the method is novel and interesting. However, the main concern is the validation of the method and the experiment setups. The reviewers think it is difficult to validate the improvement from the method based on the results provided in the paper. Therefore, the paper would need to be futher improved before it can be accepted to ICLR.
justification_for_why_not_higher_score: n/a
justification_for_why_not_lower_score: n/a
Decision: Reject