Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper studied the problem of parameter prediction for deep neural networks. Especially, the authors focused on the task of predicting parameters for various network structures with a single hypernetwork. The method is based GHN and introduces several important modifications. To evaluate the performance of the proposed method, a standardized benchmark is introduced.  The improved method showed impressive performance on CIFAR-10 and ImageNet even with large neural networks. Besides, the method is also proved to be useful in architecture representation learning.

 
main_review: Strengths:
1. The task of predicting the parameter of diverse architectures with a single hyper-network is novel and important. Though iterative optimization is the mainstream for learning deep networks, it should be encouraged to explore different directions.

2. The method is based on the existing work Graph-hypernetworks, while there are three key elements introduced to gain superior improvement. The modification is certainly well motivated which also provides impressive empirical performance.

3. The evaluation part of this paper is strict and extensive. I appreciate the efforts in providing the benchmarks for evaluating the generalization ability of the different methods which will certainly benefit the research in this area. The ablation study on different components is detailed and clear which makes the results convincing.

Questions:
1. One appealing property of the proposed method is the decreased computation burden. To this end, I suggest the authors provide some statistics about the learning procedure of the Graph-hypernetworks.

2. The experiments is conducted mainly in the image classification settings. I wonder whether the proposed method could apply to different data type, such as language and graph, or different architectures such as transformers. 


limitations_and_societal_impact: Yes.
Refer to the main reviews.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose the task of parameter prediction with a new DeepNets-1M dataset. By using a improved Graph HyperNetwork, they can achieve reasonable performance for unseen and diverse networks without iterative optimization. The learned neural architecture representation also performs better than previous works.
main_review: - The three components proposed by this paper significantly improve the original GHN performance. The authors also provide rich ablation studies to show the effectiveness of their modifications.
- The authors extend the network design space to include a lot of different ops. The proposed GHN-2 shows good generalization ability on OOD architectures.
- The neural architecture representation learned by GHN seems to be more powerful than previous works.
limitations_and_societal_impact: - Although the authors claim this parameters prediction task is more general than NAS, I think the most suitable application scenario of this work is still in NAS (given the performance of predicted parameters still have a huge gap with iterative optimized parameters). It would be interesting to see some results in NAS. I believe the network representation learned by GHN-2 should be powerful for NAS.
- Including some OOD architectures in the training set would be an interesting direction. It's great to see whether this framework can work for architectures with mixed ops (for example an architecture with both convolutions and multi-head attentions).
- It would be interesting to see whether this work can be applied to other tasks beyond image classification (e.g. object detection).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper propose a way to predict/generate the parameters of an image based deep network instead of training it via SGD. It's a thought-provoking paper and the authors appears to be very detailed in their implementation. The results are surprisingly good but not better than standard optimization algorithms.
main_review: This paper propose a way to predict/generate the parameters of an image based deep network instead of training it via SGD. It's a thought-provoking paper and the authors appears to be very detailed in their implementation. The results are surprisingly good but not better than standard optimization algorithms.

Positives of the paper:
- The idea is novel and very thought provoking and different than standard way of approaching learning. The ability to accelerate learning is a great thing to do, given the time it takes to train large models.
- The paper is well written and clear. The authors have also shown the limitations of this approach which provides good context and discussion.

Negatives of the paper:
- The results doesn't surpass He-initialization with finetuning, which would've be nice and strong suggestions in favor this this approach.
- The authors suggested that it was 'natural' to be able to predict parameters from architecture. I think this isn't quite clear and it would be good for the author to outline more benefits to predicting the parameters. Given that the weights are good but not near SoTA, it's is naturally to ask why predict the parameters?
- The proposed method predicts parameters given architecture. How does this fit in with NAS?
- How sensitive are the changes of the parameters to network architecture (of the same parameter dimension), is it possible that the good weights are 'stored' in the GHN ?

Overall I think the paper is well written and novel and interesting, even though the results are not SoTA. The authors are advised to address the comments above for the next version of the paper.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes and analyzes a method for generating trained weights already optimized for a given single task for a large class of diverse architectures using Graph hypernetworks. They introduce a new DEEPNETS-1M benchmark with in-distribution and out of distribution architectures. The work also proposed three main improvements to improve performance of their GHN: normalization of predicted parameters, enhanced long-range interactions in the GHN, and meta-batching of architectures.  They demonstrate that their method is able to produce weights that do surprisingly well on unseen and diverse networks in less than 1 second on CPU or GPU for CIFAR-10 and Imagenet. They demonstrate that even for large Resnet-50s which are out of the training distribution, they could achieve 60% accuracy on CIFAR10.
main_review: This paper analyzes the interesting task of directly predicting trained parameters for diverse feed-forward networks. This in itself is not novel since it has been done previously in many NAS works for the purposes for creating performance predictors as well works on targeting a specific class of networks. However, there is significant novelty in the scope of diverse network architectures that their method can predict reasonable weights for and this may be the first which does a focused analysis on accuracy performance of these predicted weights. The analysis of results on in distribution and out of distribution architectures is quite novel. Unfortunately in terms of practical significance, the practical use cases of using these weights to generate good initializations for fine tuning seems to currently significantly degrade performance. That in itself is still an interesting finding to invite further analysis. 

The paper provides thorough analysis of their methods against the baseline of training from scratch for a very diverse set of architectures. The work also proposed three main improvements to improve performance: normalization of predicted parameters, enhanced long-range interactions in the GHN, and meta-batching of architectures. They provide thorough experiments ablating the effect of these changes.

Some of the results could benefit from further analysis. For example, it's interesting that the CIFAR-10 trained networks seem to generalize significantly better to Resnet than the Imagenet-trained networks.

I believe that this paper's main weakness is that it would be very useful to be able to compare the performance of the GHN with some stronger competitors. While there may be little or no work on predicting weights for out-of-distribution architectures, there has been significant NAS work on training supernets. A strong baseline then would be to train a supernet on some subset of the architecture space using a uniform prior and examine performance when directly deriving networks from those subnetworks.

In addition, this paper would greatly benefit from significant analysis of the weights that are generated. How sparse are the weights generated? Is there a correlation between more weights or less weights and model accuracy? For deep networks for example is it generating a few layers and a bunch of identity or skip layers? It would be interesting to visualize the activations. 

Currently, it is also a bit difficult to judge the usability of this work since it lacks discussion of the training cost of the graph hypernetwork.

limitations_and_societal_impact: The paper adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: This paper presents a technique for prediction of parameters in deep architectures. After the initial reviews, the reviewers were concerned about 1) performance in the low-data regime 2) training cost of GHNs 3) relevance/utility for NAS application 4) improvement via finetuning.
The authors clarified 1) and 4) well. 2) is high (as one would expect) but still reasonable, considering that predictions for new architectures can be quickly generated after initial training and that practitioners often adopt even more expensive approaches, such as OFA. The advantage of the OFA framework is that each architecture can be trained in parallel on a cluster, but then again, only a finite set of architectures can be trained this way, and the cost increases linearly with the number of architectures. 3) was briefly discussed in the paper and is relatively strong for the DARTS search space on CIFAR-10, although a comparison to standard/broader NAS benchmarks would have made a stronger point here. 
Decision: Accept (Poster)