Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposed a dataset (referred to as model zoos) of model checkpoints, where each data point is a neural network checkpoint flattened into a real-valued vector. By setting up a generating design (hyperparameters, etc.), the authors generate a grid of neural networks and save the checkpoints at the end of each epoch (50 epochs in total). The authors believe that such zoos can help advance research in model analysis, investigating learning dynamics, learning rich representations and generative modeling of model parameters. The authors hope to provide a standardized dataset for these tasks. The proposed model zoo dataset involves 6 image datasets and 2 ConvNet architectures. An analysis is provided to explore the initial results of this dataset.


==================================================================================================

UPDATE:

After the rebuttal session, I increased my rating from 5 to 6 (in order to acknowledge the improvements made by the authors during the rebuttal). 
strengths: * This is a good initiative to provide a standardized model zoo dataset.
* I think this work may encourage future work on various related problems.
* The authors provided some analyses to investigate some initial results. 
weaknesses: * There are some hypotheses/claims made in the paper that are not investigated or addressed in the end. For example "models evolve on unique, smooth trajectories in weight space during training", "would form topological structures in weight space", "We think that the geometry, curvature and smoothness of these structures contain information about the state of training and can be reveal latent properties of individual models", etc.
* The authors state that their dataset is likely to be useful in many tasks or use cases, however all of them are not backed by experiments. For example, reliability, bias, fairness, or adversarial vulnerability, etc., Section 4.3 (representation learning) and Section 4.4 (Generating New Models) demonstrated the utility of the proposed model zoos on none of the proposed tasks. Without experiments, it might be difficult to judge the usefulness. If a use case is not showcased in the paper, then it cannot be counted as a valid use case. Either the authors remove the unproven use cases into the "future work" discussion, or they provide more experiments to show the usefulness on these tasks. 
* As training and saving many model checkpoints is not a difficult task in itself, it would be crucial that the model generating design is representative and diversified enough and the scale is large enough to convince other researchers to use this dataset as a standardized infrastructure. I acknowledge the efforts the authors made to promote the diversity in the model zoos, however I doubt whether it is enough. The 2 model architectures are tiny-sized (the larger one has about 10k parameters) compared to the actual models used in academia and industry. Each model is trained only for 50 epochs. The involved datasets consist of 6 image datasets with rather small-sized images. These factors harm completeness and diversity. The number of model checkpoints is large (around 2 million), but this number does not mean a lot since it is a matter of how much time to spend to generate model checkpoints. If the generating design is limited, it is like densely exploring a narrow search space but lacking the exploration elsewhere. Despite these problems, this work seems to be a good initiative. 





correctness: See Weaknesses




clarity: The paper is easy to read overall. However some writing could be improved, for example the paragraph from Line 147 to Line 150 is confusing.
relation to prior work: The authors discussed many examples of model zoos in literature, but the comparison could be made more clear. For example, why is this new model zoo more general-purpose than previous ones, what are the drawbacks of previous work that could be fixed by this new model zoo dataset. 

additional feedback: * Line 140, “The zoos include a total of around 47’360 unique neural networks.” Line 146, “approx. 2’415’360 collected model states.” The authors may want to provide the exact statistics instead of using words like "around, approx." Precise information will be more useful for user reference. 
* Line 169: “A Tensorflow counterpart will follow.” The authors may want to promise an explicit time line for this follow-up work. 
* Line 175: "Zoo cards with key values and visualizations are provided along with the zoos online." Please provide the link or precise access information as it is unclear where such information can be found. 
* Line 178: "The threshold value is set for each zoo, such that it only excludes diverging models." The authors are invited to provide more information about how this threshold is set. This information should be useful to keep consciousness of potential bias when future works use this resource.


Typo:
 
Line 5: refereed => referred

Line 7: can be reveal => can reveal

Line 24 : high => highly

Line 26 : These => This

Figure 1: make available with all meta-data => made available with all meta-data 

Line 79: describes => described

Line 119: Appendix C => Appendix A

Line 226: This => These

Line 256: then => than; raises => raise

Line 295: the => they
review_title: Review of "Model Zoo: A Dataset of Diverse Populations of Neural Network Models"

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a dataset, called a model zoo, of neural network training checkpoints and metrics. The networks are trained using a set of hyperparameters across a range of datasets. The authors hypothesize that the weights of populations of networks trained using the same hyperparameters form geometric structures in high dimensional space which can only be studied using model zoo datasets like the one presented here. The paper outlines the collection of 2,415,360 model states from training 47,360 neural networks on six datasets. Additionally, a preliminary analysis is presented to justify the collection of model zoo datasets.
strengths: The details regarding the generation of the dataset are clearly presented so that researchers can feel confident using and reproducing the results. By defining a training trajectory using the triplet: dataset, hyperparamter, and architecture, the paper makes it easy to identify and reproduce specific training trajectories. Additionally, by presenting the statistics which measure the distribution of learning outcomes in terms of accuracy, pairwise weight similarities, etc., the paper clearly demonstrates the diversity of models contained in the zoo. The data is easily accessible online with transparent documentation to support its use.
weaknesses: Although this paper does a good job of clearly explaining and presenting the dataset, the significance of the contribution is at times unclear. For example, the introduction section hypothesizes that “NN models evolve on unique, smooth trajectories in weight space during training,” which “form topological structures in weight space,” and that this structure can “only be discovered” from the collection and analysis of model zoos. However, these claims are neither supported in the paper, nor are methods proposed for investigating these claims. Instead, section 3 focuses on characterizing the distribution of fully trained models from the zoo. This characterization supports the claim that the model zoo contains a diverse collection of trained models, but does not rigorously investigate the evolution of model weights during training or the geometric properties of the topological structures in weight space. While these analyses can and should be conducted in future work, the paper does not support the claim that these kinds of investigations can only be conducted using model zoos.

Section 2.1 states that “the model zoos have to be representative of real world models, but also diverse and span an interesting range of properties.” Despite this claim, the paper does not adequately justify the inclusion of the chosen datasets, hyperparameters, and architectures. Today, real world models tend to be very large with several orders of magnitude more parameters and hyperparamters than the CNNs training in this paper. Additionally, real world datasets in use today are often more complex containing many more examples than those used in this paper. If these models, hyperparameters, and datasets are not necessarily representative of real world models, then what commonalities do they share with those larger models? Why would we expect trends found in this model zoo to transfer to larger models trained on larger more complex datasets?

correctness: Yes, to the best of my knowledge.

clarity: The paper is mostly well written, however, as stated previously, some claims are not fully supported in the text. Additionally, some grammatical errors exist and the paper would benefit from a thorough spelling and grammar check. For instance, “These leads” (line 26) should be “These lead.” “multiple training result” (line 29) should be “multiple training runs result.” “describes” (line 79) should be “described.” “To get a more in-depth insights” (line 183) should be either “To get more in-depth insights” or “To get a more in-depth insight.”
relation to prior work: Yes, however it is unclear whether these differences constitute a significant contribution. For example, the introduction warns that existing model zoos suffer from “limited general utility” and that “a new model zoo causes subsequent work to lose comparability to existing research.” As stated in the weaknesses section above, it is unclear what characteristics help this dataset provide a general utility. Are these datasets, hyperparameters, and architectures the most generally useful to study? And, if not, what will make it possible to generalize the patterns discovered here to other datasets, hyperparameters, and architectures? 
additional feedback: NA
review_title: A well documented dataset that needs a stronger justification of its significance

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper proposes a benchmark suite for the class of neural network models trained under different hyperparameter settings across a variety of datasets. This benefit of such a collection (model zoo) is to uncover topological structures in weight space, model analysis, and generative modelling. The authors also provide an analysis of the model zoo.
strengths: - a collection of 47k+ unique neural model evaluations. 
- empirical analysis of the performance across different model zoos
- provide an interesting use-case about learning model properties from the stored states. 
- public repository for easy accessibility. 
weaknesses: - The authors propose to generate the model zoos on 6 image dataset with three different configurations. The authors also suggest a triple of (dataset, hyperparameter, architecture), however the reported results in Table 1 suggest that only 2 of the 6 datasets have varying architectures. It is unclear how the architecture acts as a dimension in the model zoo.
- It is unclear how the results of Table 1 are obtained. Are they the average performance across different hyperparameters ? If so, it would be interesting to learn what the ranges of the hyperparameters are for HYP-10-Rand. 
- Regarding HYP-10-FIX, authors state that they take the same grid as HYP-10-Rand, but re-run the same hyperparameter configuration, with a fixed seed 10 times. It is unclear what would play a role in the result discrepancy across multiple runs for the same seed/hyperparameter pairs. 
- A key use-case of that could benefit from such a model zoo is hyperparameter optimization. It is unfortunate that the authors do no address it.

correctness: The paper has alot of typos that makes it difficult to read. 
- l7 : can be reveal
- l17 are generated and 
- l26 these leads
- l46 disadvantage. (i), existing .. 
- l155 a we document .. 
- inconsistent use of hyperparameter and hyper-parameter

clarity: See correctness. 
relation to prior work: related. 
additional feedback: See detailed comments.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors provide a model zoo dataset consisting of  2.4M checkpoints of different architectures, trained on different datasets with different hyper parameters to provide a foundation for research into multiple areas including model analysis, discovery of unknown learning dynamics, representation learning, and generative modelling of neural network weights and biases. A small analysis is performed on the obtained dataset.
strengths: - Interesting, novel contribution with relevance to a broader research community.
- Large model zoo
- Clearly written paper.
- The data is easily accessible.
weaknesses: - The zoo only includes two very simple (e.g. in terms of parameter count) CNNs. In general, that is fine but the achieved performance is far behind SOTA models and implies that weights may not have been converged due to insufficient capacity of the model (see "Relation To Prior Work" [1]). This may have a negative impact of the dataset quality.
- The zoo is limited to low-resolution, classification tasks and only considers CNNs.
- Missing recommended documentations frameworks: datasheets for datasets, dataset nutrition labels, data statements for NLP, or accountability frameworks. 
- The analysis is a bit short.
correctness: The claims made in this paper appear to be constructed in a sound way.  
clarity: The paper is very clearly written.
relation to prior work: The paper clearly discussed previous contributions. I would suggest adding one more paper to related work that also performed an analysis on a large model zoo (albeit on a less systematically collected model zoo and limited to convolution weights):

[1] Paul Gavrikov, Janis Keuper, "CNN Filter DB: An Empirical Investigation of Trained Convolutional Filters", In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 19066-19076.
additional feedback: Good paper.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The dataset created in this paper consists of populations of convolutional networks trained on image classification datasets with different hyper-parameters, random seeds, and architectures. Various properties of the populations of trained networks are analyzed to illustrate the usefulness of the dataset. 


strengths: The populations of trained networks and their training trajectories on different datasets is a useful source of data for understanding the properties of the trained networks and their training trajectories. 
weaknesses: (1) The properties such as diversities of the populations of the trained networks are more or less expected. It is unclear what deep insights can be gained from the proposed dataset. For a researcher interested in understanding the properties of the trained networks and the training dynamics, he or she may well design his or her own training protocols and collect his or her own dataset. 

(2) The networks and the datasets are limited in their scales, which may limit the usefulness of the dataset. 
correctness: The paper is technically correct. 
clarity: The paper is clearly written and easy to read. 
relation to prior work: The related work is clearly discussed. 
additional feedback: The authors may consider enlarging the scope of the datasets.
review_title: A population of trained networks

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a dataset of diverse populations of neural network models, called Model Zoo. In Model Zoo, six classical image datasets, two CNN model architectures and plentiful training hyperparameters are employed to generate the model checkpoints. These checkpoints record the whole training process of a specifically configured model on a specific dataset. Three different model zoos are constructed under the different diversity of hyperparameters and random seeds. The comprehensive model properties reflecting the performance and status of each checkpoint enable the downstream task of model property prediction, and the Model Zoo dataset can also be utilized for training dynamics analysis, model representation learning and model generation.
strengths: + The Model Zoo datasets is a novel contribution to the analysis of machine learning model states and dynamics. It can also facilitate many real-world applications like Neural Architecture Search. 
+ Model Zoo is constructed in a principled way, which well controls the individual impact of different datasets, model architectures and hyperparameters, and the model variance analysis is done under different diversity of random seeds. 
+ The four proposed use cases are practical and interesting, which cover both the discriminative analysis of property prediction and representation learning and also cover the generative analysis of model design. 

weaknesses: - The six selected image datasets and two selected CNN architectures are a bit out of date, which do not reflect the current trend in Computer Vision research with large-scale datasets like ImageNet and large-scale backbone model like Vision Transformer. 
- In the supplementary material, authors state that Model Zoo is generated using CPU resources only, which neglects several important GPU training hyperparameters that play an important role in the model training of this era. For GPU-based training, models' states and dynamics could be influenced by mixed-precision training, activation checkpointing and how model states are synchronized across GPUs. These factors should also be taken into consideration when generating model zoos. 

correctness: The dataset is generated in a technically sound way with good controls on broad influential factors. 
clarity: The paper is well written, which makes it easy to capture the main ideas of this work.
relation to prior work: To my best knowledge, this dataset is a novel contribution to the related field. 
additional feedback: In general, I admire the contribution of the proposed Model Zoo dataset. However, regarding the interests of modern machine learning model design, I have following questions for the author response period.

1. How will Model Zoo be extended to involve large-scale datasets like ImageNet and large-scale models like Vision Transformer? These involvements could attract more practitioners from the community.

2. How do authors plan to take the influence of GPU training hyperparameters into consideration?
review_title: Official Review for Paper 126 by Reviewer 3DjW

Meta-review: metareview: This study constitutes a gargantuan empirical effort to characterize a large number of neural network models and parameters (referred to by the authors as the model zoo). The dataset includes 27 model architectures and a total of 50'360 models when considering all hyperparameters. This large empirical investigation can be useful as benchmark for multiple tasks and also to further understand the dynamics of training in neural networks. It can also serve the purpose of testing more theoretical approaches to study learning in neural networks. 


confidence: 5: The area chair is absolutely certain
Decision: Accept