Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a self-supervised representation learning approach to uncover properties from trained neural network (NN) weights and biases. Representations are learnt through encoder-decoder architectures on the weights, using various losses. Attention-based modules are used to find interactional structure between different weights of a NN, resulting in compressed representations in different manners. To this end, data augmentation based on permutation augmentation is proposed; random erasing and noise augmentation are also explored. Empirics are performed on trained models from MNIST, Fashion-MNIST, SVHN, CIFAR-10 as well as synthetic datasets. Results explore data augmentation, architecture ablation, out-of-distribution prediction and various downstream tasks.
main_review: Overall, the paper is well written and clear. The topic is relevant to the literature and its goal relevant. The methodology is well described and seems sound. The experimental setting is detailed and experiments are extensive. The addition of a few details would improve the quality of the paper:
- the amount of compute needed is mentioned. Could you also report model sizes?
- references to the experiments developed in the appendix could be added (eg. on the impact of the compression ratio)

I recommend an accept for this paper.

Typos:
- line 57: ", so" instead of ". So"
- line 114: "a self-supervised fashion"
- citing types are often mixed up (eg lines 113, 120)

limitations_and_societal_impact: The limitations and societal impacts are discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose three data augmentations and one attention module to learn a representation of the network weights to predict the network accuracy.
main_review: Originality: The authors propose three data augmentations. The only novel one seems to be the permutation augmentation. The other two (erasing, noising) seems to be proposed for general self-supervised learning before. The attention modules is also not new. But I think applying self-supervised learning to neural network accuracy prediction itself is new.

Quality: 
1. The motivation is still not very clear. If predicting neural network accuracy from the weights can help understand fundamental characteristics that make NN successful, how does using a self-supervised representation help more on the understanding of the fundamental characteristics. Maybe the authors can give some examples on which understanding the authors gain more on the fundamental characteristics.
2. I am not sure why we need to use self-supervised representations. I guess for a different random seed, we may get a different model. And so, the data is unlimited.
3. The experiment results seems to be okay. Self-supervised representation seems to be get better performance on predicting the network accuracy. 
4. The equation under line 96 does not look correct. I do not think matrix multiplication and sigmoid function is exchangeable. Also, I am not very sure about why we want this permutation augmentation.(but this does not affect my score)

clarity:
1. The paper is clearly written.

Significance:
1. It's not clear to me "predicting neural network accuracy from the weights can help understand fundamental characteristics that make NN successful". If the authors can give some examples, it will make the contribution more significant.


limitations_and_societal_impact: The authors described the limitations of the work. And as they mentioned in the checklist, potential negative societal impact does not apply to a single method.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This is a very interesting paper. I need to be sure that I understand the main concept and review again carefully.

My review rating is really a placeholder, please **ignore my review rating**. I will give a true rating after the authors response to my queries.

Let me know if my understanding of this work is accurate. In a nutshell:

1. Train a zoo of M Neural Net, each with weights w_i, i=1,. . M.
2. All of the w_i has the same dimension N
3. Consider one of w_i as one 'data point' and devise auto encoder kind of network to extract the latent space of w_i
4. let latent space of w_i be z_i, z_i=g_\theta(w_i)
5. analyse these z_i using a linear regressor on downstream tasks -- line 174 in appendix
6. downstream tasks are accuracy prediction, generalisation gap etc

essentially if the latent variables z_i of the weights correlates well with (line 174 in appendix) downstream tasks, e.g. good R^2, we can say that the NN zoo (statistically) picks up good information about the tasks at hand.

main_review: I see this paper as more of a discovery kind of paper than a paper for an immediate real world application. However I wish to ask if the author has some eventual real world applications in mind that inspire this work.

The motivation of this work was discussed in lines 46-63. Would be good to strengthen this part of the paper further.
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose self-supervised algorithms to predict the model characteristics from the neural network weights. Authors experimented with different self-supervised objectives based on autoencoder, contrastive learning (NT_Xent), and modified contrastive loss. They utilize different architectures based on Fully-Connected Feed-Forward Networks (FF) and transformer (Att). 

Extensive experiments show that neural representation learned with self-supervision recovers latent structure from the neural network's weights. The learned representations outperform approaches based on features from weights, different modifications of PCA, UMAP, and layer-wise weight statistics (Unterthiner, 2020) in multiple downstream tasks and out-of-distribution generalization.

Additionally motivated by the unresolved question: whether model zoo should contain multiple seeds or not due to possible leakage between samples, - they generate new model zoos based on MNIST and show that seed variation for each set of the hyperparameters is beneficial to learn better representation to recover NN's properties.
main_review: Strengths:

- This work is more about a toy benchmark for learning from the NN's weights and applying the self-supervision to the specific domain. The experiments and ablations are extensive.
- The experiments clarified the question about using seeds in model zoos.

Weaknesses:

- There is no supervised baseline, and I fail to see a motivation for self-supervision. The labels and data can be easily generated and synthesized. Most of the SOTA results are still achieved with Supervised models. One could take the encoder from your architectures then add linear layers to train for regression and classification in a multi-task fashion. Then one can take representation before linear classification layers and perform linear evaluation or visualization with UMAP. 

- The experiments show that self-supervision works for this domain. However, I do not see them particularly useful for real case scenarios and insights: 
  - It will be interesting to see a different evaluation. For example, the researcher has already trained many models. Then the researcher would use the proposed characteristic prediction model to evaluate new models. Finally, let's ask how close the prediction of the characteristic model will be to the actual performance of the fully converged model given the new weights at the initialization step or after a couple of epochs. The main goal is to save compute and time while trying to find a better model.
  - The experiments do not uncover new insights about how neural networks work. 

Minor:

- Lines 298-299: " that changing only the seeds results in models with very similar evolution during learning," Could we consider seed just as a data augmentation?

- Line 84: It is unclear what the learning procedure for the model is and how it is different from hyper-parameters.

- Line 124: The equation for the modified contrastive loss, it seems, has an only part for positive pairs but not a negative part. I am not sure if it can be called contrastive, and it looks like a cosine similarity that has been used in at least two works as SimSiam [1] and SPR [2]. 
  - [1] Chen, Xinlei, and Kaiming He. "Exploring simple siamese representation learning." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2021.
  - [2] Schwarzer, Max, et al. "Data-Efficient Reinforcement Learning with Self-Predictive Representations." *International Conference on Learning Representations*. 2020.
  - The equation has missing parathesis for the exponent.

- Line 216: Given the text, it seems that the model selection was performed on the test set, which is supposed to be a hold-out set but not as a validation set.

- Line 258: I do not purely agree that vanilla transformer does well with long-range relations, as the community is still trying to solve these questions (e.g., Transformer-XL [3], XLNet [4], Feedback Transformer [5]). The authors should clarify this.
  - [3] Dai, Zihang, et al. "Transformer-XL: Attentive Language Models beyond a Fixed-Length Context." *ACL (1)*. 2019.
  - [4] Yang, Zhilin, et al. "Xlnet: Generalized autoregressive pretraining for language understanding." *Advances in neural information processing systems* 32 (2019).
  - [5] Fan, Angela, et al. "Addressing some limitations of transformers with feedback memory." *arXiv preprint arXiv:2002.09402* (2020).

### Originality:

- To my knowledge, no one has applied self-supervision for this particular domain.

### Quality: 

- Overall, it is a good paper. However, there are some issues I mentioned as a Minor.

### Clarity: 

- Overall, the paper is well written. However, the location of tables and figures in the document is quite distant from the text.

### Significance:

- I do not find this work a significant contribution due to the lack of new insight or phenomena on how neural networks work compared to related literature (e.g., [6]).
  - Nguyen, Thao, Maithra Raghu, and Simon Kornblith. "Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth." International Conference on Learning Representations. 2021. 
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative social impact of their work.
review_title: null

Meta-review: There was a robust discussion between the reviewers and the authors and some discussion amongst the reviewers themselves. The post-rebuttal consensus is that we should accept this work. From the discussion with reviewers, some of the constructive feedback for a final version would be to show more evidence of usefulness in real case scenarios. It would also be useful to explore the directions suggested by reviewer 6PDp in their conversation with the authors.
Decision: Accept (Poster)