Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors present a novel method inspired by neural representations like NeRFs, SIRENs, and NeRVs to represent convolutional neural networks. They use position encodings as the input coordinates to for the 5-layer MLP NeRN to represent the location of the convolutional kernel within the architecture. the output of the NeRN is the parameters of the corresponding kernel. They also propose a hybrid objective function and some smoothing techniques to improve the model performance. Their approach is evaluated on 3 different ResNet architectures on CIFAR-10, CIFAR-100, and ImageNet and achieves comparable performance to the performance of the original architectures while using less space.
strength_and_weaknesses: # Strengths
- The paper is very well written and easy to read.
- The results are compelling.
- The method is novel to my knowledge.
- The approach considered suggests future explorations that could be useful.

# Weaknesses
- Some sections are not as clear as they could be. For instance, the section detailing the permutation-based smoothness was hard to understand and could be better explained, perhaps with the addition of an extra diagram.
- Some of the decisions are not extremely well motivated or ablated against. For instance:
  - What is the difference between using positional encodings and using directly using coordinates like those used in the SIREN?
  - Why is smoothness designed in this way? There doesn't seem to be any strong motivation for why neighboring kernels should look similar. What assumption is this smoothness based on?
- I would like some experiments with non-ResNet CNN architectures to be explored. Right now it's hard to say if the NeRN generalizes to any other architectures even though that is what is implicitly assumed by the paper.
- The performance of the NeRNs is still consistently lower than that of the original CNNs. What might be some reasons for this? It seems like increasing the capacity of the NeRN helps but the model seems to be limited in some way based on the results.
- How does this method compare to HyperNets which also attempt to predict the weights of a network albeit in a different way?
clarity,_quality,_novelty_and_reproducibility: - The paper is very clearly written and easy to understand except for some sections I have mentioned in the Weaknesses section.
- The quality of this paper is high. The experiments are thorough, and the discussion is compelling.
- I have not attempted to reproduce the results of this paper.

# Minor corrections
- line 4 on the second paragraph of page 5: wights should be weights.
summary_of_the_review: I think this is a really interesting paper that has the opportunity to lead to some further interesting research directions in the community.
However, there are some weaknesses I pointed out in the weaknesses section that I would appreciate if the authors could address.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors have proposed several ways to enhance the smoothness of the reconstructed weights. They show that incorporating a smoothness constraint over the original network's weights aids the neural networks towards a better reconstruction. They illustrate the utility of smoothness in reconstructed weights by experimenting on several real datasets.
strength_and_weaknesses: Strengths:
1. The problem is interesting and well-motivated. The authors point out that by changing the order in which the neural network predicts the kernels, we can improve the smoothness of the weights and yield better reconstruction accuracy.
2. Extensive experiments. The authors show the effectiveness of several smoothness-enhancing techniques and validate them on various datasets.
Weakness:
1. It introduces both size and time overhead to apply permutation smoothness. I am wondering about the time complexity to find proper permutations. Is it scalable with the network size?
2. It would be better if the authors can provide their intuitions of why cross-filter permutation is in general the best strategy. Is this conclusion universal or dependent on the data the original network is trained on or dependent on the architecture of the original network?
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and has some novelty.
summary_of_the_review: Overall, the paper is well-written and well-motivated. The experiments are extensive.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper present NERN which is a technique for learning neural representations for convolutional NNs. The parametrize the weight space by (layer, filter, channel) and train a MLP to predict the  k x k convolutional kernels. They combine different loss functions, experiments on CIFAR and ImageNet as well as provide multiple ablation studies.
strength_and_weaknesses: The technique seem novel and interesting. The paper in itself is sound and thorough. I especially like the smoothness regularization and experiments around it, e.g. the positional encodings. The weakness is the comparison in performance to other related work, like knowledge distillation etc. 
clarity,_quality,_novelty_and_reproducibility: The clarity, quality, and novelty and reproducibility are strong. I quickly inspected some of the code and it looks good. The only thing missing is comparison with other methods.
summary_of_the_review: Great paper. I only miss comparison with other methods. Would love to hear authors comments on this.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a neural representation method of neural networks (NeRN) to represent the weights of neural networks, which maps coordinates to convolutional kernels. The paper also considers spatial smoothness constraints on networks’ weights  to help NeRN. Moreover, the paper also points out two applications using NeRN, including filter visualization and network structure pruning. Experiments show good reconstructions of CNNs.
strength_and_weaknesses: Strengths:
(1)	The paper provides a novel method for NeRN, which surprisingly reconstruct the weights of CNNs with implicit neural representations. 
(2)	To boost the performance, the paper also introduces a smoothness constraint on NeRN.
(3)	The paper shows two promising applications of NeRN, including weight importance visualization and meta-compression.


Weaknesses:
(1)	In the meta-compression, it is unclear how the paper performs meta-compression. The paper claims “we apply naïve magnitude-based pruning on the predictor”, why do not directly prune raw pre-trained weights? I do not understand how the reconstructed weight works. Given a NeRN, we can map coordinates to weights and can also filter large magnitudes or map to sparse structured weights. If so, how to select the channel number for different layers? The paper claims the NeRN may reduce the disk size. A drawback could increase computational cost since we must reconstruct weights each forward. I am not sure about the inference time compared with the original weights. Moreover, because it needs to reconstruct intermediate temporary networks, how does it reduce the disk size? It would be better if some more applications are introduced.

(2)The paper mainly focuses on “weight generator”, while the reference [r1] focuses on “weight discriminator”. Is it possible to design a “weight GAN” to generate realistic weights (sharp and containing high frequencies) without smoothness constraints? Or some permutation constraints (cross-filter and in-filter, e.g., chain normalization rule [r1]) on weights can be exploited as extra constraints to boost the reconstruction instead of strong smoothness assumptions? Some discussions would be nice if possible.

[r1]Understanding Weight Similarity of Neural Networks via Chain Normalization Rule and Hypothesis-Training-Testing

clarity,_quality,_novelty_and_reproducibility: The paper provides a novel method for NeRN, which surprisingly reconstruct the weights of CNNs with implicit neural representations. The paper also shows some promising applications of NeRN. Overall, in my opinion, the paper is novel and the idea of weight reconstruction is interesting. It would be a good paper that provides a possible solution to hyper-networks or meta-learning.

summary_of_the_review: (1) A novel method for NeRN for weight reconstruction of CNNs.
(2) Good experimental results of reconstructed weights of neural networks.
(3) Some weaknesses exist, but not a big question.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: I Summary:

- I.1 Investigated Problem:
    - The paper investigates the problem of neural representation for the weights of a pretrained neural network by Proposing NeRN (Neural Representations for Neural Networks)

- I.2 Proposed Solution:
    - Nern performs reconstruction of the parameters of a pretrained convolutional network using predictor MLP and incorporates smoothness constraint over the original network's weights that guides the neural networks towards a better reconstruction. 

- I.3 Validity Proof of the Proposed Solution:
    - Authors show the effectiveness of several smoothness-enhancing techniques and validate them on various datasets. Authors refer to two potential and promising applications of their proposed method:
        - weight importance analysis (visualization), where the importance is measured by NeRN’s accuracy;
        - meta-compression, where the predictor is pruned to achieve a disk-size compact representation, possibly without data.

II Strengths

- II.1 From a structural (organization) point of view:
    - The authors did a great job presenting their work. All reviews were unanimous on the quality of the presentation and the structure of the submission

- II.2 From an analytical (development) point of view: Most Reviewers appreciated:
    - The novelty of the method: reconstructing a network’s pretrained weights using a different network was not explored in previous literature, especially when treating the task as a neural representation problem.
    - The clarity of the motivation;
    - The empirical evidence provided to support the case of the proposed solution as the extensive experiments have been conducted:
        - Authors show the effectiveness of several smoothness-enhancing techniques and validate them on various datasets;
    - It is also worth mentioning and the transparency aspect of the submission as open source code is provided for reproducibility purposes

- II.3 From a perspective of soundness (unity, and coherence) and completeness (correctness):
    - The strength points mentioned above are sufficient evidence of the soundness and completeness of the paper.  An additional point reinforcing the strengths mentioned above is the active interaction of the authors during the rebuttal period and their openness to concerns and questions raised by the reviewers.

- III Addressing what can be thought of as weaknesses
    - There isn’t much to mention about weaknesses as:
        - Authors tried to address most of the concerns that reviewers raised and the questions they asked.
        - The presented work is novel of its kind and has a great potential to open up the door for new research in the community
        - Unanimously, the reviewers agree on the acceptance of the submission.

IV. Potential of the paper:

- IV.1 From a Potential perspective (Potential of the paper to the community):
    - The proposed solution has a great potential to be of benefit to the whole community of representation learning. The extension and investigation to other models can also be of great benefit as the presented work provide potential alternatives to hyper-networks and meta-learning.


justification_for_why_not_higher_score: it would be unfair to the authors of the paper to state an arbitrary reason for not attributing a higher score. The paper has significant potential to be impactful. As a meta-reviewer who tries to be fair, objective and intellectually honest, the only reason for which the paper has not been considered for the spotlight or oral is that it has a grade not strictly greater than 7.
justification_for_why_not_lower_score: Honestly, Recommending whether an accepted paper should be attributed to the mention of a poster, spotlight or an oral is very subjective if it does not follow fair and objective rules. A proxy to such desired fairness and objectivity are the reviews and the grades attributed to the paper. The benefits of the proposed solution and emphatic and all reviews were positive with a decent level of confidence (Average grade: 7 Average confidence: 4.00). For this reason, at least a spotlight mention should be attributed to the proposed solution.
Decision: Accept: notable-top-25%