NeRN: Learning Neural Representations for Neural Networks
Authors: Maor Ashkenazi
Zohar Rimon
Ron Vainshtein
Shir Levi
Elad Richardson
Pinchas Mintz
Eran Treister
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Convolutional Neural Networks
Neural Representations
Implicit Representations

Abstract: Neural Representations have recently been shown to effectively reconstruct a wide range of signals from 3D meshes and shapes to images and videos. We show that, when adapted correctly, neural representations can be used to directly represent the weights of a pre-trained convolutional neural network, resulting in a Neural Representation for Neural Networks (NeRN). Inspired by coordinate inputs of previous neural representation methods, we assign a coordinate to each convolutional kernel in our network based on its position in the architecture, and optimize a predictor network to map coordinates to their corresponding weights. Similarly to the spatial smoothness of visual scenes, we show that incorporating a smoothness constraint over the original network's weights aids NeRN towards a better reconstruction. In addition, since slight perturbations in pre-trained model weights can result in a considerable accuracy loss, we employ techniques from the field of knowledge distillation to stabilize the learning process. We demonstrate the effectiveness of NeRN in reconstructing widely used architectures on CIFAR-10, CIFAR-100, and ImageNet. Finally, we present two applications using NeRN, demonstrating the capabilities of the learned representations.

Published as a conference paper at ICLR 2023
NERN LEARNING NEURAL REPRESENTATIONS FOR NEURAL NETWORKS
Maor Ashkenazi Zohar Rimon', Ron Vainshtein", Shir Levi', Elad Richardson Pinchas Mintz Eran Treister- Ben-Gurion University of the Negev TTchhnio Israel Institute of Technology 3Penta-AI
ABSTRACT
Neural Representations have recently been shown to effectively reconstruct a wide range of signals from 3D meshes and shapes images and videos. We show that, when adapted correctly, neural representations can be used to directly rep- resent the weights of pre-trained convolutional neural network, resulting in a Neural Representation for Neural Networks (NeRN). Inspired by coordinate in- puts of previous neural representation methods, we assign a coordinate each convolutional kernel in our network based on its position in the architecture, and optimize a predictor network to map coordinates to their corresponding weights. Similarly to the spatial smoothness of visual scenes, show that incorporating a smoothness constraint over the original network's 's weights aids NeRN towards a better reconstruction. In addition, since slight perturbations in pre-trained model weights can result in a considerable accuracy loss, we employ techniques from the field of knowledge distillation stabilize the learning process. We demonstrate the effectiveness of NeRN in reconstructing widely used architectures on CIFAR- 10, CIFAR- 100, and ImageNet. Finally, we present two applications using NeRN, demonstrating the capabilities of the learned representations.
INTRODUCTION
In the last decade, neural networks have proven to be very effective at learning representations a wide variety of domains. Recently, NeRF (Mildenhall et al. (2200) demonstrated that a relatively simple neural network can directly learn to represent a 3D scene. This is done using the general method for neural representations, where the task is modeled as prediction problem from some coordinate system an output that represents the scene. Once trained, the scene is encoded in the weights of the neural network and thus novel views can be rendered for previously unobserved coordinates. NeRFs outperformed previous view synthesis methods, but more importantly, offered new view on scene representation. Following the success of NeRF, there have been various at- tempts to learn neural representations on other domains as well. In SIREN (Sitzmann et al. (0200) it is shown that neural representations frequencies. NeRV (Chen et al.) 2021) utilizes neural representations for video encoding. where the video is represented
successfully model images when adapted to handle high
mapping from timestamp the pixel values of that specific frame.
In this paper, we explore the idea of learning neural representations for pre-trained neural networks. We consider representing Convolutional Neural Network (CNN) using separate predictor neural network, resulting in a neural representation for neural networks, or NeRN. We model this task as a problem of mapping each weight's 's coordinates its corresponding values in the original network. Specifically, our coordinate system is defined as a (Layer, Filter, Channel) tuple, denoted by (I, f, c), where each coordinate corresponds the weights of & x k convolutional kernel. NeRN trained to map each input's coordinate back to the original kernel weights. One can then reconstruct the original network by querying NeRN over all possible coordinates.
While a larger predictor network can trivially learn to overfit smaller original network, we show that successfully creating a compact implicit representation not trivial. To achieve this, we propose methods for introducing smoothness over the learned signal, i.e. the original network weights,
"Email: maorash @ post.ggu.a. Code avaliable https:://gtthbb.com/mmeee

---

Published as a conference paper at ICLR 2023
either by applying regularization term in the original network training or by applying post-training permutations over the original network weights. In addition, we design a training scheme inspired by knowledge distillation methods that allows for better and more stable optimization process.
Similarly to other neural representations, a trained NeRN represents the weights of the specific neu- ral network it was trained on, which to the best of our knowledge differs from previous weight prediction papers such as Ha et al. (2016); Schürholt et al.) (2021): Knyazey et al. (2021): [Schürholt al.] (2022). We demonstrate NeRN's reconstruction results on several classification benchmarks. Successfully learning a NeRN provides some additional interesting insights. For example, NeRN with limited capacity must prioritize the original weights during training. This can then be explored, using the reconstruction error, to study importance of different weights. In addition to our pro- posed method and extensive experiments, we provide a scalable framework for NeRN built using py PyTorch (Paszke et al. 2019) that can be extended to support new models and tasks. We hope that our proposed NeRN will give a new perspective on neural networks for future research.
RELATED WORK
Neural representations have recently proven to be a powerful tool in representing various signals
using coordinate inputs fed into an MLP (multilayer perceptron). The superiority of implicit 3D
shape neural representations (Sitzmann et al. 2019 Jiang et al., 2020 Peng et al.) 2020 Chabra
et al. 2020 Mildenhall et al. 2020) over previous representations such as grids or meshes has been demonstrated in Park et al. (2019); Chen & Zhang (2019); Genova et al. (2020). Following NeRF's success, additional applications rose for neural representations such as image compression (Dupont jet al. 202T), video encoding (Chen et al.] (02T), camera pose estimation (Yen-Chen et al.) 2021) and more. Some of these redesigned the predictor network to complement the learned signal. For example, Chen et al. (202I) adopted a CNN for frame prediction. In our work, we adopt a simple MLP while incorporating additional methods to fit the characteristics of convolutional weights.
Weight prediction refers to generating a neural network's weights using an additional predictor network. In Ha et al. (2016) the weights of larger network are predicted using smaller internal network, denoted as a HyperNetwork. The HyperNetwork is trained to directly solve the task, while also learning the input vectors for parameter prediction. Deutsch (2018) followed this idea by exploring the trade-off between accuracy and diversity in parameter prediction. In contrast, we aim to directly represent a pre-trained neural network, using fixed inputs. Several works have explored mnstance,Schürhool
the idea of using a model dataset for weight prediction. For instance, /Schürhllt et al. (2021) proposes a representation learning approach for predicting hyperparameters and downstream performance. Schürholt et al. (2022) explored a similar idea for weight initialization while promoting diversity. Zhang et al. (2018a) 8a); Knyazev et (202T) leverage GNN (graph neural network) to predict the parameters of previously unseen architecture by modeling it as a graph input.
Knowledge distillation is mostly used for improving the performance of a compressed network, given a pre-trained larger teacher network. There are two main types of knowledge used in student- teacher learning. First, response-based methods (Ba & Caruanal 2014: Hinton et al., 2014 Chen
et al.
2017 (019) focus on the output classification logits. Second, feature-based methods (Romero et al.
2015 Zagoruyko & Komodakis 2017) focus on feature maps (activations) throughout the
network.
The distillation scheme can be generally categorized offline (Zagoruyko & Komodakis
2017
Huang & Wang 2017] Passalis & Tefas 2018 [Ho et al.] 2019 Mirzadeh et al.) 2020:] Li
al.
2020) online (Zhang al., 2018b) Chen al.) 2020 Xie et al. 2019). In our work,
leverage offline response and feature- based knowledge distillation for guiding the learning process.
3
METHOD
In this work we focus on representing convolutional classification networks. Our overall pipeline is presented in Figure [ with extended details below the design choices and training of NeRN.
3.1 DESIGNING NERNS
Similar to other neural representations, at its core, NeRN is composed of a simple neural network, whose input is some positional embedding representing weight coordinate in the original network
2

---

Published as a conference paper at ICLR 2023
Original
Network
Ferrre Maps
(1) Syaron
Logits
(2) Lxn (3) Leeo
Paraan: R R
Charnols
Filters
Logits
(f..c)
(c) Reconstructdd
Feature Maps
(a) Paane) R2
(b) NeRN (MLP)
Network
The NeRN pipeline. Input coordinates are transformed to positional embeddings (a), which are mapped through NeRN (b) to obtain the predicted weights of a corresponding convo- lutional kernel. The predicted weights compose the reconstructed network (c). NeRN is trained using a combination of three losses; (1) reconstruction loss between the original and reconstructed weights, (2) knowledge distillation loss between the original and predicted classification logits, and (3) feature map reconstruction loss between the original and predicted feature maps.
Figure 1:
and whose output is the weight values at that coordinate, The predicted weights on all possible coordinates compose the predicted neural network, denoted as the reconstructed network.
I/O modeling We propose to learn mapping between 3-tuple (e) to the kxk kernel at channel e of filter f in layer I. Since the output size of NeRN fixed, we set to the largest kernel size in the original network, and sample from the middle when predicting smaller kernels. We model convolutional layers only, and not others such as fully-connected or normalization layers, as their parameters are of negligible size compared to the convolution weights (see Appendix C
Positional embeddings Similarly to preceding neural representation works (Nguyen- Phuoc et al. 2022 Chen et al.] 2 Tancik al. 2020)), the input coordinates are first mapped high dimen- sional vector space. By using high dimensional space, NeRN is able to represent high-frequency variations in the learned signal. We adopt the positional embeddings used in Vaswani et al. (2017).
PF concat (r
7 (v) = [ssinbb rv), cos(ooosr))
cos(bN"1
where b, and N represent the base frequency and the number of sampled frequencies, respectively.
Architecture The NeRN predictor is 5-layer MLP, which is simplified version of the architecture used in Park et al. (2019). We omit the internal concatenation of the positional embedding, as it did not change our empirical results. The hidden layer size is fixed throughout the network.
3.2 TRAINING NERNS
In order to train a NeRN we need to define set of loss functions. The most basic loss is recon- struction loss between the original and reconstructed network's weights. However, it is clear that some weights have more effect the network's 's activations and output. Hence, introduce two additional losses: a Knowledge Distillation (KD) loss and a Feature Map Distillation (FMD) loss. As presented in subsection [.6] the reconstruction loss alone yields respectable accuracy, and the additional losses improves on it, promotes faster convergence and stabilizes the training process. Notice that our model trained with no direct task loss and thus does not need access to labeled data. We further show in section 4.4 that NeRN might not require any data at all.
The objective function for training NeRNs is comprised of the following:
Labjective - + aLKD + 3LFMDs
where Lrorons LKD. and LPMD denote the reconstruction, knowledge distillation and feature map distillation losses, respectively. The and coefficients can be used to balance the different losses.
3

---

Published as a conference paper at ICLR 2023
The weight reconstruction loss, Crecons is defined as:
Crecon 1 |W Wll.
(3)
where W = (w)) w(L] and w( is the tensor of layer I's convolutional weights in the and w(!) denotes the corresponding weights
original network. Similarly, W = [w/(0) w.(1), w.(1)...w((l ...]([ of the reconstructed network. Note that we do not normalize the error by the weight's magnitude. Next, the feature map distillation loss, LFMD, introduced in Romero et al. (2015), is defined by
LFMD
1 B
ΣΣ
( a
(1) a
12
where 1 and _ are the l2 normalized feature maps generated from the i-th sample in the mini- batch B at the (-th layer for the original and reconstructed networks, respectively. Finally, the knowl- edge distillation loss, LKD. from Hinton al. (2014) is defined as
CKD =
Σ iEB
KL
aa (out)
a (out)
(5)
[B
where (,,) is the Kullback-Leibler divergence, a; (out) and a ( are output logits generated from the i-th sample in the minibatch B by the original and reconstructed networks, respectively.
Stochastic sampling Similarly to minibatch sampling used in standard stochastic optimization algorithms, in each training step of NeRN we predict all the reconstructed weights but optimize only on a minibatch of them. This allows us to support large neural networks and empirically shows better convergence even for small ones. We explored three stochastic sampling techniques (1) entire random layer, (2) uniform sampling, where we uniformly sample coordinates from across the model, (3) magnitude-orienter
where we use uniform sampling with probability Puni, and weighted sampling with probability 1 Puni. where the probability is proportional to the individual weight's magnitude. In practice, we chose the third technique with Puns 0.8. Ablation results are presented in Section 4.6
3.3 PROMOTING SMOOTHNESS
While videos, images, and 3D objects all have some inherent smoothness, this is not the case with the weights of neural network. For example, while adjacent frames in video are likely to be similar, there is clearly no reason for adjacent kernels of a pre-trained network to have similar values. We hypothesize that by introducing task for NeRN. We now present and discuss two different methods to incorporate such smoothness.
form of smoothness between our kernels we can simplify the
Regularization-basee smoothness A naive approach to promoting smoothness in the weights of a neural network is to explicitly add a loss term in the training process of the original network that encourages smoothness. Interestingly, we show that one can successfully learn smooth network with slightly inferior performance on the original task simply by adding the smoothness term,
L- F --2
Lymooth = 3Σ 1=0 1-0
A. [f.c. ( [f (1+11c)) + + (w()) [s,c,, w() (xc+1)) (6)
where Ae stands for the cosine distance, L stands for the number of layers in the network, and Fi, C stand for the number of filters and channels in a specific layer respectively. In 1 layers, we use a l2 distance instead of the cosine distance, since the weight kernels are scalars. While conceptually interesting, this approach requires modifying the training scheme of the original network and access to its training data, and may result in degradation in accuracy due to the additional loss term.
Permutation- based smoothness To overcome the downside of regularization, we introduce a novel approach for achieving kernel smoothness, by applying permutations over the pre-trained model's weights. That is, we search for a permutation of the kernels that minimizes equation 6 without changing the actual weights. Recall that NeRN maps each coordinate to the corresponding kernel, so in practice we keep the order of weights in the original network, and only change the order in which NeRN predicts the kernels.
4

---

Published as a conference paper at ICLR 2023
To solve the permutation problem formalize it using graph theory. We denote the complete graph G₁ (Vi, Fi), vl E [1, 7.1, where each vertex in Gj is a kernel in w(),, and the edge between vertex i and j is the cosine distance between the i-th and j-th kernels in w (for 1 x kernels we replace the cosine distance with & distance). Now, the optimal reordering of the weights in layer is equivalent to the minimal-weight Hamiltonian path (a path that goes through all vertices exactly once) G₁ which is precisely the traveling salesman problem (TSP) (Reinelt 1994). While propose use approximation to the optimal solution using
TSP is known to be NP-Hard, greedy solution. That is, for G₁, start from an arbitrary vertex in Vi and recursively find the closest vertex Vi that has yet been visited, until visiting all the vertices. We evaluated this method in our experiments presented in the following section. We consider two variants of this approach:
Cross-filter permutations. In each layer, consider w(") as a list all kernels in layer l. We calculate the permutation across the entire list. The disadvantage of this approach is the overhead of saving the calculated ordering to disk. For a layer with Fi filters and C kernels each, the overhead is Ft Ct loga (Fi Ci) Fr. Ct (log, Fi loga Ci) bits. For standard ResNet variants, saving these permutations entails a 4%-66% size overhead.
In-filter permutations. To reduce the overhead of the cross-filter permutations, we introduce in-filter permutations. For layer I, we first calculate the permutation of kernels inside each filter indepen- dently and then compute the permutation of the permuted filters. Figure 2 demonstrates this process. The overhead for a layer with Fj filters and C₁ kernels each is F C₁ log2 C1 Fi logg Fi bits. For standard ResNet variants, saving these permutations entails 2%-3% size overhead. Additional de- tails are presented in appendix Intuitively, we' dd expect the cross-filter permutations to be superior. Since we adopt greedy algorithm this not guaranteed, as will be shown later.
(a)
(b)
(c)
Figure 2: The process of applying in-filter permutations for specific layer. After applying the permutations, each weight kernel is still located in its original filter. (a) The original filters in the layer. (b) The weight kernels within each filter are permuted. (c) The entire filters are permuted.
4 EXPERIMENTS
In this section we evaluate our proposed method on three standard vision classification benchmarks 10, CIFAR- 100 (Krizhevsky et al.) (2009) and ImageNet (Deng al. 22099). For all bench- marks, we use NeRN to predict the weights of the ResNet (He et al., (211aa) architectures. The ResNet architectures
chosen for (1) their popularity, (2) their non-trivial design (deep layers, which incorporate x 1 convolutions). and (3) their relatively high accuracy.
For every benchmark, we examine various NeRN hidden layer sizes and show the effectiveness of promoting smoothness in the original network's 's weights via weight permutations. For CIFAR-I0, we present complete results and additionally explore the smoothness regularization method. For CIFAR-100 and ImageNet, we show only the best setup due to space constraints, and provide the complete results Appendix E Each experiment is executed with 4 different random seeds, where the mean and confidence intervals of the results are listed. We adopt the Ranger (Wright 2019) optimizer, using a learning rate of 5 10-3 and a cosine learning rate decay. The input coordinates mapped to positional embeddings of size 240. The rest of the training scheme is presented in each individual section. We run our experiments using PyTorch on a single Nvidia RTX3090. The CIFAR experiments take about 1-3 hours, depending the sizes of the original model and NeRN. ImageNet experiments take about 20-30 hours, depending on NeRN's 's size.
Following these, discuss the idea of training NeRN without promoting any kind of smoothness to the original network, examine NeRN's ability to learn using noise images instead of the task data, and finally present some relevant ablation experiments.
5

---

Published as a conference paper at ICLR 2023
Table
NeRN CIFAR-10 reconstruction results
Architecture (Accuracy %)
Learnable Weights Size [MB]
Permutation Smoothness
NeRN Predictor
Reconstructed Accuracy %
Hidden Size
Model Size [MB]
140
0.36
89.65 = 0.33
ResNet20 (91.69)
160
0,45
90.76 = 0.13
1.03
In-filter
180
0.54
91.24 0.08
200
0.65
91.57 = 0.09
140
0.36
90.39 = 0.05
ResNer20 (91.69)
160
0.45
91.04 = 0.07
1.03
Cross-filte
180
0.54
91.43 = 0.08
200
0.65
91.68 0.06
240
0.89
91.32 + 0.07
ResNet56 (93.52)
280
1.17
92.26 = 0.12
3.25
In-filter
320
1.48
92.68 0.05
360
1.83
93.11 0.08
240
0.89
91.79 0.14
ResNet56 (93.52)
280
1.17
92.45 = 0.11
3.25
Cross-filter
320
1.48
92.86 0.09
360
1.83
93.15 0.12
94
94
Reconstructed Original Accuracy
Reconstrucedd Original Accuracy
Reconstructed Accuracy
Reconstructed Accuracy
0.98
92
0.98
92
0.96
90
0.96
90
Original Accuracy
Original Accuracy
0.94
Reconstructed Accuracy
88
0.94
Reconstructed Accuracy
88
Relative Reconstruction
Relative Reconstruction
10
10
10 6
10
Smoothness Regularization Factor (log scale)
Smoothness Regularization Factor (log scale)
Figure 3: NeRN ResNet56 reconstruction results on CIFAR-10 using different smoothness regu- larization factors, shown for the original model and the reconstructed one. The figures display the performance of NeRN with different hidden sizes, 240 on the left and 280 on the right.
4.1 CIFAR-10
For these experiments, start by training ResNet20/56 to be used the original networks. These ResNet variants are specifically designed to the low input size CIFAR by limiting the input downsampling factor x4. We train NeRN for 70k iterations, using a task input batch size of 256. In addition, batch of 212 reconstructed weights for the gradient computation is sampled in each iteration. Results are presented in Table As expected, increasing the predictor size results in significant performance gains.
Regularization-Be -Based Smoothness Here we show the results of applying smoothness via a reg- ularization term the original network training. Since incorporating additional loss term results in degradation in accuracy. there is an inherent tradeoff between the original network's accuracy and NeRN's ability reconstruct the network. An optimal regularization factor balances the two,

---

Published as a conference paper at ICLR 2023
Table NeRN CIFAR-I00 reconstruction results
Architecture (Accuracy %)
Learnable Weights Size [MB]
Permutation Smoothness
NeRN Predictor
Reconstructed Accuracy %
Hidden Size
Model Size [MB]
320
1.48
69.30 = 0.36
ResNe156 (71.35)
3.25
Cross-filter
360
1.83
70.31 = 0.20
3,25
In-filter
400
2.22
|| 70.97 = 0.14
Table 3: NeRN ImageNet reconstruction results
Learnable Weights Size [MB]
NeRN Predictor
Architecture (Top-1/Top-57)
Permutation Smoothness
Reconstructed Top-1 %
Reconstructed Top-5
Hidden Size
Model Size [MB]
1024
12.99
67.55 = 0.05
87.82 + 0.07
R
1140
15.97
68.21 0.12
88.27 0.05
41.91
Cross-
(69.76/89.08)
1256
19.27
68.74
0.03
88.57 0.03
1372
22.87
69.07 + 0.05
88.79 + 0.05
achieving high absolute reconstructed accuracy. Figure B demonstrates this using several regular- ization factors and two NeRN configurations, Using hidden size of 240, the optimal is 5 10 while using a hidden size of 280, the optimal is 10-6, Complete results appear in Appendix E.1
4.2 CIFAR-100
Here we start by training ResNet56 be used as the original model. The setup is similar to that of CIFAR-10, only we train NeRN for ~990 iterations. These experiments show similar trends, where promoting smoothness and using larger predictors results in better reconstruction. Results are presented in Table Note that since this task is more complex than CIFAR-10, 10, it requires slightly larger NeRN for the same reconstructed architecture. Complete results appear in Appendix E..
4.3 IMAGENET
Here show the flexibility of NeRN by learning to represent the ImageNet-pretranne ResNet18 from torchvision Paszke et al. (201)). Thanks to our permutation- bassed smoothness, which is ap- plied post-training, NeRN can learn represent network that was trained on a large-scale dataset even without access to the training scheme of the original model. For ResNet-18, 18, we predict only the 3 x 3 convolutions in the network which constitute more than 98% of the entire convolutional parameters, while skipping the first 7 convolution and the downsampling layers. We train NeRN for 160k iterations, using a task input batch size of 32 (4 epochs). In addition, a batch of 2 recon- structed weights for the gradient computation is sampled in each iteration. Results are presented in Table where for evaluation we use the script provided by Wightman (2019). Here, interestingly, require relatively smaller NeRN. For example, the bottom row shows satisfying results, using a NeRN of ~ 54% the size of the original model. Complete results appear in Appendix E.3
4.4 DATA-FREE TRAINING
the previous experiments, NeRN was trained with images from the training data of the original network. Ideally, we would like to be able to reconstruct network without using the original task data. This allows for a complete detachment of the original task when training NeRN. However, distilling knowledge using out-of-domain data is non-trivial task. This is evident in the recent experiments by Beyer et al. 2002), where distilling knowledge using out-of-domain data achieved significantly worse results than in in-domain data. Consequently,
would assume distilling knowl- edge from noise inputs should prove to be an even more difficult task, as the extracted features might not carry meaningful signal. Interestingly, we show that given combined losses and

---

Published as a conference paper at ICLR 2023
method, NeRN achieves good reconstruction results without any meaningful input data, i. by using uniformly sampled noise U[-1,1]. Results are presented in Table
4.5 RECONSTRUCTING NON-SMOOTH NETWORKS
Although our presented method for permutation smoothness offers a very small overhead, one might consider the possibility of reconstructing model without promoting any kind of smoothness. Re- sults are presented in table Note that although NeRN able to reconstruct non-smooth networks, the results are inferior to those of reconstructing smooth ones. While promoting smoothness im- proves results across all experiments, the accuracy gap is more significant for smaller predictors. In these cases, the predictor typically has lower capacity capture non-smooth signals.
Table 4: NcRN CIFAR-10 10 reconstruction sults using noise as input compared to omit- ting the distillation losses. Experiments were run using cross-filee filttr permutations.
Table 5: NeRN CIFAR- 10 reconstruction sults for non-smooth networks.
Architecture (Accuracy %)
NeRN
Accuracy %
Hidden Size
NeRN Hidden Size
Architecture
140
87.95 + 0.37
Inputs
t Accuracy
ResNer20 (91.69)
(Accuracy %)
160
89.90 + 0.26
180
90.82 + 0.07
x Noise
88.36 + 0.39
ResNer20 (91.69)
160
89.08 + 0.34
ResNet56 (93.52)
240
87.54 0.24
90.83 + 0.15
x Noise
90.21 = 0.32
180
320
92.11 + 0.07
90.64 = 0.22
X
90.93 = 0.28
280
ResNet56 (93.52)
Noisc
91.40 = 0.37
X
92.06 + 0.11
320
Noise
92.42 + 0.06
4.6 ABLATION EXPERIMENTS
We present a few ablation experiments for NeRN's loss functions in Table & Figure 4 The results emphasize the importance of the distillation losses, allowing for better and more stable conver- gence, and the need for a reconstruction loss. In addition, Table 7 examines the weight sampling methods for gradient computation as discussed in Section 3.2
Table 6: Loss ablation results for NeRN ResNet20 reconstruction on CIFAR-I0. We use setup of cross-filter permuta- tions and a hidden size of 160.
Reconstructed Accuracy (%
100
80
60
Crecon LKD LEND Accuracy %
X
x
88.36 = 0.39
x
10.30 = 0.48
91.04 = 0.07
20
All Losses
Reconstruction Loss
Table 7:
Weight sampling ablation re-
2
6
sults. Setup similar Table 6
Iteration
-0
Weight Sampling
t Accuracy %
Figure 4: NeRN ResNer20 training plot on CIFAR-10.
All Weights
90.99 + 0.13
Random Layer
73.77 + 5.49
Setup similar Table 6 Incorporating the distillation losses allows for a faster and more stable convergence.
Random Batch
91.00 1 0.08
Random Batch (weighted)
91.04 1. 0.07
8

---

Published as a conference paper at ICLR 2023
100
Reconstructed Accuracy (%)
80
60
40
Hidden Size=320
20
Hidden Size=280
0
10
20
30
40
50
Pruning Factor (%)
Input
Low Error
High Error
Figure 5: Two additional applications for NeRN. The left figure presents naive unstructured magni- tude pruning results. Pruning was applied NeRN trained to reconstruct ResNet56 on CIFAR-10, using cross-filter permutations. The right figure presents the averaged activations of 10 original fil- ters with the lowest/highest reconstruction error in layer4.0.com conov22 of ResNet18 trained ImageNet. For this, we used a NeRN trained with in-filter permutations and a hidden size of 1256.
ADDITIONAL APPLICATIONS
NeRNs offer a new viewpoint on neural networks, encoding the network weights themselves in another network. We believe that various research directions and applications can arise from this new representation, and below we examine some possible applications that can benefit from NeRNs.
Weight Importance Through the distillation losses guiding the optimization process, we hypothe- size that NeRN prioritizes the reconstruction of weights based on their influence on the activations and logits. This observation means that NeRN implicitly learns weight importance in a network- global manner. By extracting this information from NeRN we can visualize important filters, which will be those with the lowest relative reconstruction error. Figure 5 visualizes the average activation map of the 10 filters with the lowest/highest reconstruction error in specific ResNet18 layer. The filters with the lower reconstruction error do indeed correspond to high valued activation maps.
Meta-Com Cota-Compression NeRN offers a compact representation of the original network, which is neural network by itself. We propose that one can compress the NeRN predictor to achieve a more disk-size economical representation. To demonstrate this, apply naive magnitude-bel based pruning on the predictor. The results are presented in Figure 5 As further extensions one can also examine more sophisticated compression techniques e.g., structured pruning and quantization.
Another interesting usage the NeRN of an already pruned network to further reduce its disk size. That promising direction for two reasons, (I) NeRN predicts only weights, and having less weights to predict simplifies the task, and likely also the size of NeRN, and (2) NeRN can be used post-training, without access the task data.
CONCLUSION
We propose a technique to learn neural representation for neural networks (NeRN), where a predic- tor MLP, reconstructs the weights of pretrained CNN. Using multiple losses and a unique learning scheme, present satisfying reconstruction results popular architectures and benchmarks. We further demonstrate the importance weight smoothness in the original network, as well as ways to promote it. We finish by presenting two possible applications for NeRN, (I) weight importance analysis, where the importance is measured by NeRN's accuracy, and (2) meta-compression, where the predictor is pruned achieve disk-size compact representation, possibly without data.

---

Published as a conference paper at ICLR 2023
Reproducibility Statement An important aspect for the authors of this paper is code usability. We have developed generic and scalable framework for NeRN, which we provide in the supplementary materials. The significant information for reproducing the experiments in this paper is listed in Section 3 In addition, we provide the relevant configuration files and README file containing instructions in the supplementary material. We hope this allows the reader to reproduce the results in an accessible manner.
Acknowledgements We would like to thank Yoav Miron, Gal Metzer and Yuval Alaluf for their valuable insights throughout the research and writing of this paper. This research supported by The Israel Science Foundation (grant No. 1589/19), and in part by the Isracli Council for Higher Education (CHE) via the Data Science Research Center, BGU, Israel.
REFERENCES
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? Advances in neural information processing systems, 27, 2014.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10925-10934,
2022.
Rohan Chabra, Jan E Lenssen, Eddy llg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In European Conference on Computer Vision, pp. 608 625..25.2 Springer. 2020.
Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3430-3437, 2020.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learn- ing efficient object detection models with knowledge distillation. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates. Inc., 2017. URL hatteess/ppoeeeinnn...errrereeee e1e32e235eee wtt.:/eeee11110000000303333.11.0 Paper.. Paper.pdd
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. 2019 IEFF/CVF International Conference on Computer Vision (ICCV), pp. 3513-3521, 2019.
Hao Chen, Bo Hc, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neu- ral representations for videos. Advances in Neural Information Processing Systems, 34:21557- 21568, 2021.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9939-5948, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEFE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Lior Deutsch. Generating neural networks with neural networks. arXiv preprint arXiv:1801.0199: 2018.
Emilien Dupont, Adam Goliński, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Com- pression with implicit neural representations. arXiv preprint arXiv:2103.03: 2021.
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Samaa, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4857 2020.
10

---

Published as a conference paper at ICLR 2023
David Ha, Andrew Dai, and Quoc V Le. Hypemetworks. arXiv preprint 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Jeamnnn for image recog- nition. CoRR, abs/1512.03385. 2015a. URL BttL://pp//aaaalvooo///////.2212
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. CoRR, abs/1502.01852, 2015b. URL
httt:://axxlv.rrr/bbb//555 1552.001822
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge distillation with adver- sarial samples supporting decision boundary. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 3771 2019.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. Deep Learning and Representation Learning Workshop, NIPS, 2014.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity
transfer. CoRR, abs/1707.01219, 2017.
Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6001-6010, 2020.
Boris Knyazev, Michal Drozdzal, Graham Taylor, and Adriana Romero Soriano. Parameter prediction for unseen deep architectures. Advances Neural Information Processing Systems, 34:29433-29448, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tianhong Li, Jianguo Li, Zhuang Liu, and Changshui Zhang. Few sample knowledge distillation for efficient network compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14639-1450 14647, 2020.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Taneik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. 2020.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5191-5198, 2020.
Thu Nguyen-Phuoc, Feng Liu, and Lei Xiao. Snerf: stylized neural implicit representations for 3d scenes. arXiv preprint arXiv:2207.02363 2022.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 165-174, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 268-284, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veto,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Newral Information Processing Systems 32, pp. 8024-8035. Cur- ran Associates, Inc., 2019. URL httpa///ww...//.//////aaaaaaaaa. an imperatiiiiiit. sty ortp:/aaetti.....co........oooon
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In European Conference on Computer Vision, pp. 523-540. Springer, 2020.
11

---

Published as a conference paper at ICLR 2023
G Reinelt. The traveling salesman: Computational solutions for tsp applications, 1994.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7- 9, 2015, Conference Truck Proceedings, 2015. URL http:::/awxxx.v.......a httpg//bbs//411..... (tbos/141..6555
Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self- Self-super representation learn- ing on neural network weights for model characteristic prediction. Advances in Neural Informa- lion Processing Systems, 34: 16481-16649222 2021.
Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper- representations for pre-training and transfer learning. arXiv preprint 1xXi::2207.1955/1 2022.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein, Scene representation networks: Con- tinuous 3d-structure-awer neural scene representations. Advances in Neural Information Pro- cessing Systems, 32, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kan Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Process- ing Systems, 33:7537-7547, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. 2017. URL https://
Ross Wightman. Pytorch image models. https://gthhiaccc//////ataaa/t/t image-models 2019.
Less Wright. Ranger- synergistic httpii/ch...pp.././...././//////
Learning
zer 2019.
Jiao Xie, Shaohui Lin, Yichen Zhang, and Linkai Luo. Training convolutional neural networks with cheap convolutions and online distillation. arXiv preprint 19t:::9999..000 2019.
Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, and Tsung- Yi Lin. iNcRF: Inverting neural radiance fields for pose estimation. In IEEE/RS.J International Conference Intelligent Robots and Systems (IROS), 2021.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. In 5th International Confer- ence on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL httpps://ppnteeeee..eeeee fortt//frummmdd
Sks9_a jex
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. arXiv preprint arXiv:1810.0577 20180 8a.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4320-4328, 2018b.
12

---

Published as a conference paper at ICLR 2023
Appendices
A VISUALIZING RECONSTRUCTED KERNELS
To qualitatively demonstrate NeRN's reconstruction, train NeRN reconstruct an ImageNet- pretrained ResNet18, and visualize the reconstructed kernels. For this experiment, we use a hidden size of 1256. Figure O presents the original and reconstructed kemnlls of specific channel in the second layer of 8, using 8 x grid of 3 x 3 kernels.
(a)
(b)
Figure 6: An 8 8 grid visualization of part of the original and reconstructed 3 kernels in the second layer of ResNet18, trained on ImageNet. (a) are the original kernels, (b) are the reconstructed kernels.
B NERN INITIALIZATION
standard neural network initialization method (He al. 2015b) aspires preserve the activation's variance throughout the network. Due to its non-standard outputs, we initialize NeRN such that the initially predicted weights are of similar mean and variance to the original model's weights. This is done using similar initialization method to that of HyperNetworks (Ha et al. (2016). We empirically found this initialization allows for relatively faster convergence.
C CONVOLUTIONAL PARAMETERS SIZE
Table 8: Size of convolutional parameters in standard ResNet architectures.
Convolutional Parameters Size [MB]
Convolutional Parameters Size (%)
Architecture
Task
Total Size [MB]
ResNet20 ResNet56
CIFAR-10
1.04
1.03
99.04%
CIFAR-10
3.26
3.25
99.69%
ResNet20
CIFAR-100
1.06
1.03
97.17%
ResNet56
CIFAR-100
3.29
3.25
98.78%
ResNet18
ImageNet
44.59
42.60
95.54%
13

---

Published as a conference paper at ICLR 2023
D PERMUTATIOO-BASED SMOOTHNESS
D.1 SIZE OVERHEAD
As discussed in Section 3.3 saving the original model's weight permutations for a layer with Fi filters and C kernels each, weighs either Fi Ct- (log, Fi + log, C₁) bits for the cross-filter permu- tations variant, or Fi C₁ log2 C + Fi log2 Fl bits for the in-filter permutations variant. Table presents this size overhead for both variants on the architectures we experimented on throughout the paper.
Table 9: Size overhead for weight permutations standard ResNet architectures.
Total Size [MB]
In-channel Permutations
Cross-channel Permutations
Architecture
Task
Size [MB]
Overhead (%)
Size [MB]
Overhead (%)
ResNet20 ResNet56
CIFAR-10
1.04
0.02
1.92
0.04
3.85
CIFAR-10 10
3.26
0.065
1.99
0.128
3.93
ResNet20
CIFAR-
1.06
0.02
1.89
0.04
3.77
ResNe156
CIFAR-100
3,29
0.065
1.98
0.128
3.89
ResNet18
ImageNet
44.59
1.246
2.79
2.505
5.62
D.2 PERMUTATIONS ILLUSTRATION
In the permutation figures, the color of the weight kernels signify the filters they belong to originally, not the value of the weights. The in-filerr permutation process is demonstrated in figure 2 The cross-filter permutation process is demonstrated in figure
(a)
(b)
(c)
Figure 7: The process of applying cross-filter permutations for specific layer. After applying the permutations, the weight kernels might not be located in their original filters. (a) The original filters in the layer. (b) The filters are concatenated to single tensor of weight kemnlls. (c) The weight kernels the entire tensor are permuted. (d) The permuted tensor is reshaped into separate filters.
14

---

Published as a conference paper at ICLR 2023
E COMPLETE RECONSTRUCTION RESULTS
E. CIFARIO WITH REGULAREZATON- SMOOTHNESS
Table 10: NeRN CIFAR-10 reconstruction results using regularization-basee smoothness on ResNet56. Learnabee weights size is 3.25 MB. Notice the tradeoff between original network ac- curacy, and NeRN's ability to reconstruct the network.
Smoothness
Original Accuracy %
NeRN Predictor
Reconstructed Accuracy %
Regularization Factor
Hidden Size
Model Size [MB]
10 5 10
93.35
90.14 + 0.15
92.07
90.96 0.11
240
1 10-3
91.55
90.82 + 0.03
5 10-5
90.35
89.82 + 0.05
1 10-6
93.35
91.74 0.10
10-6
92.07
91.45 0.03
280
1.17
1. 10
91.55
91.08 + 0.04
10
90.35
90.10 0.05
E.2
CIFARION 100
Table 11: Complete NeRN CIFAR-100 100 reconstruction results
Architecture (Accuracy %)
Learnable Woights Size [MB]
Permutation Smoothness
NeRN Predictor
Reconstructed Accuracy %
Hidden Size
Model Size [MB]
320
1.48
68.76 0.08
ResNet56 (71.35)
3.25
None
360
1.83
70.09 + 0.06
400
2.22
70.83 + 0.13
320
1.48
68.92 1 0.12
ResNet56 (71.35)
3.25
In-filter
360
1.83
70.30 + 0.20
400
2.22
70.97 0.14
320
1.48
69.30 0.36
ResNet56 (71.35)
3.25
Cross-filter
360
1.83
70.31 0.20
400
2.22
70.86 0.17
E.3 IMAGENET
Table 12: NeRN ImageNet reconstruction results using ResNet18
Learnable Weights Size [MB]
NeRN Predictor
Architecture
Permutation Smootheness
Reconstructed Top-1 %
Reconstructed Top-5 %
Hidden Size
Model Size [MB]
(Top- 1/Top-5%)
1024
12.99
67.48
0.06
87.78
ResNet18
1140
15.97
68.25 + 0.03
88.29 + 0.04
41.91
In-filter
(69.76/89.08)
1256
19.27
68.71 + 0.09
88.54 +: 0.02
1372
22.87
69.03 + 0.02
88.72 +: 0.02
1024
12.99
67.55 + 0.05
87.82 + 0.07
ResNet18
1140
15.97
68.21 1 0.12
88.27 1 0.05
41.91
Cross-filter
(69.76/89.08)
1256
19.27
68.74 0.03
88.57
1372
22.87
69.07 + 0.05
88.79 - 0.05
15

---

Published as a conference paper at ICLR 2023
Table 13: NeRN ImageNet reconstruction results using SqueezeNet
Learnable Weights Size [MB]
NeRN Predictor Hidden Size
Architecture (Top-1 %)
Permutation Smootheness
Reconstructed Top-1 %
Model Size [MB]
SqueezeNet (58.19)
340
1.48
56.94 + 0.05
2.74
In-filter
340
1.65
57.44 0.05
360
1.83
57.64 0.07
F A NOTE ON POSITIONAL EMBEDDINGS
As NeRN learns to represent kernels by mapping from the positional embedding of given coor- dinate its corresponding kernel, clear that the embeddings we choose also play significant part in the learning process. As adjacent convolutional kernels are relatively similar to another after we promote smoothness, one might consider the possibility of creating similarly behaving po- sitional embeddings. These positional embeddings should be, on the hand, slowly changing with respect to adjacent kernel coordinates, and on the other hand, highly separable with respect to distant coordinates. In practice, this be achieved by finding the right basis for positional embedding, as shown in Figure 8 Although we had hypothesized that incorporating this inductive bias might assist NeRN in reconstructing smooth networks, numerous experiments have refuted this theory. Nevertheless, further investigation into the chosen positional embeddings is needed, and we leave this topic for future research.
b=1.25
Normalized Cosine Similarity
b=0.76
0.8
0.6
0.4
0.2
40
60
Embedding Index
Figure 8: Normalized cosine similarity between positional embeddings of (0,0, e), E [0,63] to embedding of coordinate (0, 0, 31), using different base frequencies.
16


Reviews:
[{"review_id": "Zye_EKFbSFr", "reviewer": "Reviewer_MGd2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors have proposed several ways to enhance the smoothness of the reconstructed weights. They show that incorporating a smoothness constraint over the original network's weights aids the neural networks towards a better reconstruction. They illustrate the utility of smoothness in reconstructed weights by experimenting on several real datasets.", "summary_of_the_review": "Overall, the paper is well-written and well-motivated. The experiments are extensive.", "strength_and_weaknesses": "Strengths:\n1. The problem is interesting and well-motivated. The authors point out that by changing the order in which the neural network predicts the kernels, we can improve the smoothness of the weights and yield better reconstruction accuracy.\n2. Extensive experiments. The authors show the effectiveness of several smoothness-enhancing techniques and validate them on various datasets.\nWeakness:\n1. It introduces both size and time overhead to apply permutation smoothness. I am wondering about the time complexity to find proper permutations. Is it scalable with the network size?\n2. It would be better if the authors can provide their intuitions of why cross-filter permutation is in general the best strategy. Is this conclusion universal or dependent on the data the original network is trained on or dependent on the architecture of the original network?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "VipWwQkFYb", "reviewer": "Reviewer_uYwd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper present NERN which is a technique for learning neural representations for convolutional NNs. The parametrize the weight space by (layer, filter, channel) and train a MLP to predict the  k x k convolutional kernels. They combine different loss functions, experiments on CIFAR and ImageNet as well as provide multiple ablation studies.", "summary_of_the_review": "Great paper. I only miss comparison with other methods. Would love to hear authors comments on this. \n", "strength_and_weaknesses": "The technique seem novel and interesting. The paper in itself is sound and thorough. I especially like the smoothness regularization and experiments around it, e.g. the positional encodings. The weakness is the comparison in performance to other related work, like knowledge distillation etc. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SnrTymK-sFV", "reviewer": "Reviewer_HBhL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a neural representation method of neural networks (NeRN) to represent the weights of neural networks, which maps coordinates to convolutional kernels. The paper also considers spatial smoothness constraints on networks’ weights  to help NeRN. Moreover, the paper also points out two applications using NeRN, including filter visualization and network structure pruning. Experiments show good reconstructions of CNNs.", "summary_of_the_review": "(1) A novel method for NeRN for weight reconstruction of CNNs.\n(2) Good experimental results of reconstructed weights of neural networks.\n(3) Some weaknesses exist, but not a big question.", "strength_and_weaknesses": "Strengths:\n(1)\tThe paper provides a novel method for NeRN, which surprisingly reconstruct the weights of CNNs with implicit neural representations. \n(2)\tTo boost the performance, the paper also introduces a smoothness constraint on NeRN.\n(3)\tThe paper shows two promising applications of NeRN, including weight importance visualization and meta-compression.\n\n\nWeaknesses:\n(1)\tIn the meta-compression, it is unclear how the paper performs meta-compression. The paper claims “we apply naïve magnitude-based pruning on the predictor”, why do not directly prune raw pre-trained weights? I do not understand how the reconstructed weight works. Given a NeRN, we can map coordinates to weights and can also filter large magnitudes or map to sparse structured weights. If so, how to select the channel number for different layers? The paper claims the NeRN may reduce the disk size. A drawback could increase computational cost since we must reconstruct weights each forward. I am not sure about the inference time compared with the original weights. Moreover, because it needs to reconstruct intermediate temporary networks, how does it reduce the disk size? It would be better if some more applications are introduced.\n\n(2)The paper mainly focuses on “weight generator”, while the reference [r1] focuses on “weight discriminator”. Is it possible to design a “weight GAN” to generate realistic weights (sharp and containing high frequencies) without smoothness constraints? Or some permutation constraints (cross-filter and in-filter, e.g., chain normalization rule [r1]) on weights can be exploited as extra constraints to boost the reconstruction instead of strong smoothness assumptions? Some discussions would be nice if possible.\n\n[r1]Understanding Weight Similarity of Neural Networks via Chain Normalization Rule and Hypothesis-Training-Testing\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5OHPj6x95q_", "reviewer": "Reviewer_jngE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors present a novel method inspired by neural representations like NeRFs, SIRENs, and NeRVs to represent convolutional neural networks. They use position encodings as the input coordinates to for the 5-layer MLP NeRN to represent the location of the convolutional kernel within the architecture. the output of the NeRN is the parameters of the corresponding kernel. They also propose a hybrid objective function and some smoothing techniques to improve the model performance. Their approach is evaluated on 3 different ResNet architectures on CIFAR-10, CIFAR-100, and ImageNet and achieves comparable performance to the performance of the original architectures while using less space.", "summary_of_the_review": "I think this is a really interesting paper that has the opportunity to lead to some further interesting research directions in the community.\nHowever, there are some weaknesses I pointed out in the weaknesses section that I would appreciate if the authors could address.", "strength_and_weaknesses": "# Strengths\n- The paper is very well written and easy to read.\n- The results are compelling.\n- The method is novel to my knowledge.\n- The approach considered suggests future explorations that could be useful.\n\n# Weaknesses\n- Some sections are not as clear as they could be. For instance, the section detailing the permutation-based smoothness was hard to understand and could be better explained, perhaps with the addition of an extra diagram.\n- Some of the decisions are not extremely well motivated or ablated against. For instance:\n  - What is the difference between using positional encodings and using directly using coordinates like those used in the SIREN?\n  - Why is smoothness designed in this way? There doesn't seem to be any strong motivation for why neighboring kernels should look similar. What assumption is this smoothness based on?\n- I would like some experiments with non-ResNet CNN architectures to be explored. Right now it's hard to say if the NeRN generalizes to any other architectures even though that is what is implicitly assumed by the paper.\n- The performance of the NeRNs is still consistently lower than that of the original CNNs. What might be some reasons for this? It seems like increasing the capacity of the NeRN helps but the model seems to be limited in some way based on the results.\n- How does this method compare to HyperNets which also attempt to predict the weights of a network albeit in a different way?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%