Meta-Learning Sparse Implicit Neural Representations
Authors: Jaeho Lee
Jihoon Tack
Namhoon Lee
Jinwoo Shin
Venue: NeurIPS 2021 Poster (2021)
Keywords: network pruning
implicit neural representation
coordinate-based representations
data compression

Abstract: Implicit neural representations are a promising new avenue of representing general signals by learning a continuous function that, parameterized as a neural network, maps the domain of a signal to its codomain; the mapping from spatial coordinates of an image to its pixel values, for example. Being capable of conveying fine details in a high dimensional signal, unboundedly of its domain, implicit neural representations ensure many advantages over conventional discrete representations. However, the current approach is difficult to scale for a large number of signals or a data set, since learning a neural representation---which is parameter heavy by itself---for each signal individually requires a lot of memory and computations. To address this issue, we propose to leverage a meta-learning approach in combination with network compression under a sparsity constraint, such that it renders a well-initialized sparse parameterization that evolves quickly to represent a set of unseen signals in the subsequent training. We empirically demonstrate that meta-learned sparse neural representations achieve a much smaller loss than dense meta-learned models with the same number of parameters, when trained to fit each signal using the same number of optimization steps.

Meta-learning Sparse Implicit Neural Representations
Jaeho Lee
Jihoon Tack** Namhoon Lee
Jinwoo Shin 4B
School of Electrical Engineering. KAIST
Kim Jaechul Graduate School of AI, KAIST
Graduate School of Artificial Intelligence, UNIST D Department of Computer Science and Engineering, UNIST anwoos](kaist @kaistt.......oaaa ac.kr, nleetunist.
{jaeho -ee,, ihoontack, jinwoos]
Abstract
Implicit neural representations are promising new avenue of representing gen- eral signals by learning continuous function that, parameterized neural net- work, maps the domain of signal to its codomain; the mapping from spatial coordinates of an image to its pixel values, for example. Being capable of convey- ing fine details in a high dimensional signal, unboundedly of its domain, implicit neural representations ensure many advantages over conventional discrete repre- sentations. However, the current approach is difficult to scale for a large number of signals or a data set, since learning a neural representation- whhch is parameter heavy by itself-for each signal individually requires a lot of memory and com- putations. To address this issue, we propose to leverage a meta-learning approach in combination with network compression under a sparsity constraint, such that it renders a well-initialized sparse parameterization that evolves quickly to represent set of unseen signals in the subsequent training. We empirically demonstrate that meta-learned sparse neural representations achieve a much smaller loss than dense meta-learned models with the same number of parameters, when trained to fit each signal using the same number of optimization steps.
Introduction
explosively growing line of research on implicit neural representations (INRs) also known as coordinate-based neural representations- of storing the signal values corresponding to the coordinate grid (e.g., pixels or voxels), we train neural network with continuous activation functions (e.g., ReLU, sinusoids) to approximate the perdinate-to-valud mapping [31, 26, 3]. As the activation functions are continuous, the trained INRs give continuous representation of the signal. The continuity of INRs brings several practical benefits over the conventional discrete representations, such as providing an out-of-the-bel method for superresolution or inpainting tasks 37, 24, 36), having its number of parameters not strictly scaling with the spatial dimension and/or spatial resolution of the signal [31, 26, 3, 8, 1, 34, 32). Furthermore, as INRs take form of trainable models, they readily amenable the idea of "learning a prior" from a set of signals, which can be utilized for various purposes including image generation [2, 37, 36] view synthesis [40, 36, 35, 27].
new paradigm of signal representation: Instead
Despite many advantages, this network-as--erepreennttiio approach is difficult to scale handle large set of signals, as having parameter-heavy neural network trained for each signal requires lot
Equal contributions
Code: https:/ //
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Random Initialization
Trained Representation
f: R R²
Training
Pruning
Re-Training
(a) INR
(b) The standard pruning pipeline for handling a single signal.
Initialization
Random
Learning
Meta-
Pruning
Learning
Meta-
Sparse Initiel Representation
Sparse Representations
(c) The proposed Meta-SparseIINR procedure for handling multiple signals.
Figure 1: Illustrations of (a) an implicit neural representation, (b) the standard pruning algorithm [9] that prunes and retrains the model for each signal considered, and (c) the proposed Mcta- MetaaSpprseeINR procedure to find a sparse initial INR, which can be trained further to fit each signal.
of memory and computations. Existing approaches to address this problem can be roughly divided into three categories. The first category of works utilizes neural network structure shared across the signals, which takes a latent code vector an input and modifies the INR output accordingly (e.g.. [36, 2, 31).. While these methods only need to store a latent vector for each signal (along with shared model parameters), they have limited ability adapt signals falling outside the learned latent space; this may lead to a degraded representation capability on new signals. The second line of works adopt meta-learning approach to train an initial INR from which each signal can be trained to fit within a small number of optimization steps [35, 40J. Unfortunately. these methods do not reduce the memory required to store the neural representations for each signal. The work belonging to the third category [4] reduces the memory required to store the INR model parameters by uniformly quantizing the weights of each INR. The method, however, does not reduce the number of optimization steps needed to train for each signal, utilize any information from other signals to improve the quantization. this sense, all three approaches have their own limitations in providing both memory- and compute-efficient method to train INRs for both seen and unseen signals.
To address these limitations, we establish new framework based on the neural network pruning, i.c., removing parameters from the neural representation by fixing them to zero (c.g.. [29]). Under this framework, the objective to train set of sparse INRs that represent diverse set of signals (either previously seen or unseen) in compute-effine compute-efficient manner. To achieve this goal, we make the following contributions:
We formulate this problem as finding sparse initial INR, which can be trained to fit each signal within a small number of optimization steps. By considering such formulation, we eliminate the need to prune the INRs separately for every single signal, each of which typically requires computationally heavy prune-retrain cycle (Fig. 1b). We give a detailed formulation Section 3.
To solve this newly formulated problem, we propose a pruming algorithm called Meta- Mitaa-SparseINR, which is the first pruning algorithm designed for the INR setups (up our knowledge). Meta- SparseINR alternately applies (1) meta-learning step to learn the INR that can be efficiently trained to fit each signal, and (2) a pruning step to removes fraction of surviving parameters from the INR using the meta-trained weight magnitudes as a saliency score (Fig. lc). See Section 4 for complete description of the algorithm.
In Section 5, we validate the performance of the proposed Meta- Meta-SparselIN scheme on three nat- ural/synthetic image datasets (CelebA [21], Imagenette [14], SDF [40]) using widely used INR
a low dimensional vector generated by forwarding each signal through some encoder.

---

architecture (SIREN [36])'. In particular, we show that, when trained to fit seen/unseen signals with small number of optimization steps, Setta-SpasseIRR consistently achieves a better PSNR than the baseline methods using similar number of parameters, which include the meta-trained dense INRs with narrower width [40] and random pruning. Furthermore, in Section 5.3, we provide results on an exploratory experiment which sheds a light on the potential limitations of pruning-at-inineee intiitiilliiaiioonn schemes for fitting multiple signals under the INR context.
2 Related works
Instead of giving general overview on the implicit neural representations and neural network prun- ing literature, we focus on describing the works that are more directly related to the main ideas of this manuscript. For an overview of two disciplines, we refer the readers to excellent surveys of Tewari et al. [41] and Hoefler et al. [12] on implicit neural representations and the neural network pruning. respectively.
Fitting implicit neural representations for multiple signals. As briefly discussed in Section 1, there are at least three different approaches to efficiently train the INRs for a large set of signals. The first approach is to have a neural network component shared across the signals, which uses a latent code (i.e., low-dimensional vector generated from each signal) to modify the INR output. [36] trains a hypernetwork to predict INR parameters from the latent code, to utilize the knowledge learned from the signal set to fit each signal better. Other works [2, 26, 31] constructs an INR architecture that takes a concatenation of latent code and coordinate vector as an input, to predict the signal values corresponding to the each signal. This approach has advantages in terms of both memory and computation: in terms of parameters, the per-signal memory cost equal to the size of the latent vector, and the per-signal training cost is total training cost of the shared network, divided by the number of samples. These methods, however, have limitations in expressing the signals that has not been observed during the training phase, as pointed out in [40]. The second approach is to meta-learn the initial INR from which each signal can be efficiently fit using a small number of gradient descent steps. [35] proposes
algorithm based this approach to learn the signed distance function (SDF) corresponding to 2D/3D shapes, and shows that such pre-training by meta- learning significantly reduces the number of optimization steps required encode new signals. [40] proposes a similar algorithm that works well on a wider range of representation tasks, including 2D image regression. We note that, despite its computational benefits, this approach does not provide any benefit in terms of the crrameter-efficieeey. The third approach is compress each INR after training. [4] propose one such method based on the weight quantization: They first fit INRs for each signal via extensive training. Then, they quantize the weights from 32-bit precision to 16-bit precision. This method does not provide any advantage in terms of computations, but can potentially be combined with other approaches. The approach taken in the present paper is most related to the second line of works [35, 40]; indeed, one of our main baselines will be the method of [40] used to train dense but narrower INRs.
Training from sparse initial models. Recent studies [6. 18, 23] have shown that one can find sparse subnetwork (i.e., a pruned network) of a randomly initialized model that can closely achieve the performance level of the original dense model when trained from scratch. This finding has brought a surge of interest toward developing algorithms that can find such "winning ticket" sub- networks by masking the original models using various notions of saliency scores (c.g., [44, 38]. Although the research direction has seen a great progress overall, it has been less studied how one can find the best subnetwork when there are multiple datasets at hand. closely related work is the paper by Morcos et al. [28], which shows that the winning ticket discovered from dataset can be transferred to a highly relevant dataset. In this paper, the aim is to find a sparse initial model that performs well on set of different tasks (which may be fitting a set of images/3D shapes in this context) when trained further on each task. We note, however, that we consider a slightly different setup than the previous works: We do not restrict ourselves to the models generated by pruning a randomly initialized model, and instead prune a meta-trained model. Indeed, our observation in Section 5.3 suggests that the models generated by pruning a randomly initialized model may not be able to significantly outperform dense and narrow basclines, in the INR context.
In Appendix B. we report additional experimental results another INR architecture called Fourier feature networks (FFN [39]), consisting of Fourier encoding layer and fulle-connecttd ReLU layers thereafter.
3

---

Meta-learning & network pruning, One of the earliest papers considering the combination of meta-learning with neural network pruning is the work by Liu al. [22]. In [22], the goal to train a PruningNet which generates set of channel-
architecture and corresponding weights, satisfying the layerwise sparsity requirements given as an input. A more recent paper by Tian et al. [43] is slightly more related to our setting: In [43], the goal is to improve the generalization property of few-shot meta-learning algorithms, without any considerations on the memory or computation limitations; the proposed algorithm alternatively removes and restores the connections from the meta-learned model, as in DSD [10] or SDNN [15]. Unlike these works, the primary aim this paper is to provide method to compress set of neural networks; to generate a shared model that can be compressed and adapted to each signal. Despite its practical importance, up to our knowledge, are the first to address this problem.
meta-learning tool
3 A framework for efficiently learning sparse neural representations
The goal of this paper to develop a memory- and compute-efficient framework for learning and storing the implicit neural representations for large number signals. We formulate this problem as finding a well-initialized sparse subnetwork structure; the sparse neural representation for each signal can be directly trained from this sparse subnetwork, without requiring further pruning by computation-heavy prune-retrain cycle (see, e.g., [9]) for each individual signal.
Formalisms. For formal description, we introduce some notations: Let T = { ,,. be a set of N target signals that we want to approximate. Each signal Tÿ RR - 1T' is a func- tion that maps k-dimensional coordinate vector an m-dimensional signal value. For exam- ple, a two-dimensional RGB image can be represented by the signal function T RR² - R with T(x. y) (R. G, B). and the occupancy of a 3D object (i.e., whether a point in 3D space in occupied by the object) can be represented by the We assume that we have an access to data pairs (x,, (x:)) for each signal j E {1. ......, N), where (x, Jiez denotes the finite set of predefined input coordinate vectors indexed by I. For two- dimensional 256 256 images, for instance, the input coordinate vectors may be 2D vectors belong- ing to the grid {0, 1/255, 2/255, .... 254/255, 1/2.
function T', R + {0, 1} R.
We use the implicit neural representation f(::0) R + R" with a trainable parameter vector 0 E &Rd to ippoximaae each signal. For SIRENS [36], ((: will be a multi-layer perceptron (MIP) with sinusoidal activation functions; for Fourier feature networks (FFN [39]), f(:; 0) will be MLP with random Fourier feature encoders and ReLU activation functions. Now, the representation risk of approximating j-th signal with parameter 0 can be written as
())
Σ [f 0) (x:)113 SE
Note that, while we are using the squared loss for a simple presentation, any other loss can be used (e.g., & loss). Under typical scenarios without any sparsity considerations, may optimize the parameters independently for each signal. other words, we solve the optimization problem
1 Σ 2,,,,)
1 Σ min C,(0,)
min 0...... ON ERG N
N
ByERd
which can be decomposed into N subproblems. In this case, the total number of parameters is d..
Sparse subnetwork and initialization. To reduce the number of parameters, a shared sparse subnetwork structure having at most k < d connections, so that the total number s.N. More formally, the subnetwork structure can be expressed binary mask M E {0, 1] with M| & k, which is applied to the neural representation by taking an element-wise product with the parameter, i.c., f(x; M O 0). We hope find zero-one mask vector M that solves the optimization
consider finding
of parameters is reduced
min C3 3 (MM))))).. BR
min (f 6 ON.....N ERd N 14llos
min
min Meto.ayd Malos
(3)
j=1
The problem (3), however, needs to be refined for practical considerations. To solve each subprob- lem in Eq. (3) without too many gradient computations, also require good initialization. Indeed,

---

it has been known that without a well-chosen initialization, the optimization of sparse networks of- ten takes significantly longer [20]. or even fail achieve the performance level that can be achieved with well-chosen one [6. 19].
To this end, augment the problem (3) with the notion of optimized initialization follows. Let
(M: ) E R denote the parameter that has been attained by taking t steps of gradient-based optimization (e.g., SGD or Adam) to fit the target signal Jj on the subnetwork with an initial value ((0) and masked by M. Given some steps-to- budget 6, we want to find a pair of mask M E {0, 1]d
and the initial value g(0) E Fd that jointly solves
N Ee j=1
min ERA
min (o. IMllos
(4)
N
We propose meta-learning approach to solve the problem (4) in Section
Before we move on, we note that the problem cannot be directly salved-withour an architectural change-by simply pooling the data pairs from N signals and applying standard pruning techniques (e.g.. [9]). This is because we are considering the task of signal representation, instead a clas- sification (or similar). Under the signal representation setup, the data pairs from different signals share the coordinate input space (xi hez. If data pairs from different signals are pooled, this leads to conflicting pairs of samples that have different output values for the same input.
4 Learning sparse neural representations with meta-learning
As described in Eq. (4), our goal is find well-initialized sparse implicit neural representation from which the neural representations for each signal can be efficiently trained. To achieve this goal, we take a meta-learning approach: We meta-learn an implicit neural representation that can quickly adapt to a set of signals, and then prune the model based on its weight magnitudes (see Algorithm 1). We call this procedure MetaaSSprseeINR (Meta-learning Sparse Implicit Neural Representation). For completeness, we provide preliminaries on the meta-learning procedure we use in Section 4.1, and give a full description of the Spars-Spr algorithm in Section 4.2.
4.1 Preliminaries: Meta-learning with neural networks
Meta-learning [42] aims to give models that can be trained efficiently on a given task, by utilizing form of knowledge gained from training on family of relevant but different tasks. This knowl- edge can be transferred in various ways, e.g., utilizing neural network structures with memory- augmentations [33]. In this section, however, we focus on describing the model-agnostic approach that explicitly trains model parameters to convey the learned knowledge (namely, MAML [5]), which we use as a part of our method. This is because (i) it allows us to use popular implicit neural representation architectures (e.g., [36, 39I) without modifications, and (ii) we will use the model parameters as saliency score for pruning, as will be described in the following subsection.
While MAML is not originally proposed for the signal representation, we describe MAML using the INR setup and notations (introduced in Section 3) for simplicity: Given set of signals T {Ti.... .,,], MAML [5] aims to learn model parameters that can adapt to each signal within a small number of gradient steps. To do so, one performs the following iterative optimization: At each outer loop, the learner draws one (or more) signal from the signal set. For now, suppose that this is the j-th iteration of the outer loop and only one signal has been drawn, which we denote by Ty. Then, the learner computes f-step SGD-updated version of the parameter ((()) (T).0) 02)) from the current parameter 0y. Then, one updates the parameter 0g by taking a stochastic gradient descent (SGD) with some outer step size B:
-3. V, (O
(5) where Lj denotes the representation risk computed for the signal Tj. as in Section 3. This ends a single outer step. We denote running 7 steps of outer loops on the current parameter 0, using the signal set T, outer step size 8, number of inner SGD steps t by (A; In our method, we will also perform MAML on the sparse networks (masked by some M). In this case, we will denote the same procedure by MAML, (0; M, T).
5

---

Algorithm/ Meta-SparselNNR: Meta-learning Sparse Implicit Neural Representations
Require: Signal set T = (Ti...., ,.......TTy... Ty}, J. desired sparsity K, per-iteration pruning ratio 7, number of pre-/re-training steps т, F, number of inner SGD steps 4,
outer loop step size B.
1: Initialize 0 E Rd using the standard initialization scheme.
2: Initialize mask M = 1,, 1) of length d.
> Begin from the unpruned network.
(r) 3: 0 <- MAML, 8.8 (0;T).
> Pretraining. See Section 4.1 for details.
4: while ||M l > do
5: 8; -M, 10:|,
E (I.....d). d).
Compute the magnitude score.
6:
$ = SortDescending(s
7: M, + 1ss; 2 1
vi E {l....,d}
> Prune connections by the magnitude. b. Retraining.
8:
+- MAML, 1.9 (0;
9: end while
4.2 Meta-SparseINR: Meta-learning sparse implicit neural representations
We now describe our method, Meta- Meta-SparseeINR (Algorithm 1). to learn the sparse implicit neural rep- resentation that can be efficiently trained to fit each signals. In nutshell, Meta-SparseINN operates by alternately meta-training the INR using MAMI, and pruning the INR using the magnitude-base pruning; this procedure can be viewed as alternating minimization of Eq. (4), where meta-training steps and pruning steps approximately perform the outer and inner minimization steps in Eq. (4), respectively. More concretely, Seta-Sparse consists of three steps:
Step 1 (Meta-learning). Given set of signals and INR randomly initialized with standard initialization schemes (e.g., as in [36] for SIRENs), we meta-learn the INR over the signals by running MAMI for steps.
Step 2 (Pruning). From the learned INR, we remove K% of surviving connections with small- est weight magnitudes. We the global magnitude criterion, i.e., we apply single magnitude threshold for every layers instead of having thresholds for each layer.
Step3 3 (Retrain and repeat). We retrain the pruned INR by running MAML for steps; required to converge to the desired performance level may often be smaller than used in Step 1. the desired global sparsity level has not been met, go back to Step 2.
As we will validate through our experiments in Section 5, meta-learning is an essential component in Meta-SparseINR. More specifically, meta-learning steps play, at least, a twofold role: First, the weights given by the meta-learning can be trained to fit each signal with a much smaller number of optimization steps. This fact may not come as a big surprise, as the meta-learning objective (5) explicitly optimizes the weight after optimizing for few steps. Indeed, without any pruning considerations, this point has been validated by Tancik et al. [40]. We observe that the same holds for sparse INRs, as we will observe in our main experiments in Section 5.2.
Second, the magnitude of the weights learned by meta-learning acts as a powerful saliency score for the pruning step (Step 2). Intuitively, the connections with large meta- meta-learned weight magnitudes are likely to have relatively large weight magnitude after a t-step adaptation to a particular signal (with a small t). In this sense, the meta-learned weight magnitudes can viewed as an aggregation of the weight magnitudes for each signal. As it is believed that weight magnitudes are an effective saliency criterion for the model trained for a single task (or signal, in this context), we expect the magnitudes of the meta-learned weights to be an effective saliency criterion for the signals, on average." Indeed, we show Section 5.2 that removing the connections with the smallest magnitude of the meta-learned weights outperforms both random pruning and dense, narrow INRs.
Assde from this intuition, the magnitude criterion is much more computationally efficient than its com- petitors, such as Hessian based ones [17, 11]: Hessian-basee ones may require the evaluation of third-order derivative, we using the meta- meta-learning loss.

---

In addition to the second point, we note that we make an empirical observation which suggests that starting from the (neta-)trainedd weights may even be necessary for sparse models outperform the dense, narrow models in the INR context. We give more details in Section 5.3.
5 Experiments
In this section, we experimentally validate the performance of the proposed Meta-SparseINR (see Algorithm 1) by measuring its performance on various 2D regression tasks. We describe the exper- imental setup and the baseline methods Section 5.1. We report the experimental results and discuss the observations in Section 5.2. In Section 5.3, we describe the experimental setup and results for an additional exploratory experiment; the experiment is designed to gauge the potential of "pruning-at at- initialization" schemes to learn sparse INR that can be efficiently trained to each signal.
5.1
Experimental setup
We evaluate the performance of the proposed Meta-SparseINR under the following two-phase sce- nario: We are given a set of signals with training and validation splits. During the training phase, meta-learn the initial INR from the training split of signals. During the test phase, randomly draw 100 signals from the validation split of signals, and train for 100 steps using the full-batch Adam to fit the INR each signal. We report the average performance (PSNR) of the model over these 100 samples. We average over three seeds for all experiments in the following subsection.
Datasets. We focus on 2D image regression tasks using three image datasets with distinct charac- teristics: CelebA (face data [21]), Imagenette (natural image [14]), and 2D SDF (geometric pattern [40].. We use the default training and validation splits. All images resized and cropped to have the resolution of 178* x 178.
Models. As our base INR, we use multi-layer perceptrons (MLPs) with sinusoidal activation func- tions (SIREN [36]), having four hidden layers with 256 neurons in each layer. This configuration popularly used for the 2D image regression task in previous works, e.g., [36, 40]. In Appendix B, we report additional experimental results on the another widely used class of architectures: MLPs using random Fourier features (FFN [39]).
Method & baselines. We evaluate the proposed Meta-SparseINR along with five baselines.
Meta-SparseINRR (ours): The method proposed in this paper (see Algorithm 1). Throughout the experiments, we prune 20% of the weight parameters in every pruning phase (i.e., 7 = 0.2).
Random Pruming: Same as Sparssaars but we randomly prune the INR parameters with uniform probability, instead of using the magnitude criterion.
Dense-Narrow: We meta-learn the dense neural representation that has a narrower width than the original INR, as [40]. The widths selected to ensure that each sparse INR has dense-narrow counterpart with an equal or slightly larger number of total parameters.
MAML+OneShot: We fit MAML-trained model on each signal for 50 epochs. Then, perform magnitude--assd pruning at a single shot, and run additional 50 epochs of training.
MAML+IMP: We prune a MAML-traine' model times, removing 20% of the surviving weights each time, and running 100
2+1 epochs before/betwween/afier each pruning phase.
Scratch: Same Dense-Narrow, but we do not meta-learn the weights and use the weights ini- tialized using the standard random initialization scheme (e.g., the one [36] for SIRENs).
Other details. For other experimental details, including the data pre-procsssing steps and hyperpa- rameters for the meta-learning phases, see Appendix A.
5.2 Main experiments
In Fig. 2, we observe that the proposed IN achieves a consistent performance gain over the all baseline methods for all three datascts. We observe that the gain is larger for in the high-sparsity regime, i.c., for INRs having a much smaller number of parameters. particular, for
'e ensure that same 100 signals for method and the baselines.

---

40
40-
35
36
80
Pook PENA
25-
25-
20-
Meta Sporso PRR (Dunt)
Random Pruring
20
Meta SporselNR (Duns)
Randon Pruning
Derse-Narron Naroon
Derse Narro
16-
MAWE-GGaaShi
Soowth
10
10
Fractio of Sunviing Parameters
Fraction of Sunviing Parametero
CelebA
(b) Imagenette
70
Dataset
Method
PSNR
#arams
60
CelehA
Meta SparseINR (Ours)
27.71
8,704
CelebA
Random Pruning
26.25
17,000
CelebA
Narrow
27.68
17,708
40-
Imagenette
Meta- SparseINR (Ours)
25.74
8,704
Imagenette
Random Pruning
24.09
17,000
30"
Mata Sparow (Dunn)
Imagenette
Dense Donse-Narrow
24.76
14,212
Random Pruring
Donge Narow
SDF
Sparse (Ours)
SDF
Random Pruning
47.48
17,000
SDF
Dense Narrow
44.61
26,978
10
Fraction of Sunviing Parameters
(c) SDF
(d) Comparison table
Figure 2: Comparison of the average PSNR of meta-learned SIRENs after 100 steps of per-signal training. (a-c) PSNR plots for CelebA, Imagenette, and SDF (d) Comparing the Meta-SparseINR having 8,704 parameters with strongest baselines (Random Pruning and Dense-Narrow) achieving similar level of PSNR. Shaded region: Mean = one standard deviation (three seeds).
25
Mett-Spurmeiht (Oum))
40- Random Pruniig
Nance-Narco
PSN
Posk PENS
10-
Randoo Pruning
Derow-Narme
100ns
60%
40%
Stepe
Sundving
(a) PSNR vs. number of optimization steps
(b) Two-step optimization
Figure 3: Fitting SIRENs meta-learned on CelebA to validation data with a smaller training budget. (a) PSNR trajectory for sparse models having 6,964 parameters and dense- dense-narrow narrow models having 7,152 parameters. (b) PSNR after two steps of training on each sample.
CelebA dataset on SIREN (Fig. 2a), Meta- Sotta-SparseINR achieves the PSNR 26. 73 with only 6,964 parameters, whereas the strongest baseline (Dense-Narrow) achieves s 20.52 with slightly more 7,152 parameters; this is x4.18 difference in terms the MSE loss. Among baselines, we observe that Random Pruning and Dense-Narrow are stronger than MAML+Oneshht and MAML+IMP This suggests that under the per-signal training budget constraints, it is beneficial to have compressed initial model to start from, instead of compressing the model while fitting each signal.
In Fig. 3, we compare the performance of the proposed Meta-SparseINNR with the strongest baselines (Random Pruning and Dense- Dense-Narrow) with a per-signal training budget smaller than 100 steps. We observe that Meta-SparseINR still outperforms the baselines such cases. Interestingly, Dense- Narrow performs worse than Random Pruning at the early stage of per-signal fitting, but achieves

---

Dense- Nerrow
#Parameters:
26,978
17,708
11,100
7,152
4,522
3,232
Meta- SparseINR (Ours)
Original
#Parameters:
26,561
17,000
10,880
6,964
4,458
2,864
Figure 4: A visual comparison of meta-learned SIRENs after training for 100 steps to fit image. (Upper row) Trained from a meta-learned dense and narrow INR, as in [40]. (Lower row) Trained from sparse model learned by the Meta-SparseeINR procedure. Sparse models retain more color information and give coarse patterns, while dense-narrow models are more monochromatic with less-structured patterns. We observed a similar behaviors in other examples (see Appendix C).
a better performance than random sparsity as the per-signal training budget increases. From this observation, suspect that the drawbacks of dense models comes from their expressive power, rather than how efficiently they can be trained.
To better understand the effect of sparsity (as opposed to dense), compare the distorted images generated by dense and narrow models with images generated by sparse models (Fig. 4). Here, we use the first image from the CelebA validation dataset. We observe that dense-narrow models tend to be more monochromatic and have finer patterns than wide-sparse models. Indeed, we observe a similar trend in other figures well; Appendix C for more images.
Lastly, we note that Meta-SparseI outperforms the baselines in terms of the PSNR achieved for training split data as well. Indeed, the PSNR achieved for the training and validation splits differ only by a tiny margin. We report the results on the training split in Appendix D.
5.3 Pruning without trained initial weights
In this subsection, we ask whether it is possible to generate a sparse initial INR that can fit a set of signals efficiently, by simply pruning randomly initialized INR directly for each signal without any trained weights. For this purpose, we compare the performance of dense and narrow models with sparse initial INRs generated by the winning ticket method [6]. which is known to achieve the best overall performance among pruning-at-initiaiinn at--niiiaallzziionnn schemes [7].
Setup. We consider fitting set of 512 x 512 natural images with RGB color channels. For the winning ticket method, we use iterative magnitude praning with weight rewinding [6]. with a global magnitude criterion. As in main experiments, we remove 20% of the surviving weights in every iteration. We train for 50k steps with full batch, using Adam with learning rate 1.0 x 10 For other experimental details (e.g., the image used and hyperparameters), see Appendix A.1.
Result. In Fig. 5, we report the PSNR achieved by both methods on SIREN (Fig. 5a) and FFN (Fig. 5b); we report the peak PSNR instead of the final PSNR, as PSNRs tend to be unstable during the end of the training.
We make two observations. First, we observe that the performance of both schemes drop consis- tently as we consider smaller number of parameters. Second, none of the two methods noticeably outperform the other. Together with the results in Section 5.2, the observations highlight the impor- tance of the training step preceding the pruning step; meta-learning gives rise sparse structure that can be trained to a higher PSNR than the dense- narrow counterpart. This sheds pessimistic light on the existence of an algorithm that can prune a randomly initialized INR to give sparse INR that can be trained to fit mutliple signals better than a dense-narrow INR, as the best known iniiiiiiatinitiiaizzatonnn scheme does not provide any gain even for a single signal case. As a minor remark, we note that pruning at initialization may have some benefits over dense-narrow models at the early stage of training, while at such stages the PSNR level is generally low, and the benefits vanish as PSNR starts to exceed 20 (at around epoch 1000).

---

Peek PSNR
Pesk PSNR
80
35
25
Dense-Narrow Narrow Winning Ticket
Nanse--Nrroow
Winning Ticket
20
10
10
10
Number of Trainable Paremeters (log scale)
Number of Trainable Parameters (log scale)
(a) SIREN
(b) FFN
Figure 5: Size-PSNR tradeoff curves for fitting 512 512 image using dense and narrow INRs (red circles) and wide-sparse INRs pruned at initialization using the "winning ticket" method (blue squares). In both plots, wide-sparse INRs do not perform noticeably well against dense and narrow ones, highlighting the importance of trained weights to enjoy the benefit of sparsity.
6 Discussion & limitations
In this paper, we proposed a framework for an efficient training of implicit neural representations, in terms of memory and computation. We formulated the problem learning sparse INR that be trained further to fit each signal within a small training budget, and proposed a first algorithm solve the problem, called Meta-SparseINR; the proposed algorithm consistently outperforms the baselines in all experimental setups considered.
Adopting the neural network pruning approach, have implicitly limited the discussion methods that (1) generate sparse parameters, and (2) does not have any shared parameters available. While such properties
come advantageous for some applications (e.g., smaller inference cost), such restrictions are not necessary for improving the parameter efficiency. For instance, one may consider fitting the signals with INRs having sparse differences from dense, shared reference model; the memory cost to store the shared model will become negligibly small as the number of signals increases. Exploring the potential of such algorithms is a promising future direction.
potential ethical side-effect of network compression algorithms is their disparate impact on various subgroups [13]. While such impact takes form of disparate accuracy drop on subgroups, the impact can take a more nuanced and think of two additional dimensions of disparate impact: the discrepancy in the training time required to arrive at desired performance level, and structurally-bitt image, c.g., as in [25]. Such potential negative impacts fairness should be carefully measured, disclosed, and mitigated before being considered for public use.
form in the context of INR training. More specifically,
distortion in the represented
Acknowledgments and Disclosure of Funding
We thank Matthew Tancik for initial suggestions on the model and hyperparameter selection, and thank Kwonyoung Ryu and Sihyun Yu for various technical suggestions the PyTTorch implemen- tation of implicit neural representations.
This work was partially supported by Institute of Information & Communications Technology Plan- ning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. N0.22019-0-00055. Arti- ficial Intelligence Graduate School Program (KAIST), and No. No.2020-0-01336 Artificial Intelligence Graduate School Program (UNIST)). This work was mainly supported by Samsung Research Fund- ing & Incubation Center of Samsung Electronics under Project Number SRFCIT1902.00
References
[1] M. Atzmon and Y. Lipman. SAL: Sign-agnostic learning of shapes from raw data. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[2] Y. Chen, S. Liu, and Wang. Learning continuous image representation with local implicit image function. In JEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
10

---

[3] Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling, In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[4] E. Dupont, A. Goliñski, M. Alizadeh, Y. W. Teh, and A. Doucet. COIN: Compression with implicit neural representations. In ICLR Neural Compression: From Information Theory to Applications Workshop, 2021.
[5] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of International Conference on Machine Learning, 2017.
[6] J. Frankle and M. Carbin. The lottery ticket hypothesis: finding sparse, trainable neural net- works. International Conference on Learning Representations, 2019.
[7] Frankle, G. K. Dziugaite, D. Roy, and M. Carbin. Pruning neural networks at initialization: Why are we missing the mark? In International Conference on Learning Representations, 2021.
[8] K. Genova, F. Cole, D. Vlasic, A. Sarna, W. T. Freeman, and T. Funkhouser. Learning shape templates with structured implicit functions. In IEEE International Conference on Computer Vision, 2019.
[9] S. Han, J. Pool, Tran, and W. J. Dally. Learning both weights and connections for efficient neural networks. In Advances in Neural Information Processing Systems, 2015.
[10] S. Han, J. Pool, S. Narang, H. Mao, E. Gong, S. Tang, E. Elsen, P. Vajda, M. Paluri, J. Tran, B. Catanzaro, and W. Dally. DSD: Dense-sparse-dense training of deep neural networks. In International Conference on Learning Representations, 2017.
[11] B. Hassibi and D. G. Stork. Second-order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems, 1992.
[12] T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. arXiv preprint 2012.00554, 2021.
[13] S. Hooker, A. Courville, G. Clark, Y. Dauphin, and A. Frome. What do compressed deep neural networks forget? arXiv preprint 1911.05248, 2019.
[14] J. Howard. Imagenette. URL httptttss//ttttiiee
www.coooaaaaaae fastai/ta//// /iimagnneteee....
[15] X. Jin, X. Yuan, J. Feng, and S. Yan. Training skinny deep neural networks with iterative hard thresholding methods. arXiv preprint 1607.05420 2016.
[16] Kodak. Kodak dataset. http: //rOk. http://OOOkuuggggphttee wws/grappics///ooooe /oookk// 1991.
[17] Y. LeCun, J. Denker, and S. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, 1989.
[18] N. Lee, T. Ajanthan, and H. Tom. SNIP: Single-shot network pruning based on connection sensitivity. In International Conference Learning Representations, 2019.
[19] N. Lee, T. Ajanthan, P. H. S. Torr, and M. Jaggi. A signal propagation perspective for pruning neural networks at initialization. In International Conference on Learning Representations, 2020.
[20] N. Lee, T. Ajanthan, P. H. S. Torr, and M. Jaggi. Understanding the effects of data parallelism and sparsity on neural network training. In International Conference on Learning Representa- tions, 2021.
[21] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In IEEE International Conference on Computer Vision, 2015.
[22] Z. Liu, H. Mu, X. Zhang, Z. Guo, X. Yang, K.-T. Cheng, and J. Sun. MetaPruning: Meta learning for automatic neural network channel pruning. In IEEE International Conference Computer Vision, 2019.
[23] Z. Liu, M. Sun, Zhou, G. Huang, and Darrell. Rethinking the value of network pruning. In International Conference on Learning Representutions, 2019.
[24] I. Mehta, M. Gharbi, C. Barnes, E. Schechtman, R. Ramamoorthi, and M. Chandraker. Mod- ulated periodic activations for generalizable local functional representations. arXiv preprint 2 2104.03960v1. 2021.
11

---

[25] S. Menon, A. Damian, S. Hu, N. Ravi, and C. Rudin. PULSE: Self-supervised photo upsam- pling via latent space exploration of generative models, In /EEE/CVV Conference on Computer Vision and Puttern Recognition, 2020.
[26] L. Mescheder, M. Occhsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3D reconstruction in function space. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[27] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
[28] A. S. Morcos, H. Yu, M. Paganini, and Y. Tian. One ticket win them all: generalizing lottery ticket initializations across datasets and optimizers. In Advances in Neural Information Processing Systems, 2019.
[29] M. C. Mozer and P. Smolensky. Skeletonization: A technique for trimming the fat from network via relevance assessment. In Advances in Neural Information Processing Systems, 1988.
[30] A. Nichol, J. Achiam, and J.dJ.Sooe Schulman. On first-order meta-learning algorithms. arXiv preprint 1803.022 300300299,,, 2018.
[31] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepSDF:
Learning con- tinuous signed distance functions for shape representation. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2019.
[32] S. Peng, M. Niemeyer, L. Mescheder, and A. Geiger. Convolutional occupancy networks. In European Conference on Computer Vision, 2020.
[33] S. Ravi and H. Larochelle. Optimization Conference on Learning Representations, 2017.
model for few-shot learning. In International
[34] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3D- structure-awe neural scene representations. In Advances in Neural Information Processing Systems, 2019.
[35] V. Sitzmann, E. R. Chan, R. Tucker, N. Snavely, and G. Wetzstein. MetaSDF: Meta-learning signed distance functions. In Advances in Neural Information Processing Systems, 2020.
[36] V. Sitzmann, J. N. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Process- ing Systems, 2020.
[37] I. Skorokhodov. S. Ignatyev, and M. Elhoseiny. Adversarial generation of continuous images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[38] H. Tanaka, Kunin, D. L. K. Yamins, and S. Ganguli. Pruning neural networks without any data by iterative conserving synaptic flow. In Advances in Neural Information Processing Systems, 2020.
[39] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keit Keil, N. Raghavan, U. Singhal, R. Ra- mamoorthi, J. T. Barron, and R. Ng. Fourier features let networks learn high frequency func- tions in low dimensional domains. In Advances in Neural Information Processing Systems, 2020.
[40] M. Tancik, Mildenhall, Wang, D. Schmidt, P. Srinivasan, J. T. Barron, and R. Ng. Learned initializations for optimizing coordinate- neural representations. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[41] A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, K. Sunkavalli, R. Martin-Brualla, T. Simon, J. Saragih, M. Nießner, R. Pandey, S. Fanello, G. Wetzstein, J.-Y. Zhu, C. Theobalt, M. Agrawala, E. Shechtman, B. Goldman, and M. Zollhöfer. State of the art on neural rendering. Eurographics, 2020.
[42] S. Thrun and L. Pratt. Learning to Learn. Springer, 1998.
[43] H. Tian, B. Liu, X.-T. Yuan, and Q. Liu. Meta-learning with network pruning. In European Conference on Computer Vision, 2020.
[44] C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference Learning Representations, 2020.
12


Reviews:
[{"review_id": "slEnJVxMrkO", "reviewer": "Reviewer_oRre", "summary": "This paper presents a method of alternately meta-learning the Implicit Neural Representations (INR) for a set of signals using the MAML method, and pruning the INR using the global magnitude-based pruning method for sparsifying the INR. The proposed method is tested on the 2D image regression task in three datasets (CeleA, Imagenette and 2D SDF). In comparisons, three baselines are used (MAML+w/ random pruning,  MAML+w/ dense-narrow, and w/o MAML + dense-narrow). The proposed method obtains better PSNR across different number of optimization steps, thus showing more efficient learning in terms of computation and memory footprint. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the proposed method seems to apply existing techniques, a meta-learning method using MAML and a pruning strategy using global magnitude-based pruning, to sparsify the INRs.  The technical novelty thus seems to be limited.  The baselines also are limited. It will be better If different meta-learning methods and pruning strategies are tested.  So, I think this paper is marginally below the bar of NeurIPS. \n\nMore specifically, \n\nPros: \n+ The paper is well motivated and easy to follow. \n+ The observations of applying MAML+Magnitude-based Pruning in sparsifying INRs are informative. \n+ The experimental results are promising with good analyses. \n\nCons: \n- Only one type of meta-learning methods, MAML and one type of pruning strategies, magnitude-based pruning are studied. More are entailed to gain more insights. \n- Vanilla INRs are used (SIREN and FFN). It will be better if some new designs are studied towards more effective and more efficient meta-learning and pruning. \n- The proposed method does not test the cross-dataset meta-learning and pruning, which seems to be more interesting for scaling up INRs in a computation and memory efficient way", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the proposed method are considered in the manuscript, but not sufficient as stated above. Potential ethical side-effects are addressed. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "about 3"}, "extra_text": {}}, {"review_id": "X4JgozSXW5t", "reviewer": "Reviewer_mx13", "summary": "The paper describes a method for representing sets of signals using coordinate-based neural representations (or INRs) by combining a meta learning approach for learning the initial weights with a network pruning approach for ensuring that this initialization is as sparse and memory-efficient as possible. The paper contributes an algorithm, Meta-SparseINR, which alternates between meta-learning weight initializations for coordinate-based networks and pruning the weights in the meta-learned initialization. The algorithm results in a sparse network initialization which can be then specialized in a few gradient steps to represent any signal in a dataset, and generalize to new signals taken from the same distribution. The paper demonstrates that this additional pruning step built into the meta learning process leads to improved results when compared to just meta-learning dense network initializations with a comparable number of parameters. The method is evaluated on an image-fitting task for multiple datasets, and it is shown that this approach does lead to a better performance versus memory trade-off when compared to just network pruning or meta-learning alone.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I will break down the review into an analysis of strengths and weaknesses of the paper. In my opinion, the strengths of the paper are:\n1. The results show that applying the combination of meta learning and network pruning iteratively as in Meta-SparseINR result in noticeable improvement over performing either of these methods alone. This is a valuable contribution, since it justifies the papers’ claim that this method can bridge the gap between meta learning methods which speed up learning INRs, and quantization or pruning methods which decrease the memory required to store a dataset of INRs.\n    - The experiments show that Meta-SparseINR outperforms just performing meta learning with a representation with the same number of parameters, especially when there are few parameters (Fig 2). Interestingly, the sparse representation also converges with less optimization steps (Fig 3a).\n    - The experiments show that Meta-SparseINR retains the benefits of meta learning and converges significantly faster to a higher quality representation than training from scratch (and presumably pruning or quantization this scratch training, since this likely does not improve the quality of the representations trained) (Fig 3a).\n2. The paper is well motivated - one of the largest challenges with coordinate-based networks is generalization to represent datasets of signals. This paper attempts to remedy this by considering both axes of computational requirements to do this - optimization time, and memory consumption. This could potentially enable representing large datasets with INRs, which would allow for easier studying of the characteristics of (and possibly generalization across) these types of signal representations.\n3. The paper is written very clearly and concisely. The experiments shown directly justify the claims that are made in the paper, especially those in Fig 2, 3, 4. The methodology is described in detail, and justification for design choices is provided. This is valuable as it makes it much easier to understand what the method is doing, and thus more likely to be adopted into future research work.\n\nThe weaknesses of the paper are:\n1. The main concern I have is related to the choice of learning a sparse initialization, instead of learning an initialization which can then be made sparse when fit to each dataset signal. I have a few questions and possible concerns:\n    - The method only learns a sparse initial INR. Once this initialization is applied to represent a new signal, is the learned mask retained and the final represented signal also sparse? It must be, otherwise memory is only saved when storing the **initialization**, and not each of the dataset signal representations.\n    - The most logical baseline to compare to would be one which first meta-learns an initialization (non-sparse), and then uses this initialization to fit each of the dataset signals independently (non-sparse), and then applies some pruning or quantization method to each of these final representations (sparse or quantized). In this case, the (non-sparse) meta-learned initialization would still provide the same speedup bonus on optimization, and the quantization or pruning on the (non-sparse) dataset signal representation could provide a similar memory benefit in storage by making it sparse or quantized. I understand that pruning is an iterative process, which requires fine tuning after making a representation sparse, so this type of baseline would likely still incur additional computational cost when fitting each dataset signal for this fine tuning. But, I still think this makes sense to compare to, in terms of computational requirement and representation quality, alongside just simply applying meta learning to a smaller INR architecture.\n    - Without some comparison like this or explanation why this baseline doesn’t make sense, I think that it’s not super well justified why the meta learning and pruning steps need to be iteratively applied to the initialization, instead of sequentially applied to the initialization and then the dataset signals. Since this is the main originality of the contributed algorithm, I think some justification of this would make the paper significantly stronger.\n2. I think there is a flaw in the conclusion of section 5.3. Figure 5 shows that pruning using the *Winning Ticket* method performs equivalently to just training a smaller representation, which is supposed to highlight some benefit of sparsity when trained with the meta-initialization. However, it is explained that these results are obtained by training each of these representations for 10k steps, while the comparable plot where the sparse initializations outperform the smaller networks is only benchmarked after training for 100 iterations (Fig 2). I suspect that meta-learning only enables improved performance for a limited number of iterations, after which the magnitude of contribution of the initialization likely doesn’t matter anymore. I.e., if Fig 2 was left to train for as long as Fig 5, it’s possible that we’d see the same trend with the meta-learned initializations as well.\n\nOverall, the paper contributes a valuable algorithm for combining network pruning with meta-learning methods. The contribution of this algorithm is well-motivated for learning to represent classes of signals using INRs, and the paper shows that it does better than existing methods which apply just meta learning or pruning. Because of this, I think the paper is already quite good. I believe that the comparison to independently performing meta-learning **and** network pruning could remove all reasonable doubt that the formulation of Meta-SparseINR for learning the initialization, and then directly optimizing this sparse representation on dataset signals, is the right way to go about learning a dataset of INRs. This, along with some clarification on the conclusions drawn from Fig 5, would make this a much stronger paper.\n\nSome other minor comments which could possibly be clarified to improve the quality of the paper:\n- For Fig 5, the supplement mentions that this was obtained from a single image of a Fox? Does this trend hold for multiple images.\n- Can the repeated application of sparsity throughout the meta learning process reduce the memory required to train the model? One big issue with MAML and other gradient-based meta learning algorithms is the memory requirement of differentiating through the optimization process. If making this network sparser can reduce this, this could be very impactful.\n\n**Update after Author Response**:\nThe additional experiments provided by the authors have addressed the concerns brought up in my original review. I retain my score, and think that this is a good paper which should be accepted.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impacts of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "In my opinion, there are no ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "Uc5Hd4jGpe-", "reviewer": "Reviewer_kwH6", "summary": "Authors propose a method for learning sparse implicit functional representations for signal data. `Their method combines ideas from MAML to learn a good initialization from which a sparse neural network representing the signal can be efficiently estimated. Experiments on a few datasets suggest that the proposed method achieves better PSNR than more naive approaches for the same levels of sparsity. Overall, the paper is in a good shape but still could benefit from a few additions.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty:\nTo my knowledge, both the application and the proposed method are novel.\n\nClarity:\nOverall the paper is well-written and easy to follow.\n\nA minor comment is that I would prefer more rigour in presenting eq. 4 and connecting it to algorithm 1, particularly, in how the iterative procedure is actually performing the inner-loop optimization explicitly writtein eq. 4. Does the algorithm at least provably improves on the objective? It is not completely obvious to me and I would like to know exactly what guarantees Algorithm 1 has.\n\nSignificance:\n\nTo me it's fairly interesting to be able to sparsely represent natural signals both from abstract point of view and also because it's opening certain applications such as more efficient image or audio compression algorithms.\n\nI'm this very curious if there is any chance of comparing their method to standard image compression algorithms such as JPEG in the paper which should not be very difficult.\n\nQuality:\n\nOverall I find the proposal method technically sound and experimental evaluation adequate (also see my comment in the clarity section).\n\nOne of the baselines which I think is missing from the experimental evaluation is some kind of a functional/implicit-style auto-encoder with the varying dimensionality of the hidden variable. I understand that it won't be fully comparable to the model because even if the code itself is short, the decoder network still needs to be transmitted somehow, but arguably one can do this beforehand, so I still think it is a valid experiment (also see question 1).\n\nI also don't fully understand if a more standard lottery-ticket method can applied for the task of simply learning a sparse representation of a single signal. Albeit, again, not being fully comparable to the proposed method, it would probably achieve even better PSNRs and thus provide an important, perhaps gold-standard data-point.\n\nOther questions:\n1) Have authors tried an alternative approach where the neural network representing the function is not sparse itself, but only a few parameters of it are allowed to be modified and supposed to contain information about a particular signal? \nAn example of this approach be found in [1] (although there the setting was quite different), where certain weights in some of the fully-connected layers are modifyable. \n2) Why did authors use Imagenette and not the standard ImageNet? I would be more confident (perhaps, falsely) in the results if they were performed on a more diverse dataset. I don't think this would require much work compared to Imagenette experiments and I'm very curiuous to see the results.\n\nReferences:\n[1] Bartunov, S., Rae, J. W., Osindero, S., & Lillicrap, T. P. (2019). Meta-learning deep energy-based memory models. arXiv preprint arXiv:1910.02720.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Discussed adequately.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to learn functional representations of signals such as images using meta learning and network pruning, so that an initial sparse network can be perturbed to fit each training or testing image. The proposed method is novel, simple and clean, and it can be useful in applications based on functional representations. The experiments are solid and thorough, especially with the additional experiments performed during the rebuttal period. 

After rebuttal, this paper received three final ratings of 7, 7, 7 (the reviewer who initially assigned a rating of 5 upgraded the rating to 7, but was unable to make the change formally due to technical error). The rebuttals have addressed all the concerns of the reviewers. 

This paper makes a valuable contribution and can be accepted. 
Decision: Accept (Poster)