Meta-Learning Sparse Compression Networks
Authors: Jonathan Schwarz
Yee Whye Teh
Venue: Accepted by TMLR ()

Abstract: Recent work in Deep Learning has re-imagined the representation of data as functions mapping from a coordinate space to an underlying continuous signal. When such functions are approximated by neural networks this introduces a compelling alternative to the more common multi-dimensional array representation.  Recent work on such Implicit Neural Representations(INRs) has shown that - following careful architecture search - INRs can outperform established compression methods such as JPEG (e.g. Dupont et al., 2021). In this paper, we propose crucial steps towards making such ideas scalable: Firstly, we employ state-of-the-art network sparsification techniques to drastically improve compression. Secondly,introduce the first method allowing for sparsification to be employed in the inner-loop of commonly used Meta-Learning algorithms, drastically improving both compression and the computational cost of learning INRs. The generality of this formalism allows us to present results on diverse data modalities such as images, manifolds, signed distance functions, 3D shapes and scenes, several of which establish new state-of-the-art results.

Published in Transactions on Machine Learning Research (08/2022)
Meta-Learning Sparse Compression Networks
Jonathan Richard Schwarz
schuarajn
DeepMind
University College London
Yee whye Teh DeepMind
yuteh yutch aggoogle.coooooooooooooo com
Reviewed on OpenReview: https: // openreview. net/ net/ forum? id- Cct Hkg
Abstract
Recent work Deep Learning has re-imagined the representation of data as functions mapping from a coordinate space to an underlying continuous signal. When such functions are approximated by neural networks this introduces compelling alternative to the more common multi-dimensional array representation. Recent work on such Implicit Newral Representations (INRs) has shown that following careful architecture scarch INRs can outperform established compression methods such as JPEG (e.g. Dupont et al., 2021). In this paper, we propose crucial steps towards making such ideas scalable: Firstly, we employ state- of-the-art network sparsification techniques to drastically improve compression. Sccondly, introduce the first method allowing for sparsification to be employed in the inner-loop of commonly used Meta-Learning algorithms, drastically improving both compression and the computational cost of learning INRs. The generality of this formalism allows us to present results on diverse data modalities such as images, manifolds, signed distance functions, 3D shapes and scenes, several of which establish new slate-of-the-arl results.
1 Introduction
An emerging sub-field of Deep Learning has started to re-imagine the representation of data items: While traditionally, we might represent an image or 3D shape as a multi-dimenssonal array, continuous representations of such data appear to be a more natural choice for the underlying signal. This can be achieved by defining a functional representation: mapping from spatial coordinates (x, y) to (r, g, b) values the case of an image. The problem of learning such a function is then simply a supervised learning task, for which we may employ neural network- an idea referred to Implicit Neural Representations (INRs). An advantage of this strategy is that the algorithms for INRs are data agnostic we may simply re-define the coordinate system and target signal values for other modalities and readily apply the same procedure. Moreover, the learned function can be queried at any point, allowing for the signal to be represented at higher resolutions once trained. Finally, the size of the network representation can be chosen by an expert or as we propose in this work, an algorithmic method to be lower than the native dimensionality of the array representation. Thus, this perspective provides a compelling new avenue into the fundamental problem of data compression. INRs are particularly altractive in cases where array representations scale poorly with the discretisation level (e.g. 3D shapes) the underlying signal is inherently continuous such in neural radiance fields (NerF) (Mildenhall et al., 2020) or when discrotisation is non-trivial, for example when data lies on manifold.
So far, the difficulty of adopting INRs as a compression strategy has been a trade-off between network size and approximation quality requiring architecture search (e.g. Dupont et al., 2021) or strong inductive biases single data point
(c.g. Chan et al., 2021; Mchta et al., 2021). Furthermore, the cost of fitting a network vastly exceeds the computational cost of standard compression methods such as JPEG (Wallace, 1992), an issue that is compounded when additional architecture search is required.

---

Published in Transactions on Machine Learning Research (08/2022)
Dense initialisation (Heta-Laamedi
0
Decoding
Oc
Denae Reconstruction
Sparse Adaptation
0
netorr & (dree)) (spame)
decompress
80
comprees
Encoding
Bit stream
Figure 1: Overview of MSCN compression method. In order to compress data item, perform sparse adaptation of a meta-learned initialisation, leading to small subset of changes 80 encoding the item. Sparsity reduces compression cost and avoids costly architecture search. Meta-Learning drastically cuts compression time. 50 can be subsequently compressed and encoded using standard techniques.
In this work, specifically focus improving the suitability of INRs as compression method by tackling the aforementioned problems. First, we recognise insights of recent deep learning studies (e.g. Frankle & Carbin, 2018) which show how only a small subset parameters encode the predictive function. We thus employ recent state-of-the-art sparsity techniques (Louizos et al., 2017) to explicitly optimise INRs using as few parameters as possible, drastically improving their compression cost.
Secondly, in order tackle the computational cost. of learning INRs, follow recent work (e.g. Lee et al., 2021; Tancik et al., 2021; Dupont et al., 2022a) by adopting Meta-Learning techniques such MAML (Finn et al., 2017) which allow learning an INR representing single signal by fine-tuning from learned initialisation using only a handful of optimisation steps. Crucially, our sparsity procedure allow efficient Backpropagation through this learning procedure, resulting an initialisation that is specifically optimised for sparse signals. This allows us to re-imagine the sparsity procedure as uncovering a network structure most suitable for the task hand. This noticeably different from related work which decouples Meta-Leat Mitta-Lerning from the sparsity procedure (Tian et al., 2020; Lee al., 2021).
Figure shows overview of our technique for compression. The key novelty of our work lies in the sparse adaplation phase, significantly reducing compression cost.
Finally, our framework is flexible and allows for Weight,, Representation- Group- and Gradiont-Sparssty with minimal changes and is thus suitable for many applications outside of the core focus of this work.
2
Background
We start by reviewing relevant aspects the Meta-Learning and INR literature that will constitute the fundamental building blocks of our method.
2.1 Implicit Neural Representations
Throughout the manuscript, represent INRs functions fe X Y parameterised by neural network with parameters 0 mapping from coordinate space X' to the output space y. INRs are instanco-specific, i.c. they are unique nelworks representing single data items. Its approximation quality is thus measured across all coordinates making up the data item, represented by the discrete index set I. As an example, for a 3D shape the coordinate space X' 3 (x, y, z) and y = [0, are Voxel occupancies. I the 3D-grid (O..... D)3 where D the discretisation level. We can thus formulate the learning of an INR a minimisation problem of the squared error between the INR's prediction and the underlying signal:
2

---

Published in Transactions on Machine Learning Research (08/2022)
min C(fo: x E y) min
> [lfo((s)) -vll IEI
(1)
which is typically minimised via Gradient Descent. As a concrete choice for for recent INR breakthroughs propose the combination Multi-Layer Perceptrons (MLPs) with either positional encodings (Mildenhall et al., 2020; Tancik al., 2020) or sinusoidal activation functions (Sitzmann et al., 2020b). Both methods significantly improve the reconstruction error of standard ReLU Networks (Nair & Hinton, 2010).
Of particular importance to the remainder of our discussion around compression is the recent observation that signals can be accurately learned using merely data-item specific modulations to a shared base network (Perez et al., 2018; Mchta et al., 2021; Dupont et al., 2022a). Specifically, in the forward pass of a network, each layer represents the transformation x + f(W(x + b(0) + m(),, where (W(), b(y) are weights and biases shared between signal with only modulations ( being specific each signal. This has the advantage of drastically reducing compression costs and is thus of particular interest to the problem considered by us.
2.2 Meta-Learning
It is worth pointing out that the minimisation of Equation (1) is extraordinarily expensive: Learning a single NeRF ((Mildenhall et al., 2020)) scene can take up to an entire day on a single GPU (Dupont et al., 2022a); even the compression of a single lww-dimeniional image requires thousands iterative optimisation steps. Fortunately, we need not resort to Tabula rasa optimisation for each data item in turn: In recent years, developments in Meta-Learning (e.g. Thrun & Pratt, 2012; Andrychowiez et al., 2016) have provided formalism that allow great deal of learning to be shared among related tasks in case signals in dataset.
Model-agnostie Meta Learning (MAML) (Finn et al., 2017) provides an pplimisationn-aaee based approach to finding an initialisation from which we can specialise the network to an INR representing signal in merely a handful of optimisation steps. Following custom notation, we will consider a set of tasks or signals ( J Finding a minimum of the loss on each signal is achieved by Gradient-based lcarning on the task-specific data (xT, yT.)- Writing LT, a shorthand for L(So, YT)), single gradient step from shared initialisation Og takes the form:
0 (VLT,,
(2)
which we can trivially iterate for multiple steps. The key insight MAML to define Mota-objective for learning the initialisation On as the minimisation of an expectation of the lask-specific loss after its update:
0g- arg min (T, arg min
(3)
where P(T) a distribution over signals dataset. The iterative optimisation of (3) is often referred to the MAML "outer loop" and (2) as the "inner loop" respectively. Note that this is a second-order optimisation objective requiring the differentiation through learning process, although first-order approximations exist. (Nichol & Schulman, 2018).
Indeed, this procedure has been widely popular in the work on INRs, e.g. being successfully used for NeRF scenes in Tancik et al. (2021) or signed distance functions (Sitzmann el al., 2020a). Finally, it should be noted that the idea of learning modulations explored in the previous section relies on the MAML process. Thus, the learning of weights and biases is achieved using (3), although only modulations are adopted in the inner-loop (2). Meta-learning subset of parameters in the inner-lopp corresponds to the MAML-derivative CAVTA (Zintgraf et al., 2019).
3

---

Published in Transactions on Machine Learning Research (08/2022)
3
Meta-Learning Sparse Compression Networks
This section introduces the key contributions of this work, providing a framework for sparse Mela-Learning which we instantiate in two concrete algorithms for learning INRs.
3.1 Lo Regularisation
While the introduction of sparsity in the INR setting is highly attractive from a compression perspective, our primary difficulty in doing so is finding a procedure compatible with the MAML, process described the previous section. This requires (i) differentiability and (ii) fast learning of both paramcters and the subnetwork structure.
We can tackle (i) by introducing Lo Regularisation (Louizos al., 2017), re- re-parameterised Lo objective using stochastic gates parameters. Consider again the INR objective (1) with sparse reparamcterisation f of dense set of parameters 0: 0 ④ z; ž3 E {0, }} and an LO Regularisation term on the gates with regularisation coefficient x. We can learn subnetwork structure by optimising distributional parameters π of distribution (Z(TT) the gates leading the regularised objective:
dlm(r)
min L (fg. = Ex((((r) 0,
Σ iE
11fooz Vill2
+ A
(4)
j=1
where A Ej=1 π, penalises the probability of gates being non-zero.
())
The key overcoming non-differentiabilit due to the discrete nature of is smoothing the objective: This is achieved by choosing an underlying continuous distribution (() and transforming its random variables by a hard rectification: = min(1, max(0.s)) = g(s). In addition, we note that the LO penalty can be naturally expressed by using the CDF of q to penalise the probability the gate being non zero (1 - Q(s - 0)Φ;)):
dim(
min C (f.. Ø) =Eg(s}ø) 0.
[1fooo(s) -y:13 ier
Q(s (00)55000) 0(0,)
(5)
A suitable choice for q(z π is a distribution allowing for the reparameterisation trick (Kingma & Welling, 2013), i.c. the expression of the expectation in (5) as an expectation a parameter-free noise distribution p(e) from which we obtain samples of through a transformation (): This allows simple Monte-Carlo estimation of (5):
dim (p)
min C CR xb)
Σ 1- Q(s S 0(0b))se ~ p(E)
(6)
EE
The choice for q(s()) in Louizos et. al. (2017) is the Hard concrete distribution, obtained by stretching the concrete (Maddison et al., 2016) allowing for reparameterisation, evaluation of the CDF and exact zeros in the gates/masks masks z. A suitable estimator is chosen at test time.
3.2 Sparsity in the inner loop
We are now place build our method from the aforementioned building blocks. Concretely, consider the application of Lo Regularisation in the MAML inner loop objective. Using a single Monte- Carlo sample for simplicity, re-write the MAML meta-objective
4

---

Published in Transactions on Machine Learning Research (08/2022)
dim(
On arg min ETT) C% (Sor (r)) (fo
arg min B [CT, ((1+
+
Σ 1 Q(;; s 0(01)|
(7)
= g((e,, )); ø' = 0n- + dΦ and p(E)
where 80 and deb are updates to both parameters and gate distributions computed in the inner loop (2). Note that this constitutes a fully differentiable sparse Meta-Learning algorithm.
However, a moment of reflection reveals concerns with (7): (i) The joint learning of both model and mask parameters in the inner-loop expected to pose a more difficult optimisation problem due to trade-off between regularisation cost. and lask performance. This is particularly troublesome as the extension of MAML long inner-loops is computationally very expensive and hence still an active area of research (e.g. Flennerhag et al., 2019; 2021). (ii) The sparsification (0 (0+ 50) @ z' is sub-optimal from compression standpoint: As the signal-specific compression cost is 80, there is no need to compute the INR using sparse network, provided 80 is sparse. 0g is
set of parameters and its compression cost thus amortised.
Our remedy to (i) is take inspiration from improvements on MAML that propose learning further parameters of the inner optimisation process (Li el al., 2017) such as the slep size (known as MetaSGD). In particular, we learn both an initial set of parameters 0g and gates Φo through the outer loop, i.c.
dim(
0,, Φu arg min Ein Cr (fotss)oott)))) + 0.6
Σ j=1
1 Q(s S 5((0++6)))))
(8)
This has interesting consequences: While we previously relied solely on the inner-loop adaptation procedure to "pick out" an appropriate network for each signal, we have now in effect reserved a sub-network that provides a particularly suitable initialisation for the set of signals at hand. This sub-network can either be taken to be fixed (such as when the adopted networks are used to learn a prior or generative model (Dupont et al., 2022a)) or adopted within an acceptable budget of inner steps, providing a mostly overlapping but yet signal-specific set of gates.
With regards to concern (ii) it is worth noting that gates z may be applied at any point in the network. This is attractive as it provides us with a simple means to implement various forms of commonly encountered sparsity: 1. Unstructured sparsity by a direct application to all parameters 2. Structured sparsity by restricting the sparsity pattern 3. Group sparsity by sharing a single gate among sets parameters 4. representational sparsity by gating activations or 5. Gradient Sparsity by masking updates in the inner loop. This highlights strength of our method: the principles discussed so far allow for framework in which spansity can be employed in a Meta-Learning process in variety of different ways depending the requirements of the problem at hand. We refer to this framework with shorthand of this manuscript's 's title: MSCN.
With regards to compression considerations, more natural objective would thus involve a term (0, + (0 6 in the inner loop, which ensures that we directly optimise for per-signal performance using an update to as few parameters as possible the real per-signal compression cost. Note that as Op dense, the resulting to + 80 (sparse) is still dense, thus allowing the use of more capacity comparison to fully sparse network. We suggest two concrete forms of 50 in the next section.
3.3 Forms of 80
3.3.1
Unstructured Sparse Gradients
Perhaps the most natural form of implementing a direct sparsification of 80 is through gating of the gradients. Assuming a budget of inner-loop steps T and writing R for the regularised Lo objective and 0, for the state of the parameters after i updates, this takes the form of unstructured sparse gradients:
5

---

Published in Transactions on Machine Learning Research (08/2022)
G 80
2 O 60
z Va. ((o...))
i=1
8n
X f(W x + b + +2 @ 9)
G 80
(b) Structured Sparse Modulations
(a) Unstructured Sparse Gradients
Figure 2: Instantiations of the MSCN framework.
50 - Σ & (o,
(9)
and thus forces updates to concentrate parameters with the highest plasticity in few-shot learning setting. We impose no restrictions on the gates, applying them to both weight and biases gradients and thus allow for complex gradient sparsity patterns to emerge. In this setting, we found the use of aforementioned MetaSGD (Li et al., 2017) to be particularly effective.
3.3.2 Structured Sparse Modulations
An alternative form allows known inductive biases to inform our application of sparsity: In the second proposed instantiation of the MSCN framework we work in the case where only modulations (m are allowed to adapt to each task-specific instance (see Section 2.1). The sparsification of those modulations is straight-forward and is achieved by introducing laycr-spocific gate:
f(W() (DD 2 @ (!)
(10)
such that the only non-zero entries in 50 are for sparse modulations. For of notation omit zero-entries and simply wrile 80 = (2) @ compression due to the comparably low dimensionality of m() Further sparsifying the modulations has the advantage of allowing the use of very deep or wide base networks Gg, ensuring that common structure is modelled as accurately as possible while a small communication cost continues to be paid for 80. Note also that an intuitive argument for deep networks that modulations in early layers have a large impact on the rest of of the network. Both forms shown in Figure
2(4) @ m(4). This provides particularly attractive form for
4 Related work
4.1 Neural network sparsity
While dating back least to the early 1990s, recent interest in sparsifying Deep Neural Networks has come both from experimental observations such as the lottery ticket hypothesis (Frankle & Carbin, 2018) and the unparalleled growth of models (e.g. Brown et al., 2020; Rae et al., 2021), often making the cost of training and inference prohibitive for all but the largest institutions in machine learning.
While more advanced methods have been studied (e.g. LeCun et al., 1989; Thimm Fiesler, 1995), most contemporary techniques rely on the simple yet powerful approach of magnitudo-based pruning removing a pre-determined fraction by absolute value. Current techniques can be mainly categorised as the iterative sparsification of densely initialised network (Gale et al., 2019) techniques that maintain constant sparsity

---

Published in Transactions on Machine Learning Research (08/2022)
throughout learning (e.g. Evci et al., 2020; Jayakumar et al., 2020). Critically, many recent techniques would not be suitable for learning in the inner-loop of a meta-learning algorithm due to non-differentiabiiiy, motivating our choice of a relaxed Lo regularisation objective.
4.2 Sparse Meta-Learning
Despite Meta-Learning and Sparsity being well established research areas, the intersection both topics has only recently started to attract increased attention. Noteworthy early works are Liu al. (2019), who design a network capable of producing parameters for any sparse network structure, although an additional evolutionary procedure is required to determine well-performing networks. Also using MAML Meta-learner, Tian et al. (2020) employ weight sparsity as means to improve generalisation.
Of particular relevance to this work is MetaSparseINR (Lee et al., 2021), who provide the first spanse Meta- Learning approach specifically designed for INRs. We can think of their procedure as the aforementioned iterative maguitude-based pruning technique (Ström, 1997; Gale et al., 2019) applied in the outer loop of MAML training. While has the advantage of avoiding the difficulty of computing gradients through a pruning operation; iterative pruning is known to produce inferior results (Schwarz al., 2021) and limits the application to an identical sparsity pattern for each signal. Due to the direct relevance to our work, we will re-visit MetaSparscINR a key bascline in the experimental Section.
4.3 Implicit Neural Representations
The compression perspective of INRs has recently been explored in the aforementioned COIN (Dupont et al., 2021) and in Davies et al. (2020) who focus on 3D shapes. Work on videos has naturally received increased attention, with (Chen et al., 2021) focusing on convolutional networks for video prediction (where the time stamp is provided as additional input) and Zhang et al. (2021) proposing to learn differences between frames via flow warping. While still lacking behind standard vidco codees, fast progress is being made and carly results are encouraging, An advantage of our contribution is that the suggested procedures can be readily applied to almost all deep-learning based architectures.
The recently proposed Functa (Dupont et al., 2022a) is a key bascline in our work as it also works on the insights of the modulation- network which maps a low dimensional latent vector to the full set of modulations. The instance-specific communication cost is thus the dimensionality of that latent vector. A minor disadvantage is the additional processing cost of running the latent vector to modulations network.
approach. Rather than using sparsity, the authors introduce a second
A particularly interesting observation is that quantisation of such modulations is highly effective (Strümpler et al., 2021; Dupont et al., 2022b), showing that simple uniform quantisation and arithmetic coding can significantly improve compression results.
Finally, is also worth noting that work on INRs related to the literature on multimodal architectures which have so far been mostly implemented through modality-speciffe feature extractors (e.g Kaiser al., 2017), although recent work has used single shared architecture (Jaegle et al., 2021).
5 Experiments
We now provide an extensive empirical analysis of the MSCN framework in the two aforementioned instan- tiations. Our analysis covers wide range of datasets and modalities ranging from images to manifolds, Voxels, signed distance functions and scenes. While network depth and width vary based on established best practices, in all cases we use SIREN-style (Sitzmann et al., 2020b) sinusoidal activations. Throughout the section, we make frequent use of the Peak signal-to- to noise ratio (PSNR), a metric commonly used to quantify reconstruction quality. For data standardised to the [0, 1] range this is defined as: PSNR = log to (MSE) where MSE is the mean-squared error. Further experimental details such as data processing steps and hyperparameter configurations in the Appendix.
7

---

Published in Transactions on Machine Learning Research (08/2022)
5.1 Unstructured sparse Gradients
In the unstructured spansity case (Section 3.3.1), the closest related work is MetaSparseINR (Lee et al., 2021) which will be the basis for our evaluations. Experiments in this section focus on Images, covering the CelebA (Liu et al., 2015) & ImageNotte (Howard, 2022) datascts as well as Signed Distance Functions (Tancik et al,, 2021), all widely used in the INR community. All datasets have been pre-processed to a size of 178 x 178. We follow the MetaSparseINN authors in the choice of those datasets and thus compare directly to that work as well as the baselines discussed. We provide a description of those baselines in the Appendix.
Narrow
Mota See SeIN
Spta-Spar
MAML- HP
Schtthh
MSCN (na MetwSStD)
Dense Narrow
MAML Onushot
MAPLIIMP
MBCN
MANL-BIP BIP
MSCN
Scrotch
MSCN
MAML+
MANLSOrwhhot
60
PSNR
40
PSWR
25
20
15
20
20
15
30
20
15
10
Fraction sunviing Parameters 194)
Fraction of survivig Parameters (36
Fraction of surviving Parameters (8)
(a) CelebA
(b) SDF
(c)
ImageNette
Figure 3: Quantitative results for unstructured sparsity. Results show the mean and standard deviation over three runs of each method.
Closely following the MctaSparseINR sctup, the network for all tasks is a 4-layer MLP with 256 units, which we mela-train for 2 inner loop steps using a batch size of 3 examples. We use the entire image to compute gradients in the inner loop. In order to correct for the absence of MetaSGD (Li et al., 2017) in MetaSparseINR, we also provide results using a fixed learning rate for SDF & CclebA as a fair comparison, although we strongly suggest to use MetaSGD as a default. Thanks to the kind cooperation of the MetaSparseINN authors, all baseline results are directly taken from their evaluation, ensuring an apples-to-apples comparison.
Full quantitative results across all datasets for varying levels of target sparsity are shown in Figure 3. In all cases notice a significant improvement, especially at high sparsity levels which are most important for good compression results. At its best, MSCN (with MetaSGD) on CelebA outperforms MetaSparseINR by over 9 decibel at the highest and over 4 PSNR at the lowest sparsity levels considered. As PSNR is calculated on a log-scale this is a significant improvement. To provide a more intuitive sense of those results we provide qualitative results in Figure 4b. Note that at extreme sparsity levels the MetaSparseINR result is barely distinguishable while facial features in MSCN reconstructions are still clcarly recognisable. Further qualitative examples are shown in the Appendix.
An interesting aspect of our method is the analysis of resulting sparsity patterns. Figure 4a shows the distribution of sparsity throughout the network at various global sparsity levels. We notice that such patterns vary both based on the overall sparsity level as well as the random initialisation, suggesting that optimal pattern are specific to each optimisation problem and thus cannot be specified in advance to a high degree of certainty. It also worth pointing out that existing hand-designed sparsity distributions (c.g. Mocanu et al., 2018; Evei et al., 2020) would result in a different pattern, allocating equal sparsity to layers 2-4, whereas our empirical results suggest this might not be optimal in all cases.
8

---

Published in Transactions on Machine Learning Research (08/2022)
(b)
Figure 4: Results on CelebA. (a) Sparsity Pattern across layers at various levels of overall network sparsity. Results shown over 3 random seeds. (b) Qualitative results on CelebA for various levels of sparsity.
5.2
Structured sparse modulations
Datasct, array size
Model
Performance at modulation size
64
128
256
512
1024
ERA5, 181 360
Funeta
43.2
43.7
43.8
44.0
44.1
MSCN
44.6
45.7
16.0
46.6
16.9
CelebA-HQ,
23.3
28.0
MSCN
21.8
23.8
25.7
28.1
30.9
ShapeNet 10, 64
Functa
99.30
99.40
99.44
99.50
99.55
MSCN
99.50
99.63
SRN Cars, 128 x 128
Functa
22.4
23.0
23.1
23.2
23.1
MSCN
22.8
24.0
24.3
24.5
24.8
Table 1: Quantitative results for each dataset. Shown the mean reconstruction error for various modulations sizes. Metrics are Voxel accuracy for ShapeNet 10 and PSNR for others. Corresponding sparsity levels for MSCN are: 64: 99.1%, 128: 98.2%, 256: 96.4%, 512: 92.9%, 1024: 85.7%.
Figure 5: Modulations show a
high level of reuse (top). Modula- tion distribution for 1024 (middle) & 64 modulations (bottom).
In this section, consider the evaluation of the sparse structured modulation setting (Section 3.3.2). The closest competitor is Functa (Dupont et al., 2022a), which we take to be our baseline method. We evaluate on Vozels using the top 10 classes of the ShapeNet dataset (Chang et al., 2015), NeRF scenes using SRN Cars (Sitzmann et al., 2019), manifolds using the ERA-5 dataset (Hersbach et al., 2019) and on images using the CelebAHQ dataset (Karras el al., 2017).
Due to the relatively low dimensionality of modulations, networks in this section are significantly deeper, comprising 15 layers of 512 units cach. In accordance with Functa, report results using MetaSGD with 3 inner-loop sets and varying batch sizes (see Appendix for details) due to memory restrictions. We show full quantitative results in Table 1, noting that we outperform Functa by a noticeable margin in almost all settings.
Resulting sparsity patterns shown in Figure (middle bottom) are particularly interesting, showing that our mothod Icads to the intuitive result of preferring allocate most of its modulation budget carlier layers, as such modulations have the potential to have large effect on the whole network. Indeed Dupont et al. (2022a) write "reconstructions can reduce the number of shift modulations by only using them for the first few layers of the MLP". A possible explanation for that observation is that while early modulations do make the largest difference, our method continues to make use of modulations in later layers. This is another argument for learned sparsity patterns over pre-defined distributions, which would result in a mostly uniform allocation throughout the network.
more sensitive to earlier modulation layers then later ones. Hence
Interestingly, Figure 5 (top) shows that modulations show high-degree of reuse. We plot the fraction of modulations that are re-used when starting an optimisation process from the same random initialisation but

---

Published in Transactions on Machine Learning Research (08/2022)
allowing for larger number of modulations (x-axis) relative to network with fewer modulations (distinguished by colours). In all cases the fraction of re-use is much higher than with a random allocation.
Furthermore, we provide qualitative results in Figures 6 and 7. In both cases we observe noticeable improvements over Functa. To provide a qualitative notion of gains afforded by larger number modulations, we show results for MSCN Figure 6d.
(b)
(d)
Figure 6: Quantitative results for structured sparsity on ShapeNet10. (a)-(c) Comparison with Functa for 1024 Modulations (d) Reconstruction quality for increasing number of modulations.
fureta (PSMR: 43.20)
(c)
(a) Functa
(b) MSCN
Figure 7: Qualitative results on ERA5 (1024 modulations). (a)-(b) Prediction errors shown on the manifold
for 1024 modulation. Best viewed as a gif (Functa, MSCN). (c) Full error shown over the entire map.
5.3 Compression performance
Returning to of the key motivations of work, now provide comparison with various commonly used compression schemes. The closest method to our work is COIN+ (Dupont et al., 2022b), which is an application of the previously discussed Functa. for compression. The authors apply uniform quantisation and entropy-coding to the dense latent vector and compare to a wide range of traditional and learned compression algorithms.
In order for MSCN to be competitive quantisation scheme, we follow the COIN+ approach to quantisation and entropy coding, which we apply to any non-zero modulations remaining after the trained architecture has has been adapted to single data item. Identical to the observations for COIN+ find that modulations can be sparsified to high level, allowing the quantisation of a standard 32bit representation to only 5-6 bits with little loss of performance. For the large images found in the Kodak datasct, we split large images into smaller patches that are represented separately. As any shared weights are identical for each example, we consider their cost amortised (i.e. they are part the compression program) and are thus not reported in the following results. This standard practice. For this reason, we force the use of identical sparsity patterns for each example (i.e. we do not update @ in Equation equation 8) avoid the otherwise necessary cost of communicating the sparsity pattern. Finally, we found the structured sparsity formulation of Section 3.3.2 to
10

---

Published in Transactions on Machine Learning Research (08/2022)
be most suitable for optimal compression, combining inductive biases with structure learning algorithms. We provide further details on these experiments in the Appendix.
37.5
35.0
2013
35
52.5
2
Sh-ne thad
(a) CIFAR10
(b) KODAK
(c) Qualitative comparison
Figure Rate-distortion plots on CIFAR10 (a) and Kodak (b). (c) shows qualitative comparison on CIFAR10. The 128/ 1024 modulation rows correspond to evaluating to the leftmost and rightmost point of the MSCN curve in (a). Results almost indistinguishable for 1024 modulations. Best viewed computer.
(a) Original
(b) Compressed
(c) Residuals
(d) Original
(e) Compressed
(f) Residuals
Figure 9: Qualitative examples on the Kodak dataset. The PSNRS achieved are 25.58 (left) and 31.77 (right).
We provide comparison with various compression schemes for CIFAR10 in Figure 8a and Kodak (for which pre-train on the DIV2K dataset (Agustsson & Timofte, 2017) in (Strümpler et al., 2021)) in Figure 8b. The x-axis shows the bit-rate in sets-per-pixell 3. We note that MSCN consistently outperforms COIN++, suggesting that the previously observed performance improvements over Functa also hold for compression applications. We also find that our method is competitive specifically in the low bit-rate regime, where we achieve strong performance improvements over JPEG/JPEG2000, Note that the strong improvement over COIN++ on Kodak may be a result on the pre-training on Div2k (COIN++ uses frames of Vimeo90k), whereas the results on CIFAR10 in line with what can be expected given the improvement over Functa in the previous section. Qualitative results shown in Figures 8c and 9.
As hypothesised in COIN+ the resulting gap to state-of-the-arl codecs like BMS & CST may be due to their use of deep generative models for entropy coding which could be added to our formulation in future work with little conceptual work. Moreover, the use more intelligent post-training compression has recently proven fruitful (Strümpler et al., 2021) and should further improve results. is however important to state that such methods require significantly higher encoding limes, thus being in conflict with one of our key motivations, We thus avoid direct comparison as this would likely fail communicate the inherent trade-off of quality versus compression speed. As with deep entropy coding, such techniques could be straight-forwarde used in conjunction with MSCN, demonstrating the flexibility of our method.
5.4 Discussion & Future work
In this work we have introduced a principled framework for sparse Meta-Learning, demonstrating two instantiations particularly suitable for Implicit Neural Representations. Our extensive evaluation show competitive results, outperforming various state-of-the-ar techniques. is worth mentioning that the ideas introduced this work can be foraigl combined with some of the considered baselines. In the Functa case for instance, would be reasonable to expect a latent variable approach to be more competitive if only sparse subset of modulations needs to be reconstructed.
IPP
Bita per Number param.
Data dim.
11

---

Published in Transactions on Machine Learning Research (08/2022)
A key motivation was the goal of avoiding costly architecture search procedures used in related work (e.g. Dupont et al., 2021; Strümpler et al., 2021). We have shown that in both cases of structured and unstructured sparsity, we observe sparsity patterns that differ from previously hand-designed distributions and also adopt to the specific initialisation of the weights. In the structured sparsity case, we observe that this can be combined with inductive biases which have previously been found to useful.
Our method continues the trend of rapid advances made with INRs for compression. As the field continues to challenge the state-of-art in compression, we observe that sparsity is likely to be key element in this endeavour. We further hypothesise that additional improvements Learning techniques which avoid the high memory requirements of MAML. In addition, we expect methods with smarter quantisation and based on deep entropy coding to siguificantly improve results over our simple baseline. Current progress nevertheless inspires optimism for the prospect of a single learning algorithm that can be applied as a compression method to vast set up modalities.
likely to come from alternative Meta-
In the wider context of Meta-Learning, we anticipate this framework to be particularly suitable in applications where fast inference is required. Sharing the gates introduced in Section 3 among sets of parameters is an easy way of introduced group sparsity and thus reducing floating point operations required in the forward pass. A popular example would be on-device recommender systems (Galashov et al., 2019)). Furthermore, the introduced framework could be used in Continual Meta Learning as previously done in (e.g. Mallya & Lazebnik, 2018; Von Oswald et al., 2021; Schwarz et al., 2021).
References
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurol information processing systems, 33:1877-1901, 2020.
Fric R Chan, Connor Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, al. Efficient geometry-awant 3d generative adversarial networks. arXiv preprint MXXx:2112200995. 2021.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arxiv:151 03012, 2015.
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representation ions for videos. Advances in Neural Information Processing Systems, 34, 2021.
Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. On the effectiveness of weight-encoded neural implicit 3d shapos. arXio preprint arXiv:2009. 09808, 2020.
Emilien Dupont, Adam Goliúski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Compression with implicit neural representations. arXiv preprint (:Xii:2103.0312) 2021.
Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point function and you should treat it like one. arXiv preprint arXiv:2201112201, 2022a.
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Golińskii Yee Whye Teh, and Arnaud Doucet. Coin++:: Data agnostic neural compression. arXis preprint arXiv:2011122000 12904, 2022b.
12

---

Published in Transactions on Machine Learning Research (08/2022)
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Internalional Conference on Machine Learning, pp. 2943 29222295. PMT.R, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. agnotiigi meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp- 1126-1135. PMT.R, 2017.
Sebastian Flennerhag, Andrei Rusu, Razvan Pascamu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. arXiv preprint arXiv:1909.0000 2019.
Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, and Satinder Singh. Bootstrapped meta-learning. arXiv preprint arXiv:2109.00 04044, 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. Xiviv preprint Xiv: 100.00955,,,, 2018.
Alexandre Galashov, Jonathan Schwarz, Hyunjik Kim, Marta Garnelo, David Saxton, Pushmeet Kohli, SM Eslami, and Yee Whye Tch. Meta-learning surrogate models for sequential decision making. arXiv preprinl Xiv: 1903. 1 1907, 2019.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. orXiv preprint 1rXi::1902.000 2019.
H Hersbach, B Bell, P Berrisford, Biavati, A Horányi, J Muñoz Sabater, J Nicolas, C Peubey, R Radu, Rozum, et al. Eraō monthly averaged data on single levels from 1979 present. Copernicus Climate Change Service (C3S) Climate Data Store (CDS), 10:252 266, 2019.
Howard. Imagenette. https://githi: httpg://ggttabb..mm///ttat//// http///ttttta////aaanettttaa 2022. Version
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with itcrative attention. In International Conference on Machine Learning, pp. 4651- 4664. PMLR, 2021.
Siddhant Jayakumar, Razvan Pascanu, Jack Rac, Simon Osindcro, and Erich Elsen. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 03:20744-20754, 2020.
Lukasz Kaiser, Aidan N Gomez, Noam Shazcer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model learn them all. arXiv preprint crXie:170.005133 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXie preprint arXiv:1710.101900 2017.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiw:1312.6114, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural informalion processing systems, 2, 1989.
Jaeho Lee, Jihoon Tack, Namhoon Lee, and Jinwoo Shin. Meta-learning sparse implicit neural representations. Advances in Neural Information Processing Systems, 34, 2021.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang L.i. Meta-sgd: Tearning to learn quickly for few-shot learning. arXiv preprint ar Xiv: 1xx::1707.0000 700.009855 2017.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Inn209metapnuniig. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3296-3305, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEFE international conference on computer vision, pp. 3730 -738,, 2015.
13

---

Published in Transactions on Machine Learning Research (08/2022)
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_ regularization. arXiv preprint arXiv: 1122.0111,, 2017.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A contimuous relaxation of 2016.
discrete random variables. arXiv preprint
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruming. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7763-7773, 2018.
Ishit Mehta, Michaël Gharbi, Counelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chan- draker. Modulated periodic activations for generalizable local functional representations. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 14214-1422, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405 421. Springer, 2020.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madcleine Gibescu, and Antonio Liolta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science, Nature communications, 9(1):1 2018.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Jeml, 2010.
Alex Nichol and John Schulman. Reptile: scalable metalearning algorithm. arXie preprint Xiv: Xrx 1808.88999. 2(3):4, 2018.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoming with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. ar preprint or 2i 1221122.11 11466, 2021.
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter Latham, and Yee Teh. Powerpropagation: A sparsity inducing weight reparameterisation. Advances in Neural Information Processing Systems, 34, 2021.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-strrcctuc-awware neural scene reprosentations. Advonces in Neural Information Processing Systems, 32, 2019.
Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. Advances in Neural Information Processing Systems, 13:10136-101-1 2020a.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33: 7462 7473, 2020b.
Nikko Ström. Sparse connection and pruning in large dynamic artificial neural networks. In Fifth European Conference on Speech Commumication and Technology. Citeseer, 1997.
Yannick Strümpler, Janis Postels, Ren Yang, Lue Van Gool, and Federico Tombari. Implicit neural represen- tations for image compression. arXiv preprint arXie:21 2t::2112.00p 22.42677 2021.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keie Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks Iearn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
14

---

Published in Transactions on Machine Learning Research (08/2022)
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2846 2855, 2021.
Georg Thimm and Fmile Fiesler. Evaluating pruning methods. In Proceedings of the International Symposium Artificial nearal networks, pp. 20-25, 1995.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science Business Media, 2012.
Hongduan Tian, Bo Liu, Tino-Tong Yuan, and Qingshan Liu. Meta-learning with network pruning. In European Conference on Computer Vision, pp. 675 700. Springer, 2020.
Johannes Von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, and João Sacramento. Learning where learn: Gradient sparsity in meta and continual learning. Advances in Neural Information Processing Systems, 34:5250-548 5263, 2021.
Gregory Wallace. The jpeg still picture compression standard. IEEE trunsactions on consumer electronics, 38(1):xvii xxxiv, 1992.
Yunfan Zhang, Ties van Rozendaal, Johann Brehmer, Markus Nagel, and Taco Cohen. Implicit neural video compression. arXie preprint arXiv:2112.11312 2021.
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via mcta-learning In International Conference on Machine Learning, pp. 7693-7702. PMLR, 2019.
15


Reviews:
[{"review_id": "3iINzHPMwK", "reviewer": "Reviewer_B3Hu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The method has broad applicability, not only to INR, but also other meta-learning tasks where a sparse model (or model update) is required -- although further experiments may be needed to verify this.\n\n2. The method is conceptually straightforward, and allows end-to-end training and straightforward implementation with standard tools from deep learning.\n\n\nWeaknesses:\n\n1. Lack of comparison with baselines that directly targets the bit-rate cost of compression with INR. \nAs the author write early on in the manuscript, \"we specifically focus on improving the suitability of INRs as a compression method\", and therefore argue (correctly) that we should focus on the compression cost of the parameter difference $\\delta \\theta$ that is transmitted during data compression, instead of the updated model $\\theta_0 + \\delta \\theta$.  Now, the sparsity of $\\delta \\theta$ is only a surrogate for its compressibility, and at the end of the day compression algorithms are evaluated on the file size (bitrate). The lack of such comparison makes it hard to evaluate the success/failure of the sparsity-based method towards the stated goal of improving data compression with INR.\n\n2. It's unclear what is being compared across some of the methods --- the sparsity of $\\delta \\theta$, or $\\theta$ ($\\theta_0 + \\delta \\theta$) itself. Clearly, the proposed method targets the sparsity of $\\delta \\theta$ -- \"Note that as $\\theta_0$ is dense, the resulting\n$\\theta_0 + \\delta \\theta$ is still dense\". By contrast, the Meta-SparseINR (Lee et al., 2021) baseline aims to sparsity $\\theta_0$, and subsequently $\\theta_0 + \\delta \\theta$. It's not surprising that, with a denser $\\theta_0 + \\delta \\theta$  (although still sparse $\\delta \\theta$), the proposed MSCN method outperforms Meta-SparseINR in reconstruction quality. It would be rather meaningless if the comparison is between the sparsity level of $\\delta \\theta$ for the proposed MSCN method, v.s. the sparsity level of $\\theta_0 + \\delta \\theta$ for the Meta-SparseINR baseline. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript proposes a new method for sparsifying parameter updates in meta-learning, specifically within the context of data compression with INRs (implicit neural representations). The method is based on imposing an additional L_0 penalty (Louizos et al., 2017) on the model update ($\\delta \\theta$) in the MAML framework. Following Louizos et al., 2017, the Gumbel-Softmax trick is used to differentiate through samples of the binary distributions masking the weights, so the modified MAML inner loop is still differentiable (for the purpose of computing the gradient of the overall meta-learning objective), allowing for end-to-end (meta) training. \nThe proposed method is evaluated on several datasets, demonstrating superior sparsity v.s. reconstruction quality tradeoff, especially compared to a closely related method (Meta-SparseINR; Lee et al., 2021) which performs iterative pruning in the MAML outer loop. However, I have concerns about how meaningful the experimental comparisons are.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. As per weakness #1, adding results on a standard image compression benchmark in terms of bitrate v.s. PSNR,  against INR compression baselines such as (Strümpler et al., 2021; Dupont et al., 2022b), would make a much more convincing case for the viability of the proposed method for compression. \n\n2. As per weakness #2, more details for the evaluation setup regarding where the sparsity numbers come from ($\\delta \\theta$ v.s. something else) for each method would add significant insight into the comparisons and behavior of the different methods.\n\n3. Minor typo on page 4,  right above Eq(8): \"the other loop\" -> \"the outer loop\".\n\n", "broader_impact_concerns": "There are no obvious ethical implications of this work that need to be addressed in the manuscript. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R14ep3512h", "reviewer": "Reviewer_R1Eu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I don’t know much about meta-learning, so I’ll comment on the aspects regarding compression.\n\nOverall the paper looks technically correct and I don’t see any major technical issues with the method. However, the experiments and discussions for lossy compression are a bit lacking. The exact setup of the experiments that resulted in Figure 7 need to be clarified. \n\n**Note that it is quite possible that I did not completely understand how MAML works or is used.**\n\nIn detail:\n\n1. How exactly would a compression scheme with this method work? You are given a sequence of images $X_1, \\dots, X_n$ to compress. MAML is used to learn $(\\theta_0 + \\delta\\theta_i)\\circ\\mathbf{z}^\\prime$ for $i=1,\\dots,n$ images. Then, what exactly is used to calculate the rate in Figure 7? Is it just the parameters $\\\\{\\delta\\theta_i\\circ\\mathbf{z}^\\prime\\\\}_{i=1}^n$ or does $\\theta_0\\circ\\mathbf{z}^\\prime$ also enter the calculation? I'm assuming both are accounted for in the rate, as that would be the correct calculation, but the authors should clarify this.\n\n2. Is the method highly sensitive to the number of images being compressed (i.e. $n$ above)? As you compress more images, $\\theta_0$ needs to be a good initialization point for more and more samples. If so, it would only make sense to talk about the rate-distortion curve (Figure 7) as a function of $n$, especially because JPEG/JPEG2000 won't have a reduction in performance as $n$ increases (assuming it was applied individually to each image).\n\n4. The idea of MAML was to reduce the overall compute necessary to compress a set of images with respect to previous methods using INRs. Maybe the author could comment and give a sense of how much less time/steps/resources were required in the experiments.\n\n5. The wording of the introduction, abstract and final paragraph of section 5.2 suggest that the main objective is compression. The paper mostly focuses on evaluating the resulting sparseness of the method. This makes sense because the weights are not entropy coded and are instead stored to disk using 16 or 32bit precision. However, this is mentioned only at the very end of page 9 which kept me wondering while reading the paper why the authors focused on the sparsity and not the actual rate (e.g. BPP). Maybe this could be mentioned at the start of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main topic of this paper is lossy compression (LC) using implicit neural representations (INRs). In previous works on LC with INRs, encoding a message (e.g. image) is done by overfitting a neural network to learn a function from the spatial coordinates of images to the RGB values. The weights of the network are then stored and represent the code for that specific message. To decode, the network weights are restored and the function is evaluated to recover the message.\n\nThe author's contributions are two-fold: \n1. (Reducing the required rate) The size of the network defines the final code-length. Therefore the authors incorporate state-of-the-art (according to them) learned-sparsity techniques to reduce the number of weights which should bring down code-length. This technique is backpropagation-compatible, which is crucial for point (2.) below.\n\n2. (Reducing the required compute) Instead of overfitting a network to each message, a meta-learning approach (MAML) is used to learn a common initialization across a set of messages. \n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. (critical) I believe the authors should clarify how the experiments were conducted that resulted in figure 7. See previous section of this review.\n\nAll below are optional, but I'd highly recommend the following to strengthen the paper:\n\n2. Usually other distortion measures, such as MS-SSIM, are placed alongside PSNR as they capture perceptual quality better (empirically).\n\n3. This paper might be of interest and can serve as a motivation for the introduction: https://proceedings.mlr.press/v151/isik22a.html. They show that any good model compressor, under certain conditions, must perform some sort of pruning/sparsity.\n\n4. Maybe the author could comment and give a sense of how much less time/steps/resources were required in the experiments, as mentioned before.\n\n5. Writing suggestions. Nothing major, but might want to consider.\n- I found the discussion of part (ii) in section 3.2 a bit confusing. I assume it refers to using $\\theta + \\delta\\theta\\circ\\mathbf{z}’$ as discussed in the third paragraph on page 5. If so, I’d recommend merging that later paragraph into part (ii) to make it clearer.\n- I believe equation (6) has a typo. The first 2 summations have the same indexing variable $i$.\n- What does “highest plasticity” mean on page 5? Maybe make that clearer (unless it is a widely used term that I’m not aware of).\n- Typo on page 6: “for ease of ~notion~ notation”\n- Typo on page 9: “modulations in ~layer~ later layers”\n- Vector notation is inconsistent. Some vectors are bold such as $\\mathbf{x}$ and $\\mathbf{y}$, but $\\theta, \\phi, s$ are not.\n", "broader_impact_concerns": "No broader impact statement was available. The authors may want to consider discussing known algorithmic biases for model pruning, and how that may disproportionately affect underrepresented features in the data when working with images of humans. See https://arxiv.org/abs/2010.03058", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lch30VvxRK", "reviewer": "Reviewer_tps1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea of this paper makes sense to me, and the project is well executed. The paper is clearly written, and explains the important ideas well. Experimental validation is fairly extensive.\n\nThe main weakness is that due to the way results are presented, it is not clear how this method compares to state of the art (neural) compression methods. The main issues are that the datasets are non-standard in the compression literature, and the results are not presented as R/D curves (bits per pixel vs PSNR) but rather as \"Fraction of surviving parameters\" vs PSNR. I'm sure the later can be turned into BPP, but it would be nice to not leave this job to the reader. Also, the result would be most convincing if the BPP is actually computed from the filesize on disk rather than a theoretical calculation (many practical issues can come up when trying to actually encode the data). Another good metric to report is BD-rate.\n\nRegarding the dataset, it would be nice if the authors could run their method on one of the standard benchmarks, such as KODAK which is quite small. \n\nEven if the method is not competitive with the state of the art (non-INR based methods), I would likely recommend acceptance because the method is interesting and promising (perhaps with further innovations), and method is very flexible, likely yielding at least decent results on a wide range of data modalities, with fairly minimal effort.\n\nRegarding novelty, I am not completely up to date with the literature, but it seems like Strumpler et al. also already considered sparsification, which is not mentioned although Strumpler is cited. They use an L1 penalty instead of L0. Comparing to this method would be nice. Other papers based on optimizing parameters for compression include Yang et al. and van Rozendaal et al.\n\n\nStrumpler, Postels, Yang, Van Gool, Tombari, Implicit Neural Representations for Image Compression\nYang, Bamler, Mandt, Improving inference for neural image compression\nvan Rozendaal, Huijben, Cohen, Overfitting for Fun and Profit: Instance-Adaptive Data Compression", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a method for data compression based on the idea of encoding the data in the weights of a neural network which maps coordinates to signal values. This has been explored before, but the present work uses a better method for sparsification of the weights based on a differentiable L0 penalty, and shows how this can be combined with a meta learning approach that significantly improves encoding time as well as compression rates. The method is evaluated on a range of datasets with quite different kinds of data such as images, data on manifolds and shapes.\n\n\n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Report standard metrics on standard datasets to enable direct comparison to existing methods based on INRs and other methods, as discussed above.\n- Discuss difference to Strumpler and other related work if deemed relevant\n- Eq. 3, missed a min and a subscript 0, I think\n", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that the paper was innovative and technically correct. The authors have successfully addressed the reviewers' concerns on clarity, novelty, metrics, and baseline comparisons during the review period. These extensive improvements helped make the approach more comparable to earlier work in the field. No further revisions are necessary. 
Decision: Accept (poster)