COIN++: Neural Compression Across Modalities
Authors: Emilien Dupont
Hrushikesh Loya
Milad Alizadeh
Adam Golinski
Yee Whye Teh
Arnaud Doucet
Venue: Accepted by TMLR ()

Abstract: Neural compression algorithms are typically based on autoencoders that require specialized encoder and decoder architectures for different data modalities. In this paper, we propose COIN++, a neural compression framework that seamlessly handles a wide range of data modalities. Our approach is based on converting data to implicit neural representations, i.e. neural functions that map coordinates (such as pixel locations) to features (such as RGB values). Then, instead of storing the weights of the implicit neural representation directly, we store modulations applied to a meta-learned base network as a compressed code for the data. We further quantize and entropy code these modulations, leading to large compression gains while reducing encoding time by two orders of magnitude compared to baselines. We empirically demonstrate the feasibility of our method by compressing various data modalities, from images and audio to medical and climate data.

Published in Transactions on Machine Learning Research (11/2022)
COIN++: Neural Compression Across Modalities
Emilien Dupont
dupont@stals... uk loya@state, oz. ac. uk milad. olizade des. oz ox.ac. uk adamg &ro bots. or..o.z..o ac uk y. w. Ssaass oz. ac. douce t tstats. oz. ac. uk
Hrushikesh Loya*
Milad Alizadeh
Adam Goliński
Yee Whye Teh
Arnaud Doucet
University of Orford
Reviewed on OpenReview: https: // openreview. net/ forum? <d- NXBO+EM2TQ
Abstract
Neural compression algorithms are typically based autoencoders that require specialized encoder and decoder architectures for different data modalitios. In this paper, we propose COIN++ a neural compression framework that scamlessly handles a wide range of data modalities. Our approach is based on converting data to implicit neural representations i.e. neural functions that map coordinates (such as pixel locations) to features (such as RGB values). Then, instead of storing the weights of the implicit neural representation directly, we store modulations applied to meta-learned base nelwork compressed code for the data. We further quantize and entropy code these modulations, leading to large compression gains while reducing encoding time by two orders of magnitude compared to baselines. We empirically demonstrate the feasibility of our method by compressing various data modalities, from images and audio to medical and climate data.
1 Introduction
It is estimated that several exabytes of data are created everyday Domo 2018). This data is comprised of a wide variety of data modalities, each of which could benefit from compression. However, the vast majority of work in neural compression has focused only on image and video data (Ma et al. 2019). In this paper, we introduce a new approach for neural compression, called COIN+ which is applicable to a wide range of data modalities, from images and audio to medical and climate data (see Figure
Most neural compression algorithms are based autoencoders Theis al. 2017 Ballé al. 2018 Minnen et al. 2018 Lee et al. 2019). An encoder maps image to latent representation which is quantized and entropy coded into bitstream. The bitstream is then transmitted to docoder that reconstructs the image. The parameters of the encoder and decoder are trained to jointly minimize reconstruction error, or distortion, and the length of the compressed code, or rate. To achieve good performance, these algorithms heavily rely on encoder and decoder architectures that are specialized images (Cheng al., 22020b Xie et al. 2021 Zou al. 2022; Wang al., 2022). Applying these models to new data modalities then requires designing encoders and decoders which usually challenging.
Figure 1: COIN++ converts a wide range of data modalities to neural networks via optimization and then stores the parameters of these neural networks compressed codes for the data. Different. data modalities can be compressed by simply changing the input and output dimensions of the neural networks.

---

Published in Transactions on Machine Learning Research (11/2022)
Recently, new framework for neural compression, called COIN (COmpression with Implicit Neural representations); was proposed which bypasses the need for specialized encoders and decoders (Dupont et al. 2021a). Instead of compressing images directly, COIN fits a neural network mapping pixel locations to RGB values to an image and stores the quantized weights of this network as a compressod code for the image. While Dupont et al. (0211a) only apply COTN to images, it holds promise for storing other data modalities. Indeod, neural networks mapping coordinates (such as pixel locations) to features (such as RGB values), typically called implicit newral represenlations (LNR), have been used to represent signed distance functions Park et al.) 2019), voxel grids (Mescheder et al. 2019), 3D scenes (Sitzmann et al. 2019 Mildenhall et al. 2020), temperature fields (Dupont et al. 2021b), videos [Li et al. 2021b), audio (Sitamann et al. 2020b) and many more. COIN-like approaches that convert data to INRs and compress these are therefore promising for building flexible neural codecs applicable to a range of modalities.
In this paper, we identify and address several key problems with COIN and propose a compression algorithm applicable to multiple modalitics, which we call COIN++. More specifically, we identify the following issues with COIN: 1. Encoding is slow: compressing a single image can take up to an hour, 2. Lack of shared structure: each image is compressed independently, there is no shared information between networks, 3. Performance is well below state of the art (SOTA) image codecs. We address these issucs by: 1. Using meta-learning to reduce encoding time by more than two orders of magnitude to less than a second, compared to minutes or hours for COIN, 2. Learning a base network that encodes shared structure and applying modulations to this network to encode instance specific information, 3. Quantizing and entropy coding the modulations. While our method significantly exceeds COIN both terms of compression and speed, it only partially closes the gap to SOTA codecs well-studied modalities such images. However, COIN++ is applicable to wide range of data modalitics where traditional methods are difficult to use, making it promising tool for neural compression in non-standard domains.
2 Method
In this paper, we consider compressing data that can be expressed in terms of sets of coordinates x E X and features y E y. An image for example
be described by set of pixel locations x = y) in RR and their MRI scan can be described by set of
(r.g,b) b) in 1, 255)3 Similarly, positions in 3D space x = (z, y, 2) and an intensity value y E Rt. Given single datapoint as collection of coordinate and feature pairs d = (x), (for example an image as a collection of n pixel locations and RGB values), the COIN approach consists in fitting a neural network fo : X + y with parameters 0 to the datapoint by minimizing
corresponding RGB values
L( = Σ [)((x)) - y2ll2-
(1)
The weights 0 are then quantized and stored as a compressed representation of the datapoint d. The neural network fo is parameterized by SIREN (Sitzmann et al. 2020b), i.c. MLP with sine activation functions, which is necessary to fit high frequency data such as natural images (Mildenhall et al. 2020 Tancik et al., 2020b Sitzmann et al. 2020bb). More specifically, a SIREN layer is defined by an elementwise sin applied to a hidden feature vector E I as
Sh) = sin(wo(Wh - b))
where We Rdxd is weight matrix, b E Rd a bias vector and wo E R+ a positive scaling factor.
While this approach is very general, there are several key issues. Firstly, as compression involves minimizing equation I. encoding is extremely slow. For example, compressing a single image from the Kodak dataset [Kodak] 1991) Lakes nearly an hour on a 1080Ti GPU (Dupont et al. 2021a)). Secondly, as each datapoint d is fitted with a separate neural network fe, there is no information shared across datapoints. This is clearly suboptimal when several datapoints are available: natural images for example share a lot of common structure that does not need to be repeatodly stored for each individual image. In the following sections, we show how our proposed approach, COIN++, addresses these problems while maintaining the generality of COIN.
2

---

Published in Transactions on Machine Learning Research (11/2022)
2.1 Storing modulations
While COIN stores each image separate neural network, we instead train base net shared across datapoints and apply modulations to this network to parameterize individual datapoints. Given base network, such as a multi-layer perceptron (MLP), we use FiLM layers (Perez et al. 2018). to modulate the hidden features E R of the network by applying elementwise scales y E R and shifts , E R as
(h) = 7 o B
(3)
Given a fixed base MIP, we can therefore parameterize families of neural networks by applying different scales and shifts at each layer. Each neural network function therefore specified by a set of scales and shifts, which are collectively referred as modulations (Perez et al. 2018). Recently, the FiLM approach has also been applied in the context of INRs. Chan et al. (2021) parameterize the generator in generative adversarial network by SIREN network and generate samples by applying modulations to this network as sin(y @ (Wh + b) + 3). Similarly, Mehla iet al. (2021) parameterize families of INRs using a scale factor via a @inninWhh Both of these approaches can be modified to use low dimensional latent vector mapped to set of modulations instead of directly applying modulations. Chan et. al. (2021) map latent vector to scales and shifts with MLP, while Mehta et al. (2021) map the latent vector through an MLP of the same shape as the base network and use the hidden act ivations of this network as modulations. However, we found that both of these approaches performed poorly terms of compressibility, requiring a large number of modulations to achieve satisfying reconstructions.
X
fe
Figure 2: COIN++ architecture. La- tent modulations & (in green) are mapped through a hypernetwork to modulations (in blue) which are added to activations of the base network fe (in white) to pa- rametorize single function that can be evaluated coordinates to obtain fea- tures y.
Instead, we propose new parameterization of mod- ulations for INRs which, on top of yielding better compression rates, is also more stable to train. More specifically, given a base SIREN network, we only apply shifts B E Rd as modulations using
(11)
(2)
(
sin(wo (Wh h + b b+b +))
(4)
at every layer of the MLP. To further reduce storage, we use a latent vector which is linearly mapped to the modulations as shown in Figure 2 In slight overload of notation, we also refer to this vector as modulations or latent modulations. Indeod, we found empirically that using only shifts gave the same per- formance as using both shifts and scales while using only scales yielded considerably worse performance. In addition, linearly mapping the latent vector to modulations worked better than using a deep MLP as in Chan et al. (2021). Given this parameteriza- tion, we then store datapoint (such as an image) as a set of (latent) modulations Φ. To decode the datapoint, simply evaluate the modulated base network () at every coordinate
fo
6(1)
((2)
((3)
Figure 3: By applying modulations 6( (622) ((3)
to base network fe. obtain different functions that can be decoded into datapoints d(1), d d(3) by evaluating the functions at various coordinates. While we show images in this figure, the same principle can be applied a range of data modalities.
y = fo(x;))
(5)
as shown in Figure 3 To fit a set of modulations o to datapoint d, we keep the parameters 0 of the
3

---

Published in Transactions on Machine Learning Research (11/2022)
base network fixed and minimize
a
((0,0,d= Σ [Ifo(x;;)) Yill2
(6)
over Φ. In contrast to COIN, where each datapoint d is stored a separate neural network fo, COIN++ only requires storing (k) modulations (or less when using latents) opposed to O(k2) weights, where k is the width of the MLP. In addition, this approach allows us to store shared information in the base network and instance specific information in the modulations. For natural images for example, the base network encodes structure that is common to natural images while the modulations store the information required to reconstruct individual images.
2.2
Meta-learning modulations
Given a base network fa. we can encode datapoint d by minimizing equation G However, we are still faced with two problems: 1. We need to Iearn the weights of the base network, 2. Encoding a datapoint via equation G is slow, requiring thousands of iterations of gradient descent. COIN++ solves both of these problems with meta-learning.
Recently, Sitzmann et al. (2020a): : Tancik et al. (0200a) have shown that applying Model-Agnostic Mcta- Learning (MAML) (Finn et al. 2017) to INRs can reduce fitting at lest time to just a few gradient steps. Instead of minimizing C(O, d) directly via gradient descent from a random initialization, we can meta-learn an initialization 0* such that minimizing C(O, d) can be done in few gradient steps. More specifically, assume we are given a dataset of N points {d()) 1 fj=1- 1- Starting from an initialization 9, a step of the MAML inner
loop datapoint d()) is given by
(0)) - V.ol
where a is the inner loop learning rate. We are then interested in learning good initialization 0* such that the loss C(0, (()) is minimized after few gradient steps across the entire set of datapoints ( To update the initalization 8, we then perform a step of the outer loop, with an outer loop learning rate 8, via
0 BV. Σ 1== ((06)) d6)).
(8)
In our case, MAML cannot be used directly since at test time we only fit the modulations & and not the shared parameters 0. We therefore need to meta-learn an initialization for 0 and & such that, given a new datapoint, the modulations o can rapidly be computed while keeping 8 constant. Indeed, we only store the modulations for each datapoint and share the parameters 0 across all datapoints. For COIN+ a single step of the inner loop is then given by
(9)
where 0 is kept fixed. Performing the inner loop on a subset of parameters has previously been explored by Zintgraf et al. (2019) and is referred to as CAVIA. As observed in CAVIA, meta-learning the initialization for & is redundant as it can be absorbed into a bias parameter of the base network weights 0. We therefore only
need to meta-learn the shared parameter initialization 0. The update rule for the outer loop is then given by
0 3V. Ey-, C(0, 660, d()).
(10)
The inner loop then updates the modulations while the outer loop updates the shared parameters 0. This algorithm allows us to meta-learn a base network such that each set of modulations can easily and rapidly be fitted (see Figure 4. In practice, we find that as few as 3 gradient steps gives compelling results, compared with thousands for COIN.
2.3 Patches, quantization and entropy coding for modulations
Patches for large scale data. While meta-learning the base network allows us rapidly encode new datapoints into modulations, the training procedure is expensive, MAMT. must. take gradients through the
4

---

Published in Transactions on Machine Learning Research (11/2022)
(
((1)
(3(3
0
0
Figure 4: (Teft) Starting from random initialization 0, we meta-learn parameters 0 of the base network (with training progress shown as a solid line) such that modulations & can easily be fit in a few gradient steps (with fitting shown in dashed lines). (Right) During training we sample patches randomly, while at test time we partition the datapoint into patches and fit modulations to each patch.
inner loop (Finn et al. 2017]. p. For large datapoints (such as high resolution images or MRI scans), this can become prohibitively expensive. While first-order approximations exist (Finn ct al. 2017 Nichol et al. 2018 Rajeswaran et al. 2019), we found that they severely hindered performance. Instead, to reduce memory usage, we split datapoints into random patches during training. For large scale images for example, we train on 32x32 patches. At train time, we then learn a base network such that modulations can easily be fit to patches. At test time, we split a new image into patches and compute modulations for each them. The image is then represented by the set of modulations for all patches (see Figure 4 We use a similar approach for other data modalities, e.g. MRI scans are split into 3D patches.
Quantization. While COIN quantizes the neural network weights from 32 bits to 16 bits to reduce storage, quantixing beyond this severely hinders performance (Dupont et al. 2021ah. In contrast, we find that modulations are surprisingly quantizable. During meta-learning, modulations are represented by 32 bit floats. To quantize these to shorter bitwidths, we simply use uniform quantization. We first clip the modulations to within 3 standard deviations of their mean. We then split this interval into 2 equally sized bins (where is the number of bits).. Remarkably, we found that reducing the number of bits from 32 to (i.c. reducing the number of symbols from more than 10 to only 32) resulted only in small decreases in reconstruction accuracy. Simply applying uniform quantization then improves compression by factor of 6 little cost in reconstruction quality.
Entropy coding. A core component of almost all codecs is entropy coding, which allows for lossless compression of the quantized code, using e.g. arithmetic coding (Rissanen & Langdon 1979). This relies on a model of the distribution of the quantized codes. As with quantization, we use a very simple approach for modcling this distribution: we count the frequency of each quantized modulation value in the training set and this distribution for arithmetic coding at test. time. In our experiments, this reduced storage 8-15% at cost in reconstruction quality. While this simple entropy coding scheme works well, expect more sophisticated methods to significantly improve performance, which is exciting direction for future work.
Finally, we note that we only transmit the modulations and assume the receiver has access to the shared base network. As such, only the modulations are quantized, entropy coded and count towards the final compressod file size. This is similar to the typical neural compression setting where the receiver is assumed to have access to the autoencoder and only the quantized and entropy coded latent vector is transmitted.
3 Related Work
Neural compression. Learned compression approaches are typically based on autoencoders that jointly
minimize rate and distortion, as initially introduced in Ballé et al. (2017): Theis et al. (2017). Ballé et al.
(2018) extend this by adding a hyperprior, while Mentzer el al. (2018) Minnen el al. (2018): Lee el al. [2019) use an autoregressive model to improve entropy coding. Cheng et al. (2020b) improve the accuracy of the entropy models by adding attention and Gaussian mixture models for the distribution of latent codes, while Xie et al. (2021) use invertible convolutional layers further enhance performance. While most of these are optimized traditional distortion metrics such as MSE or SSIM, other works have explored the use of
5

---

Published in Transactions on Machine Learning Research (11/2022)
35
30.0
27.5
25.0
(38)
22.5
PSNR (dB]
25
20.0
17.5
15.0
Ours
Chan
Scales
Snifts
12.5
Mehta et
shifts
0
50000
10CCCO
150000
200000
250000
30000C
0
25000 50000
75000
100000
125000
150000
175000 200000
nerations
Iterations
Figure 5: (Left) Test PSNR on CIFAR10 during training using fixed number of modulations (see appendix for experimental details). Our method outperforms both baselines, improving PSNR by 2dB for the same number of parameters. (Right) Comparison of using shifts, scales and scales & shifts for modulations on MNIST (note that shifts and scales & shifts overlap). As can be seen, shifts perform significantly better than scales.
generative adversarial networks for optimizing perceptual metrics
(Agustsson et al. 2019 Mentzer et al.,
2020 Neural compression has also been applied to video (Lu et al. 2019 Goliński et al. 2020 Agustsson
ct al. 2020 and audio (Kleijn et al. 2018 Valin & Skoglund 2019 Zeghidour et al. 2021).
Implicit neural representations and compression. In addition to COIN, several recent works have explored the use of INRs for compression. Davies et al. (2020) encode 3D shapes with neural networks and show that this can reduce memory usage compared with traditional decimated meshes. Chen et al. (2021) represent videos by convolutional neural networks that take as input a time index and output a frame in the video. By pruming, quantizing and entropy coding the weights of this network, the authors achieve compression performance close to slandard video codecs. Lee et al. (2021) meta-learn sparse and parameter efficient initializations for INRs and show that this can reduce the number of parameters required to store image at a given reconstruction quality, although it is not yet competitive with image codecs such JPEG. Lu et al. 20211 Isik et al. 22021) explore the use of INRs for volumetric compression. [Zhang et al. (2021) compress frames in videos using INRs (which are quantized and entropy coded) while learning a flow warping to model differences between frames. Results on video benchmarks are promising although the performance still lags behind standard video codocs. In concurrent work, Strümpler et al. (2021) propose a method for image compression with INRs which is closely related to ours. The authors also mela-learn an MLP initialization and subsequently quantize and entropy code the weights of MLPs fitted to images, leading to large performance gains over COIN. In particular, for large scale images their method significantly they do not use patches. However, their approach still requires
outperforms both COIN and COIN++ tens of thousands of iterations at test time to fully converge, unlike ours which requires 10 iterations (three orders of magnitude faster). Further, the authors do not employ modulations but directly learn the weights of the MLPs at test time. Finally, unlike our work, their approach is not applied to a wide range of modalities, including audio, medical and climate data. Indeed, to the best of our knowledge, none of these works have considered INRs for building a unified compression framework across data modalitics.
4 Experiments
We evaluate COIN++ on four data modalities: images, audio, medical data and climate data. We implement all models in PyTorrch (Paszkc et al., 2019) and train on single GPU. We use SGD for the inner loop with a learning rate of le-2 and Adam for the outer loop with a learning rate of 1e-6 or 3e-6. We normalize coordinates x to lie in 1] and features y to lie in [0, Full experimental details required to reproduce all the results can be found in the appendix. We train COIN++ using MSE between the compressod and ground truth data. As standard, we measure reconstruction performance (or distortion) using PSNR (in dB), which is defined PSNR 100 1o (MSE). We measure the size of the compressed data (or
6

---

Published in Transactions on Machine Learning Research (11/2022)
Original COIN+ Residual
BPG
Residual
PSWR [c8)
COIN++
30
COIN
BPG
25
PEG2000
BMS
CST
Bitr [bpp]
Figure 6: (Left) Rate distortion plot on CIFARIO. COIN++ outperforms COIN, JPEG and JPEG2000 while partially closing the gap to state of the art codecs. (Right) Qualitative comparison of compression artifacts for models similar reconstruction quality. COIN+ achieves 32.4dB at 3.29 bpp while BPG achieves 31.9dB 1.88 bpp.
rate) in terms of bits-per-pixe (bpp) which is given by number of bits
number of pixels and kilobits per second (kpbs) for audio. We benchmark COIN+ against large number of basclines including standard image codecs
JPEG (Wallace 1992), JPEG2000 (Skodras et al. 2001), 1 BPG (Bellard 2014) and VTM (Bross el al.
2021] autoencoder based neural compression BMS (Ballé et al. 2018). MBT (Minnen et al. [018]] and CST (Cheng et al. 2020b) standard audio codecs MP3 (MP3) 1993 and COIN (Dupont et al. 2021a). For clarity, we use consistent colors for different codecs and plot learned codecs with solid lincs and standard codecs with dashed lines. The code to reproduce all experiments the paper can be found http://ttttbb..cmm//iiiiieeeap//
4.1 Comparisons to other INR parameterizations
We first compare parameterization of INRs with the methods proposed by Chan et al. (2021) and Mehta
et al. (2021) as described in Section 2.1 As can be seen in Figure 5) our method significantly outperforms both in terms of compressibility, improving PSNR by 2dB with the same number of parameters Further, using shift modulations is more effective than using scales and shifts, and performs significantly better than scales alone. We also note that our method allows us to quickly INRs using only a few hundred parameters. This is in contrast existing works on meta-learning for INRs (Sitzmann al. 2020a Tancik al. 2020a which typically require fitting orders of magnitude more parameters at test time.
4.2 Images: CIFAR10
We train COIN++ on CIFARIO using 128, 256, 384, 512, 768 and 1024 latent modulations. As can be seen in Figure 0 COIN+ vastly outperforms COIN, JPEG and JPEG2000 while partially closing the gap to BPG, particularly at low bitrates. To the best of our knowledge, this is the first time compression with INRs has outperformed image codecs like JPEG2000. Part of the gap between COIN+ and SOTA codecs (BMS, CST) is likely due to entropy coding: we use the simple scheme described in Section 2.3 while BMS and CST use deep generative models. We hypothesize that using deep entropy coding for the modulations would siguificantly reduce this gap. Figure shows qualitative comparisons between our model and BPG to highlight the types of compression artifacts obtained with COIN+ In order to thoroughly analyse and evaluate each component of COIN++. perform number of ablation studics.
Quantization bitwidth. Quantizing the modulations to lower bitwidth yields more compressed codes al the cost of reconstruction accuracy. To understand the tradeoff between these, we show rate distortion plots when quantizing from 3 to bits in Figure 7a As can be seen, the optimal bitwidths are surprisingly low:
For image data "pixcl" corresponds to single dimension of the data. 2 Despite significant experimental effort, we were uable achieve better performance using meta-learning with Mehta al.
2021

---

Published in Transactions on Machine Learning Research (11/2022)
-2.
-5.0
30
-7.5
28
-10.0
MIC
PSNR
-12.5
COIN+
COIN
20
bits
20.00
10
12
14
16
BR rate bop)
Number of bits
(a)
(b)
0.0
38
drop (8)
-7.5
PS04 (ORO)
32
-10.0
250
384
28
-15.0
-17.5
768
1024
Base
cuanbzation
20..0
entropy coctin
rit (bpo]
(c)
(d)
Figure (a) Rate distortion plot on CIFAR10 when quantizing the modulations & to various bitwidths. As can be seen, using or bits is optimal.
(b) Drop in PSNR for COIN and COIN+ quantization. COIN+ significantly more robust to quantization. (c) Drop in PSNR when quantizing the modulations @ to various bitwidths, for various latent dimensions. Larger latent dimensions have larger drops in PSNR.
(d) Effect of of quantization (to 5 bits) and entropy coding on CIFAR10. As can be seen, both quantization and entropy coding improve rate distortion performance.
5 bits is optimal at low bitrates while 6 is optimal at higher bitrates. Qualitative artifacts obtained from quantizing the modulations are shown in Figure 14 in the appendix.
Quantization COINvs COIN+ We compare the drop in PSNR due to quantization for COIN and COIN++ in Figure 7b As can seen, modulations are remarkably quantizable: when quantizing the COTN weights directly, performance decreases significantly around 14 bits, whereas quantizing modulations yields small drops in PSNR even when using bits. However, shown in Figure 7d the drop PSNR from quantization is larger for larger models.
Entropy coding. Figure 7d shows rate distortion plots for full precision, quantized and entropy coded modulations. As can be seen, both quantization and entropy coding significantly improve performance.
Encoding/decoding speed. Table 1 shows the average encoding and decoding time for BPG, COIN and COIN+ on CTFAR10. As BPG runs on CPU while COIN and COIN+ run on GPU, these times are not directly comparable. However, we follow standard practice the literature and run all neural codecs on GPU and standard codecs on CPU (see appendix B.1 for hardware details). In terms of encoding, COIN++ compresses images 300x faster than COIN while achieving a 4x better compression rate. Note that these results are obtained from compressing cach image separately. When using batches of images, we can compress the entire CIFAR10 Lest set (10k images) in 4mins when using 10 inner loop sleps (and in just over minute when using 3 steps). In addition, as shown in Figure 15 in the appendix, COIN+ requires only 3 gradient steps to reach the same performance as COIN does 10,000 steps, while using 4x less storage. In terms of decoding, COIN+4 + is slower than COIN as uses larger shared network and entropy coding, However, decoding with COIN+ remains fast (on the order of millisecond).
8

---

Published in Transactions on Machine Learning Research (11/2022)
BPG
COIN
+
Encoding (ms)
5.19
2.97 10
94.9
Decoding (ms)
1.25
0.46
1.29
Table 1: Average encoding and decoding time on CIFAR10 for BPG, COIN and COIN+ As can be seen, COIN+ encodes images orders of magnitude faster than COIN, while only being marginally slower than BPG at decoding time.
Original
COIN+
Residual
PSNR (08)
COIN+
BPG
JPEG2000
JPEG
0.8
1.0
12
Bibrate [bppl
Figure 8: (Teft) Rate distortion plot on ERA5. COIN++ vastly outperforms all baselines. (Right) COIN+ ++ compression artifacts on ERA5. See appendix E.5 for more samples.
4.3 Climate data: ERA5 global temperature measurements
To demonstrate the flexibility of our approach, we also use COIN++ to compress data lying on a manifold. We global temperature measurements from the ERA5 dataset (Hersbach et al. 2019 with the processing and splits from Dupont et al. (2021b). The dataset contains 8510 train and 2420 test globes of size 46x90, with temperature measurements at equally spaced latitudes À and longitudes (p on the Earth from 1979 to 2020. To model this data, we follow Dupont et al. (2021b) and use spherical coordinates x = (cos Ả cos yp, cos Asin sin A) for the inputs. As a baselinc, we compare COIN+ against JPEG, JPEG2000 and BPG applied to flat map projections vastly outperforms all baselines. These strong results highlight the versatility of the COIN+ approach: unlike traditional codecs and autoencoder based methods (which would require spherical convolutions for the encoder), manifold. Indeed, COIN++ achieves 3000x compression rate while having an RMSF of O.5%C, highlighting the potential for compressing climate data. More generally, it is likely that many highly compressible data modalities are not compressed in practice, simply because an applicable codec does not exist. We hope the flexibility of COIN+ will help make neural compression more generally applicable such modalities.
the data. As can be seen in Figure S COIN+
easily apply our method to wide range of data modalities, including data lying
We also note that very few baselines exist for compressing data on manifolds. A notable exception is McEwwn et al. (2011) which builds codec analogous to JPEG2000 by using wavelet transforms on the sphere. However, their method is not likely to outperform ours it is not learned codec and cannot take advantage of the low entropy of the climate data. Further, their method is only applicable to the sphere, while our method is applicable to any manifold where coordinate system can be defined to pass as input to the INR. Finally, we note that, while autoencoder based methods may be able to outperform COIN++ on this data modality, building such an autoencoder would require a non-trivial amount of work (including the use of spherical convolutions both for the encoder /dcooer decoder and hyperprior). In contrast, with COIN++ we simply change the input coordinates.
9

---

Published in Transactions on Machine Learning Research (11/2022)
38
Original
COIN
Residual
34
(68)
COIN+
COIN
30
BPG
28-
JPEC2000
JPEG
BM5
MBT
VIM
Bitrate [hpp]
Figure 9: (Left) Rate distortion plot on Kodak. While COIN++ performs slightly better than COIN, the use of patches reduces compression performance. (Right) COIN+ compression artifacts on Kodak. Soe appendix E.5 for more samples.
4.4 Compression with patches
To evaluate the patching approach from Section 2.3 and to demonstrate that COIN++ can scale to large data (albeit at cost in performance), we test our model on images, audio and MRI data.
Large scale images: Kodak. The Kodak dataset (Kodak 1991] contains 24 large scale images of size 768x512. To train the model, we use random 32x32 patches from the Vimeo90k dataset (Xue et al. 2019), containing 154k images of size 448x256. At ovaluation time, cach Kodak image then split into 384 32x32 patches which are compressod independently. As we do not model the global structure of the image, we therefore expect significant drop in performance compared to the case when patching is required. As be seen in Figure g the performance of COIN++ indeed drops, but still outperforms COIN and JPEG at low bitrates. We expect that this can be massively improved by modeling the global structure of the image (e.g. two patches of blue sky are nearly identical, but that information redundancy is not exploited in the current setup) but leave this to future work.
Audio: LibriSpeech. To evaluate COIN+ on audio, we use the LibriSpeech dataset (Panayotov et al. 2015) containing several hours of speech data. recorded at 16kHz, As baseline, compare against the widely used MP3 codec (MP3 1993). We split each audio sample into patches of varying size and compress each these to obtain models at various bit-rates (we refer Lo appendix B.5 for full experi- mental details). As can be seen in Figure [0] even though audio is a very different modality from the rest considered in this paper, COIN+ can still be used for compression, highlighting the versatility of our approach. However, in terms of performance, COIN+ lags behind well-establised audio codecs such as MP3.
50.0
PSWR (48)
47.5
45.0
42.5
40.0
37.5
MP3
0
40 50
80
100
120
Bit rate (lbps)
Figure 10: Rate distortion plot on LibriSpeech. While COIN+ does not outperform specialized codecs like MP3, it still performs fairly well on audio, even though this is vastly different data modality.
Medical data: brain MRI scans. Finally, we train our model on brain MRI scans from the FastMRI dataset (Zbontar et al. 2018 The dataset contains 565 train volumes and 212 test volumes with sizes ranging from 16x320x 240 to 16x384x384 (see appendix A.2 for full dataset details). As baseline, we compare our model against JPEG, JPEG2000 and BPG applied independently to each slice. Due to memory constraints, we train COIN++ on 16x 16x 16 patches. We therefore store roughly 400 independent patches at test time (as opposed to 16 slices for the image codecs). Even then COIN++ performs reasonably well, particularly at low bitrates (sce Figure 11}. As a large number of patches are nearly identical, espocially close to the edges, we expect that large gains can be made from modeling the global structure of the data. Qualitatively, our model also performs well although it has patch artifacts at low bitrates (see Figure 11)
10

---

Published in Transactions on Machine Learning Research (11/2022)
Original
COIN+
Residual
FSVR (68)
84
BPG
JPEG2000
JPEG
0.0
0.2
Bibrate [bpo]
Figure 11: (Left) Rate distortion plot on FastMRI. As can be seen, even when using patches, + performs better than JPEG. However, our methods lags behind BPG and JPEG2000. (Right) + compression artifacts on FastMRI. See appendix E.5 for more samples.
5 Conclusion, limitations and future work
Conclusion. We introduce COIN++ the first (to the best of our knowledge) neural codec applied to a wide range of data modalities. Our framework significantly improves performance compared to COIN both in terms of compression and encoding time, while being competitive with well-established codecs such as JPEG. While COIN++ does not match the performance of SOTA codees, we hope our work will help expand the range of domains where neural compression is applicable.
Limitations. The main drawback of COIN++ is that, because of the second-order gradients required for MAML, training the model is memory intensive. This in turn limits scalability and requires us to use patches for large data. Devising effective first-order approximations or bypassiing meta-learning altogether would mitigate these issues. In addition, training COIN+ ++ can occasionally be unstable, although the model typically recovers from loss instabilities (see Figure in the appendix). Further, there are several common modalities our framework cannot handle, such as text or tabular data, as these are not easily expressible as continuous functions. Finally, COIN+ lags behind SOTA codecs. Indeed, while we have demonstrated the case with which COIN++ is applicable to diffcrent modalities compared autoencoders (as they do not require specialized and potentially complicated modality specilic encoders and decoders), it is not clear whether COIN+ can outperform autoencoders specialized to each modality. However, we believe there are several interesting directions for future work to improve the performance of INR basod codocs.
Future work. In ils current form, COIN employs very basic methods for both quantization and entropy coding using more sophisticated techniques for these steps could likely lead to large performance gains. Indeed, recent success in modcling distributions of functions (Schwarz et al., 2020 Anokhin et al.) 2021
Skorokhodov et al. 2021 Dupont et al. 2021b) suggests that large gains could be made from using deep generative models to learn the distribution of modulations for entropy coding. Similarly, better post-training quantization (Nagel et al. 2019 Li et al. 2021a) or quantization-awarr training (Krishnamoorthhi 2018 Esser ct al. 2020) would also improve performance. More generally, there are a plethora of methods from the model compression literature that could be applied COIN++ (Cheng al.) 2020aa Liang et al.] 2021 For large scale data, would be intoresting to model the global structure of patches instead encoding and entropy coding them independently. Further, the field of INRs is progressing rapidly and these advances
likely to
improve COIN++ too. For example, Martel al. (2021) use adaptive patches to scale INRs to gigapixel images such partition of the input is similar to the variable size blocks used in BPG [Bellard 2014). In addition, using better activation functions (Ramasinghe & Lucey 2021) to increase PSNR and equilibrium models (Haang et. al. 2021) to reduce memory usage are exciting avenues for future research.
Finally, as COIN ++ replaces the encoder traditional neural compression with a flexible optimization procedure and the decoder with a powerful functional representation, believe compression with INRs has great potential. Advances in INRs, combined with more sophisticated entropy coding and quantization may
11

---

Published in Transactions on Machine Learning Research (11/2022)
allow COIN-like algorithms to equal even surpass SOTA codecs, while potentially allowing for compression currently unexplored modalities.
Acknowledgments
We would like to thank Hyunjik Kim, Danilo Rezende, Dan Rosenbaum and Ali Eslami for helpful discussions. We thank Jean-Francois Ton for helpful discussions around meta-learning and for reviewing early version of the paper. We also thank the anonymous reviewers for their valuable feedback which helped improve the paper. Emilien gratefully acknowledges his PhD funding from Google DecpMind.
References
Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Generative adversarial networks for extreme learned image compression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 221-231, 2019.
Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-Space Flow for End-to-Enu End Optimized Video Compression. In CVPR, 2020.
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov.
Image generators with conditionally-indeeeende pixel synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp.
2021.
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-cn optimized image compression. In International Conference on Learning Representations, 2017.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In International Conference on Learning Representations, 2018.
Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. Compressai: pytorch library and evaluation platform for end-lo-end compression research. orXiv preprinl (xXx:2011.00029, 2020.
Fabrice Bellard. Bpg image format. https://bellaadd.orgg org/bp// 2014. [Online; accessed 27-Dec-2021].
Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J Sullivan, and Lens-Rainer Ohm. Overview of the versatile video coding (vvc) standard and its applications. IEEE Transactions on Circwils and Systems for Video Technology, 31(10):3736 3764, 2021.
Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 5799-5809, 2021.
Hao Chen, Bo Hc, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations ions for videos. In Advances in Neural Information Processing Systems, 2021.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXie preprint orXiv:1710 0920009,,2 2020a.
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In CVPR, 2020b.
Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. On the effectiveness of weightt-nncoded neural implicit 3d shapes. arXiv preprint orXiv:200000988 2020.
Domo. Data never sleeps 5.0. http:////www.oocccccc/////////aa OOlinee accessed 14-Sep-2021].
2018.
Emilien Dupont, Adam Goliński, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Compression with implicit neural representations. arxiv preprint arXiv:2103.03123, 2021a.
12

---

Published in Transactions on Machine Learning Research (11/2022)
Emilien Dupont, Yee Whye Tch, and Arnaud Doucet. Generative Models Distributions of Functions. arzio preprint ar arXi::2102.00 0776, 2021b.
Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswany, and Dharmendra S. Modha. Learned step size quantization. In International Conference on Learning Representations, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
Adam Goliński, Reza Pourreza, Yang Yang, Guillaume Sautiére, and Taco Cohen. Feedback recurrent autoencoder for video compression. In Proceedings of the Asian Conference on Computer Vision, 2020.
H. Hersbach, B. Bell, P. Berrisford, G. Biavali, A. Horányi, J. Muñoz Sabater, J. Nicolas, C. Peubey, R. Radu, I. Rozum, D. Schepers, A. Simmons, C. Soci, D. Dee, and J-N. Thépaut. ERA5 monthly averaged data on single levels from 1979 to present. Copernicus Climate Change Ser- vice (C3S) Climate Data Store (CDS). https://cdes lefcdscclimate clw.ate..........mmmmmmmmmmaaaee copernicus cus. wetpenicssuuuueeeeeeeeeeeeeeeeee wu/ cddapptt/datasett /dataset/ reanalysis Seanalysis-era--snng-ee lereglee-eeelsmmmnttee
(Accessed 97-09, 2019.
Zhichun Huang, Shaojie Bai, and J Zico Kolter. Implicit?: Implicit layers for implicit representations. In Advances in Neural Information Processing Systems, 2021.
Berivan Isik, Philip Chou, Sung Jin Hwang, Nick Johnston,, and George Toderici. Lvac: Learned volumetric attribute compression for point clouds using coordinate based networks. arXiv preprint Xi::2111.0089588 2021.
W Bastiaan Kleijn, Felicia SC Lim, Alejandro Luebs, Jan Skoglund, Florian Stimberg, Quan Wang, and Thomas C Walters. Wavenet based low rate speech coding. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 676 680. LEEE, 2018.
Florian Knoll, Jure Zbontar, Anuroop Sriram, Matthew J Muckley, Mary Bruno, Aaron Defazio, Marc Parente, Krzysztof J Geras, Joe Katsnelson, Hersh Chandarana, et al. fastmri: A publicly available raw k-space and dicom dataset of knee images for accelerated mr image reconstruction using machine learning. Radiology: Artificial Intelligence, =110=e1900 2020.
Kodak. Kodak Dataset.
1991.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arzic preprint (ror::1806.08342. 2018.
Jacho Lee, Jihoon Tack, Namhoon Lee, and Jinwoo Shin. Mota-learning sparse implicit neural reprosentations. Advances in Neural Information Processing Systems, 34, 2021.
Jooyoung Lee, Seunghyun Cho, and Boung-Kwon Beack. Context-adaptive entropy model for end-to-end optimized image compression. In International Conference on Learning Representations, 2019.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. BRECQ: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations, 2021a.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pullern Recognition, pp. 6498 6508, 2021b.
Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and quantization for deep neural network acccleration: A survey. arXiv preprint "Xi::2011.09671 2021.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dve: An end-to-end deep vidco compression framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Poltern Recognation, pp. 11006-11015, 2019.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Yuzhe Lu, Kairong Jiang, Joshua A Levine, and Matthew Berger. Compressive neural representations of volumetric scalar fields. In Computer Graphics Forum, volume 40, pp. 135-146. Wiley Online Library, 2021.
Siwei Ma, Xinfeng Zhang, Chuanmin Jia, Zhenghui Zhao, Shiqi Wang, and Shanshe Wang. Image and video compression with neural networks: A review. IEEE Transactions on Circuits and Systems for Video Technology, 30(6):1683-1699) 2019.
Julien NP Martel, David B Lindell, Connor Lin, Eric R Chan, Marco Monteiro, and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. arXiv preprint 1Xxt:2105.02788, 2021.
Jason D McEwen, Yves Wiaux, and DM Eyers. Data compression on the sphere. Astronomy & Astrophysics, 531:A98, 2011.
Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chan draker. Modulated periodic activations for generalizable local functional representations. arXiv preprint arXiv:2104.0300 2021.
Fabian Mentzer, Eirikur Agustsson, Michael Tschannon, Radu Timofte, and Luc Van Gool. Conditional probability models for deep image compression. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognátion, pp. 4394-4402, 2018.
Fabian Mentzer, George Toderici, Michael Tschanmen, and Eirikur Agustsson. High-fidelity generative image compression. arXie preprint arxiv:20 09966, 2020.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 4460 4470, 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, 2020.
David Minnen, Johannes Ballé, and George Toderici. Joint Autoregressive and Hierarchical Priors for Learned Image Compression. In Advances in Neural Information Processing Systems, 2018.
MP3. MP3 codec. https: wwwp:/iso..rr/sttaadrr/22222222
1993.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. Xiviv preprint orXio:1906.0479/ 2019.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 1003100.0000 180::0293,,, 2018.
Vassil Panayotor, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispecch: an asr corpus based on public domain audio books. In 2015 IEFF international conference acoustics, speech and signal processing (ICASSP), pp. 5206 5210. IEEE, 2015.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems, 2019.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Advances in Newral Information Processing Systems, 2019.
Sameera Ramasinghe and Simon Lucey. Beyond periodicity: Towards a unifying framework for activations in coordinate-mlps. arxiv preprint 0rXi::211.15111 151355 2021.
Jorma. Rissanen and Glen G Langdon. Arithmetic coding. IBM Journal of Research and Development, 23(2): 149 162, 1979.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. arXiv preprint arXiv:2007. 04444 2020.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structuro-aware neural scene representations. In Advances in Neural Information Processing Systems, 2019.
Vincent. Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Welzstein. MetaSDF: Meta- learning Signed Distance Functions. In Advances in Neural Information Processing Systems, 2020a.
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit Neural Representations with Periodic Activation Functions. In Advances in Newral Information Processing Systems, 2020b.
Athanassios Skodras, Charilaos Christopoulos, and Touradj Ebrahimi. The jpeg 2000 still image compression standard. IEEE Signal Processing Magazine, 18(5):36-56) 58, 2001.
Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17754, 2021.
Yannick Strümpler, Janis Postels, Ren Yang, Lue Van Gool, and Federico Tombari. Implicit neural represen- tations for image compression. Xr preprint orXiv:2112.04207, 2021.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. arriv preprint w:Xiv:2012.02180 2020a.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Aduances in Neural Information Processing Systems, 2020b.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferene Huszár. Lossy image compression with compressive autoencoders. arXiv preprint arXiv:170s 00395, 2017.
Jean-Mare Valin and Jan Skoglund. LPCNet: Improving neural speech synthesis through linear prediction. In IEEE International Conference Acoustics, Speech and Signal Processing, pp. 5891-5895. IEEE, 2019.
Gregory K Wallace. The jpeg still picture compression standard. IEEE Transcttins on Consumer Electronics, (xxx)xxviiixxxxx 1992.
Dezhao Wang, Wenhan Yang, Yueyu Hu, and Jiaying Liu. Neural data-dependent transform for learned image compression. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pattern Recognition, pp. 17879-17388, 2022.
Yueqi Xie, Ka Leong Cheng, and Qifeng Chen. Enhanced invertible encoding for learned image compression. In Proceedings of the 29th ACM International Conference on Multimedio, pp. 162-170, 2021.
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision, 127(8):1106-1125 2019.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Jure Zbontar, Florian Knoll, Amuroop Sriram, Tullic Murrell, Zhengnan Huang, Matthew J Muckley, Aaron Defazio, Ruben Stern, Patricia Johnson, Mary Bruno, et al. fastmri: An open dataset and benchmarks for accelerated mri. arXiv preprint orXic:1811.8899, 2018.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 2021.
Yunfan Zhang, Ties van Rozendaal, Johann Brehmer, Markus Nagel, and Taco Cohen. Implicit neural video compression. Xiv preprint orXiv:21 12.11211111 11312, 2021.
Luisa Zint gral, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning, pp. 1693-7702 PMLR, 2019.
Renjie Zou, Chunfeng Song, and Zhaoxiang Zhang. The devil is in the details: Window-based attention for image compression. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recomnition, pp. 17492 17501, 2022.
16

---

Published in Transactions on Machine Learning Research (11/2022)
A Dataset details
A.1
Vimeo90k
We use the Vimeo90k triplet dataset, Xue et al. (2019) containing 73,171 3-frame sequences from videos at resolution of x48x256, We processed the dataset following Bégaint et al. (2020). The resulting dataset contains 153,939 training images and 11,346 test images.
A.2 FastMRI
To generate the datasct, et al. 2018). This contains 1378 fully sampled brain MRI images obtained through variety of sources T1, T1 post-contrast, T2 and FLAIR images. We then filter the dataset to only use scans from the T2 source. In addition, as the vast majority of volumes have 16 slices, we also filter by volumes with 16 slices. We then randomly split the filtered scans into a 565 training volumes and 212 testing volumes. The train dataset contains the following shapes (with their counts):
the validation split from the FastMRI brain multicoil database (Zbontar
(16, 384, 384): 329
(16, 320, 320): 229
(16, 384, 312): 2
(16, 320, 260): 2
(16, 320, 240):
(16, 384, 342): 1
(16, 320, 270): 1
While the test dataset contains the following shapes (with their counts):
(16, 384, 384): 124
(16, 320, 320): 86
(16, 320, 260): 2
We also normalize the data to lie in [0, 1] (while COIN++ can handle data in any range, cannot apply the image compression baselines if the data is not in [0, 1)). As the data contains outliers, we first compute a histogram of the data distribution and choose the maximum value such that 99.99% of the data has value less than this. We then normalize by the minimum and maximum value and clip any value lying outside this range (<0.01% % of the data).
Disclaimer required when using the FastMRI dataset: "Data used in the preparation of this article were obtained from the NYU fastMRI Initiative database (fastmri. www.iiimdd..ggo. ntw.....ddddd...dduu... [Zbontor et al. 2018 Knoll et al. 2020). As such, NYU fastMRI invesligators provided dala but did not participale in analysis wriling of thés report. A listing of NYU fastMRI investigators, subject to updates, can be found at:fastmri wed.nyu.dunncmmmmmmmmm edu. The primary goal of fastMRl is to lest whether machine learning can oid in the reconstruction of medical images."
A.3 ERA5
The climate dataset was extracted from the ERA5 database (Hersbach et al. 2019) using the processing and splits from Dupout et al. (2021b) (see this reference for details). The resulting dataset contains 12,096 grids of size 96x90, with 8510 training examples, 1166 validation examples and 2420 test examples.
A.4 LibriSpeech
The LibriSpeech dataset (Panayotov et al. 2015) contains several hours of read English Speech recorded at 16kHz. For training, the train-clean-100 split containing 28,539 examples and the test-clean split.
17

---

Published in Transactions on Machine Learning Research (11/2022)
containing 2,620 examples. We train and evaluate the first seconds of every example, corresponding to 48,000 audio samples per example.
B
Experimental details
B.1 CIFAR10
For all models, we set Wo = 50 and used an inner learning rate of 1e-2, an outer learning rate of 3e-6 and batch size 64. All models were trained for 500 epochs (400k iterations). We used the following architectures:
latent dim: 128, 10 layers of width 512
latent dim: 256, 10 layers of width 512
latent dim: 384, 10 layers of width 512
latent dim: 512, 15 layers of width 512
latent dim: 768, 15 layers of width 512
latent dim: 1024, 15 layers of width 512
We used 10 inner steps test time for all models.
While only transmit modulations (which have between 128 and 1024 parameters), the receiver is assumed to have access to the based shared network, which has on the order of millions of parameters. For example, the latent dimension 384 model has 3.8 million parameters, corresponding to model size of 15.5MB. We note that autoencoder based models typically have a similar size (Dupont et al. 2021a).
INR baselines. We used a latent dimension 384 for all models. For Chan al. 2021), we used a ReLU MIP with a single hidden layer with 512 units to map the latent vector to the modulations (we found that more layers performed worse). For Mehta et al. 22021) used the exact model proposed in their paper and set the dimension of the vector to 384.
COINbaseline, We manually searched for the best architecture for each bpp level. We followed all other hyperparrmeters from COIN (Dupont et al. (021a) and trained for 10k iterations (we found this was enough to converge on CIFAR10). Surprisingly, we found that for CIFAR10 depth did not improve performance and that increasing the width of the layers was better. This may be because the layers are already very small.
bpp: 3.6, 2 layers of width 12
bpp: 4.6, 2 layers of width 14
bpp: 5.8, 2 layers of width 16
bpp: 7.1, 2 layers of width 18
bpp: 8.5, 2 layers of width 20
bpp: 10.0, layers of width 22
For the COIN quantization experiments, used uniform quantization for the weights and biases separately. We chose the number of standard deviations & at which to define the quantization range using the formula k == 3 + mumber of bits-1 Le. when using bit, we use 3 standard deviations and when using 16 bits we use 6 standard deviations. Indeed, there is tradeoff between how much data we are cutting off and how finely we can quantize the range. We found that this formula generally gave robust results across different bit values.
15
Autoencoder baselines. All autoencoder basclines were trained using the CompressAL implementations Bégaint et al. 2020). In order for these models to handle 32x32 images from the CIFAR10 dataset, we
18

---

Published in Transactions on Machine Learning Research (11/2022)
modified the architectures both for BMS and CST. Specifically, for BMS we changed the last two convolutional layers in the encoder from kernel size 5, stride 2 convolutions to kernel size 3 stride convolutions, in order to preserve the spatial size (we made similar changes for the transposed convolutions in the decoder). For CST we replaced the first three residual blocks in the image encoder with stride convolutions instead of stride 2, hence preserving the size of the image. Similarly, we replaced the upsampling operations in the decoder with stride upsampling (i.e. dimension preserving convolutions) instead of stride 2. Otherwise, we used the default parameters provided by CompressAI, i.e. for BMS, we used N-128 and M-192 and for CST N-128. We trained all models for 500 epochs with learning rate of le-4. We trained models for each of the following À values: [0.0016, 0.0032, 0.0075, 0.015, 0.03, 0.05, 0.1, 0.15, 0.3, 0.5]. As particularly CST could be unstable to train, we trained two models for each value of À and kept the best model for the rate distortion plot.
Standard image codec baselines. We use three image codec baselines: JPEG [Wallace 1992). JPEG2000 (Skodras et al. 2001) and BPG (Bellard 2014). For each of these, we perform a search over either the quality, quantization level compression ratio find the best quality image (in terms of PSNR) at given bpp level.
We use the JPEG implementation from Pillow version 8.1.0. We use the OpenJPEG version 2.4.0 implemen- tation of JPEG2000, calling the binary file with
com_ross -i <in filepath> -I <compression ratio> -o <out filepath>. We use BPG version 0.9.8, calling the binary file with bpgenc -f 444 -q <quantization level> -o <out filepath> <in filepath>.
Encoding and decoding time. We measure the encoding time of COIN and COTN+ on 1080Ti GPU and the decoding time on 2080Ti GPU. For COIN we fit separate neural network for each image in the CIFAR10 test set and report the average encoding time. For COIN+ we similarly fit modulations for each image in the test set and report the average encoding time. For BPG, we measured encoding time on an AMD Ryzen 5 3600 (12) at 3.600GHz with 32GB of RAM.
B.2 Kodak and Vimeo90k
For all models, we set wp = 50 and used an inner learning rate of 1c-2, an outer lcarning rate of 1o-6 and batch size 64. All models were trained for 600 epochs (1.4 million iterations). We used the following architectures:
latent dim: 16, 10 layers of width 512
lalent dim: 32, 10 layers of width 512
latent dim: 64, 10 layers of width 512
latent dim: 96, 10 layers of width 512
latent dim: 128, 10 layers of width 512
We used 32x32 patches from the Vimeo90k dataset to train the model and evaluated on the full Kodak images. We used 3 inner steps for the latent dim 32 and 64 models and 10 inner steps for the latent dim 16, 96 and 128 models this gave the best results. We quantized all modulations to 5 bits.
We experimented with different patch sizes (8x8, 16x16, 32x 64x64) but found that 32 was optimal in our case. Indeed while using larger patches should help compression performance, it also requires more memory and hence a smaller batch size when memory is limited. As we use GPUs with only 11GB of RAM, the batch size we had to use for 64x64 was prohibitively small, leading to unstable training and worse performance.
B.3 MastMRI
For all models, we set Wg = 50 and used an inner learning rate of 1c-2, an outer learning rate of 3c-6 and batch size 16. All models were trained for 32,000 epochs (1.1 million iterations). We used the following architectures:
19

---

Published in Transactions on Machine Learning Research (11/2022)
latent dim: 16, 10 layers of width 512
latent dim: 32, 10 layers of width 512
latent dim: 64, 10 layers of width 512
latent dim: 128, 10 layers of width 512
We trained on 16x 16 patches and evaluated on the full volumes. We used inner steps at encoding time as this gave the best results. On the rate distortion plot, the first two points are the latent dim 16 model, quantized to 5 and 6 bits, then the latent dim 32 model, quantized to 5 bits, then the latent dim 64 model quantized to 6 bits and finally the latent dim 128 model, quantized to 5 bits and 6 bits.
B.4 ERA5
For all models, we set wo 50 and used an inner learning rate of 1c-2, an outer learning rate of 3c-6 and batch size 32. All models were trained for 800 epochs (210k iterations). We used the following architectures:
latent dim: 4, 10 layers of width 384
latent dim: 10 layers of width 384
latent dim: 12, 10 layers of width 384
We used 3 inner steps at encoding time this gave the best results. On the rate distortion plot, the first two points are the latent dim 4 and 8 models quantized to 5 bits, then the latent dim 8 model quantized to 6 and bits and finally the latent dim 12 model quantized to 7 and 8 bits.
B.5 LibriSpeech
For all models, we set wo 50 and used an inner learning rate of le-2, outer learning rate of 1e-6 and batch size 64. We further scaled the coordinates to lie in [-5, 5] we found this improved performance (similar observations were made by SSizmann et al. (2020b)). All models were trained for 1000 epochs (445k iterations), except the latent dim 256 model which was trained for 2000 epochs (890k iterations). We used the following architectures:
latent dim: 128, 10 layers of width 512, patch size 1600
latent dim: 128, 10 layers of width 512, patch size 800
latent dim: 128, 10 layers of width 512, patch size 400
latent dim: 128, 10 layers of width 512, patch size 200
latent dim: 256, 10 layers of width 512, patch size 200
We used 3 inner steps at encoding time. On the rate distortion plot, each point corresponds to one of the above models quantized to 5, 6, 6, 7 and 7 bits respectively.
Audio codec baselines. We use the MP3 implementation from LAME version 3.100, calling the binary file with
lmme -b <bit rate> <in filepath> <out filepath>.
20

---

Published in Transactions on Machine Learning Research (11/2022)
C
Figure details
(COIN vs COIN+ quantization). The results in this figure are averaged across the entire CIFARIO Lest set. We used COIN and COIN++ models that achieve roughly the same PSNR (30-31dB), corresponding to the bpp 7.1 model for COIN and the latent dim 384 model for COIN+
Figure
7b
Encoding time). The BPG model uses 1.25 bpp (PSNR: 28.7dB), the COIN+ model 1.14 bpp (PSNR: 28.9dB) and the COIN model 7.1 bpp (PSNR: 30.7dB).
Table
Figure Kodak qualitative samples). The COIN++ model used for this plot has bpp of 0.537 (latent dim
128).
Figure 11 FastMRI qualitative samples). The COIN++ model used for this plot has bpp of 0.168 (latent dim 128).
Figure (ERA5 qualitative samples). The COIN+ model used for this plot has bpp of 0.012 (latent dim
8).
Figure
14 QQalitative quantization). This figure uses the COIN++ model with latent dim of 768.
D Things we tried that didn't work
As MAML is very memory intensive, we experimented with first-order approximations. We ran first- order MAML as described in Finn et al. (2017). but found that this severely hindered performance. Further, methods such REPTILE NNchol al. 2018) not applicable to our problem, as the weights updated in the inner and outer loop not the same.
Mehta et al. 2021 use a similar approach to us for fitting INRs (without meta-learning) by using overlapping patches in images. However, we found that using overlapping patches yiclded worse tradeoff between reconstruction accuracy and number of modulations and therefore used non-overlapping patches throughout.
We experimented with using a deep MLP (and the architecture from Mehta et al. 2021 for mapping the latent vector modulations but found that this decreased performance. As MLPs are strictly more expressive than linear mappings, we hypothesize that this is due to optimizatiin issues arising from the meta-learning. If the base network is learned without meta-learning, it is likely deep MLP would improve performance over a linear mapping.
21

---

Published in Transactions on Machine Learning Research (11/2022)
E
Additional results
E.1
Meta-learning curves
28
PStRR (cla]
PSTRR (dR)
26
28
256
25
16
28
384
32
24
768
96
22
1024
23
128
50000
100000 150000
200000
250000 300000
350000
Iterations
Iterations
1e6
48
34
46
PSNR [cl)]
PSNR [cR]
42
40
28
64
28
0.0
0.4
0.6
1.0
25000
50000
75000
100000 125000
150000
175000 200000
Iterations
1e6
Iterations
70
60
PSNR [dbb
55
50
45
- 1800
40
128 400
35
128-200
30
200000
400000
600000
800000
Iterations
Figure 12: Validation PSNR (3 inner steps) during meta-learning on CIFAR10 (top left., Kodak (top right),, FastMRI (middle left), ERA5 (middle right) and LibriSpeech (bottom). Note that for LibriSpeech, the legend corresponds to 'latent dimension patch size'.
22

---

Published in Transactions on Machine Learning Research (11/2022)
E.2 CIFAR10 ablations
PSNR (681
stops 10 steps staps
0
Di-rete [opp]
Figure 13: Effect of number of inner steps on CIFARIO for model that has been quantized to 5 bits, with entropy coding. While we use 3 inner steps for meta-learning, performing 10 steps at test time leads to an increase reconstruction performance of 0.5-1.5dB, while fitting for more than 10 steps generally does not improve performance. Indeod, curves for and 50 steps almost fully overlap.
E.3 Qualitative quantization results
Data
9
4
/
Recs
9
4
/
bit
4
2 bits
7
H
3 bits
0 7
4
4 bits
9
4
5 bits
9
4
/
Figure 14: Qualitative effects of quantization. The top row shows ground truth data from MNIST and CIFAR10, the second shows the reconstructions from full precision (32 bit) modulations. The subse- quent rows show reconstructions when quantizing to various bitwidths. As can be seen, with only 5 bits, reconstructions are nearly perfect. Using as few as 1 or 2 bits, the class of the object is generally recognizable.
23

---

Published in Transactions on Machine Learning Research (11/2022)
E.4
Encoding curves
35
25
PSNR (00)
20
20
10
COIN+
COIN+
COIN
1000 2000 3000
4000
5000
6000
7000
8000 9000
10000
10
Encoding time [iterations]
Encoding time fiterations]
Figure 15: Encoding curves for COIN and COIN+ on CIFAR10 (full curve on the left, zoomed in version on the right). The COIN model has a bpp of 7.1, while COIN+ has a bpp 2.2.
E.5
Additional qualitative results
Figure 16: Qualitative compression artifacts ERA5 using the latent dim 8 model with 0.012 bpp (original in first column, C in second column and residual in third column).
24

---

Published in Transactions on Machine Learning Research (11/2022)
Figure 17: Qualitative compression artifacts on Kodak (original in first column, COIN++ in second column and residual in third column). The first 4 rows correspond the model with latent dim 128 (0.537 bpp), while the bottom two correspond to the model with latent dim 64 (0.398 bpp).
25

---

Published in Transactions on Machine Learning Research (11/2022)
Figure 18: Qualitative compression artifacts on FastMRT using the latent dim 128 model with 0.168 bpp (original in first column, COIN++ in second column and residual in third column).
26


Reviews:
[{"review_id": "86Dz69OlcM", "reviewer": "Reviewer_U5rU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors clearly state the tradeoff between specific methods (such as image compression) versus multimodal compression: the proposed COIN++ method performs well but does not bridge the gap with specific methods, trading off a wide range of applicability for some loss in performance.\n- The method is clearly exposed and well-executed. We thank the authors for providing experimental details in Section 4 and code in the appendix (the reviewer did not execute the code). A lot of experiments acorss modalities support the main claims of the paper.\n- The paper is well written and easy to follow. In particular, the authors leverage previous work and clearly expose ths underlying methods such as MAML.\n\nIn sumamry, I would say that the paper is well written and that the method is well executed. The claims are matched with convincing experimental results. Hence, this work seems to be a good candidate for acceptance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose to tackle the problem of data compression with the goal of having one method that is works across moalidites. To do so, the authors build on the recent COIN method that shifts away from the traditional encoder/decoder for data compression. Instead, the idea is to train a network on a proxy task (such as predicting pixel values from pixel indexes) and to use the network's weights as compressed representation.\n\nMore specifically, the authors adress two shortcomings of COIN, namely slow compression time (requiring network training) and lack of shared structure (training network from scratch to compress two images from the same database). Namely, the authors propose to learn a common network that is particularized to each data point using modulations (data-specific learnt vectors) that are injected at specific layers (instance-adaptation like). They reduce training time by leveraging meta-learning (MAML) and quantize and entropy-code the modulations. \n\nThe authors present a wide set of experiments (image, climate and medical data) demonstrating that COIN++ performs well among multimodal methods while stil not on a par with data type-specific compression methods (such as image compression).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No critical changes are required for acceptance. A minor change to strengthen the paper would be as follows:\n- The authors rightfully state that the modulations are easier to compress than the original models. This echoes some work in Federated Learning showing that it's generally easier to compress model deltas (that are client specific, ie the equivalent of the modulations) than models themselves. Adding some references on this could help substantiate this claim (for instance [1]).\n\n\"Reconciling Security and Communication Efficiency in Federated Learning\", Prasad et al.", "broader_impact_concerns": "Broader impact statement seems not to be present in the main paper. No particular concerns for this work. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W8hLQ7Vh5v", "reviewer": "Reviewer_DSnf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nI think the core idea of using meta-learning to reduce the computational burden of using INRs for compression is sound, and demonstrating that it works reasonably well without much tuning is very promising and a good contribution. Moreover, the paper contains a lot of empirical verification of the idea and has a reasonable amount of ablation study, making it quite appealing.\n\n## Weaknesses\nUnfortunately, the presentation of the ideas is poor and confusing, with some dubious claims.\n - I don't think the authors' argument that INR-based methods are more easily applicable to different data types than other neural methods is backed-up in any way. E.g. the authors claim that: \"These strong results highlight the versatility of the COIN++ approach: unlike traditional codecs and autoencoder based methods (which would require spherical convolutions for the encoder), we can easily apply our method to a wide range of data modalities, including data lying on a manifold\". I appreciate that INRs can be applied more easily in the sense that they don't require specialized layers (e.g. spherical convolutions), but I don't see how this is an advantage. In every experiment, the authors show that without fine-tuning the INR (e.g. its architecture) for a particular data type, their method is outperformed by any of the more specialized compression methods. This surely demonstrates that at least there is good potential in using specialized layers for INRs. In case the authors would like to retain this claim in their paper, they need to show that their method outperforms at least some sort of simple neural baselines on less conventional data types (climate data, MRI). They only compare against traditional codecs that were not designed to compress these data types. Otherwise, this claim should be removed from the paper.\n - I found Figure 1 misleading. It makes it seem like the authors are proposing to simultaneously meta-learn across different data types, which I believe is not the case. If this is a misunderstanding on my part, then the authors need to clarify the writing because there is no further implication in the paper that they would be performing the meta-learning for all data types simultaneously.\n - I couldn't parse Figure 2. The component with the latent modulation (green) is not connected to the rest of the figure. Also, there are two blue components in the figure, so I'm not sure what is mapped to what. Please redesign this figure or remove it.\n - (✓) The left plot of Figure 4 is confusing. I believe that $\\theta^*$ is the only quantity that is ever used during test time, so it is not clear what the point of depicting $\\theta$ is. I believe the plane shown is the parameter space, but this is not mentioned anywhere. Furthermore, connecting the $\\phi$ points to $\\theta^*$ is confusing. In general, I don't see how this plot is supposed to aid in understanding the ideas in the paper. Could the authors clarify this for me please?\n - The label sizes in Figures 6-10 need to be increased as they are currently hard to read.\n - (✓) What are the plots in Figure 5 depicting? The performance during training or test time?\n - (✓) In Figure 7d) the comparison appears to be tenuous, as I believe the compression is done using a GPU for the neural methods, while BPG only requires a CPU. \n - (✓) The symbol $n$ appears to be overloaded: it represents both the input size (e.g. in Eq 1 & 3) and the width of the modulated MLP in the discussion following Eq 3.\n - In the paragraph following Eq 2, the authors claim: \"Secondly, as each datapoint $d$ is fitted with a separate neural network $f_\\theta$, there is no information shared across datapoints. This is clearly suboptimal: natural images for example share a lot of common structure that does not need to be repeatedly stored for each individual image\". I am not quite sure what the authors mean by suboptimal, but this claim is most likely false. For single-shot image compression, optimizing the INRs for individual data points is the optimal thing to do, at least in theory. Amortizing the representation across multiple images should only be optimal when considering compressing multiple images simultaneously.\n - (✓) The section describing how $\\theta$ is learned using meta-learning is confusing. It appears that the authors use the learning rules in Eqs 7 and 8, but I believe they only use them in Eqs 9 and 10.\n - (✓) The \"Entropy coding\" paragraph in Section 4.2 is entirely dedicated to describing a plot from the paper's appendix. This figure should be moved to the main body of the paper. Otherwise, the paragraph does not have a place.\n\n## Questions:\n - (✓) How much does patch size matter for compressing large-scale images? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider using implicit neural representations (INRs) for compressing data. INRs make use of the observation that data can be represented as a function that maps a location to some value, e.g. in the case of images mapping a pixel to its RGB value. One overfits a neural network (NN) to this function, which becomes a lossy compressed representation of the original data, called the INR. INRs are a promising approach for neural data compression, but encoding a single data point is extremely expensive, as it requires fitting a neural network.\n\nTo reduce the heavy computational burden, the authors suggest meta-learning a base INR, which is way cheaper to adapt to a given data point. The adaptation is performed by learning a shift for the pre-activations of the INR. The compressed representation is thus reduced to these learned shifts, which the authors call \"modulations\". The authors discuss how to further reduce the codelength of the representations by quantizing and entropy coding them.\n\nThe authors test their method by compressing small and large image data, climate data on a sphere, MRI scans and audio data and demonstrate large gains compared to previous INR-based methods. Their method shows promising performance compared to state-of-the-art (SOTA) methods for each data type.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "For most necessary changes, see the \"Weaknesses\" section above. The authors would need to address all concerns I outline there for me to recommend acceptance.\n\nAdditionally:\n - MAML, as an abbreviation, is undefined.\n - Several citations are badly bracketed.\n", "broader_impact_concerns": "I don't believe that a Broader Impact Statement is required.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Kbc4jObL0a", "reviewer": "Reviewer_sNx8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* Motivation is clear.\n* I found the paper to be easy to follow in terms of writing and content.\n* The paper outlines a general method for compressing many modalities of data.\n* Experiments are within scope of the claims.\n\n**Weaknesses:**\n* I found some of the captions on the figures to be lacking a conclusion or analysis.\n* References to encoding speed are somewhat vague. For example, the number of iterations is used as a proxy for speed, but the time per iteration or any constant-time preprocessing (e.g., an up-front meta-learning cost) are not directly addressed. Decoding time is not addressed.\n* Questions about the constants in space usage (e.g., the shared stem) are also not clear. It’s not clear what weights are considered part of the bit stream. For example, if the shared parts are not part of the bit stream due to having a single “fixed” shared model, then it’s not clear if this “fixed” model (along with the modulations) will degrade in performance once the overfitting capacity is stretched. In that case, either a new model is re-transmitted (incurring bitstream cost) or distortion increases.’\n* The primary metric is PSNR but there are other metrics, such as SSIM that could have also been used.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces COIN++, an improvement of the methods introduced by COIN.\nCOIN++ is a implicit neural representation method, where individual data points are encoded in neural network weights (i.e., via overfitting the data).\nBoth COIN and COIN++ use sinusoidal layers (SIRENS).\nCOIN++ improves COIN’s encoding/training time by adding meta-learning, it improves COIN’s storage redundancy by factoring the COIN network into a shared “stem” and a set of modulations, and it improves COIN’s lack of quantization/entropy coding by implementing them.\nCOIN++ is evaluated on a range of data modalities, covering natural images, audio, MRI scans, and climate data (the latter two being 3D).\nCOIN++ is much faster at encoding than COIN and results in competitive (though not optimal) compression performance across experiments, demonstrating that a single neural compression method can have general and practical utility.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* [Critical] The information density of some of the captions seems unnecessarily low. Make the captions summarize key observations, such that the figure is more self-contained. I think, for example, mentions of the appendix are not necessary in the caption, but conclusions should be in the caption. Figure 5, in particular, has the conclusion of \"improving PSNR by 2dB with the same number of parameters\" in the text while the caption is mostly restating the labels on the plots. Similarly, the rate-distortion plots are captioned as rate-distortion plots without a conclusion.\n* [Critical] From a presentation point of view, it should be clear what the space usage calculation is. Maybe I missed something, but the assumptions about the setting under study are not apparent e.g., how shared model weights are considered.\n* [Would Strengthen] Directly measuring and ablating factors of encoding/decoding speeds (i.e., wallclock time and not iteration count) and calculations involving space usage. Figure 7d is a good start but it's not a complete characterization.\n* [Would Strengthen] Adding additional metrics e.g., SSIM to evaluations. PSNR is strongly related to the (MSE) objective function used and is not a typical end-goal of compression in itself.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers and myself agree that the idea is sound: applying meta learning to solving the slow compression issue of COIN, also sharing part of the architectures across data can allow better representation learning for the base network.

However there are also some suggestions in the reviews that I suggest the authors to include them in final revision:
1. Clarify the claim better in terms of "implicit neural representation based compression methods are easier to apply to multiple types of data";
2. Include a discussion of codecs on manifolds to better situate the climate experiments in the literature;
Decision: Accept (poster)