Compression with Bayesian Implicit Neural Representations
Authors: Zongyu Guo
Gergely Flamich
Jiajun He
Zhibo Chen
José Miguel Hernández-Lobato
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Neural Compression
Implicit Neural Representation
Relative Entropy Coding
Bayesian Neural Network
TL;DR: We propose to compress data as variational Bayesian implicit neural representations with relative entropy coding that supports joint rate-distortion optimization.

Abstract: Many common types of data can be represented as functions that map coordinates to signal values, such as pixel locations to RGB values in the case of an image. Based on this view, data can be compressed by overfitting a compact neural network to its functional representation and then encoding the network weights. However, most current solutions for this are inefficient, as quantization to low-bit precision substantially degrades the reconstruction quality. To address this issue, we propose overfitting variational Bayesian neural networks to the data and compressing an approximate posterior weight sample using relative entropy coding instead of quantizing and entropy coding it. This strategy enables direct optimization of the rate-distortion performance by minimizing the $\beta$-ELBO, and target different rate-distortion trade-offs for a given network architecture by adjusting $\beta$. Moreover, we introduce an iterative algorithm for learning prior weight distributions and employ a progressive refinement process for the variational posterior that significantly enhances performance. Experiments show that our method achieves strong performance on image and audio compression while retaining simplicity.

Compression with Bayesian Implicit Neural Representations
Zongyu Guo
Gergely Flamich*
University of Science and Technology of China guozyčmail usw.usuc.oo.. edu. cn
University of Cambridge gf3320cam ac
Jiajun He University of Cambridge jh23830cam. ac. uk
Zhibo Chen
José Miguel Hernández-Lobate University of Cambridge jmh2330cam ac. uk
University of Science and Technology of China chenzhibo@ustc. edu, cn
Abstract
Many common types data can be represented as functions that map coordi- nates to signal values, such as pixel locations to RGB values in the case of an im- age. Based on this view, data can be compressed by overfitting a compact neural network to its functional representation and then encoding the network weights. However, most current solutions for this are inefficient, as quantization to low-bit precision substantially degrades the reconstruction quality. To address this issue, we propose overfitting variational Bayesian neural networks to the data and com- pressing approximate posterior weight sample using relative entropy coding instead of quantizing and entropy coding it. This strategy enables direct optimiza- tion of the rate-distortion performance by minimizing the 3-ELBO, and target different rate-distortion trade-offs for a given network architecture by adjusting 3. Moreover, we introduce iterative algorithm for learning prior weight distribu- tions and employ progressive refinement process for the variational posterior that significantly enhances performance. Experiments show that our method achieves strong performance on image and audio compression while retaining simplicity. Our code is available at https: //gi giihub........ com/ cambridge mlg/ combiner.
Introduction
With the celebrated development of deep learning, we have seen tremendous progress of neural data compression, particularly in the field of lossy image compression [1-4]. Taking inspiration from deep generative models, especially variational autoencoders (VAEs, [5]), neural image compression models have outperformed the best manually designed image compression schemes, in terms of both objective metrics, such as PSNR and MS-SSIM [6, 7] and perceptual quality [8, 9]. However, these methods' success largely thanks their elaborate architectures designed for particular data modality. Unfortunately, this makes transferring their insights across data modalities challenging.
A recent line of work [10-12] proposes to solve this issue by reformulating it model compression problem: we treat single datum as a continuous signal that maps coordinates to values, to which overfit small neural network called its implicit neural representation (INR). While INRs were originally proposed [13] to study structural relationships in the data, Dupont et al. [10] have demonstrated that can also them for compression by encoding their weights. Since the data
"Equal Contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Coordinates A
Coordinates x
Datum
D= (1...
Approximate Coding Cast
((z)
Model Prior
((x)
Ver
Pi
Pw
qw
W
Distortion Mecsure
Ni ding(a
sample pau
Relative Entropy Coding
Variational Inference
Transmit Index."
f(x(w)
f(z (ww)
Figure 1: Framework overview of COMBINER. It first encodes datum DD into Bayesian implicit neural representations, as variational posterior distribution (w- Then an approximate posterior sam- ple w* communicated from the sender the receiver using relative entropy coding.
conceptualised as an abstract signal, INRs allow us to develop universal, moodality-agnostie neural compression methods. However, despite their flexibility, current INR-based compression methods exhibit a substantial performance gap compared to modality-speciff neural compression models. This discrepancy exists because these methods cannot optimize the compression cost directly and simply quantize the parameters to a fixed precision, as opposed to VAB-based methods that rely on expressive entropy models [2, 3, 14-17] for end-to-end joint rate-distortion optimization.
In this paper, we propose simple yet general method resolve this issue by extending INRs to the variational Bayesian setting, i.e., we overfit a variational posterior distribution Iw over the weights w to the data, instead of point estimate. Then, to compress the INRs, we use a relative entropy coding (REC) algorithm [18-20] to encode a weight sample w Qw from the posterior. The average coding cost of REC algorithms approximately DKI. Pw]. where Pw is the prior over the weights. Therefore, the advantage of our method is that we can directly optimize the rate-distortion trade-off of our INR by minimising its negative 3-ELBO [21], in a similar fashion to VAE-based methods [22, 2]. We dub our method Compression with Bayesian Implicit Neural Representations (COMBINER), and present a high-level description of it in Figure 1.
We propose and extensively evaluate two methodological improvements critical to enhancing COM- BINER's performance further. First, we find that a good prior distribution over the weights is cru- cial for good performance in practice. Thus, we derive an iterative algorithm to learn the optimal weight prior when our INRs' variational posteriors are Gaussian. Second, adapting technique from Havasi et al. [23]. we randomly partition our weights into small blocks and compress our INRs pro- gressively. Concretely, we encode a weight sample from one block at a time and perform a few gradient descent steps between the encoding steps to improve the posteriors over the remaining un- compressed weights. Our ablation studies show these techniques can improve PSNR performance by more than 4dB on low-resolution image compression.
We evaluate COMBINER on the CIFAR- 10 [24] and Kodak [25] image datasets and the LibriSpeech audio dataset [26], and show that it achieves strong performance despite being simpler than its competitors. In particular, COMBINER is not limited by the expensive meta-learning loop present in current state-of- ooftte----h--arr the art INR-based works [11, 12]. Thus we can directly optimize INRs on entire high-resolution images and audio files instead of splitting the data into chunks. As such, our INRs can capture dependencies across all the data, leading to significant performance gains.
To summarize, contributions are as follows:
We propose variational Bayesian implicit neural representations for modality-agnostic data compression by encoding INR weight samples using relative entropy coding. We call our method Compression with Bayesian Implicit Neural Representations (COMBINER).
We propose an iterative algorithm learn a prior distribution on the weights, and pro- gressive strategy to refine posteriors, both of which significantly improve performance.
We conduct experiments on the CIFAR-10, Kodak and LibriSpeech datasets, and show that COMBINER achieves strong performance despite being simpler than related methods.

---

2 Background and Motivation
In this section, we briefly review the three core ingredients of our method: implicit neural repre- sentations (INRs: [10]) and variational Bayesian neural networks (BNNs; [21]), which serve as the basis for model of the data, and relative entropy coding, which we use to compress our model.
Implicit neural representations: We can conceptualise many types of data as continuous signals, such as images, audio and video. Based on neural networks' ability to approximate any continu- ous function arbitrarily well [27], Stanley [13] proposed to use neural networks to represent data. In practice, this involves treating datum 'D as point set, where each point corresponds to coordinate-signal value pair (x,y). y), and overfitting a small neural network f(x w), usually a multilayer perceptron (MLP) parameterised by weights w. which is then called the implicit neural representation (INR) of D. Recently, Dupont et al. [10] popularised INRs for lossy data compres- sion by noting that compressing the INR's weights amounts to compressing D. However, their method has a crucial shortcoming: they assume a uniform coding distribution over w, leading to a constant rate, and overfit the INR only using the distortion the loss. Thus, unfortunately, they can only control the compression cost by varying the number of weights since they show that quantizing the weights low precision significantly degrades performance. In this paper, we solve this issue using variational Bayesian neural networks, which we discuss next.
Variational Bayesian neural networks: Based on the minimum description length principle, can explicitly control the network weights' compression cost by making them stochastic. Concretely, introduce a prior p( p(w) (abbreviated as Pw) and variational posterior q(w(D) (abbreviated as 9w) over the weights, in which case their information content is given by the Kellback-Leibber (KL) divergence DKL Iw [pw], shown in [28]. Therefore, for distortion measure A and a coding budget of C bits, we can optimize the constrained objective
min Σ (a,y)ED
Ew--q (ay. f(x w)], subject to DKI.ow/lpw] & C.
(1)
In practice, we introduce a slack variable 3 and optimize the Lagrangian dual, which yields:
L CD (w: 9w:
Pw:)
Σ Ew Ay f(x w)] 8- PKL/aw|p /pll + const.,
(a.a))
with different settings of | corresponding to different coding budgets C. Thus, optimizing (D, Sw. Pw) is equivalent to directly optimizing the rate-distortion trade-off for given rate C.
Relative entropy coding with A * coding: We will use relative entropy coding directly encode single random weight sample w ~ qw instead of quantizing point estimate and entropy coding it. This idea was first proposed by Havasi et al. [23] for model compression, who introduced minimal random coding (MRC) to encode a weight sample. In our paper, we use depth-limited, dlobal-bouna A* coding instead, which we refer as A* coding hereafter for brevity's s sake [29, 20]. We present it in Appendix A for completeness. A coding is an importance sampling algorithm that draws' independent samples W1,.... wN from the prior Pw for some parameter t > O, and computes their importance weights ra = log (gw(w:) (w.)). Then, in similar fashion to the Gumbel-max trick [31], it randomly perturbs the importance weights and selects the sample with the greatest perturbed weight. Unfortunately, this procedure returns an approximate sample with distribution fw- However, Theis and Yosri [32] have shown that the total variation distance 19w fw Iy vanishes exponentially quickly as t - Øo. Thus, t can be thought of as a free parameter of the algorithm that trades off compression rate for sample quality. Furthermore, A* coding is more efficient than MRC [23] in the following sense: Let NMRE and NA be the codes returned by MRC and * coding, respectively, when given the same target and proposal distribution as input. Then, TH[ [[[NA=] & H[[MRC). hence using * coding is always strictly more efficient [32].
N = (onkaL[wlll |2
3 Compression with Bayesian Implicit Neural Representations
We now introduce method, dubbed Compression with Bayesian Implicit Neural Representations (COMBINER). It extends INRs to the variational Bayesian setting by introducing a variational pos- terior (w over the network weights and fits INRs to the data D by minimizing Eqquation (2). Since
IIn practice, we use quasi-random number generation with multi-dimensional Sobol sequences [30] to sim- ulate our random variables to ensure that they cover the sample space evenly as possible.
3

---

Algorithm Learning the model prior
Require:
Training data (D:) (Di, D2, ..., Dar). The model posteriors qw = N diag(o;)) every training datum D,. Initialize: The model priors Pw:0, = N (pp, diag(or,))
Initialize :
repeat until convergence for i <- to M do (8) (w } arg min (ww)) [(0ps (9w))
Gradient descent for optimizing posteriors
end for 0,- arg ming, [(pp. (9w))
Closed-fore solution in Equation (5)
end repeat Return Pw: = (Hp, diag pp))
encoding the model weights is equivalent to compressing the data D, Equation (2) corresponds to jointly optimizing given rate-distortion trade-off for the data. This COMBINER' main advan- lage over other INR-based compression methods, which optimize the distortion only while keeping the rate fixed and cannot jointly optimize the distoristo Moreover, another important difference is that we encode a random weight sample w ~ 9w from the weight posterior using A* coding [20] instead of quantizing the weights and entropy coding them. At high level, COMBINER applies the model compression approach proposed by Havasi et al. [23] to encode variational Bayesian INRs, albeit with significant improvements which we discuss in Sections 3.1 and 3.2.
In this paper, we only consider networks with diagonal Gaussian prior Pw = (M diag( (o,))
and posterior gw = (py, diag(os o.)) for mean and variance vectors μps f4g, on g. Here, diag(v) denotes diagonal matrix with v on the main diagonal. Following Havasi et al. [23]. we optimize the
variational parameters f&g and o using the local reparameterization trick [33] and, in Section 3.1.
we derive an iterative algorithm to learn the prior parameters μu and x..
3.1 Learning the Model Prior on the Training Set
To guarantee that COMBINER performs well in practice, it is critical that find good prior Pw over the network weights, since serves as the proposal distribution for A coding and thus directly impacts the method's coding efficiency. To this end, in Algorithm we describe iterative algorithm to learn the prior parameters Op = {pp, op) that minimize the average rate-distortion objective over some training data (Di,..., DM
M
Cg pm (1
M Σ (D.s s Pw:0,))
1
(3)
In Equation (3) we write qu N((ug diag(o!))) and Pw:0, N( (iag(orp)) explicitly denoting the prior's dependence on its parameters. Now, we propose a coordinate descent algorithm minimize the objective Equation (3), shown in Algorithm 1. To begin, we randomly initialize the model prior and the posteriors, and alternate the following two steps to optimize {gw) and 0p:
1. Optimize the variational posteriors: We fix the prior parameters Op and optimize the posteriors using the local reparameterization trick [33] with gradient descent. Note that, given Op. optimizing (0,, (gw) }) be split into M independent optimization prob- lems, which we can perform in parallel:
for each i = 1,..., M: qw arg min (D (.wo,,)
2. Updating prior: We fix the posteriors (ow and update the model prior by computing
Op arg ming E. (Op, (g(0)
(0) )). In the Gaussian case, this admits closed-form solution:
M
1 M Ei + (4)) - -1pp) =1
Mp M
H
op
(5)
M

---

We provide the full derivation of this procedure in Appendix B. Note that by the definition of co- ordinate descent, the value of C, (9W)) decreases after each iteration, which ensures that our estimate of Op converges to some optimum.
3.2 Compression with Posterior Refinement
Once the model prior is obtained using Algorithm 1, the sender uses the prior to train the variational posterior distribution for a specific test datum, as illustrated by Equation (2). To further improve the performance of INR compression, we also adopt progressive posterior refinement strategy, concept originally proposed in [23] for Bayesian model compression.
To motivate this strategy, we first consider the optimal weight posterior gw Fixing the data D. trade-off parameter 3 and weight prior Pw, qn is given by Iw = arg min, Lz(D. gg Pw), where the minimization is performed over the set of all possible target distributions q, To compress DD using our Bayesian INR, ideally we would like to encode a sample w" qw. as it achieves optimal performance average by definition. Unfortunately, finding 9w is intractable in general, hence we restrict the search over the set of all factorized Gaussian distributions in practice, which yields rather crude approximation. However, note that for compression, we only care about encoding a single, good quality sample using relative entropy coding. To achieve this, Havasi et al. [23] suggest partitioning the weight vector w into K blocks Wi:k {W1..... wx) For example, might partition the weights per MLP layer with w; representing the weights on layer i, or into a preset number of random blocks; at the extremes, we could partition w per dimension, we could just set K = for the trivial partition. Now, to obtain a good quality posterior sample given a partition WI:K, we start with our crude posterior approximation and obtain
qw = 9w: x
x qwx arg min (D q1 x... x qK, Pw):
(6)
uu...K
where each of the K minimization procedures takes place over the appropriate family of factorized Gaussian distributions. Then, we draw sample wi 9w: and refine the remaining approximation:
Sw/ww: = Iwo (w X ... X 9wx x /101 = arg min C3 (D, q2 x ... x 9K Pw wt),
(7)
qx....
where C3(- w/) indicates that w is fixed during the optimization. We now draw W2 ~ qwa w to obtain the second chunk of final sample. In total, we iterate the refinement procedure K times, progressively conditioning on more blocks, until we obtain our final sample w W1:k- Note that already after the first step, the approximation becomes conditionally factorized Gaussian, which makes it far more flexible, and thus it approximates sw much better [18].
Above, we assumed that after each re- finement step k, we draw the next weight block wA ~ Owa/ 1. However, as suggested in [23], can also extend the scheme to incorporate relative entropy coding, by encoding an approximate sample Wyg ~ (walsb 1 with A coding instead. This way, we actually feed two birds with one scone: the refinement process allows us to obtain a better overall approximate sample w by extend- ing the variational family and by correcting for the occasional bad quality chunk tūg at the same time, thus making COMBINER more robust in practice.
Combining the refinement procedure with compression:
3.3 COMBINER in Practice
Given a partition Wi:k of the weight vector w, we use A coding to encode a sample wy from each represent the KI. divergence in block k after the completion 1 refinement steps, where we have already simulated and encoded samples from the
block. Let 6N DKL. [ /=..-, /ww of the first first k k- 1 blocks. As we discussed in Section 2, need to simulate | samples from the prior Pwr to ensure that the sample wZ encoded by A* coding has low bias. Therefore, for our method to be computationally tractable, it is important to ensure that there is no block with large divergence 5g. In fact, to guarantee that COMBINER's 's runtime is consistent, we would like the divergences across all blocks to be approximately equal, i.e., 8, 6, for 0 & i,j bit- bud-et of x bits per block and below we describe the to techniques we used to ensure óN R k for all k- I,..., K. Unless stated otherwise, we A-- 16 bits and /. in our experiments.
K. To this end, set
First, we describe how we partition the weight vector based on the training data, to approximately enforce budget average. Note that we control COMBINER's rate-distortion trade-off by
5

---

varying 5 in its training loss in Equation (3). Thus, when we run Algorithm 1 to learn the prior, we also estimate the expected coding cost of the data given & as c3 M شi=1 DKLLow |pw] Then, we set the number of blocks as Ks.x = Iea/ k]] and we partition the weight vector such that the average divergence òy of each block estimated on the training data matches the coding budget, i... 8g a bits. Unfortunately, allocating individual weights to the blocks under this constraint is equivalent to the NP-hard bin packing problem [34]. However, we found that randomly permuting the weights and greedily assigning them using the next-fit bin packing algorithm [35] worked well in practice.
(5)
Relative entropy coding-aware fine-tuning: Assume we now wish to compress some data D, and already selected the desired rate-distortion trade-off B. ran the prior learning procedure, fixed bit budget K for each block and partitioned the weight vector using the procedure from the previous paragraph. Despite our effort set the blocks that the average divergence dR R in each block on the training data, if we optimized the variational posterior (w using LD(D, (ww, ), it is unlikely that the actual divergences ón would match in each block. Therefore, we adapt the optimization procedure from [23], and we use a modified objective for each of the l posterior refinement steps:
Dxaxx (D, Iw possk 1 PwP = Σ Ew-qu [D w)] + Σ
A
(8)
(a.D)E
t=
where AE:K = (A....., } are slack variables, which we dynamically adjust during optimization. Roughly speaking, each optimization step, we compute each 6, and increase its penalty term A; if it exceeds the coding budget (i.e., 6, > k) and decrease the penalty term otherwise. See Appendix D for the detailed algorithm.
The comprehensive COMBINER pipeline: We now provide a brief summary of the entire COM- BINER compression pipeline. To begin, given dataset (Di...., Dark, select an appropriate INR architecture, and run the prior learning procedure (Algorithm 1) with different settings for / to obtain priors for a range of rate-distortion trade-offs.
To compress a new data point DD we select a prior with the desired rate-distortion trade-off and pick a blockwise coding budget k. Then, we partition the weight vector w based on K, and finally, we run the relative entropy coding-aware fine-tuning procedure from above, using A coding to compress the weight blocks between the refinement steps to obtain the compressed representation of D.
4 Related Work
Despite their short history, neural image compression methods' rate- distortion performance rapidly surpassed traditional image compression standards [16, 7, 9]. The
Neural Compression:
current state-of-the-ar methods follow a variational autoencoder (VAE) framework [2]. optimizing
the rate-distortion loss jointly. More recently, VAEs were also successfully applied to compressing other data modalities, such video [36] or point clouds [37]. However, mainstream methods quantize the latent variables produced by the encoder for transmission. Since the gradient of quantization is zero almost everywhere, learning the VAE encoder with standard back-propagation is not possible [38]. popular solution [22] to use additive uniform noise during training to approximate the quantization error, but it suffers from train-test mismatch [39]. Relative entropy coding (REC) algorithms [19] eliminate this mismatch, as they can directly encode samples from the VAEs' latent posterior. Moreover, they bring unique advantages compression with additional constraints, such as lossy compression with realism constraints [40, 41] and differentially private compression [42].
Compressing with INRs: INRs are parametric functional representations of data that offer many benefits over conventional grid-based representations, such as compactness and memory-efficient [43-45]. Recently, compression with INRs has emerged as a new paradigm for neural compres- sion [10], effective in compressing images [46], climate data [11], videos [47] and 3D scenes [48]. Usually, obtaining the INRs involves overfitting a neural network to a new signal, which is compu- tationally costly [49]. Therefore, to ease the computational burden, some works [11, 46, 12] employ meta-learning loops [50] that largely reduce the fitting times during encoding. However, due to the expensive nature of the meta-learning process, these methods need to crop the data into patches to make training with second-order gradients practical. The biggest difficulty the current INR-based methods face that quantizing the INR weights and activations can significantly degrade their per- formance, due to the brittle nature of the heavily overfitted parameters. Our method solves this issue

---

COMBINER (ours)
MSCN [12]
COIN+ [11]
COIN [10]
CVPR2020 [4]
ICLR2018 [2]
BPG
JPEG2000
42
38
32
34
30
RGB PSNR
28
26
26
22
24
18
22
2
3
6
7
0
0.1
0.2
0.3
0.4
0.5
Rate (bpp)
Rate (bpp)
(a) CIFAR-10 dataset
(b) Kodak dataset
Figure 2: Rate-distortion curves two image datasets. In both figures, solid lines denote INR- based methods, dotted lines denote VAE-based methods and dashed lines denote classical methods. Examples of decoded Kodak images are provided in Appendix F.3
by fitting a variational posterior over the parameters, from which we can encode samples directly using REC, eliminating the mismatch caused by quantization. Concurrent to our work, Schwarz et al. [51] introduced method to learn a better coding distribution for the INR weights using VAE, a similar vein to our prior learning method in Algorithm 1. Their method achieves impres- sive performance
image and audio compression tasks, but is significantly more complex than method: they run an expensive meta-learning procedure to learn the backbone architecture for their INRs and train VAE to encode the INRs, making the already long training phase even longer.
Experiments
To assess COMBINER's performance across different data regimes and modalities, ducted experiments compressing images from the low-resolution CIFAR-10 dataset [24], the high- resolution Kodak dataset [25], and compressing audio from the LibriSpeech dataset [26]: the periments and their results are described in Sections 5.1 and 5.2. Furthermore, in Section 5.3, we present analysis and ablation studies on COMBINER's ability adaptively activate prune the INR parameters, the effectiveness of its posterior refinement procedure and on the time complexity of its encoding procedure.
5.1 Image Compression
Datasets: We conducted our image compression experiments the CIFAR-10 [24] and Kodak [25] datasets. For the CIFAR-10 10 dataset, which contains 32 x 32 pixel images, we randomly selected model all 10,000
2048 images from the training set for learning the model prior, and evaluated images the test set. For the high- bigh-resolution image compression experiments we use 512 randomly cropped 768 x 512 pixel patches from the CLIC training set [52] to learn the model prior and tested on the Kodak images, which have matching resolution.
Following previous methods [10-12], we utilize SIREN [43] as the network architecture. Input coordinates æ are transformed into Fourier embeddings [44] before being fed into the MLP network. depicted as ((x) in Figure 1. For the model structure, experimentally find 4-layer MLP with hidden units per layer and 32 Fourier embeddings works well on CIFAR-I0. When training CLIC and testing on Kodak, we use models of different sizes cover multiple rate points. We describe the model structure and other experimental settings in more detail in Appendix E. Remarkably, the networks utilized in experiments quite small. Our model for compressing CIFAR- images has only 1,123 parameters, and the larger model for compressing high-resolution Kodak images contains merely 21,563 parameters.
Models:
Performance: Figure 2, we benchmark COMBINER'S rate-distortion performance against clas- sical codecs including JPEG2000 and BPG, and codecs including COIN [10], COIN+

---

28.9
RGB PSNR (dB)
46
RGB PSNR (dB)
RGB PSNR (dIs)
30
28.7
COMBINER
aptimal
COMBINER
No fine tining
MP3
20
No proo leamiing
COMBINER
10
20
2
Rate (kbps)
Rate (bpp)
Number of iterations
Figure 3: COMBINER's s audio compression performance sus MP3 on the LibriSpeech dataset.
Figure 4: Ablation study on CIFAR-10 verifying the effec- tiveness of the fine-tuning and prior learning procedures.
Figure COMBINER's per- formance improvement function the number fine- tuning steps.
[1]]. and MSCN [12]. Additionally, we include results from VAE-based codecs such as ICLR2018 [2] and CVPR2020 [4] for reference. We observe that COMBINER exhibits competitive perfor- mance on the CIFAR-I0 dataset, on par with COIN+ and marginally lower than MSCN. Fur- thermore, our proposed method achieves impressive performance on the Kodak dataset, surpassing JPEG2000 and other INR-based codecs. This superior performance is in part due to our method not requiring an expensive meta-learning loop [11, (1,,46, 12], which would involve computing second- order gradients during training. Since we avoid this cost, we can compress the whole high-resolution image using single MI.P network, thus the model can capture global patterns the image.
5.2 Audio Compression
To demonstrate the effectiveness of COMBINER for compressing data in other modalities, we also conduct experiments on audio data. Since our method does not need compute the second-order gradient during training, we can directly compress long audio segment with a single INR model. We evaluate our method on LibriSpeech [26], speech dataset recorded at a 16kHz sampling rate. We train the model prior with 3-second chunks of audio, with 48000 samples per chunk. The detailed experimental setup described in Appendix E. Due to COMBINER's time-consuming encoding process, we restrict our evaluation to 24 randomly selected audio chunks from the test set. Since we lack COIN+ statistics for this subset 24 audio chunks, we only compare our method with MP3 (implemented using the ffmpeg package), which has been shown to be much better than COIN+ on the complete test set [11]. Figure shows that COMBINER outperforms MP3 at low bitrate points, which verifies its effectiveness in audio compression. We also conducted another group of experiments where the audios are cropped into shorter chunks, which we describe in Appendix F.2.
5.3 Analysis, Ablation Study and Time Complexity
Model Visualizations: To provide more insight into COMBINER's ' behavior, we visualize its pa- rameters and information content on the second hidden layer of two small 4-layer models trained on two CIFAR-10 images with 3 10-5, We use the KL in bits as an estimate of their coding cost, and not encode the weights with A coding or perform fine-tuning.
In Figure 6, we visualize the learned model prior parameters Nº and op in the left column, the variational parameters of distinct images in the second and third column and the KL divergence DKL[qw [ in bits in the rightmost column. Since this layer incorporates 16 hidden units, the weight matrix of parameters has a 17 16 shape, where weights and bias concatenated (the bias is represented by the last row). Interestingly, there are seven "active" columns within or, indicating that only seven hidden units of this layer would be activated for signal representation at this rate point. For instance, when representing image that is randomly selected from the CIFAR- 10 test set, four columns are activated for representation. This activation is evident the four blue columns within the KL map, which require few bits transmit the sample of the posterior distribution. Similarly, three hidden units are engaged in the representation of image 2. their variational Gaussian distributions have close zero variance, the posterior distributions these

---

0.055
3015
0102
5020
2,000
10
-0413
-0.25
-2413
05
3,030
-0.950
-5.830
-00
Prior mean
Posterior mean, image
Posterior mean, image 2
KL /its, image
6.12
2.012
25
0.013
5.009
2.00
20
0.033
5.066
100
0004
2002
Prior s.d.
Posterior s.d., image
Posterior s.d., image 2
KL /its, image
Figure 6: Visualizations of the weight prior, posterior and information content of a variational INR trained on two CIFAR-IO images. We focus on the INR's weights connecting the first and second hidden layers. Each heatmap is 17 x 16 because both layers have 16 hidden units and we concate- nated the weights and the biases (last row). We write s.d. for standard deviation.
activated columns basically approach Dirac delta distribution. In summary, by optimizing the rate-distortion objective, our proposed method can adaptively activate or prune network parameters.
Ablation Studies: We conducted ablation studies on the CIFAR- 10 dataset to verify the effective- ness of learning the model prior (Section 3.1) and posterior fine-tume to--itunngg (Section 3.2). In the first ablation study, instead of learning the prior parameters, we follow the methodology of Havasi ([53), p. 73) and use a layer-wise zero-mean isotropic Gaussian prior ps NO, ael), where pe is the weight prior for the 9th hidden layer. We learn the ag's jointly with the posterior parameters by optimizing Equation (3) using gradient descent, and encode them at 32-bit precision alongside the A*-coded posterior weight samples. In the second ablation study, omit the fine-tuming steps be- tween encoding blocks with A coding, i.c. we never correct for bad quality approximate samples. In both experiments, with the theoretically optimal scenario: we draw an exact sample from each blocks's 's variational posterior between refinement steps instead of encoding approximate sample with A coding, and estimate the sample's 's codelength with the block's KL. divergence.
compress each block using 16 bits. Finally, reference, also compare
We compare the results of these experiments with our proposed pipeline (Section 3.3) using the above mentioned techniques Figure 4. We find that both the prior learning and posterior refine- ment contribute significantly to COMBINER's performance. In particular, fine-tuning the posteriors is more effective at higher bitrates, while prior learning increases yields a consistent 4dB gain in PSNR across all bitrates. Finally, see that fine-tuning cannot completely compensate for the occasional bad approximate samples that A coding yields, as there a consistent 0.8 1.3dB discrepancy between COMBINER's ' and the theoretically optimal performance.
In Appendix C, we describe a further experiments we conducted to estimate how much each fine- tuning step contributes the PSNR gain between compressing two blocks. The results are shown in Figure 7, which demonstrate that quality the encoded approximate posterior sample doesn't 't just monotonically increase with each fine-tuning step, see Appendix C for explanation.
Time Complexity: COMBINER's encoding procedure is slow, as it requires several thousand gra- dient descent steps to infer the parameters of the INR's 's weight posterior, and thousands more for the progressive fine-tuning. To get a better understanding of COMBINER's practical time complexity, we evaluate its coding time on both the CIFAR- 10 and Kodak datasets at different rates and report our findings in Tables and 2. We find that can take between 13 minutes (0.91 bpp) to 34 minutes (4.45 bpp) to encode 500 CIFAR-10 images in parallel with single A GPU, including posterior inference (7 minutes) and progressive fine-tuning. Note, that fine-tuning takes longer for higher bitrates, as the weights are partitioned into more groups as each weight has higher individual in- formation content. To compress high-resol images from the Kodak dataset, the encoding time varies between 21.5 minutes (0.070 hpp) and 79 minutes (0.293 bpp).

---

Encoding (500 images, GPU A100 80G)
bit-rate
Decoding (1 image, CPU)
Learning Posterior
REC Fine-tuning
Total
0.91 bpp
- min
13 min
2.06
1.39 bpp
~9 min
~16 min
2.09 ms
2.28 hpp
~ min
min s
~1 min 30 s
2.86 ms
3,50 bpp
21 min 30 s
~2 min 30 s
3.82 ms
4.45 bpp
~27 min
-34 min
3,88 ms
Table 1: The encoding time and decoding time of COMBINER on CIFAR-10 dataset.
Encoding (1 image, GPU A100 80G)
bit-rate
Decoding (1 image, CPU)
Learning Posterior
REC + Fine-tuning
Total
0.07 bpp
~22 min 30
~21 min 30
348.42 ms
0.11 bpp
~9 min
18 mins
~27 min
381.53 ms
0.13 bpp
22 min
~31 min
405.38 ms
0.22 bpp
50 min
~61 min
597.39 ms
~-1 min
0.29 hpp
68 min
~7 min
602.32 ms
Table 2: The encoding time and decoding time of COMBINER on Kodak dataset.
To assess the effect the fine-tuning procedure's length, we randomly selected a CIFAR-10 10 image and encoded it using the whole COMBINER pipeline, but varied the number of fine-tuning steps between 2148 and 30260; we report the results of our experiment in Figure 5. We find that running the fine-tuning process beyond certain point has diminshing returns. In particular, while used around 30k iterations in our other experiments, just using 3k iterations would sacrifice a mere 0.3 dB in the reconstruction quality, while saving 90% on the original tuning time.
On the other hand, COMBINER has fast decoding speed, since once we decode the compressed weight sample, we can reconstruct the data with single forward pass through the network at each coordinate, which can be easily parallelized. Specifically, the decoding time of a single CIFAR-10 10 image is between ms and 4 ms using A100 GPU, and less than 1 second for Kodak image.
6 Conclusion and Limitations
In this paper, proposed COMBINER, new neural compression approach that first encodes data as variational Bayesian implicit neural representations and then communicates an approximate posterior weight sample using relative entropy coding. Unlike previous INR- neural codecs, COMBINER supports joint rate-distortion optimization and thus can adaptively activate and prune the network parameters. Moreover, introduced iterative algorithm for learning the prior pa- rameters on the network weights and progressively refining the variational posterior. Our abla- tion studies show that these methods significantly enhance the COMBINER'S 's rate-distortion perfor- mance. Finally, COMBINER achieves strong compression performance on low and high-resolution image and audio compression, showcasing its potential across different data regimes and modalities.
COMBINER has several limitations. First, as discussed in Section 5.3, while decoding process is fast, its encoding time is considerably longer. Optimizing the variational posterior distributions requires thousands of iterations, and progressively fine-tuning them is also time-consuming. Second, Bayesian neural networks are inherently sensitive to initialization [21]. Identifying the optimal initialization setting for achieving training stability and superior rate-distortion performance may require considerable effort. Despite these challenges, believe COMBINER paves the way for joint rate-distortion optimization of INRs for compression.
Acknowledgements
ZG acknowledges funding from the Outstanding PhD Student Program at the University of Science and Technology of China. ZC is supported in part by National Natural Science Foundation of China under Grant U1908209, 62021001. GF acknowledges funding from DeepMind.
10

---

References
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. International Conference on Learning Reprexentation, 2017.
[2] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In International Conference on Learning Representations, 2018. 2,
6, 7, 8
[3] David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hierarchical priors for
learned image compression. Advances Neural Information Processing Systems, 2018. 2
[4] Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized Gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, 2020. 1, 7,8
[5] Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
[6] Dailan Ziming Yang. Weikun Peng, Rui Ma, Hongwei Qin, Yan Wang. ELIC: efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[7] Jinming Liu, Heming Sun, and Jiro Katto. Learned image compression with mixed transformer-Co architectures. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 1,6 6
[8] Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Lue Van Gool. Generative adversarial networks for extreme learned image compression. In Proceedings of the IEEE/CVF Interna- tional Conference Computer Vision, pages 221 2019. 1
[9] Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity generative image compression. In Advances Neural Information Processing Systems, volume 33, pages 11913- 11924, 2020. 1,6
[10] Emilien Dupont, Adam Golinski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. COIN: Com- pression with implicit neural representations. In Neural Compression: From Information Theory to Application-WWorssoppe ICLR, 2021. 3.
[11] Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Golinski, Y Whye Teh, and Arnaud Doucet. COIN+ Neural compression across modalities. Transactions on Machine Learning Research, 2022(11), 2022. 2, 6, 7, 8, 15
[12] Jonathan Richard Schwarz and Yee Whye Teh. Meta-learning sparse compression networks. Transactions Machine Learning Research, 2022. ISSN 2835-8856. 1, 2, 7, 8, 15
[13] Kenneth O Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines, 8(2):131-12,, 2007. 1,3
[14] David Minnen and Saurahh Singh. Channel-wise autoregressive entropy models for learned image com- pression. In 2020 IEFE International Conference on Image Processing (ICIP). IEFF, 2020. 2
[15] Dailan He, Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin. Checkerboard context model for elficient learned image compression. In Proceedingy of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
[16] Zongyu Guo, Zhizheng Zhang, Runsen Feng, and Zhibo Chen. Causal contextual prediction learned image compression. IEEE Transactions on Circuits and Systems for Video Technology, 32(4), 2021. 6
[17] Ahmet Burakhan Koyuncu, Han Gao, Atanas Bocv, Georgii Gaikov, Elena Alshina, and Eckehard G. Steinbach. Contextformer: transformer with spatio-channel attention for context modeling in Iearned image compression. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, Octo- ber 23-27, 2022, Proceedings, Part XIX. Springer, 2022.
[18] Marton Havasi, Jasper Snoek, Dustin Tran, Jonathan Gordon, and José Miguel corraandzz--obbtee Re- fining the variational posterior through iterative optimization. In Bayesian Deep Learning Workshop @ NeurIPS, 2019. 5
11

---

[19] Gergely Flamich, Marton Havasi, and José Miguel Hernández-Lobato. Compressing images by encoding their latent representations with relative entropy coding. In Advances in Neural Information Processing Systems, volume 33, pages 16131-16141, 2020.
[20] Gergely Flamich, Stratis Markou, and José Miguel Hernández-Lobato Fast relative entropy coding with A coding. In International Conference on Machine Learning, pages 6548-6577. PMIR, 2022. 2, 3, 4,
14
[21] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In International Conference on Machine Learning, pages 1613-1622. PMLR, 2015. 2, 3, 10, 17
[22] Johannes Ballé, Valero Laparra, and Fero P. Simoncelli. End-to-end optimized image compression. In International Conference on Learning Representations, 2017. 2,
[23] Marton Havasi, Robert Peharz, and José Miguel Hernández-Lobato. Minimal random code learning: Getting bits back from compressed model parameters. In International Conference on Learning Repre- sentations, 2019. 2, 3, 4, 5, 6, 16
[24] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Canadian Institute for Advanced Research, 2009. 2,
[25] Eastman Kodak. Kodak Lossless True Color Image Suite (PhotoCD PCD0992). http://rOk.uu us/ traphice/kodak// 1993. 7,
[26] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech; An ASR corpus based public domain audio books. In ICASSP, pages 5206-5210. IEEE, 2015. 2. 17
[27] George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 304):303-314, 1989. 3
[28] Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the descrip- tion length of the weights. Proceedings of the sixth annual conference on Computational learning theory, pages 5-13, 1993. 3
[29] Chris J. Maddison, Daniel Tarlow, and Tom Minka. A sampling. In Advances in Neural Information Processing Systems, volume 27, 2014. 3
[30] II' ya Meerovich Sobol'. On the distribution of points in a cube and the approximate evaluation of integrals. Zhurnal Vychislitel niiiteeeeliiiiieeeii Matematiki i Matematicheskoi Fiziki, (()):784-02,, 1967. 3
[31] George Papandreou and Alan L Yuille. Perturb-and and- random fields: Using discrete optimization to learn and sample from energy models. In 2011 International Conference Computer Vision, pages 193-200. IEEE, 2011.
[32] Lucas Theis and Noureldin Yosri. Algorithms for the communication of samples. In International Conference on Machine Learning, pages 21308-21328. PMLR, 2022. 3, 14
[33] Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems, volume 28, 2015. 4, 16
[34] Silvano Martello and Paolo Toth. Bin-packing problem. Knapsack problems: Algorithms and computer implementations, pages 221-245. 1990. 6
[35] David S Johnson. Near-optimal bin packing algorithms. PhD thesis, Massachusetts Institute of Technol- ogy. 1973. 6
[36] Guo Lu, Wanli Ouyang. Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. DVC: an end-to-end deep video compression framework. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11006 11015, 2019.
[37] Yun He, Xinlin Ren, Danhang Tang, Yinda Zhang, Xiangyang Xue, and Yanwei Fu. Density-preserving deep point cloud compression. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2333-2342, 2022. 6
[38] Zongyu Guo, Zhizheng Zhang, Runsen Feng, and Zhibo Chen. Soft then hard: Rethinking the quanti- zation in neural image compression. In International Conference on Machine Learning. PMLR, 2021.
12

---

[39] Eirikur Agustsson and Lucas Theis. Universally quantized neural compression. In Advances in Neural Information Processing Systems, volume 33, 2020.
[40] Lucas Theis and Eirikur Agustsson. On the advantages of stochastic encoders. In Neural Compression Workshop at ICLR, 2021. 6
[41] Lucas Theis, Tim Salimans. Matthew D. Hoffmann and Fabian Mentzer. Lossy compression with Gaussian diffusion, arXiv preprint arXiv:2206.0668 08888, 2022.
[42] Abhin Shah, Wei-Ning Chen, Johannes Balle, Peter Kairouz, and Lucas Theis. Optimal compression of locally differentially private mechanisms. In International Conference Artificial Intelligence and Statisticy, 2022.
[43] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Advances Neural Information Processing Systems, volume 33, pages 7462-7473, 2020. 6,
[44] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems, volume 33, pages 7537 7547, 2020.
[45] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 60(1):99-106, 2021.
[46] Yannick Strümpler. Janis Postels, Ren Yang. Luc Van Gool, and Federico Tombari. Implicit neural repre- sentations for image compression. In ECCV (26), volume 13686 Lecture Notes Computer Science, pages 74-91. Springer, 2022. 6, 8
[47] Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, and Abhinav Shrivastava. NeRV; Neural representations for videos. In Advances in Neural Information Processing Systems, volume 34, pages 21557-21568, 2021.
[48] Thomas Bird, Johannes Ballé, Saurabh Singh, and Philip A. Chou. 3D scene compression through entropy penalized neural representation functions. In 202/ Picture Coding Symposiun (PCS), pages 1-5. IEEE, 2021. 6
[49] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-he -hassd neural representations. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2846-2855, 2021.
[50] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pages 1126-115 135. PMLR, 2017.
[51] Jonathan Richard Schwarz, Jihoon Tack, Yec Whye Tch, Jacho Lee, and Jinwoo Shin. Modality-annostic variational compression implicit neural representations. International Conference on Machine Learning, 2023. 7
[52] 5th Workshop and Challenge on Learned Image Compression. http://compeassonn cc, 2022, 7,
[53] Marton Havasi. Advances in compression using probabilistic models. PhD thesis, University of Cam- bridge, 2021. 9
13

---

A Relative Entropy Coding with A# Coding
Algorithm A encoding
Require: Proposal distribution and target distribution Qw-
Initialize N, Go; u² No 2|01, 00,
00
for i = ,
NN do
N samples from proposal distribution
w; Pw
G; TruncGumbel(Gi]]
L G; + log (4w (w; P/W (w;))
>> Perturbed importance weight
if L, & L then
L
w** N u;
end if
end for return w* N
Transmit the index N
Algorithm A decoding
Simulate (w:) Receive N*
t> Simulate N samples from Pw with the shared seed
return w* w
Receive the approximate posterior sample
Recall that we would like to communicate sample from the variational posterior distribution 9w using the proposal distribution Pw- In our experiments, we used global-bound depth-limited A coding to achieve this [20]. We describe the encoding procedure in Algorithm 2 and the decoding procedure Algorithm 3. For brevity, we refer to this particular variant of the algorithm as coding for the rest of the appendix.
A* coding is an importance sampler that draws samples Wi...., WN Pw from the pro- posal distribution Pw, where N is a parameter we pick. Then, it computes the importance weights (w,)///))))) r(w,) qw (n,) ) Pw (,)),, and sequentially perturbs them with truncated Gumbel noise:
Fn r(w,) + G G TruneCumbellGrr-)) Gu
Then, can be shown that by setting
N arg max nE[1:N]
(10)
we have that wys Tw is approximately distributed according to the target, i.c. Tw & Qw. More preciesly, have the following result:
Lemma A.1 (Bound the total variation between Tu and gw (Lemma D.I in [32])). Let set the number of proposal samples simulated by Algorithm 2 to 2Pa 14 w p Ht for some parameter
t > O. Let qw denote the approximate distribution of the algorithm's output for this choice of N. Then,
Drv Fw) 4e,
(11)
where
1/2
(2- + +2 PZ [log, r(Z) DRI[Q|IP] t/2]
(12)
This result essentially tells that should draw at least around D Inw] samples ensure low sample bias, and beyond this, the bias decreases exponentially quickly as - J0. However,
'The PDF of standard Gumbel random variable truncated to (-00,6) is given by TruneGumbel(s b) 1[x exp(-x exp(-2) + exp(-b)).
14

---

note that the number of samples we need also increases exponentially quickly with t. In practice, we observed that when Dk. (9 Pw] sufficiently large (around 16-20 bits), setting gave good results. To encode N* we built an empirical distribution over indices using our training datasets and used for entropy coding to find the optimal variable-length code for the index.
already
In short, on the encoder side, N random samples are obtained from the proposal distribution Pw- Then we select the sample w, and transmit its index N that has the greatest perturbed importance weight. On the decoder side, those N random samples can be simulated with the same seed held by the encoder. The decoder only needs find the sample with the index N Therefore, the decoding process of our method is very fast.
B Closed-Form Solution for Updating Model Prior
In this section, we derive the analytic expressions for the prior parameter updates in our iterative prior learning procedure when both the prior and the posterior are Gaussian distributions. Given (1) set of training data (D:) (Di, D2, ...; Das), fit variational distribution gw represent each of the D,s. To do this, we minimize the loss (abbreviated as L later)
1
Cs(0p.
Σ
Co(Di, :..::,
(13)
MM 1-1
1
E Ew (Ay f(x w)] + B. DKLLow/lp..e.] }.
(14)
M
(m.y)ED
Now calculate the derivative w.r.t. the prior distribution parameter Pw:0,*
OC
1
@DKL [ Pw.e,1
(15)
M
20,
Considering choose factorized Gaussian as variational distributions, the KL divergence is
DKIlLa Pw.0, DKI, DKI,(N(Hu diag(o:)))lN( diag(o:))]
(16)
1 pp
+ (14
(6)
12p)
log
+
(17)
2
(t)
op
2
To compute the analytical solution, let
M
a d
www e,1
(18)
M
a0,
Note here a refers to variance rather than standard deviation. The above equation is equivalent to
(8)
Hp
pip
0,
dp
1-1
(19)
09
(6)
+
(14
1p) )
0. o
da
1=1
We finally can solve these equations and get
1 M 1D = M
M
O M E6
( + (pg 1,))
(20)
as the result of Equation (5) in our main text. In short, this closed-form solution provides an efficient way to update the model prior from bunch of variational posteriors. It makes our method simple in practice, unlike some previous methods [11, 12] that require expensive meta-learning loops.
15

---

240
REC midiinddd innelarin
38.5
33.59
23.0
11.00
13.96
22.5
11.06
31.
33,02
320
23.2
Riee gee-noning
agc Group
Wrnetuning
ABC Group
free nomiing
50
130
250
200
toe men-m
PBC Gomup 217 Fime men-aannng
RBC group 218 fime-tuning
Compressed Groups
Figure 7: The approximated PSNR value changes as the fine-tuning process goes on.
C The Approximated PSNR Changes As Fine-tuning Goes On
We compressed some of the parameters using A coding, directly sampled the rest from the poste- rior distributions, and used their corresponding KL. divergence to estimate the coding cost. At the same time, the decoding quality. As shown in Figure 7, the PSNR tends to increase as the fine-tuning pro- cess goes on. However, tends drop when the fine-tuning process is nearing completion. This phenomenon occurs because, at the initial fine-tuning stage, the fine-tuning gain is more than the loss from A coding, many uncompressed groups can be fine-tuned to correct the errors of A* coding. But when the fine-tuning process nears completion, there are fewer uncompressed groups which could compensate for the bad sample of A coding. Therefore, the general PSNR curve tends to decrease when it approaches the end of fine-tuning, This figure shows that while A* coding's s sample results may have a distance the accurate posterior, our proposed progressive fine-tuning strategy effectively mitigates most of these discrepancies.
obtain the approximated PSNR value by using the posterior samples estimate
D Dynamic Adjustment of B
When learning the model prior, the value of 3 that controlling the rate-distortion trade-off is defined in advance to train the model prior at a specific bitrate point. After obtaining the model prior, we will first partition the network parameters into K groups WI:K (WI.... wx) according the average approximate coding cost of training data, as described in Section 3.3. Now for training the variational posterior for given test datum, to ensure the coding cost of each group is close to
= 16 bits, we adjust the value of 3 dynamically when optimizing the posteriors. The detailed algorithm is illustrated here in Algorithm 4.
The algorithm is improved from Havasi et al. [23] to stabilize training, in the way that we set an interval [x 0.4, x] buffer area where we do not change the value of dς. Here we only adjust A every 15 iterations to avoid frequent changes at the initial training stage.
E Experiment Details
We introduce the experimental settings here and summarize the settings in Table 3.
E.1 CIFAR-10
We use l-yerr MLP with 16 hidden units and Fourier embeddings for the CIFAR-10 dataset. The model prior is trained with 128 epochs to ensure convergence. Here, the term "epoch" is used to refer to optimizing the posteriors and updating the prior in the Algorithm I in the main text. For each epoch, the posteriors of all 2048 training data are optimized for 100 iterations using the local reparameterization trick [33], except the first epoch that contains 250 iterations. We use the Adam optimizer with learning rate 0.0002. The posterior variances initialized 9 x 10-6.
16

---

Algorithm 4 Dynamic 3 adjustment for optimizing the posteriors
Require: 8, Wi1K {W1, ( ................................ wK}
Initialize: A 3, 1.....,....
Initialize: variational posterior Twe h = 1, K
for i - Numberlter do
dR. DKLLow. pw.
Qw1.K - Variati prrratiionalldateeCS...o
CX.. is defined in Equation 8 in the main text
if (i mod 15) then
if 0g > x then 2 2 1.05
end if
if oµ < K - 0.4 then d X 1.05
end if
end if
end for
return Owe AR. k ,,..,
After obtaining the model prior, given a specific test CIFAR-10 image to be compressed, the pos- terior of this image is optimized for 25000 iterations, with the same optimizer. When we finally progressively compress and fine-tune the posterior, the posteriors of the uncompressed parameter groups are fine-tuned for 15 iterations with the same optimizer once previous group compressed.
E.2 Kodak
For Kodak dataset, since training on high- image takes much longer time, the model prior learned using fewer training data, i.e., only 512 cropped CLJC images [52]. We also reduce the learning rate of the Adam optimizer to 0.0001 to stabilize training. In each epoch, the posterior of each image is trained for 200 iterations, except the first epoch that contains 500 iterations. We also reduce the total epoch number to 96 which is empirically enough to learn the model prior.
We use models with different capacity for compressing high-resolution Kodak images. The smaller model is 6-layer SIREN with 48 hidden units and 64 Fourier embeddings. This model is used get the three low-bitrate points in Figure 2b in our main text, where the corresponding beta is set as {10 1 4 x 10 10-8). Another larger model comprises 7-layer MLP with 56 hidden units and 96 Fourier embeddings, which used for evaluation at the two relatively higher bitrate points in Figure 2b in our main text. The betas of these two models have the same value 2 x 10-9. We empirically adjust the variance initialization from the set (4 10 4 x 10-10) and find they can affect the converged bitrate and achieve good performance. In particular, the posterior variance initialized as 4 100 to reach the highest bitrate point in the rate rite-distortion curve. The posterior variance of other bitrate-points Kodak dataset are all initialized as 4 10-6.
Important note: It required significant empirical effort to find the optimal parameter settings described above, hence our note in the Conclusion and Limitations section that Bayesian neural networks are inherently sensitive to initialization [21].
F.3 LibriSpeech
We randomly crop 1024 audio samples from LibriSpeech "train-clean- 100" set [26] for learning the model prior and randomly crop 24 test samples from "test-clean" set for evaluation. The model structure is the same as the small model used for compressing Kodak images. We evaluate on four bitrate points by setting 3 = {10-7, 3 x 10-8, 10-8 10-9). There are 128 epochs, and each epoch has 100 iterations with learning rate as 0.0002. The first epoch has 250 iterations. In addition, the posterior variance initialized as 10 1 The settings for optimizing and fine-tuning posterior of a test datum are the same as the experiments on Kodak dataset.
17

---

Kodak
CIFAR-10
LibriSpeech
Smaller Model
Larger Model
Network Structure
number MLP layer
6
hidden unit
48
Fourier embedding
32
64
96
64
numher parameters
1123
21563
12675
Learning Model Prior from Training Data
number of training data
2048
512
512
1024
epoch number
128
learning rate
0.0002
0.0001
0.0001
0.0002
iteration/ / epoch
100
200
200
100
(except the first epoch)
steration number
250
the first epoch initialization
9
10
10
4 x 10 9,4 10
4 x 10
posterior variance
2
9,5 10
9.2
10-2,100$,
10 4 x 10-8
x 10
7.3 10
9.5
10
",10 1010
Optimize the Posterior of Test Datum
iteraion number
25000
25000
25000
25000
leamiing rate
0.0002
0.0001
0.0001
0.0002
truiming with
1/4 the points (pixels)
number of group (KL budget=
(58, 89, 146,
(1005, 2924,
(1729, 2962, 3264)
(5503, 7176)
224, 285)
4575, 6289)
16 bits / group)
bitrute, (hpp for images,
(0.91, 1.39, 2.28,
(5.36, 15.59,
(0.070, 0.1 0.13
(0.224, 0.293)
Kbps for audios)
3,50, 4.45)
24.40, 33.54)
(0.91, 1.39, 2.28,
(5.36, 15.59,
PSNR, dB
(0.070, 100, O. 0.132)
(0.224, 0.293)
3.50, 4.45)
24.40, 33.54)
Table 3: Hyper parameters in our experiments.
F
Supplementary Experimental Results
F.
Number of Training Samples
Since the model prior is learned from a few training data, the number of training data may influence the quality of the learned model prior. We train the model prior with different number of training images from the CIFAR-10 training set and evaluate the performance on 100 randomly selected test images from the CIFAR-10 test Surprisingly, 16 training images can help to learn good model prior. Considering the randomness of training and testing, the performance on this test subset almost the same when the number of training data exceeds 16. This demonstrates that the model prior is quite robust and generalizes well to test data. In our final experiments, the number of training samples is set to 2048 (on CIFAR-10 dataset) to ensure the prior converges to a good optimum.
shown in Figure 8, found that merely
F.2 Compressing Audios with Small Chunks
The proposed approach does not need to compute the second-orde gradient during training, which helps learn the model prior of the entire datum. Hence, compression with a single Bayesian INR network helps to fully capture the global dependencies of datum. That is the reason for our strong performance on Kodak and LibriSpeech datasets. Here, we also conduct a group of experiment to compare the influence of cropping audios into chunks. Unlike the experimental setting in our main text that compresses every 3-second audio (1 x 48000) with single MI.P network, here we try to crop all the 24 audios into small chunks, each of the chunk has the shape of 1 x 200. We use the same
18

---

PSNR(dB)
PSNR(dB)
27.5
27.0
COMBINER
COMBINER X
BINER
2
2
27
2
2
2
training samples
Kbps
Figure 8: Impact of the number of training data.
Figure 9: Compressing audios.
network used for compressing CIFAR- 10 images for our experiments here. As shown in Figure 9, we do not compress the audio as an entire entity, the performance will drops for around 5 dB. demonstrates the importance of compressing with single MIP network capture the inherent redundancies within the entire data.
F.3 Additional Figures
We provide some examples of the decoded Kodak images in Figure 10.
Ground Truth
0.0703 hpp. 23.02 dB
0.2928 hpp, 25.43 dB
Ground Truth
0.0703 hpp, 29.73 dB
0.2928 hpp, 33.59 dB
Figure 10: Decoded Kodak images.
19


Reviews:
[{"review_id": "Oj0kah74x9", "reviewer": "Reviewer_WLVL", "summary": "The paper proposes a new method for compressing general signals, by using Variational Bayesian implicit neural representations. It proposes an algorithm for learning a prior distribution over the implicit representation weights, as well as a pipeline for inferring the posterior distribution corresponding to every given signal to be encoded. The authors employ several tricks (e.g. Relative Entropy Coding) borrowed from previous works, that are meant to increase compression efficiency.\n\nThe method is evaluated over two image datasets (namely the CIFAR10 dataset of very small images and the Kodak dataset containing 24 larger images) and one audio dataset, while comparing performance with several image compression methods and the classic MP3 audio compression method, respectively.", "questions": "As I'm not entirely familiar with this field, the authors are welcome to direct my attention to certain points they feel that I overlooked, and I'll gladly consider increasing my initial rating.", "limitations": "Limitations were candidly discussed in the paper.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Generally speaking, the paper is presented well - despite the fact that compression is not my main expertise, I was able to follow and understand most of it.\nThe combination of the different ideas used in this method is interesting and appealing.\nLimitations, as well as concurrent work, are discussed candidly.", "weaknesses": "Besides the method's complexity, which could (at least partially) be attributed to my lack of expertise in this field, I found two major flows:\n\nPerformance:\nThe performance curves presented in Fig. 2 and 5 in the paper do not indicate an advantage of the proposed method over many of the existing methods, in terms of the rate-distortion trade-off. In the case of audio (Fig. 5), some advantage is shown only over the classic MP3 compression, and only for part of the kbps range.\n\nSpeed/Complexity:\nThe encoding process of this method is long and expensive. Besides being a big limitation (as noted by the authors) and hence a weakness of the method compared to some of the other ones, I'm missing some detailed comparison of these aspects.\n\nA third, relatively more minor flow has to do with the experimental setup itself, as experiments were conducted over only two small image datasets and one audio dataset, and compared only with MP3 in the latter case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's time and effort to review our paper and address your concerns below.\n\n## Performance\n\nIn our paper, we focus on beating the previous INR-based compression methods. The performance of VAE-based methods is shown in Figure 2 using dotted lines for reference, similar to all previous INR-based compression papers [10][11][12].  \nWhile VAE-based methods have long been dominant in neural image compression, compressing with INRs has emerged as a promising alternative but has certain limitations given its novelty and short development history. As Reviewer D8f2 commented, research would get stuck in a local minimum if we require new approaches to outperform established ones too soon in the research cycle. Our proposed method supports joint rate-distortion optimization, a significant improvement compared to previous INR-based compression methods. \n\n## Complexity\n\nFirst, after improving the practical implementation of entropy coding, we update our method's encoding and decoding times in Tables 1 and 2, as shown in the one-page rebuttal document. Now both the encoding time and decoding time are acceptable for practical application. Specifically, on the CIFAR-10 dataset, it only takes around 33 minutes for our largest bitrate model to encode 500 images from the CIFAR-10 dataset, meaning each CIFAR-10 image takes around 3.96 seconds for encoding. We can compare our method COMBINER with previous methods like COIN [10] and COIN++ [11] (according to the coding time reported in the paper of COIN++), showing the comparisons in the below table. \n\n|                   |  COIN             |  COIN++           |  COMBINER (Ours)  |\n|-------------------|-------------------|-------------------|-------------------|\n| Encoding Time     |    2.97 s         |    0.095 s        |    3.96 s         |\n| Decoding Time     |    0.46 ms        |    1.29 ms        |    3.90 ms        |\n\nSecond, we can reduce the number of iterations during the finetuning process to shorten the encoding time. As shown in Figure 1 in the rebuttal document, we can reduce the iteration number from about 30000 to less than 3000, while the PSNR only drops for 0.3 dB. It demonstrates the time of progressive finetuning, which constitutes the bulk of the encoding time, can be largely reduced by only sacrificing a little compression performance. \n\n## Experimental Setup\n\nEvaluating on Kodak dataset with 24 images is a standard setting for research in data compression. It is because compression models are not so sensitive to imbalanced data distribution, and the image distribution on the Kodak dataset is relatively representative. \nIn addition, most previous INR-based works are constrained by MAML's computational demands and hence cannot be directly applied to high-resolution image compression like COMBINER can.\nTherefore, these works report their performance on the low-resolution Cifar-10 dataset. \nSince these works are our important baselines, we also follow this setting and compare the results on the Cifar-10 dataset for consistency.\n\n_We are happy to answer any further questions the reviewer might have. However, if we answered the reviewer's questions adequately, we kindly invite the reviewer to consider raising their score._"}, {"review_id": "cx6KWlHGrP", "reviewer": "Reviewer_D8f2", "summary": "This paper addresses the problem of lossy data compression (evaluation is on images and audio) using implicit neural representations (INRs). In this approach, a neural network is designed that maps coordinates (e.g., x,y locations in an image or time for audio) to samples (RGB values or audio amplitude). Then the overfit network itself is the representation that is stored or transmitted since the data can be approximately recovered by running the network over appropriate coordinates.\n\nINRs for data compression are not new, but this paper includes two key novelties: (1) a variational Bayesian approach is adopted, which (2) allows the method to jointly optimize for rate (the entropy of the neural network parameters) and distortion (the quality of the reconstructed data). Previous INRs for compression used less powerful formulations and separately optimized for rate and distortion, which typically leads to worse overall performance (and the empirical evaluation in this paper demonstrates that here).\n\nWithin this setup, the authors describe an iterative algorithm for learning the prior over network weights, and they describe a progressive refinement method that improves performance by dividing the network parameters into blocks and conditionally coding each block given previous blocks.\n\nThe method is evaluated on images (CIFAR-10 and Kodak) and audio (LibriSpeech). In both cases, the method is shown to outperform previous INR-based models. For images, it does not outperform the best VAE-based compression methods.", "questions": "Section 5.1 (\"Models\") discusses the model architecture saying that it was empirically determined. How sensitive are the results to the base model architecture, and how easily can the encoding process prune (reduce parameter entropy to near zero) large parts of a model architecture that is much larger than the RD optimal model?", "limitations": "adequately addressed", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Originality: the problem is not new and jointly optimizing rate and distortion is quite common in neural compression in general, but I have not seen it done before for INRs. Nor have I seen an approach that uses variational Bayes and relative entropy coding with INRs.\n\nQuality and clarity are both excellent.\n\nI think the significance of this paper is quite high for the neural compression community and especially researchers looking at INRs. VAE-based (often called \"nonlinear transform coding\" or NTC) is the dominant approach right now, but INRs are quite interesting and there's a growing literature using such methods for compression. This work is significant because it does the right thing (joint optimization of rate and distortion), which has not been done before with INRs, and because it sets a new SOTA for compression with this approach.", "weaknesses": "1. Obviously the paper would be stronger if the empirical results were better. Focusing on Fig. 2b (RD curves on the Kodak image set), COMBINER trails CVPR2020 by more than 2dB at 0.2 bpp (a huge gap), and CVPR2020 is no longer a SOTA approach. That said, COMBINER is SOTA for INR-based methods (as far as I know), which is an important result, and I think research can get stuck in a local minimum if we reviewers require new approaches to outperform established ones too soon in the research cycle, e.g., Fig. 2b shows that COMBINER slightly outperforms the VAE-based method from ICLR2018.\n\n2. Neural model compression is a large subfield so some discussion or comparison with a model compression method (beyond simple weight quantization) seems like an important baseline that's missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful reading and constructive feedback and respond to the reviewer's concerns below.\n\n> Obviously, the paper would be stronger if the empirical results were better. COMBINER is SOTA for INR-based methods (as far as I know), which is an important result, and I think research can get stuck in a local minimum if we reviewers require new approaches to outperform established ones too soon in the research cycle. \n\nThanks for the positive comments. As the reviewer says, it is often unrealistic to expect a new approach to surpass all these well-established approaches in a short time. But we do believe that our paper will provide insights into this field, and we will keep improving our method continually. We believe this framework will soon be developed to achieve more impressive compression performance. \n\n> Neural model compression is a large subfield so some discussion or comparison with a model compression method (beyond simple weight quantization) seems like an important baseline that's missing. \n\nHavasi et al. [23] applied relative entropy coding to model compression and demonstrated state-of-the-art performance, and they also compared it with some model compression baselines. Another paper [Ref1] also compared with some model compression methods and found using relative entropy coding is more effective for Bayesian model compression than other model compression methods, especially when the model size is small. This is shown, for example, in Table 1 of [Ref1] (Minimal Random Coding Learning achieves the best performance on LeNet5-Caffe). As these prior works already demonstrate the superior performance of using relative entropy coding for model compression, we omit these results in our paper.\n\n## References\n\n[Ref1] Scalable Model Compression by Entropy Penalized Reparameterization. Oktay et al., ICLR 2020."}, {"review_id": "4JBWw5wSRd", "reviewer": "Reviewer_7dCG", "summary": "The paper improves INR-based (image) compression by introducing majorly two techniques: 1) a relative entropy coding based model compressing framework as an alternative to commonly used quantization - entropy coding pipeline; 2) a semi-amortized approach to train the model prior which is similar to beta-VAE and enables RD tradeoff with relative entropy coding. With the proposed novel pipeline, one can adopt COIN-like INR compression with better RD performance because it gets rid of quantization errors.", "questions": "See above", "limitations": "The authors have discussed the limitations, which is convincing.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written with dedicated figures and easy to follow. Overall I like the story of adopting beta-VAE/REC in INR coding methods. It is intuitive that given a dataset, all data neural representations share some common knowledge, and a potential amortized approach tends to exist. Thus, a meta-model describing the prior model distribution is soundable. This re-connects compression with VAEs, though a different way from feature-based learned image compression methods. And this re-connection makes a finer rate control of INR more promising. I believe this work contributes to society and can inspire future studies.", "weaknesses": "My major concern is about practicality. Seems that to train a model prior, we should first train many model posteriors. As also discussed by the authors, this training makes the entire training (encoding) time extremely slow. Is it possible to adopt this model posterior to further speed up the encoding of samples out of the training set? i.e. we may imagine efficiently finetuning a new INR (posterior) from the obtained prior. If this finetuning requires less time to convergence, the enlarged encoding time may be amortized.\n\nAnother issue for me is the somewhat marginal performance improvement. Is it worth costing such a large encoding time for the model prior, instead of simply training a COIN/COIN++ model longer to compensate for the quantization error? It would be better to report the converging speed of the models e.g. something similar to PSNR-iteration curves.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments and feedback on our work; we address the reviewer's concerns below.\n\n> Seems that to train a model prior, we should first train many model posteriors. As also discussed by the authors, this training makes the entire training (encoding) time extremely slow. Is it possible to adopt the model posterior of training samples to further speed up the encoding of samples out of the training set? We may imagine efficiently finetuning a new INR (posterior) from the obtained prior. \n\nThis appears to be a misunderstanding; we don't need to re-learn the prior each time we encode an image. We learn the model prior once from a few training images and their corresponding posteriors, akin to learning the weights of a neural network before deployment. Then, during test time, we fix the prior we learnt earlier and only optimize the variational posterior of the INR corresponding to the data. \n\nWe experimented with training a new INR model initialized from the established prior. However, we observed its convergence speed is similar to an appropriately randomized initialization. Consequently, given a model prior and a test image, we trained the corresponding INR posterior from scratch. However, it is an interesting question whether we could find better initialization heuristics for our INRs to speed up convergence. We will note this as a possible avenue for future work in the camera-ready version of the paper.\n\n> Is it worth costing such a large encoding time to compensate for the quantization error? \n\nWith the sacrifice of encoding time, we contribute to two critical improvements compared with previous methods: (1) joint rate-distortion optimization; (2) memory-efficient training. The first improvement is critical because the development path of VAE-based compression indicates that joint rate-distortion is necessary for superior performance. The second improvement means we do not need to crop high-resolution signals such as Kodak images into patches like previous methods. In that way, implicit neural representations have much more potential to capture the whole image's correlation structure and deliver better compression performance. \n\nCompression with implicit neural representations is a new and promising paradigm in data compression. Many researchers are currently interested in whether INR-based compression can surpass VAE-based compression in terms of rate-distortion performance. It is quite similar when Minnen et al. [3], for the first time, employed the autoregressive context model to improve the compression performance, which increases the decoding time dramatically. But several follow-up works successfully solved this issue soon after the work of Minnen et al. [3]. We believe our work can provide insights into this field as a new INR-based compression approach with several advantages, and we are hopeful that many of our method's practical limitations will be removed in future work.\n\n> It would be better to report the converging speed of the models e.g. something similar to PSNR-iteration curves. \n\nWe appreciate your constructive suggestion. We conducted additional experiments to address this, adjusting the finetuning strategy across varying iteration numbers. Our analysis on a Kodak image (Kodim03.png) is described in a graph juxtaposing PSNR against iteration number. Figure 1 in our rebuttal document shows that performance improves marginally as iterations increase, and the finetuning time grows linearly according to the iteration number. The results in Figure 2b of our main paper match the highest iteration number. Notably, we can reduce encoding time at the small cost of a 0.3dB drop in PSNR by setting the iteration number to 10%."}, {"review_id": "hNcbZp9B7d", "reviewer": "Reviewer_5R2p", "summary": "This paper proposes overfitting variational Bayesian neural networks to the data and compressing an approximate posterior weight sample using relative entropy coding, which enables direct optimization of the rate-distortion performance by minimizing the $\\beta$-ELBO. Moreover, an iterative algorithm for learning prior weight distributions is introduced and a progressive refinement process for the variational posterior is employed for improved performance. Experiments were conducted on image and audio compression.", "questions": "- It would be nice to compare the proposed method with the concurrent work [52[ published in ICML 2023.\n- Will the encoding time be the obstacle for applying he porposed method to video compression or 3D data compression?\n- Code availability helps reproducibility although enough implementation details are provided.\n", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- the work encodes data with variational Bayesian implicit neural representations, which enables direct optimization of the rate-distortion performance by minimizing the $\\beta$-ELBO. \n- the work presents an iterative algorithm for learning prior weight distributions and a progressive refinement process for the variational posterior for improved performance. \n", "weaknesses": "The presentation can be improved, e.g.,\n- $\\theta_p$ indicates $\\mu_p, \\sigma_p$, which should be clearly described.\n- Some typos, e.g., Line 183: represent -> represents; Line 190: the to -> the; Line 209: the its -> its; Line 296: subscript \"2\".\n- Title of [5] is missed, \"Auto-Encoding Variational Bayes\". Similar problem in [35]. And [35] actually repeats [34].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful reading and detailed review. We will carefully proofread the paper again and correct the typos and references in our final version. Moreover, we address the reviewer's questions below.\n\n> It would be nice to compare the proposed method with the concurrent work [52] published in ICML 2023. \n\nVC-INR [52] adopts a similar MAML framework also used by prior works, including MSCN [13] and COIN++ [11], and achieves impressive results. Our approach is novel and different from all these works; thus, we still have a gap with that paper. However, we would like to note the significant gap in complexity between the two methods: besides using the MAML-based approach, VC-INR uses a complex gating mechanism, low-rank weight factorization and variational auto-encoders to encode the weight factors to obtain its good performance. On the other hand, COMBINER simply encodes a variational INR with relative entropy coding.\n\n> Will the encoding time be the obstacle for applying the proposed method to video compression or 3D data compression?\n\nThanks for your question. Although videos or 3D data usually have more pixels/points than image data, our method still applies to these data formats. As a simple, practical solution, we can sample a subset of pixels during training without significantly compromising performance. For example, when training the model prior on a high-resolution image dataset, we sample 25% pixels to accelerate training, which we found to have negligible impact on the final performance. On the other hand, we can reduce the iteration number during the progressive finetuning process. As shown in Figure 1 in the rebuttal document, as we decrease the iteration number, the finetuning time reduces from nearly 1 hour (corresponding to nearly 30000 iterations) to 334 seconds (less than 3000 iterations), which only sacrifices 0.3dB reconstruction quality.\n\n> Code availability helps reproducibility although enough implementation details are provided.\n\nIt took us some time to clean up the code after submitting the manuscript. We have completed cleaning up part of the code and sent the anonymous link to the AC guided by the NeurIPS rebuttal policy."}, {"review_id": "34ZyUFXs3L", "reviewer": "Reviewer_rYFn", "summary": "## Summary\nThe authors propose an approximated correlation communication approach to compress Bayesian INF. Two practical considerations, such as prior fitting and posterior refinement are proposed. It achieves an R-D performance comparable to the SOTA INF method on image compression. Moreover, the audio compression is also considered.\n\n========================================\n## Post rebuttal\nThe authors have addressed my concern wrt to the experiments, in a quite unexpected way. I raise my rating to accept.\n", "questions": "See weakness.", "limitations": "The limtation is discussed.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "## Strength\n* The idea of using correlation communication to compress image INR is novel. The compression of bayesian INR is also novel and likely to have impact outside compression community. Furthermore, it also considers audio compression and achieves reasonable results.\n", "weaknesses": "## Weakness\n* To the best of my knowledge, there is no algorithm that communicates correlation using finite number of samples. The adaptative reject sampling [Harsha 2010, The Communication Complexity of Correlation] and Possion Functional Representation [Li 2018, Strong Functional Representation Lemma and Applications to Coding Theorems] both requires infinite number of samples in the worst case to communicate exact posterior. As [Theis 2021, Algorithms for the Communication of Samples] has shown in Fig. 2, the expected code length of relative entropy coding [Flamich 2020], even if converge expotentially fast, still have quite some distance to the true posterior. And the expected code length of [Flamich 2020] is also inferior to [Harsha 2010] [Li 2018], which is $D + \\log D + \\log\\log D +,... $ The [Flamich 2020] does not emphasis this gap, while it is supposed to negatively effect the performance of the proposed approach. And this inaccurate posterior gap also exists for A* coding [Flamich 2022]. Then my question is, how bad such posterior approximation it is? Or to say, if we assume that we can use infinite sample optimal PFR to achieve exact posterior, what is the R-D curve like?\n* Another weakness is that the advantage of the proposed approach over MSCN [Schwarz 2022, Meta-Learning Sparse Compression Networks] is unclear. From the perspective of R-D performance, the proposed approach is only comparable to MSCN on Kodak and is slightly outperformed by MSCN on CIFAR. However, the encoding time of the proposed approach seems to be quite long due to the progressive posterior refinement. As neither the author nor original MSCN paper report encoding time of MSCN, I can not evaluate the advantage of the proposed approach pver MSCN. Or to say, if the authors can show that the run time of propoased approach is significantly faster than MSCN, this paper is more acceptable. This is especially true as the authors claim that their approach is simpler (L54,66,253). They need evdience, such as faster runtime, to support their claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and the valuable questions. We respond to their concerns below.\n\n**Concerns regarding channel simulation:** While the reviewer states important questions about the quality of our coding scheme's posterior approximation quality (which we address below), we also note that the reviewer conflates exact and approximate channel simulation. Harsha et al.'s rejection sampler, PFR and A* coding are all exact channel simulation algorithms, whose optimal, practically achievable expected codelengths are at most $\\mathbb{I}[\\mathcal{D} : w] + \\log(\\mathbb{I}[\\mathcal{D} : w] + 1) + 5$ bits (Theorem 1; Li & El Gamal, 2018), where $\\mathbb{I}[\\mathcal{D} : w]$ denotes the mutual information between the data $\\mathcal{D}$ and the INR weights $w$. Furthermore, given mild constraints that are virtually always satisfied in practice (bounded target-proposal density ratio), all these algorithms terminate with probability 1. However, as we mention in line 96, we use depth-limited A* coding in our work, which is an approximate channel simulation algorithm with finite runtime and the bias of which is formally characterised by Theorem 3.2 of Havasi et al. [23].\n> How bad such posterior approximation it is? Or to say, if we assume that we can use infinite sample optimal PFR to achieve exact posterior, what is the R-D curve like?\n\nWe thank the reviewer for this question. First, as we explain in lines 174-180, we use our finetuning procedure precisely to mitigate the approximation gap. However, we have also added an extra group of experiments to show the theoretical R-D curve in Figure 2 in the rebuttal document that we will also include in the camera-ready version of our paper. In these new experiments, we still finetune the model parameters progressively. However, instead of A* coding, we directly sample from the posterior distribution of the current block and assume that the sample is transmitted to the decoder at maximum efficiency, using $D_{KL}[Q || P]$ bits. We see that on the CIFAR-10 dataset, the theoretically optimal performance is 1 dB higher than the practical performance, demonstrating that there is still much room for improvement for our coding scheme.\n\nIn addition, we conducted another experiment to investigate how the PSNR value changes during the finetuning process, the results of which are shown in the rebuttal document as Figure 3. We compressed some of the parameters using A* coding, directly sampled the rest from the posterior distributions, and used their corresponding KL divergence to estimate the coding cost. Interestingly, the PSNR tends to increase as the finetuning process goes on, however, it tends to drop when the finetuning process is nearing completion. This phenomenon occurs because, at the initial finetuning stage, the finetuning gain is more than the loss from A* coding, as many uncompressed groups can be finetuned to correct the errors of A* coding. But when the finetuning process nears completion, there are fewer uncompressed groups which could compensate for the bad sample of A* coding. Therefore, the general PSNR curve tends to decrease when it approaches the end of finetuning. \n\nIn summary, these two figures show that while A* coding's sample results may have a distance to the accurate posterior, our proposed progressive finetuning strategy effectively mitigates most of these discrepancies. \n> Another weakness is that the advantage of the proposed approach over MSCN is unclear.\n\nWe recently caught an error in the evaluation of the MSCN paper, which we confirmed with the authors in private communication. In particular, the image compression performance they report on the Kodak dataset has some errors. Since they cannot directly apply their method to the high-resolution images due to computational constraints, they split the images into patches and encoded them separately. However, they calculate the PSNR for the images by averaging the PSNR values of their patches, which is incorrect. The correct would have been to reassemble the patches first and compute the PSNR on the whole image. Thus, COMBINER actually performs much better than MSCN on the Kodak dataset (more than 1.5dB gain).\n> As the authors claim that their approach is simpler, they need evidence, such as faster runtime, to support their claim.\n\nWe thank the reviewer for their suggestion, we elaborate on COMBINER's simplicity and runtime below.\n\n**Simplicity:** Our claim refers to the technical simplicity of our method, which consists only of two parts: (1) train a variational Bayesian implicit neural representation on some data and (2) transmit a sample from the INR's weight posterior to encode the data it represents. \n\n**Runtime:** After improving the practical implementation of entropy coding, we update our method's encoding and decoding times in Tables 1 and 2, as shown in the one-page rebuttal document. Now both the encoding time and decoding time are acceptable for practical application. Specifically, on the CIFAR-10 dataset, it only takes around 33 minutes for our largest bitrate model to encode 500 images from the CIFAR-10 dataset, meaning each CIFAR-10 image takes around 3.96 seconds for encoding. We can compare our method COMBINER with previous methods like COIN [10] and COIN++ [11] (according to the coding time reported in the paper of COIN++), showing the comparisons in the below table.\n\n|                   |  COIN             |  COIN++           |  COMBINER (Ours)  |\n|-------------------|-------------------|-------------------|-------------------|\n| Encoding Time     |    2.97 s         |    0.095 s        |    3.96 s         |\n| Decoding Time     |    0.46 ms        |    1.29 ms        |    3.90 ms        |\n\n_We are happy to address any further questions the reviewer might have. However, should we have addressed the reviewer's concerns adequately, we kindly ask them to consider raising their score._"}]

Author rebuttal:
We extend our gratitude to all the reviewers for their comprehensive feedback and time spent reviewing our manuscript. It is heartening that all the reviewers agree that the idea proposed in this paper is novel and valuable. We have addressed their concerns in our respective responses.

In addition, after submitting the manuscript, we cleaned up the code and improved the practical implementation. On the one hand, we attach an anonymous link of our code to the AC for reproducibility guided by the NeurIPS rebuttal policies. On the other, we also update the practical encoding and decoding time, as shown in Tables 1 and 2 of the one-page rebuttal document. Specifically, encoding for 500 CIFAR-10 images now ranges from 13 to 33 minutes (0.91 to 4.45 bpp), which means it only takes a few seconds to encode each CIFAR-10 image. For high-resolution Kodak images, the encoding time is now only 21.5 minutes at 0.07 bpp. Moreover, by adjusting the number of iterations during the progressive finetuning phase, we can reduce the finetuning time by 90 %, with only a minimal 0.3 dB performance trade-off. We will include these statistics in the final version of our paper.

Meta-review: Author rebuttal addressed most of the reviewer concerns and all reviewers are leaning accept after final discussion.

While implicit representations are catching on for compression applications, the paper proposes to fit a variational BNN and then uses relative entropy coding (as opposed to quantization and entropy coding, which is most common). This combination is quite unique and flexible, which in conjunction with other contributions, yields strong results worthy of highlighting at the conference via spotlight.
Decision: Accept (spotlight)