RECOMBINER: Robust and Enhanced Compression with Bayesian Implicit Neural Representations
Authors: Jiajun He
Gergely Flamich
Zongyu Guo
José Miguel Hernández-Lobato
Venue: ICLR 2024 poster (2024)
Keywords: implicit neural representations
neural data compression
relative entropy coding
combiner
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We improve COMBINER, an INR-based neural data compression method by employing a richer variational posterior approximation, learned positional encodings and an expressive hierarchical prior.

Abstract: COMpression with Bayesian Implicit NEural Representations (COMBINER) is a recent data compression method that addresses a key inefficiency of previous Implicit Neural Representation (INR)-based approaches: it avoids quantization and enables direct optimization of the rate-distortion performance. However, COMBINER still has significant limitations: 1) it uses factorized priors and posterior approximations that lack flexibility; 2) it cannot effectively adapt to local deviations from global patterns in the data; and 3) its performance can be susceptible to modeling choices and the variational parameters' initializations. Our proposed method, Robust and Enhanced COMBINER (RECOMBINER), addresses these issues by 1) enriching the variational approximation while retaining a low computational cost via a linear reparameterization of the INR weights, 2) augmenting our INRs with learnable positional encodings that enable them to adapt to local details and 3) splitting high-resolution data into patches to increase robustness and utilizing expressive hierarchical priors to capture dependency across patches. We conduct extensive experiments across several data modalities, showcasing that RECOMBINER achieves competitive results with the best INR-based methods and even outperforms autoencoder-based codecs on low-resolution images at low bitrates. Our PyTorch implementation is available at https://github.com/cambridge-mlg/RECOMBINER/.

Published as a conference paper at ICLR 2024
RECOMBINER: ROBUST AND ENHANCED COMPRESSION WITH BAYESIAN IMPLICIT NEURAL REPRESENTATIONS
Jiajun He*
Gergely Flamich* * University of Cambridge gf3320cam ac uk
University of Cambridge jh23830cas cam. ac. uk
Zongyu Guo
José Miguel Hernández-Lobato University of Cambridge jmh233 20h2330cam, @cam. ac uk
University of Science and Technology of China guozy@mail. uste. edu.cn
ABSTRACT
COMpression with Bayesian Implicit NEural Representations (COMBINER) is a recent data compression method that addresses a key inefficiency of previous Implicit Neural Representation (INR)-based approaches: it avoids quantization and enables direct optimization of the rate-distortion performance. However, COMBINER still has significant limitations: 1) uses factorized priors and pos- terior approximations that lack flexibility; 2) it cannot effectively adapt to lo- cal deviations from global patterns in the data; and 3) its performance can be susceptible to modeling choices and the variational parameters' initializations. Our proposed method, Robust and Enhanced COMBINER (RECOMBINER), ad- dresses these issues by 1) enriching the variational approximation while retain- ing a low computational cost via linear reparameterization of the INR weights, 2) augmenting our INRS with learnable positional encodings that enable them to adapt to local details and 3) splitting high-resolution data into patches to in- crease robustness and utilizing expressive hierarchical priors capture depen- dency across patches. We conduct extensive experiments across several data modalities, showcasing that RECOMBINER achieves competitive results with the best INR-based methods and even outperforms autoencoder codecs on low- resolution images low bitrates. Our PyTorch implementation available https: //gi thgg............ com/ cambridge mlg RECCOMBINER
INTRODUCTION
Advances deep learning recently enabled a new data compression technique impossible with classical approaches:
train neural network to memorize the data (Stanley, 2007) and then encode the network's ' weights instead. These networks are called the implicit neural representation (INR) of the data, and differ from neural networks used elsewhere in three significant ways. First, they treat data as a signal that maps from coordinates to values, such as mapping (X,Yr Y) pixel coordinates (R, G, B) color triplets in the case of an image. Second, their architecture consists of many fewer layers and units than usual and tends to utilize SIREN activations (Sitzmann et al., 2020). Third, aim to overfit them the data as much as possible.
Unfortunately, most based data compression methods cannot directly and jointly optimize rate- distortion, which results in a wasteful allocation of bits leading to suboptimal coding performance. COMpression with Bayesian Implicit NEural Representations (COMBINER; Guo et al., 2023) ad- dresses this issue by picking variational Gaussian mean-field Bayesian neural network (Blundell et al., 2015) as the INR of the data. This choice enables joint rate-distortion optimization via maxi- mizing the INR's fevvidence lower bound (8-ELBO), where 3 controls the rate-distortion trade-off.
cqual contribution.

---

Published as a conference paper at ICLR 2024
Signal
Signal Regresentation
Reconstruction
ippresntation
splt reshups
uptample
Connironns
Outputs (9))
(a) COMMINER.
Rep esentation
concateratt with Founey entendiing Siiaals dewbinch ber 6 D. MMW)
M
[xc)
(9:).
(b) RECOMBINER.
(e) The input in RECOMBINER.
Figure 1: Schematic of (a) COMBINER and (b) RECOMBINER, our proposed method. See Sections and 3 for notation. As the INR's input, RECOMBINER uses ha upsampled to pixel-wise positional encodings concatenated with Fourier embeddings. (c) closer look at how RECOMBINER maps h2 to the INR input, taking images as example. FE: Fourier embeddings; FC: fully connected layer.
Finally, the authors encode weight sample from the INR's variational weight posterior to represent the data using relative entropy coding (REC; Havasi al., 2018; Flamich et al., 2020).
Although COMBINER performs strongly among INR-based approaches, it falls short of the state- of-the-art codecs on well-established data modalities both terms of performance and robustness. In this paper, we identify several issues that lead to this discrepancy: 1) COMBINER employs fully-factorized Gaussian variational posterior over the INR weights, which tends to underfit the data (Dusenberry et al., 2020), going directly against our goal of overfitting: 2) Overfitting small INRs used by COMBINER is challenging, especially at low bitrates: a small change to any weight can significantly affect the reconstruction at every coordinate, hence optimization by stochastic gradient descent becomes unstable and yields suboptimal results. 3) Overfitting becomes more problematic on high-resolution signals. As highlighted by Guo et al. (2023), the method is sensitive to model choices and the variational parameters' initialization and requires considerable effort to tune.
We tackle these problems by proposing several non-trivial extensions to COMBINER, which signifi- cantly improve the rate-distortion performance and robustness to modeling choices. Hence, we dub our method robus! and enhanced COMBINER (RECOMBINER). Concretely, our contributions are:
We propose a simple yet effective learned reparameterization for neural network weights specifically tailored for INR-based compression, yielding more expressive variational pos- teriors while matching the computational cost of standard mean-field variational inference.
We augment our INR with learnable positional encodings whose parameters only have a lo- cal influence on the reconstructed signal, thus allowing deviations from the global patterns captured by the network weights, facilitating overfitting the INR with gradient descent.
We split high-resolution data into patches improve robustness to modeling choices and the variational parameters' initialization. Moreover, we propose an expressive hierarchical Bayesian model to capture the dependencies across patches enhance performance.
We conduct extensive experiments verify the effectiveness of our proposed extensions across several data modalities, including image, audio, video and protein structure data. In particular, we show that RECOMBINER achieves better rate-distortion performance than VAE-based approaches on low-resolution images at low bitrates.
2 BACKGROUND
This section reviews the essential parts of Guo et al. (2023)'s compression with Bayesian implicit neural representations (COMBINER), as it provides the basis for our method.
Variational Bayesian Implicit Neural Representations: We assume the data wish to compress can be represented as a continuous function f : I - RDD from I-dimensional coordinates to O- dimensional signal values. Then, our goal is approximate with small neural network g(: w)
2

---

Published as a conference paper at ICLR 2024
with weights w. Given L hidden layers in the network, we write w = [[("] ....., wl']], which represents the concatenation of the L weight matrices w(',, ..... w(*], each flattened into row- vector. Guo et al. (2023) propose using variational Bayesian neural networks (BNN; Blundell et al., 2015) that place prior Pw and a variational posterior gw the weights. Furthermore, they use Fourier embeddings (xx) for the input data (Tancik et al., 2020) and sine activations at the hidden layers (Sitzmann et al., 2020). To infer the implicit neural representation (INR) for some data D, we treat D as a dataset of coordinate-valu pairs { (x:, y+)) e.g. for an image, x; can be an (X, Y) pixel coordinate and y, the corresponding (R, G. B) triplet. Next, we pick a distortion metric A (e.g., mean squared error) and a trade-off parameter / to define the 3-rate-distortion objective:
D
(DD, qwo Pw: 3) = B. DKL[[W [Pw] + D
Σ E.g. [A(ys, w)]:
(1)
where DKL ( Pw] denotes the Kullback-Leible divergence of qw from Pw: and as we explain below, it represents the compression rate of single weight sample w ~ qw. Note that Equation (1) corresponds to a negative 3-evidence lower bound under mild assumptions on A.
We infer the optimal posterior by computing ah = arg ming L(D, Iw: Pww 9) over an appropri- ate variational family Q. Guo et al. (2023) set to be the family of factorized Gaussian distributions.
Training COMBINER:
Once we selected a network architecture 9 for our INRs, a crucial element of COMBINER is select good prior on the weights Pw- Given a training set {Di,.... Dar and an initial guess for pw. Guo et al. (2023) propose the following iterative scheme to select the optimal prior: 1) Fix Pw and infer the variational INR posteriors Iw.m for each datum Dn, by minimizng Equation (1): 2) Fix the m 8 and update the prior parameters Pw based on the parameters of the posteriors. When the qw are Gaussian, Guo et al. (2023) derive analytic formulae for updating the prior parameters. To avoid overloading the notion of training, we refer to learning pw and the other model parameters training, and to learning &w as inferring the INR.
Compressing data with COMBINER: Once we picked the INR architecture g and found the optimal prior Pw: can COMBINER to compress new data D in two steps: 1) We first infer the varia- tional INR posterior 9w for D by optimizing Equation (1), after which 2) we encode an approximate sample from 9w using relative entropy coding (REC), whose expected coding cost is approximately DKI[Ww/Pe [w]] (Havasi et al., 2018; Flamich et al., 2020). Following Guo et al. (2023), we used depth-limited global-bound A coding (Flamich et al., 2022), to which we will refer as just A* cod- ing. Unfortunately, applying A* coding to encode a sample from qw is infeasible in practice, as the time complexity of the algorithm grows as (cxp(( D D [9w [[/ppw)) Hence, Guo et al. (2023) suggest breaking up the problem into smaller ones. First, they draw uniformly random permutation a on dim (w)) elements, and use to permute the dimensions of w as a a(w)= [Wa(1)..... W (addii()))
Then, they partition a(w) into smaller blocks, and compress the blocks sequentially. Permuting the weight vector ensures that the KL divergences are spread approximately evenly across the blocks. As an additional technical note, between compressing each block, we run a few steps of finetuning the posterior of the weights that are yet to be compressed, see Guo et al. (2023) for more details.
METHODS
In this section, we propose several extensions to Guo et al. (2023)'s framework that significantly improve its robustness and performance: 1) we introduce linear reparemeterization for the INR's weights which yields a richer variational posterior family; 2) we augment the INR'S input with learned positional encodings to capture local features in the data and to assist overfitting; 3) we scale our method to high-resolution image compression by dividing the images into patches and introducing an expressive hierarchical Bayesian model over the patch-INRs, and 4) we introduce minor modifications to the training procedure and adaptively select 3 to achieve the desired coding budget. Contributions 1) and 2) are depicted in Figure 1, while 3) shown in Figure 2.
3.1 LINEAR REPARAMETERIZATUN FOR THE NETWORK PARAMETERS
A significant limitation of the factorized Gaussian variational posterior used by COMBINER is that it posits dimension-win independent weights. This assumption is known to be unrealistic (Izmailov et al.. 2021) and underfit the data (Dusenberry et al., 2020), which goes directly against our goal of overfitting the data. On the other hand, using a full-covariance Gaussian posterior approximation would increase the INR's 's training and coding time significantly, even for small network architectures.
3

---

Published as a conference paper at ICLR 2024
Hence, propose a solution that lies in-between: high level, we learn linearly-train tranfooddrad factorized Gaussian approximation that closely matches the full-covariance Gaussian posterior on average over the training data. Formally, for each layer will = hWd A where the A are square matrices, and we place a factorized Gaussian prior and variational posterior on hw instead. We learn each A during the training stage, after which we fix them and only infer factorized posteriors 4, when compressing new data. To simplify notation, Alll)
,,...,, L, we model the weights as
we collect the A in a block-diagonal matrix A = diag( A (4., and the hw in a single h[') so that now the weights are given by w h. We found this layer-wise weight reparameterization as efficient as using joint one for the entire weight vector w. Hence, we use the layer-wise approach,
row-vector hw = [bW
more parameter and compute- efficiient.
This simple yet expressive variational approximation has a couple of advantages. First, it provides an expressive full-covariance prior and posterior while requiring much less training and coding time. Specifically, the KI. divergence required by Equation (1) is still between factorized Gaussians and we not need to optimize the full covariance matrices of the posteriors during coding. Second, this parameterization has scale redundancy: for any R have h (1/c hc) A). Hence, we initialize hw suboptimally during training. A can still learn to compensate for it, making our method more robust. Finally, note that this reparameterization is specifically tailored for INR-based compression and would usually not be feasible in other BNN use-cases, since we learn A while inferring multiple variational posteriors simultaneously.
3.2 LEARNED POSITIONAL ENCODINGS
A challenge for overfitting INRs, especially at low bitrates is their global representation of the data, in the sense that each of their weights influences the reconstruction every coordinate. To mitigate this issue, we extend our INRs to take learned positional input at each coordinate x;: g(x,, Z; w).
However, is usually wasteful to introduce vector for each coordinate in practice. Instead, we use a sower-dimensionll row-vector representation h2 that we reshape and upsample with a learnable function o. In the case of W H image with F-dimensional positional encodings, we could pick he such that dim h2) & F. W. H, then reshape and upsample it to be F W x H by picking to be some small convolutional network. Then, we set z; = (h2)x. to be the positional encoding at location x,. We placed factorized Gaussian prior and variational posterior on h2 Hereafter, we refer to h2 as the latent positional encodings, ob (h2)) and z, as the upsampled positional encodings.
3.3 SCALING To HIGH-RESOLUTION DATA WITH PATCHES
With considerable effort, Guo al. (2023) successfully scaled COMBINER to high-resolution images by significantly increasing the number of INR parameters. However, they note that the training pro- cedure was very sensitive to hyperparameters, including the initialization of variational parameters and model size selection. Unfortunately, improving the robustness of large INRS using the weight reparameterization we describe in Section 3.1 is also impractical, because the size of the transforma- tion matrix A grows quadratically in the number of weights. Therefore, we split high- high-resolution data into patches and infer separate small INR for each patch, in line with other INR-based works as well (Dupont et al., 2022; Schwarz & Teh, 2022; Schwarz et al., 2023). However, the patches' INRs are independent by default, hence we re-introduce information sharing between the patch-INRE weights via a hierarchical model for hw. Finally, we take advantage of the patch structure to parallelize data compression and reduce the encoding time RECOMBINER, as discussed the end of this section.
We posit global representation for the weights hw, from which each patch-INR can deviate. Thus, assuming that the data D is split into P patches, for each patch π 1...., P. we need to define the conditional distributions of patch representa- tions hw hw- However, since we wish to model deviations from the global representation, it is natural decompose the patch representation as h((*) An + hw and specify the conditional distribution of the differences AhW hw instead, without any loss of generality. In this paper, place factorized Gaussian prior and variational posterior on the joint distribution of the global
RECOMBINER's 'ss hierarchical Bayesian model:
4

---

Published as a conference paper at ICLR 2024
entire signal
low
H within
group
permuto across rows
patch
h
partition into blocks
P/G
coding
same
bugget
Three-level hicrarchical model and the corresponding graphical model.
(b) Permutation.
Figure 2: Illustration of (a) the three-level hierarchical model and (b) our permutation strategy.
representation and the deviations, given by the following product of P Gaussian measures:
(N diag( (w)) x
I M (m)
Addag(a (²
Ph Ahg-P) (PPP))
==1
P
9 The Ah New, diag
N N
(T) diag(p ())))
(3)
where P is the slice notation, i.e. Ahw (1:P) Ah posterior approximation in Equation (3) assumes that the global representation and the differences are independent, hw and h remain correlated. Note that optimizing Equation (1) requires to compute (9g: Ipp g. Unfortunately, due the complex dependence between the h(Ts. this calculation is infeasible. Instead, we can minimize an upper bound to it by observing that p) [ppee & DKL[9 (lphg:
Ah Importantly, while the
DKL (aulIn (.g.))
DKL ( (winh (R,PRR,, Rw.ng)))) ())
= DKI.9 Tw be (1:P)) |121 hw.. [P:p)]
DKLL Aa (PPPPP) P Ah(-P))
(4)
Hence, when training the patch-INRS, we replace the KL term in Equation (1) with the divergence in Equation (4), which is between factorized Gaussian distributions and cheap to compute. Finally, we remark that we can view hw as side information also prevalent in other neural compression codecs (Ballé al., 2018), auxiliary latent variables enabling factorization (Koller & Friedman, 2009).
While Equations (2) and (3) describe a two-level hierarchical model, we can easily extend the hi- erarchical structure by breaking up patches further into sub-patches and adding extra levels to the probabilistic model. For our experiments on high-resolution audio, images, and video, we found that three-level hierarchical model worked best, with global weight representation hw. second/group- level representations E(::0) and third/patch-level representations k()P), illustrated Figure 2a. Em- pirically, hierarchical model for h, did yield significant gains, thus we only use for hw-
Compressing high-resolution data with RECOMBINER: An advantage of patching is that we can compress and fine-tune INRS and latent positional encodings of all patches in parallel. Unfortunately, compressing P patches in parallel using COMBINER 's procedure is suboptimal, since the information content between patches might vary significantly. However, by carefully permuting the weights across the patches' representations we can 1) adaptively allocate bits to each patch to compensate for the differences in their information content and 2) enforce the same coding budget across each parallel thread to ensure consistent coding times. Concretely, we stack representations of each patch matrix at each level the hierarchical model. For example, in our three-level model we set
H
h()),,
(
(g) hw
H hw
where we use slice notation to denote the ith row as H.,: and the jth column as H. Furthermore, let S, denote the set of permutations on n elements. Now, at each level f, assume ( has C₁ columns and R2 rows. We sample single within-row permutation uniformly from Se, and for each column of H() ( we sample an across-rows permutation ag uniformly from SR, elements.
Then, we permute (t) as H = ((() (()))))))) Finally, we split the H into blocks row-wise, and encode and fine-tune each row in parallel. We illustrate the above procedure in Figure 2b.
5

---

Published as a conference paper at ICLR 2024
3.4 EXTENDED TRAINING PROCEDURE
In this section, we describe the ways in which RECOMBINER's training procedure deviates from COMBINER'S. s. To begin, we collect the RECOMBINER'S representations into one vector. For non- patching cases we set h [hw h and for the patch case using the three-level hierarchical model we set h = vec[[H (0) H H (1) For simplicity, we denote the factorized Gaussian prior and variational posterior over h as Ph = NNu, diag(o)) and Qh (v, diag(p)). where μ and are the means and a and are the diagonals of covariances of the prior and the posterior, respectively.
Training RECOMBINER: Our objective for the training stage is to obtain the model parameters A, H, (2023) control the coding budget implicitly by manually setting different values for 3 in Equation (I). In this paper, we adopt an explicit approach and tune 3 dynamically based on our desired coding budget of bits. More precisely, after every iteration, we calculate the average KL divergence of the training examples, i.e., 8 = TV 2m-1 DKLLa [Ph]] mlPp).
Mt, a given a training dataset (D)...., _DM} and a coding budget C. In their work, Guo et al.
1
If 5 > C, we update 3 by 3 - B x (1+ TC);
if I < C cC, we update 3 by 3 - 3/(1 + TC). Here <c is threshold parameter to stabilize the training process and prevent overly frequent updates to B, and To the adjustment step size. Unless otherwise stated, we set Te 0.5 in our experiments. Empirically, we find the value of 33 stabilizes after 30 to 50 iterations. We present the pseudocode of this prior learning algorithm in Algorithm Then, our training step is three-step coordinate descent process analogous to Guo et al. (2023)'s:
1. Optimize variational parameters, linear transformation and upsampling network: Fix the prior Ph. and optimize Equation (I) or its modified version from Section 3.3 via gradient descent. Note, that C is a function of the linear transform A and upsampling network parameters á too:
M
M Vu Pm) m==1 A, Φ
arg min (MemPPm Ar
Σ
C(Dm: m. Ph, A. A.0.33
(6)
M
2. Update prior: Update the prior parameters by the floeed-ormm solution:
M Σ
M
# - M
Vm
o M Σ
[um 14)² + pm
(7)
3. Update 3: Set B - B x + (c) or 3 - 33/1111- + TC) based on the procedure described above.
Note that unlike other INR-based methods (Dupont et al., 2022; Schwarz & Teh, 2022; Schwarz et al., 2023) our training procedure is remarkably stable, as we illustrate in Appendix D.4.
4 RELATED WORKS
Nonlinear transform coding: Currently, the dominant paradigm in neural compression is nonlinear transform coding (NTC; Ballé et al., 2020) usually implemented using variational autoencoders (VAE). NTC has achieved impressive performance in terms of both objective metrics (Cheng al., 2020; He et al., 2022) and perceptual quality (Mentzer et al., 2020), mainly due to their expressive learned non-linear transforms (Ballé et al., 2020; Zhu et al., 2021; Liu et al., 2023) and elaborate entropy models (Ballé et al., 2018; 8; Minnen et al., 2018; Guo et al., 2021).
Compressing INRS can also be viewed as a form of NTC: we use gradent descent to transform data into an INR. The idea to quantize INR weights and entropy code them was first proposed by Dupont al. (2021), whose method has since been extended significantly (Dupont al., 2022; Schwarz & Teh, 2022; Schwarz et al., 2023). The current sfate-of-the-ans INR-based method, VC- INR (Schwarz et al., 2023), achieves impressive results across several data modalities, albeit at the cost of significantly higher complexity and still falling short of autoencoder- NTC methods on images. Our method, following COMBINER (Guo et al., 2023), differs from all of the above methods, as it uses REC to encode variational INRs, instead of quantization and entropy coding.
Linear weight reparameterizaation: Similar to our proposal in Section 3. Oktay et al. (2019) leam affine reparameterrizatin of the weights of large neural networks. They demonstrate that scalar quantization in the transformed space leads to significant gains in compression performance. However, since they performing one-shot model compression, their linear transformations have As slight abuse of notation, we use denote both the upsampling function and its parameters.

---

Published as a conference paper at ICLR 2024
COMBINER
COIN (Duponi 2001)
COIN++ (Dupont al., 2022)
RECOMBINER (ours)
(Gro al,, 2023)
MSCN (Schwarz 2022)
VC-INR
JPEG2000
BPG
[Schwarz
Belllndd 2014]
(JVFT, 2020)
VTM
Ballé (2018)
Cheng et (2020)
(He ot al., 2022)
ELIC
36
RGGH PSNR (dR)
38
33
3:43dB
34
30
aa 1.55
27
24
hitate (hpp)
(a) curve on CIFAR- (left) and Kodak (right). We provide full-resolution plots in Appendix F.
1.2
RECOMBINER
20
Puldansp (hullaos)
50
Feddoney (Do:
63
RECOMBINER Ssnn
H
GAIBINER teero gualtyy
PnC! (Co)
(SC
(Kaps)
thep) (c) RD curve on video.
(b) RD curve on audio.
(d) RD curve on protein data.
0.12 hpp
28.68 dB
Bitrate 11.17 bpa
RMSD 0.99
Bitrate 35.17 bpa RMSD 0.12 A
1.18 bpp
40.24 dB
Ground Truth
Decoded Structure
(e) Decoded videos and residuals.
(f) Decoded protein structure examples.
Figure 3: Quantitive evaluation and qualitative examples of RECOMBINER image, audio, video, and 3D protein structure. Kbps stands for kilobits per second, RMSD stands for Root Mean Square Deviation, and bpa stands for bits per atom. For all plots, codecs, dotted lines to denote VAE-based codecs, and dashed lines to denote classical codecs.
solid lines to denote INR-based
very few parameters as they need to transmit them alongside the quantized weights, limiting their ex- pressivity. On the other hand, RECOMBINER learns the linear transform during training after which it is fixed and shared between communicating parties, thus it does not cause any communication overhead. Therefore, linear transformation be significantly more expressive.
Positional encodings: Some recent works have demonstrated that learning positional features is beneficial for fitting INRs (Jiang et al., 2020; Kim et al., 2022; Müller et al., 2022; Ladune et al., 2023). Sharing a similar motivation, our method essentially incorporates implicit representations with explicit ones, forming hybrid INR framework (Chen et al., 2023).
5 EXPERIMENTAL RESULTS
In this section, evaluate RECOMBINER image, audio, video, and 3D protein structure data and demonstrate that it achieves strong performance across all modalities. We also perform extensive ablation studies the CIFAR- 10 and Kodak datasets which demonstrate RECOMBINER robustness and the effectiveness of each of our proposed solutions. For all experiments, we use a 4-layer, 32- hidden unit SIREN network (Sitzmann al., 2020) as the architecture unless otherwise stated, and a small 3-layer convolution network as the upsampling network é, as shown in Figure 6 in the appendix. See Appendix C for the detailed description of our experimental setup.

---

Published as a conference paper at ICLR 2024
(a) w/o positional encodings; bitrate 0.287 bpp: PSNR 25.62 dB. bitrate 0.316 bpp; PSNR 26.85 dB. bitrate 0.178 bpp: PSNR 25.05 dB.
(h) with positional encodings; (c) with positional encodings;
Figure 4: Comparison between kodim24 2 details compressed with and without learnable positional encodings. (a)(b) have similar bitrates and (a)(c) have similar PSNRs.
5.1 DATA COMPRESSION ACROSS MODALITIES
We evaluate RECOMBINER the CIFAR-10 (Krizhevsky et al., 2009) and Kodak (Kodak, 1993) image datasets, and show its rate-distortion (RD) performance in Figure 3a, and compare against recent INR and VAE-based methods, as well VTM (IVET, 2020)², BPG (Bellard, 2014) and JPEG2000. RECOMBINER displays remarkable performance on CIFAR-10, 10, especially at low bitrates, outperforming even VAE-based codecs. On Kodak, it outperforms most INR INR-based codecs and is competitive with the more complex VC-INR method of Schwarz et al. (2023). Finally, while RECOMBINER still falls behind VAE-based codecs, it significantly reduces the performance gap.
Image:
Audio: Following the experimental set-up of Guo et al. (2023), we evaluate our method on the LibriSpeech (Panayotov al., 2015) dataset. In Figure 3b, we depict RECOMBINER'S 's RD curve on the full test set, alongside the curves of VC-INR, COIN++, and MP3. We can see RECOMBINER outperforms both COIN++ and MP3 and matches with VC-INR. Since Guo al. (2023) only tested COMBINER on 24 test clips, we do not include COMBINER in this plot but put an extra comparison in Figure 13 in Appendix F, where we can also see that RECOMBINER clearly outperforms COMBINER.
Video: We evaluate RECOMBINER on UCF-101 action recognition dataset (Soomro et al., 2012), following Schwarz et al. (2023)'s experimental setup. However, as they do not report their train- test split and due to the time-consuming encoding process of approach, we only benchmark our method against H.264 and H.265 on 16 randomly selected video clips. Figure 3c shows RECOM- BINER achieves comparable performance the classic domain-specific codecs H.264 and H.265, especially at lower bitrates. However, there is still gap between our approach and H.264 and H.265 when they are configured to prioritize quality. Figure 3e shows non-cherry-picke video compressed with RECOMBINER at two different bitrates and its reconstruction errors.
3D Protein Structure: To further illustrate the applicability of our approach, we use it to compress the 3D coordinates of Ca atoms in protein fragments. We take domain-specine lossy codecs as baselines, including Foldcomp (Kim al., 2023). PDC (Zhang Pyle, 2023) and PIC (Staniscia & Yu, 2023). Surprisingly, as shown in Figure 3d, RECOMBINER'S 's performance is competitive with specificc codecs. Furthermore, it allows us to tune its rate-distortion performance, whereas the baselines only support a certain compression rate. Since the experimental resolution of 3D structures is typically between 1-3 Á (RCSB Protein Data Bank, 2000), RECOMBINER could help with reducing the increasing storage demand for protein structures without losing key information. Figure 3f shows non-cherry-pice piccked examples compressed with our method.
highly
5.2 EFFECTIVENESS OF OUR SOLUTIONS, ABLATION STUDIES AND RUNTIME ANALYSIS
This section showcases RECOMBINER'S 's robustness to model size and the effectiveness of each com- ponent. Appendix D.1 provides additional visualizations for deeper understanding of our methods.
Positional encodings facilitate local deviations: Figure 4 compares images obtained by RECOM- BINER with and without positional encodings matching bitrates and PSNRs. As we can see, po- sitional encodings preserve intricate details in fine-textured regions while preventing noisy artifacts in other regions of the patches, making RECOMBINER 's reconstructions more visually pleasing.
Pntppa //vogtt hht wtw.s/..htt..trtrtoe www.nhafer..rccmmm de/ / WWw.otttaaaaa..o...ooo httpt/aawaeeeeeeeeeeeee ott. t_yypeeaggs
8

---

Published as a conference paper at ICLR 2024
50
54
54
37
COMBINER
31
30
LRR
LR
HIM RP
KP
lidken uniss
26
EJ
REP
hidden
12
25
20
13
hidden
22
PR
24
0.2
0.00
0.66
bizzac thpp?
bimii (epp)
brinatc (opp)
(a) Rubusiness la model sizee
(b) Ablation study on CIFAR-10
(c) Ablation study on Kodak.
Figure 5: (a) RD performances of COMBINER and RECOMBINER with different numbers hidden units. (b)()) Ablation studies on CIFAR-IO and Kodak. LR: linear reparameterization: PE: posi- tional encodings; HM: hicrarchical model; RP: random permutation across patches. We describe the details of experimental settings for ablation studies in Appendix C.3.
RECOMBINER is more robust to model size: Using the same INR architecture, Figure 5a shows COMBINER and RECOMBINER's RD curves as we vary the number of hidden units. RECOMBINER displays minimal performance variation and also consistently outperforms COMBINER. Based on Figure in Appendix D, this phenomenon is likely due to RECOMBINER'S ' linear weight reparame- terization allowing it to more flexibly prune its weight representations.
Ablation study: In Figures 5b and 5c, we ablate our linear reparameterization, positional encodings, hierarchical model, and permutation strategy CIFAR-10 10 and Kodak, with five key takeaways:
1. Linear weight reparameterization consistently improves performance both datasets, yiclding up to 4dB gain on CIFAR- at high bitrates and over 0.5 dB gain on Kodak in PSNR.
Learnable positional encodings provide more substantial advantages at lower bitrates. On CIFAR-10, the encodings contribute up 0.5 dB gain when the bitrate falls below bpp. On Kodak, the encodings provide noteworthy gains of 2 dB at low bitrates and dB at high bitrates.
3. Surprisingly, the hierarchical model without positional encodings can degrade performance. We hypothesize that this is because directly applying the hierarchical model poses challenges in optimizing Equation (1). A potential solution is to warm up the rate penalty 3 level by level akin to what is done in hicrarchical VAEs (Sønderby et al., 2016), which we leave for further work.
However, positional encodings appear to consistently alleviate this optimization difficulty, yield- ing 0.5 dB gain when used with hierarchical models.
Our proposed permutation strategy provides significant gains of 0.5 dB low bitrates and more than 1.5 dB at higher bitrates.
We list RECOMBINER's encoding and decoding times in Appendix D.5. Unfor- tunately, our approach exhibits long encoding time, similar to COMBINER. However, our decoding process is still remarkably fast, matching the speed of COIN and COMBINER, even on CPUs.
Runtime Analysis:
6 CONCLUSIONS AND LIMITATIONS
In this paper, we propose RECOMBINER, new codec based on several non-trivial extensions to COMBINER, encompassing the linear reparameterization for the network weights, learnable posi- tional encodings, and expressive hierarchical Bayesian models for high-resolution signals. Experi- ments demonstrate that our proposed method sets new state-of- of-the-art on low-resolution images at low bitrates, and consistently delivers strong results across other data modalities.
A major limitation of our work is the encoding time complexity and tackling it should be of primary concern in future work. possible avenue for solving this issue reduce the number of parame- ters to optimize over and switch from inference over weights to modulations using, e.g. FiLM layers (Perez et al., 2018), as is done in other INR-based works. A second limitation that while com- pressing with patches enables parallelization and higher robustness, it is suboptimal as it leads to block artifacts, as can be seen in Figure 4. Third, as Guo et al. (2023) demonstrate, the approximate samples given by A coding significantly impact the methods performance, e.g. by requiring more fine-tuning. An interesting question is whether an exact REC algorithm could be adapted to solve this issue, such as the recently developed greedy Poisson rejection sampler (Flamich, 2023).

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
The authors would like to thank Runsen Feng for helping us ensure that our baseline for our exper- iments video compression is correctly set up. GF acknowledges funding from DeepMind. ZG acknowledges funding from the Outstanding PhD Student Program at the University of Science and Technology of China.
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2017.
Johannes Ballé, Philip Chou, David Minnen, Saurabh Singh, Nick Johnston, Eirikur Agustsson, Sung Jin Hwang, and George Toderici. Nonlinear transform coding. IEEE Journal of Selected Topics in Signal Processing, 2020.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In International Conference Learning Represen- tations, 2018.
Fabrice Bellard. BPG image format. 09-27.
2014. Accessed: 2023-
Charles Blundell, Julien Corneiise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, 2015.
Hao Chen, Matthew Gwilliam, Ser-Nam Lim, and Abhinav Shrivastava. Hnerf: A hybrid neural representation for videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
Emilien Dupont, Adam Golinski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Com- pression with implicit neural representations. Neural Compression: From Information Theory to Applications-Workshppp @orkshop@@ ICLR 2021, 2021.
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Golinski, Y Whye Tch, and Arnaud Doucet. Coin++: Neural compression across modalities. Transactions on Machine Learning Research, 2022.
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In International conference on machine learning. 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In International conference machine learning. 2017.
Gergely Flamich. Greedy Poisson rejection sampling. In Advances in Neural Information Process- ing Systems, 2023.
Gergely Flamich, Marton Havasi, and José Miguel Hernández-Looato. Compressing images by encoding their latent representations with relative entropy coding. In Advances in Neural Infor- mation Processing Systems, 2020.
Gergely Flamich, Stratis Markou, and José Miguel Hernández-Lobato. Fast relative entropy coding with * coding, In International Conference on Machine Learning, 2022.
Zongyu Guo, Zhizheng Zhang, Runsen Feng, and Zhibo Chen. Causal contextual prediction for learned image compression. IEEE Transactions on Circuits and Systems for Video Technology, 2021.
10

---

Published as a conference paper at ICLR 2024
Zongyu Guo, Gergely Flamich, Jiajun He, Zhibo Chen, and José Miguel Hemándezz-Lbatoo. Com- pression with Bayesian implicit neural representations. In Advances in Neural Information Pro- cessing Systems, 2023.
Marton Havasi, Robert Peharz, and José Miguel Herándeez-Lobato. Minimal random code learning: Getting bits back from compressed model parameters. In International Conference on Learning Representations, 2018
Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. Blic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are Bayesian neural network posteriors really like? In International conference on machine learning, 2021.
Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
JVET. VVC offical test model. https:// hhi. www.wfffruu..ccccmmmmm de, 2020. 2024-03-05.
Accessed:
Hyunbin Kim, Milot Mirdita, and Martin Steinegger. Foldcomp: a library and format for compress- ing and indexing large protein structure sets. Bioinformatics, 2023.
Subin Kim, Sihyun Yu, Jacho Lee, and Jinwoo Shin. Scalable neural video representations with learnable positional features. In Advances in Neural Information Processing Systems, 2022.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. In Advances in Neural Information Processing Systems, 2015.
Eastman Kodak. Kodak Lossless True Color Image Suite (PhotoCD PCD0992). http://rOk. us/ uis/graphice iss /kokkk/okk 1993.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Théo Ladune, Pierrick Philippe, Félix Henry, Gordon Clare, and Thomas Leguay. Cool-chic: Coordinate-based low complexity hierarchical image codec. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Jinming Liu, Heming Sun, and Jiro Katto. Learned image compression with mixed transformer- cnn architectures. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, 2023.
Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity gen- erative image compression. In Advances in Neural Information Processing Systems, 2020.
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. In Advances in neural information processing systems, 2018.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. ACM Transactions Graphics, 2022.
Deniz Oktay, Johannes Ballé, Saurabh Singh, and Abhinav Shrivastava. Scalable model compression by entropy penalized reparameterization. In International Conference on Learning Representa- fions, 2019.
11

---

Published as a conference paper at ICLR 2024
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI conference on artificial intelligence, 2018.
RCSB Protein Data Bank. PDB Statistics: PDB data distribution by resolution. https: // www. resb. http:///ttatttttttttottttoooeeee
2000. Accessed: 2023-09-27.
Jonathan Richard Schwarz and Yee Whye Tch. Meta-learning sparse compression networks. Trans- actions Machine Learning Research, 2022.
Jonathan Richard Schwarz, Jihoon Tack, Yee Whye Teh, Jaeho Lee, and Jinwoo Shin. Modality- agnostic variational compression of implicit neural representations. In International conference on machine learning. 2023.
Vincent Sitzmann, Julien N. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wet- zstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems, 2020.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kane Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in neural information processing systems, 2016.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCFI01: A dataset of 101 human actions classes from videos in the wild. arXiv preprint
2012.
Luke Staniscia and Yun William Yu. Image-centric compression of protein structures improves space savings, BMC Bioinformatics, 2023.
Kenneth O Stanley. Compositional pattern producing networks: novel abstraction of development.
Genetic programming and evolvable machines, 2007.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal. Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Pro- cessing Systems, 2020.
Suramya Tomar.
Converting video formats with FFmpeg. Linux Journal, 2006.
Brian Trippe and Richard Turne. Overpruning in variational Bayesian neural networks. In Advances in Approximate Bayesian Inference workshop at NIPS 2017, 2017.
Chengxin Zhang and Anna Marie Pyle. PDC: highly compact file format to store protein 3D coordinates. Database (Oxford), 2023.
Yinhao 7hu, Yang Yang, and Taco Cohen. Transformer-based transform coding. In International Conference on Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2024
A
NOTATIONS
We summarize the notations used in this paper in Table 1:
Notation
Name
8
rate penalty hyperparameter in Equation (1)
C
coding budget
TC
step size for adjusting B
threshold parameter to stabilize training when adjusting / weights in INR ith coordinate ith signal value
X
y
RECOMBINER'S upsampled positional encodings at coordinate X,
hw
RECOMBINER'S latent weights
hg
RECOMBINER'S s latent positional encodings
h
(x)
latent INR weights for æth patch (lowest level of the hierarchical model)
h
latent positional encodings for 7th patch (lowest level of the hierarchical model)
gth representation in the second level of the hierarchical model third level representations of the hierarchical model mean of the Gaussian posterior mean of the Gaussian prior
hw
μ
P
diagonal of the covariance matrix of the Gaussian posterior diagonal of the covariance matrix of the Gaussian prior
A
RECOMBINER'S 's linear transform on INR weights
H
matrix stacking representations in the éth level defined in Equation (5)
H
matrix for representations in the éth level after permutation signal data point (as a dataset with coordinate-value pairs)
D
Sn
of all permutations on n elements Fourier embedding to coordinates
7()
((),.())
permutation
upsampling network for positional encodings INR with weights w
90. w)
Table 1: Notations.
13

---

Published as a conference paper at ICLR 2024
B RECOMBINER'S TRAINING ALGORITHMS
We describe the algorithm to train RECOMBINER in Algorithm 1.
Algorithm Training RECOMBINER: the prior, the linear transform A and upsampling network ø
Require: Training data 1,.,. : desired bitrate C. Initialize: Qh.m = N (um; diag (Pm)) for every training instance Dm-
Initialize: Ph N (m,, diag (o))
Initialize: A, Φ.
repeat until convergence
Step I: Optimize posteriors, linear reparameterization matrix, and upsampling network
(mm Pm)) m=1 A, a < arg !n 2m= L(Dm: Ih.m, Ph. A, Φ. (3]
Optimize by Equation (6)
Step 2: Update prior
1 2 [Vmm 11)2 2))++ + pm
" M
2m=11
Mms
M
Update by Equation (7)
# Step 3: Update B
5 = 1 S.M
M 2mm1 DKL (hnm
Calculate the average training KI.
if 5 C then
- x (1 11TT0 TC)
Increase if budget exceeded
end if
8 < C fc then 3/(1 TO)
Decrease / if budget is not fully occupied
end if end repeat
Return: Ph = N (M, diag (a)), A, Φ.
C SUPPLEMENTARY EXPERIMENTAL DETAILS
C.I DATASETS AND MORE DETAILS ON EXPERIMENTS
In this section, we describe the dataset and experimental settings. We depict the upsampling network we used Figure and summarize the hyperparameters for each modality in Table 2. Besides, we present details for the baselines in Appendix C.2.
Note, that as the proposed linear reparameterizztiion yields full-covariance Gaussian posterior over the weights in the INR, the local reparameterization trick (Kingma et al., 2015) is not applicable in RECOMBINER. Therefore, in the above experiments, when inferring the posteriors of a test signal, employ a Monte Carlo estimator with samples estimate the expectation in 33ELBO in Equa- tion (1). While during the training stage, we still use sample. In Appendix D.3, we provide an analysis of the sample size's influence. is worth noting that using just I sample during inferring does not significantly deteriorate performance, and therefore we have the flexibility to reduce the sample size when prioritizing encoding time, with marginal performance impact.
CIFAR-10: CIFAR-10 is set of low-resolution images with a size of 32 x 32. It has training set of 50,000 images and training set for the training stage and evaluate RD performance all test images. we use SIREN network (Sizzmann et al., 2020) with layers and hidden units as the INR architecture.
of 10,000 images. We randomly select 15,000 images from the
Kodak: Kodak dataset is commonly used image compression benchmark, containing 24 images with resolutions of either 768 x 512 or 512 x 768. our experiments, we split each images into 96 patches with size 64 64. Lacking standard training set, we randomly select and crop 83 images with the same size (splitting into 7,968 patches) from the DIV2K dataset (Agustsson & Timofte, 2017) as the training set. We compress each Kodak image in 64 64 patches. For each patch, use the same INR setup as that for CIFAR-10, 10, ie.. SIREN network (Sitzmann et al., 2020) with layers and 32 hidden units. Besides, apply three-level hierarchical Bayesian model to Kodak
14

---

Published as a conference paper at ICLR 2024
The lowest level has 96 patches. Every 16 (4 4) patches grouped together in the second level, and in total there are groups. The highest level consists of global representation for the entire image.
patches.
Audio: LibriSpeech (Panayotov et al., 2015) is a speech dataset recorded at a 16kHz sampling rate. We follow the experiment settings by Guo al. (2023), taking the first 3 seconds of every recording, corresponding to 48,000 audio samples. We compress each audio clip with 60 patches, each of which has 800 audio samples. For each patch, we use the same INR architecture as CIFAR- 10 except the output of the network has only one dimension. We train RECOMBINEA on 197 training instances (corresponding
11,820 patches) and evaluate on the test set split by Guo et al. (2023), consisting of 24 instances. We also apply a three-level hierarchical model. The lowest level consists of 60 patches. Every 4 patches are grouped together in the second level, and total there are 60/4 = 16 groups. The highest level consists of global representation for the entire signal.
dataset of human actions. It consists of 101 action classes, over 13k clips, and 27 hours of video data. We follow Schwarz et al. (2023) center-cropping each video clip to 240 x 240 x 24 and then resizing them to 128 x 128 x 24. Then we compress each clip with 16x 16x 24 patches. We train RECOMBINER on 75 video clips (4,800 patches) and evaluate it on 16 randomly selected clips. For each patch, we still use the INR with layers and 32 hidden units. We also apply the three-level hierarchical model. The lowest level consists of 64 patches. Every 16 4 x 4 patches are grouped together in the second level, and in total, there are 4 groups. The highest level consists of a global representation for the entire clip. 3D Protein structure: We evaluate RECOMBINER on the Saccharomyces cerevisiae proteome from the AlphaFold DB v4 To standardize the dataset, for each protein, we take the Ca atom of the first 96 residues (i.e., amino acids) as the target data to be compressed. The input coordinates are the indices of the Ca atoms (varying between 1-96, and normalized between 0- and the outputs of INRs are their corresponding 3D coordinates. We randomly select 1,000 structures as the test set and others as the training set. We still use the same INR architecture as CIFAR- 10, i.e., SIREN network with layers and 32 hidden units in each layer. We use the standard MSE as the distortion measure. Note that our method can also be extended to take the fact that the 3D structure is rotation and translation invariant into account by using different losses.
Video: UCF-101 (Soomro et al., 2012)
upsample 4 x
conv 128, 64,5
upsample 2 x
conv 64, 64,3
upsample 2 x
conv 64, 16,3
LeakyReLU
LeakyReLU
Figure 6: Architecture of the up-sampling network ó for learnable positional encodings. The num- bers in the convolution layer represent the number of input channels, the number of output channels, and kernel size respectively. sane padding mode used in all convolution layers. The kernel dimension depends on the modality, for instances, we use kernels with sizes of 5, 3, 3 for audio and proteins, kernels with sizes of 5 x 5, 3, 3 x 3 for images, and kernels with sizes of 5 x 5
5,
3 x 3 x 3, 3 x 3 x 3 for video.
C.2 BASELINE SETTINGS The baseline performances, including JPEG2000, BPG, COIN, COIN+ Ballé et al. (2018) and Cheng et al. (2020) on CIFAR-10 and Kodak, and MP3 and COIN+ the full test set of Lib- riSpeech, are taken from the COIN+4's GitHub repo*. Statistics for VC-INR and MSCN are pro-
Phttps: htttt/tt.bbb/ ebi ac. uk YEAST_ tarr .ar /https:/ / thub, com Kmi wwwm//aiiii/mappet//eeeeooo /en/uupent/coingg
_5590990022111559292___
15

---

Published as a conference paper at ICLR 2024
Image
Audio
Video
Protein
Cifar-10
Kodak Putching
patch not
/
K
800
hicrarchical model levels
3
namber of patches (lowest level)
96
60
number groups patches (middlec number of groups of groups (highest level)
16
Positional Encodings
latent positional encoding shape latent positional enerding param number upsampled positional encoding shupe
1
128 4 44
128
128
1-1
128
512
2560
6400
128
768
INR Architecture
layers
hidden units
32
Fourier emhedddings dimension
16
16
16
2 not integer)
16
output dimension
1
number of panameters
3267
3267
3201
3331
3201
Training Stage
training size epochs optimizer
15000
83 (7968 patches)
197 (11820 patches)
75 (4800 patches)
4691
$50
Adam (I==0.0002)
sample size to estimate 3-ELBO
gradicut iteration
updating prior
the first gradicnt iteration initial posterior variance initial posterior mean
200 10
SIREN initialization
initial All values
A 26(- 1/s); a degloai where
and doos the input output dimcusion layer 0.0S bpp
0.3 hpp
0.5 khgs
0,3 hpp
0.3 bya
Adaptively adjusted. Initial value 10
Posterior Inferring und Compression Stuge
gradicat desecat itcration
30000
optimizer
(1r=0.0002)
sample size to estimate 3-ELBO
(1819, 3187, 4373,7770.
blocks per signal (total number of blocks)
[1066,
[2827_ 5992, 14858, 290733
[67.
123:214,281)
12004, 23898)
4146, 8182)
503, 637)
per block
[17, 30.
[15, 31.
(34. 71. 198,
blocks the lowest level (patch)
41, 73,
233)
122)
[17, 34,
55.
[109, 284, 427, 561)
blooks the middle level
52,
145, 211}
[85,103.
blocks the highest level
150,
(41,64. 112)
[215.315 312, 478, 6533
190, 264]
Table Hyperparameters for images, audio, video, and protein structure compression.
vided by the authors in the paper. We also include comparison of RECOMBINER and COMBINER on 24 test audio clips since the authors of COMBINER did not test on the full test set. For this com- parison, the performances of COMBINER and MP3 on 24 test audio clips are provided by the authors of COMBINER.
Below, we describe details about the baseline of the video and protein structure compression.
C.2.1 VIDEO BASELINES
Video compression baselines are implemented by ffmpeq (Tomar, 2006), with the following com- mands.
H.264 (best speed):
fmpeg. exe -i INPUT.avi -c.v 1ibx264 -preset ultrafast. -crf $CRF OUTPUT.mke mkv
H.265 (best speed):
ffmpeg. exe -i INPUT.avi -c:V 1ibx265 -preset ultrafast -orf $CRF OUTPUT mkv
16

---

Published as a conference paper at ICLR 2024
H.264 (best quality):
ffmpeg. exe -i INPUT.avi -CIV 1ibx264 -preset veryslow -crf $CRF OUTPUT mkk
H.265 (best quality):
ffmpeg. exe -i INPUT. -c.v 1ibx265 -preset veryslow -crf $CRF OUTPUT m
The argument $CRF varies in 15 20 25 30 35 40.
C.2.2 PROTEIN BASELINES
Softwares implementing PIC, PDC and Foldcomp are available at https: /ttp::///itthhh com/ www.sukttiiiii//Pe fttps://githin com/kad maad-ecoii/ee i/dc and https:// github. ctm/ stei cegeeeteeeeeegegelllll..o.
PIC first employs a lossy mapping, converting the 3D coordinates of atoms to an image, and then lossless compresses the image in PNG format. We use the PNG image size to calculate the bitrate.
As for PDC and Foldcomp, since they directly operate on PDB files containing other information like the headers, sequences, B factor, etc., we cannot use the file size directly. Therefore, we use their theoretical bitrates as our baseline. Below we present how we calculate their theoretical bitrates.
PDC uses three 4-byte integers to save the coordinates of the first Co atom, and three 1-byte integers for coordinate differences of all remaining Ca atoms. Therefore, in theory, for a 96-residue length protein, each Ca atom is assigned with (8 x 3 x 95 + 4 8 x 3 1)/96 bits.
Foldcomp compresses the quantized dihedral/bond angles for each residue. Every residue needs bits. Besides, Foldcomp saves uncompressed coordinates for every 25 residues anchors, which requires 36 bytes. Therefore, the thcoretical number bits assigned to each Ca given by (36 x 8 + 59 255/25. However, since Foldcomp is designed to encode all backbone atoms (C, N, Co) instead of merely Co., is unfair to compare in this way. We thus also report its performance on all backbone atoms for reference.
C.3 ABLATION STUDY SETTINGS
In this section, we describe the details settings for ablation studies in Figures 5b and 5c.
Experiments without Linear Reparameterization: We simply w hw without the linear matrix A. Besides, since in this case, w follows mean-field Gaussian, we use the local reparameter- ization trick with sample to reduce the variance during both training and inferring.
Experiments without Positional Encodings: Recall that the inputs of INRS in RECOMBINER is
the concatenation of Fourier transformed coordinates 7x.) and the upsampled positional encodings at the corresponding position only input the Fourier transformed coordinates to the INR. To keep the INR size consistent, we also increase the dimension of the Fourier transformation, that dim( (x())) dim(7(xx)) dim(z.). Also, we no longer need to train the upsampling network Ø.
d(hz)x,- In the experiments without positional encodings, we
Experiments without Hierarchical Model:
We assume all patch-INRs are independent and simply assign independent mean-field Gaussian priors and posteriors over (x(=) for each patch.
Experiments without Random Permutation across patches: Recall in RECOMBINER, for each level in the hierarchical model, we stack the representations together into matrix, where each row is one representation. We then (a) apply the same permutation over all rows. This is the same as COMBINER and is to ensure KL is distributed uniformly across the entire representation for each patch. Then (b) for each column, uniformly across patches. In the ablation study, we do not only apply the permutation in (b) but still perform the permutation in (a).
apply its permutation to encourage KL be distributed
17

---

Published as a conference paper at ICLR 2024
D SUPPLEMENTARY EXPERIMENTS AND RESULTS
D.1 METHODS VISUALIZATION
(a) Visualization of 4 channels in the upsampled positional encodings for kodim03 0.488 bpp. Patches stitched together for a clearer visualization.
(b) Visualization of the information contained encoded hw for kodim0' 3 0.488 bpp. Patches stitched together.
0.04
0.02
Dimendens
-0.02
0.02
0.04
140
140
60
100
ponding
Dimersions panding
(c) Visualization of A 0.074 bpp.
Visualization of A 0.972 bpp.
Figure 7: Visualizations.
In this section, bring insights into methods by visualizations. Recall that each signal is rep- resented by hz and h together in RECOMBINER. We visualize the information contained in each of them. Besides, visualize the linear transform A understand how it improves performances.
Positional encodings: We take kodim03 3 at 0.488 bpp as an example, and visualize channels of its upsampled positional encodings (h,) in Fig 7a. Interestingly, before fed into the INR, the positional
18

---

Published as a conference paper at ICLR 2024
encodings already present a pattern of the image. This is an indication of how the learnable positional encodings help with the fitting. When the target signal is intricate, and there is strict bitrate constraint, the INR capacity is insufficient for learning the complex mapping from coordinates to signal values directly. On the other hand, when combined with positional encodings, INR simply needs to extract, combine, and enhance this information, instead of "creating" information from scratch. This aligns with the findings of the ablation study, which indicate that learnable positional encodings have a more significant impact on CIFAR-10 at low bitrates and the Kodak dataset, but small effect on CIFAR- 10 at high bitrates.
Information contained in hw: To visualize the information contained in hws we also take kodim03 at 0.488 bpp an example. We reconstruct the image using hw for this image but mask out hz by the prior mean. The image reconstructed in this way is shown in Fig 7b.
From the figure, we can clearly see hw mostly captures the color specific to each patch, in compar- ison to the positional encodings containing information more about edges and shapes. Moreover, interestingly, we can see patches close to each other share similar patterns, indicating the redun- dancy between patches. This explains why employing the hierarchical model provides substantial gains, especially when applying it together with positional encodings.
To interpret how the linear reparameterization works, we take the Kodak dataset as an example, and visualize A for the second layer (i.e., A(2)) at 0.074 and 0.972 bpp in Fig 7c and 7d. Note that this layer has 32 hidden units and thus A has shape of 1056 x 1056. We only take a subset of 150 x 150 in order to have clearer visualization. Recall w = hw A, and thus rows correspond to dimensions in hw and columns correspond dimensions in w.
Linear Transform A:
It can be seen that when the bitrate is high, many rows in A are active, enabling flexible model. Conversely, at lower bitrates, many rows become 0, effectively pruning out corresponding dimen- sions. This explains clearly how A contributes to improve the performance: first, A greatly pro- motes parameter sharing. For instance, at low bitrates, merely 10 percent of the parameters get involved in constructing the entire network. Second, the pruning in hw is more efficient than that in w directly. The predecessor of RECOMBINER, i.e., COMBINER, utilizes standard Bayesian neural networks. It controls its bitrates by pruning activating the hidden units. When a unit is pruned, the entire column in the weight matrix will be pruned out (Trippe & Turner, 2017). In other words, in COMBINER, the pruning in w is always conducted in chunks, which highly limits the flexibility of the network. On the contrary, in our approach, the linear reparameterization enables a direct pruning or activating of each dimension in hw individually, ensuring the flexibility of INR while effectively managing the rate.
Another interesting observation is the matrix A essentially learns low-rank pattern without manual tuning. This is in comparison with VC-INR (Schwarz et al., 2023) where the low-rank pattern is explicitly enforced by manually setting the LoRA-style (Hu al., 2021) modulation.
D.2 EFFECTIVENESS OF RANDOM PERMUTATION
In this section, we provide an example illustrating the effectiveness of random permutation across patches. Specifically, we encode kodim23 0.074 bpp. both with and without random permu- tation, and visualize their residual images in Figure 8. We can see that, without permutation, the residuals for complex patches are significantly larger than simpler patches. This due to the fact that, in RECOMBINER, the bits allocated to each patch are merely determined by the number of blocks, which is shared across all the patches. On the other hand, after the permutation, we can see more balanced distribution of residuals across patches: complex patches achieve better recon- structions, whereas simple patches' performances only degrade marginally. This is because, after the permutation across patches, each block can have different patches' representations, enabling an adaptive allocation of bits across patches. Overall, random permutation yields a 1. 1.00 dB gain on this image.
D.3 INFLUENCE OF SAMPLE SIZE
As discussed in Appendix C.I, in our experiments, we use samples to estimate the expectation in the B-ELBO in Equation (1), when inferring the posterior of test datum. Here, we provide the RD
19

---

Published as a conference paper at ICLR 2024
(a) with permutation, PSNR 29.16 dB
(b) without permutation, PSNR 28.16 dB
Figure 8: Comparison of residuals of kodim23 0.074 bpp, with and without random permutation across patches.
curve using 1, 5 and 10 samples, on 500 randomly selected Cifar-10 10 test images and kodim05 as examples, illustrate the influence of different choices of sample sizes.
As shown Figure 9, the sample size mainly impacts the performance at high bitrates. Besides, further increasing the sample size to 10 only brings minor improvement. Therefore, choose 5 samples our experiments to balance between encoding time and performance. It is also worth noting that using just sample does significantly reduce the performance. Therefore. have the flcxibility of choosing smaller sample sizes when prioritizing encoding time, with minor perfor- mance impacts.
40.0
Sample size 10
Sample size Sample size 5
Sample size =
37.5
38
35.0
PSNR (dB)
PSNR (dB)
32.5
38.5
30.0
39.5
27.5
0.39 dB
32
0.28 dB
25.0
39.0
22.5
4.3
4.4 4.5
0.95
1,00
0.4
0.6
0.8
1.0
bitrate (kbps)
bitrate (kbps)
(a) CIFAR-10
(b) kodim03
Figure 9: Influence of Sample size. (a) RD curve evaluated 500 randomly selected CIFAR-10
images. (b) RD curve evaluated on kodim03.
D.4 ROBUSTNESS DURING TRAINING
Different from previous INR-based codecs based on MAML (Finn et al., 2017) including C (Dupont al., 2022), MSCN (Schwarz & Teh, 2022) and VC-INR (Schwarz et al., 2023), our proposed RECOMBINER does not require nested gradient descent and thus features higher stability during training period.
To demonstrate this advantage, present a visualization of the average 3-ELBO during training CIFAR- 10 across three bitrates in Figure 10. We can see that the training curves exhibit an initial dip followed by consistent increase. The dip at the beginning is result of our adjustment of 3 during training (Step 3 in Algorithm 1). Importantly, this adjustment does not impact training robustness; and that is quickly adjusted, and the training proceeds smoothly.
20

---

Published as a conference paper at ICLR 2024
-50
-2000
1000
4000
1.9bpp
1.9bpp
-50
0.9bpp
0.3bpp
0.3bpp
200
400
500
200
400
Training tooaaions
Training Iterations
(a) 3-ELBO w.r.t training steps.
(b) Zoom-in plot.
Figure 10: Average training 3-ELBO on Cifar-10 at three different bitrates. The initial dip is because we also adjust 3 during training ensure the coding budget (Step 3 in Algorithm 1). We can see the initial quickly adjusts in the first several steps, and then the training proceeds smoothly.
D.5 CODING TIME
In this section, we provide details regarding the encoding and decoding time of RECOMBINER, The encoding speed is measured on a single NVIDIA A100-SXM-80GB GPU. On CIFAR-IO and protein structures, we compress signals in batch, with a batch size of 500 images and 1,000 structures, respectively. On Kodak, audio, and video datasets, compress each signal separately. We should note that the batch size does not influence the results. Posteriors of signals within one batch are optimized in parallel, and their gradients not crossed. The decoding speed is measured per signal on CPU.
Similar COMBINER, our approach features high encoding time complexity. However, the decoding process remarkably fast, even on CPU, matching the speed of COIN and COMBINER. Note that the decoding time listed here encompasses the retrieval of samples for each block. In practical applications, this process can be implemented and parallelized using lower-level languages such C++ C, which can lead further acceleration of execution.
Encoding Time (GPU, 500 instances)
Decoding Time (CPU, per instance)
Bitrate
0.297 bpp
-63 min
0.00386 0.00429
0.719 bpp
665 min
0.938 hpp
~68 min
0.00461
1.531 bpp
~72 min
0.00514
1.922 bpp
755 min
0.00581
3.344 bpp
87 min
0,00776
4.391 bpp
99 min
0.01050
Table 3: Coding time for CIFAR-10.
Encoding Time
Decoding Time (CPU, per instance)
Bitrate
(GPU, per instance, 96 patches)
0,074 bpp
59 min
0.25848 s
0.130 bpp
66 min
0.29117 s
0.178 bpp
667 min
0.30875 s
0.316 bpp
72 min
0.29690 s
0,488 bpp
80 min
0.34237 s
0.972 bpp
99 min
0.41861 s
Table 4: Coding time for Kodak.
21

---

Published as a conference paper at ICLR 2024
Encoding Time
Decoding Time (CPU, per instance)
Bitrate
(GPU, per instance, 50 patches)
5.69 kbps
18 min
0.05564
10.66 kbps
21 min
0.06003
22.11 kbps
22 min
0.06166
43.64 kbps
22 min
0.07350
Table 5: Coding time for Audio.
Encoding Time
Decoding Time (CPU, per instance)
Bitrate
(GPU, per instance, 64 patches)
0,115 bpp
49 min
0.31936 s
0.244 bpp
62 min
0.33416
0.605 bpp
78 min
0.33448 s
1.183 bpp
~ 102 min
0.35665 s
Table 6: Coding time for Video.
Encoding Time (GPU, 1000 instance)
Decoding Time (CPU, per instance)
Bitrate
11.17 bpa
72 min
0.00704
35.17 bpa
~ 123 min
0.00948
60.67 bpa
175 min
0.01429
83.83 bpa
222 min
0.01778
106.17 bpa
~274 min
0.02014
Table 7: Coding time for Protein.
E THINGS WE TRIED THAT DID NOT WORK
in RECOMBINER, apply linear reparameterization INR weights, which transfers the weights linearly into a transformed space. Perhaps a natural extension is to apply more complex transformations, e.g., neural networks, flows. We experimented with this idea. but it did not provide gains over the linear transformation.
in RECOMBINER, we propose a hierarchical Bayesian model, equivalent to assigning erarchical hyper-priors and inferring the hierarchical posteriors the means of the INR weights. A natural extension can be assigning hyper-priors/postrrioss to both means and variances. But did not find any gain by this.
in RECOMBINER, the hierarchical Bayesian model is only applied to the latent INR weights hw- is natural to apply the same hierarchical structure to the latent positional encodings hg. However, found it does provide visible gain.
MORE RD CURVES
Here, show the full-resolution RD curves for image compression in Figures 11 and 12. Besides, we also provide a further comparison between RECOMBINER with COMBINER on 24 test audio clips from LibriSpeech in Figure 13.
22

---

Published as a conference paper at ICLR 2024
40
3,43dB
35
RGB PSNR (dB)
30
25
20
1
2
3
4
5
6
bitate (bpp)
COMBINER
COIN
RECOMBINER (ours)
(Guo et al., 2023)
(Dupont et al., 2021)
COIN+ (Dupont et al., 2022)
MSCN (Schwarz & Teh, 2022)
VC-INR
(Schwarz et al., 2023)
BPG
JPEG2000
VTM
(Bellard, 2014)
Ballé et al. (2018)
Cheng et al. (2020)
Figure 11: RD curve CIFAR-10.
23

---

Published as a conference paper at ICLR 2024
42
39
36
33
RGB PSNR (dB)
30
0..9dB
27
24
21
0.5
1
1.5
2
2.5
3.5
bitate (bpp)
COMBINER (Guo et al., 2023)
COIN
RECOMBINER (ours)
(Dupont et al., 2021)
COIN+ (Dupont et al., 2022)
MSCN (Schwarz Tch, 2022)
VC-INR
(Schwarz al., 2023)
BPG
JPEG2000
VTM
(Bellard, 2014)
ELIC (He et ul., 2022)
Ballé al. (2018)
Cheng (2020)
Figure 12: RD curve Kodak.
24

---

Published as a conference paper at ICLR 2024
60
55
50
PSNR (dB)
45
40
RECOMBINER (ours)
35
COMBINER
MP3
30
0
10
20
30
40
50
bitrate (kbps)
Figure 13: RD curve of MP3, COMBINER and RECOMBINER on 24 test audio clips from LibriSpeech test set.
G RD VALUES
CIFAR-10:
rate [0.297, 0. 719, 0. 0.938, 1.531, 1.922, 3.344, 4.391]
PSNR = [23.592, 27.222, 28.505, 30.911, 32.168, 35. 73,, 38. 38.139]
Kodak:
rate = (0.074, 0.130, 0.178, 0.316, 0.488, 0. 0.972, 1.567, 3.320]
PSNR (26.158, 27.653, 2945 30.4 439, 31. 953, 34.54 540, 36.547, 40. 426]
Audio:
On full test set:
rate [5.685) 685, 10.661, 22.112, 43. 637]
PSNR = (42.612, 47 47.101, 196, 58.199
On 24 test examples (to compare with COMBINER):
rate [5.168, 10.805, 22.112, 43. 637]
PSNR 782.789, 47.106, 52.206, 58.322 (87]
Video:
rate [0.115, 0.244, 0. 605, 1.183]
PSNR = [28.722, 31.494, 35. 717, 39.171]
Protein:
rate = (11.17, 35.17, 50, 83.83, 106.17) 17]
RMSD
[0.9242, 0.1388, 0.0709, 0.0506, 0. 0436]
25

---

Published as a conference paper at ICLR 2024
H
MORE DECODED EXAMPLES
H.
CIFAR-10
Ground Truth
Bitrate 0.297bpp PSNR 21.13
Bitrate 4.391 bpp PSNR 35.18
Truth
Bitrate 0.297bpp
bpp
Ground Truth
Bitrate 0.297bpp 24.38
Bitrate 4.391 bpp PSNR 39.08
Figure 14: Decoded CIFAR-10 images and residuals.
H.2
KODAK
Ground Truth
Bitrate 0.074 hpp PSNR 23.11 dB
Bitrate 0.316 bpp PSNR 26.61 dB
Bitrate 0.488 bpp PSNR 27.87 dB
Bitrate 0.972 bpp PSNR 30.17 dB
(a) Decoded images and residuals of kodim01.
Ground Truth
Bitrate 0.074 bpp 29.16
Bitrate 0.316 bpp PSNR 35.23
bpp
bpp
(b) Decoded images residuals kodim23.
Figure 15: Examples of decoded Kodak images and their residuals.
26

---

Published as a conference paper at ICLR 2024
H.3 AUDIO
Decoded Audios
Ground Truth
5.17 kbps, 46.78 dB
10.81 kbps, 51.53 dB
22.11 kbps, 56.45 dB
here
here
here
here
Table 8: Decoded audio examples.
H.4
VIDEO
Ground Truth
Bitrate 0.12 bpp PSNR 33.14 dB
Bitrate 0.24 bpp PSNR 37.01 dB
Bitrate 0.60 bpp PSNR 42.63
Bitrate 1.18 bpp PSNR 46.31
Figure 16: Examples of decoded videos and residuals. Animation visualization is available here.
H.5 PROTEIN STRUCTURE
11.17 bits atom; RMSD 0.99 A
35.17 bits per atom; RMSD 0.12 A
60.67 bits per atom; RMSD 0.07
Ground Truth
Decoded Structure
(a) Example 1. 3D view available at here.
11.17 bits atom; RMSD 0.81 A
35.17 bits atom; RMSD 0.12 A
60.67 bits per atom; RMSD 0.06
www.
www.mma
Decoded
(b) Example 2. 3D view available here.
Figure 17: Examples of decoded protein structures and their ground truths.
27


Reviews:
[{"review_id": "iwe3oyruWg", "reviewer": "Reviewer_oN2Z", "summary": "This paper proposes three additional techniques that can be used to improve COMBINER, an INR-based modality-agnostic data compression framework. (1) Linear Reparameterization: One places Gaussian factorized priors on the latent codes, instead of the weight parameters themselves. (2) Learnable Positional Encodings: One generates positional encoding from some lower-dimensional learnable latent code, which is mapped to the full-dimensional space through upsampling and convolution. (3) Hierarchical Bayesian modeling on patchified data: One divides the data into patches and models it with hierarchical Bayesian approach. These changes make the modified version (called RECOMBINER) achieve the performance comparable to SOTA codecs and a competing INR-based approach (VC-INR).", "questions": "In addition to the requests in \"weaknesses,\" I have one more question:\n- The figure 5c is very interesting, in the sense that using the tricks without random permutation is almost the worst among all choices (brown). Do you have any explanation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- The performance gain over the previous attempt in this direction (i.e., COMBINER) is indeed very impressive. The innovations introduced in this paper seems to make the combiner-like approach a competitive paradigm for INR-based data compression.\n- All three proposed modifications are technically well-designed and well-motivated. Especially, the hierarchical Bayes approach in section 3.3 looks quite clear and intuitive.\n- The writing is very clear. The paper is one of the most easy-to-read papers among all papers I read in the past several months.\n- The empirical validation is quite extensive, covering image/video/audio to protein structures.\n- Appendix E, which describes the things that didn't work, is very useful and a good academic practice.", "weaknesses": "- **R-D tradeoff on image.** While it is definitely good to see that recombiner outperforms combiner, the performance does not clearly outperform VC-INR, an even older baseline. I do not see this as a very big drawback, but this observation makes it quite questionable whether combiner-like approach has any great potential in the long run.\n\n- **Comparison on audio/video.** I wonder how the RECOMBINER compare with VC-INR and COIN++ on audio/video datasets. The paper says that it does not compare with these baselines, because \"the works use different data splits.\" I do not think this is a good excuse to not compare with these baselines. It seems evident that recombiner outperforms combiner, but in the end, we would like to understand whether the combiner-like approach is indeed a useful framework, when compared with other INR-based paradigms (and furthermore, VAE-based ones).\n\n- **Limited range of bitrate in comparison---problems in extending to higher bitrates?.** Comparing with baselines, the range of bitrates considered is considerably smaller. For Kodak, VC-INR compares on the bitrate up to 3.5, while this work only considers up to 1.2. For videos, the range is again up to 1.2, while VC-INR does it to over 4. The same for the audio. I wonder why the authors made this choice. Does this mean that recombiner has difficulties in training in high bpp regime?\n\n- **(minor) Practicality.** Not a big issue for a research-in-progress, but the long encoding/decoding time is a severe practical limitation (appendix d.4). I wonder how these computational costs compare with the baselines; it would be a great help if authors could give an explicit head-to-head comparison with combiner, coin++, and vc-inr.\n\n- **(minor) Limited Impact of Technical Contributions.** Given the lukewarm performance gain over competing paradigms, it would be great if the proposed technical innovations are very novel or have a wider applicability to other fields of machine learning. I am not sure if this is the case; the linear reparameterization and learned positional encodings are either not very new or highly specialized to the context of recombiner. I do appreciate the technicality of the hierarchical Bayes part, but some components are somewhat mysterious to me (the strange importance of random permutation) and the idea may not have a wider impact outside this specific context.\n\n- (minor) the last row of the legend in figure 5c is wrongly ordered?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6lNab7QIj9", "reviewer": "Reviewer_TTUk", "summary": "This paper improves the most recent implicit neural representation (INR) compression method, called the COMBINER, and proposes a robust and enhanced COMBINER, thus named as RECOMBINER. The enhancements are mainly from the layer-wise block diagonal matrix in factorised Gaussian assumptions to increase the flexibility of overfitting data, additional positional embedding to address the local patterns, as well as the hierarchical model to compress high-resolution images. Experimental results verify the effectiveness of this improved COMBINER method.", "questions": "Please see my weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper improves the existing COMBINER method, with robust and enhanced performances on data compression. The improvements are reasonably established, upon the stringent factorised Gaussian assumption as proposed in the original COMBINER method. A block-wise diagonal method does much help during training and inferencing.\n2. This paper proposes the hierarchical strategy to accommodate the compression for high-resolution images. The optimisation is supported by minimising the upper bound when splitting into patches.\n3. Experimental results have verified the effectiveness of the proposed RECOMBINER method. Although not beating the state-of-the-art VAE based methods, I still value this work to be a promising alternative direction for learnt data compression.", "weaknesses": "1. The quality of this paper needs to be comprehensively improved, whereby many typos exist. For example, \"have a neural network memorize the data (Stanley, 2007) and encode the network weights instead.\".\n2. For the linear reparameterization module, why A^[l] is updated during the training stage? Also the authors claim that the block-wise diagonal matrix operates as good as the full covariance matrix. Is any verification on this, from either theoretically or emperically?\n3. For the learnt positional embeddings, I am a bit confused on using additional position cues. Since x_i also includes the coordinate information, why using z_i helps to address the global representation challenge?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DZri1qpsQ8", "reviewer": "Reviewer_DDkv", "summary": "The paper introduces RECOMBINER, an extension of the previous COMBINER implicit neural representation method for achieving neural compression. The new work includes a few model variations from the original COMBINER, including 1) a model parameter reparametrization, 2) the inclusion of positional encodings, and 3) a patch processing mechanism to handle large images. For these new model complexities, the paper also proposes an altered training procedure for fitting the parameters and the variational model. The paper considers numerical experiments for image, audio, video, and protein data. On image data, the proposed method outperforms VAE-based alternatives at low bitrates. On audio data, the proposed method outperforms the compared methods. On video data, the proposed method outperforms H.264/H.265 when H.264/H.265 are not in quality mode. The paper also validates its changes with ablation experiments.", "questions": "1. Do you think there is a mechanism where RECOMBINER could be adapted to a perceptual codec?\n2. Did you consider other neural audio compression methods? Encodec (Defossez, 2022) is one of particular note.\n3. How are Fourier embeddings computed for protein and video data?\n\nDéfossez, Alexandre, et al. \"High fidelity neural audio compression.\" arXiv preprint arXiv:2210.13438 (2022).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a series of changes in COMBINER that result in good improvements to performance on all tasks where there is data for both methods (RECOMBINER and COMBINER were not compared for video and protein data).\n2. The paper looks a large variety of tasks - most compression papers would only focus on one of these tasks.\n3. The mathematics for the reparametrization are presented clearly and intuitively.\n4. The patch-level processing for high-resolution data is a particularly welcome modification for image compression.", "weaknesses": "I am waffling on this paper a little bit largely due to the comparisons. Some of the tasks seem a little bit selective in terms of baselines. I raise a few points below.\n\n1. Despite the improvements, overall INRs continue lag behind performance of competing methods on 3 out of 4 tasks.\n2. INR performance comes at a compute cost penalty. This would be particularly large in the case of the video and image codec comparisons.\n3. Errors for the protein data seem quite large, and I don't think the benefit of rate control is particularly useful for this setting. The paper presents RECOMBINER as an option here, but when the rate is lower than the competing methods, the error approaches the machine resolution. RECOMBINER only matches competitor compression performance at high rates.\n4. Many neural methods for compression (particularly for audio) rely on perceptual compression rather than rate-distortion as is experimented with in the present paper. The paper does not consider how RECOMBINER could be adapted to become a perceptual codec.\n5. Older image compression methods are used as baselines. The handcrafted baseline of choice is now VTM, and the baseline neural image compression is ELIC (He, 2022).\n\nHe, Dailan, et al. \"Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)