Overfitting for Fun and Profit: Instance-Adaptive Data Compression
Authors: Ties van Rozendaal
Iris AM Huijben
Taco Cohen
Venue: ICLR 2021 Poster (2021)
Keywords: Neural data compression
Learned compression
Generative modeling
Overfitting
Finetuning
Instance learning
Instance adaptation
Variational autoencoders
Rate-distortion optimization
Model compression
Weight quantization

Abstract: Neural data compression has been shown to outperform classical methods in terms of $RD$ performance, with results still improving rapidly.
At a high level, neural compression is based on an autoencoder that tries to reconstruct the input instance from a (quantized) latent representation, coupled with a prior that is used to losslessly compress these latents.
Due to limitations on model capacity and imperfect optimization and generalization, such models will suboptimally compress test data in general.
However, one of the great strengths of learned compression is that if the test-time data distribution is known and relatively low-entropy (e.g. a camera watching a static scene, a dash cam in an autonomous car, etc.), the model can easily be finetuned or adapted to this distribution, leading to improved $RD$ performance.
In this paper we take this concept to the extreme, adapting the full model to a single video, and sending model updates (quantized and compressed using a parameter-space prior) along with the latent representation. Unlike previous work, we finetune not only the encoder/latents but the entire model, and - during finetuning - take into account both the effect of model quantization and the additional costs incurred by sending the model updates. We evaluate an image compression model on I-frames (sampled at 2 fps) from videos of the Xiph dataset, and demonstrate that full-model adaptation improves $RD$ performance by ~1 dB, with respect to encoder-only finetuning.

Published as a conference paper at ICLR 2021
OVERFITTING FOR FUN AND PROFIT: INSTANCE-ADAPTIVE DATA COMPRESSION
Ties Rozendaal Qualcomm Al Research f tiesêqti. wtt.epp.....lllcm...oommm com
Iris A.M. Huijben"
Qualcomm Al Research
Department of Electrical Engineering Eindhoven University of Technology i. a m. hui jbenntteeeeee..... nl
Taco S. Cohen
Qualcomm AI Research I
tacos@ati qual mimalcom..om com
ABSTRACT
Neural data compression has been shown to outperform classical methods in terms of rate-distortion (RD) performance, with results still improving rapidly. At high level, neural compression is based on an autoencoder that tries to recon- struct the input instance from a (quantized) latent representation, coupled with a prior that is used to losslessly compress these latents. Due to limitations on model capacity and imperfect optimization and generalization, such models will subopti- mally compress test data in general. However, one of the great strengths of learned compression is that if the test-time data distribution is known and relatively low- entropy (e.g. camera watching static scene, a dash cam in an autonomous etc.), the model can easily be finetuned or adapted to this distribution, leading to improved RD performance. In this paper we take this concept to the extreme, adapting the full model to single video, and sending model updates (quantized and compressed using parameter-space prior) along with the latent represen- tation. Unlike previous work, we finetune not only the encoder/latents but the entire model, and during finetuning take into account both the effect of model quantization and the additional costs incurred by sending the model updates. We evaluate an image compression model on I-frames (sampled 2 fps) from videos of the Xiph dataset, and demonstrate that full-model adaptation improves RD performance by dB, with respect to encoder-only finetuning.
INTRODUCTION
The most common approach to neural lossy compression is to train variational autoencoder (VAE). like model on a training dataset to minimize the expected RD cost D + 3R (Theis et al. 2017 Kingma & Welling 2013). Although this approach has proven to be very successful (Ballé al. 2018), a model trained to minimize expected R.D cost over a full dataset is unlikely to be optimal for every test instance because the model has limited capacity, and both optimization and generalization will be imperfect. The problem of generalization will be especially significant when the testing distribution is different from the training distribution, as is likely be the case in practice.
Suboptimality of the encoder has been studied extensively under the term inference suboptimality (Cremer al., 2018)), and has been shown that finetuning the encoder or latents for particular instance can lead to improved compression performance (Lu et al. 2020 Campos et al., 2019 Yang al. 2020b] Guo al., (2020). This approach appealing as no additional information needs be added to the bitstream, and nothing changes on the receiver side. Performance gains however are limited, because the prior and decoder can not be adapted.
* "qqual contribution
"Qullcomm AI Research is an initiative of Qualcomm Technologies, Inc.
Work done during internship Qualcomm AT Research

---

Published as a conference paper at ICLR 2021
In this paper present method for full-model instance-adaptive compression, i.e. adapting the entire model to a single data instance. Unlike previous work, our method takes into account the costs for sending not only the latent prior, but also the decoder model updates, well quantization of these updates. This is achieved by extending the typical RD loss with an additional model rate term M that measures the number of bits required send the model updates under newly introduced model prior, resulting in combined RDM loss.
As an initial proof of concept, we show that this approach can lead very substantial gains in RDD performance (~ 1 dB PSNR gain at the same bitrate) on the problem of I-frame video coding, where a set of key frames, sampled from video at fps, are independently coded using an I-frame (image compression) model. Additionally. we show how the model rate bits are distributed across the model, and (by means of an ablation study) quantify the individual gains achieved by including a model-rate loss and using quantization-awan finetuning.
The rest of this paper is structured as follows. Section 2 discusses the basics of neural compression and related work on adaptive compression. Section 3 presents our method, including details on the RDM loss, the choice of the model prior, its quantization, and the (de)coding procedure. In Sections 4 and we present our experiments and results, followed by a discussion in Section 6
PRELIMINARIES AND RELATED WORK
2.1 NEURAL DATA COMPRESSION
The standard approach to neural compression can be understood as a particular kind of VAE (Kingma & Welling 2013). In the compression literature the encoder (x(=(æ) is typically defined by neural network parameterized by ø, with either deterministic output (so qa xx) is one-hot) (Habibian et al. 2019) or with fixed uniform [0. 1] noise on the outputs (Ballé et al. 2018). In both cases, sampling
(z(2)) is used during training while quantization is used at test time.
The latent # is encoded to the bitstream using entropy coding in conjunction with a latent prior po(z), so that coding z takes about log (o( bits (up to discretization). On the receiving side, the entropy decoder is used with the same prior (2) to decode z and then reconstruct å using the decoder network pe (z) 2z) (note that we use the same symbol 0 to denote the parameters of the prior and decoder jointly, as in our method both will have to be coded and added to the bitstream).
From these considerations terms in the following loss:
clear that the rate R and distortion can be measured by the two
LRD( (0, 0) B (x |- log po()]] + E Byg(=|a) |- log po (x]22)
R
D
This loss is equal (up to the tradeoff parameter 3 and additive constant) to the standard negative evidence lower bound (ELBO) used in VAE training. The rate term of ELBO written as a KI. divergence between encoder and prior, but since (KL(q, = R [q]. and the encoder entropy H [q] is constant in our case, minimizing the KL loss is equivalent to minimizing the rate loss.
Neural video compression is typically decomposed into the problem of independently compressing set of key frames (i.e. I-frames) and conditionally compressing the remaining frames (Lu et al., 2019 Liu et al., 2020 (u et al. 2018 Djelouah et al., 2019 Yang al. 2020a). In this work, we specifically focus on improving --rame compression.
2.2 ADAPTIVE COMPRESSION
A compression model is trained on a dataset D with the aim of achieving optimal RD performance on test data. However, because of limited model capacity, optimization difficulties, or insufficient data (resulting in poor generalization), the model will in general not achieve this goal. When the test data distribution differs from that of the training data, generalization will not be guaranteed even in the limit of infinite data and model capacity, and perfect optimization.
convenient feature of neural compression however is that model can easily be finetuned on new data or data from a specific domain. A model can for instance (further) be trained after deployment,
2

---

Published as a conference paper at ICLR 2021
and Habibian et al. (0199) showed improved R.D gains after finetuning video compression model on footage from a dash cam, an approach dubbed adaptive compression (Habibian et al. 2019).
In adaptive compression, decoding requires access to the adapted prior and decoder models. These models (or their delta relative to a pretrained shared model) thus need to be signaled. When the amount of data coded with the adapted model is large, the cost of signaling the model update will be negligible as it amortized. However, a tradeoff exists, the more restricted the domain of adaptation, the more we can expect to gain from adaptation (e.g. an image compared videos). In this paper we consider the case where the domain of adaptation is a set of I-frames from a single video, resulting in costs for sending model updates which become very relevant.
video or collection of
2.3 CLOSING THE AMORTIZATION GAP
Coding model updates can easily become prohibitively expensive when the model is adapted for every instance. However, if we only adapt the encoder or latents, no model update needs to be added the bitstream, since the encoder is not needed for decoding as the latents are sent anyway. We can thus close, or at least reduce, the amortization gap (the difference between Is (s (z|az az) and the optimal encoder; Cremer et al. )2018) without paying any bits for model updates. Various authors have investigated this approach: Aytekin et al. (2018); Lu et al. (2020) adapt the encoder, while (aaless et al. (2019); Yang al. (2020b); Guo et al. (2020) adapt the latents directly. This simple approach was shown to provide a modest boost in RTD) performance.
2.4 ENCODING MODEL UPDATES
As mentioned, when adapting (parts of) the decoder or prior to an instance, model updates have to be added to the bitstream order to enable decoding. Recent works have proposed ways to finetune parts of the model, while keeping the resulting bitrate overhead small. For instance Klopp et al. (2020) train a reconstruction error predicting network at encoding time, quantize its parameters, and add them to the bitstream. Similarly (Lam et al., 2019 2020) propose to finetune all parameters or only the convolutional biases, respectively, of an artifact removal filter that operates after decoding. and
A sparsity-enfordet is applied to even more strictly enforce sparsity. The update vector is thereafter quantized using k-means clustering. Finally, Zou et al.] (2020) finetune the latents in meta-learning framework, in addition to updating the decoder convolutional biases, which are quantized by k-means and there- potentially
suppressing penalty is leveraged, and additional thresholding
after transmitted. All these methods perform quantization post-training, leading unbounded reduction in performance. Also, albeit the use of regularizing loss terms, no valid proxy for the actual cost of sending model updates is adopted. Finally, none of these methods performs adaptation of the full model.
The field of model compression is related our work as the main question to be answered is how to most efficiently compress a neural network without compromising on downstream task performance (Han et al., 2016; Kuzmin al., 2019). Bayesian compression is closely related, where the model weights are sent under a model prior (Louizos et al. 2017, Havasi et al. 2018) as is the case in our method. Instead of modeling uncertainty in parameters, we however assume a deterministic posterior (i.c. point estimate). Another key difference with these works is that send the model parameter updates relative to an existing bascline model, which enables extreme compression rates (0.02-0.2 bits/param). This concept of compressing updates has been used before in the context of federated learning (McMahan et al. 2017] Alistarh et al.) 2017) as well, We distinguish ourselves from that context, there the model has be compressed during every iteration, allowing for error corrections in later iterations. We only transmit the model updates once for every data instance that finetune on.
FULL-MODEL INSTANCE- COMPRESSION
In this section we present full-model finetuning on one instance, while (during finetuning) taking into account both model quantization and the costs for sending model updates. The main idea is described in Section (.I after which Section 3.2 and 3.3 respectively provide details regarding the model prior and its quantization. The algorithm described in Section 3.4
3

---

Published as a conference paper at ICLR 2021
3.1 FINETUNING AT INFERENCE TIME
Full-model instance-adaptive compression entails finetuning of set of global model parameters {oD, (obtained by training on dataset D) on a single instance z. This results in updated param- eters 6, 8, of which only / has to be signaled in the bitstream. In practice we only learn the changes with respect to the global model, and encode the model updates 5 = OD of the decoding model. In order encode 6, we introduce continuous model prior p(6) to regularize these updates, and use the quantized counterpart p|ō] for entropy (de)coding them (more on quantization in Section 3.3
The overhead for sending quantized model update 8 is given by model rate M log plo]. and can be approximated by its continuous counterpart M = p()) (see Appendix A. for justifi- cation). Adding this term to the standard RD loss using the same tradeoff parameter 19, obtain the instance-adaptivy compression objective:
LRDM(O, 6) LRD(O. + +8) 3 (-llog(())
M
At inference time, this objective can be minimized directly to find the optimal model parameters for transmitting datapoint æ. It takes into account the additional costs for encoding model updates (M term), and incorporates model quantization during finetuning (LRD evaluated 0 = OD + 8).
3.2 MODEL PRIOR DESIGN
A plethora of options exist for designing model prior p(8) any probability density function (PDF) could be chosen, but a natural choice for modeling parameter update å = 0 - Op is to leverage Gaussian distribution, centered around the zero-update. Specifically, we can define the model prior on the updates as a multivariate zero-ceng pero-centered Gaussian with zero covariance, and single shared (hyperparameter) σ, denoting the standard deviation: p(5) (66600.
Note that this is equivalent to modeling 0 by p(0) = NO (D.
When entropy (de)coding the quantized updates 8 under p(ö), we must realize that even the zero- update, i.e. = 0, is not for free. We define these initial static costs as Mo = - - log = 0). Because the mode of the defined model prior is zero, these initial costs Mg equal the minimum costs. Minimization of eq. (2) thus ensures that after overcoming these static costs any extra bit spent on model updates will be accompanied by a commensurate improvement in RD performance.
Since our method works best when signaling the zero-update is cheap, we want to increase the probability mass p|β = 0]. We propose to generalize our carlier proposed model prior to a so-called spike-and-slab prior (Ročková & George 2018), which drastically reduces the costs for this zero- update. More specifically, we redefine the PDF as a (weighted) sum two Gaussians a wide (slab) and narrow (spike) distribution:
p(6) Potab(O) + a Pspike (0)
with
1
Palat (6) = NN oT) and Pspike (6) = N6
(3)
where a E 1Rz0 a hyperparameter determining the height of the spiky Gaussian with respect the wider slab, is the the bin width used for quantization (more details in Section (.3). By choosing the standard deviation of the spike to be 6. the mass within six standard deviations (i.e. 99. of the total mass) is included in the central quantization bin after quantization. Note that the (slab-only) Gaussian prior is a special case of the apike-and-slab Gaussian prior in eq. (3) where 0. As such, p(o) refers to the spike-and-slad prior in the rest of this work. Appendix A.2 compares the continuous and discrete spike-and-she prior and its gradients.
(.3). and >> f
Adding the spike distribution, not only decreases Mo. it also more heavily enforces sparsity on the updates via regularizing term M in eq. ( In fact, a high spike (i.e. large a) can make the bits for signaling a zero-update a binary choice; a parameter either worth updating at the cost of some additional rate, or its not updated and the 'spent' bits are negligible.
small (i.e. almost negligible), that the model effectively learns to make
4

---

Published as a conference paper at ICLR 2021
3.3 QUANTIZATION
In order to quantize scalar ó; (denoted by á in this section to avoid clutter), we use N equispaced bins of width I, and define the following quantization function:
8 = Q6 clip ( min (N - 1) max (N
As both rounding and clipping are non-differentiable, the gradient of Qr is approximated using the Straight-Troough estimator (STE), proposed by [engiio et al. (2013) That is, we assume (Qr(6)/06 1.
The bins are intervals the smallest integer such that the region covered by bins (i.e. the interval [- - 1)t/2, (N (N--))/2),, covers at least 1 2- of the probability mass of p(6). Indeed the number of bins N proportional to the ratio of the width of p(o) and the width of the bins: N x a/t. The number of bins presents a tradeoff between finetuning flexibility and model rate costs M, so the a /t-ratio an important hyperparameter. The higher N, the higher these costs due finer quantization, but simultaneously the lower the quantization gap, enabling more flexible finetuning.
We view as a hyperparameter, and define N be
Since 8 Q:) the discrete model prior plō is the pushforward of p(6) through Q::
8+2/2
plo = /
p(0)d6
d( P(8 - P(6 < 8 t/2).
(5)
(8)
1/-1/2
That is, p|ō] equals the mass of (6) in the bin of 8, which can be computed as the difference of the cumulative density function (CDF) of p(8) evaluated at the edges of that bin.
3.4 ENTROPY CODING AND DECODING
After finetuning of the compression model on instance by minimizing the loss in eq. (2) both the latents and the model updates are entropy coded (under their respective priors pū(z) and p(0) into bitstream Decoding starts by decoding using plő). followed by decoding using pa (where 0 8 + 0p). and finally reconstructing using Pa (x z). The whole process is shown in Figure and defined formally in Algorithms and
EXPERIMENTAL SETUP
4.1 DATASETS
The experiments in this paper use images and videos from the following two datasets:
The CLIC19 dataset contains a collection of natural high resolution images. con- ventionally divided into a professional set, and set of mobile phone images. Here we merge the existing training folds of both sets and use the resulting dataset D to train our I-frame model. The corresponding validation folds are used to validate the global model performance.
CLIC19 T
Xiph-5N 2fps
The Xiph dataset contains a variety videos of different formats. We select a representative sample of five videos (5N) from the set of 1080p videos (see Appendix B for details regarding selection of these samples). Each video is temporally subsampled to fps to create a dataset of I-frames, referred to Xiph-5N 2fps. Frames in all videos contain 1920 1080 pixels, and the set of I-frames after subsampling to 2 fps contain between 20 and 42 frames. The five selected videos are single-scene but multi-shot, and come from variety of sources. Xiph-5N 2fps is used to validate our instance-adaptive data compression framework.
4.2 GLOBAL MODEL ARCHITECTURE AND TRAINING
Model rate can be restricted by (among others) choosing a low-complexity neural compression model, and amortizing the additional model update costs over a large number of pixels.
httaass/www.ccmmrrsiioooc/////1 https://melia.iiih..g///tddoddd
1enge/
5

---

Published as a conference paper at ICLR 2021
model prlor P[S]
noddll prlo P(R)
shared knowledge
bg
sender
receiver
E11
ED
D1
8
D2
1 PE10
lnsert Cr(
Op (rls)
pr(e)
petxte)
EB
1
1 D4
00
BO
ED
E11
D3
Figure 1: Visualization of encoding (Algorithm 1 and decoding (Algorithm 2 of our full-model instance- adaativv method. Each step denoted with a code, e.g. E9, which refers to line 9 of the encoding algorithm. EE and ED denote entropy encoding and decoding, respectively. Both the latent representation and the parameter updates ó are encoded their respective bitstreams by and bz- Model prior plö] entropy decodes by. after which the latent prior can decode b2.
Algorithm Encoding
Input: Global model parameters fΦb, 0p} trained on training set D, model parameter quantizer Qs model prior pl0), datapoint to be compressed æ.
Output: Compressed bitstream = (bz, bz)
1: Initialize model parameters: ø ΦD. and I 0D
2: for step in MAX STEPS do
3: Sample single I-frame: Ff
4: Quantize transmittable parameters: 0 Qx 0p. with 5 = 0 0p
5: Forward pass: x ~ qo (z x and evaluate Pa (z) and pz(z).
6: Compute loss LROM , 8) y according eq.
aCRDM
0CRDM
7:
Backpropagate using STE for Qz, then update Φ, 0 using gradients
and
0Φ
00
8: end for
9: Compress x to 2 ~ (x))..
10: Compute quantized model parameters: = 0D + ō, with 8 Q. 0p).
11:
Entropy encode: by enc( plō) and b enc( p;()))
Algorithm 2 Decoding of x
Input: Global model parameters Op trained on training set D, model prior plő]. bitstream b b= (bg, by).
Output: Decoded datapoint a
1: Entropy decode: 8 = (ec(by;p(0))
2: Compute updated parameters: 0 - OD + 8.
3: Entropy decode latent under finetuned prior: z = deccbb:;ppp22
4: Decode instance the mean of the finetuned decoder: pa (2(x)2
The most natural case for full-modes adaptation is therefore to finetune a low-complexity model on video instance. Typical video compression setups combine an image compression (I-frame) model to compress key frames, and a predict- or between-frame model reconstruct the remaining frames. Without loss of generality for video-adaptive finetuning, showcase full-model finetuning framework for the I-frame compression problem, being a subproblem of video compression.
Specifically, we use the (relatively low-complexity) hyperprior-mode proposed by Ballé et al. (0018), including the mean-scale prior (without context) from Minnen et al.) (2018). Before fine- tuning, this model is trained the training fold of the CLIC19 dataset, using the R.D objective given in eq. (I. Appendix provides more details on both its architecture and the adopted training procedure.

---

Published as a conference paper at ICLR 2021
4.3 INSTANCE- FINETUNING
Each instance in the Xiph-5N 2fps dataset (i.e. a set of I-frames belonging to one video) is (sepa- rately) used for full-model adaptation. The resulting model rate costs are amortized over the set of I-frames, rather than the full video. As a benchmark, we only finetune the encoding procedure, as it does not induce additional bits in the bitstream. Encoder-only finetuning can be implemented by either finetuning the encoding model parameters o, or directly optimizing the latents as in Campos al. 2 We implement both benchmarks, as the former an ablated version of our proposed full-model finetuning, whereas the latter is expected to perform better due to the amortization gap Cremer et al. (2018).
Global models trained with RD tradeoff parameter / E {3c-3, 1e-3, 2.5c-4, 1e-4] are used to initialize the models that are then being finetuned with the corresonding value for B. Both encoder- only and direct-latent finetuning minimize the RD loss given in eq. ( For encoder-only tuning we use a constant learning rate of 1e-6, whereas for latent optimization a learning rate of 5e-4 is used for the low bitrate region (i.e. two highest /3 values), and 1 for the high rate region. In case of direct latent optimization, the pre-quantized latents are finetuned, which are initialized using forward pass of the encoder.
Our instance- instance-adaptive adaptive full-model finetuning framework extends encoder-only finetuning by jointly updating and 0 using the RDM loss from eq. ( In this case we finetune the global model that is trained with 3 = 0.001, independent of the value of 33 during finetuning. Empirically, this resulted in negligible difference in performance compared to using the global model of the corresponding finetuning B, while it alleviated memory constraints thanks to the smaller size of this low bitrate model architecture (see Appendix C.. The training objective in eq. (2) is expressed in bits per pixel and optimized using a fixed learning rate of 1c-4. The parameters for the model prior were chosen as follows: quantization bin width 0.005, standard deviation o 0.05, and the multiplicative 1000. We empirically found that sensitivity to changing & in the range 50-5000 was low. Realize that, instead of empirically setting a, its value could also be solved for a target initial cost Mo. given the number of decoding parameters and pixels in the finetuning instance.
factor of the spike
All finetuning experiments (both encoding-only and full-model) ran for 100k steps, each containing one mini-batch of a single, full resolution I-frame") We used the Adam optimizer (default settings) Kingma& & Ba 2014), and best model selection was based on the RDM loss over the set of I-frames.
5 RESULTS
5.1
RATE-DISTORTION GAINS
Figure 2h shows the compression performance for encoder-only finetuning, direct latent optimiza- tion, and full-model finetuning, averaged over all videos in the Xiph-5N 2fps dataset for different rate-distortion tradeoffs. Finetuning of the entire parameter-space results in much higher RDM gains (on average approximately dB for the same bitrate) compared to only finetuning the en- coding parameters Φ or the latents directly. Figure 8 in Appendix E shows this plot for each video separately. Note that encoder-only finetuning performance is on par with direct latent optimization, implying that the amortization gap (Cremer et al. 2018) close to zero when finetuning the encoder model on a moderate number of I-frames.
Table provides insight in the distribution of bits over latent rate R and model rate M, which both increase for lower values of B. However, the relative contribution of the model rate increases for the higher bitrate regime, which could be explained by the fact that this regime the latent rate can more heavily be reduced by finetuning. Figure 2 indeed confirms that the the total rate reduction is higher for the high bitrate regime, which thus fully originates from the reduction in latent rate after finetuning. Table I also shows that the static intial costs Mo only marginally contribute to the total model rate. Figure shows for one video how finetuning progressed over training steps, confirming that the compression gains already at the beginning of finetuning cover these initial costs. This effect was visible for all tested videos (see Appendix E
3 All frames in the videos in Xiph- 5N 26ps are of spatial resolution 1920 x 1080. In order to make this shape compatible with the strided convolutions in our model, we pad each frame to 1920 x 1088 hefore encoding. After reconstructing àt, it is cropped back its original resolution for evaluation.
7

---

Published as a conference paper at ICLR 2021
Average Xiph-SN 21ps Result
Finetuning Progression for sunflower 2tps
Ablation for sunflower 2fps (42 fremes)
42
41
14
40
PONR (50)
30
-00
PSN
PSNW
-05
Flectaring
Evaluation
-6
Baselite
Tre lainigg Step
t0 Trained Bassine
Instance Adaptive Models
20200
34
34
Encoden-Orly Finsuuring W 0)
50300
40200
50303
32
11
R
97
33
Full Rodel Fineturing IM 00
103030
-1-
RID Traeedttiona Rocrlre
plaee
(a)
(b)
(c)
Figure 2: (a) Averaged RDM performance over all videos of the Xiph-5N 2fps datase for four dif- ferent rate-distortion tradeoff's with = 3e-3, le-3. full-model finetuning outperforms encoder-only and direct latent optimization with approximately dB gain for the same rate. (b) Finetuning progression of the sunflower video over time. Be- tween each dot, 500 training steps are taken, showing that already at the start of finetuning large RDM gains are achieved and the RDM performance continues to improve during finetuning. (c) Ablation where we show the effect of both quantization- (Q)( and model rate aware (M Loss) finetuning. Case VI shows the upper bound on achievable finetuning performance when (naively) not taking into account quantization and model update rate.
2.5e-4, and le-4 (from left to right). Our
5.2 ABLATIONS
Figure 2 shows several ablation results for one video. Case is our proposed full-model finetun- ing, optmizing the RDM loss, while simultaneously quantizing the updates to compute distortion (denoted with (6)). One can see that not doing quantization-aware finetuning (case II) deteri- orates the distortion gains during evaluation. Removing the (continuous) model rate penalty from the finetuning procedure (case III) imposes an extreme increase in rate during evaluation, caused by unbounded growing of the model rate during finetuning. As such, the models result in a rate even much higher than the baseline model's rate, showing that finetuning without model rate awareness provides extremely poor results. Case IV shows performance deterioration in the situation of both quantization- and model rate unaware finetuning. Analyzing these runs while (naively) not taking into account the additional model update costs (cases V and VI) provides upper bounds on com- pression improvement. Case VI shows the most naive bound; RD finetuning without quantization, whereas case V is a tighter bound that does include quantization. The gap between V and VI is small, suggesting that the used quantization strategy does only mildly harm performance.
An ablation study on the effect of the number of finetuning frames is provided in Appendix D It reveals that, under the spike-and-slab model prior, full-model finetuning works well for a large range of instance lengths. Only in the worst case scenario, when finetuning only one frame in the low bitrate regime, full-model finetuning was found to be too costly.
5.3 DISTRIBUTION OF BITS
Figure 3 shows for different (mutually exclusive) parameter groups the distribution of the model updates ó (top) and their corresponding bits (bottom). Interestingly one can see how for (almost) all groups, the updates are being clipped by our carlier defined quantizer Q. This suggests the need for large, expensive updates in these parameter groups, for which the additional R.D gain thus appears to outweigh the extra costs. At the same time, all groups show an elevated center bin, thanks to training with the spike-and-slad prior (see Appendix F.. By design of this prior, the bits paid for this zero- update are extremely low, which can best be seen in the bits histogram (Fig. Boottom of the Codec Decoder Weight and Biases. The parameter updates of the Codec Decoder IGND group are the only ones that are sonysmmeetricllly distributed across the zero-updated, which can be explained by the fact that IGDN (Ballé et al.] 2016)) is an (inverse) normalization layer. The Codec Decoder Weights were found to contribute most to the total model rate M.
8

---

Published as a conference paper at ICLR 2021
HyperPrior
HyperCodec. Wee.e..ee.Weggh.....
HyperCoder. Dec.Blas
Godee.Dec.Weagge
Codec.Dec.Bio
Codec.Dec..ce DeC.IGDN
K-1 lapu-
18 bapase
Med, Leit tjo beaani
Wed 2e.5 saine tearan
Mod. Se-5 Egex 0.17 buian
Mid Sed hipe bapon
M-d2e Eggor bytarees
Count
(bits)
10
520
010
510
030 000
040 500
000-25 013
500
Figure 3: Empirical distribution of parameter updates for the model finetuned on the sunflower ower video with / = 2. Columns denote parameter groups. Top: Histograms of the model updates Bottom:
Histogram of bit allocation for 8. Subtitles indicate the total number of bits for each parameter group, both expressed in bits per pixel (b/px) and bits per parameter (b/param).
e
PSNR (dB)
R+ M
R
M
Mo
M
M
(bits/pixell
(bits/parameter)
(kB/frame)
3.0e-03
34.0
0.175
0.174
0.001
0.00033
0.026
0.39
1.0e-03
36.2
0.304
0.301
0.003
0.00033
0.044
0.67
2.5e-04
38.9
0.601
0.593
0.008
0.00033
0.129
1.99
1.0c-04
40.7
0.921
0.905
0.017
0.00033
0.282
4.35
Table I: Distribution of bitrate for different rate-distortion tradeoff's averaged over the videos in the Xiph-5N 2fps dataset. The number of bits are distributed over the latent rate R and the model rate M, which is computed using the quantized model prior plō].
6 DISCUSSION
This work presented instance-adaptive neural compression, the first method that enables finetuning of full compression model set of I-frames from single video, while restricting the additional bits for encoding the (quantized) model updates. To this end, the typical rate-distortion loss was extended by incorporating both model quantization and the additional model rate costs during fine- tuning. This loss guarantees pure RDM performance gains after overcoming a small initial cost for encoding the zero-update. We showed improved RDM performance on all five tested videos from the Xiph dataset, with an average distortion improvement of approximately I dB for the same bitrate.
Among vidcos, we found a difference in achieved finetuning RDM gain (see Appendix Possible causes can be three-fold. First, the performance of the global model differs per video, therewith influencing the maximum gains to be achieved by finetuning. Second, video characteristics such as (non-) (oonn)ssationarity greatly influence the diversity of the set of I-frames, thereby affecting the ease of model-adaption. Third, the number of I-frames differs per video and thus trades off model update costs (which are amortized over the set of I-frames), with ease of finetuning.
The results of the ablation in Fig. 2 show that the quantization gap (V vs VI) considerably smaller than the performance deterioration due to (additionally) regularizing the finetuning using the model prior (I vs V). Most improvement in future work is thus expected to be gained by leveraging more flexible model priors, e.g. by learning its parameters and/or modeling dependencies among updates.
We showed how instance- adaptive full-model finetuning greatly improves RDM performance for I-frame compression, a subproblem in video compression. Equivalently, one can exploit full-model finetuning to enhance compression of the remaining (non-key) frames of video, compressing the total video even further. Also, neural video compression models that exploit temporal redundancy. could be finetuned, as long as the model's complexity is low enough to restrict model rate. Leverag- ing such a low-complexin video model moves computational complexity of data compression from the receiver to the sender by heavily finetuning this small model on each video. We foresee conve- nience of this computational shift in applications where bandwidth and receiver compute power is scarce, but encoding compute time less important. This use case in practice happens e.g. for (non- live) video streaming to low-power edge devices. Finding such low-complexity video compression models is however non-trivial as it's capacity must still be enough to compress an entire video. We foresee great opportunities here, and will therefore investigate this in future work.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-effiien sgd via gradient quantization and encoding. Advances in Neural Information Processing Systems, pp. 1709-1720, 2017.
Caglar Aytekin, Xingyang Ni, Francesco Cricri, Jani Lainema, Emre Aksu, and Miska M Hannuk- sela. Block-optimized variable bit rate neural image compression. In CVPR Workshops, pp. 2551-2554, 2018.
Johannes Ballé, Valero Laparra, and Eero Simoncelli. Density modeling of images using a general- ized normalization transformation. In 4th International Conference on Learning Representations, ICLR 2016, 2016.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In International Conference on Learning Represen- tations, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 130..343.,. 2013.
Joaquim Campos, Simon Meierhans. Abdelaziz Djelouah, and Christopher Schroers. Content adap- tive optimization for neural image compression. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autoen- coders. In International Conference on Machine Learning, pp. 1078-1086, 2018.
Abdelaziz Djelouah, Joaquim Campos, Simone Schaub-Meyer, and Christopher Schroers. Neural inter-frame compression for video coding. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6421-6429 2019.
Tiansheng Guo. Jing Wang. Ze Cui, Yihui Feng, Yunying Ge, and Bo Bai. Variable rate image compression with content adaptive optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp.
2020.
Amirhossein Habibian, Ties van Rozendaal, Jakub Tomezak, and Taco S Cohen. Video compres- sion with rate-distortion autoencoders. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7033-7042. ppenaccess.theevf.com 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. arXiv preprint 110.0:151 2016.
Marton Havasi, Robert Peharz, and José Miguel Hernández-Lobale Minimal random code learning: Getting bits back from compressed model parameters. In International Conference on Learning Representations, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2014.
Diederik Kingma and Max Welling. Auto-Encoding variational bayes. arXiv: 14, 2013.
arXiv preprint
Jan Klopp, Liang-Gee Chen, and Shao-Yi Chien. Utilising low complexity cnns lift non-local redundancies in video coding. IEEE Transactions on Image Processing, 2020.
Andrey Kuzmin, Markus Nagel, Saurabh Pitre, Sandeep Pendyam, Tijmen Blankevoort, and Max Welling, Taxonomy and evaluation of structured compression of convolutional neural networks. arXiv preprini arXiv: (w..199229999 2019.
Yat- nat-Hong Lam, Alireza Zare, Caglar Aytekin, Francesco Cricri, Jani Lainema, Emre Aksu, and Miska Hannuksela. Compressing weight-updates for image artifacts removal neural networks. In Proceedings of the IEEÉ Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
10

---

Published as a conference paper at ICLR 2021
Yat-Hong Lam, Alireza Zare, Francesco Cricri, Jani Lainema, and Miska M Hannuksela. Efficient adaptation of neural network filter for video compression. In Proceedings of the 28th ACM Inter- national Conference Multimedia, pp.
2020.
Haojie Liu, Han Shen, Lichao Huang, Ming Lu, Tong Chen, and Zhan Ma. Learned video compres- sion via joint spatial-temporal correlation exploration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 1 11580- 11587. 2020.
Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning. In Advances in neural information processing systems, pp. 3288-3298, 2017.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dve: An end- to-end deep video compression framework. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Guo Lu, Chunlei Cai, Xiaoyun Zhang, Li Chen, Wanli Ouyang, Dong Xu, and Zhiyong Gao, Content adaptive and error propagation aware deep video compression. arXiv preprint 1tt::2000311288: 2020.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficii learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1282. PMLR, 2017.
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. In Advances in Neural Information Processing Systems, pp- 10771-10780, 2018.
Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander G Anderson, and Lubomir Bour- dev. Learned video compression. In Proceedings of the IEFE International Conference on Com- puter Vision, pp. 1444-34663 2019.
Veronika Ročková and Edward I George. The spike-ande sakk-aand-slab lasso. Journal of the American Statis- tical Association, 1 113(521):431 444, 2018.
L. Theis, W. Shi, A. Cunningham, and F. Huszár. Lossy image compression with compressive autoencoders. In International Conference on Learning Representations, 2017. URL https: //openzeview. net//c////.c http://wwwww.eeeeeeddd/ddddedeed (iardddddd=JJ-wwgge
Chao-Yuan Wu, Nayan Singhal, and Philipp Krahenbuhl. Video compression through image inter- polation. Proceedings of the European Conference on Computer Vision (ECCV), pp. 416-431, 2018.
Yang Yang, Guillaume Sautière, Jon Ryu, and Taco S Cohen. Feedback recurrent autoencoder. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3347-3351. IEEE, 2020a.
Yibo Yang, Robert Bamler, and Stephan Mandt. Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33, 2020b.
Nannan Zou, Honglei Zhang, Francesco Cricri, Hamed R Tavakoli, Jani Lainema, Miska Hannuk- sela, Emre Aksu, and Esa Rahtu. L'C-Icarning to learn to compress. In 2020 IEEE 22nd Inter- national Workshop on Multinedia Signal Processing (MMSP), pp. 1-6. IEEE, 2020.
11

---

Published as a conference paper at ICLR 2021
A MODEL RATE LOSS
A.1 GRADIENT DEFINITION
The proof below shows that Vg. is unbiased first-order approximation of M, validating its during finetuning.
The gradient of the continuous model rate loss M towards ô defined as:
V V (ogg(()) Vsp(o) p(6)
(6)
The gradient of the non-differentiable discrete model rate loss M towards can be defined by ex- ploiting the Straight-Througe gradient estimator (Bengio et al. 2013), ie. V 1.
As suct
Va logo < 5 2 P(ό < <3-3 L
t
V&M- = - Voglo
2
2
- 4)
P(d < P -) P(8 < - 4)
By first-order approximation we can write:
V Vap(d) ~ pl6
+ p
p(ō -
(8)
and
2 )) P(8 < 8 ; p(ō)t.
P(5 < 8
(9)
Using eq. 8 and eq.
p
express the gradient of the discrete model rate costs V.M as:
V. M Vap(ō)t
Vap(o)
(10)
p(o)t
p(6)
which is thus first-order approximation of V.M.
A.2 CONTINUOUS DISCRETE MODEL RATE PENALTY
The proof provided in the previous section is not restricted to specific designs for p()), and thus holds both for the spike-and-slab prior including spike (a > 0), or the special case where spike is used (a 0).
Figure shows quantization of the model updates (top-leftt and its corresponding gradient (left- bottom), exploiting the Straight-Throupe gradient estimator [Begiio et al. (2013). Also the discrete (true) model rate M and its continuous analogy M (middle-top) with their corresponding gradients (middle-bottom) are shown for the special case of not using a spike, i.e. a = O. One can see that the continuous model rate proxy is shifted version of the discrete costs. Since such translation does not influence the gradient (and neither gradient-brr -ased optimization), the continuous model rate loss can be used during finetuning, preventing instable training thanks its smooth gradient.
'Due to cutting (a maximum) of -8 mass from the tails of p(o) to enable quantization, the difference in cumulative masses in eq. (7) should be renormalized by 1 - 2- As this constant division inside a log. it results in subtraction of V. log - 2 = O. Since this normalization does not influence the gradient, we omitted it for the sake of clarity.
12

---

Published as a conference paper at ICLR 2021
Quantization
rate overnead (a=0.00
overhead (a=4.0)
function
03
presy
presy
oradient
53
dNl
Figure 4: (Left) Illustrative example of the quantization effect and the corresponding gradient (using the Straight-Through estimator) for parameter update 5. (Middle) The true bitrate overhead (bluc) and its continuous proxy (orange) of (slab-only) Gaussian prior (a = 0) and their gradients with respect to unquantized 8. (Right) The true bitrate overhead (blue) and its continuous proxy (orange) of spike-and-slab prior (a = 4) and their gradients with respect to unquantized d. One can see how the effect of the spike (almost) fully disappears in the gradient of the quantized bitrate overhead, as the largest amount of mass of the spike distribution is part of the center quantization bin.
right shows the same figure but for model prior that includes a spike. Comparing true (dis- crete) model rate costs of the slab-only prior (middle-top), to these costs using the spike-and-slab prior (right-top), shows how the introduced spike reduces the number of bits to encode the zero- update (i.e. the center bin), at the cost of making larger updates more expensive in number of bits. Another interesting phenomena is visible when comparing the gradients of the discrete and continu- model rates for slab-only smiddle-bottom versus spike-and-slab prior night-bottoom. The effect of the spike almost fully disappears in the gradient of the discrete model rate. This is caused by the fact that most of the spike's ' mass is (by design) positioned inside the center quantization bin after quantization. Mathematically, this can be seen from filling in eq. (7) for the spike-and-slab prior:
Patab (5 + 1)) Paiat () + Pspike (5 + 2) Pspike (ō ())
V&M
Palab (6 < < + 5) (0 8- ()) + a P Papito (8 < 8 + ))) - a Pspike (6 <
(11)
We can distinguish different behavior of eq. 11 for two distinct ranges of 8:
=
Symmetry around zero-update:
Palab (8+b((++)) ())== Patab () and Pspike (8+ () Papiko (8- -)).
V.M
8 4
Since 1 = spike:
Pspiko (0+ 4 2 0,
Pspiko (0 (0-1)= R0 0,
Papiko < xz+5) 20 0,
P Papike < <x- O.
Pean( -)
V.M = Penl6=<++5) Ponnlloo
Note that this approximation becomes less tight for large a, as the small probabilities and cumulative densities of the spike distribution are multiplied by
From the 8- = case, one can see that inclusion of the spike leaves the gradient unbiased. The f O case shows that the spike does not influence the gradient in the limit when the spike's mass entirely positioned within the center quantization bin. As the standard deviation of the spiky Gaussian was chosen to be 8 a total of 0.3% of it's mass is in practice being quantized in an off- center quantization bin. This explains the slight increase of the off-center bins in the gradient of the discrete model rate costs in Fig.
4 (right- bottom.. Comparing the gradient of the discrete versus continuous model rate costs for the spike-and-slab prior in Fig. (right- we can see that the first order approximation between the two introduces a larger error than in the slab-only case (Fig. This can be explained by the fact that the introduced spiky Gaussian has a higher tangent due to its support being more narrow than that of the Gaussian slab. Nevertheless, the use of the continuous model rate loss is preferred for finetuning as it (much more) strictly enforces zero- updates (thanks the present gradient peaks around the center bin) than its discrete counterpart.
4
13

---

Published as a conference paper at ICLR 2021
B SAMPLE SELECTION FROM XIPH DATASET
XIPH DATASET
The Xiph test videos can be found
oipre//r/eiooo/eeeee/edrrrr Like Rip-
pel et al. (2019) we select all 1080p videos, and exclude cumputer-generatee videos and videos with
inappropriate licences, which leaves with the following videos:
aspen 1080p
ducks take of 1080p50
red kayak 1080p
speed bag 1080p
blue.aky.1080pe Setrroleel.bunn.0053 crowd_run 1080p50 dinner_1
In.to.tree 1080p50
civerbed.1080p23 fush.field.cute s_100Cp
statioo2_1080ee sunflower.1080p2
old.2om.ccoss.1080P park,joy 100_00p50 poddsttrion.orro.11000000
rush_hour_1080p2
tractor_1080p2.
anov.mrt_1080p
west_wwindjeaay.10088
B.2 XIPH-5N 2 FPS DATASET
Due to computational limits, we draw sample of five videos, which we refer to the Xiph-5N dataset. When drawing such a small sample randomly, a high probability arises of drawing an unrepresentative sample, including for example too many videos with either low high finetuning potential. To alleviate this problem, we use the following heuristic to select N(= 5) videos:
1. Evaluate the global model's RD performance all videos for all values of B.
2. Per 33 value, rank all videos based on their respective LRD loss.
3. For each video, average the CRD rank over all values of B.
4. Order all videos according to their average rank, and select N videos based on N + 1 evenly spaced percentiles.
The global model's RD performance for all videos from the Xiph dataset is shown in Figure 5 The five videos part Xiph-5N are indicated with colors, and Table provides more details about these five videos. The column titled RD-tank percentile shows the actual percentile at which the selected videos are ranked. The computed (target) percentiles for N are 1/6, 2/6, .... 5/6. For each percentile selected the video closest these target percentiles. The last column denotes the number of I-frames after subsampling to fps. Videos which the original sampling frequency was not integer divisible by a factor 2, were subsampled with factor resulting in the I-frame sampling frequency closest to 2 fps.
Xiph 5m Datapoint Selection
42.5
400
375
dataset
Figure 5: RD performance global baseline for all datapoints Xiph and Xiph-5N.
Video
RD-rank percentile
Target percentile
width height
Original fps
Duration (s)
Nr. of I-frames
frames
fps
in.to.tree
0.22
0.167
1920 x 1080
10.0
20
aspen
0.37
0.333
1920 x 1080 x 570
30
19.0
38
controlled
0,50
0.500
1920 x 1080 570
30
19.0
38
sunflower
0.69
0.667
1920 x 1080 500
25
20.0
42
podestrian.area
0.82
0.833
1920 x 1080
25
15.0
32
AVERAGE
0.52
0.500
1920 x 1080
16.6
34
Table 2: Characteristics of the five selected videos in Xiph-5N 2fps.
14

---

Published as a conference paper at ICLR 2021
C GLOBAL MODEL ARCHITECTURE AND TRAINING
For our neural compression model, we adopt the architecture proposed by BBllé et al. (0018), in- cluding the mean-scale prior from Minnen et (2018). We use shared hyperdecoder to predict the mean and scale parameters, Like (Ballé et al. 2018 we use model architecture with fewer pa-
rameters for the low bitrate regime (32 le- 1e-3). Table indicates both the model architecture and the number of parameters, grouped per sub-model. The upper row in this table links the terminology proposed by [Balé et al. (2118) conventional VAE terminology which we follow in this work.
Figure 6 provides a visual overview of the model architecture, where we use and 21 to indicate the latent and hyper-latent space respectively (referred to as y and z in the original paper of BBall et al. (018)). Note that even though we adopt hierarchical latent variable model, we simplity the notation by defining a single latent space 2 {z1, 22} throughout this work.
During training adopt mixed quantization strategy, where the quantized latent are used to calculate the distortion loss (with their gradients estimated using Straight-Trone estimator from Bengio et al. (2013)) while
noisy samples for 21, #2 when computing the rate loss.
Optimization of eq. ① on the training fold of the CLICI9 dataset, was done using the Adam opti- mizer with default settings (Kingma & Ba 2014), and took 1.5M steps for the low bitrate models, and 2.0M steps for the high bitrate models. Each step contained 8 random crops of 256 x 256 pixels, and the initial learning rate
to 1e-4 4 and lowered to le 5 at 90% of training.
Codec
X
2
Z2
decoder po(a)
Z
Hyper- codec
encoder P2(o,1ng) 1b2)
latent prior (at)))))))
Hyper- prior
latent prior (t,).)
Figure 6: The mean-scale hyperprior model architecture visualized using the VAE framework.
Table 3: Model architecture and parameter count for the adopted hyperprior model. As suggested by [allé et al. 2018), distinct architecture is used for the low and high bitrate regime. We report the number of output channels per layer, for the exact model architecture we refer Ballé et al. (2018). Note that the last column refers to the total number of parameters that needs to be known at the receiver side.
Transmitter
Receiver
Encoder qo (2 xx)
Hyper Encoder
Hyperprior (2))))
Hyper Decoder
Decoder Po (a: 22)
Nr. of receiver parameters
lo (21 (22)
(22
Low bitrate model
Layers x Output channels Parameter count
4x192 2.89M
3x128 1.04M
3x3 5.50k
2x + Ix256
3x192 1x3 2.89M
1.26M
4.16M
High bitrate model
Layers x Output channels
4x320
3x192 2.40M
3x3
2x192 + 1x384
3x320 1x3
Parameter count
8.01M
8.26k
2.95M
8.01M
10.97M
15

---

Published as a conference paper at ICLR 2021
D TEMPORAL ABLATION
In this experiment we investigate the tradeoff between the number frames that the model fine- tuned on, and the final RDM performance. The higher this number of frames, the higher (po- tentially) the diversity (making finetuning more difficult), but the lower the bitrate overhead (in bits/pixels) due model updates. This ablation repeats our main experiment the video (Fig. for a varying number of I-frames.
We sample number of frames (equispaced) from the full video, starting the zero' th index. The experiment is run for f E {1, 2,2,,, 5, 10, 25, 50, 100, 250, 500), and the two outmost rate-distortion tradeoffs: 3 (0.003, 0.0001): ). Note that the original experiment was done with frames sampled at 2 fps, resulting in f 42 for the sunfl flwerr video.
Lowest Bitrate (8 = 0.003)
Highest Bitrate ($ 0.0001)
2.3
352
3.09
foxnel koodin
PSM
0.5
030
Nulioodd Eietuing: Kerky
0008
3.62
Mesttiettee Ful Boloopalibabbnng 2n/
13
100
220
500
20
25
56
100
258
500
Number Frames?
Frames
Figure 7: Compression performance as a function of number of finetuning frames for the sunflower video. The dashedd red line indicates sampling at fps (used in our main experiments).
Figure shows (for the low and high bitrate region) the total RDM loss, and its subdivision in distortion and the different rate terms, as a function of numbers of finetuning frames. Full-model finetuning outperforms encoder-only finetuning in all cases, except for regime. In this case, the model rate M is causing the total rate R + M to become too high to be competitive with the baselines. This in turn is mainly caused by the initial cost My. which can only be amortized over single frame. In general, for other values of f. these initial costs were found to contribute only little to the total rate (with even a negligible contribution in the low and high bitrate 25 and f 10).
in the low bit-rate
regions for respectively
Note that the global model's performance varies noticeably as a function of f. Apparently, the first frame of this video is easy to compress, therewith lowering the total loss for small sets of I-frames. To make fair comparisons, one should thus only consider relative performance of encoder-only and full-model finetuning with respect the RD trained baseline. In line with main findings in Fig. 2b. full-model finetuning shows the biggest improvements for the high bitrate setting.
Interestingly, when comparing the low and high bitrate regimes, the total relative RDM gain of full-model finetuning follows similar pattern for varying values of f (higher gain for higher f). However, the subdivision of this gain in rate and distortion gain differs due to leveraging another tradeoff setting S. For the high bitrate, mainly distortion is diminished (row 2), whereas for the low bitrate, rate predominantly reduced (row 3). These rate and distortion reduction plots clearly show how the flexibility of full-model (compared to encoder-only) finetuning can improve results in various conditions.
This experiment has shown that the potential for full-model finetuning (under the current model architecture and prior) seems highest for video compression purposes, as gains are negative (due to relative high static initial costs in the low bitrate regime) or only marginal (in the high bitrate regime) when overfitting on a single frame. Yet, we hypothesize that full-model finetuning could still be useful for (single) image compression well, given other choices for the model architecture and/or model prior. Also, the provided ablation run on one video only, so further research is needed investigate full-model finetuning in image compression setup.
16

---

Published as a conference paper at ICLR 2021
E RATE-DISTORTION FINETUNING PERFORMANCE PER VIDEO
Figure shows the RDM plots for the different videos after finetuning for 100k steps. Full-model finetuning outperforms the global model, encoder-only and direct latent optimization for all videos. The blue lines, indicating global model performance, differ per video, which might influence the finetuning gains, which also differ per video, e.g. controll wowoleed.burr versus sunflowe: True entropy-coded results are used to create these graphs, rather than the computed RDM values. Devi- ations between entropy-cooe cooed and computed rates were found to be negligible (mean deviation was 1.94e-04 bits/pixel for and 1.06e-03 bits/pixel for M). Throughout this paper, all training graphs and ablations therefore provided using the computed values, rather than the entropy coded sults.
Figure shows for each of these videos the finetuning progression training steps. Also here, differences in performance are visible among videos. The videos that result in highest finetuning gains, e.g. sunfl ower, show quicker performance improvement after the start of finetuning, and also continue to improve more over time.
video aspen 2fps (38 frames)
video controlled burn 2dps (38 frames)
video in to tree 2fps (20 frames)
PSNR (dB3
30
36
32
30
pedestrian, area 21ps Trames))
video sunflower 2lps (42 frames)
42
R M (bits pixel)
40
Baseline
PSNR (dB)
30
Trained
Instance Adaptive Models
Encoder-Oriy Finetuning (8
34
Direct Latent Finsturing (M -01
32
Finctuning (R
30
0.0
0.5
1.0
1.5
2.0 0.0
0.5
1.0
1.5
2.0
pixel)
(bits pixel)
Figure 8: RDM performance for instance-adaptive encoder-only and full-model finetuning, com- pared with the performance of the global model, split per video. The instance-adaptive models used to create each graph are finetuned on the corresponding single video.
video aspen 2fps (38 frames)
video controlled burn 2fps (38 frames]
tree Video in tree 2fps (20 framest
44
30
37
36
PSNN (0B)
38
38
34
36
33
32
32
30
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
1.0
1.2
1.4
pixel)
video pedestrian
frames)
videa sunflower 2fps frames)
42
43
42
Training Step
PSNR (d6)
40
40000
80000
36
R A (bits pixel)
R + (bits pixel)
Figure 9: Progression of finetuning over time for all videos in Xiph-5N 2fps.
17

---

Published as a conference paper at ICLR 2021
F
MODEL UPDATES DISTRIBUTIONS
showss how the (quantized) model updates become much sparser (top row) when finetuning includes the spike-and-slab model rate loss M, compared unregularized finetuning (bottom row).
HyperPrior
HyperCodec. Dec Weight HyperCodec. Dec. Bias
Codec. Dedec..Dc..hee Weight
Codec. Dec. Bias
Codec. DeN.
Count
Count
Figure 10: Histograms showing the distribution of quantized model updates ă when finetuning with (top row) and without (bottom row) the model rate regularizer M.
18


Reviews:
[{"review_id": "cYTAWjKYttx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\nThis paper extends neural compression approaches by fine-tuning the decoder on individual instances and including (an update to) the decoder in the bit-stream for each image/video. The proposed approach is evaluated on the UVG dataset and the authors find a 1db improvement (PSNR) relative to their own baseline.\n\n\nQuality (5/10)\n----------\nThe proposed approach is sound and it would have been interesting to see the gains which can be achieved by fine-tuning the decoder of common neural compression approaches. Unfortunately, the few results provided in the paper are not just failing to answer this question but are misleading. By choosing a weak baseline, the reader is led to believe that fine-tuning a decoder will lead to large gains when realistic models are likely to benefit significantly less.\n\nThe authors motivate their simple baseline by noting that their approach is \"model-agnostic\". However, while the approach is model-agnostic, the results and conclusions are not. And it is mostly the empirical results which will be of interest to the reader. (A reader familiar with compression will be very well aware that a neural decoder _could_ be included in the bit-stream, making the conceptual contributions less interesting.)\n\nThe evaluated model encodes each frame of a 600 frame video sequence _independently_. A more realistic decoder would be conditioned on information in previously encoded frames, changing its behavior. It is reasonable to expect that similar change in behavior is encoded in the model updates. That is, the proposed approach is likely less effective in a more realistic setting.\n\nIf model complexity was a concern, the authors could have evaluated their approach on images instead of videos. The results would have looked less impressive but would have been more useful. Alternatively, they could have chosen a different video compression architecture of low complexity but one which is still practically relevant. E.g., one motivated by computational constraints.\n\n\nSignificance (4/10)\n----------------\nNeural compression is of interest to many people in the the ICLR community and exploring the fine-tuning of decoders would be a useful contribution to this field. The significance of this contribution is only limited by the lack of a meaningful results.\n\n\nOriginality (4/10)\n--------------\nIncluding model information in the bit-stream is an old idea in compression and not limited to neural compression. For example, Netflix is optimizing their classical video codecs at a \"shot\" level. Even JPEG (1992) allows us to fine-tune the Huffman table for an individual image (\"optimized JPEG\").\n\nIt is also common for compression challenges to require the model to be included in the bit-stream (e.g., the Hutter prize or the P-frame challenge of CLIC 2020).\n\nMany papers have been written on the related topic of _model compression_ (e.g., Han et al., 2016), which should at least be acknowledged. Compressed model updates are also used in parallelized implementations of SGD (e.g., Alistarh et al., 2017).\n\n\nClarity (8/10)\n---------\nThe paper is well written and clear.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TREz-LxdknP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\n\nThe paper describes an instance specific finetuning method for image and video compression including finetuning the decoder. Based on the shown experiments, the required additional bits for sending the updated finetuned model parameters are worth the achieved increase in RD performance. However, the method has only been evaluated on one video dataset and with respect to its own baseline and not with respect to any other existing method.\n\n**Strength**\n\n= Method which also considers to finetune/adapt the decoder side of image compression network, for improved performance. \n\n= Paper is self-contained by recapping the necessary basic formulations.\n\n**Weakness**\n\n= Method has only been evaluated with respect to its own baseline method (image compression model without finetuning).\n\n= Method has only been evaluated on one video dataset, but by compressing frame by frame, therefore not taking advantage of temporal redundancy. \n\n= Given that it is an image compression method, the proposed instance adaptive method could also be evaluated on the e.g. clic validation set.\n\n*Some open questions*\n\nIs $\\bar{M}$ computed for the whole video and averaged per frame for the results in Table 1 and therefore dependent on the length of the video?\n\nDo the authors have some intuition, why some videos are easier to finetune than others?\n\n*Minor*\n\nReferences of arxiv papers, which have been published before submission deadline, can be updated with the respective conference.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IsBt0RHOup", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of per-instance model adaptation for neural data compression, and proposes a new method for end-to-end finetuning the model that is quantization-aware, by introducing an additional term that measures the compression cost of model update to the typical rate-distortion loss. Evaluation on the UVG dataset shows encouraging performance, with an average distortion improvement of approximately 1 dB for the same bit rate compared to the naive baseline (without fine-tuning).\n\n------------------------\n\nPros:\n1. The paper is well written and concepts are clearly explained.\n2. The method is sound, and incorporating the entropy cost of model update during fine-tuning offers a conceptually appealing (and likely more performant, though not empirical verified (see below)) approach compared to previous methods (Lam et al., 2020, Zou et al., 2020) that tackles model update quantization after fine-tuning.\n\n------------------------\n\nCons:\nThe experiment section is the weakest point. Particularly:\n1. It's unclear from the description if the evaluation on UVG actually \"adapts the entire model to a single data instance\" (i.e., *for each image*) as claimed, or amortizes the model update cost over a batch of all the images in a video. The paper claims that \"In this paper we consider the extreme case where the domain of adaptation is a single instance, resulting in costs for sending model updates which become very relevant\", but this would highly misleading if all the experiments were conducted in a batch compression setting.\n2. If the experiment did perform per-instance model adaptation, then it would be much more convincing to evaluate on standard datasets like Kodak and Tecnick from the image compression literature, instead of frames of UVG videos.\n3. Since the paper's contribution is about improving the existing fine-tuning strategy that tackles model update quantization after fine-tuning (e.g., Zou et al., 2020), the proposed method should then also compare to these baselines to really assess its performance.\n4. It would also be interesting to compare with approaches that optimize the encoded latents (e.g., Yang et al., 2020), which also achieve close to 1 PSNR improvement at equal bitrate without the overhead of decoder updates.\n\n------------------------\n\nQuestions:\n1. Can the author comment on how \"the quantization bin width t and standard deviation σ of p[\\bar δ]\" (Sec 4.3) are chosen? How sensitive is the compression performance to their choice, e.g., is it possible to discretize so finely that no amount of RD improvement can overcome the model update cost?\n2. The use of the continuous density for the M (model update cost) term in Eq 2 is established in the Appendix A by showing that the gradient of the discrete cost \\bar M has the same gradient (up to first order) as that of -log p(δ) based on the density p(δ). Did I understand this correctly?  But M = -log p(δ) doesn't actually give an estimate of the cost after discretization \\bar M = -log p[\\bar δ]. Instead, the typical thing to do in literature (due to Balle et al.) is to actually minimize -log p[\\bar δ], where \\bar δ = round(δ), and the rounding can be either approximated by uniform noise injection or STE.   Can the authors comment on this choice of their method?\n\n------------------------\n\nTypos and minor mistakes/fixes:\n1. p. 2, under eq (1): The R-D loss is equivalent to the *negative* ELBO in VAEs;\n2. Does Figure 3 bottom show the histogram of bit allocation for \\bar δ? If so then the caption can just say \"Bottom: histogram of bit allocation for \\bar δ\" as it's clearer.\n\n------------------------\nUpdate after author response:\n\nI have increased my score in light of the substantial improvement to the manuscript and experiments.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3ffXrnHKWD9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates how to improve the test time performance of learned image compression models through finetuning of the full model. The authors finetune the model (both the model parameters and the prior on the latent space) for every test-time instance, appending the model updates to the bitstream. The model updates are coded according to a discretised, mean-zero Gaussian distribution with a single learned variance. They demonstrate that this approach yields a superior rate-distortion curve than the non-finetuned model on a set of I-frame video data.\n\nOverall I like the paper. It is clearly and simply written, with good motivation given for the concepts introduced. The method itself is also straightforward to understand, and seems like a sensible approach. Although on the surface the idea of doing instance-specific fine-tuning might seem to be impractical, it benefits from the fact that the extra encoding time of fine-tuning the model is paid by the sender. The receiver only has to pay the extra cost of decoding the model updates, which is fast if the coding distribution is factored (as it is in this paper). These asymmetrical coding times are often acceptable, as the authors note, since encoding-decoding is often a one-to-many relation.\n\nI think the results demonstrated by the method are positive enough to warrant the extra overhead introduced, with a ~1dB gain for a given bitrate. I also appreciate the breakdown of where the extra model delta bits are allocated as per Figure 3, and the visualisation of the training performance in Figure 2b. I think these give a nice feel for the way the method works and the finetuning progresses on this particular instance.\n\nDo the authors have any comment on why the encoding-only finetuning yields barely any benefit, as shown in Figure 2a? My interpretation might be that finetuning only the encoder is sub-optimal because the latent prior is fixed. The prior will have been learned jointly with the encoder on the global model, such that the encoder maps to parts of the latent space that the prior assigns mass to. As such, if the prior is fixed and you then finetune the encoder, the encoder still has to map to parts of space that are assigned mass in order to avoid the rate becoming too large. It might be interesting to see the results if the encoder and prior are finetuned but not the decoder. Although if you are finetuning (and communicating side information for the prior updates) then it is probably very little extra cost to also update the decoder. The results also seem to indicate that most bits for the model updates are spent on the decoder weights.\n\nI also think it would have been good to include results using a learned prior to code the model updates, not a Gaussian. The authors do mention this as a possibility in the discussion, but surely it would have been very easy to implement? Given that they are already doing so for the latent space itself. Another small point about the Gaussian quantisation, is that an alternative discretisation is that of assigning equal mass to all bins, as per https://arxiv.org/abs/1901.04866 (see Appendix B). This results in simple coding - the discrete distribution is uniform, since the bins all have equal mass - and ensures that the discretisation is appropriate for the Gaussian.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper suggests a procedure to efficiently adapting a learned neural compression model to a new test distribution. If this test distribution has low entropy (e.g., a video as a sequence of interrelated frames), large compression gains can be expected. To achieve these gains, the method adapts the decoder model to the new instance, transmitting not only the data but also a compressed model update. Experiments are carried out on compressing I-frames from videos, while comparisons comprise baseline approaches that finetune the latent representations of videos as opposed to the decoder. 

The paper’s main contribution is very timely and relevant. While it was well-known in the classical compression literature that model updates could be sent along with the data (e.g., as already done in “optimized JPEG”), this is the first time the idea was implemented in neural compression. The experiments are arguably the paper’s weaker part and were originally a concern, but they have been significantly improved during the review period such that all reviewers voted for acceptance. We encourage the authors to further strengthen their experimental results by adding more challenging baselines on well-established tasks (e.g., image compression).

Decision: Accept (Poster)