Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary**

The paper describes an instance specific finetuning method for image and video compression including finetuning the decoder. Based on the shown experiments, the required additional bits for sending the updated finetuned model parameters are worth the achieved increase in RD performance. However, the method has only been evaluated on one video dataset and with respect to its own baseline and not with respect to any other existing method.

**Strength**

= Method which also considers to finetune/adapt the decoder side of image compression network, for improved performance. 

= Paper is self-contained by recapping the necessary basic formulations.

**Weakness**

= Method has only been evaluated with respect to its own baseline method (image compression model without finetuning).

= Method has only been evaluated on one video dataset, but by compressing frame by frame, therefore not taking advantage of temporal redundancy. 

= Given that it is an image compression method, the proposed instance adaptive method could also be evaluated on the e.g. clic validation set.

*Some open questions*

Is $\bar{M}$ computed for the whole video and averaged per frame for the results in Table 1 and therefore dependent on the length of the video?

Do the authors have some intuition, why some videos are easier to finetune than others?

*Minor*

References of arxiv papers, which have been published before submission deadline, can be updated with the respective conference.

review_title: Instance specific finetuning method for image and video compression but with weaknesses in the experimental section

final_score: rating: 7: Good paper, accept
review_content: This paper investigates how to improve the test time performance of learned image compression models through finetuning of the full model. The authors finetune the model (both the model parameters and the prior on the latent space) for every test-time instance, appending the model updates to the bitstream. The model updates are coded according to a discretised, mean-zero Gaussian distribution with a single learned variance. They demonstrate that this approach yields a superior rate-distortion curve than the non-finetuned model on a set of I-frame video data.

Overall I like the paper. It is clearly and simply written, with good motivation given for the concepts introduced. The method itself is also straightforward to understand, and seems like a sensible approach. Although on the surface the idea of doing instance-specific fine-tuning might seem to be impractical, it benefits from the fact that the extra encoding time of fine-tuning the model is paid by the sender. The receiver only has to pay the extra cost of decoding the model updates, which is fast if the coding distribution is factored (as it is in this paper). These asymmetrical coding times are often acceptable, as the authors note, since encoding-decoding is often a one-to-many relation.

I think the results demonstrated by the method are positive enough to warrant the extra overhead introduced, with a ~1dB gain for a given bitrate. I also appreciate the breakdown of where the extra model delta bits are allocated as per Figure 3, and the visualisation of the training performance in Figure 2b. I think these give a nice feel for the way the method works and the finetuning progresses on this particular instance.

Do the authors have any comment on why the encoding-only finetuning yields barely any benefit, as shown in Figure 2a? My interpretation might be that finetuning only the encoder is sub-optimal because the latent prior is fixed. The prior will have been learned jointly with the encoder on the global model, such that the encoder maps to parts of the latent space that the prior assigns mass to. As such, if the prior is fixed and you then finetune the encoder, the encoder still has to map to parts of space that are assigned mass in order to avoid the rate becoming too large. It might be interesting to see the results if the encoder and prior are finetuned but not the decoder. Although if you are finetuning (and communicating side information for the prior updates) then it is probably very little extra cost to also update the decoder. The results also seem to indicate that most bits for the model updates are spent on the decoder weights.

I also think it would have been good to include results using a learned prior to code the model updates, not a Gaussian. The authors do mention this as a possibility in the discussion, but surely it would have been very easy to implement? Given that they are already doing so for the latent space itself. Another small point about the Gaussian quantisation, is that an alternative discretisation is that of assigning equal mass to all bins, as per https://arxiv.org/abs/1901.04866 (see Appendix B). This results in simple coding - the discrete distribution is uniform, since the bins all have equal mass - and ensures that the discretisation is appropriate for the Gaussian.
review_title: A nice idea, well communicated

final_score: rating: 7: Good paper, accept
review_content: This paper considers the problem of per-instance model adaptation for neural data compression, and proposes a new method for end-to-end finetuning the model that is quantization-aware, by introducing an additional term that measures the compression cost of model update to the typical rate-distortion loss. Evaluation on the UVG dataset shows encouraging performance, with an average distortion improvement of approximately 1 dB for the same bit rate compared to the naive baseline (without fine-tuning).

------------------------

Pros:
1. The paper is well written and concepts are clearly explained.
2. The method is sound, and incorporating the entropy cost of model update during fine-tuning offers a conceptually appealing (and likely more performant, though not empirical verified (see below)) approach compared to previous methods (Lam et al., 2020, Zou et al., 2020) that tackles model update quantization after fine-tuning.

------------------------

Cons:
The experiment section is the weakest point. Particularly:
1. It's unclear from the description if the evaluation on UVG actually "adapts the entire model to a single data instance" (i.e., *for each image*) as claimed, or amortizes the model update cost over a batch of all the images in a video. The paper claims that "In this paper we consider the extreme case where the domain of adaptation is a single instance, resulting in costs for sending model updates which become very relevant", but this would highly misleading if all the experiments were conducted in a batch compression setting.
2. If the experiment did perform per-instance model adaptation, then it would be much more convincing to evaluate on standard datasets like Kodak and Tecnick from the image compression literature, instead of frames of UVG videos.
3. Since the paper's contribution is about improving the existing fine-tuning strategy that tackles model update quantization after fine-tuning (e.g., Zou et al., 2020), the proposed method should then also compare to these baselines to really assess its performance.
4. It would also be interesting to compare with approaches that optimize the encoded latents (e.g., Yang et al., 2020), which also achieve close to 1 PSNR improvement at equal bitrate without the overhead of decoder updates.

------------------------

Questions:
1. Can the author comment on how "the quantization bin width t and standard deviation σ of p[\bar δ]" (Sec 4.3) are chosen? How sensitive is the compression performance to their choice, e.g., is it possible to discretize so finely that no amount of RD improvement can overcome the model update cost?
2. The use of the continuous density for the M (model update cost) term in Eq 2 is established in the Appendix A by showing that the gradient of the discrete cost \bar M has the same gradient (up to first order) as that of -log p(δ) based on the density p(δ). Did I understand this correctly?  But M = -log p(δ) doesn't actually give an estimate of the cost after discretization \bar M = -log p[\bar δ]. Instead, the typical thing to do in literature (due to Balle et al.) is to actually minimize -log p[\bar δ], where \bar δ = round(δ), and the rounding can be either approximated by uniform noise injection or STE.   Can the authors comment on this choice of their method?

------------------------

Typos and minor mistakes/fixes:
1. p. 2, under eq (1): The R-D loss is equivalent to the *negative* ELBO in VAEs;
2. Does Figure 3 bottom show the histogram of bit allocation for \bar δ? If so then the caption can just say "Bottom: histogram of bit allocation for \bar δ" as it's clearer.

------------------------
Update after author response:

I have increased my score in light of the substantial improvement to the manuscript and experiments.
review_title: Good idea and sound method, but experiments can be better executed.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
-------------
This paper extends neural compression approaches by fine-tuning the decoder on individual instances and including (an update to) the decoder in the bit-stream for each image/video. The proposed approach is evaluated on the UVG dataset and the authors find a 1db improvement (PSNR) relative to their own baseline.


Quality (5/10)
----------
The proposed approach is sound and it would have been interesting to see the gains which can be achieved by fine-tuning the decoder of common neural compression approaches. Unfortunately, the few results provided in the paper are not just failing to answer this question but are misleading. By choosing a weak baseline, the reader is led to believe that fine-tuning a decoder will lead to large gains when realistic models are likely to benefit significantly less.

The authors motivate their simple baseline by noting that their approach is "model-agnostic". However, while the approach is model-agnostic, the results and conclusions are not. And it is mostly the empirical results which will be of interest to the reader. (A reader familiar with compression will be very well aware that a neural decoder _could_ be included in the bit-stream, making the conceptual contributions less interesting.)

The evaluated model encodes each frame of a 600 frame video sequence _independently_. A more realistic decoder would be conditioned on information in previously encoded frames, changing its behavior. It is reasonable to expect that similar change in behavior is encoded in the model updates. That is, the proposed approach is likely less effective in a more realistic setting.

If model complexity was a concern, the authors could have evaluated their approach on images instead of videos. The results would have looked less impressive but would have been more useful. Alternatively, they could have chosen a different video compression architecture of low complexity but one which is still practically relevant. E.g., one motivated by computational constraints.


Significance (4/10)
----------------
Neural compression is of interest to many people in the the ICLR community and exploring the fine-tuning of decoders would be a useful contribution to this field. The significance of this contribution is only limited by the lack of a meaningful results.


Originality (4/10)
--------------
Including model information in the bit-stream is an old idea in compression and not limited to neural compression. For example, Netflix is optimizing their classical video codecs at a "shot" level. Even JPEG (1992) allows us to fine-tune the Huffman table for an individual image ("optimized JPEG").

It is also common for compression challenges to require the model to be included in the bit-stream (e.g., the Hutter prize or the P-frame challenge of CLIC 2020).

Many papers have been written on the related topic of _model compression_ (e.g., Han et al., 2016), which should at least be acknowledged. Compressed model updates are also used in parallelized implementations of SGD (e.g., Alistarh et al., 2017).


Clarity (8/10)
---------
The paper is well written and clear.
review_title: Misleading results [updated]

Meta-review: The paper suggests a procedure to efficiently adapting a learned neural compression model to a new test distribution. If this test distribution has low entropy (e.g., a video as a sequence of interrelated frames), large compression gains can be expected. To achieve these gains, the method adapts the decoder model to the new instance, transmitting not only the data but also a compressed model update. Experiments are carried out on compressing I-frames from videos, while comparisons comprise baseline approaches that finetune the latent representations of videos as opposed to the decoder. 

The paper’s main contribution is very timely and relevant. While it was well-known in the classical compression literature that model updates could be sent along with the data (e.g., as already done in “optimized JPEG”), this is the first time the idea was implemented in neural compression. The experiments are arguably the paper’s weaker part and were originally a concern, but they have been significantly improved during the review period such that all reviewers voted for acceptance. We encourage the authors to further strengthen their experimental results by adding more challenging baselines on well-established tasks (e.g., image compression).

Decision: Accept (Poster)