Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set

Abstract: We introduce a video compression algorithm based on instance-adaptive learning. On each video sequence to be transmitted, we finetune a pretrained compression model. The optimal parameters are transmitted to the receiver along with the latent code. By entropy-coding the parameter updates under a suitable mixture model prior, we ensure that the network parameters can be encoded efficiently. This instance-adaptive compression algorithm is agnostic about the choice of base model and has the potential to improve any neural video codec. On UVG, HEVC, and Xiph datasets, our codec improves the performance of a low-latency scale-space flow model by between 24% and 26% BD-rate savings, and that of a state-of-the-art B-frame model by 17 to 20% BD-rate savings. We also demonstrate that instance-adaptive finetuning improves the robustness to domain shift. Finally, our approach reduces the capacity requirements on compression models. We show that it enables a state-of-the-art performance even after reducing the network size by 72%.

Reviews and author responses:

 Thank the author for replying to my questions and targeted modifications.

Only applied experiments, in principle, the workload is not enough, and there is a lack of more in-depth theoretical analysis. However, the author has also made some contributions to promote the development of small model application in the field  of video compression. I would like to keep the initial rating.

 We thank the reviewer for their careful review and helpful comments and suggestions. We are glad that they find our experiments and the empirical results convincing and found the manuscript well-written.

> How many optimization steps are used to produce the final reported performance in Figure.1 as well as in some other figs and tables.

As can be seen in Figure 2, the RD performance of our method and the ffmpeg baselines is a function of encoding time. For all figures and tables, we selected the best possible results as shown in Figure 2. For our method, this corresponded to an average finetuning time of 30 minutes/frame, or about 300 000 steps. Note however that the biggest performance improvements happen quickly, and we clearly outperform the baseline models after finetuning only for a few seconds per frame.

> The proposed method looks stronger at high bitrate but close to the baselines at low bitrate.

This is a good point. We clarified this in the manuscript and added a new Figure 4 to the appendix to highlight the relative performance between different methods. Indeed, our method performs stronger at higher bitrates when compared to classical codecs (ffmpeg and HM), Klopp et al. (2021) (which is based on a classical codec and an instance-adaptive neural enhancement method), and ELF-VC (Rippel et al., 2021) (a neural codec that does particularly well in the low-bitrate region). On the other hand, the improvements over the base model (SSF and B-EPIC) and other neural baselines are relatively constant over the full range. This makes us optimistic that with a different base model, for instance ELF-VC, InstA could become even better in the low-bitrate region. At extremely low bitrate, InstA will however be limited by the model rate overhead. 

> What is the precise bitrate range used for BD-rate comparison.

We use the largest PSNR range for which we have results for all main methods and baselines. We have added the precise numbers to Appendix E. The corresponding bitrate ranges depend on the method. For example, for InstA-SSF18 on HEVC class B, this range is 0.03 bpp to 0.59 bpp. 

>  a related work about implementing content adaptive algorithm in learned video compression is suggested for discussion or comparison

We thank the reviewer for suggesting this related work, and we have included it in our related work section as well as in our experimental evaluation. It is closely related to our "encoder-finetuned" baseline, but uses a different base model. Our empirical results show that full-model finetuning leads to larger gains than encoder-only finetuning (as in Lu et al), despite the additional rate overhead from model updates to the prior and decoder.

> The paper is well written in general. However, the overall contribution may not be sufficient enough to reach the acceptance threshold.

It is true that we do not develop a new algorithm or architecture in this paper. Nevertheless, we believe that there is substantial empirical novelty in our work:
  1. Our work is the first to apply instance-adaptive finetuning of a full compression model to complete video sequences.
  2. We demonstrate large empirical performance improvements over the base model (and in some settings and bitrate regions SOTA results). It was a priori not clear how well this idea would work, as optical-flow-based video codecs are already highly efficient.
  3. We show that instance-adaptive finetuning lets us use smaller networks and reduce decoder-side compute by 72 % with only a minimal reduction in compression performance (still outperforming SSF). Decoder-side compute is a major bottleneck in real-life applications, especially on mobile devices; such a reduction can have a large impact.
  4. We also provide additional valuable insight not included in van Rozendaal et al. (2021). We provide an extensive experimental analysis of the properties and trade-offs of instance-adaptive video compression, including
      * experiments that quantitatively analyze the trade-off between encoder-side compute and compression performance (Figure 2),
      * experiments to show how instance-adaptive compression can decrease the performance gap for out-of-domain data instances (Figure 3, 6, 7, 9),
      * experiments that show how robust this method is to different base model architectures (B-frame or P-frame neural codecs) and model size (Figure 1, 8),
      * experiments that analyze the trade-off for different finetuning strategies for longer videos (Figure 9), and
      * an analysis on the bitrate allocation across latents and model updates as well as across modules of the finetuned network (Figure 10, Table 7).

We thank the reviewer again for their comments. We have made a number of changes to our paper to address them and added some new results (see our top-level comment for an exhaustive list). We believe that this made our paper quite a bit better and hope that now makes sufficiently large contributions.


 Thank the authors for the modification of the paper and the response.

The answers from the authors addressed my concerns, especially the experimental results of the breakdown of the total rate.

However, I still think that the novelty and contribution are limited as the video compression architecture and the instance-adaptive scheme are both borrowed from previous works. Considering that it is an interesting attempt on video compression and it achieves good performance, and the experiments analyze lots of informative things, I would like to keep the initial rating as "6: marginally above the acceptance threshold"

This paper tried to close the domain gap between the training and testing dataset for the learned video compression. An instance-adaptive video compression approach is proposed by updating the model parameters of the video codecs and the optimal parameters are transmitted to the decoder. Besides, the authors modeled the updated model parameters under a mixture prior model, which further reduces the bit cost. Experimental results on several datasets demonstrated the effectiveness of this approach.  -Strengths
1. Good motivations. The domain gap between the training and testing datasets existed in the practical applications and it is important to solve this problem. 
2. The proposed approach is straightforward and neat. Transmitting the model parameters significantly improves the compression performance. 
3. Comprehensive analysis. A lot of results are provided in the ablation study and it is very helpful. 

-Weaknesses
1. Novelty. This paper extended the previous work Rozendaal et al. (2021) to video compression and may have limited novelty. The extension from image compression to video compression is expected. The major modification is the prior model for entropy coding, which is not enough. 
2. Another concern is computational complexity. This new approach increases a lot of computational burdens, which is another issue. This method is even slower than HM in some cases. 
3. The authors only evaluate the proposed method on the UVG and Class B datasets, which are two high-resolution(1920x1080) videos. How about the experimental results on other datasets, like Class c or D?
4. There are also some related works that should be discussed.   (a) Content Adaptive and Error Propagation Aware Deep Video Compression. Lu et al. (b) Efficient Video Compression via Content-Adaptive Super-Resolution. Khani et al.  (c) Online-Trained Upsampler for Deep Low Complexity Video Compression. Klopp et al. 

 The main concern is the novelty as a similar approach has been used in image compression. Basically, it is a borderline paper and I am open to other reviews. 

 We have uploaded another revision today. It includes two changes which are highlighted below.

## Model rate ablation
We have extended our analysis of the breakdown of the bitrate. The new Figure 11 shows how many bits are spent on updates to the various neural components. We specify the number of parameters, bits/parameter, and total rate in a single figure, where we can compare between InstA-SSF18 and InstA-SSF5. This highlights the following fact: Our InstA-SSF5 model learns to spend more bits per parameter to signal model updates. This allows it to achieve a competitive rate-distortion performance (around 20% BD-rate savings over Augustsson et al., 2021) despite its small size (72% less decoding operations than Augustsson et al., 2021).

## Extended related work section
We would like to thank our reviewers again for pointing us to various related works. We used their suggestions to extend our related work section. In particular, we have added a detailed categorization of neural instance-adaptive compression methods in Appendix A.

The activity in this field highlights the relevance of instance-adaptive video compression. Out of all works we are aware of, our paper presents the most scalable solution for instance-adaptive neural video codecs so far.

We once again thank the reviewers for their suggestions. We believe these two additional improvements further strengthen our manuscript. 


 We thank the reviewer for their thorough review and constructive suggestions. We are happy they found the motivation of our work convincing and enjoyed our method and empirical analysis.

> Novelty. This paper extended the previous work Rozendaal et al. (2021) to video compression and may have limited novelty.

Indeed, our model extends the algorithm presented in van Rozendaal et al. (2021). We do not present a new algorithm, nor do we introduce new architectures. However, we believe that there is substantial empirical novelty in our work:

  1. Our work is the first to apply instance-adaptive finetuning of a full neural compression model (not just the encoder) to an end-to-end neural video codec.
  2. We demonstrate large empirical performance improvements over the base model. It was a priori not clear how well this idea would work, as video codecs based on optical flow warping are already highly efficient, and we are able to provide an answer to this question.
  3. For the first time we provide evidence that instance-adaptive finetuning allows us to reduce decoder-side complexity with only a small reduction in compression performance. Decoder-side compute is a major bottleneck in real-life applications, especially on mobile devices; such a reduction can have a large impact.
  4. We are glad that the reviewer appreciated our empirical analysis. We see this as a key contribution of our work. In the original submission, we already presented an analysis of the trade-off between encoder-side compute and compression performance, an analysis of how instance-adaptive compression can decrease the performance gap on out-of-domain data, ablation studies covering different base architectures and model sizes, and an analysis of different finetuning strategies for longer videos. We now added a detailed breakdown of the contributions to the model rate and a study of the performance on lower-resolution data. We believe these analyses provide valuable insight beyond the algorithm developed in van Rozendaal et al. (2021).

> Another concern is computational complexity. This new approach increases a lot of computational burdens, which is another issue. This method is even slower than HM in some cases.

Yes, our new approach increases the computational complexity on the encoder side. However, our algorithm allows us to choose by how much. To get the absolute best compression performance, we indeed need to finetune for a long period. But our instance-adaptive video compression method can be stopped at any time. As can be seen in Figure 2, we can encode for a few seconds per frame and thus encode much faster than HM. In this case, the performance will not be as good as HM, but still clearly better than the base model.

Furthermore, we would like to emphasise that the compute overhead is almost entirely on the encoder side. On the decoder side, the overhead is minimal: the initial decoding of the model updates takes a fraction of a second. After that, the codec is as efficient as the base model. In fact, because instance-adaptive compression allows us to reduce the model size without a large drop in performance, it can even reduce the computational complexity on the decoder side.

Our work thus enables a strong asymmetry between encoding and decoding compute. This is certainly not ideal for all applications, but can be a perfect match for one-to-many transmission scenarios (e.g. Netflix).

> The authors only evaluate the proposed method on the UVG and Class B datasets, which are two high-resolution (1920x1080) videos. How about the experimental results on other datasets, like Class c or D?

We thank the reviewer for their suggestion and have added an evaluation on HEVC class C. This is an interesting dataset to compare to: we expect our method to perform worse on this lower-resolution data, as the model update overhead is amortized over fewer pixels.  Indeed, the performance compared to HM, ffmpeg, and FVC is not as good as on the other datasets. Nevertheless, InstA-SSF18 still clearly outperforms the SSF18 baseline model.

Please note that these results are preliminary, the instance-adaptive finetuning has only run briefly and has the potential to further improve (see Figure 2).

> There are also some related works that should be discussed.

We are grateful to the reviewer for pointing us to these related works. We have added them to our paper, see in particular the new Figure 4 in the appendix.

Lastly we noted that the reviewer gave a rating of 1 for technical correctness. We kindly ask the reviewer to point us to where they deem our work incorrect so that we can improve it adequately.

We would like to thank the reviewer again for the detailed and helpful comments. We incorporated them in our updated paper and added several new results (see our overall reply for an overview). These changes clearly improved our paper and we hope that it now presents a useful contribution to the neural video compression field.

 ## Summary

We would like to thank all the reviewers for their detailed comments, spot-on criticisms, and helpful suggestions. We are glad that they appreciated the key contributions of our work, in particular the motivation for instance-adaptive finetuning, the SOTA performance, and the detailed empirical analysis. We are humbled that they found our paper clearly written and appreciated the extensive supplementary materials. Below we address their main concerns and present some additional experiments and analyses.

## Novelty

The reviewers rightly note that the basic idea of our paper is to apply the instance-adaptive finetuning method as introduced in van Rozendaal et al. (2021) to video compression models. They express concern about the limited novelty of our work. Indeed, we do not present an entirely new algorithm or new architectures. However, we believe that there is substantial empirical novelty in our work:
  1. Our work is the first to apply instance-adaptive finetuning of a full neural compression model (not just the encoder) to video sequences.
  2. We demonstrate large empirical performance improvements over the base model (and in some settings and bitrate regions SOTA results). It was a priori not clear how well instance-adaptive compression would work on video data, as optical-flow-based video codecs are already highly efficient.
  3. We show that instance-adaptive finetuning lets us use smaller networks and reduce decoder-side compute by 72% with only a minimal reduction in compression performance (still outperforming SSF). Decoder-side compute is a major bottleneck in real-life applications, especially on mobile devices; such a reduction can have a large impact.
  4. Unlike van Rozendaal et al. (2021), we provide an extensive experimental analysis of the properties and trade-offs of instance-adaptive video compression including:
      * experiments that quantitatively analyze the trade-off between encoder-side compute and compression performance,
      * experiments to show how instance-adaptive compression can decrease the performance gap for out-of-domain data instances,
      * experiments that show how robust this method is to different base model architectures (B-frame or P-frame neural codecs) and model size,
      * experiments that analyze the trade-off for different finetuning strategies for longer videos, and
      * an analysis on the bitrate allocation across latents and model updates as well as across modules of the finetuned network.

## Datasets

We believe that one our experimental evaluation was already quite rigorous, with results on 4 datasets of high-resolution videos and several baselines. However, two reviewers suggested to also present results on lower-resolution sequences. We find this an excellent idea: even though these smaller resolutions are increasingly less common, they let us evaluate under which settings instance-adaptive video compression works. Low-resolution data is a challenging setting for our algorithm, as the model rate overhead is amortized over fewer pixels.

In the current version of the manuscript, we therefore added results on HEVC class C. (Because of limited compute, we did not add class D.) Indeed, our method does not outperform all baselines in this setting, but it still leads to a clear improvement over the SSF base model with 20% BD-rate savings. 

Please note that these results are preliminary, the instance-adaptive finetuning has only run briefly and has not converged yet and has the potential to further improve (see Figure 2). We will update these results later.

## Baselines

We are grateful that the reviewers pointed us to a number of related works that we missed. We added these to our paper. For six of these works, we found experimental results and added them as baselines to our empirical analysis. In order to prevent clutter, we have added Figure 4 that shows the relative rate differences between the models much more clearly. Between all these new baselines, our instance-adaptive video compression methods still achieve an impressive performance, especially at higher bitrates.

## Rate breakdown analysis

We added an in-depth  empirical analysis of the various contributions to the total bitrates, see Figure 10 and Table 7. We find that on most datasets, less than 1% of the total rate is spent on model updates. However, on out-of-domain data (in or case, animated sequences), the model learns to adapt more strongly and allocate up to 3% of the total rate to model updates.

## Conclusions

We thank the reviewers again for their helpful feedback. We have updated our manuscript, improved text and formatting, implemented many of their suggestions, and added new experiments, baselines, and analyses. We believe that this clearly improved our paper and hope that it now presents a useful contribution to the neural video compression community.

 We thank the reviewer for their careful review, positive feedback, and constructive suggestions. We are happy that they appreciated that InstA relaxes the need for matching distributions at train and test time and liked the potential for model-size reduction.

In the following, we will address the reviewer's criticisms and questions.

> a The key idea of this paper comes from the instance-adaptive data compression, which is only applied to the scale-space flow model and B-frame compression architecture. The method of this paper is lack of innovation.

The reviewer correctly points out that the method of the paper originates from that of van Rozendaal et al. (2021). Indeed, we do not claim that this paper presents algorithmic or architectural novelty. Instead, we believe that we made a contribution with substantial empirical novelty. We are glad that the reviewer noted our key contributions, in particular:
  1. Our work is the first to apply instance-adaptive finetuning of a full neural compression model (not just the encoder) to video sequences.
  2. We demonstrate large empirical performance improvements over the base model (and in some settings and bitrate regions SOTA results). It was a priori not clear how well instance-adaptive compression would work on video data, as video codecs based on optical flow warping are already highly efficient, and we are able to provide an answer to this question.
  3. We show that instance-adaptive finetuning lets us use smaller networks and reduce decoder-side compute by 72% with only a minimal reduction in compression performance (still outperforming SSF). Decoder-side compute is a major bottleneck in real-life applications, especially on mobile devices; such a reduction can have a large impact.
  4. Unlike van Rozendaal et al. (2021), we provide an extensive experimental analysis of the properties and trade-offs of instance-adaptive video compression including:
      * experiments that quantitatively analyse the trade-off between encoder-side compute and compression performance (Figure 2),
      * experiments to show how instance-adaptive compression can decrease the performance gap for out-of-domain data instances (Figure 3, 6, 7, 9),
      * experiments that show how robust this method is to different base model architectures (B-frame or P-frame neural codecs) and model size (Figure 1, 8),
      * experiments that analyze the trade-off for different finetuning strategies for longer videos (Figure 9), and
      * an analysis on the bitrate allocation across latents and model updates as well as across modules of the finetuned network (Figure 10, Table 7).

> The difference between the model used and the scale-space flow described in the appendix: the size of the first layer convolution kernel is changed from 5 * 5 to 3 * 3. No reasonable explanation was given.

Originally this change was made by accident. Since we only spotted this difference in kernel size after we had run many of our experiments, we decided it would be a waste of compute to re-run the baseline and all instance-adaptive experiments, which is why we reported on the difference in our implementations.

However, we extensively benchmarked our re-implementation against the results reported by Augustsson et al., including exactly reproducing their reported ffmpeg baselines and running entropy-coding with our own SSF model. We found that our implementation gives a similar, but generally even slightly better performance than the results reported by Augustsson et al.

> c In the experiment, only UVG and class B of the standard test sequences were compared. How does it compare with FVC on the HEVC Class C and HEVC Class D ?

We thank the reviewer for their suggestion and have added an evaluation on HEVC class C in Figure 5 of the Appendix as well as in the BD-rate tables. This is an interesting dataset to compare to: we expect our method to perform worse on this lower-resolution data, as the model update overhead is amortized over fewer pixels.  Indeed, the performance compared to HM, ffmpeg, and FVC is not as good as on the other datasets. Nevertheless, InstA-SSF18 still clearly outperforms the SSF18 baseline model.

Please note that these results are preliminary, the instance-adaptive finetuning has only run briefly and has not converged yet and has the potential to further improve (see Fig. 2). We will update these results later.

> d There are some spelling and grammatical problems in the paper.
> e The reference format of the paper is not uniform.

We thank the reviewer for their careful feedback and apologise for these inaccuracies, which we have fixed in the updated paper version.

Overall, we believe the reviewer's feedback greatly improved our manuscript. We have incorporated many of the suggestions and added multiple results, references, and ablations; see our general response for a complete list. We hope that our paper is now in a state fit for ICLR.


 We thank the reviewer for their helpful comments. We are glad that they found our method interesting and appreciated its strong compression performance. We were particularly happy to see that they liked  our discussion of the trade-off between encoding time and compression performance.

We will now respond to the reviewer's concerns and comments.

> Limited novelty and contribution. 
> ...
> Although it is new and interesting to explore the effectiveness of these technics in learned video compression, the reviewer thinks that the novelty and contribution are limited.

Indeed, we apply the instance-adaptive finetuning method as introduced in van Rozendaal et al. (2021) to video compression models with P- and B-frames. We do not introduce an entirely new algorithm or new architectures. However, we believe that there is substantial empirical novelty in our work:

  1. We show that instance-adaptive finetuning lets us use smaller networks and reduce decoder-side compute by 72 % with only a minimal reduction in compression performance (still outperforming SSF). Decoder-side compute is a major bottleneck in real-life applications, especially on mobile devices; such a reduction can have a large impact (Figure 1,2,8).
  2. Unlike van Rozendaal et al. (2021), we provide an extensive experimental analysis of the properties and trade-offs of instance-adaptive video compression including:
      - experiments that quantitatively analyze the trade-off between encoder-side compute and compression performance (Figure 2),
      - experiments to show how instance-adaptive compression can decrease the performance gap for out-of-domain data instances (Figure 3, 6, 7, 9),
      - experiments that show how robust this method is to different base model architectures (B-frame or P-frame neural codecs) and model size (Figure 1, 8),
      - experiments that analyze the trade-off for different finetuning strategies for longer videos (Figure 9), and
      - an analysis on the bitrate allocation across latents and model updates as well as across modules of the finetuned network (Figure 10, Table 7).

> The ablation studies are missed. How many overhead bits are consumed due to the parameters' update? What is the proportion of the bits used for encoding quantized updates $\bar{\theta}$, quantized codes $\bar{z}$ and and the prior $p(\bar{\theta})$?
> ...
> These are necessary to analyze the proposed method.

We thank the reviewer for these excellent suggestions! We have added two ablations to our manuscript. One ablation shows the breakdown of the total rate (Figure 10). It can be seen that for video sequences that are similar to the training data, only a tiny fraction (< 1%) of the overall rate is allocated to the model updates. However, when encoding out-of-domain data (in our case the animated Big Buck Bunny sequences), the model learns to adapt more strongly by spending up to 3% of the overall rate on model updates. In addition, we added an ablation which shows how the model rate is distributed over the different layers of the network (Table 8). We find that most of the bits are allocated to updates for the I-frame and residual model.

> Some representative existing works are missed from related works and experimental comparisons. For instance, ...

We thank the reviewer for their suggestions and have incorporated all suggested baselines into our manuscript. In order to prevent clutter, we added a new Figure 4 to the appendix, which shows rate savings against PSNR and makes the differences between the various methods more transparent. For the RD plots in Figure 2, we only show the best-performing models, as well as other instance-adaptive methods (which are closely related to our work).


We would like to thank the reviewer once more for their review and comments. We have updated our manuscript according to the reviewer's suggestions (see the general reply for a complete overview) and believe that that greatly improved the quality of our paper. We hope that the reviewer agrees with this added value of our changes.


This paper proposed a video compression algorithm based on instance-adaptive learning. Experiments show that this method can significantly improve the performance of existing learned video compression codec with acceptable decoding complexity and latency. Strength:

This paper is an extention of instance-adaptive image compression to video compression. Main advantage of the proposed method is that it's compatible with many existing learned video compression frameworks and the increase of computational complexity at the decoder side is negligible. Besides, the experiments are sufficient and the performance gain is convincing. 

Weakness:

The overall novelty seems limited since the instance-adaptive method is from existing work with no primary changes. Here are some main questions and concerns:

1). How many optimization steps are used to produce the final reported performance in Figure.1 as well as in some other figs and tables? 

2). The proposed method looks stronger at high bitrate but close to the baselines at low bitrate. What is the precise bitrate range used for BD-rate comparison?

Besides, a related work about implementing content adaptive algorithm in learned video compression is suggested for discussion or comparison:

Guo Lu, et al., "Content Adaptive and Error Propagation Aware Deep Video Compression." ECCV 2020.
 The paper is well written in general. However, the overall contribution may not be sufficient enough to reach the acceptance threshold.

This paper proposes a video compression algorithm based on instance-adaptive learning. On each video sequence to be transmitted, the proposed method finetunes a pretrained compression model. The parameters' change are transmitted to the decoder with little bit-rate overhead. The experiments show the outstanding performance of the proposed method. The paper introduces an interesting on-line parameter updating scheme into video compression. The experiments show good RD performance and the analyses on the trade-off between encoding time and RD performance show the flexibility of encoding time.

In my point of view, the paper may have the following weakness:

1. Limited novelty and contribution.  The video compression architecture is borrowed from scale-space flow (SSF), and the parameter updating scheme follows the similar idea on image compression (Rozendaal et al., 2021), and other detailed technics are also from previous works, e.g., spike-and-slab prior (Johnstone & Titterington, 2009; Rozendaal et al., 2021). Although it is new and interesting to explore the effectiveness of these technics in learned video compression, the reviewer thinks that the novelty and contribution are limited. 

2. The ablation studies are missed. How many overhead bits are consumed due to the parameters' update? What is the proportion of the bits used for encoding quantized updates \bar{\theta}, quantized codes \bar{z} and and the prior p(\bar{\theta})? These are necessary to analyze the proposed method.

2. Some representative existing works are missed from related works and experimental comparisons. For instance,

Cheng, Zhengxue, et al. "Learning image and video compression through spatial-temporal energy compaction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

Habibian, Amirhossein, et al. "Video compression with rate-distortion autoencoders." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.

Lin, Jianping, et al. "M-LVC: multiple frames prediction for learned video compression." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

Yang, Ren, et al. "Learning for video compression with hierarchical quality and recurrent enhancement." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

Liu, Haojie, et al. "Learned video compression via joint spatial-temporal correlation exploration." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020.

Lu, Guo, et al. "Content adaptive and error propagation aware deep video compression." European Conference on Computer Vision. Springer, Cham, 2020.

Hu, Zhihao, et al. "Improving deep video compression by resolution-adaptive flow coding." European Conference on Computer Vision. Springer, Cham, 2020.

Yang, Ren, et al. "Learning for video compression with recurrent auto-encoder and recurrent probability model." IEEE Journal of Selected Topics in Signal Processing 15.2 (2021): 388-401.

Golinski, Adam, et al. "Feedback recurrent autoencoder for video compression." Proceedings of the Asian Conference on Computer Vision. 2020.

 + The method is interesting and effective.

- The contribution and novelty are limited.

- The experiments (and ablations) are incremental.

Overall speaking, the reviewer thinks that this paper stands at the borderline, but marginally above the acceptance threshold.

This paper mainly introduces a video compression algorithm. The main feature is that it can simultaneously optimize the codec model parameters while compressing the transmitted video, and dynamically adapt to different video styles. The author's main contributions are:

a. In this paper, the instance-adaptive data compression method is applied to the compression of complete video sequences. During the test of each video, by fine tuning the model parameters of the codec, the robustness to deal with various fields of video is improved. This method can reduce the model capacity, because a smaller network may suffice to model a single instance, and relax the assumption that the distribution of training set and test set always match. It can be applied to any neural network video codec model.

b. This paper demonstrates the effectiveness of the method through experiments on two different architectures: scale-space flow with low delay setting and B-frame based model. Advantages:

This paper introduces an instance-adaptive video compression method, which determines a pretrained neural compression model on each video sequence to be transmitted, adjusts the network parameters, and is transmitted together with the latent coding. It is verified by the latest scale-space flow model and B-frame compression model, and some bitstream savings are achieved.

Disadvantages:

a The key idea of this paper comes from the instance-adaptive data compression, which is only applied to the scale-space flow model and B-frame compression architecture. The method of this paper is lack of innovation.
b The difference between the model used and the scale-space flow described in the appendix: the size of the first layer convolution kernel is changed from 5 * 5 to 3 * 3. No reasonable explanation was given.
c In the experiment, only UVG and class B of the standard test sequences were compared. How does it compare with FVC on the HEVC Class C and HEVC Class D ?
d There are some spelling and grammatical problems in the paper, such as page 3, the third line of the paragraph above formula 2, parameter “can be be transmitted very effectively”. In the references on page11, the abbreviation of the conference “SiPS” should be capitalized, just as “Dvc” should be expressed by “DVC”.
e The reference format of the paper is not uniform. For example, on page10, the author of the seventh is inconsistent with other formats. Make sure that some references have been officially published, such as the last one on page 11, which has been published on ICLR. Please check others. On page 12, the conference name of the third reference should be capitalized in accordance with other references. On page 11 there is a problem with the format of the 10th reference. Please check other parts carefully. This paper applies the instance-adaptive image compression method to video compression, and proves the effectiveness of the instance-adaptive data compression method in video through the implementation on two existing models, but I want to weak accept it because of the following problems:

a The method of this paper is lack of innovation. In fact, most of the work is done by combining other people's methods. There is no doubt that this paper applies the method of I-frame adaptive compression to video and combines it with other better models to get a certain effect. Authors need to highlight their innovative contributions.

b In addition, there are many errors in the references of the manuscript, such as inconsistent format, author abbreviations, etc.

Meta-review: Overall, this paper receives negative reviews due to limited technical novelty and contributions. The authors' rebuttal does not address all the raised concerns. As such, the area chair agrees with the reviewers and does not recommend it be accepted at this conference

Decision: Reject