Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a video compression algorithm based on instance-adaptive learning. Experiments show that this method can significantly improve the performance of existing learned video compression codec with acceptable decoding complexity and latency.
main_review: Strength:

This paper is an extention of instance-adaptive image compression to video compression. Main advantage of the proposed method is that it's compatible with many existing learned video compression frameworks and the increase of computational complexity at the decoder side is negligible. Besides, the experiments are sufficient and the performance gain is convincing. 

Weakness:

The overall novelty seems limited since the instance-adaptive method is from existing work with no primary changes. Here are some main questions and concerns:

1). How many optimization steps are used to produce the final reported performance in Figure.1 as well as in some other figs and tables? 

2). The proposed method looks stronger at high bitrate but close to the baselines at low bitrate. What is the precise bitrate range used for BD-rate comparison?

Besides, a related work about implementing content adaptive algorithm in learned video compression is suggested for discussion or comparison:

Guo Lu, et al., "Content Adaptive and Error Propagation Aware Deep Video Compression." ECCV 2020.

summary_of_the_review: The paper is well written in general. However, the overall contribution may not be sufficient enough to reach the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a video compression algorithm based on instance-adaptive learning. On each video sequence to be transmitted, the proposed method finetunes a pretrained compression model. The parameters' change are transmitted to the decoder with little bit-rate overhead. The experiments show the outstanding performance of the proposed method.
main_review: The paper introduces an interesting on-line parameter updating scheme into video compression. The experiments show good RD performance and the analyses on the trade-off between encoding time and RD performance show the flexibility of encoding time.

In my point of view, the paper may have the following weakness:

1. Limited novelty and contribution.  The video compression architecture is borrowed from scale-space flow (SSF), and the parameter updating scheme follows the similar idea on image compression (Rozendaal et al., 2021), and other detailed technics are also from previous works, e.g., spike-and-slab prior (Johnstone & Titterington, 2009; Rozendaal et al., 2021). Although it is new and interesting to explore the effectiveness of these technics in learned video compression, the reviewer thinks that the novelty and contribution are limited. 

2. The ablation studies are missed. How many overhead bits are consumed due to the parameters' update? What is the proportion of the bits used for encoding quantized updates \bar{\theta}, quantized codes \bar{z} and and the prior p(\bar{\theta})? These are necessary to analyze the proposed method.

2. Some representative existing works are missed from related works and experimental comparisons. For instance,

Cheng, Zhengxue, et al. "Learning image and video compression through spatial-temporal energy compaction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

Habibian, Amirhossein, et al. "Video compression with rate-distortion autoencoders." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.

Lin, Jianping, et al. "M-LVC: multiple frames prediction for learned video compression." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

Yang, Ren, et al. "Learning for video compression with hierarchical quality and recurrent enhancement." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

Liu, Haojie, et al. "Learned video compression via joint spatial-temporal correlation exploration." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020.

Lu, Guo, et al. "Content adaptive and error propagation aware deep video compression." European Conference on Computer Vision. Springer, Cham, 2020.

Hu, Zhihao, et al. "Improving deep video compression by resolution-adaptive flow coding." European Conference on Computer Vision. Springer, Cham, 2020.

Yang, Ren, et al. "Learning for video compression with recurrent auto-encoder and recurrent probability model." IEEE Journal of Selected Topics in Signal Processing 15.2 (2021): 388-401.

Golinski, Adam, et al. "Feedback recurrent autoencoder for video compression." Proceedings of the Asian Conference on Computer Vision. 2020.


summary_of_the_review: + The method is interesting and effective.

- The contribution and novelty are limited.

- The experiments (and ablations) are incremental.

Overall speaking, the reviewer thinks that this paper stands at the borderline, but marginally above the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper mainly introduces a video compression algorithm. The main feature is that it can simultaneously optimize the codec model parameters while compressing the transmitted video, and dynamically adapt to different video styles. The author's main contributions are:

a. In this paper, the instance-adaptive data compression method is applied to the compression of complete video sequences. During the test of each video, by fine tuning the model parameters of the codec, the robustness to deal with various fields of video is improved. This method can reduce the model capacity, because a smaller network may suffice to model a single instance, and relax the assumption that the distribution of training set and test set always match. It can be applied to any neural network video codec model.

b. This paper demonstrates the effectiveness of the method through experiments on two different architectures: scale-space flow with low delay setting and B-frame based model.
main_review: Advantages:

This paper introduces an instance-adaptive video compression method, which determines a pretrained neural compression model on each video sequence to be transmitted, adjusts the network parameters, and is transmitted together with the latent coding. It is verified by the latest scale-space flow model and B-frame compression model, and some bitstream savings are achieved.

Disadvantages:

a The key idea of this paper comes from the instance-adaptive data compression, which is only applied to the scale-space flow model and B-frame compression architecture. The method of this paper is lack of innovation.
b The difference between the model used and the scale-space flow described in the appendix: the size of the first layer convolution kernel is changed from 5 * 5 to 3 * 3. No reasonable explanation was given.
c In the experiment, only UVG and class B of the standard test sequences were compared. How does it compare with FVC on the HEVC Class C and HEVC Class D ?
d There are some spelling and grammatical problems in the paper, such as page 3, the third line of the paragraph above formula 2, parameter “can be be transmitted very effectively”. In the references on page11, the abbreviation of the conference “SiPS” should be capitalized, just as “Dvc” should be expressed by “DVC”.
e The reference format of the paper is not uniform. For example, on page10, the author of the seventh is inconsistent with other formats. Make sure that some references have been officially published, such as the last one on page 11, which has been published on ICLR. Please check others. On page 12, the conference name of the third reference should be capitalized in accordance with other references. On page 11 there is a problem with the format of the 10th reference. Please check other parts carefully.
summary_of_the_review: This paper applies the instance-adaptive image compression method to video compression, and proves the effectiveness of the instance-adaptive data compression method in video through the implementation on two existing models, but I want to weak accept it because of the following problems:

a The method of this paper is lack of innovation. In fact, most of the work is done by combining other people's methods. There is no doubt that this paper applies the method of I-frame adaptive compression to video and combines it with other better models to get a certain effect. Authors need to highlight their innovative contributions.

b In addition, there are many errors in the references of the manuscript, such as inconsistent format, author abbreviations, etc.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tried to close the domain gap between the training and testing dataset for the learned video compression. An instance-adaptive video compression approach is proposed by updating the model parameters of the video codecs and the optimal parameters are transmitted to the decoder. Besides, the authors modeled the updated model parameters under a mixture prior model, which further reduces the bit cost. Experimental results on several datasets demonstrated the effectiveness of this approach. 
main_review: -Strengths
1. Good motivations. The domain gap between the training and testing datasets existed in the practical applications and it is important to solve this problem. 
2. The proposed approach is straightforward and neat. Transmitting the model parameters significantly improves the compression performance. 
3. Comprehensive analysis. A lot of results are provided in the ablation study and it is very helpful. 

-Weaknesses
1. Novelty. This paper extended the previous work Rozendaal et al. (2021) to video compression and may have limited novelty. The extension from image compression to video compression is expected. The major modification is the prior model for entropy coding, which is not enough. 
2. Another concern is computational complexity. This new approach increases a lot of computational burdens, which is another issue. This method is even slower than HM in some cases. 
3. The authors only evaluate the proposed method on the UVG and Class B datasets, which are two high-resolution(1920x1080) videos. How about the experimental results on other datasets, like Class c or D?
4. There are also some related works that should be discussed.   (a) Content Adaptive and Error Propagation Aware Deep Video Compression. Lu et al. (b) Efficient Video Compression via Content-Adaptive Super-Resolution. Khani et al.  (c) Online-Trained Upsampler for Deep Low Complexity Video Compression. Klopp et al. 


summary_of_the_review: The main concern is the novelty as a similar approach has been used in image compression. Basically, it is a borderline paper and I am open to other reviews.
review_title: null

Meta-review: Overall, this paper receives negative reviews due to limited technical novelty and contributions. The authors' rebuttal does not address all the raised concerns. As such, the area chair agrees with the reviewers and does not recommend it be accepted at this conference
Decision: Reject