Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The author propose a video compression framework that leverages context model in place of conventional residual compressor.
main_review: pros:
1. use a context model to improve the compression performance
2. Better rate-distortion performance over DVCpro baseline

cons:
1. Need more neural baselines.
2. Conditional entropy model is not a novel idea for video compression. (see point 4)
3. Spatial autoregressive prior is not a proper solution for video, as it will significantly increase both encoding and decoding time. Its autoregressive feature makes it hard to be parallelized. In addition, entropy coding usually happens in CPU but network inference happens in NPU/GPU, the accumulated communication cost in autoregressive calls will further worsen the coding efficiency. Given the fact that video decoding is sensitive to runtime (high frame rate), I disagree to use any spatial autoregressive model here.
4. I'm skeptical about the effectiveness of the context model. 
    - Author tries to concatenate $\bar{\textbf x}_t$ in en/decoder to demonstrate that it's better than residual, but I didn't see there is any theoretical difference. This can simply be regarded as a different way to incorporate residual, and the concatenation method requires more model parameters and conv layers. The invertibility of the original residual also makes the model easier to train.
    - Similar idea was already proposed to improve the compression (Yang et al.). They interpret the residual compression as an invertible transform (or normalizing flow) and propose a similar(also simpler) conditional prior.
    - $\textbf{Comments}$: The context $\bar x$ also acts as a condition of the conditional entropy coder. As the concatenation method is simply a different form of residual, I think the performance improvement may mainly come from the conditional entropy coder. Extra conditioning makes the prior fit the distribution of the latent variable better (for better probabilistic model); residual is a cheap way to reduce the redundancy and removing high-frequency information (for better transform). They are aimed at solving the same problem in slightly different ways. 

Yang, Ruihan, et al. "Hierarchical Autoregressive Modeling for Neural Video Compression." International Conference on Learning Representations. 2021.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper extends an existing neural video compression method (DVCPro) by adding a two crucial components: (1) the usage of optical flow in warping latents; (2) the usage of optical flow to design a non-additive decoder.  These additions result in a state of the part (as stated by authors, but not verified by me) performance reported on a few (small) datasets usually used for video compression evaluation, outperforming HEVC.
main_review: This paper tackles the task of neural video coding.

This task, in one of its most common forms takes key-frames, which get encoded/decoded separately, and uses these to seed the start of a sequence of compressed frames. The sequence of compressed P-frames ("predicted") uses the previously decoded frame(s) and motion to come up with a hypothesis for what the current frame should be. The difference (residual) is then encoded in the bitstream.

This paper essentially does exactly this, but with a few crucial twists.

The motion information is used for warping latents (i.e., the quantized tensors which are used for either encoding/decoding or conditioning), which as far as I know is relatively uncommon. These are then used in various places within the encoder/decoder/entropy coder for the P-frame residual. 

Secondly, in a typical video compression framework, the output ("decoded") frame is computed as a warp of the previously decoded frame plus the residual ("additive reconstruction"). This typically causes some problems, which this paper avoids by providing the decoder instead with latent values computed from the previously decoded frame, with the decoder's task being to perform a full reconstruction from that and the additional information.

In addition to the hyper-prior (done per frame), the paper also employs a temporal prior model to boost performance.

The authors perform various ablations, which can be found in the supplementary materials. I feel this paper is incomplete by a large margin if we exclude the supplementary, where most of the details about the paper are found.

Overall, I think this is a pretty good paper, but ONLY if we take into account the supplementary. Without them I felt scratching my head multiple times, but all the missing information that made me wonder "why is this not discussed/presented/tested" was present in supplementary.
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this work the authors propose a new model for video compression that, instead of warping the previous frame and predicting the current frame, extract and warp features using the motion vectors. As a result, the encoder doesn’t rely on residual compression (residual between ground truth and prediction) but directly encodes the necessary information needed to decode the frame.
main_review: **Originality**

Such an approach to video compression is relatively new as most related works indeed focus on predicting a full frame then encoding a residual image. It has however strong links with frame interpolation methods such as:

-Softmax Splatting for Video Frame Interpolation

-Depth-aware video frame interpolation

which also warp context information before image prediction. The authors also didn’t discuss some other important related work such as “Learned Video Compression, ICCV 2019” which also goes a little into the direction of a conditioning not limited to residuals only.

There is also a strong link with [11, 14, 15] but this is sufficiently discussed in the paper

**Quality**

The idea presented in the paper is intuitive and leverages all the knowledge gained in both video compression and frame interpolation. Quantitative evaluation shows state of the art results on a large variety of test sets. I still have the following issues:

-The evaluation only presents a small set of methods. Since it is done on the same datasets it would be important to present other top performing methods for better context. Is there a reason not to include them?

-Only one quality evaluation is presented in supplementary material. Such a figure should appear in the main paper. In addition to this, comparisons with more methods should be included as well. 

-Temporal stability is not discussed and the authors didn’t provide a video as supplementary material which I think is quite important to evaluate the work. Also the authors didn’t explain how to maintain similar quality between the key frame and the interpolated frames

-The discussion around the information represented in the feature domain lacks precision. In particular the discussion on the link between the channel 1 in Figure.3 and the motion vectors. Visually one can argue that high intensity values correspond to the “red players”. An ablation study would be required to better understand the content: all values to 0 or set to the average and see the effect on the image. 


**Clarity**

In general the paper is clear and easy to follow.

-“MEMC” is used without being introduced. I suppose it stands for Motion Estimation and Motion Compensation? Same comment about “MV” 

-l.122: … of network. => “of neural networks”?

-The discussion section is a little rough and would benefit from a revision.

**Significance**

The authors proposed a new approach for learning based video compression which seems to be setting a new state of the art result. 

**After Rebuttal**

The authors answered most of my questions and concerns in their rebuttal. I would insist they include more visual and qualitative examples in the revised version.

limitations_and_societal_impact: The authors discussed some limitations, in particular the fact that the proposed solution is another conditioning which is slightly more complex and rich than image prediction but other alternatives remain to be explored. I think the discussion could be extended to other considerations like temporal stability.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes to replace the minus/plus operations commonly used in neural video compression (to calculate/add the residual) with neural networks. The resulting contextual encoder/decoder networks are conditioned on the same context that is used for the entropy model, which is calculated by warping features extracted from the previously decoded frame.
main_review: [Post rebuttal update: all my concerns were addressed].

The paper contains a solid idea (replacing minus/plus) and shows promising R-D results. My issues with the paper are:

a) Ablation studies: from the text it is not clear how the proposed method relates to "DVCPro", and Section 4.3. is written unclearly. Is the only difference that the proposed method uses contextual encoder/decoder nets and the generation net? If that is the case, the presented ablation studies make sense, but this has to be made more clear. I.e. I need a paragraph on what exactly the differences between the "DVCPro" and the "DCVC (context in pixel domain)" lines in Fig. 7 are. I am hoping that the second line can be renamed to sth like "DVCPro + context in pixel domain". If there are more novel components, the ablation study is not useful. In particular, I'm wondering what is meant with "DVC and DVCPro are retested using the same intra frame coding" (L229) and whether the training strategy mentioned in Section 2 of the Appendix is used for all?

b) Runtime concerns: there is only one line on complexity (L252). It is unclear which complexity is meant (number of weights, FLOPS, fps, ...?). I would expect the feature extraction and refinement operations (which happen at full res) to be costly, and they are needed for encoding and decoding. As such, I'm surprised how the "complexity" can be the same as DVCPro? 

c) Contributions list is overly vague and it's unclear which things are novel. "[...] jointly utilizing the spatial and temporal correlation, [...]" (L66) does not seem novel. "[...] more accurate probability estimation can be obtained" (L67) is not ablated in the text. "We define the condition as the context in feature domain" seems vaguely familiar to what Rippel et al. were doing in their 2018 paper.

d) Entropy Coding? L169: "the gap between the actual bitrate ... and the cross-entropy is negligible": not always, depends on the precision of your arithmetic. This statement makes me wonder whether the presented graphs are using the real, entropy coded bitrate?

e) ffmpeg settings: I did not follow why the arguments "-r", and "-vframes" are needed.

f) Various issues with the text:
- L8 and throughout: I think what is called "condition" should be called "conditioning signal". 
- L28: Reference [11] does not contain the entropy formula, and intuitively it is unclear how the entropy of the substraction is higher, I think entropy is the wrong concept to be used here, or I'm missing something in which case clarification is needed.
- L122: "MEMC which is totally composed of network": not sure what this means or how it's different from the proposed method.
- L167, L209: the "true" probability mass function of y is of course unknown, we are using the empirical distribution to estimate the rate. Similarily, the statement on L169, "the gap betweeen the actual bitrate ... and the cross-entropy ..." 
- L179: Any reason/intuition for using Laplace?
- L186ff seem similar to L133ff? What is meant with a "plain" network on L186?

Summary:
Overall I like the idea, but I think the paper does not sufficiently place it within the context of previous works.
limitations_and_societal_impact: The authors have not addressed societal impact.
review_title: null

Meta-review: The authors propose a new video compression model that uses motion vectors to extract and warp features instead of warping the previous frame for predicting the current frame. As a result, the encoder doesn’t rely on residual compression but directly encodes the necessary information needed to decode the frame. 

The paper leads to a new state of the art in video compression and merits publication. Unfortunately, the main paper lacked several citations and baseline comparisons. Some of these comparisons were shown only in the appendix, making the main paper appear incomplete. I strongly urge the authors to present these results in the main paper. In addition, I also encourage the authors to run additional baseline comparisons of the related works mentioned by the reviewers (such as Ruihan Yang et al., ICLR 2021). 
Decision: Accept (Poster)