Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a neural video compression method that is designed around the idea of using a voxel flow instead of the more traditional pixel warping idea. The key difference here is that a volume of reference frames is used, and flow is computed to all of them. This allows the compression method to be quite liberal in terms of what information from the GOP can be used, yielding a better RD performance than methods that rely on a single reference frame.

Overall I think this is a step in the right direction, but I fear that such a method is completely impractical for the foreseeable future.
main_review: This method makes use of a frame volume that can be equal to the number of frames in the GOP. To give some perspective, normal video codecs tend to be very memory sensitive, and in general it's not possible to have more than a small set of reference frames due to the huge memory pressure that this imposes, and of course cost (more frames = more RAM needed). This problem is exacerbated by the fact that we see a trend towards higher resolutions which grow in size quadratically, making the cost of multiple reference frames completely prohibitive. This is my main worry about the applicability of this paper... I am not sure how realistic the assumption is that a copy of N reference frames can be kept in RAM at decoding time....where N >> 2. Similarly the number of flows is also on the order N, making my worry even worse.

Ignoring the practical aspect, the paper presents an interesting approach to improve compression. I haven't seen something done exactly this way in any other area so I think that we have an increased level of originality here. The main differentiator is the multiple flow volume idea (multiple reference frames have been done many times).

I applaud the authors for not fearing to compare against HM, because that's the best possible case for HEVC. I am not familiar enough with the settings to comment whether the modifications made to the profile used make sense, and I would invite a reviewer with more familiarity to comment on this.

I also think it's good that the details of the network are presented in the supplementary materials.

My second biggest concern, though is related to runtime. This method sounds very very expensive and at least in the body of the main paper this issue is not really addressed. I think in the final version THIS MUST be explicitly addressed.

Overall, I am inclined to accept this paper.
limitations_and_societal_impact: The limitations are really not discussed at all. This method has some glaring problems I mentioned in the review and they're not at all acknowledged.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Learned video compression is a promising field and has demonstrated that it can be as effective as the classic video compression method. In terms of rate distortion it has caught up with the classical methods greatly. However binding the prediction mode and fixed network is what hinders the field to go forward. This paper proposes a versatile learned video compression framework that learns one mode to predict all possible modes. The paper proposes versatile compression where the motion compensation applies 3D motion vector fields, trilinear warping. Motion estimation here acts as a decoupler of prediction modes away from framework design, allowing its dependence to be not affected by framework design. For multiple reference frame prediction, the network predicts the motion fields with a unified polynomial function. The flow prediction can lead to reduction of computation by reducing voxel flow transmission. 
main_review: The strengths of this paper is that it explains the motivation behind the proposed method in an understandable way and the articulation is sufficient in that it is easy to understand. The introduction well describes the intuition and motivation behind designing a versatile motion compensation module that can handle multiple prediction mode scenarios. It also explains about the polynomial motion trajectory. 
Overall, the paper is well written and easy to follow with good figures explaining the concept. The experimental results also show that the proposed method performs well against other SOTA methods. 
A concern is that this paper does not show any analysis about computational complexity in the main paper. The greatest reason why learning approaches are not being used widely compared to classical methods is due to computation complexity. I would suspect that incorporating a versatile 3D voxel flow module to handle all prediction mode cases will give a great impact on computational speed. Also, solving for the polynomial coefficients for the motion trajectories does not sound computationally efficient. The analysis on computational efficiency was lacking in my opinion. There needs to be more analysis on this aspect. The computational impact will be different for each prediction mode for the classical method, however, the computational load will more or less be similar for the proposed methods since it uses a unified framework for predicting the trajectories for all modes. Also more analysis is needed on the computational load for polynomial coefficient prediction. 
It would be wise to compare the computational load with the classical methods as well for different prediction modes.
Given that the model size is 103MB, about 3 times as large as PWC-Net, the model seems sizable and may most likely translate to high computational complexity. Nonetheless, I believe a comparison with other Learning based methods at least is needed.

limitations_and_societal_impact: For video compression I cannot imagine a suitable case for ethical impact. However, any piece of technology can contribute to wrong use and ethical implications. This work did not address such concern to my knowledge.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper introduces multi-frame references for learned video coding, which estimates multiple optical flows as voxel flows for motion compensation. With the weighted map, it can achieve weighted trilinear warping to combine the multiple predictions into  the final prediction. It is a good idea to obtain a better predcition to search information across multiple frames.  The author introduces  the polynominal motion modeling to use multiple backward optical flow to generate the single optical flow. Based on the method of softmax spaltting, they convert the backward flow into a forward flow using the flow reversal layer. Then use the forward flow and softmax spaltting for motion compensation. The overall architecture is a hybrid video coding system and the main contribution is inter coding. Residual coding is applied on the feature domain. 
main_review: 1) Voxel flows. The concept of voxel flow is proposed in the paper [1] for weighted prediction which can utilize the different information from different frames partially for a final prediction. The author uses this idea for the multi-frame reference for video coding with a rate-distortion optimization. Although decoding multiple flows sometimes only add some consumption of bit rates, can the author give some data about the comparison between single flow and voxel flow. Meanwhile, the author should include the MLVC [2] for comparison because this paper also uses multiple frame as references.
2) Forward flow with softmax spaltting. We think the author is aimed to use the forward flow with softmax spaltting for compensation to solve the problem of the worse predcition around the motion edges and occlusions in some extent. However, voxel flows can also solve this problem with the long-term prediction with the weighted maps. Can the author give some results when only using the generated backward flow for compensation?
3) We think the main contributions are voxel flows and the different wapring method based the proposed polynominal motion modeling. They are mainly from the existing published papers and just apply these methods to the learned video coding from the frame interpolation and prediction. The overall framework and concept are still follow the hybrid video coding system.
4) Experiments only show the results on UVG and ClassC and ClassD. How about the results on ClassE and ClassB and MCL-JCV?
5) More learned video coding methods should be added for comparison such as DVC_Pro [3], [4],[5], [6].
6) We also find that the bit rate on ClassC and ClassD are larger than 0.1 bpp. As a video coder, performance between 0.01 ~ 0.1 bpp is significant. Please provide the results between the bit rate range and include the results of the listed learned video coder.




[1]. Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis 371
using deep voxel flow. In Proceedings of the IEEE International Conference on Computer Vision , pages 372
4463–4471, 2017.
[2]. Jianping Lin, Dong Liu, Houqiang Li, Feng Wu. M-LVC: Multiple Frames Prediction for Learned Video Compression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[3]. Guo Lu; Xiaoyun Zhang; Wanli Ouyang; Li Chen; Zhiyong Gao; Dong Xu. An End-to-End Learning Framework for Video Compression. IEEE Transactions on Pattern Analysis and Machine Intelligence 2020. 
[4]. Haojie Liu, Ming Lu, Zhan Ma, Fan Wang, Zhihuang Xie, Xun Cao, Yao Wang. Neural video coding using multiscale motion compensation and spatiotemporal context model. IEEE Transactions on Circuits and Systems for Video Technology 2021.
[5]. Ren Yang, Fabian Mentzer, Luc Van Gool, Radu Timofte. Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model. JSTSP 2021.
[6]. Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, George Toderici;Scale-Space Flow for End-to-End Optimized Video Compression.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 8503-8512
limitations_and_societal_impact: Please see the comments below.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors introduce VLVC, a new neural video compression codec. Thanks to a new 3D motion compensation structure based on spatio-temporal interpolation, a single trained model can be run in arbitrary prediction modes (such as low-delay or random-access settings). In addition, each frame can be compressed with an arbitrary number of reference frames. There are a number of smaller changes to established neural video codecs. The resulting VLVC codec is demonstrated on standard datasets, showing good rate-distortion results (in particular when trained and evaluated on MS-SSIM).
main_review: Neural video compression is without question a problem of large practical importance. It seems reasonable that a NN codec that can operate both in low-delay and random-access modes is beneficial, though I find that the authors overstate this case. Blanket statements like "existing learned video compression schemes [...] are inapplicable for various scenarios" really need to be supported by more arguments or examples.

VLVC brings novelty to the neural video compression landscape. The independence of a model from the prediction mode is made possible by a new warping algorithm based on ultiple 3D voxel flows and spatio-temporal polynomial interpolation. However, I believe this is not the first work in this direction: T. Ladune et al [1] also propose a framework in which a single model can be used to compress frames in different ways. A comparison of these methods would enhance the paper.

The proposed method is overall sound, as far as I can tell. In addition to the flexibility with respect to coding scheme / reference structure, the structure with multiple flows allows the model to deal better with occlusions and to model motion uncertainty to some extent, both of which may be relevant in practice.

Some aspects I found confusing; in particular, I am not sure if I understood the generalized optical flow prediction (section 3.2) correctly. Perhaps more importantly, especially for a paper that focuses so much on the independence between reference structures: which reference configurations are actually used? In line 216 the authors describe it as "the model will have access to various reference structures" without going into detail. The labels "randomaccess" (RA) and "lowdelay" (LD) also do not fully specify this. If there was a definition of these somewhere in the paper, I missed it and would very much appreciate a pointer. Are the reference structures described for HM and VTM in the supplementary material used for VLVC as well? How are I-frames / key frames handled in this approach? 

I find that several aspects of this approach deserve more discussion. For instance:
1. When should one use which coding configuration (reference structure)? Can the model be extended to make such decisions dynamically based on video content?
2. What exactly are the benefits of having a single model that supports arbitrary coding configurations? Can the authors for instance argue or demonstrate that with the same overall computational budget, a common model achieves a better performance than separate LD and RA models?
3. How fast is the proposed video codec during training, encoding, and decoding relative to other neural video codecs? How efficient is the new 3D voxel warping operation? The supplementary material mentions encoding and decoding around 0.7 fps, can the authors further comment on how this could be improved?
4. It would be interesting to have a detailed comparison of the proposed 3D warping and the scale-space flow (SSF) warping proposed by E. Agustsson et al [2]. Both methods use 3D flows, although the third dimension in SSF refers to a blur direction rather than time. Both have the interesting feature that they improve the modelling of uncertainty in the optical flow, for instance in the presence of movement or occlusion (see Fig. 6). Can the authors comment more on the similarity and differences between these approaches?
5. How important is the off-the-shelf PWC-Net in this approach? How does the use of this pretrained optical flow estimator fit together with Sec. 3, where the authors argue that end-to-end training with implicit flow networks  performs better than pretrained optical flow networks and simplifies the network architecture?  

The empirical results show a strong rate-distortion performance on standard datasets, especially in the low-bitrate region. Comparing to HM and VTM is laudable, though I do not see why the authors changed the default RA reference structure in these classical codecs. The authors also compare to several neural baselines. Some strong baselines like SSF [2], ELF-VC [3], and B-EPIC [4] are missing (except for SSF these all came out very recently though).

While the results are strong, they do not quite justify all of the claims that the authors make. VLVC does not "yield competitive R-D performance compared to [...] VVC" (last sentence of the conclusions) when evaluated on PSNR, i.e. the metric that classical codecs are largely optimized for. I also do not see that the "proposed method significantly outperforms existing learned video compression methods" when the strongest neural baselines are missing from the comparison. (I tried to compare VLVC to SSF, ELF-VC, and B-EPIC manually and it does seem to me that VLVC outperforms them slightly at low bitrates! But the authors need to show this to make SOTA claims). I wonder why the authors hide the meaningful classes B and E of the HEVC test datasets in the supplementary material, while showing results for classes C and D with their extremely small resolutions (HEVC class D has a resolution of 240x416...). Finally, it would be great to provide more BD-rate savings (for all considered datasets and also for the MSE models).

The authors present a good set of ablations and extensive supplementary materials. The main ablation I am missing is a comparison to the same method when it is trained on only one prediction mode.

The presentation of the paper could be improved. I believe the explanations could be clearer, important aspects could be highlighted better, and (as discussed above) several aspects seem to be missing. There are also several typos.

All in all, the authors present an interesting neural video compression method with several new features and a strong empirical performance. Unfortunately, this paper does not do the method justice. Both the description of the method, the discussion of its properties and use cases, and the empirical evaluation can be improved in several ways. A revised version that answers my question and addresses these points might make a good paper, but in its current form I do not think it is ready to be accepted.

References:
- [1] T. Ladune et al, Conditional coding for flexible learned video compression, ICLR 2021 workshop neural compression
- [2] E. Agustsson et al, Scale-space flow for end-to-end optimized video compression, CVPR 2020
- [3] O. Rippel et al, ELF-VC: Efficient Learned Flexible-Rate Video Coding, arXiv:2104.14335
- [4] R. Pourreza and T. Cohen, Extending Neural P-frame Codecs for B-frame Coding, arXiv:2104.00531

limitations_and_societal_impact: The authors do not discuss the societal impact that neural video codecs could have. The paper could be improved by discussing risks for instance from biases in the training data, but also due to algorithmic choices.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: First, the paper introduces the idea of using multiple predictions (e.g. M=25) for learning-based video coding. Second, to reduce the required motion bits, a flow prediction method using polynomial approximation is developed. The idea is reasonable, and the reported performance seems promising. However, the paper is written rather poorly, and how the algorithm is actually designed is not clear. Thus, the reported performance is not convincing either. 
main_review: ***
This paper is slightly longer than the page limit (9 pages). Anyway, it violates the rule and can be rejected. I leave the decision to AC. 

***

Except for the survey of previous work, the writing and description are poor. There are many typos and grammatical errors. For example,
1. gop -> GOP
2. More attention should be paid to articles (a, the) and tenses.
3. "Considering a practical scene where multiple objects of the same types appear at the same time, more than one areas could 
 referred for reconstructing the local region." What does "the local region" specify?
4. More possible sampling location
5. "$D$ is the number of reference frames used for motion compensation." Motion estimation? You use only 2 frames for motion compensation. Is that right?
6. In this process, the predicted optical flow reduces the spatial displacement need to be encoded
7. there are two optical flow describe the motion between the reference frame and  the target frame 
8. the flow describe
9. pixels is unavailable. 

In fact, there are too many errors to list them all, especially near the end of the paper. 



 

There are many technical details, which cannot be clearly understood even after reading the supplemental document. 
1. Please describe the prediction structure for random access at $r=2$ when the GOP size is 7. 
2. When there are only two reference frames ($r=2$), what is the role of $g_z$ ? Why do you need it anyway? You are using multiple voxel flows. So half of them can be constrained from reference 1, and the other from reference 2. 
3. Figure 6 is hard to understand. Please provide the color bars. 
4. How do you encode intra frames? Using the same residual encoder? Are they considered in the rate-distortion curves?
5. How do you set $k$ in Eq. (2) and (3)? Is a higher-order polynomial (beyond quadratic) really necessary? Also, here, the variable is $\Delta t = t_j - t_i$. Is it ok to ignore the distance from the current frame $t$?






limitations_and_societal_impact: NA
review_title: null

Meta-review: The work proposes and implements a learning-based video compression solution, which is able to handle multiple prediction modes and achieves good rate-distortion performance. The key building blocks are not new, but the actual model and its successful implementation bring an interesting contribution to the video compression problem. 
The submitted manuscript can be completed by the multiple additional results discussed with the reviewers, and a complexity analysis; the presentation should finally be improved before an eventual publication. In particular, the work should be more clearly positioned wrt to recent competitors, including 'T. Ladune et al, Conditional coding for flexible learned video compression, ICLR 2021 workshop neural compression'. 
While the work is really interesting, the revisions that result from the active discussion with the reviewers will lead to a quite different, improved, manuscript, which probably deserves another review before a positive outcome. 
Decision: Reject