Deep Contextual Video Compression
Authors: Jiahao Li
Bin Li
Yan Lu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep video compression
conditional coding
predictive coding
residue coding
context

Abstract: Most of the existing neural video compression methods adopt the predictive coding framework, which first generates the predicted frame and then encodes its residue with the current frame. However, as for compression ratio, predictive coding is only a sub-optimal solution as it uses simple subtraction operation to remove the redundancy across frames. In this paper, we propose a deep contextual video compression framework to enable a paradigm shift from predictive coding to conditional coding. In particular, we try to answer the following questions: how to define, use, and learn condition under a deep video compression framework. To tap the potential of conditional coding, we propose using feature domain context as condition. This enables us to leverage the high dimension context to carry rich information to both the encoder and the decoder, which helps reconstruct the high-frequency contents for higher video quality. Our framework is also extensible, in which the condition can be flexibly designed. Experiments show that our method can significantly outperform the previous state-of-the-art (SOTA) deep video compression methods. When compared with x265 using veryslow preset, we can achieve 26.0% bitrate saving for 1080P standard test videos.

Deep Contextual Video Compression
Jiahao Li, Bin Li, Yan Lu Microsoft Research Asia jli.joaho, libin, yanlu] @ microsoft.com
Abstract
Most of the existing neural video compression methods adopt the predictive coding framework, which first generates the predicted frame and then encodes its residue with the current frame. However, as for compression ratio, predictive coding is only sub-optimal solution as it uses simple subtraction operation remove the redundancy across frames. In this paper, we propose a deep contextual video compression framework enable paradigm shift from predictive coding conditional coding. In particular, we try to answer the following questions: how to define, use, and learn condition under a deep video compression framework. To tap the potential of conditional coding, we propose using feature domain context as condition. This enables rich information to both the encoder and the decoder, which helps reconstruct the high-frequency contents for higher video quality. Our framework is also extensible, in which the condition can be flexibly designed. Experiments show that our method can significantly outperform the previous state-of state-of-thee-rt (SOTA) deep video compression methods. When compared with x265 using veryslow preset, we can achieve 26.0% bitrate saving for 1080P standard test videos. The codes are at pettps://githne Seiibbbb.ccm//eep//-DD//////
leverage the high dimension context to carry
Introduction
From H.261 [ developed in 1988 to the just released H.266 [2] in 2020, all traditional video coding standards are based predictive coding paradigm, where the predicted frame is first generated by handcrafted modules and then the residue between the current frame and the predicted frame is encoded and decoded. Recently, many deep learning (DL.)based video compression methods 11 also adopt the predictive coding framework to encode the residue, where all handcrafted modules are merely replaced by neural networks.
simple yet efficient manner for video compression, considering the strong temporal correlations among frames. However, residue coding not optimal encode the current frame 2, given the predicted frame žt, because it only uses handcrafted subtraction operation to remove the redundancy across frames. The entropy of residue coding is greater than equal to that conditional coding 12]: H(x, - F+) 2 H [zy (t), where represents the Shannon entropy. Theoretically, one pixel in frame 22 correlates all the pixels in the previous decoded frames and the pixels already been decoded in x4. For traditional video codec, is impossible to use the handcrafted rules explicitly explore the correlation by taking all of them into consideration due to the huge space. Thus, residue coding is widely adopted as a special extremely simplified case of conditional coding, with the very strong assumption that the current pixel only has the correlation with the predicted pixel. DL opens the door to automatically explore correlations in a huge space. Considering the success of DL in image compression [13] [4] which just uses autoencoder explore correlation image, why not use network to build the conditional coding- based autoencoder to explore correlation in video rather than restricting our vision into residue coding?
Encoding residue
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Residue coding
Conditional coding
Enc
Dec
Enc
Dec
Ri predicted frame in RGB damain
Prediction
Generation
Commanly used residue cading-bin based video compression
Our deep contextual video compression
Paradigm shift from residue coding- based framework to conditional coding- based frame- work. 2g is the current frame. dt and Φ the current and previous decoded frames. The orange dashed line means that the context also used for entropy modeling.
Figure I:
When we design the conditional coding-based solution, series of questions naturally come up: What is condition? How to use condition? And how to learn condition? Technically speaking, condition can be anything that may be helpful compress the current frame. The predicted frame can be used as condition but it is not necessary to restrict it as the only representation of condition. Thus, define the condition learnable contextual features with arbitrary dimensions. Along this idea, we propose a deep contextual video compression (DCVC) framework to utilize condition in unified, simple, yet efficient approach. The diagram of our DCVC framework is shown in Fig. 1 The contextual information is used as part of the input of contextual encoder, contextual decoder, well as the entropy model. In particular, benefiting from the temporal prior provided by context, the entropy model itself is temporally adaptive, resulting in a richer and more accurate model. As for how learn condition, we propose using motion estimation and motion compensation (MEMC) at feature domain. The MEMC can guide the model where to extract useful context. Experimental results demonstrate the effectiveness of the proposed DCVC. For 1080p standard test videos, our DCVC can achieve 26.0% bitrate saving over x265 using veryslow preset, and 16.4% bitrate saving over previous SOTA DL-based model DVCPro
Actually, the concept of conditional coding has appeared in 15 only designed for partial module (e.g., only entropy model encoder) need handcrafted operations to filter which content should be conditionally coded. By contrast, our framework is a more comprehensive solution which considers all of encoding, decoding, and entropy modeling, In addition, the proposed DCVC is an extensible conditional coding-based framework, where the condition can be flexibly designed. Although this paper proposes using feature domain MEMC to generate contextual features and demonstrates its effectiveness, we still think it is an open question worth further investigation for higher compression ratio.
16
However, these works
Our main contributions are four-folded:
We design deep contextual video compression framework based on conditional coding. The definition, usage, and learning manner of condition are all innovative. Our method can achieve higher compression ratio than previous residue coding-basee based methods.
We propose a simple yet efficient approach using context to help the encoding, decoding, well as the entropy modeling. For entropy modeling, we design model which utilizes spatial-temporal correlation for higher compression ratio or only utilizes temporal correlation for fast speed.
We define the condition as context in feature domain. The context with higher dimensions can provide richer information help reconstruct the high frequency contents.
Our framework is extensible. There exists great potential boosting compression ratio by better defining, using, and learning the condition.

---

2
Related works
Deep image compression Recently there are many works for deep image compression. For exam- ple, the compressive autoencoder [8]] could get comparable results with JPEG 2000. Subsequently, many works boost the performance by more advanced entropy models and network structures. For example, Ballé et al. proposed the factorized [19] and hyper prior [1]] entropy models. The method based on hyper prior catches up H.265 intra coding. The entropy model jointly utilizing hyper prior and auto regressive context outperforms H.265 intra coding. The method with Gaussian mixture model [20] is comparable with H6266 intra coding. For the network structure, some RNN (recurrent methods 23 were proposed in the early development stage, but most of recent methods are based on CNN (convolutional neural network).
neural
Deep video compression Existing works for deep video compression can be classified into two categories. i.e. non-delay-constranne and delay- celay-constrained. For the first category, there is no restriction on reference frame location, which means that the reference frame can be from future. For example, Wu et al. [IO] proposed interpolating the predicted frame from previous and future frames, and then the frame residue encoded. Djelouah et al. [ also followed this coding structure and introduced the optical flow estimation network to get better predicted frame. Yang et al. [6] designed a recurrent enhancement module for this coding structure. In addition, 3D autoencoder was proposed to encode group of pictures in [4 (25). This is natural extension of deep image compression by increasing the dimension of input. It is noted that this coding manner will bring larger delay and the GPU memory cost will be significantly increased. For the delay- consttcoodt methods, the reference frame is only from the previous frames. For example, Lu et al. [3] designed the DVC model, where all modules in traditional hybrid video codec are replaced by networks. Then the improved model DVCPro which adopts more advanced entropy model from [14] and deeper network was proposed al. designed more advanced optical flow estimation in scale space. Hu es al. [26] considered the rate distortion optimization when encoding motion vector (MV). In [7), the single reference frame is extended multiple reference frames. Recently, Yang et al. [6] proposed an RNN-based MV/residue encoder and decoder. In [I]]
in []. Following the similar framework with DVC, Agustsson
the residue is adaptively scaled by learned parameter.
Our research belongs to the delay-constrainee method as it can be applied more scenarios, e-g. real time communication. Different from the above works, design conditional ooding-based framework rather than following the commonly-used residue coding. Other video tasks show that utilizing temporal information condition is helpful [27 28 For video compression, recent works in 15 16), and 12 17 have some investigations on condition coding. In [15], the conditional coding is only designed for entropy modeling. However, due to the lack of MEMC. the compression ratio is not high, and the method in [5] cannot outperform DVC in terms of PSNR. By contrast, our conditional coding designed for encoding, decoding, and entropy modeling can significantly outperform DVCPro. In [16), only encoder takes the conditional coding. However, for decoder, the residual coding is still adopted. As latent state is used, the framework in [T6] is difficult train
[7]. By contrast, we use explicit MEMC to guide the context learning, which is easier to train. In [7], the video contents need to be explicitly classified into skip and non-skip modes, where only the contents with non-skip mode use the conditional coding. By contrast, our method does not need the handcrafted operation to decompose the video. In addition, the condition in DCVC the context in feature domain, which has much larger capacity. In summary, when compared with [15, [66. and
12 17, the definition, usage, and learning manner of the condition in DCVC are all innovative.
3 Proposed method
In this section, we present the details of the proposed DCVC. We first describe the whole framework of DCVC. Then we introduce the entropy model for compressing the latent codes, followed by the approach of learning the context. At last, we provide the details about training.
3.1 The framework of DCVC
In traditional video codec, the inter frame coding adopts the residue coding, formulated as:
2, faee (Leen =i)1)) + ã, with H, = Spredict (2)-1).
(I)
3

---

Encoding
Input frame x
Contextual encoder
Contextual decader
Decoded frame &
Bitstream
Enc
Dec
modal
Decoding
Contest
Context refinement
Decoded frame f
estimation
Motion
m
Enc
81
MV Dec
Warp
R
Feature extractor
Figure 2: The framework of our DCVC.
Ty is the current frame. & and -;-1 are the current and previous decoded frames. For simplification. we use single reference frame in the formulation. [ena(:) and saee(-) are the residue encoder and decoder. [.] is the quantization operation. [predict()) represents the function for generating the predicted frame .. In traditional video codec, Spredict (-) is implemented in the manner of MEMC, which uses handcrafted coding tools to search the best MV and then interpolates the predicted frame. For most existing DL-based video codecs (39), fpredict (-) is the MEMC which is totally composed of neural networks.
In this paper, we do not adopt the commonly-used residue coding but try to design a conditional coding-base based framework for higher compression ratio. Actually, one straightforward conditional coding manner is directly using the predicted frame ãt as the condition:
Jdec (x. 5t) with i, = (
However, the condition still restricted pixel domain with low channel dimensions. This will limit the model capacity. Now that the conditional coding is used, why not let model learn the condition by itself? Thus, this paper proposes a contextual video compression framework, where we use network to generate context rather than the predicted frame. Our framework can be formulated as:
fdee (1fenc (x; [5)]] 2.) with T. (
Somttent( (.) represents the function of generating context Tt. Sena(.) and s are the contextual encoder and decoder, which are different from residue encoder and decoder. Our DCVC framework is illustrated in Fig. 2
To provide richer and more correlated condition for encoding Iz, the context the feature domain with higher dimensions. In addition, due to the large capacity of context, different channels therein have the freedom to extract different kinds of information. Here we give an analysis example in Fig. 3 In the figure, the upper right part shows four channel examples in context. Looking into the four channels, we can find different channels have different focuses. For example, the third channel seems to put lot of emphases on the high frequency contents when compared with the visualization of high frequency
Ty, By contrast, the second and fourth channels look like extracting color information, where the second channel focuses on green color and the fourth channel emphasizes the red color. Benefiting from these various contextual features, our DCVC can achieve better reconstruction quality, especially for the complex textures with lots of high frequencies. The bottom right image in Fig. 3 shows the reconstruction error reduction of DCVC when compared with residue coding-based framework. From this comparison, DCVC can achieve non-trivial error decrease on the high frequency regions in both background and foreground, which are hard to compress for many video codecs.
As shown Fig. the encoding and decoding of the current frame are both conditioned on the context Tr. Through contextual encoder, æ, is encoded into latent codes yt- Yt is then quantized as jr via rounding operation. Via the contextual decoder, the reconstructed frame à, finally obtained. In our design, we use network to automatically learn the correlation between I; and Ze and then

---

Input frame X
Channel examples context &
Previous decoded frame -
Motion vector f,
High frequency X
Reduction of reconstruction
15
O: high frequency region in background
high frequency region in foreground
new content region
Figure 3: Visual examples from videoSRCO5 in MCL-JCV (9 dataset. The high frequency in zt decomposed by discrete cosine transform. It shows that DCVC improves the reconstruction of high frequency contents in both background with small motion and foreground with large motion. In addition, DCVC is good encoding the new content region caused by motion, where the reconstruction error can be significantly decreased compared with residue coding- framework DVCPro 4]. The BPP (bits per pixel) of DCVC (0.0306) is smaller than that of DVCPro (0.0359).
remove the redundancy rather than using fixed subtraction operation in residue coding. From another perspective, our method also has the ability to adaptively use the context. Due to the motion in video, contents often appear in the object boundary regions. These find a good reference in previous decoded frame. In this situation, the DL-based video codec with frame residue coding is still forced to encode the residue. For the new contents, the residue can be very large and the inter coding via subtraction operation may be worse than the intra coding. By contrast, conditional coding framework has the capacity to adaptively utilize the condition. For the new contents, the model can adaptively tend to learn intra coding. As shown in reconstruction reduction in Fig. the reconstruction error of contents be significantly reduced.
contents probably cannot
In addition, this paper not only proposes using the context H, to generate the latent codes, but also proposes utilizing it to build the entropy model. More details are introduced in the next subsection.
3.2 Entropy model
According to 330, the cross-entropy between the estimated probability distribution and the actual latent code distribution is tight lower bound of the actual bitrate, namely
R(úr) 2 Eg. (- logaPs. (ft)).
Pa. (ūr) and qg (ft) are estimated and true probability mass functions of quantized latent codes Ic- respectively. Actually, the arithmetic coding almost can encode the latent codes at the bitrate of cross-entropy. The gap between actual bitrate R(ū.) and the bitrate of cross-entropy is negligible. So our target is designing an entropy model which can accurately estimate the probability distribution of latent codes Pt. (.c). The framework of our entropy model is illustrated in Fig. First, we use the hyper prior model [I3] to learn the hierarchical prior and use auto regressive network (14] to learn the spatial prior. The two priors (hierarchical prior and spatial prior) are coseonly-used in deep image compression. However, for video, the latent codes also have the temporal correlation. Thus, we propose using the context H, to generate the temporal prior. As shown Fig. 4 we design temporal prior encoder to explore the temporal correlation. The prior fusion network will learn to
5

---

Context&
Side information
Temporal correlation
Temporal
HPE
prior
Spatial correlation
Auto
Prior fusion
regnessive
(1p del
Yt
AE
9:
Figure 4: Our entropy model used to encode the quantized latent codes ÿt. HPE and HPD are hyper prior encoder and decoder. Q means quantization. AE and AD are arithmetic encoder and decoder.
fuse the three different priors and then estimate the mean and scale of latent code distribution. In this paper, we follow the existing work [31 and assume that Po. (ūc) follows the Laplace distribution. The formulation of Pa. (ūx) is:
1 1 2 (ir.s) Pe: (Ue Te, 2e) = II (C(M, uu(11-p 2' 2
(5)
with Pe. T2, for (frna(2e), for (Dt, Sipe (ū:)).
The index i represents the spatial location. fhpa(-) is the hyper prior decoder network. far(-) is the auto regressive network. (.) is the specially designed temporal prior encoder network. fps (-) denotes the prior fusion network. In our entropy model, far (Vt.<s) and fipe(Et) provide the spatial and temporal priors, respectively. fhp)(&)) provides the supplemental side information which cannot be learned from spatial and temporal correlation.
The entropy model formulated in Eq. Buuiiiizss spatial prior for higher compression ratio. However, the operations about spatial prior are non non-parallel and then lead to slow encoding/decoding speed. By contrast, all operations about the proposed temporal prior are parallel. Thus, we also provide a solution which removes spatial prior but relies on temporal prior for acceleration, namely (r. oz, for (E)) (T)).
3.3 Context learning
For how to learn the context, one alternative solution is directly using plain network composed by several convolutional layers, where the input is the previous decoded frame ăt-1 and the output T.. However, it is hard for plain network to extract useful information without supervision. Video often contains various contents and there exist lot of complex motions. For a position in the current frame, the collocated position in the reference frame may have less correlation. In this situation, the collocated position in context H, also less correlated to the position in Tr, and the less correlated context cannot facilitate the compression of x,, For this reason, we also adopt the idea of MEMC. But different from commonly usage of applying MEMC pixel domain, we propose performing MEMC in feature domain. The context generation function feontext is designed as:
Scontext (k, = (u
(6)
We first design a feature extraction network ž,-1 = (ft-1) to convert the reference frame from pixel domain feature domain. At the same time, we use the optical flow estimation network (32 to learn the MV between the reference frame - and the current frame It. The MV m; is then encoded and decoded. The decoded m guides the network where extract the context through warping operation म, = warp(it-1, m.). The 3e is kind of relatively rough context because the warping operation may introduce some spatial discontinuity. Thus, we design a context refinement network (-) to obtain the final context v, = (Fz). In function Sconteare (-), MV m,, not only guides the network where to extract the context also enables network to learn context from larger reference region when compared with the solution without MEMC

---

Table 1: The BD-Bitrate comparison
Method
MCI.IIV
UVG
HEVC Class
HEVC Class C
HEVC Class
HEVC Class E
DCVC (proposed)
-23.9%
-2.3%
-26.0%
-5.8%
-17.5%
-11.9%
DVCPro
4
-4.1%
-7.9%
-9.0%
7.2%
-6.9%
17.2%
x265 (veryslow)
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
DVC 3
13.3%
17.2%
7.9%
15.1%
7.2%
21.1%
x264 (veryslow)
32.7%
30.3%
35.0%
19.9%
15.5%
50.0%
The anchor is x265 (veryslow). Negatíve number means bitrate saving and positive number means bitrate increase.
3.4 Training
The target video compression is using least bitrate to get the best reconstruction quality, Thus, the training loss consists of two metrics:
--
controls the trade-off between the distortion D and the bitrate cost can be MSE (mean squared eπror) MS-SSIM (multiscale structural similarity) for different targets. During the training, R is calculated as the cross-entropy between the true and estimated probability of the latent codes.
For the learning rate, it is set as le-4 at the start and le-5 at the fine-tuning stage. The training batch size set as For comparing DCVC with other methods, we follow and train models with different À (MSE: 256, 512, 1024, 2048; MS-SSIM: 8, 16, 32, 64).
4 Experimental results
4.1 Experimental settings
Training data We use the training part in Vimeo-904 septuplet dataset B3 as our training data. During the training, we will randomly crop videos into 256x256 patches.
Testing data The testing data includes HEVC Class B (1080P), C. (480P), D (240P), E (720P) from the common conditions [34) used by codec standard community. In addition, The 1080p videos from MCL-JCV 29 and UVG (35) datasets are also tested.
Testing settings The GOP (group of pictures) size is same with 4 namely 10 for HEVC videos and 12 for non- HECH videos. As this paper only focuses on inter frame coding, for intra frame coding, directly existing deep image compression models provided by CompressAl [36]. We use cheng2020-ancchn 20 for MSE target and use hyperprior [3] for MS-SSIM target.
According to the performance comparison in 37 38 31 DVCPro is one previous SOTA DL- based codec among recent works 8 9 26 Thus, we compare DVCPro in our paper. addition, its predecessor DVCB is also tested. It is noted that, for fair comparison, DVC and DVCPro are retested using the same intra frame coding with DCVC. For traditional codecs, x264 and x265 encoders [39] are tested. The settings of these two encoders are same with [4] except two options. One that we use veryslow preset rather than veryfast preset. Veryslow preset can achieve higher compression ratio than veryfast preset. Another is rather than constant rate factor setting to avoid the influence of rate control.
the constant quantization parameter setting
4.2 Performance comparison
Compression ratio Fig. 5]and Fig. show the rate- rate-distortion curves among these methods, where the distortion Fig. 5iis measured by PSNR and the distortion in Fig. [o measured by MS-SSIM. From these figures, we can find that our DCVC model can outperform DVCPro for all bitrate ranges. Table ggives the corresponding BD-Bitrate 40 results in terms of PSNR. When compared with x265 using veryslow preset, DVCPro achieves 4.1%, 7.9%, 9.0%, and 6.9% bitrate saving on MCL-JCV, UVG, HEVC Class B, and D, respectively. However, for HEVC Class and E, DVCPro performs worse, and there are 7.2% and 17.2% bitrate increases. By contrast, our DCVC can outperform x265 on all

---

MCL-JCV
UVG
HEVC Class B
15
15 (00)
POWR (RR)
DMC
Dut Pm
Dycho
DCVC (pososed)
DCVC laropoccal (verysber) lrngylow
DCst (poxscoed) becystoed
52-
0.04
0.00
0.30
0.36
0.05
0.22
HEVC Class
HEVC Class D
HEVC Class
30-
-22
(001
22
PORR
DMC
DVC
DWCho
28
X283 (vrysloa)
(verpaie))
0.30
0.15
0.20
0.55
0.40
0.15
0.20
0.25
6.30
0.35
0.40
02
6FF
DPP
Figure 5: PSNR and bitrate comparison. The horizontal axis is bits per pixel (BPP) representing bitrate cost and vertical axis is PSNR representing reconstruction quality.
datasets. For three 1080p datasets (MCL-JCV, UVG, HEVC Class B), the bitrate savings are 23.9%, 25.3%, and 26.0%, respectively. For low resolution videos HEVC Class and D, the improvements are also adequate, i.e. 5.8% and 17.5%. For HEVC Class E with relatively small motion, the bitrate saving 11.9 1..9. From these comparisons, we can find that our DCVC can significantly outperform DVCPro and x265 for various videos with different resolutions and different content characteristics.
In addition, we can find that DCVC can achieve larger improvement on high resolution videos. This is because that high resolution video contains more textures with high frequency. For this kind of video, the feature domain context with higher dimensions is more helpful and able to carry richer contextual information to reconstruct the high frequency contents.
Complexity The MACs (multiply-accumulate operations) 2268G for DCVC and 2014G for DVCPro, and there about 13% increase. However, the actual inference time per 1080P frame is 857 ms for DCVC and 849 ms for DVCPro P40 GPU, and there is only about 1% increase, mainly due to the parallel ability of GPU.
4.3 Ablation study
Conditional coding and temporal prior In our DCVC, we propose using concatenation- based conditional coding to replace subtraction- based residue coding. At the same time, we design the temporal prior for entropy model. To verify the effectiveness of these ideas, we make the ablation study shown in Table 2 where the baseline is our final solution (i.e. temporal prior + concatenating context feature). From this table, can find that both concatenating RGB prediction and concatenating context feature improve with residue coding. In addition, we can find that the improvement of concatenating context feature much larger than that of concatenating RGB prediction. It shows the advantage of context in feature domain. From Table 2 we also find that the proposed temporal prior further boosts the performance, and its improvement under conditional coding (no matter concatenating RGB prediction or concatenating context feature) is larger than that under residue coding. These results demonstrate the advantage of our ideas.
compression ratio. verifies the benefit of conditional coding compared
Entropy model In DCVC, besides the hyper prior model, the entropy model for compressing the quantized latent codes ÿt utilizes both spatial and temporal priors for higher compression ratio. However, the drawback of spatial prior is slow encoding/decoding speed as it brings spatial depen- dency and non-paralle.. By contrast, operations about the proposed temporal prior are parallel. Thus, DCVC also supports removing spatial prior but relies temporal prior for acceleration.

---

MCL-JCV
UVG
HEVC Class
0.05
0.835
0.960
6.925
0.975
0.022
0.85
0.055
DVC
0.992
DCVC
0.50
0.500
<h6d (snrydoe)
<003 isayacel
4355 beeyssal
0.45
1254 (verssce
1255 Orrys01
0.25
2.10
0.15
0.20
0.25
2.30
3.55
9.0
1.0
3.20
0.40
0.55
030
0.33
5,29
0.50
0.5
0.40
HEVC Class
HEVC Class
HEVC Class
0/793
2.975
0J993
2.900
0.033
0.992
HD 35.55
0.953
6.952
2.805
0:33
DBC
DVC DACRs
3.500
DVChh
DCKC groponee:
6:955
DCC (anaasse)
0925
0.973
6PP
DFP
Figure 6: MS-SSIM and bitrate comparison. The DL--based codecs fine-tuned for MS-SSIM.
Table 2: Ablation study on conditional coding and temporal prior
Temporal prior
Concatenate context feature Concatenate RGB prediction
Bitrate increase
0.0%
5.4%
4.6%
8.7%
11.2%
12.9%
Benefiting from the rich temporal context, the model without spatial prior only has small bitrate increase. Table compares the performance influence of spatial and temporal priors. From this table, find that the performance has large drop if both priors either of these two priors, the performance can be improved a lot. When enabling both of them, the performance can be further improved. However, considering the trade-off between complexity and compression ratio, the solution only using hyper prior and temporal prior is better. These results show the advantage of temporal prior-based entropy model.
disabled. When enabling
4.4 Influence of intra frame coding and comparison with more baselines
To build the best DL-based video compression framework, we use the SOTA DL-based image compression as our intra frame coding. is noted that, fair comparison, DVC and DVCPro are retested using the same intra frame coding with DCVC, and their results are better than that reported in
Table 3: Ablation study entropy model
Entropy model
Bitrate increase
hyper prior spatial prior temporal prior
0.0%
hyper prior temporal prior
3.8%
hyper prior spatial prior hyper prior
4.6%
60.9%

---

HEVC Class
36
35
PSNR (cB)
CVPR20 (public)
SU_CCC22 (public)
ECCV20 (public)
DVC (public)
DVCPro tpublic)
DVC frame: chen2020 anchor)
DVCPro (I frame: chen2020-anchar)
chen2020-ancher)
0.05
0.10
0.15
0.20
0.25
0.30
0.35
BPP
Figure Performance comparison with the public results. The results of method with suffix *(public) are provided by [38] B1. The results of method with suffix *(i frame: cheng2020 use theng2020-anchhn as the intra frame coding.
their original papers. Fig. shows the performance comparison between retested DVC/DVCPro and their public results provided by [88 31. addition, Fig. also shows the results the recent works RY CVPR20 66. LU ECCV20 [5], and HU ECCV20[26], provided by [38] BT]. From the public results in this figure, we can find that DVCPro one SOTA method among recent works. The results of LU LU_ECCV20 and HU ECCV20 are quite close with DVCPro. When the intra frame coding of DVC and DVCPro uses SOTA DL-based image compression model cheng2020-anchor provided by Compress (36), their performance has large improvement. When using the same intra frame coding, our proposed DCVC method can significantly outperform DVCPro, as shown in Fig. 7
5 Discussion
In this paper, we make efforts on designing conditional coding-based deep video compression frame- work which has lower entropy bound than the commonly-used residue coding-based framework. Residue coding-based framework assumes the inter frame prediction is always most efficient, which is inadequate, especially for encoding new contents. By contrast, our conditional coding enables the adaptability between learning temporal correlation and learning spatial correlation. In addition, the condition is defined as feature domain context in DCVC. Context with higher dimensions can provide richer information to help the conditional coding. especially for the high frequency contents. In the future, high resolution video is more popular. High resolution video contains more high frequency contents, which means the advantage of our DCVC will be more obvious.
When designing a conditional coding-based framework, the core questions are What is condition? How use condition? And how to learn condition? In this paper, DCVC is solution which answers these questions and demonstrates its effectiveness. However, these core questions are still open. Our DCVC framework is extensible and worthy more investigation. There exists great potential in designing more efficient solution by better defining, using and learning the condition.
In this paper, do not add supervision on the channels in context during the training. There may exist redundancy across channels, and this is not conducive to making full advantage of context with high dimensions. In the future, will investigate how to eliminate the redundancy across the channels to maximize the utilization of context. For context generation, this paper only uses single reference frame. Traditional codecs have shown that using more reference frames can significantly improve the performance. Thus, how the design the conditional coding-based framework given multiple reference frames very promising. In addition, currently do not consider temporal stability of reconstruction quality, which can be further improved by post processing or additional training supervision (e.g., loss about temporal stability).
10

---

References
[1] B. Girod, E. G. Steinbach, and N. Faerber, "Comparison of the H. 263 and H. 261 video compression standards," " in Standards and Common Interfaces for Video Information Systems: Critical Review, 1995.
[2] B. Bross, J. Chen, J.-R. Ohm, G. Sullivan, and Y.KK. Wang. "Developments in international video coding standardization after AVC, with overview of Versatile Video Coding (VVC)," Proceedings of the IEEE, 2021.
[3] G. Lu, W. Ouyang. D. Xu, X. Zhang, C. Cai, and Z. Gao, "DVC: end-to-ent end deep video compression framework," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11006-1 1015, 2019.
[4] G. Lu, X. Zhang, W. Ouyang. L. Chen, Z. Gao, and D. Xu, "An end-to-end learning framework for video compression," IEEE transactions on pattern analysis and machine intelligence, 2020.
[5] G. Lu, C. Cai, X. Zhang, I.. Chen, W. Ouyang, D. Xu, and Z. Gao, "Content adaptive and error propagation aware deep video compression," in European Conference on Computer Vision, pp. 456-472, Springer, 2020.
[6] R. Yang. F. Mentzer, L. V. Gool, and R. Timofte, "Learning for video compression with hierarchical quality and recurrent enhancement," in Proceedings of IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020.
[7] J. Lin, D. Liu, H. Li, and F. Wu, "M-LVC: multiple frames prediction for learned video compression," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[8] A. Djelouah, Campos, S. Schaub-Meyer, and C. Schroers, "Neural inter-frame compression for video coding," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[9] E. Agustsson, D. Minnen, N. Johnston, J. Balle, S. J. Hwang, and G. Toderici, "Scale-space flow for end-to-end optimized video compression," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8503-8512, 2020.
[10] C.-Y. Wu, N. Singhal, and Krähenbühl, "Video compression through image interpolation," ECCV, 2018.
[11] R. Yang, Y. Yang. J. Marino, and S. Mandt, "Hierarchical autoregressive modeling for neural video compression." 9th International Conference on Learning Representations, ICLR, 2021.
[12] T. Ladune, P. Philippe. W. Hamidouche, L. Zhang, and O. Déforges. "Optical flow and mode selection for lcarning-based video coding," in 22nd IEEE International Workshop on Multimedia Signal Processing, 2020.
[13] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, "Variational image compression with scale hyperprior." 6th International Conference on Learning Representations, ICLR, 2018
[14] D. Minnen, J. Ballé, and G. Toderici, "Joint autoregressive and hierarchical priors for learned image compression," " arXiv preprint 180.:18090007366 2018
[15] J. Liu, S. Wang, W.-C. Ma. M. Shah, R. Hu, P. Dhawan, and R. Urtasun, "Conditional entropy coding for efficient video compression." " arXiv preprint 2rXiv:2008.09180 2020.
[16] O. Rippel, S. Nair, C. Lew, S. Branson, A. G. Anderson, and L. Bourdev, "Leanndd video compression," in Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3464-3463, 2019.
[17] T. Ladune, P. Philippe, W. Hamidouche, I. Zhang, and O. Déforges, "Conditional coding for flexible learned video compression," in Neural Compression: From Information Theory to Applications Workshop @ ICLR, 2021.
[18] L. Theis, W. Shi, A. Cunningham, and F. Huszár, "Lossy image compression with compressive autoen- coders," arXiv preprint arXiv:7703000395, 2017.
[19] J. Ballé, V. Laparra, and E. P. Simoncelli, "End-to-end optimized image compression," arXiv preprint arXiv: 1et.11111.000000 2017.
[20] Z. Cheng, H. Sun. M. Takeuchi, and J. Katto, "Leamed image compression with discretized gaussian mixture likelihoods and attention modules." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.

---

[21] G. Toderici, S. M. O'Malley. S. J. Hwang. D. Vincent, D. Minnen, S. Baluja, M. Covell, and R. Sukthankar, "Variable rate image compression with recurrent neural networks." e arXiv preprint arXiv:151100085. 2015.
[22] G. Toderici, D. Vincent, N. Johnston, S. Jin Hwang, D. Minnen, J. Shor, and M. Covell, "Full resolution image compression with recurrent neural networks," in Proceedings the IEEE Conference Computer Vision and Pattern Recognition. pp. 5306-5314, 2017.
[23] N. Johnston, D. Vincent, D. Minnen, M. Covell, S. Singh, T. Chinen, S. J. Hwang. J. Shor, and G. Toderici, "Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4385 4393, 2018.
[24] Pessoa, Aidos, P. Tomás, and M. A. Figueiredo, "End-to-end learning of video compression using spatio-temporal autoencoders." " in 2020 IEEE Workshop on Signal Processing Syxtems (SiPS), pp. 1-6, IEEE, 2020.
[25] A. Habibian, T. v. Rozendaal, J. M. Tomezak, and T. S. Cohen, "Video compression with rate-distortion autoencoders," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7033- 7042, 2019.
[26] Z. Hu, Z. Chen, D. Xu, G. Lu, W. Ouyang, and S. Gu, "Improving deep video compression by resolution- adaptive flow coding," in European Conference Computer Vision, pp. 193 209, Springer, 2020.
[27] W. Bao, W.-S, Lai. C. Ma. X. Zhang, Z. Gao. and M.-H. Yang. "Depth-aware video frame interpolation." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3703-3712, 2019.
[28] S. Niklaus and F. Liu, "Softmax splatting for video frame interpolation." in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5437-5446, 2020.
[29] H. Wang, W. Gan, S. Hu, J. Y. Lin, L. Jin, L. Song, P. Wang. I. Katsavounidis, A. Aaron, and C.-C. J. Kuo, "MCL-JCV: IND-based 264/AVC vidco quality assessment dataset," 2016 IEEE International Conference on Image Processing (ICIP), pp. 1509-155 1513, IEEE, 2016.
[30] C. Shannon. "A mathematical theory of communication," ACM SIGMOBILE mobile computing and communications review, vol. 5, no. 1, 3-55, 2001.
[31] "PyTorch line: accessed 12 April 2021.
On-
[32] A. Ranjan and M. J. Black, "Optical flow estimation using a spatial pyramid network," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4161-41700 2017.
[33] T. Xue, Chen, Wu, D. Wei, and W. T. Freeman, "Video enhancement with task-oriented flow." International Journal of Computer Vision (IJCV), vol. 127, no. 8, pp. 1106-1125, 2019.
[34] F. Bossen et al., "Common test conditions and software reference configurations," JCTVC-L1100, vol. 12, 2013.
[35] "Ultra video group test sequences." htttp://ultraviddeo.ccom tut. fi Online: accessed 12 April 2021.
[36] J. Bégaint, F. Racapé, S. Feltman, and A. Pushparaja, "CompressAl: PyTorch library and evaluation platform for end-to-end compression research," arXiv preprint arXiv:20/ 2020.
[37] D. Xu, G. Lu, R. Yang, and R. Timofte, "Learned image and video compression with deep neural networks." in 2020 IEEE International Conference on Visual Communications and Image Processing, VCIP 2020, Macau, China, December 1-4, 2020, pp. 1-3, IEEE, 2020.
[38] D. Xu, Lu, R. Yang, and R. Timofte, "Tutorial: Learndd image and video compression with deep neural networks." "https:/ /// /t:::://driii /diieedveee/ee//eeee/eeeeeeeee google. walom/file//ae a/ view Online; accessed 12 April 2021.
[39]
"Ffmpeg." httpp:://wttt http:://wwwwfffpg...o ffmpeg.org/
Online; accessed 12 April 2021.
[40] G. Bjontegaard, "Calculation average PSNR differences between RD-curves." VCEG-M33, 2001.
12


Reviews:
[{"review_id": "lgO9l-5Bnhe", "reviewer": "Reviewer_JZR2", "summary": "This paper extends an existing neural video compression method (DVCPro) by adding a two crucial components: (1) the usage of optical flow in warping latents; (2) the usage of optical flow to design a non-additive decoder.  These additions result in a state of the part (as stated by authors, but not verified by me) performance reported on a few (small) datasets usually used for video compression evaluation, outperforming HEVC.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper tackles the task of neural video coding.\n\nThis task, in one of its most common forms takes key-frames, which get encoded/decoded separately, and uses these to seed the start of a sequence of compressed frames. The sequence of compressed P-frames (\"predicted\") uses the previously decoded frame(s) and motion to come up with a hypothesis for what the current frame should be. The difference (residual) is then encoded in the bitstream.\n\nThis paper essentially does exactly this, but with a few crucial twists.\n\nThe motion information is used for warping latents (i.e., the quantized tensors which are used for either encoding/decoding or conditioning), which as far as I know is relatively uncommon. These are then used in various places within the encoder/decoder/entropy coder for the P-frame residual. \n\nSecondly, in a typical video compression framework, the output (\"decoded\") frame is computed as a warp of the previously decoded frame plus the residual (\"additive reconstruction\"). This typically causes some problems, which this paper avoids by providing the decoder instead with latent values computed from the previously decoded frame, with the decoder's task being to perform a full reconstruction from that and the additional information.\n\nIn addition to the hyper-prior (done per frame), the paper also employs a temporal prior model to boost performance.\n\nThe authors perform various ablations, which can be found in the supplementary materials. I feel this paper is incomplete by a large margin if we exclude the supplementary, where most of the details about the paper are found.\n\nOverall, I think this is a pretty good paper, but ONLY if we take into account the supplementary. Without them I felt scratching my head multiple times, but all the missing information that made me wonder \"why is this not discussed/presented/tested\" was present in supplementary.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "hVu8fUP6ZQc", "reviewer": "Reviewer_E81t", "summary": "The paper proposes to replace the minus/plus operations commonly used in neural video compression (to calculate/add the residual) with neural networks. The resulting contextual encoder/decoder networks are conditioned on the same context that is used for the entropy model, which is calculated by warping features extracted from the previously decoded frame.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Post rebuttal update: all my concerns were addressed].\n\nThe paper contains a solid idea (replacing minus/plus) and shows promising R-D results. My issues with the paper are:\n\na) Ablation studies: from the text it is not clear how the proposed method relates to \"DVCPro\", and Section 4.3. is written unclearly. Is the only difference that the proposed method uses contextual encoder/decoder nets and the generation net? If that is the case, the presented ablation studies make sense, but this has to be made more clear. I.e. I need a paragraph on what exactly the differences between the \"DVCPro\" and the \"DCVC (context in pixel domain)\" lines in Fig. 7 are. I am hoping that the second line can be renamed to sth like \"DVCPro + context in pixel domain\". If there are more novel components, the ablation study is not useful. In particular, I'm wondering what is meant with \"DVC and DVCPro are retested using the same intra frame coding\" (L229) and whether the training strategy mentioned in Section 2 of the Appendix is used for all?\n\nb) Runtime concerns: there is only one line on complexity (L252). It is unclear which complexity is meant (number of weights, FLOPS, fps, ...?). I would expect the feature extraction and refinement operations (which happen at full res) to be costly, and they are needed for encoding and decoding. As such, I'm surprised how the \"complexity\" can be the same as DVCPro? \n\nc) Contributions list is overly vague and it's unclear which things are novel. \"[...] jointly utilizing the spatial and temporal correlation, [...]\" (L66) does not seem novel. \"[...] more accurate probability estimation can be obtained\" (L67) is not ablated in the text. \"We define the condition as the context in feature domain\" seems vaguely familiar to what Rippel et al. were doing in their 2018 paper.\n\nd) Entropy Coding? L169: \"the gap between the actual bitrate ... and the cross-entropy is negligible\": not always, depends on the precision of your arithmetic. This statement makes me wonder whether the presented graphs are using the real, entropy coded bitrate?\n\ne) ffmpeg settings: I did not follow why the arguments \"-r\", and \"-vframes\" are needed.\n\nf) Various issues with the text:\n- L8 and throughout: I think what is called \"condition\" should be called \"conditioning signal\". \n- L28: Reference [11] does not contain the entropy formula, and intuitively it is unclear how the entropy of the substraction is higher, I think entropy is the wrong concept to be used here, or I'm missing something in which case clarification is needed.\n- L122: \"MEMC which is totally composed of network\": not sure what this means or how it's different from the proposed method.\n- L167, L209: the \"true\" probability mass function of y is of course unknown, we are using the empirical distribution to estimate the rate. Similarily, the statement on L169, \"the gap betweeen the actual bitrate ... and the cross-entropy ...\" \n- L179: Any reason/intuition for using Laplace?\n- L186ff seem similar to L133ff? What is meant with a \"plain\" network on L186?\n\nSummary:\nOverall I like the idea, but I think the paper does not sufficiently place it within the context of previous works.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not addressed societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "dxt90TF0Wbd", "reviewer": "Reviewer_jmWx", "summary": "The author propose a video compression framework that leverages context model in place of conventional residual compressor.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "pros:\n1. use a context model to improve the compression performance\n2. Better rate-distortion performance over DVCpro baseline\n\ncons:\n1. Need more neural baselines.\n2. Conditional entropy model is not a novel idea for video compression. (see point 4)\n3. Spatial autoregressive prior is not a proper solution for video, as it will significantly increase both encoding and decoding time. Its autoregressive feature makes it hard to be parallelized. In addition, entropy coding usually happens in CPU but network inference happens in NPU/GPU, the accumulated communication cost in autoregressive calls will further worsen the coding efficiency. Given the fact that video decoding is sensitive to runtime (high frame rate), I disagree to use any spatial autoregressive model here.\n4. I'm skeptical about the effectiveness of the context model. \n    - Author tries to concatenate $\\bar{\\textbf x}_t$ in en/decoder to demonstrate that it's better than residual, but I didn't see there is any theoretical difference. This can simply be regarded as a different way to incorporate residual, and the concatenation method requires more model parameters and conv layers. The invertibility of the original residual also makes the model easier to train.\n    - Similar idea was already proposed to improve the compression (Yang et al.). They interpret the residual compression as an invertible transform (or normalizing flow) and propose a similar(also simpler) conditional prior.\n    - $\\textbf{Comments}$: The context $\\bar x$ also acts as a condition of the conditional entropy coder. As the concatenation method is simply a different form of residual, I think the performance improvement may mainly come from the conditional entropy coder. Extra conditioning makes the prior fit the distribution of the latent variable better (for better probabilistic model); residual is a cheap way to reduce the redundancy and removing high-frequency information (for better transform). They are aimed at solving the same problem in slightly different ways. \n\nYang, Ruihan, et al. \"Hierarchical Autoregressive Modeling for Neural Video Compression.\" International Conference on Learning Representations. 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "U23F_hPX52d", "reviewer": "Reviewer_JwvE", "summary": "In this work the authors propose a new model for video compression that, instead of warping the previous frame and predicting the current frame, extract and warp features using the motion vectors. As a result, the encoder doesn’t rely on residual compression (residual between ground truth and prediction) but directly encodes the necessary information needed to decode the frame.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nSuch an approach to video compression is relatively new as most related works indeed focus on predicting a full frame then encoding a residual image. It has however strong links with frame interpolation methods such as:\n\n-Softmax Splatting for Video Frame Interpolation\n\n-Depth-aware video frame interpolation\n\nwhich also warp context information before image prediction. The authors also didn’t discuss some other important related work such as “Learned Video Compression, ICCV 2019” which also goes a little into the direction of a conditioning not limited to residuals only.\n\nThere is also a strong link with [11, 14, 15] but this is sufficiently discussed in the paper\n\n**Quality**\n\nThe idea presented in the paper is intuitive and leverages all the knowledge gained in both video compression and frame interpolation. Quantitative evaluation shows state of the art results on a large variety of test sets. I still have the following issues:\n\n-The evaluation only presents a small set of methods. Since it is done on the same datasets it would be important to present other top performing methods for better context. Is there a reason not to include them?\n\n-Only one quality evaluation is presented in supplementary material. Such a figure should appear in the main paper. In addition to this, comparisons with more methods should be included as well. \n\n-Temporal stability is not discussed and the authors didn’t provide a video as supplementary material which I think is quite important to evaluate the work. Also the authors didn’t explain how to maintain similar quality between the key frame and the interpolated frames\n\n-The discussion around the information represented in the feature domain lacks precision. In particular the discussion on the link between the channel 1 in Figure.3 and the motion vectors. Visually one can argue that high intensity values correspond to the “red players”. An ablation study would be required to better understand the content: all values to 0 or set to the average and see the effect on the image. \n\n\n**Clarity**\n\nIn general the paper is clear and easy to follow.\n\n-“MEMC” is used without being introduced. I suppose it stands for Motion Estimation and Motion Compensation? Same comment about “MV” \n\n-l.122: … of network. => “of neural networks”?\n\n-The discussion section is a little rough and would benefit from a revision.\n\n**Significance**\n\nThe authors proposed a new approach for learning based video compression which seems to be setting a new state of the art result. \n\n**After Rebuttal**\n\nThe authors answered most of my questions and concerns in their rebuttal. I would insist they include more visual and qualitative examples in the revised version.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed some limitations, in particular the fact that the proposed solution is another conditioning which is slightly more complex and rich than image prediction but other alternatives remain to be explored. I think the discussion could be extended to other considerations like temporal stability.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new video compression model that uses motion vectors to extract and warp features instead of warping the previous frame for predicting the current frame. As a result, the encoder doesn’t rely on residual compression but directly encodes the necessary information needed to decode the frame. 

The paper leads to a new state of the art in video compression and merits publication. Unfortunately, the main paper lacked several citations and baseline comparisons. Some of these comparisons were shown only in the appendix, making the main paper appear incomplete. I strongly urge the authors to present these results in the main paper. In addition, I also encourage the authors to run additional baseline comparisons of the related works mentioned by the reviewers (such as Ruihan Yang et al., ICLR 2021). 
Decision: Accept (Poster)