Versatile Learned Video Compression
Authors: Runsen Feng
Zongyu Guo
Zhizheng Zhang
Zhibo Chen
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Video Compression
Deep Learning

Abstract: Learned video compression methods have demonstrated great promise in catching up with traditional video codecs in their rate-distortion (R-D) performance. However, existing learned video compression schemes are limited by the binding of the prediction mode and the fixed network framework. They are unable to support various inter prediction modes and thus inapplicable for various scenarios.In this paper, to break this limitation, we propose a versatile learned video compression (VLVC) framework that uses one model to support all possible prediction modes.Specifically, to realize versatile compression, we first build a motion compensation module that applies multiple 3D motion vector fields (\ieno, voxel flows) for weighted trilinear warping in spatial-temporal space. The voxel flows convey the information of temporal reference position that helps to decouple inter prediction modes away from framework designing.Secondly, in case of multiple-reference-frame prediction, we apply a flow prediction module to predict accurate motion trajectories with a unified polynomial function. We show that the flow prediction module can largely reduce the transmission cost of voxel flows.Experimental results demonstrate that our proposed VLVC not only supports versatile compression in various settings but also achieves comparable R-D performance with the latest Versatile Video Coding (VVC) standard in terms of MS-SSIM.

Versatile Learned Video Compression
Anonymous Author(s)
Affiliation Address
email
Abstract
Learned video compression methods have demonstrated great promise in catching up with traditional video codecs in their rate-distortion (R-D) performance. How- ever, existing learned video compression schemes are limited by the binding of the prediction mode and the fixed network framework. They are unable to support various inter prediction modes and thus inapplicable for various scenarios. In this paper, to break this limitation, we propose versatile learned video compression (VLVC) framework that uses one model to support all possible prediction modes. Specifically, realize versatile compression, module that applies multiple 3D motion vector fields (i.e., voxel flows) for weighted trilinear warping in spatial-temporal space. The voxel flows convey the information of temporal reference position that helps to decouple inter prediction modes away predic-
3
5
first build motion compensation
10
11
12
from framework designing. Secondly, in case of multiple-reference-fiee tion, we apply flow prediction module to predict accurate motion trajectories with unified polynomial function. We show that the flow prediction module can largely reduce the transmission cost of voxel flows. Experimental results demonstrate that our proposed VLVC not only supports versatile compression various settings but also achieves comparable R-D performance with the latest Versatile Video Coding (VVC) standard terms of MS-SSIM.
13
14
15
16
17
18
Introduction
19
Video occupies more than 80% of network traffic and the amount of video data is increasing rapidly [. Thus, the storage and transmission video become more challenging. series of hybrid video coding standards have been proposed, such AVC/H.264 264 (2), HEVC/H.265 [3] and the latest video coding standard VVC/H.260 [4] These traditional standards are manually designed, evolving for decades. However, the development within the hybrid coding framework is gradually saturated.
21
23
25
Recently, the performance of video compression is mainly improved by designing more complex prediction modes, leading to increased coding complexity.
27 Deep neural networks are currently promoting the development data compression. Despite the remarkable progress on the field of learned image compression [5] the area of learned video compression still in early stages. Existing methods for learned video compression can grouped into three categories, including frame interpolation-basee based methods [2], 3D autoencoder- based methods 13 [4], and predictive coding methods with optical flows 15H17 So far, among them, video compression with optical flow presents the best performance [18), where optical flow represents pixel-wise motion vector (MV) fields utilized for inter frame prediction. In this paper, we also focus this predictive coding architecture. Previous works with optical flow are proposed to support 35 specific prediction mode, including unidirectional or bidirectional, single or multiple frame prediction. They too cumbersome to support versatile compression in various settings since they bind the inter 37 prediction mode with the fixed network framework. is important to design a more flexible model to handle all possible settings like traditional codecs. In this paper, propose versatile learned video
28
29
31
33
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

(a)
(b)
(c)
(d)
Figure Different motion compensation (inter frame prediction) methods. (a) Unidirectional prediction with 2D optical flow [13). (b) Bidirectional prediction with two optical flows and weight
coefficients [12]. (c) Prediction with single voxel flow, freely sampling the reference frames in
space-time. (d) Prediction with multiple voxel flows via weighted trilinear warping.
39
compression (VLVC) framework that achieves coding flexibility as well as compression performance.
voxel flow based motion compensation module adopted for higher flexibility, which is then extended into multiple voxel flows to perform weighted trilinear warping. In addition, in case of multiple-reference-fiee frame prediction, polynomial motion trajectories based flow prediction module is designed for better compression performance. Our motivations are described as follows.
41
43
44
Motion compensation with multiple voxel flows. Previous works such [15] apply 2D optical flows for low-delay prediction using single reference frame (unidirectional prediction, see Fig. Tap. For the practical random access scenario, bidirectional reference frames available for more accurate frame interpolation [1] (Fig. [bb.. However, the reference positions in these works are determined by pre-defined prediction modes. They cannot adapt various inter prediction modes where reference positions are various. In this paper, we apply 3D voxel flows to describe not only the spatial MVs, but also the information of temporal reference positions (Fig. Icl. We perform voxel flow based motion compensation via trilinear warping, which is applicable to single or multiple, unidirectional or bidirectional reference frames. Unlike (16] that adopts scale space flow with trilinear warping, we apply voxel flows for inter prediction in spatial-temporal space, which naturally renders our model more robust different coding scenarios. Furthermore, beyond using single MV in every position of the current frame, we propose to use multiple voxel flows to describe multiple possible reference relationships (Fig. ld). Then the target pixel is synthesized by weighted fusing of the warping results. We show that without increasing the coding cost of motion information, the motion compensation is thus more accurate, yielding less residuals and more efficient compression.
45
47
49
51
52
59
54
55
56
57
58
59
Flow prediction with polynomial motion trajectories. Exploiting multiple reference frames usu- ally achieves better compression performance since more reference information is provided. A versatile learned video compression model should cover this multi-reference case. While previous work [17 designs complex flow prediction network reduce the redundancies of 2D MV fields, the number and structure of reference frames are inherent and fixed within the framework. In this paper, design a more intelligent method for flow prediction, i.e., modelling the prediction modes with polynomial coefficients. We formulate different motion trajectories in a time interval by unified polynomial function. The polynomial coefficients are solved by establishing a multivariate equation (see Section 3.2). Since this polynomial function models the accurate motion trajectories, it serves as
61
82
63
64
85
86
87
68
a basic discipline that constrains the predicted motion be reasonable. We show the transmission cost of voxel flows is reduced obviously with the help of additional motion trajectory information.
69
Thanks the above technical contributions, proposed VLVC is not only applicable for various practical compression scenarios with different inter prediction modes, but also delivers impressive R-D performance on standard test sequences. Extensive experimental results demonstrate that our method is the first one to achieve comparable performance with VVC in terms of MS-SSIM in both
71
72
73
74 low delay and random access configurations. Comprehensive ablation studies and discussions are provided to verify the effectiveness of our method.
75
2
Related Work
77 Learned Image Compression Recent advances in learned image compression 5 have shown the great success of nonlinear transform coding. Many existing methods are built upon hyperprior-

---

79 based coding framework (6], which are improved with more efficient entropy models 7 8 variable- rate compression [19] and more effective quantization ( [0]. While the widely used autoregressive entropy models provide significant performance gain in image coding, the high decoding complexity not suitable for practical video compression. We thus employ the hyperprior model [6] without
80
81
82
88
context models in our video compression framework.
84
20 24 can be roughly divided 24 3D nutoencoder-base methods 13 14,, and predictive coding methods with optical flows 17 Currently, researchers are more interested in the latter two methods. Although 3D autoencoderrbaseet methods requires less time complexity, they barely achieve comparable performance with x265 in MS-SSIM [14], Meanwhile, predictive coding methods with optical flows have outperformed HM in terms of PSNR [18.
Learned Video Compression Existing approaches 15| 17
18
85
into three categories: frame interpolation-basel methods [1] 12
86
87
88
89
90
Predictive-based video compression approaches [13 20 [2 sequentially perform motion estimation, motion compression, motion compensation and residual compression. Chen et al. [24 first propose to predict block of pixels using learned neural network (DNN), and the residual is compressed by RNN-based autoencoder. Wu et al. [[]] propose interpolation bassodd----dddd--- approach using traditional MVs. Lu et al. [15] propose an fully end-to-end trainable framework, where all key components in the classical video codec are implemented with neural networks. Rippel et al. (21 jointly compress the motion and residual information, and propose a latent state to memorize the information from the past. Djelouah et al. [22 perform interpolation by the decoded optical flow and blending coefficients. They reuse the same autoencoder of I-frame compression and directly quantize the corresponding latent space residual. Liu al. [23] combine the optical flow estimation and motion compression networks into one-stage, and remove the redundancy of quantized flow representations using joint
91
92
94
95
96
97
98
99
100
101
priors. Yang et al. (20 propose a video compression framework with three hierarchical quality layers and recurrent enhancement. In [17], multiple frames motion prediction are introduced into the P-frame coding. Lu et al. [22 propose an content adaptive and error propagation aware method to reduce error accumulation and achieve adaptive coding. Agustsson et al. [16] replace the bilinear warping operation with scale-space flow which allows the model adaptively blur the reference content for better warping results. However, most existing methods are designed for particular prediction modes, resulting in inflexibility for different scenarios.
102
103
104
105
106
107
108
Video Interpolation The task of video interpolation is closely related to video compression. One pioneering work (25) proposes to use deep voxel flow to synthesize new video frames. Some works of video interpolation (66-2 directly generate the ppatially-ddaptiv convolutional kernels for each motion vectors by neural networks. Most recently. 229 (30] proposed to relax the kernel shape, allowing the models to freely select multiple sampling points in space or space-time. In this paper, our employed multiple voxel flows is motivated by the accurate interpolation result in [30]
109
110
111
112
113
3
Versatile Learned Video Compression
114
115
To compress video, the original video sequence is first divided into groups of pictures (gop). Let
116 = {æi, (2,,, denote the frames in one gop unit where the gop size is T. To take advantage of previous decoded frames, our model predicts the current frame D; from n reference frame(s),
117
118
i.e., the lossy reconstruction results compared the original frames. Here, we denote the reference frames as (á.,, -t2,.............. .... R.,, J. where {t1, f2. tx } is the index of temporal reference position. If multiple frames are taken as the reference (i.e., n 1), the reference frames are divided into two groups: one is used only for flow prediction, and the other is used for both flow prediction and motion compensation. In other words, the reference involved for motion compensation is only a sub-set of
119
120
121
122
( Ltz, ..., }. J. which could be concatenated into a volume denoted by X,. If only one reference frame is available, the volume for warping X, = {
123
124
125 An overview of our video compression framework is shown in Fig. Previous work [6 demon- strates that an implicit flow encoder can outperform pre-trained optical flow network and simplify the network structures simultaneously. In our paper, we also abandon the use of pre-trained optical
126
127
128
flow network in motion encoder. The motion encoder and decoder are similar to image compression network [5] While the work of [16] sends current frame and previous reconstruction into motion encoder, we make some modifications on the input of motion encoder. Specifically, in our framework,
129
130
the motion encoder is fed with the current frame D; concatenated with predicted frames (represented
3

---

Reconstruted frame it
Reference frames
Voxel flows
We iighted Tri I near Warping
V Volume for warping
L..
gr
Generalized O ptical Flow Prediction
Motion Decoder
Residual Decoder
Predicted optical flow fe- fe-ta
+ Compensation frame
Bi linear Warping og
m,
Conca tenat i on
Motion Encoder
Predicted frames
Residual Encoder
Optical Flow Encoder
Conca tenat i
Current frame
Figure 2: Overview of our inter-frame coding framework.
132 as x in Fig. 2.
Here, the predicted frame x/ is an estimation of current frame Dt. The flow prediction module will predict 2D optical flows fi-t, to warp corresponding reference frames Lt,, each of which will generate predicted frame x/,, The predicted frames reveal how much information the decoder knows about current frame.
134
136
On the decoder side, a motion decoder will generate voxel flows which is used for motion compensa- tion via trilinear warping (details are explained in Section 3.1] In addition. when multiple reference frames are available, the flow prediction module turned on (the red dashed box). As shown in
137
138
139
Fig. 2 the flow prediction module is auxiliary for motion encoding and decoding, which reduces the transmission cost of the quantized motion latent m,, The specific mechanism of flow prediction
140
obtain prediction of current frame as Tt. The residual encoder and decoder are then used to compress the remaining residuals between the original frame et and the predicted frame ãt, yielding the final reconstructed frame &; and quantized
can be found in Section 3.2 After motion compensation,
142
144
latent f..
145
3.1
Motion compensation with multiple voxel flows
Voxel flow 25 is per-pixel 3-D motion vector that describes relationships in spatial-temporal domain. Compared to 2D optical flow, voxel flow can inherently allow the codec to be aware of the sampling positions in the temporal dimension for various prediction modes. Given arbitrary number of reference frames, the model is expected to select the optimal reference frame for better reconstructing the current frame to be compressed. Such 3-D motion descriptor helps to build prediction-modellggnnsti video compression framework, i.e., versatile learned video codec.
147
149
150
151
In addition, single flow field is hard to represent complex motion (e.g. blurry motion), which may result in inaccurate motion compensation or high coding cost of motions. When reconstructing a local region, its reference information may not come from only one souree. Considering practical scene where multiple objects of the same types appear at the same time, more than one areas could be referred for reconstructing the local region. Thereby, in this work, further propose to use multiple voxel flows to perform weighted trilinear warping by sampling in , for multiple times. We remind our readers that X is volume consisting of some reference frames. Assume the dimension of X, is x H x W (usually reshaped into H x W x D for warping), where is the number of reference frames used for motion compensation. the motion decoder will generate multiple voxel
153
155
157
159
160
181
flows by outputting (4M) x H x W tensor. Here, M refers to the number of flows. Therefore, every voxel flow is a 4-channel field that describes the 3-channel voxel flow g = (g Ty g with corresponding weight channel g Here, (1 S S M) is the index of voxel flow. To synthesize the target pixels in current frame, the weights g', normalized by softmax function across M voxel
162
163

---

185
flows. We finally obtain the target pixel Z[z, v) in spatial location [x. v] by calculating the weighted sum of sampling results, formulated as:
186
T c v]
you (2r, v) X,( (2,,)) oy (z,
(I)
167
We experimentally find that compared with single voxel flow, the transmission cost of multiple voxel flows does not increase largely. The model is able assign appropriate number of flows under the rate-distortion optimization goal. In other word, the model is optimized to avoid the transmission of unnecessary flows. Meanwhile, due to more accurate inter frame prediction, the transmission cost of residuals decreases obviously by using multiple voxel flows for weighted warping.
168
189
170
171
172
3.2
Generalized optical flow prediction
173
In our proposed VLVC framework, as illustrated in Fig. 2 we compress the information via motion encoder and decoder. The concatenation of the predicted frames (i.e., bilinear warping results using the predicted optical flow) and the current frame are fed into the motion encoder. The 3-D motion descriptor, i.e., the voxel flows, are then decoded by the motion decoder given the quantized motion latent and the feature of predicted optical flow. In this process, the predicted optical flow reduces the spatial displacement need to be encoded, and also serves as the conditions to promote the generation of voxel flows. Thus, the optical flow prediction is clearly of great importance to reduce the redundancies of consecutive voxel flows in case of using multiple reference frames.
motion
174
175
176
177
178
179
181
Specifically, there are two optical flow describe the motion between the reference frame &t and the target frame zy: ft, and fi-t,- The flow fi-t, describe the motion of each pixel from Zt. and therefore we can sample &t, for each pixel in the target frame x; via bilinear (backward) warping. However, fi-t, is unknown at decoder side because the pixels of target frame is unavailable. Fortunately, the pixels from reference frames are known both encoder and decoder. We can first estimate the optical flow of pixels from reference frame bolir,, to other reference frames, and then predict the flow ft, While we obtain fty it cannot be directly used for motion compensation with bilinear warping.
182
183
184
185
186
187
189
Recently work (31) for video interpolation proposed a forward warping method interpolate the target frame o, by directly using the flow fe,-t- For video compression, we aim to predict a approximation of the flow fi-1, to reduce the redundancies of the proposed voxel flows for better rite-distortion performance. We therefore employ the forward warping method [31] (named softmax
190
191
192
193
splatting) project the flow fi,-1 to f which is a kind of flow reversal methods similar to [32] In the following part, we will describe novel polynomial motion modeling method to predict ft,
194
195
given arbitrary reference frames and any target time stamp t. And flow reversal layer based on softmax splatting is introduced for the final flow prediction.
196
197
Polynomial motion modeling For each pixel at ljs we model the motion Jiy by the k-order (k n) polynomial functions:
(t- x
(t- +,. au (t tz)*
(2)
199 where ag. 01 ..., Og are the polynomial coefficients. To solve the coefficients, we set equals to the top-k nearest time stamp {t,, 1 around t, within the set of reference time stamp. Then we can 201 obtain the following equation:
(1i (4) (b, 12
(Gn - 44)
Seg-ots fty
02
(t1, t2)
(tz, 12) (t)-t)))
(3)
ak
(tix tj) (t), 22 (t,, - )2
202 where fey 1 ft .... fty can be obtained using off- the-shelf flow estimation network.
203 Then we can derive the polynomial coefficients and apply them to Eq. {3) predict the forward flow
from tj to any time stamp
5

---

t
n
fú
f 7
fc+t
O ptical Flow Estimation
Polynomial
Flow Reversal
Motion Modeling
L
Generalized Flow Prediction
Figure 3: Generalized flow prediction module.
205
Flow reversal via softmax splatting While the forward flow fty is predicted by the polynomial functions, cannot be directly used for motion compensation. Therefore, we introduce a flow reversal layer to forward warping ft, by softmax splatting [3I]
206
Σ (exp(Z) (- (-ft,-ihi Sty
f t-
(4)
Σ (exp(Z), (t,
208 where Σ is the summation splatting defined in 31. and Z is an importance mask generated from small network q as:
Z = q
[e, w (Zt: f (1,+);))l2).
(5)
k
210
where w the bilinear backward warping operator.
211
3.3
Loss function
In previous works, the reference frames determined according to pre-defined prediction modes. For example, the work of [17] applics four unidirectional reference frames, where the reference set 213
is { 3 2 -.-1), ). The work of [12) applies 1 (#,-2, .2+2), (.,-3, 2z+3) 215 as the reference set for bilinear prediction. In this paper, to optimize versatile video compression model, the model will have access to various reference structures during training adapt different
217
prediction modes. Therefore, we apply the loss function to cover all the frames in the entire gop as:
L Σ
Ra f.lB .... 2ry) + D(E Zo B, ... -1
Here, T is the gop size during training. The maximum value of T is seven in our experiments since 7-frame gop can cover most prediction modes. (æ.,, ..., &., represents different reference set that may vary in different mini-batches. R, P.) is the rate of motion and residual. For simplicity, omit the process of intra frame compression (at / = 1) in this loss function.
219
220
221
Experiments
222
223
4.1
Experimental setup
224
Model details The motion/residual compression modules are two auto-encoder style networks,
225
where the bit-rate are estimated by the factorized and hyperprior entropy model 7.7 respectively. We
employ the off-the-shelf PWC-net 33 as the optical flow estimation network only in our generalized
227
flow prediction module. We employ feature residual coding (34 instead of pixel residual coding for better performance. Detailed architecture can be found in supplementary.

---

HEVC Cass dataset
40
34
33
34
VINE Homasppp
30
30
+ah 21 jondowarees
196-6.22 (rodelay
14--4..3 (owdeay)
184-6-21 (INNERRs)
22
U (CC09922)
(MDC) ICVTR
(MS8) HEDVY
Lo (CVFRIE
0.0
0.1
32
01
0.1
dataset
0.99
HEVC Class C detanet
D'daaapet
0.50
0.90
VDC beneaaaeen
VIVC Janetrraccent
0:97
FM--S 23 tanderraceenl
VIN S2AM SOCTINASS
VTS-120 10002:0 (randerscceen
Harts 23 Howlddy
trandoraccens
199 28 opeaoraccms
(EDCV220
0.96
0.2
0.1
011
0.2
0.5
04
23
04
(a)
(b)
(c)
Figure 4: Rate-distortion Performance.
229
Training sets The models were trained on the Vimeo-90k septuplets dataset (355 which consists of
230
89800 video clips with diverse content. The video clips are randomly cropped to 128 x 128 or 256 x
256 pixel for training.
232
Testing sets The HEVC common test sequences 3 and the UVG dataset 33 are used for evalua- tion. The HEVC Classes B,C,D and E contain 16 videos with different resolution and content. The UVG dataset contains seven 1080p HD video sequences with 3900 frames in total.
233
234
235
Implementation details We optimize four models for MSE and four models for MS-SSIM [37.
236
The video clip length T is set to 7 for training. We use the Adam optimizer (38] with batch size of 8 and initial learning rate of 5 10-5, It is difficult to stably train the whole models from scratch. We first separately pre-train the intra-frame coding models and -nter-frame coding models for MSE, with
237
238
239
128 x 128 video crops and 1,200,000 training steps. Then we jointly optimize both the models with
the gop loss Eq. (6) for 100,000 steps using different metrics and X values. Finally, we fine-tuning all the models for 20, 000 with crop size of 256 256 and a reduced learning rate of 1 x 10-5
241
Evaluation Setting We measure the quality of reconstructed frames using PSNR and MS- SSIM (37 in the RGB colorspace. The bits per pixel (bpp) is used to measure the average number of bits. We compare our method with the traditional video coding standards H.265/HEVC and H.266/VVC, as well as the state-of-ar of-art learning based methods including [5] 22
243
245
17.
Recent works for learned video compression usually evaluate H.265 by using FFmpeg, with perfor- mance is much lower than official implementation. In this paper, we evaluate H.265 and H6 by using the implementation of the standard reference software HM 16.21 (9) and VTM 12.0[40) respec- tively. We use the default low delay and random access configuration and modify the gop structure and key frame interval for fair comparision. Detailed configuration can be found in supplementary.
247
249
4.2
Performance
252
We evaluate our model with the state-of-ther -he--ar learned video compression approaches, including the P- frame based methods of [15 22 23 17], the interpolation based methods of 12]. As shown
7

---

HEVC Class C dataset
HEVC Class C dataset
HEVC Class C dataset
34
Proport on of voxel ftows (%%
6x
34
33
PSUR(dR)
PSMR(dB)
32
M-25
RA wlo GFP
M-25
M--5
LD
M-4
30
M-1
M-1
RA (r-1)
0.2
0.3
0.4
0.5
0.3
0.4
0.5
0.2
0.3
0.4
0.5
Bpp
Total bitrate (8pp)
Bpp
(a)
(b)
(c)
Figure 5: (a) Ablation the number of voxel flows. (b) The Proportion of voxel flows in total bitrate.
(c) Ablation on different coding configurations.
254
it can be observed that our proposed method significantly outperforms exiting learned video compression methods in both PSNR and MS-SSIM. Note that the VLVC (randomaccess) and
in Fig.
255
256
VLVC (lowdelay) are two different configurations from the same models. Besides, our model the
257
first end-to-end learned video compression method that achieves comparable R-D performance with
258
H. 11.266 terms of MS-SSIM.
259
4.3
Ablation Study and Analysis
260
All the models reported in ablation studies are trained for MSE using 128 x 128 video clips. More ablation study results and visual results can be found the supplementary.
261
The effect of the voxel flow number As shown in Fig. 5a the number M of voxel flows signifi- cantly influence the overall rate-distortion performance. More voxel flows provide more possible sampling location for accurate motion compensation. Our proposed weighted trilinear warping with multiple voxel flows achieves about IdB gain compared with the conventional trilinear warping with single voxel flow. Note that the performance gain is nearly saturated for M 25, which is used as the default value in our models.
263
265
267
We also investigate the additional bitrate cost of multiple voxel flows. As shown in Fig. 5b the 268
proportion of multiple voxel flows in the total bitrate of video coding increases about 1 at the same bitrate. In other words, our model can learn to improve the overall compression performance by transmitting proper amount of additional motion information, which represented as voxel flows.
269
270
271
272
Versatile coding configurations The proposed methods can deal with various set of prediction modes. To evaluate the effectiveness of coding flexibility as well as the effectiveness of the proposed generalized flow prediction module, we simply change the input coding configurations of the same trained model different bitrate points. Random access and low delay coding settings are denoted RA and LD. respectively. As shown in Fig. 5cl the coding mode RA with bidirectional reference frames achieves a compression gain of about 0.4dB, compared with the unidirectional coding modes LD. Furthermore, the performance dropped about OdB.B0..3 when turn off the generalized flow prediction module for different coding settings, noted as w/o GFP. We also illustrate the bitrate reduction of the voxel flows shown in Fig. 5b where M=25 reduce the bitrate of voxel flows about compared to M=25 w/o GFP. Finally, we change the number of the reference frames for warping, which is set to 2 default. We reduce the number to 1 in random access mode, noted as RA (r=1), which performance is even worse than low delay setting.
273
275
277
279
281
283
Visualization of voxel flows The proposed voxel flows contain multiple 3-channel voxel flows IM (g By. 9: | and their weights ( We separately visualize the weighted temporal and spatial flow maps. The mean temporal flow map 9. Σ, gi. g¹ describes the weighted centroid of
285
287
voxel flows along the time axis. As shown the fourth column Fig. [aa the Da performs like occlusion map for bidirectional frame prediction. The pixels in black area cannot be found in the
8

---

Reference framel
Reference frame2
Current frame
Mean temporal flow
Bidirectionnl
Mean spatial flowl
Mean spatial flow2
Std spatial flow1
Std spatial flow2
(a)
Reference framel
Reference frame2
Current frame
Mean temporal flow
Unidiroctione
Mean spatial flowl
Mean spatial flow2
Std spatial flowl
Std spatial flow2
(b)
Figure 6: Visualization the voxel flows for the same target frame with different reference frames, generated by the same model. (a) Bidirectional reference frames. (b) Unidirectional reference frames
first reference frame because the basketball player in red covers the background. Hence the voxel flows pay more attention on the second frame, resulting in large weights. The white area can be explained in the similar way for the first frame, and the gray area means that the voxel flows pay equally attention for both frames. unidirectional frame prediction, the I. generated by the same model almost black everywhere, demonstrating the flexibility of trilinear warping for different prediction mode.
290
292
294
295
We also visualize the weighted mean and weighted standard deviation of spatial flow maps (noted as mean spatial flow and std spatial flow) investigate the spatial distribution of voxel flows. We first round and group the g to the nearst integer location of reference frames (e.g. 0 or 1), then separately calculate the mean flow map and std flow map for each group of voxel flows. shown in
296
297
298
the second and fourth raws of Fig. [ the grouped spatial mean of voxel flows has similar distribution with optical flow. Different from optical flow, the voxel flows have large variance in the area of motion, occlusion and blur, shown in the std spatial flow map. Single optical flow is not able to find accurate reference pixel and results inefficient motion compensation. Multiple flow warping with weighted coefficients provide choice to perform motion compensation using multiple reference pixels with better rate-distortion performance.
300
302
304
5
Conclusion
308
In this paper, we propose a versatile learned video coding (VLVC) framework that allows us train one model to support various inter prediction modes. To this end, we apply voxel flows as motion
307
308
information descriptor along both spatial and temporal dimensions, and we perform reconstruction
309
via proposed weighted trilinear warping using voxel flows for more effective motion compensation.
310
Through formulating different inter prediction modes by unified polynomial function, we design
novel flow prediction module to predict accurate motion trajectories. In this way, significantly
312
reduce the bits cost of encoding motion information. Thanks to above novel motion compensation and flow prediction, VLVC not only achieve the support of different inter prediction modes but also

---

yield competitive R-D performance compared to conventional VVC standard, which fosters practical 315 applications of learned video compression technologies.
314
References
316
317
[1] VNI Cisco. Cisco visual networking index: Forecast and trends, 2017-2022. White Paper, 1, 2018.
318
[2] Thomas Wiegand, Gary J Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/ave video coding standard. IEEE Transactions on circuits and systems for video technology. 13(7):560 576, 2003.
[3] Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. IEEE Transactions circuits and systems for video technology, 22((2): 1649-1668, 2012.
321
322
323
[4] Shan Liu fu- Wang Benjamin Bross, Jianle Chen. Versatile video coding (draft 9). Draft, JVET, 2020.
324
[5] Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. arXiv prepriat arXiv:1611.011 1.01004 2016.
325
326
[6] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang. and Nick Johnston. Variational image compression with scale hyperprior. arXiv preprint 102.v418 2018.
327
328
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. In Advances in Neural Information Processing Systems. pages 10771-10780. 2018.
329
330
331
[8] Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with dis- cretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7939-7948, 2020.
332
338
334
Firikur Agustsson and Lueas Theis. Universally quantized neural compression. Information Processing Systems, 33, 2020.
Advances in Neural
335
336
[10]
Zongyu Guo, Zhizheng Zhang, Runsen Feng, and Zhibo Chen. Soft then hard: Rethinking the quantization in neural image compression, 2021.
337
338
[11] Chao- Yuan Wu, Nayan Singhal, and Philipp Krahenbuhl. Video compression through image interpolation. In Proceedings of the Furopean Conference on Computer Vision (ECCV), pages 416-431, 2018
[12] Abdelaziz Djelouah, Joaquim Campos, Simone Schaub- Meyer, and Christopher Schroers. Neural inter- frame compression for video coding. In Proceedings of the IEEE International Conference on Computer Vision, pages 6421 1-229, 2019.
341
342
343
[13] Amirhossein Habibian, Ties van Rozendaal, Jakub M Tomezak, and Taco S Cohen. Video compression with rate-distortion autoencoders. In Proceedings of the IEEE International Conference on Computer Vision, pages 7033-7042, 2019.
344
[14] Jemry Liu, Shenlong Wang, Wei-Chiu Ma, Meet Shah, Rui Hu, Pranaab Dhawan, and Raquel Urtasun. Conditional entropy coding for efficient video compression, 2020.
348
[15] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dve: An end-to-end deep video compression framework. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 11006-1 1015, 2019.
349
350
351
[16] Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
353
354
[17] Jianping Lin, Dong Liu, Houqiang L.,, and Feng Wu. M-lve: Multiple frames prediction for learned video compression. arXiv preprint arXiv:2004. 10290, 2020.
355
356
[18] Oren Rippel, Alexander Anderson, Kedar Tatwawadi, Sanjay Nair, Craig Lytle, and Lubomir Bourdev. Elf-ve: Efficient learned flexible-rate vidco coding. arXiv preprint arXiv:2 140.353544,,,,,,,, 2021.
357
358
[19] Ze Cui, Jing Wang, Bo Bai, Tiansheng Guo, and Yihui Feng. G-vae: A continuously variable rate deep image compression framework. arXiv preprint arXiv:2003.02012, 2020.
10

---

380
[20] Ren Yang, Fabian Mentzer, Luc Van Gool, and Radu Timofte. Learning for video compression with hierarchical quality and recurrent enhancement. arXiv preprint arXivv2003.0196. 2020.
382
[21] Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander G Anderson. and Lubomir Bourdev. Learned video compression. In Proceedings of the IEEE International Conference Computer Vision. pages 3454-3463, 2019.
364
[22] Guo Lu, Chunlei Cai, Xiaoyun Zhang, Li Chen, Wanli Ouyang, Dong Xu, and Zhiyong Gao. Content adaptive and error propagation aware deep video compression. arXiv preprint arXiv:2003. 1282, 2020.
366
367
[23] Haojie Liu, Lichao Huang, Ming Lu, Tong Chen, Zhan Ma, et al. Learned video compression via joint spatial-temporal correlation exploration. arXiv preprint arXiv: 1912.06348, 2019.
389
[24] Zhibo Chen, Tianyu He, Xin Jin, and Feng Wu. Learning for video compression. IEEE Transactions on Circuits and Systems for Video Technology. 30(2):666-576 2020.
371
[25]
Ziwei Liu, Raymond A Ych, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. In Proceedings of the IEEE International Conference on Computer Vision, pages 4463 4471, 2017.
373
[26] Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive convolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 670-679, 2017.
375
376
[27] Fitsum A Reda, Guilin Liu, Kevin J Shih, Robert Kirby, Jon Barker, David Tarjan, Andrew Tao, and Bryan Catanzaro. Sde-net: Video prediction using ppatially-displaae convolution. In Proceedings of the European Conference on Computer Vision (ECCV), pages 718-733, 2018.
378
[28]
Wenbo Bao. Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, and Ming Hsuan Yang. netc Motion estimation and motion compensation driven neural network for video interpolation and enhancement. JEEE transactions on pattern analysis and machine intelligence, 2019.
380
381
382
[29] Hycongmin Lee, Tacoh Kim, Tac-young Chung, Dachyun Pak, Yuscok Ban, and Sangyoun Lec. Adacof: adaptive collaboration of flows for video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5316-5325, 2020.
383
384
385
[30] Zhihao Shi, Xiaohong Liu, Kangdi Shi, Linhui Dai. and Jun Chen. Video frame interpolation via generalized deformable convolution. IEEE Transactions on Multimedia, 2021.
386
387
[31] Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5437-5446, 2020.
388
389
[32] Xiangyu Xu, Siyao, Wenxiu Sun, Qian Yin, and Ming- Husun Yang. Quadratic video interpolation. arXiv preprint 19Xiv:1911.00627 2019.
390
[33] Deqing Sun, Xiaodong Yang, Ming-Yu Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8934-8943, 2018.
382
393
394
[34] Runsen Feng. Yaojun Wu, Zongyu Guo, Zhizheng Zhang, and Zhibo Chen. Learned video compression with feature-level residuals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition Workshops, pages 120-1211 2020.
395
396
397
[35] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with mas--orienedd flow. International Journa of Computer Vision, 127(8): 1708))1100610 100-111,,, 2019.
398
399
[36] Alexandre Mereat, Marko Viitanen, and Jarno Vanne. Uvg dataset: 50/120fps 4k sequences for vidco codec analysis and development. In Proceedings of the 1thh ACM Multimedia Systems Conference, pages 297-302 2020.
400
402
[37] Zhou Wang, Fero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assess- ment. In The pages 1998-1422. Ieee. 2003.
403
Asilomar Conference on Signals, Systems & Computers, 2003, volume 2.
405
[38] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv: 142,664 2014.
arv preprint
407
[39] HEVC Offical Test Model HM. htttps://hee.... hhi fraunhofer .de
408
[40]
VVC Offical Test Model VTM. https: /tttp:///jvt.. hhi fraunhofer. .de
11

---

Checklist
409
410
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. . You are strongly encouraged to include justification to your answer. either by referencing the appropriate section of your paper or providing a brief inline description. For example:
412
413
414
Did you include the license to the code and datasets? [Yes] See Section ??.
415
Did you include the license the code and datasets? [No] The code and the data are proprietary.
416
417
Did you include the license to the code and datasets? [N/]]
418
Please do not modify the questions and only use the provided macros for your answers. Note that the
419
Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
420
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
423
424
(b) Did you describe the limitations of your work? [No]
425
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
427
428
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
430
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
432
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? We describe datasets, network architecture and more details in the supplementary.
433
435
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
436
437
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
438
439
(d) Did you include the total amount of compute and the type of resources used (c.g., type
of GPUs, internal cluster, cloud provider)? [No] In the supplementary.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
442
(a) If your work uses existing assets, did you cite the creators? [Yes)
443
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material URL? [No]
445
(d) Did you discuss whether and how consent was obtained from people whose data you' rre're
using/curating? [No]
446
447
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
448
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
451
452
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
453
454
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
455
12


Reviews:
[{"review_id": "ynWDRgtLYap", "reviewer": "Reviewer_hMDd", "summary": "The authors introduce VLVC, a new neural video compression codec. Thanks to a new 3D motion compensation structure based on spatio-temporal interpolation, a single trained model can be run in arbitrary prediction modes (such as low-delay or random-access settings). In addition, each frame can be compressed with an arbitrary number of reference frames. There are a number of smaller changes to established neural video codecs. The resulting VLVC codec is demonstrated on standard datasets, showing good rate-distortion results (in particular when trained and evaluated on MS-SSIM).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Neural video compression is without question a problem of large practical importance. It seems reasonable that a NN codec that can operate both in low-delay and random-access modes is beneficial, though I find that the authors overstate this case. Blanket statements like \"existing learned video compression schemes [...] are inapplicable for various scenarios\" really need to be supported by more arguments or examples.\n\nVLVC brings novelty to the neural video compression landscape. The independence of a model from the prediction mode is made possible by a new warping algorithm based on ultiple 3D voxel flows and spatio-temporal polynomial interpolation. However, I believe this is not the first work in this direction: T. Ladune et al [1] also propose a framework in which a single model can be used to compress frames in different ways. A comparison of these methods would enhance the paper.\n\nThe proposed method is overall sound, as far as I can tell. In addition to the flexibility with respect to coding scheme / reference structure, the structure with multiple flows allows the model to deal better with occlusions and to model motion uncertainty to some extent, both of which may be relevant in practice.\n\nSome aspects I found confusing; in particular, I am not sure if I understood the generalized optical flow prediction (section 3.2) correctly. Perhaps more importantly, especially for a paper that focuses so much on the independence between reference structures: which reference configurations are actually used? In line 216 the authors describe it as \"the model will have access to various reference structures\" without going into detail. The labels \"randomaccess\" (RA) and \"lowdelay\" (LD) also do not fully specify this. If there was a definition of these somewhere in the paper, I missed it and would very much appreciate a pointer. Are the reference structures described for HM and VTM in the supplementary material used for VLVC as well? How are I-frames / key frames handled in this approach? \n\nI find that several aspects of this approach deserve more discussion. For instance:\n1. When should one use which coding configuration (reference structure)? Can the model be extended to make such decisions dynamically based on video content?\n2. What exactly are the benefits of having a single model that supports arbitrary coding configurations? Can the authors for instance argue or demonstrate that with the same overall computational budget, a common model achieves a better performance than separate LD and RA models?\n3. How fast is the proposed video codec during training, encoding, and decoding relative to other neural video codecs? How efficient is the new 3D voxel warping operation? The supplementary material mentions encoding and decoding around 0.7 fps, can the authors further comment on how this could be improved?\n4. It would be interesting to have a detailed comparison of the proposed 3D warping and the scale-space flow (SSF) warping proposed by E. Agustsson et al [2]. Both methods use 3D flows, although the third dimension in SSF refers to a blur direction rather than time. Both have the interesting feature that they improve the modelling of uncertainty in the optical flow, for instance in the presence of movement or occlusion (see Fig. 6). Can the authors comment more on the similarity and differences between these approaches?\n5. How important is the off-the-shelf PWC-Net in this approach? How does the use of this pretrained optical flow estimator fit together with Sec. 3, where the authors argue that end-to-end training with implicit flow networks  performs better than pretrained optical flow networks and simplifies the network architecture?  \n\nThe empirical results show a strong rate-distortion performance on standard datasets, especially in the low-bitrate region. Comparing to HM and VTM is laudable, though I do not see why the authors changed the default RA reference structure in these classical codecs. The authors also compare to several neural baselines. Some strong baselines like SSF [2], ELF-VC [3], and B-EPIC [4] are missing (except for SSF these all came out very recently though).\n\nWhile the results are strong, they do not quite justify all of the claims that the authors make. VLVC does not \"yield competitive R-D performance compared to [...] VVC\" (last sentence of the conclusions) when evaluated on PSNR, i.e. the metric that classical codecs are largely optimized for. I also do not see that the \"proposed method significantly outperforms existing learned video compression methods\" when the strongest neural baselines are missing from the comparison. (I tried to compare VLVC to SSF, ELF-VC, and B-EPIC manually and it does seem to me that VLVC outperforms them slightly at low bitrates! But the authors need to show this to make SOTA claims). I wonder why the authors hide the meaningful classes B and E of the HEVC test datasets in the supplementary material, while showing results for classes C and D with their extremely small resolutions (HEVC class D has a resolution of 240x416...). Finally, it would be great to provide more BD-rate savings (for all considered datasets and also for the MSE models).\n\nThe authors present a good set of ablations and extensive supplementary materials. The main ablation I am missing is a comparison to the same method when it is trained on only one prediction mode.\n\nThe presentation of the paper could be improved. I believe the explanations could be clearer, important aspects could be highlighted better, and (as discussed above) several aspects seem to be missing. There are also several typos.\n\nAll in all, the authors present an interesting neural video compression method with several new features and a strong empirical performance. Unfortunately, this paper does not do the method justice. Both the description of the method, the discussion of its properties and use cases, and the empirical evaluation can be improved in several ways. A revised version that answers my question and addresses these points might make a good paper, but in its current form I do not think it is ready to be accepted.\n\nReferences:\n- [1] T. Ladune et al, Conditional coding for flexible learned video compression, ICLR 2021 workshop neural compression\n- [2] E. Agustsson et al, Scale-space flow for end-to-end optimized video compression, CVPR 2020\n- [3] O. Rippel et al, ELF-VC: Efficient Learned Flexible-Rate Video Coding, arXiv:2104.14335\n- [4] R. Pourreza and T. Cohen, Extending Neural P-frame Codecs for B-frame Coding, arXiv:2104.00531\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss the societal impact that neural video codecs could have. The paper could be improved by discussing risks for instance from biases in the training data, but also due to algorithmic choices.", "review_text": "", "extra_scores": {"ethical_concerns": "I see no ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Wm6kAgZbGW", "reviewer": "Reviewer_GPmx", "summary": "This paper introduces multi-frame references for learned video coding, which estimates multiple optical flows as voxel flows for motion compensation. With the weighted map, it can achieve weighted trilinear warping to combine the multiple predictions into  the final prediction. It is a good idea to obtain a better predcition to search information across multiple frames.  The author introduces  the polynominal motion modeling to use multiple backward optical flow to generate the single optical flow. Based on the method of softmax spaltting, they convert the backward flow into a forward flow using the flow reversal layer. Then use the forward flow and softmax spaltting for motion compensation. The overall architecture is a hybrid video coding system and the main contribution is inter coding. Residual coding is applied on the feature domain. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) Voxel flows. The concept of voxel flow is proposed in the paper [1] for weighted prediction which can utilize the different information from different frames partially for a final prediction. The author uses this idea for the multi-frame reference for video coding with a rate-distortion optimization. Although decoding multiple flows sometimes only add some consumption of bit rates, can the author give some data about the comparison between single flow and voxel flow. Meanwhile, the author should include the MLVC [2] for comparison because this paper also uses multiple frame as references.\n2) Forward flow with softmax spaltting. We think the author is aimed to use the forward flow with softmax spaltting for compensation to solve the problem of the worse predcition around the motion edges and occlusions in some extent. However, voxel flows can also solve this problem with the long-term prediction with the weighted maps. Can the author give some results when only using the generated backward flow for compensation?\n3) We think the main contributions are voxel flows and the different wapring method based the proposed polynominal motion modeling. They are mainly from the existing published papers and just apply these methods to the learned video coding from the frame interpolation and prediction. The overall framework and concept are still follow the hybrid video coding system.\n4) Experiments only show the results on UVG and ClassC and ClassD. How about the results on ClassE and ClassB and MCL-JCV?\n5) More learned video coding methods should be added for comparison such as DVC_Pro [3], [4],[5], [6].\n6) We also find that the bit rate on ClassC and ClassD are larger than 0.1 bpp. As a video coder, performance between 0.01 ~ 0.1 bpp is significant. Please provide the results between the bit rate range and include the results of the listed learned video coder.\n\n\n\n\n[1]. Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis 371\nusing deep voxel flow. In Proceedings of the IEEE International Conference on Computer Vision , pages 372\n4463–4471, 2017.\n[2]. Jianping Lin, Dong Liu, Houqiang Li, Feng Wu. M-LVC: Multiple Frames Prediction for Learned Video Compression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.\n[3]. Guo Lu; Xiaoyun Zhang; Wanli Ouyang; Li Chen; Zhiyong Gao; Dong Xu. An End-to-End Learning Framework for Video Compression. IEEE Transactions on Pattern Analysis and Machine Intelligence 2020. \n[4]. Haojie Liu, Ming Lu, Zhan Ma, Fan Wang, Zhihuang Xie, Xun Cao, Yao Wang. Neural video coding using multiscale motion compensation and spatiotemporal context model. IEEE Transactions on Circuits and Systems for Video Technology 2021.\n[5]. Ren Yang, Fabian Mentzer, Luc Van Gool, Radu Timofte. Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model. JSTSP 2021.\n[6]. Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, George Toderici;Scale-Space Flow for End-to-End Optimized Video Compression.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 8503-8512", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the comments below.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "60"}, "extra_text": {}}, {"review_id": "Q32ZUXXCIEL", "reviewer": "Reviewer_Eb8Q", "summary": "First, the paper introduces the idea of using multiple predictions (e.g. M=25) for learning-based video coding. Second, to reduce the required motion bits, a flow prediction method using polynomial approximation is developed. The idea is reasonable, and the reported performance seems promising. However, the paper is written rather poorly, and how the algorithm is actually designed is not clear. Thus, the reported performance is not convincing either. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "***\nThis paper is slightly longer than the page limit (9 pages). Anyway, it violates the rule and can be rejected. I leave the decision to AC. \n\n***\n\nExcept for the survey of previous work, the writing and description are poor. There are many typos and grammatical errors. For example,\n1. gop -> GOP\n2. More attention should be paid to articles (a, the) and tenses.\n3. \"Considering a practical scene where multiple objects of the same types appear at the same time, more than one areas could \n referred for reconstructing the local region.\" What does \"the local region\" specify?\n4. More possible sampling location\n5. \"$D$ is the number of reference frames used for motion compensation.\" Motion estimation? You use only 2 frames for motion compensation. Is that right?\n6. In this process, the predicted optical flow reduces the spatial displacement need to be encoded\n7. there are two optical flow describe the motion between the reference frame and  the target frame \n8. the flow describe\n9. pixels is unavailable. \n\nIn fact, there are too many errors to list them all, especially near the end of the paper. \n\n\n\n \n\nThere are many technical details, which cannot be clearly understood even after reading the supplemental document. \n1. Please describe the prediction structure for random access at $r=2$ when the GOP size is 7. \n2. When there are only two reference frames ($r=2$), what is the role of $g_z$ ? Why do you need it anyway? You are using multiple voxel flows. So half of them can be constrained from reference 1, and the other from reference 2. \n3. Figure 6 is hard to understand. Please provide the color bars. \n4. How do you encode intra frames? Using the same residual encoder? Are they considered in the rate-distortion curves?\n5. How do you set $k$ in Eq. (2) and (3)? Is a higher-order polynomial (beyond quadratic) really necessary? Also, here, the variable is $\\Delta t = t_j - t_i$. Is it ok to ignore the distance from the current frame $t$?\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "FQXOHq3g4W_", "reviewer": "Reviewer_htG5", "summary": "Learned video compression is a promising field and has demonstrated that it can be as effective as the classic video compression method. In terms of rate distortion it has caught up with the classical methods greatly. However binding the prediction mode and fixed network is what hinders the field to go forward. This paper proposes a versatile learned video compression framework that learns one mode to predict all possible modes. The paper proposes versatile compression where the motion compensation applies 3D motion vector fields, trilinear warping. Motion estimation here acts as a decoupler of prediction modes away from framework design, allowing its dependence to be not affected by framework design. For multiple reference frame prediction, the network predicts the motion fields with a unified polynomial function. The flow prediction can lead to reduction of computation by reducing voxel flow transmission. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of this paper is that it explains the motivation behind the proposed method in an understandable way and the articulation is sufficient in that it is easy to understand. The introduction well describes the intuition and motivation behind designing a versatile motion compensation module that can handle multiple prediction mode scenarios. It also explains about the polynomial motion trajectory. \nOverall, the paper is well written and easy to follow with good figures explaining the concept. The experimental results also show that the proposed method performs well against other SOTA methods. \nA concern is that this paper does not show any analysis about computational complexity in the main paper. The greatest reason why learning approaches are not being used widely compared to classical methods is due to computation complexity. I would suspect that incorporating a versatile 3D voxel flow module to handle all prediction mode cases will give a great impact on computational speed. Also, solving for the polynomial coefficients for the motion trajectories does not sound computationally efficient. The analysis on computational efficiency was lacking in my opinion. There needs to be more analysis on this aspect. The computational impact will be different for each prediction mode for the classical method, however, the computational load will more or less be similar for the proposed methods since it uses a unified framework for predicting the trajectories for all modes. Also more analysis is needed on the computational load for polynomial coefficient prediction. \nIt would be wise to compare the computational load with the classical methods as well for different prediction modes.\nGiven that the model size is 103MB, about 3 times as large as PWC-Net, the model seems sizable and may most likely translate to high computational complexity. Nonetheless, I believe a comparison with other Learning based methods at least is needed.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "For video compression I cannot imagine a suitable case for ethical impact. However, any piece of technology can contribute to wrong use and ethical implications. This work did not address such concern to my knowledge. \n", "review_text": "", "extra_scores": {"ethical_concerns": "See above\n", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "DrmH_x0-wn", "reviewer": "Reviewer_BSR1", "summary": "This paper proposes a neural video compression method that is designed around the idea of using a voxel flow instead of the more traditional pixel warping idea. The key difference here is that a volume of reference frames is used, and flow is computed to all of them. This allows the compression method to be quite liberal in terms of what information from the GOP can be used, yielding a better RD performance than methods that rely on a single reference frame.\n\nOverall I think this is a step in the right direction, but I fear that such a method is completely impractical for the foreseeable future.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This method makes use of a frame volume that can be equal to the number of frames in the GOP. To give some perspective, normal video codecs tend to be very memory sensitive, and in general it's not possible to have more than a small set of reference frames due to the huge memory pressure that this imposes, and of course cost (more frames = more RAM needed). This problem is exacerbated by the fact that we see a trend towards higher resolutions which grow in size quadratically, making the cost of multiple reference frames completely prohibitive. This is my main worry about the applicability of this paper... I am not sure how realistic the assumption is that a copy of N reference frames can be kept in RAM at decoding time....where N >> 2. Similarly the number of flows is also on the order N, making my worry even worse.\n\nIgnoring the practical aspect, the paper presents an interesting approach to improve compression. I haven't seen something done exactly this way in any other area so I think that we have an increased level of originality here. The main differentiator is the multiple flow volume idea (multiple reference frames have been done many times).\n\nI applaud the authors for not fearing to compare against HM, because that's the best possible case for HEVC. I am not familiar enough with the settings to comment whether the modifications made to the profile used make sense, and I would invite a reviewer with more familiarity to comment on this.\n\nI also think it's good that the details of the network are presented in the supplementary materials.\n\nMy second biggest concern, though is related to runtime. This method sounds very very expensive and at least in the body of the main paper this issue is not really addressed. I think in the final version THIS MUST be explicitly addressed.\n\nOverall, I am inclined to accept this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are really not discussed at all. This method has some glaring problems I mentioned in the review and they're not at all acknowledged.", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work proposes and implements a learning-based video compression solution, which is able to handle multiple prediction modes and achieves good rate-distortion performance. The key building blocks are not new, but the actual model and its successful implementation bring an interesting contribution to the video compression problem. 
The submitted manuscript can be completed by the multiple additional results discussed with the reviewers, and a complexity analysis; the presentation should finally be improved before an eventual publication. In particular, the work should be more clearly positioned wrt to recent competitors, including 'T. Ladune et al, Conditional coding for flexible learned video compression, ICLR 2021 workshop neural compression'. 
While the work is really interesting, the revisions that result from the active discussion with the reviewers will lead to a quite different, improved, manuscript, which probably deserves another review before a positive outcome. 
Decision: Reject