Uncertainty-Aware Deep Video Compression with Ensembles
Authors: Wufei Ma
Jiahao Li
Bin Li
Yan Lu
Venue: ICLR 2022 Submitted (2022)
Keywords: Video compression
uncertainty
ensemble learning

Abstract: Deep learning-based video compression is a challenging task and many previous state-of-the-art learning-based video codecs use optical flows to exploit the temporal correlation between successive frames and then compress the residual error. Although these two-stage models are end-to-end optimized, errors in the intermediate errors are propagated to later stages and would harm the overall performance. In this work, we investigate the inherent uncertainty in these intermediate predictions and present an ensemble-based video compression model to capture the predictive uncertainty. We also propose an ensemble-aware loss to encourage the diversity between ensemble members and investigate the benefit of incorporating adversarial training in the video compression task. Experimental results on 1080p sequences show that our model can effectively save bits by more than 20% compared to DVC Pro.

Under review as a conference paper at ICLR 2022
UNCERTAINTY-AWAR DEEP VIDEO COMPRESSION WITH ENSEMBLES
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning-based video compression is challenging task and many previous state-of-the-art learning-based video codecs use optical flows to exploit the tem- poral correlation between successive frames and then compress the residual error Although these two-stage models are end-to-end optimized, errors in the inter- mediate errors are propagated to later stages and would harm the overall perfor- mance. In this work, we investigate the inherent uncertainty in these intermediate predictions and present an ensemble-based video compression model to capture the predictive uncertainty. We also propose an ensemble-aware loss to encourage the diversity between ensemble members and investigate the benefit of incorpo- rating adversarial training in the video compression task. Experimental results on 1080p sequences show that our model can effectively save bits by more than 20% compared to DVC Pro (Lu et al. 2020b
INTRODUCTION
Video contents are reported to account for 82% percent of all consumer Internet traffic by 2021 and are growing rapidly with an increasing demand for high-resolution videos (c.g., 4K movies) and live streaming services (Cisco) 2020). It is therefore critical for us to improve the video compression performance to transmit video with higher quality given limited Internet bandwidth. In recent years, there has been a surge of deep learning-basee
video compression models (Rippel et al.] [019 Lu al. 2020b; Agustsson al. 2020)) and some of them have achieved comparable or even better performance than previous traditional video codecs, such as x.264 (Tomar 2006).
Despite previous deep learning-based video codecs have achieved improved performance on many challenging datasets, most of the ofate-of-the-art models estimate deterministic predictions for in- termediate representations, such as optical flows and residuals. These models fail to represent the aleatoric uncertainty inherent in the model inputs or the epistemic uncertainty in the model pa- rameters and would blindly assume the predictions to be accurate, which is not always the case (Der Kiureghian & Ditlevsen 2009: Kendall & Gal 2017). In terms of video compression, such models produce deterministic motion vectors (or optical flows) and residuals for each pixel location, ignoring the fact that optical flows may not be estimated accurately in occluded regions and around object boundaries, and the quantization operation for lossless entropy coding also introduces addi- tional noises the inputs of the decoders. Underlying errors in such overconfident intermediate predictions are propagated to later stages of the P-frame model and even to subsequent frames for models built on temporal correlation, leading to suboptimal performance of the compression system.
Predictive uncertainty is crucial for us to understand how certain the model is about the predictions, especially for -fiittiiitiiitno data. However, most neural networks do not offer such information and tend to produce overconfident predictions (Gal 2016,] Lakshminarayanan et al. 2017). Bayesian neural networks (Ma 1992. Hinton Neal 1995) are widely used to quantify predictive un- certainty. but lack practicality duc greatly increased computation complexity and do not scale well to high dimensional data. Gal Ghahramani (2016) proposed Monte Carlo dropout that performs test-time dropout. It is simple to implement but is not suitable for deep learning- based compression since it requires multiple decoding-ting inferences and yields non-deterministic outputs.
In terms of deep learning- based video compression, two -on-BByesian approaches considered to represent the predictive uncertainty: (1) modeling the uncertainty explicitly by regressing the empir- ical uncertainty of the model outputs (Nix & Weigend 1994); and (2) using ensembles for predictive

---

Under review as a conference paper at ICLR 2022
uncertainty estimation @Lakshminarayanan et al. 2017). Agustsson et al. (2020) took the first ap- proach and proposed seale-space flow for motion compensation. Specifically, y, their model predicts a scale field besides the standard 2-dimension flow field, representing the uncertainty associated with each MV. Gaussian blurring is then applied to the reference frame and the scale parameter used to control the scale of the Gaussian kernel. Experimental results show that the self-supervised scale parameter increases around the boundaries of the objects and in regions where the motion is large, which confirms that the scale field learns to estimate the uncertainty of the optical flow prediction.
In this work, we consider the second approach and represent the underlying uncertainty with an ensemble of outputs. Ensembles of models have been shown to improve predictive performance (Dietterich] 2000). and ensembles of neural networks have also been used to boost model perfor- mance on many challenging benchmark datasets (Szegedy et al.) 2015). Instead of producing a deterministic prediction, ensemble methods perform model combination and reflect the uncertainty of out-of out-of-distribution data. Here we propose an ensemble-based multi-head decoding module that can generate ensemble of intermediate outputs, such as motion vectors and residuals, and im- plicitly represent the predictive uncertainty with the variance of the Gaussian mixture prediction. We no longer assume that any individual intermediate prediction is accurate and explicitly consider multiple candidates of them. Such uncertainty is then propagated to later stages and all modules in our framework are optimized in end-to-end fashion.
To further improve the performance of our ensemble- bbsed video compression model, we propose an ensemble-aware loss to encourage diversity between different branches and incorporate ad- versarial training strategy, fast gradient sign method (FGSM) (Goodfellow et al., 2014)) to smooth the intermediate latent representations. Our experiments show that our model achieves an average bitrate saving of 15% when the distortion is measured in PSNR.
The contributions of this work can be summarized as follows:
We are the first to leverage deep ensembles in a video compression model and demonstrate that our approach can be used to capture the underlying uncertainty of intermediate repre- sentations and prevent the model from being overconfident with intermediate predictions.
We demonstrate that our approach can effectively improve the performance of deep- learning based video compression and can be widely applied to optical flow-based video codecs with negligible complexity increase.
Our experiments show that our model outperforms previous state-of-the-ar models such as DVC Pro (Lu et al., 2020b) and Lu et al. (2020a), and our ablation study proves the effectiveness of each module.
2 RELATED WORK
Learned video compression. Previous learning-based video compression methods can catego- rized into two groups: (i) one-stage models, such methods based on 3D autoencoders (Pessoal
ct al. 2020: Habibian et al.) 2019); and (ii) two-stage models, which is adopted by most previous ofate-of-the-art methods, consist of predicted frame generation and residual coding. [Lu et al. (2019) proposed an end-to-end trainable video codec, DVC, that utilizes an optical-flow network (Ranjan & Black 2017) for motion compensation and then compresses the residuals. An extension of DVC, known as DVC Pro (Lu et al. (2020b), improves the compression performance by introducing refine- ment modules and auto-regressive entropy models. Feng al. (220)) considered residual coding in the feature domain and choose between pixel-level residuals and feature- residuals with RDO encoding time. Agusssson et al. (2020) proposed scale-space flow blur the intermediate recon- structions when motion vectors are not estimated well. [Chen et al. (202T) proposed to represent videos with neural networks and achieved promising results for video compression.
Model uncertainty. Predictive uncertainty can be grouped into aleatoric uncertainty and epistemic uncertainty (Der Kiureghian & Ditlevsen 2009). Aleatoric uncertainty captures the noises inherent in the observations and cannot be explained away with more data, while epistemic uncertainty ac- counts for uncertainty in the model structure or parameters and can be reduced with more training data. Bayesian neural networks (MacKay) 1992 Hinton & Neal 1995) is widely used approach for modeling predictive uncertainty that extends the traditional neural networks by learning a pos- terior distribution of model parameters from the observed data. Various non-Bayesian approaches
2

---

Under review as a conference paper at ICLR 2022
Frame
Encoder
Decoder
Reconstrction
Mation Comperiation
Biliner Warping
Add
MV
Relerence Froms
Malien Comperaation Prediction
MV Decoder Head
Decoder Head
Mation
CFCI MV Bisstraam
Docoded Frome Buffer
Residal Bistream
(a) DVC Lu et al.
2020b
(b) Multi-head decoder
Figure I: (a) A low latency optical flow-based video compression, DVC (Lu et al. 2020b). (b) Our proposed multi-head decoder structure (Section 3.2) 'MV' stands for motion vectors, and 'Res' stands for residuals.
have also been proposed, such utilizing the probabilities of softmax distributions (Hendrycks & Gimpel, 2016) and Specialists+1 I Ensemble for representing the predictive uncertainty for adversar- ial samples (Abbasi & Gagné 2017). Gal & Ghahramani (2016) proposed Monte Carlo dropout by performing multiple inferences with Dropout at test time. Lakshminarayanan et al.) (2017) proposed use an ensemble of neural networks for quantifying predictive uncertainty.
Deep Ensembles. The neural networks community has been investigating ensembles of deep net- works since the early 1990s (Hansen & Salamon 1990 Wolpert 1992: Perrone & Cooper 1993).
Krogh & Vedelsby (1955) proved the bias-variang trade-off for ensemble models which suggested the importance of the diversity among ensemble members. [ee et al. (2015) investigated several training strategies to train an ensemble and proposed ensemble-aware oracle loss to encourage di- versity. GoogLeNet (Szegedy al., (015), one the best-performing models on ILSVRC 2014, is an ensembles of CNNs. Lakshminarayanal et al. (2017) proposed to estimate predictive uncertainty by training multiple stand-alone neural networks. |Garipov et al. (2018); Fort et al. (2020) showed that deep ensembles can learn different modes of function with ensemble members that only differ in initialization weights.
UNCERTAINTY-WWAR DEEP VIDEO COMPRESSION
This section presents our main contributions. We introduce the theoretical background and the motivation of our proposed approach in Section 3.1 Then we introduce the ensemble-based multi- head structure decode multiple candidates of motion vectors and residuals in Section order to encourage the diversity among the ensemble members and to improve the overall performance, we propose an ensemble-aware loss for multi-head decoders in Section 3.3 Finally, we introduce an adversarial training strategy that find beneficial for the learning-bast based video compression task in Section 3.4
3.1 UNCERTAINTIES IN DEEP VIDEO COMPRESSION
We consider the predictive coding-based based framework. DVC, proposed by Lu al. (2020b) (Fig. Let the current frame be x; and the reconstructed previous frame from the buffer be -;-1- We estimate a motion vector (MV) map f, with a motion estimation network. The optical flow is then sent to motion auto-encoder for transform coding, yielding quantized bits à; and the reconstructed optical flow f.- Bilinear warping is used for motion compensation (MC) and MC prediction ă; with residual r; = I; 1 -; is obtained. The residual r; is then compressed with a residual encoder and decoder, outputting quantized residual bit stream and the decoded residual Å. The reconstructed current frame is the sum of the MC prediction and the decoded residual written
c
Bilincar Warp( fi) fr.
3

---

Under review as a conference paper at ICLR 2022
Current frame 2i
(h) Estimated MV
Decoded MV
(d) Aleatoric uncertainty
(e) Epistemic uncertainty
(I) Predictive uncertainty
Figure 2: A preliminary experiment on the underlying uncertainty of the optical flows. (a) The cur- rent frame 2; to be compressed. (b) The estimated MV ft- (c) The decoded MV ft. (d) Alcatoric uncertainty measured as the 1.2 distance between two optical flows with and without small per- turbation the bit stream. (e) Epistemic uncertainty measured by motion vectors that cannot be estimated well. (f) The predictive uncertainty represented by the multi-head decoder. Visualization of predictive uncertainty other sequences can be found in Appendix A.6
Aleatoric uncertainty. Although at encoding time we have full information necessary to decode Fi for lossy compression at certain bit rates, we quantize the bit stream that is passed to the decoder, and inevitably introduce aleatoric uncertainty at decoding time. Since the aleatoric uncertainty cannot be reduced with more training data, well-trained codec cannot reduce the quantization noise or fully recover the estimated MV fs.
Consider the MV auto-encoder in the DVC framework above. The lossy compression of the motion vectors can be summarized as
= MVEncoder((i)
((a:) a; +
f. MVDecoder((.))
(2)
If the MV decoder is implemented with a linear model parameterized by w, the impact of the quan- tization noise 77 on the decoded MV given by
w à w (a; + TT,
Since 7) is introduced by the quantization operation, have ||7|l0o S 1/2 s and it follows that the upper bound of the effects from the quantization operation is given by
"ww'
Tou as
w T nll S
e
w sign
lw sign(w)ll
(4)
Da;
While in practice the MV decoder is usually implemented with stack of convolution layers and nonlinear activation layers, such as leaky ReLUs, the transformation the MV decoder may be too linear to reject the quantization noise (Goodfellow et al. 2014).
We visualize the aleatoric uncertainty by adding small perturbation || Rs 0.2 || oo to the bit stream (see Appendix A.2) and the impact on the decoded MVs is depicted in Fig. [d] It shows that
there is more aleatoric uncertainty in regions where the motion complicated and the impact due to the aleatoric uncertainty is nontrivial.
Epistemic uncertainty. Due to limited observed data during training, epistemic uncertainty ac- counts for the uncertainty in the estimated motion vectors. Motion vectors near the object boundaries and occluded regions tend not to be estimated well, and warping erroneous motion vectors would propagate errors to the residual coding. We may roughly visualize such uncertainty by optimizing
4

---

Under review as a conference paper at ICLR 2022
motion estimation network with regards the mean squared enrors (MSE) between the current frame 23 and the warped frame
(T, BilinearWarp(2111.1
(5)
We depict the results in Fig. 2e Since the motion estimation is optimized to minimize the MSE, re- gions where the MSE is large are likely to have a larger epistemic uncertainty and the corresponding motion vectors cannot be estimated well given the limited training data.
Ideally, we could save MV bits by not encoding MVs that are not estimated well and save residual bits by not warping MVs that would not help to reduce residuals. This often difficult to implement as end-to-end trained deep neural network. In the next section, we will show that with the help of multi-head decoders, the model could learn to exploit available information in the bit stream and handle those predictions with larger uncertainty.
3.2 MULTI-HEAD DECODER
Our proposed multi-head decoder structure decodes multiple groups of motion vectors (MV) for motion compensation and multiple groups of residuals for final reconstruction. The multi-head MV decoder and multi-head head residual decoder are depicted in Fig. [b Take the multi-head MV decoder as an example. The MV decoder base first decodes MV feature from the quantized MV bitstream ā,. Then he groups of MVs, denoted by { h = 1...., h..) are decoded from the MV feature with respective MV decoder heads. We obtain Amo warped frames { k = 1...., Ame) by bilinearly warping each f. on the reference frame - The Amo warped frames are then concatenated for motion compensation and retained for final reconstruction.
Many previous ensemble- based models train an ensemble of stand-alone neural networks (Szegedy) al. 2015 Abbasi & Gagné 2017: Wang et al. 2020). While they can outperform the single-model baseline by wide margin, the number of parameters are greatly increased, well the inference complexity. Lee et al. (2015) proposed to share backhone parameters with TreeNets, but the models achieve the best performance when very few layers are shared. In our multi-head decoder structure, each decoder branch shares most of the convolution layers, making each decoder head light weight. This design effectively improves the overall performance with negligible complexity increase (see Section 4.4)..
The ensemble of decoded MVs can be represented by equally weighted Gaussian mixture model given by
f.
hmw Σ N(F I D. Σ
k
0
(6)
of F.y.
where off and of are the variance in æ and y directions respectively. The mean and variance of the Gaussian mixture model are respectively (see Appendix A.3
Amw
1 Ame
Bifi
Hj.
2
E
2
(otta) 2+2+ (
hmm
Empirically, we can visualize the predictive uncertainty represented by this ensemble model with the variance of the Gaussian mixture model by setting (o
(o )² 1, which gives
2
+1
homo
The predictive uncertainty for the first two frames in the BasketballDrill sequence depicted in Fig. 2e We that the predictive uncertainty properly capture both the aleatorie uncertainty and the epistemic uncertainty shown in Fig. _ the basketball has large aleatoric uncertainty due to rapid motion and the object boundaries have large epistemic uncertainty.
Relation to scale-space flow. Agustsson et al.] (2020) estimated a scale field 92 besides the 2- dimensional optical flow (ga, ). We may represent the decoded MV with multivariate Gaussian
5

---

Under review as a conference paper at ICLR 2022
distribution given by
9mo ~ (((x,999) Σ), E = Ix
0
9=
and the scale-space warp gives weighted mean of the warped value obtained from Jmv- The Gaussian mixture prediction from our proposed multi-head decoder can represent a more diverse predictive uncertainty distribution than the single Gaussian distribution in the scale-space flow. It has been shown in Garipov et al. (2018); Fort et al.] (2019) that ensemble models produce diverse results by learning different modes of the function, rather than interpolating around a given mean in the output space. It should be noted that while we only regress the mean of MV predictions in each ensemble branch and model the predictive uncertainty with the variance of the Gaussian mixture prediction, the multi-head decoder can easily be extended to an ensemble of scale-space flows.
3.3 ENSEMBLE- AWARE TRAINING
Intuitively, diversity is a key factor for ensemble models. Ensemble members similar in the param- eter space are unlikely provide any more useful information than their single-model counterpart. Krogh & Vedelsby (1995) proved the bias-variance trade-off in ensemble, E = E - A, which sug- gested that the inherent variance is the key for the ensemble models to be effective and we should encourage the diversity among the ensemble members.
In the previous literature, multiple approaches considered, including random initialization, hag- ging, and boosting. Randomly initializing the model parameters is simple but effective approach induce randomness and is quite suitable for deep ensembles (Lee et al., 2015). Bagging trains ensemble members on independently drawn examples with bootstrap sampling. Lee et al. (2015) showed that bagging may harm the model performance since each model may see only 63% of the available data and would perform poorly when there is high correlation inherent in the data (Bartlett) al. 1998). Boosting generates the ensemble models sequentially and can be quite time-consuming in terms of neural networks.
For deep compression models with our proposed multi-head decoder structure, there a single model with multiple parallel ensemble branches. Bagging boosting would not work for such network design since the model is trained on randomly sampled data but different ensemble branches would see the same data. Therefore, we choose to randomly initialize the network parameters and initial experiments show the effectiieness. To further encourage the diversity among the ensemble members, we propose ensemble-awand loss for deep ensembles to induce additional randomness.
Given h decoded outputs from the multi-head decoder, we obtain h predictions (motion- compensated predictions or final reconstructions) z¹ for i = 1,...,h. h. The ensemble-aware loss given by
Lensemble-aware (I, _1
t Σ
H ΣEN
minl -r.jll ISISHSSIS
(10)
and the minimum can be relaxed to choosing the smallest & out of the h norms. Our ensemble- aware loss encourage diversity between heads by relaxing the magnitude of the loss, and at the same time ensure performance of individual heads by updating every head at every iteration. Since the gradients propagated to each head depend on the head with the minimal MSE, rather than the MSE of the head itself, we can effectively induce randomness and therefore encourage diversity. Although the oracle loss proposed in Lee al. (2015) helps encourage diversity and boost the oracle accuracy. it would significantly harm the performance of individual ensemble members since each head only sees small portion of all training data. Experiments on learned video compression also showed that adopting the oracle set loss will decrease the overall performance. Instead, our ensemble-aware loss can effectively encourage diversity among ensemble members, and the same time, each individual member sees all training data and achieves reliable performance.
3.4 ADVERSARIAL TRAINING WITH FGSM
Adversarial examples (Szegedy et al. 2013) are training samples with a small but non non-random per- turbation that misclassified by neural networks with high confidence. Goodfellow al. (2014)

---

Under review as a conference paper at ICLR 2022
proposed the fast gradient sign method (FGSM) that applies linear but intentionally corst-case per- turbation to the training samples, which is given by
rj = sign( x. v))
(11)
where J(0,2,, 2r, y) is the cost function, and controls the norm of the perturbation. This adversarial training strategy has been shown to boost the image classification performance, as well as improve the model's robustness to adversarial examples. [akshminarayanan as an efficient solution to smooth the predictive distributions by increasing the likelihood of the target around an c-neighborhood of the observed training samples.
al. (2017) interpreted FGSM
We find adversarial training with FGSM closely related to learned lossy compression and an effective approach improve the performance of learned video codecs. In transform coding, we want the latent representation to be as smooth as possible, since after quantization, all latent representations the e-neighborhood, {a + 7 ||>/|| < <), corresponds to the same decoded output. Learning smooth latent representation would help to make the outputs more robust to quantization noise. Although this could be a natural result of an end-to-end optimized video codec, the experimental results show that FGSM can effectively improve the rate-distortion performance.
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Model architecture. Our base model architecture follows the design in Lu et al. (202bb) and we use auto-regressive and hierarchical priors for both the motion vector and residual compression. In order to optimize the model in an end-to-end manner, we need to relax the bits estimation since quantizing the latent bits would make the gradients zero almost everywhere. Following [Ballé al. (2016), we substitute the quantization operation with additive uniform noise during training and perform actual quantization during inference. The structure of multi-head decoder modules are depicted in Figure 1b and more technical details on the implementation are available in Appendix A.I
Training datasets. Our model trained on 64,612 video sequences from the training part in Vimeo- 90K settuplet dataset (Xue et al. (2019). Each video clip has 7 frames with resolution of 448 x 256. During training, we randomly crop the video sequences into 256 x 256 pixels. Given two successive frames from random sequence, treat the first frame as the reference frame and our model is trained to minimize the rate-distortion cost of encoding and decoding the second frame.
Training details. In our experiments, we adopt the progressive training strategy and warm up the inter-coding module for 150,000 steps with the ensemble-aware motion compensation loss in Equation 10 Then the model is end-to-end optimized with the rate-distortion loss given by
LRD (Rn + R2 (2:)) + X D(r,, f:))
(12)
where (w and (2:) represent the numbers of bits used to encode the motion vectors and the residual, D(F: F:) measures the distortion in mean squared error MSE or multi-scale structural similarity MS-SSIM (Wang et al. 2003), and À the hyperparameter controlling the trade-off. Four models are trained with different quality rates by setting À 256, 512, 1024, 2048. We use the AdamW optimizer [Loshchilov & Hutter 2017 with an initial learning rate of 1 x 10-4, which is then decreased x 10-2 Each model is trained on one NVIDIA V100 GPU.
4.2 QUANTITATIVE RESULTS
To show the effectiveness of our proposed incertainty-awarg model, we test our model on the first
100 frames from video sequences in HEVC (Sullivan et al. al. (012)) with GoP size 10, and the first 120 frames from sequences in UVG (Mercat et al. 2020). MCL-JCV (Wang et al., 2016) with GoP size 12. To balance the trade-off between complexity and performance, we choose kye = frres = 4. In order to build the best learning-baat based video codec, we adopt the state-of-the-arr image compression model (Cheng et al. 2020) for intra frame coding. In order to fairly compare the performance, we test the DVC and DVC Pro with the same same intra frame model, denoted DVC (cheng2020) and DVC Pro (cheng2020). For some of the previous state-of-te the-art deep video compression methods that are not available, such HU ECCV20 (Hu et al. 2020) and LU ECCV20 (Lu et al. 2020a).
7

---

Under review as a conference paper at ICLR 2022
Table Quantitative comparisons between different terningg-basce video compression models mea- sured in BD rate. The anchor model is x265 (veryslow). Negative number means bitrate saving and positive number means bitrate increase.
MODEL
HEVC B
HEVC C
HEVC
HEVC E
UVG
MCL-JCV
x264 (veryslow)
35.0%
19.9%
15.5%
50.0%
32.7%
30.3%
x265 (veryslow)
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
DVC (public)
26.7%
41.5%
31.1%
17.8%
21.9%
16.0%
DVC Pro (public)
-0.4%
11.5%
4.5%
-3.8%
-12.6%
-5.3%
DVC (cheng2020)
7.9%
15.1%
7.2%
21.1%
17.2%
13.3%
DVC Pro (cheng2020)
-9.0%
7.2%
-6.9%
17.2%
-7.9%
-4.1%
HU HULECCV20
2.4%
13.0%
10.8%
-8.6%
-5.4%
-12.6%
LULECCV20
5.0%
8.4%
3.6%
11.7%
8.8%
8.4%
Ours
-22.3%
-6.0% %
-19.0%
-24.3%
-25.5%
-18.2%
HEVC B
UVG
38
37
34
36
K261
PSNR
X264
PSNR
35
x265
DVC (public)
X265
DVC Pro (public)
DVC (public)
34
HU, ECCV20
DVC Pro (public)
LU, ECCV20
ECCV20
33
Agustsson CVPR20
ECDV20
NeRV
31
DVC (cheng2020)
DVC (cheng2020)
32
DVC (cheng2020)
Pro (cheng2020)
Our
Ours
31
0.05
0.10
0.15
0.20
0.25
0.30
0.05
0.10
0.15
0.20
0.25
bpp
bpg
Figure 3: Rate-distortion comparison between our model and x264 (veryslow), x265 (veryslow), Hu ECCV20 (Hu et al. 2020). LU ECCV20 (Lu et al., (020a), Agustsson CVPR20 (Agustsson et al. 2020). and NeRV (Chen et al. (02T) on sequences from HEVC Class B and UVG. Results on other HEVC sequences and MCL-JCV are reported in Appendix A.4 Best viewed color.
We calculate the BD-rate (Bjontegaard 200T) of different learning-basee based video compression models using x.265 as the anchor model and the results on HEVC, UVG, MCL-JCV are reported Table We also plot the RD curves of different codec models in Fig. More RD curve plots are available in Appendix A.4
use the public results reported in Hu 2020
From the results reported Table Fig. we could see that our proposed model can effectively save bits compared to our strong baseline and outperform previous state-of-the-an learning-bast based video codecs by a wide margin in all testing datasets.
4.3 QUALITATIVE RESULTS
In Fig. 2 we visualize the aleatoric uncertainty and epistemic uncertainty in the first two frames of the BasketballDrill sequence, as well as the predictive uncertainty represented by the multi-head see, the predictive uncertainty is larger in regions where the motion cannot
MV decoder. As be estimated well or too complicated to encode. Our uncertainty- represent the underlying uncertainty with an ensemble of decoded MVs and such uncertainty is retained until the final reconstruction. More visualizations the unsupervised predictive uncertainty can be found in Appendix A.6
model learns to effectively
predictivee
8

---

Under review as a conference paper at ICLR 2022
H
Bits Saving
HEVC D
93NR
3.3
MH-MV
MH-MV
31
0.04
0.06
0.00
0.10
0.32
0.14
2
Nur Heads
(a)
(b)
Figure 4: (a) Effectiveness of various proposed module. (b) Ablation study on the number of heads in multi-head decoders.
4.4 ABLATION STUDY
Effectiveness various proposed modules. We evaluate the effectiveness of multi-head decoders, ensemble-aware loss, and adversarial training with FGSM by running ablation experiments on the first 30 frames from sequences the HEVC dataset. We adopt the short training strategy for fast experimentation. The RD curves are presented in Fig. 4a and the BD-rates are reported in A.8 MH-MV and MH-Res Res stand for multi-heal MV decoder and multi-head residual decoder, and EA-L refers to training with ensemble-awan loss. The results show the effectiveness of the multi-head decoder modules and the training strategies.
Ablation study the number of decoder heads. We investigate the model's performance with different number of heads in the multi-head decoder on HEVC sequences. As shown Fig. 4b we train eight models with hime 1, 2, 3, 3, 5, 6, 7, where h. 1 is the baseline without ensem- bling. We see that the multi-head decoder module is effective even with only two decoder heads and the performance is improved with more decoder heads. Quantitative results are reported in A.8
Complexity analysis. Most previous deep ensembles train multiple stand-alone models (Szegedy al. 2015 Abbasi Gagné 2017 Wang et al., 2020) share very few shallow layers (Lee al., 2015 for the model to be ellective. With an ensemble of 6 models, the inference complexity (in MACs) and model size (in number parameters) easily increase by 500%. With the help of multi- head decoders, our ansemble-based model share the backbone features and achieve superior results with limited complexity increase. For one extra MV and residual decoder added, the complexity increases by approximately 7% and just 1% in the model size. For the largest model we consider, where Nime = free= = 8, there is only a 48% increase in complexity and 10% in model size.
DISCUSSION
In this paper, we studied the aleatoric uncertainty and epistemic uncertainty in deep learning-bass video compression and proposed utilize an ensemble of intermediate predictions to represent the predictive uncertainty at decoding time. With multi-head decoders, our model can properly model the uncertainties in the decoded MVs or residuals, which would effectively avoid propagating underlying errors in single deterministic prediction.
We investigated the performance of our uncertainty-aware decoding module and proposed novel ensemble-awan loss to boost the diversity among the parallel ensemble branches in single model. We also proposed to incorporate adversarial training for learning- based video codecs. Experimental results show the effectiveness of our approach.
baased
Compared to one-stage learning-basse based video compression models, such as those based on 3D au- toencoders (Pessoa et al., 2020; Habibian al. 2019), two-stage motion compensation- mod- els can decode high-quality frames with low-latency. However, intermediate predictions in these two-stage pipelines are not always accurate and erroneous predictions could severely harm the per- formance of later stages, especially for out-of-distributiio data. Therefore, it critical to represent the predictive uncertainty and our proposed multi-head decoder is a simple but very effective ap- proach to capture such uncertainty. Future directions could involve modules on the encoder side to model and propagate the uncertainty the decoders for end-to-end uncertainty-awant

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
Key results in this paper are reproducible and in this section, we will refer to details helpful to reproduce the results. The implementation of our model are modified from the codes released by Bégaint et al. (2020) and Hu (220)). We propose the multi-head decoders in Section 3.2 and can be easily implemented by following the flow charts in F technical details in Appendix In Section 4.1 we introduce the dataset we use to train the model. The Vimeo-90K settuplet dataset (Xue et al. 2019) publicly available and we our model on the training part of the dataset. Also in Section 4.1 summarize training strategy, which includes ensemble-aware loss for model warm-up and then minimizing the rate-distortion cost until convergence. The motivation and formality of the ensemble-awen loss be found in Section 3.3 This loss can be easily implemented in Py PyTorch. Since directly clamping the loss gives zero gradients almost everywhere, create custom autograd function to forward the gradients before and after the clamp operation and properly update the weights of each head.
REFERENCES
Vtm reference software for vvc. WVCSoftware VTM Accessed: 2021-11-17.
httts::/vegit..tt Fraunhofer .ttpsferrrreeeeeee ivet/
Mahdieh Abbasi and Christian Gagné. Robustness to adversarial examples through ensemble of specialists. arXiv preprint arXiv:1702.068: 2017.
Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized video compression. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimization of nonlinear trans- form codes for perceptual quality. In 2016 Picture Coding Symposium (PCS), pp. 1-5. IEEE, 2016.
Peter Bartlett, Yoav Freund, Wee Sun Lee, and Robert E Schapire. Boosting the margin: A new explanation for the effectiveness of voting methods. The annals of statistics, 26(5):1651-166 1998.
Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. Compressai: a py- torch library and evaluation platform for end-to-end compression research. arXiv preprint urXiv:2011.0300 2020.
G. Bjöntegaard. Calculation of average psnr differences between rd-curves. 2001.
Hao Chen, Bo He, Hanyu Wang. Yixuan Ren, Ser-Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos s. In NeurIPS, 2021.
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jíro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Cisco. Cisco annual internet report (2018-2023) white paper. 2020. URL httpt::t//// //wp::://www.. www. col cisco. //scoo.coo////e/////////// http://en/nooo.caaaaeeeeeeeeeee attera//eeeeuuiiiiteeaa www.iveeeeeeeteteeeeee ivess
annual-inte www.rttttttt.. ol 741490. html
Armen Der Kiureghian and Ove Ditlevsen. Aleatory epistemic? does it matter? Structural safety,
3152):105-112, 2009.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop multi- ple classifier systems, pp. 1-15. Springer, 2000.
Runsen Feng, Yaojun Wu, Zongyu Guo, Zhizheng Zhang, and Zhibo Chen. Learned video com- pression with feature-level residuals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 120-121, 2020.
10

---

Under review as a conference paper at ICLR 2022
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective. arXiv preprint drXiv:1112.02777, 2019.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective, 2020.
Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL /ttps://prooceeddngsss http://eee.......oo.oooooooo///a cc/pppper/21///ilee (23007774811044444444466663633-3
pdd
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577 2014.
Amirhossein Habibian, Ties van Rozendaal, Jakub M Tomezak, and Taco S Cohen. Video compres- sion with rate-distortion autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7033-7042, 2019.
L.K. Hansen and P. Salamon. Neural network ensembles. IEEE Transactions on Puttern Analysis and Machine Intelligence, 121100993--001, 1990. doi: 10.1109/34.5887.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1rX1v:1610.022366 2016.
Geoffrey E. Hinton and R. Neal. Bayesian learning for neural networks. 1995.
Zhihao Hu. Pytorch video compression.
https://github.. http:///ittuub.ccco/hhaahaa www.:hihohhhh//////
PyTorchiiideoComprsss
2020.
Zhihao Hu, Zhenghao Chen, Dong Xu, Guo Lu, Wanli Ouyang, and Shubang Gu. Improving deep video compression by adaaption-adaative flow coding. In European Conference on Computer Vision, pp. 193-209. Springer, 2020.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? arXiv preprint arXiv:1703.04977. 2017.
Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In G. Tesauro, D. Touretzky, and T. Leen Advances in Neural Information Processing Sys- tems, volume 7. MIT Press, 1995. URL http:httt////ttt httttps:://proccedinng.....
www.:eiiiiinnnnnnninn..mee neurips cc/ /aper/ .dd
1994
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scal- able predictive uncertainty estimation using deep ensembles. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan. and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL ht Fa272244477occcaacaaaaaaaaaaaaa2
/tt.ppps..//aaerr/1////11 pdf
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1711.05 2017.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dve: An end-to-end deep video compression framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11006-11015, 2019.
11

---

Under review as a conference paper at ICLR 2022
Guo Lu, Chunlei Cai, Xiaoyun Zhang. Li Chen, Wanli Ouyang, Dong Xu, and Zhiyong Gao. Con- tent adaptive and error propagation aware deep video compression. In European Conference on Computer Vision, pp. 456-472. Springer, 2020a.
Guo Lu, Xiaoyun Zhang, Wanli Ouyang, Li Chen, Zhiyong Gao, and Dong Xu. An end-to-end learning framework for video compression. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2020b. doi: 10. 10.1109//PAM..2220.222 29884533.
David JC MacKay, A practical bayesian framework for backpropagation networks. Neural compu- tation, (3)):4484477 1992.
Alexandre Mercat, Marko Viitane,, and Jarno Vanne. Uvg dataset: 50/120fps 4k sequences for video codec analysis and development. In Proceedings of the 11th ACM Multimedia Systems Conference, pp. 297-302, 2020.
D.A. Nix and A.S. Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94), volume 1, pp. 55-60 60 vol.1, 1994. doi: 10. 1109//CNN.1194.3744....
Michael Perrone and Leon Cooper. When networks disagree: Ensemble methods for hybrid neu- ral networks. Neural networks for speech and image processing, 08 1993. doi: 10. 1422/
9789812795885_002.
Jorge Pessoa, Helena Aidos, Pedro Tomás, and Mário AT Figueiredo. End-to-end learning of video compression using spatio-temporal autoencoders. In 2020 IEEE Workshop on Signal Processing Systems (SiPS), pp. 1-6. IEEE, 2020.
Anurag Ranjan and Michael J. Black. Optical flow estimation using a spatial pyramid network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander Anderson, and Lubomir Bour- dev. Learned video compression. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3454-3463, 2019.
Gary Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (hevc) standard. IEEE Transactions on circuits and systems for video technology, (611)):644-1168, 2012.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312.619999 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015. doi: 20.11099/VPR..215.22285.
Suramya Tomar. Converting video formats with ffmpeg. Linux Journal, 2006(144):114 2006.
Haiqiang Wang. Weihao Gan, Sudeng Hu, Joe Yuchich Lin, Lina Jin, Longguang Song, Ping Wang, Ioannis Katsavounidis, Anne Aaron, and C-C Jay Kuo. Mcl-jev: a jnd-based h. 264/ave video quality assessment dataset. In 2016 IEEE International Conference on Image Processing (ICIP), pp. 1509-1513. IEEE, 2016.
Yefei Wang. Dong Liu, Siwei Ma, Feng Wu, and Wen Gao. Ensemble leamningbasse distoriio optimization for end-to-end image compression. IEEE Transactions on Circuits and Systems for Video Technology, 2020.
Zhou Wang, Eero Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pp. 1 1398-1402. leee, 2003.
12

---

Under review as a conference paper at ICLR 2022
David H. Wolpert. Stacked generalization. Neural Nenworks, 5(2):241-23) 229, 1992. ISSN
0893-6080. doi: http://cooaao//looodod0003000000 URL https: www.
seiencedirect.co.tee httpa///c.....o...o........aaaat
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement
with task-oriented flow. International Journal of Computer Vision, 127(8): 1778)):106-1125) 2019.
A APPENDIX
A.1 IMPLEMENTATION OF MULTI-HEAD DECODERS
As depicted in Figure lb multi-head decoders consist of a decoder backbone and multiple decoder heads. The decoder heads are light-weight and include two convolution layers with one leaky ReLU in between. For multi-head MV decoder, the decoder backbone first decodes MV feature from the MV bitstream â, and then h MVs are decoded with respective MV decoder heads. From the decoded MVs and the previous decoded frame. obtain h MC predictions with bilinear warping. The h MC predictions are concatenated with the previous decoded frame and sent to the Prediction Refine Net, from which we get refined MC predictions. For multi-head residual decoder, the de- coder backbone first decodes residual feature from the residual bitstream bú. and then h residuals decoded with respective residual decoder heads. From the h decoded residuals and the h refined MC predictions, we obtain reconstructions. The h reconstructions are concatenated with h refined MC predictions and sent to the Reconstruction Refine Net, from which we get refined reconstruction as the final decoded frame. All modules in our model, including decoder backbone, decoder head, and refine nets, are implemented with neural networks and optimized in an end-to-end manner.
A.2 VISUALIZING THE ALEATORIC UNCERTAINTY
To visualize the aleatoric uncertainty introduced from the quantization operation, conduct pre- liminary experiments. We add a small perturbation Yo to the quantized bit stream and obtain vi Wi + 7/o- The perturbation тo only added positions where the quantization gap is least 0.1 and corresponds to only 20% of the size of the perturbation gap. We model the alcatoric uncertainty with the LI norm between the two optical flows decoded from bit streams that differs in small perturbation. Results on the first two frames of the BasketballDrill sequence are shown in Figure 2d As we can see, the aleatoric uncertainty is not uniform across the whole image. Instead, there are more aleatoric uncertainty around the object boundaries and regions where the motion is large. While by definition, such aleatoric uncertainty cannot be reduced away, blindly assuming the optical flows to be accurate would lead to larger intermediate residual errors and cost more bits in the residual coding.
A.3 MV PREDICTION AS GAUSSIAN MIXTURE MODEL
The meati-head MV decoder predicts an ensemble of motion vectors (. k == 1, hmw from the MV bit stream and we can represent the prediction an equally-weighted Gaussian mixture model given by
0
N(f
2
Σ
FLn
(13)
Since the Gaussian distributions equally the mean of the mixture distribution is
B
Mj.
(14)
13

---

Under review as a conference paper at ICLR 2022
and the variance of the mixture distribution in each direction is
2
Pj.
(1::e) u
E
Hmw
k=1
Amw
(ofte) ( -13
(15)
hmv
In order visualize the predictive uncertainty represented by this ensemble model, we (o),))
(o = and obtain
2
1
(1,00 )
)²)
fcs
hyno
k=1
P/
+1
(16)
Nyme
Rymor
A.4 QUANTITATIVE RESULTS
32
x264
PSNR
31
x264
DVC (public)
DVC (public)
30
(public)
Pro (sublic)
HU ECCV20
HU, ECCV20
29
DVC (cheng2020)
DVC (cheng2020)
28
(cheng2020)
Pro (cheng2020)
Ours
Ours
27
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
bpp
bpp
(a)
(b)
Figure 5: Rate-distortion comparison between our model and x264 (veryslow), x265 (veryslow), Hu ECCV20 (Hu et al. 2020). LU ECCV20 (Lu et al. 2020a), Agustsson 20 (Agustsson al. 2020 sequences from HEVC Class C and HEVC Class D. Best viewed in color.
14

---

Under review as a conference paper at ICLR 2022
HEVC E
MCL-|CV
40
38
37
2SNR
35
X264
X264
36
x265
DVC (public)
DVC (public)
ECCV20
DVC Pro (public)
ECCV20
35
ECCV20
Agustssan CVPR20 NeRV
LU, ECCV20
DVC (cheng2020)
(cheng2020)
DVC Pro (cheng20201
DVC Pro (cheng2020)
31
0.04
0.025 0.050
0.075
0.100
0.150 0.175 0.200
(a)
(b)
Figure Rate-distortion comparison between our model and x264 (veryslow), x265 (veryslow), Hu ECCV20 (Hu et al., 2020), LU_ECCCC ULECCV20 (Lu et al.) 2020a), CVVP22 al. 2020) on sequences from HEVC Class E and MCL-JCV, Best viewed in color.
(Agustsson
HFVC 8
HFVC
0.97
0.96
MS-5SIM
0.95
MS
X264
0.95
X264
X265
x265
DVC (public)
DVC (public)
Pro (public)
0.94
(public)
HU ECCV20
HU, ECCV20
LU ECCV20
0.93
IU ECCV20
0.92
bpp
bgp
(a)
(b)
Figure Rate-distortion comparison between our model and x264 (veryslow), x265 (veryslow), Hu ECCV20 (Hu et al.) 2020). LU EUCEC (Lu et al.] 2020a),, Eguutsson_CVPRRS (Agustsson ct al. 2020) on sequences from HEVC Class B and HEVC Class C. Distortion measured in MS.
SSIM. Best viewed in color.
15

---

Under review as a conference paper at ICLR 2022
HEVC
MC1-JCV
0.99
0.98
0.98
0.97
M5.5 M5-55MM
MS-55MM
0.96
0.96
0.95
x265
0.95
X264
DVC (public)
X265
(public) HU_ECCV2
DVC
0.94
MU_E0O220
LU ECC220
LU ECCV2U
0.20
0.35
0.45
0.175
app
app
(a)
(b)
Figure Rate-distortion comparison between our model and x264 (veryslow), x265 (veryslow), Hu ECCV20 (Hu et al. 2020), LU ECCC20 (Lu et al., 2020aa), Coustsson_CVRR20 (Agustsson al. 2020), and NeRV (Chen et al. 2021) on sequences from HEVC Class D and MCL-JCV. Distortion measured in MS-SSIM. Best viewed in color.
A.5 MODEL PERFORMANCE USING DIFFERENT I-FRAME MODELS
In this section, investigate the performance of proposed model using different I-frame models on HEVC Class В and UVG. In Section 4.2 we report the experimental results of our proposed model using Cheng2020 (Cheng al.) 2020) as the I-frame model. Compared
DVC Pro, our proposed model with multi-head decoders can effectively reduce the bpp while maintaining the same reconstruction quality. shown Figure 3 this brings our model to a range of lower bitrates. In order to investigate our model performance in higher bitrates, we retrain our model with À = 4096 and use VTM (VTM) as the I-frame model. The performance
our model and DVC Pro using VTM are reported in Figure 9 We could see that using VTM instead of cheng2020 the I-frame model brings both our proposed model and DVC Pro to higher bitrates and similar conclusions be made. Our proposed model still outperform previous state-of-the- models in higher bitrates and the multi-head decoders can bring significant bitrate savings regardless the I-frame model used the range of bitrates.
16

---

Under review as a conference paper at ICLR 2022
HEVC B
UVG
35
X264
36
X285
X264
DVC (public)
X265
335
DVC Pro (aublic)
DVC (public)
ECCV20
DVC Pro (public)
LU, EUCC22
32
ECCV20
Agustsson CVVPR2
LU ECCV20
NeRV
DVC (cheng2020)
(cheng2020)
31
DVC Pro icheng2020)
DVC Pro (cheng2020)
30
Ours (VTM)
Ours (VTM)
DVCPro
DVCPro (VTM)
0.10
0.15
0.20
0.30
0.15
0.20
0.25
hpp
(a)
(b)
Figure Rate-distortion comparison between our model and x264 (veryslow), x265 (veryslow), Hu ECCV20 (Hu et al.) 2020), U_ECC220 (Lu et al.) 2020a),, Egustsson.CVPR2) (Agustsson et al. 2020). and NeRV (Chen et al. 2021) on sequences from HEVC Class and UVG. Distortion measured in PSNR. Best viewed in color.
17

---

Under review as a conference paper at ICLR 2022
A.6 VISUALIZATION OF UNSUPERVISED PREDICTIVE UNCERTAINTY
Current frame of BasketballDrive
Predictive uncertainty
3
(c) Current frame F, of RaceHorses
(d) Predictive uncertainty
(e) Current frame F: of Kimonol
(f) Predictive uncertainty
Figure 10: Visualization of the predictive uncertainty represented by our proposed multi-head de- coder on BasketballDrive, RaceHorses, and Kimonol.
A.7 QUALITATIVE RESULTS DECODED FRAMES
In this section, we show the qualitative examples of decoded frames for subjective comparison. We compare the performance of our proposed model with the baseline DVC Pro to show the strength and weakness of multi-head decoders. In sequence BQTerrace from HEVC B, proposed model achieves a bitrate saving 20.5% over DVC Pro. Sequence 20 from MCL-JCV is the only sequence where model failed outperform DVC Pro with bitrate increase of 2.2%.
18

---

Under review as a conference paper at ICLR 2022
Figure 11: Comparison between our proposed model with DVC Pro on a decoded P-frame from sequence BQTerrace in HEVC B. In this sequence, our proposed model has a bitrate saving of 20.5% over DVC Pro. Top: our proposed model (0.01472 bpp); middle: DVC Pro (0.02587 bpp); bottom: predictive uncertainty.
19

---

Under review as a conference paper at ICLR 2022
Figure 12: Comparison between our proposed model with DVC Pro on a decoded P-frame from sequence 20 in MCL-JCV. This the only sequence where our proposed model failed to outperform DVC Pro with a bitrate increase of 2.2%. Top: our proposed model 10.10027 bpp); middle: DVC Pro (0.11786 bpp); bottom: predictive uncertainty.
A.8 DETAILED ABLATION STUDY RESULTS
To show the effectiveness of various proposed modules and to compare between models with dif- ferent number of decoder heads, we train multiple models on the Vimeo-90K dataset with the fast training strategy and test the models the first 30 frames in each HEVC sequence. We measure
20

---

Under review as a conference paper at ICLR 2022
the model performance with BD rates and the quantitative results reported in Table 2 and Table 3 respectively.
Table 2: Ablation study on the effectiveness of each module. The performance are measured in BD rates using our baseline model as the anchor. MH-MV: multi-head MV decoder, MH-Res: multi- head residual decoder, EA-L: ensemble-aware loss, FGSM: adversarial training with FGSM. All models have Am
hres = 4 considering the trade-off between performance and complexity.
Setting
НЕVСВ B
HEVCC
HEVC
HEVCE
MH-MV MH-MV + EA-L MH-MV MH-Res EA-L
-5.8
-3.1
-4.0
-3.7
-7.0
-4.0
-6.7
-4.8
-8.7
-6.4
-8.5
-6.7
MH-MV + MH-Res + EA-L + FGSM
-12.7
-7.9
-11.2
-11.4
Table 3: Ablation study on the number of heads in multi-head decoder. The performance are measured in BD rates using our single-head baseline as the anchor. For simplicity, we set hime = hres and train eight models with hmv hrea I,..., 8 using the fast training strategy.
Setting
HEVC B
CEVC
HEVED
HEVC
hmv hres 1
0.0
0.0
0.0
0.0
haa hyes 2
-7.8
-4.8
-8.3
-5.7
DmN hres 3
-8.2
-5.0
-8.2
-6.1
hw hyea 4
-8.7
-6.4
-8.5
-6.7
hmv = hrea = 5
-8.9
-6.1
-9.5
-7.6
hy hres 6
-9.4
-6.9
-9.7
-8.2
hav Bres = 7
-10.5
-7.2
-9.9
-8.7
hm bres 8
-10.4
-6.8
-9.8
-8.2
21


Reviews:
[{"review_id": "hW50cAxtABp", "reviewer": "Reviewer_BZ5c", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Here are issues or concerns regarding this paper. \n(1) Both F_i and x_i are used to denote the current frame, e.g., in Fig. 2 (a) and in Sec. 3.1. \n(2) What are the differences between the raw prediction and motion compensation prediction? \n(3) Section 3.3 explains that the ensemble-aware loss can be relaxed to have the k-smallest outputs. \nCan the relaxed prediction improve the prediction performance? If the k=1 is the best choice, why is that so? \n(4) There is a broken sentence at the end of Sec. 4.1. \n(5) What are the definitions for the Prediction Refine Net and Reconstruction Refine Net in Fig. 1(b)? \nAre they neural networks that are trained in the framework? \nIf so, does the prediction refine net take multiple predictions and refine them simultaneously? \nIf not, I think the authors should clarify how the underlying uncertainties are predicted and utilized to better compress the video in the entire framework. \n(6) Predictive uncertainties are analyzed by assuming the predictions from multi-head decoders follow the mixture of Gaussian model. \nI think the number of decoders should be sufficiently large to support this claim.\n(7) Check for typos, punctuations, and articles. \n(8) Dependency on the optical flow estimation method. \n(9) Failure cases? \n\n\n\n\n", "summary_of_the_paper": "This paper presents an uncertainty-aware video compression framework with an ensemble of MV/residual decoders. To train the network, they use the ensemble-aware loss function which minimizes the loss function for the k best predictions. Moreover, this paper also analyzes underlying uncertainties in video compression, i.e., aleatoric and epistemic uncertainties, with visualization techniques for those uncertainties. Experimental results show that the proposed method can improve the compression performance by about 20% while not significantly increasing the computational complexity.", "summary_of_the_review": "Overall, the paper was interesting because of the in-depth analysis and insights. However, I have a few concerns as stated in the main review. I would like to hear the answers to (3), (5), (6), (9) from the authors. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UzPQscgW7h", "reviewer": "Reviewer_ZZDv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The paper is well-structured. \n+ The proposed model outperforms previous state-of-the-art models of deep learning video codecs, and the ablation study proves the effectiveness of each module.\n\nWeaknesses:\n- There are quite some theoretical analysis in the paper to support the need of using ensemble/multi-head, which seems nice, but I am not sure if this is necessary. The idea of multi-head has been widely used in transformer models and it is well-known that this can significantly improve performance in practice. Can the authors better contrast their idea with the multi-head idea used in typical transformers?\n- In Eq 10, what does i stand for? each pixel? Also, why take summation of j=1..h, isn't the min is over j which means at 1 single location, only 1 prediction is used for computing L2? Further it would be very helpful if the authors explain how this is designed beyond just citing Lee et al. Why this loss function can encourage diversity? Pls add this in the main text as this is one of the main contributions of this paper.\n- For results in Fig 3, why the proposed method's curve stops earlier than other approaches? How would the method compare with other methods when bpp is high such as> 0.15 in (a), and can the proposed approach achieve the overall better PSNR when allowing larger bpp?\n- Compared to traditional codecs, the deep learning compressors have large model size in general, so that hard to deploy on low-capability device. I assume this proposed approach would have similar issue?", "summary_of_the_paper": "This paper works on end-to-end deep learning video compression. The authors study the inherent uncertainty and accordingly propose a so-called ensemble approach which is in effect multi-head decoder. A so-called ensemble-aware loss is proposed to encourage the diversity between ensemble members. Further, adversarial training is incorporated. Experiments show that the proposed model outperforms previous state-of-the-art models such as DVC Pro (Lu et al., 2020b) and Lu et al. (2020a), and the ablation study proves the effectiveness of each module.\n", "summary_of_the_review": "I am on borderline as there are specific concerns outlined above regarding contribution of multi-head this simple idea, presentation of the ensemble-aware loss, results when bpp is high, etc. But I also believe deep codes is a promising direction and we shall encourage more work in this field and may go with a lower bar. If the authors can address my questions, I will lean to accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DBq_ur-ZMz", "reviewer": "Reviewer_3SEn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(+) Experimental results suggest that the ensemble and multihead based proposed method outperforms prior works. \n(-) Novelty of the paper is not clear. It seems that the general framework of the proposed compression scheme follows DVC but with multihead mv decoder based on equally-weighted Gaussian mixture. \n(-) The ensemble-based idea does not seem original as there have been several prior networks (Lakshmnarayanan et al, Agustsson et al. etc) as alluded by the authors. The difference between the proposed and prior work should be concisely and explicitly described. \n(-) Readability should improve with the notation used in the text (section 3.1) correlates with the figure 1. \n(-) Should make a performance comparison with the recent paper which is readily available on archive\nMissing reference \nHao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, Abhinav Shrivastava, NeRV: Neural Representations for Videos NeurIPS 2021\n", "summary_of_the_paper": "This paper proposes an ensemble-based video compression model to capture the predictive uncertainty of intermediate predictions. A loss is constructed to encourage diversity between ensemble members, and the paper investigates the benefit of incorporating adversarial training in video compression. The experimental result shows that the proposed model can save more than 20% compared to DVC Pro. ", "summary_of_the_review": "It is difficult to pinpoint the novelty of the proposed compression algorithm as it stitches various ideas from various areas (ensemble, multi-head, FGSM) that have already been proposed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2XUyniVJLRJ", "reviewer": "Reviewer_Fyqz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is interesting and a good try as a deep learning method for video compression. The experimental results seem strong. The ablation study looks extensive. However, the reviewer has several concerns.\n\n1, the motivation of multi-head, i.e., the ensemble, is not clear. Why ensemble? What is its benefit?\n2, in the introduction, the authors claim that existing methods suffer from the inaccurate estimation of optical flow. However, the proposed method also employs motion estimation, how can the proposed method avoid this issue?\n3, in the ablation study, multiple heads look useful. In this case, why not use more heads?\n4, poor representation. For example, the figures are not self-contained. One needs to refer to different parts in the texts to understand the figure, such as MV, Res in Fig. 1. There are many wrong sentences with grammar issues. Some are given but they are definitely not all.\na, caption in figure 2, \"are cannot be\"\nb, pp4, \"it follows that the upper ... is given by\"\nc, pp5, MEMSE\nd, pp4, Fig.9e should be Fig.2d?\nf, pp5. Fig.9f should be Fig.2e?\n\n", "summary_of_the_paper": "This paper aims to capture the predictive uncertainty and proposes an ensemble-based video compression approach. In the proposed approach, an ensemble-aware loss is also introduced to seek diversity among ensemble members. Experimental results show its performance.", "summary_of_the_review": "In general, this paper solves a classic problem with deep learning, which is interesting. While there are several issues (see the main review section), which prevent the reviewer from scoring it high with the current status.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper exposes a method for video compression based on multi-head models.
The reviewers seem to agree that the results are interesting, and worth publishing.
On the other hand, there are many concerns raised on the quality of the writing, with grammatical mistakes and confusing parts. The motivation for the multi-head models, as well as its novelty, has been questioned in all reviews. Although the authors rebuttal has lead some reviewers to increase their score, it's still very concerning that authors needed to explain the main point of the paper to each reviewers. I think that the authors should polish this paper, taking into account the reviewers feedback, which would make a stronger paper, and submit it again in a future venue. I therefore recommend reject for this paper.
Decision: Reject