Year: 2021

Abstract:
Deep learning-based video compression is a challenging task and many previous state-of-the-art learning-based video codecs use optical flows to exploit the temporal correlation between successive frames and then compress the residual error. Although these two-stage models are end-to-end optimized, errors in the intermediate errors are propagated to later stages and would harm the overall performance. In this work, we investigate the inherent uncertainty in these intermediate predictions and present an ensemble-based video compression model to capture the predictive uncertainty. We also propose an ensemble-aware loss to encourage the diversity between ensemble members and investigate the benefit of incorporating adversarial training in the video compression task. Experimental results on 1080p sequences show that our model can effectively save bits by more than 20% compared to DVC Pro (Lu et al., 2020b).

heading: 1 INTRODUCTION
text: Video contents are reported to account for 82% percent of all consumer Internet traffic by 2021 and are growing rapidly with an increasing demand for high-resolution videos (e.g., 4K movies) and live streaming services (Cisco, 2020). It is therefore critical for us to improve the video compression performance to transmit video with higher quality given limited Internet bandwidth. In recent years, there has been a surge of deep learning-based video compression models (Rippel et al., 2019; Lu et al., 2020b; Agustsson et al., 2020) and some of them have achieved comparable or even better performance than previous traditional video codecs, such as x.264 (Tomar, 2006).
Despite previous deep learning-based video codecs have achieved improved performance on many challenging datasets, most of the state-of-the-art models estimate deterministic predictions for intermediate representations, such as optical flows and residuals. These models fail to represent the aleatoric uncertainty inherent in the model inputs or the epistemic uncertainty in the model parameters and would blindly assume the predictions to be accurate, which is not always the case (Der Kiureghian & Ditlevsen, 2009; Kendall & Gal, 2017). In terms of video compression, such models produce deterministic motion vectors (or optical flows) and residuals for each pixel location, ignoring the fact that optical flows may not be estimated accurately in occluded regions and around object boundaries, and the quantization operation for lossless entropy coding also introduces additional noises to the inputs of the decoders. Underlying errors in such overconfident intermediate predictions are propagated to later stages of the P-frame model and even to subsequent frames for models built on temporal correlation, leading to suboptimal performance of the compression system.
Predictive uncertainty is crucial for us to understand how certain the model is about the predictions, especially for out-of-distribution data. However, most neural networks do not offer such information and tend to produce overconfident predictions (Gal, 2016; Lakshminarayanan et al., 2017). Bayesian neural networks (MacKay, 1992; Hinton & Neal, 1995) are widely used to quantify predictive uncertainty, but lack practicality due to greatly increased computation complexity and do not scale well to high dimensional data. Gal & Ghahramani (2016) proposed Monte Carlo dropout that performs test-time dropout. It is simple to implement but is not suitable for deep learning-based compression since it requires multiple decoding-time inferences and yields non-deterministic outputs.
In terms of deep learning-based video compression, two non-Bayesian approaches are considered to represent the predictive uncertainty: (1) modeling the uncertainty explicitly by regressing the empirical uncertainty of the model outputs (Nix & Weigend, 1994); and (2) using ensembles for predictive
uncertainty estimation (Lakshminarayanan et al., 2017). Agustsson et al. (2020) took the first approach and proposed scale-space flow for motion compensation. Specifically, their model predicts a scale field besides the standard 2-dimension flow field, representing the uncertainty associated with each MV. Gaussian blurring is then applied to the reference frame and the scale parameter is used to control the scale of the Gaussian kernel. Experimental results show that the self-supervised scale parameter increases around the boundaries of the objects and in regions where the motion is large, which confirms that the scale field learns to estimate the uncertainty of the optical flow prediction.
In this work, we consider the second approach and represent the underlying uncertainty with an ensemble of outputs. Ensembles of models have been shown to improve predictive performance (Dietterich, 2000), and ensembles of neural networks have also been used to boost model performance on many challenging benchmark datasets (Szegedy et al., 2015). Instead of producing a deterministic prediction, ensemble methods perform model combination and reflect the uncertainty of out-of-distribution data. Here we propose an ensemble-based multi-head decoding module that can generate an ensemble of intermediate outputs, such as motion vectors and residuals, and implicitly represent the predictive uncertainty with the variance of the Gaussian mixture prediction. We no longer assume that any individual intermediate prediction is accurate and explicitly consider multiple candidates of them. Such uncertainty is then propagated to later stages and all modules in our framework are optimized in an end-to-end fashion.
To further improve the performance of our ensemble-based video compression model, we propose an ensemble-aware loss to encourage diversity between different branches and incorporate an adversarial training strategy, fast gradient sign method (FGSM) (Goodfellow et al., 2014), to smooth the intermediate latent representations. Our experiments show that our model achieves an average bitrate saving of 15% when the distortion is measured in PSNR.
The contributions of this work can be summarized as follows:
• We are the first to leverage deep ensembles in a video compression model and demonstrate that our approach can be used to capture the underlying uncertainty of intermediate representations and prevent the model from being overconfident with intermediate predictions.
• We demonstrate that our approach can effectively improve the performance of deeplearning based video compression and can be widely applied to optical flow-based video codecs with negligible complexity increase.
• Our experiments show that our model outperforms previous state-of-the-art models such as DVC Pro (Lu et al., 2020b) and Lu et al. (2020a), and our ablation study proves the effectiveness of each module.

heading: 2 RELATED WORK
text: Learned video compression. Previous learning-based video compression methods can be categorized into two groups: (i) one-stage models, such as methods based on 3D autoencoders (Pessoa et al., 2020; Habibian et al., 2019); and (ii) two-stage models, which is adopted by most previous state-of-the-art methods, consist of predicted frame generation and residual coding. Lu et al. (2019) proposed an end-to-end trainable video codec, DVC, that utilizes an optical-flow network (Ranjan & Black, 2017) for motion compensation and then compresses the residuals. An extension of DVC, known as DVC Pro (Lu et al., 2020b), improves the compression performance by introducing refinement modules and auto-regressive entropy models. Feng et al. (2020) considered residual coding in the feature domain and choose between pixel-level residuals and feature-level residuals with RDO at encoding time. Agustsson et al. (2020) proposed scale-space flow to blur the intermediate reconstructions when motion vectors are not estimated well. Chen et al. (2021) proposed to represent videos with neural networks and achieved promising results for video compression.
Model uncertainty. Predictive uncertainty can be grouped into aleatoric uncertainty and epistemic uncertainty (Der Kiureghian & Ditlevsen, 2009). Aleatoric uncertainty captures the noises inherent in the observations and cannot be explained away with more data, while epistemic uncertainty accounts for uncertainty in the model structure or parameters and can be reduced with more training data. Bayesian neural networks (MacKay, 1992; Hinton & Neal, 1995) is a widely used approach for modeling predictive uncertainty that extends the traditional neural networks by learning a posterior distribution of model parameters from the observed data. Various non-Bayesian approaches
have also been proposed, such as utilizing the probabilities of softmax distributions (Hendrycks & Gimpel, 2016) and Specialists+1 Ensemble for representing the predictive uncertainty for adversarial samples (Abbasi & Gagné, 2017). Gal & Ghahramani (2016) proposed Monte Carlo dropout by performing multiple inferences with Dropout at test time. Lakshminarayanan et al. (2017) proposed to use an ensemble of neural networks for quantifying predictive uncertainty.
Deep Ensembles. The neural networks community has been investigating ensembles of deep networks since the early 1990s (Hansen & Salamon, 1990; Wolpert, 1992; Perrone & Cooper, 1993). Krogh & Vedelsby (1995) proved the bias-variance trade-off for ensemble models which suggested the importance of the diversity among ensemble members. Lee et al. (2015) investigated several training strategies to train an ensemble and proposed ensemble-aware oracle loss to encourage diversity. GoogLeNet (Szegedy et al., 2015), one of the best-performing models on ILSVRC 2014, is an ensembles of CNNs. Lakshminarayanan et al. (2017) proposed to estimate predictive uncertainty by training multiple stand-alone neural networks. Garipov et al. (2018); Fort et al. (2020) showed that deep ensembles can learn different modes of function with ensemble members that only differ in initialization weights.

heading: 3 UNCERTAINTY-AWARE DEEP VIDEO COMPRESSION
text: This section presents our main contributions. We introduce the theoretical background and the motivation of our proposed approach in Section 3.1. Then we introduce the ensemble-based multihead structure to decode multiple candidates of motion vectors and residuals in Section 3.2. In order to encourage the diversity among the ensemble members and to improve the overall performance, we propose an ensemble-aware loss for multi-head decoders in Section 3.3. Finally, we introduce an adversarial training strategy that we find beneficial for the learning-based video compression task in Section 3.4.

heading: 3.1 UNCERTAINTIES IN DEEP VIDEO COMPRESSION
text: We consider the predictive coding-based framework, DVC, proposed by Lu et al. (2020b) (Fig. 1a). Let the current frame be xi and the reconstructed previous frame from the buffer be x̂i−1. We estimate a motion vector (MV) map fi with a motion estimation network. The optical flow is then sent to a motion auto-encoder for transform coding, yielding quantized bits âi and the reconstructed optical flow f̂i. Bilinear warping is used for motion compensation (MC) and a MC prediction x̃i with residual ri = xi−x̃i is obtained. The residual ri is then compressed with a residual encoder and decoder, outputting quantized residual bit stream b̂i and the decoded residual r̂i. The reconstructed current frame is the sum of the MC prediction and the decoded residual written as
x̂i = BilinearWarp(x̂i−1, f̂i) + r̂i. (1)
Aleatoric uncertainty. Although at encoding time we have full information necessary to decode F̂i, for lossy compression at certain bit rates, we quantize the bit stream that is passed to the decoder, and inevitably introduce aleatoric uncertainty at decoding time. Since the aleatoric uncertainty cannot be reduced with more training data, a well-trained codec cannot reduce the quantization noise or fully recover the estimated MV fi.
Consider the MV auto-encoder in the DVC framework above. The lossy compression of the motion vectors can be summarized as
ai = MVEncoder(fi) âi = q(ai) = ai + η
f̂i = MVDecoder(âi) (2)
If the MV decoder is implemented with a linear model parameterized by w, the impact of the quantization noise η on the decoded MV is given by
w>âi = w >(ai + η) = w >ai + w >η (3)
Since η is introduced by the quantization operation, we have ‖η‖∞ ≤ 1/2 = ε and it follows that the upper bound of the effects from the quantization operation is given by
‖w>η‖1 ≤ ε ∥∥∥∥w>sign(∂w>ai∂ai )∥∥∥∥ 1 = 1 2 ‖w>sign(w)‖1 (4)
While in practice the MV decoder is usually implemented with a stack of convolution layers and nonlinear activation layers, such as leaky ReLUs, the transformation of the MV decoder may be too linear to reject the quantization noise (Goodfellow et al., 2014).
We visualize the aleatoric uncertainty by adding a small perturbation ‖η0‖ ≈ 0.2 · ‖η‖∞ to the bit stream (see Appendix A.2) and the impact on the decoded MVs is depicted in Fig. 2d. It shows that there is more aleatoric uncertainty in regions where the motion is complicated and the impact due to the aleatoric uncertainty is nontrivial.
Epistemic uncertainty. Due to limited observed data during training, epistemic uncertainty accounts for the uncertainty in the estimated motion vectors. Motion vectors near the object boundaries and occluded regions tend not to be estimated well, and warping erroneous motion vectors would propagate errors to the residual coding. We may roughly visualize such uncertainty by optimizing
an motion estimation network with regards to the mean squared errors (MSE) between the current frame xi and the warped frame
L = MSE(xi,BilinearWarp(x̂i−1, fi)) (5)
We depict the results in Fig. 2e. Since the motion estimation is optimized to minimize the MSE, regions where the MSE is large are likely to have a larger epistemic uncertainty and the corresponding motion vectors cannot be estimated well given the limited training data.
Ideally, we could save MV bits by not encoding MVs that are not estimated well and save residual bits by not warping MVs that would not help to reduce residuals. This is often difficult to implement as an end-to-end trained deep neural network. In the next section, we will show that with the help of multi-head decoders, the model could learn to exploit available information in the bit stream and handle those predictions with larger uncertainty.

heading: 3.2 MULTI-HEAD DECODER
text: Our proposed multi-head decoder structure decodes multiple groups of motion vectors (MV) for motion compensation and multiple groups of residuals for final reconstruction. The multi-head MV decoder and multi-head residual decoder are depicted in Fig. 1b. Take the multi-head MV decoder as an example. The MV decoder base first decodes MV feature from the quantized MV bitstream âi. Then hmv groups of MVs, denoted by {f̂ki | k = 1, . . . , hmv}, are decoded from the MV feature with respective MV decoder heads. We obtain hmv warped frames {x̃ki | k = 1, . . . , hmv} by bilinearly warping each f̂ki on the reference frame x̂i−1. The hmv warped frames are then concatenated for motion compensation and retained for final reconstruction.
Many previous ensemble-based models train an ensemble of stand-alone neural networks (Szegedy et al., 2015; Abbasi & Gagné, 2017; Wang et al., 2020). While they can outperform the single-model baseline by a wide margin, the number of parameters are greatly increased, as well as the inference complexity. Lee et al. (2015) proposed to share backbone parameters with TreeNets, but the models achieve the best performance when very few layers are shared. In our multi-head decoder structure, each decoder branch shares most of the convolution layers, making each decoder head light weight. This design effectively improves the overall performance with negligible complexity increase (see Section 4.4).
The ensemble of decoded MVs can be represented by an equally weighted Gaussian mixture model given by
f̂i ∼ 1
hmv hmv∑ k=1 N (f | f̂ki ,Σki ), Σki = [ σki,x 0 0 σki,y ] (6)
where σki,x and σ k i,y are the variance in x and y directions respectively. The mean and variance of the Gaussian mixture model are respectively (see Appendix A.3)
E[f̂i] = µf̂i = 1
hmv hmv∑ k=1 f̂ki ; σ 2 f̂i,x = E[f̂2i,x]− µ2f̂i = 1 hmv hmv∑ k=1 (( σki,x )2 + ( f̂ki,x )2) − µ2 f̂i (7)
Empirically, we can visualize the predictive uncertainty represented by this ensemble model with the variance of the Gaussian mixture model by setting (σki,x) 2 = (σki,y) 2 = 1, which gives
σ2 f̂i,x
= 1
hmv hmv∑ k=1 ( f̂ki,x )2 −
( 1
hmv hmv∑ k=1 f̂ki
)2 + 1 (8)
The predictive uncertainty for the first two frames in the BasketballDrill sequence is depicted in Fig. 2e. We can see that the predictive uncertainty properly capture both the aleatoric uncertainty and the epistemic uncertainty shown in Fig. 2: the basketball has large aleatoric uncertainty due to rapid motion and the object boundaries have large epistemic uncertainty.
Relation to scale-space flow. Agustsson et al. (2020) estimated a scale field gz besides the 2- dimensional optical flow (gx, gy). We may represent the decoded MV with a multivariate Gaussian
distribution given by
gmv ∼ N ((gx, gy)>,Σ), Σ = [ gz 0 0 gz ] (9)
and the scale-space warp gives a weighted mean of the warped value obtained from gmv . The Gaussian mixture prediction from our proposed multi-head decoder can represent a more diverse predictive uncertainty distribution than the single Gaussian distribution in the scale-space flow. It has been shown in Garipov et al. (2018); Fort et al. (2019) that ensemble models produce diverse results by learning different modes of the function, rather than interpolating around a given mean in the output space. It should be noted that while we only regress the mean of MV predictions in each ensemble branch and model the predictive uncertainty with the variance of the Gaussian mixture prediction, the multi-head decoder can easily be extended to an ensemble of scale-space flows.

heading: 3.3 ENSEMBLE-AWARE TRAINING
text: Intuitively, diversity is a key factor for ensemble models. Ensemble members similar in the parameter space are unlikely to provide any more useful information than their single-model counterpart. Krogh & Vedelsby (1995) proved the bias-variance trade-off in ensemble, E = Ē − Ā, which suggested that the inherent variance is the key for the ensemble models to be effective and we should encourage the diversity among the ensemble members.
In the previous literature, multiple approaches are considered, including random initialization, bagging, and boosting. Randomly initializing the model parameters is a simple but effective approach to induce randomness and is quite suitable for deep ensembles (Lee et al., 2015). Bagging trains ensemble members on independently drawn examples with bootstrap sampling. Lee et al. (2015) showed that bagging may harm the model performance since each model may see only 63% of the available data and would perform poorly when there is high correlation inherent in the data (Bartlett et al., 1998). Boosting generates the ensemble models sequentially and can be quite time-consuming in terms of neural networks.
For deep compression models with our proposed multi-head decoder structure, there is a single model with multiple parallel ensemble branches. Bagging or boosting would not work for such network design since the model is trained on randomly sampled data but different ensemble branches would see the same data. Therefore, we choose to randomly initialize the network parameters and initial experiments show the effectiveness. To further encourage the diversity among the ensemble members, we propose an ensemble-aware loss for deep ensembles to induce additional randomness.
Given h decoded outputs from the multi-head decoder, we obtain h predictions (motioncompensated predictions or final reconstructions) x̄t for t = 1, . . . , h. The ensemble-aware loss is given by
Lensemble-aware(x, x̄1, . . . , x̄h) = h∑
t=1
1 H ×W ∑
1≤i≤H ∑ 1≤j≤W min t ‖x̄ti,j − xi,j‖22 (10)
and the minimum can be relaxed to choosing the smallest k out of the h norms. Our ensembleaware loss encourage diversity between heads by relaxing the magnitude of the loss, and at the same time ensure performance of individual heads by updating every head at every iteration. Since the gradients propagated to each head depend on the head with the minimal MSE, rather than the MSE of the head itself, we can effectively induce randomness and therefore encourage diversity. Although the oracle set loss proposed in Lee et al. (2015) helps to encourage diversity and boost the oracle accuracy, it would significantly harm the performance of individual ensemble members since each head only sees a small portion of all training data. Experiments on learned video compression also showed that adopting the oracle set loss will decrease the overall performance. Instead, our ensemble-aware loss can effectively encourage diversity among ensemble members, and at the same time, each individual member sees all training data and achieves reliable performance.

heading: 3.4 ADVERSARIAL TRAINING WITH FGSM
text: Adversarial examples (Szegedy et al., 2013) are training samples with a small but non-random perturbation that are misclassified by neural networks with high confidence. Goodfellow et al. (2014)
proposed the fast gradient sign method (FGSM) that applies linear but intentionally worst-case perturbation to the training samples, which is given by
η = · sign(∇xJ(θ, x, y)) (11)
where J(θ, x, y) is the cost function, and controls the norm of the perturbation. This adversarial training strategy has been shown to boost the image classification performance, as well as improve the model’s robustness to adversarial examples. Lakshminarayanan et al. (2017) interpreted FGSM as an efficient solution to smooth the predictive distributions by increasing the likelihood of the target around an -neighborhood of the observed training samples.
We find adversarial training with FGSM closely related to learned lossy compression and an effective approach to improve the performance of learned video codecs. In transform coding, we want the latent representation to be as smooth as possible, since after quantization, all latent representations in the -neighborhood, {â + η | ‖η‖∞ < }, corresponds to the same decoded output. Learning a smooth latent representation would help to make the outputs more robust to quantization noise. Although this could be a natural result of an end-to-end optimized video codec, the experimental results show that FGSM can effectively improve the rate-distortion performance.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP
text: Model architecture. Our base model architecture follows the design in Lu et al. (2020b) and we use auto-regressive and hierarchical priors for both the motion vector and residual compression. In order to optimize the model in an end-to-end manner, we need to relax the bits estimation since quantizing the latent bits would make the gradients zero almost everywhere. Following Ballé et al. (2016), we substitute the quantization operation with additive uniform noise during training and perform actual quantization during inference. The structure of multi-head decoder modules are depicted in Figure 1b and more technical details on the implementation are available in Appendix A.1.
Training datasets. Our model is trained on 64,612 video sequences from the training part in Vimeo90K settuplet dataset (Xue et al., 2019). Each video clip has 7 frames with a resolution of 448×256. During training, we randomly crop the video sequences into 256×256 pixels. Given two successive frames from a random sequence, we treat the first frame as the reference frame and our model is trained to minimize the rate-distortion cost of encoding and decoding the second frame.
Training details. In our experiments, we adopt the progressive training strategy and warm up the inter-coding module for 150,000 steps with the ensemble-aware motion compensation loss in Equation 10. Then the model is end-to-end optimized with the rate-distortion loss given by
LRD = (Rw(ŵi) +Rz(ẑi)) + λ ·D(xi, x̂i) (12)
where Rw(ŵi) and Rz(ẑi) represent the numbers of bits used to encode the motion vectors and the residual, D(Fi, F̂i) measures the distortion in mean squared error MSE or multi-scale structural similarity MS-SSIM (Wang et al., 2003), and λ is the hyperparameter controlling the trade-off. Four models are trained with different quality rates by setting λ = 256, 512, 1024, 2048. We use the AdamW optimizer (Loshchilov & Hutter, 2017) with an initial learning rate of 1 × 10−4, which is then decreased to 1× 10−5. Each model is trained on one NVIDIA V100 GPU.

heading: 4.2 QUANTITATIVE RESULTS
text: To show the effectiveness of our proposed uncertainty-aware model, we test our model on the first 100 frames from video sequences in HEVC (Sullivan et al., 2012) with GoP size 10, and the first 120 frames from sequences in UVG (Mercat et al., 2020), MCL-JCV (Wang et al., 2016) with GoP size 12. To balance the trade-off between complexity and performance, we choose hmv = hres = 4. In order to build the best learning-based video codec, we adopt the state-of-the-art image compression model (Cheng et al., 2020) for intra frame coding. In order to fairly compare the performance, we test the DVC and DVC Pro with the same same intra frame model, denoted as DVC (cheng2020) and DVC Pro (cheng2020). For some of the previous state-of-the-art deep video compression methods that are not available, such as HU ECCV20 (Hu et al., 2020) and LU ECCV20 (Lu et al., 2020a),
we use the public results reported in Hu (2020). We calculate the BD-rate (Bjøntegaard, 2001) of different learning-based video compression models using x.265 as the anchor model and the results on HEVC, UVG, MCL-JCV are reported in Table 1. We also plot the RD curves of different codec models in Fig. 3. More RD curve plots are available in Appendix A.4.
From the results reported in Table 1 and Fig. 3, we could see that our proposed model can effectively save bits compared to our strong baseline and outperform previous state-of-the-art learning-based video codecs by a wide margin in all testing datasets.

heading: 4.3 QUALITATIVE RESULTS
text: In Fig. 2 we visualize the aleatoric uncertainty and epistemic uncertainty in the first two frames of the BasketballDrill sequence, as well as the predictive uncertainty represented by the multi-head MV decoder. As we can see, the predictive uncertainty is larger in regions where the motion cannot be estimated well or too complicated to encode. Our uncertainty-aware model learns to effectively represent the underlying uncertainty with an ensemble of decoded MVs and such uncertainty is retained until the final reconstruction. More visualizations of the unsupervised predictive uncertainty can be found in Appendix A.6.

heading: 4.4 ABLATION STUDY
text: Effectiveness of various proposed modules. We evaluate the effectiveness of multi-head decoders, ensemble-aware loss, and adversarial training with FGSM by running ablation experiments on the first 30 frames from all sequences in the HEVC dataset. We adopt the short training strategy for fast experimentation. The RD curves are presented in Fig. 4a and the BD-rates are reported in A.8. MH-MV and MH-Res stand for multi-head MV decoder and multi-head residual decoder, and EA-L refers to training with ensemble-aware loss. The results show the effectiveness of the multi-head decoder modules and the training strategies.
Ablation study on the number of decoder heads. We investigate the model’s performance with different number of heads in the multi-head decoder on HEVC sequences. As shown in Fig. 4b, we train eight models with hmv = 1, 2, 3, 4, 5, 6, 7, 8 where hmv = 1 is the baseline without ensembling. We see that the multi-head decoder module is effective even with only two decoder heads and the performance is improved with more decoder heads. Quantitative results are reported in A.8.
Complexity analysis. Most previous deep ensembles train multiple stand-alone models (Szegedy et al., 2015; Abbasi & Gagné, 2017; Wang et al., 2020) or share very few shallow layers (Lee et al., 2015) for the model to be effective. With an ensemble of 6 models, the inference complexity (in MACs) and model size (in number of parameters) easily increase by 500%. With the help of multihead decoders, our ensemble-based model share the backbone features and achieve superior results with limited complexity increase. For one extra MV and residual decoder added, the complexity increases by approximately 7% and just 1% in the model size. For the largest model we consider, where hmv = hres = 8, there is only a 48% increase in complexity and 10% in model size.

heading: 5 DISCUSSION
text: In this paper, we studied the aleatoric uncertainty and epistemic uncertainty in deep learning-based video compression and proposed to utilize an ensemble of intermediate predictions to represent the predictive uncertainty at decoding time. With multi-head decoders, our model can properly model the uncertainties in the decoded MVs or residuals, which would effectively avoid propagating underlying errors in a single deterministic prediction.
We investigated the performance of our uncertainty-aware decoding module and proposed a novel ensemble-aware loss to boost the diversity among the parallel ensemble branches in a single model. We also proposed to incorporate adversarial training for learning-based video codecs. Experimental results show the effectiveness of our approach.
Compared to one-stage learning-based video compression models, such as those based on 3D autoencoders (Pessoa et al., 2020; Habibian et al., 2019), two-stage motion compensation-based models can decode high-quality frames with low-latency. However, intermediate predictions in these two-stage pipelines are not always accurate and erroneous predictions could severely harm the performance of later stages, especially for out-of-distribution data. Therefore, it is critical to represent the predictive uncertainty and our proposed multi-head decoder is a simple but very effective approach to capture such uncertainty. Future directions could involve modules on the encoder side to model and propagate the uncertainty to the decoders for an end-to-end uncertainty-awareness.
REPRODUCIBILITY STATEMENT
Key results in this paper are reproducible and in this section, we will refer to details helpful to reproduce the results. The implementation of our model are modified from the codes released by Bégaint et al. (2020) and Hu (2020). We propose the multi-head decoders in Section 3.2 and can be easily implemented by following the flow charts in Figure 1b and technical details in Appendix A.1. In Section 4.1 we introduce the dataset we use to train the model. The Vimeo-90K settuplet dataset (Xue et al., 2019) is publicly available and we fit our model on the training part of the dataset. Also in Section 4.1 we summarize our training strategy, which includes an ensemble-aware loss for model warm-up and then minimizing the rate-distortion cost until convergence. The motivation and formality of the ensemble-aware loss can be found in Section 3.3. This loss can be easily implemented in PyTorch. Since directly clamping the loss gives zero gradients almost everywhere, we create a custom autograd function to forward the gradients before and after the clamp operation and properly update the weights of each head.

heading: A APPENDIX
text: A.1 IMPLEMENTATION OF MULTI-HEAD DECODERS
As depicted in Figure 1b, multi-head decoders consist of a decoder backbone and multiple decoder heads. The decoder heads are light-weight and include two convolution layers with one leaky ReLU in between. For multi-head MV decoder, the decoder backbone first decodes MV feature from the MV bitstream âi, and then h MVs are decoded with respective MV decoder heads. From the h decoded MVs and the previous decoded frame, we obtain h MC predictions with bilinear warping. The h MC predictions are concatenated with the previous decoded frame and sent to the Prediction Refine Net, from which we get h refined MC predictions. For multi-head residual decoder, the decoder backbone first decodes residual feature from the residual bitstream b̂i, and then h residuals are decoded with respective residual decoder heads. From the h decoded residuals and the h refined MC predictions, we obtain h reconstructions. The h reconstructions are concatenated with h refined MC predictions and sent to the Reconstruction Refine Net, from which we get 1 refined reconstruction as the final decoded frame. All modules in our model, including decoder backbone, decoder head, and refine nets, are implemented with neural networks and optimized in an end-to-end manner.
A.2 VISUALIZING THE ALEATORIC UNCERTAINTY
To visualize the aleatoric uncertainty introduced from the quantization operation, we conduct preliminary experiments. We add a small perturbation η0 to the quantized bit stream and obtain ŵ′i = ŵi + η0. The perturbation η0 is only added to positions where the quantization gap is at least 0.1 and corresponds to only 20% of the size of the perturbation gap. We model the aleatoric uncertainty with the L1 norm between the two optical flows decoded from bit streams that differs in a small perturbation. Results on the first two frames of the BasketballDrill sequence are shown in Figure 2d. As we can see, the aleatoric uncertainty is not uniform across the whole image. Instead, there are more aleatoric uncertainty around the object boundaries and regions where the motion is large. While by definition, such aleatoric uncertainty cannot be reduced away, blindly assuming the optical flows to be accurate would lead to larger intermediate residual errors and cost more bits in the residual coding.
A.3 MV PREDICTION AS A GAUSSIAN MIXTURE MODEL
The multi-head MV decoder predicts an ensemble of motion vectors {f̂ki | k = 1, . . . , hmv} from the MV bit stream and we can represent the prediction as an equally-weighted Gaussian mixture model given by
f̂i ∼ 1
hmv hmv∑ k=1 N (f | f̂ki ,Σki ), Σki = [ σki,x 0 0 σki,y ] (13)
Since the Gaussian distributions are equally-weighted, the mean of the mixture distribution is
E[f̂i] = µf̂i = 1
hmv hmv∑ k=1 f̂ki (14)
and the variance of the mixture distribution in each direction is
σ2 f̂i,x = E[f̂2i,x]− µ2f̂i
= 1
hmv hmv∑ k=1 E [( f̂ki,x )2] − µ2 f̂i
= 1
hmv hmv∑ k=1 (( σki,x )2 + ( f̂ki,x )2) − µ2 f̂i (15)
In order to visualize the predictive uncertainty represented by this ensemble model, we set (σki,x) 2 = (σki,y) 2 = 1 and obtain
σ2 f̂i,x
= 1
hmv hmv∑ k=1 ( 1 + ( f̂ki,x )2) − µ2 f̂i
= 1
hmv hmv∑ k=1 ( f̂ki,x )2 −
( 1
hmv hmv∑ k=1 f̂ki
)2 + 1 (16)
A.4 QUANTITATIVE RESULTS
A.5 MODEL PERFORMANCE USING DIFFERENT I-FRAME MODELS
In this section, we investigate the performance of our proposed model using different I-frame models on HEVC Class B and UVG. In Section 4.2, we report the experimental results of our proposed model using Cheng2020 (Cheng et al., 2020) as the I-frame model. Compared to DVC Pro, our proposed model with multi-head decoders can effectively reduce the bpp while maintaining the same reconstruction quality. As shown in Figure 3, this brings our model to a range of lower bitrates. In order to investigate our model performance in higher bitrates, we retrain our model with λ = 4096 and use VTM (VTM) as the I-frame model. The performance of our model and DVC Pro using VTM are reported in Figure 9. We could see that using VTM instead of cheng2020 as the I-frame model brings both our proposed model and DVC Pro to higher bitrates and similar conclusions can be made. Our proposed model can still outperform previous state-of-the-art models in higher bitrates and the multi-head decoders can bring significant bitrate savings regardless of the I-frame model used or the range of bitrates.
A.6 VISUALIZATION OF UNSUPERVISED PREDICTIVE UNCERTAINTY
A.7 QUALITATIVE RESULTS OF DECODED FRAMES
In this section, we show the qualitative examples of decoded frames for subjective comparison. We compare the performance of our proposed model with the baseline DVC Pro to show the strength and weakness of multi-head decoders. In sequence BQTerrace from HEVC B, our proposed model achieves a bitrate saving of 20.5% over DVC Pro. Sequence 20 from MCL-JCV is the only sequence where our model failed to outperform DVC Pro with a bitrate increase of 2.2%.
A.8 DETAILED ABLATION STUDY RESULTS
To show the effectiveness of various proposed modules and to compare between models with different number of decoder heads, we train multiple models on the Vimeo-90K dataset with the fast training strategy and test the models on the first 30 frames in each HEVC sequence. We measure
the model performance with BD rates and the quantitative results are reported in Table 2 and Table 3 respectively.

Review summaries:
This paper aims to capture the predictive uncertainty and proposes an ensemble-based video compression approach. In the proposed approach, an ensemble-aware loss is also introduced to seek diversity among ensemble members. Experimental results show its performance.

This paper proposes an ensemble-based video compression model to capture the predictive uncertainty of intermediate predictions. A loss is constructed to encourage diversity between ensemble members, and the paper investigates the benefit of incorporating adversarial training in video compression. The experimental result shows that the proposed model can save more than 20% compared to DVC Pro. 

This paper works on end-to-end deep learning video compression. The authors study the inherent uncertainty and accordingly propose a so-called ensemble approach which is in effect multi-head decoder. A so-called ensemble-aware loss is proposed to encourage the diversity between ensemble members. Further, adversarial training is incorporated. Experiments show that the proposed model outperforms previous state-of-the-art models such as DVC Pro (Lu et al., 2020b) and Lu et al. (2020a), and the ablation study proves the effectiveness of each module. 

This paper presents an uncertainty-aware video compression framework with an ensemble of MV/residual decoders. To train the network, they use the ensemble-aware loss function which minimizes the loss function for the k best predictions. Moreover, this paper also analyzes underlying uncertainties in video compression, i.e., aleatoric and epistemic uncertainties, with visualization techniques for those uncertainties. Experimental results show that the proposed method can improve the compression performance by about 20% while not significantly increasing the computational complexity.