Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set

1 INTRODUCTION . Neural compression methods have enormous potential to improve the efficiency of video coding . With video constituting the large majority of internet traffic , this has significant implications for the internet at large ( Sandvine , 2019 ) . State-of-the-art algorithms model each frame by warping the previous one with a neural estimate of the optical flow ( Lu et al. , 2019 ) or scale-space flow ( Agustsson et al. , 2020 ) and adding residuals modeled by another network . Both the optical flow and the residuals are compressed with variational autoencoders . Such neural codecs have recently achieved results on par with popular classical codecs ( Agustsson et al. , 2020 ; Rippel et al. , 2021 ; Pourreza & Cohen , 2021 ) such as H.264 / AVC ( Wiegand et al. , 2003 ) and H.265 / HEVC ( Sullivan et al. , 2012 ) . However , relatively little research has focused on their computational complexity , and matching the rate-distortion performance of H.266 / VVC ( Bross et al. , 2018 ) is still an open problem . Neural video codecs depend critically on generalization : they are developed assuming that a good performance on training and validation datasets translates to a good performance at test time . However , this assumption does not always hold in practice , for instance because of limited training data or imperfect optimization . Domain shift is a particularly challenging problem given the variety in video content and styles ; for instance , neural video codecs trained on natural scene video data often perform poorly on animated sequences ( Agustsson et al. , 2020 ) . Here we address this problem by optimizing the autoencoder — finetuning its weights — for each single video to be compressed at test time . After training on the test set in this way , the finetuned network parameters need to be made available on the receiver side : they are compressed , quantized , and entropy-coded under a suitable prior to the bitstream along with the latent code . Optimizing the compression network for every video in this way relaxes the requirement of training data that closely matches the test distribution as well as the necessary model capacity . It also allows trading off encoding compute for better compression efficiency . The essential idea of instance-adaptive finetuning was recently proposed by van Rozendaal et al . ( 2021 ) . The authors demonstrated the idea on I-frame compression , i. e. compressing a set of similar images , but did not apply it to video sequences yet . In this work we extend the idea to the compression of full videos , in which only periodic key frames are compressed as images and most frames are encoded relative to one or two reference frames . Our method is general and can be applied to various settings and base models . We first demonstrate it in a low-latency setting with I-frames and P-frames using a scale-space flow architecture ( Agustsson et al. , 2020 ) as base model . Next we show its performance in a B-frame setting , using the base model proposed by Pourreza & Cohen ( 2021 ) . On the UVG-1k , HEVC class-B , and Xiph-5N datasets , our new instance-adaptive video compression codec yields BD-rate savings of 17 to 26 % over the respective base models and 5 to 44 % over the popular ffmpeg ( x265 ) implementation of the H.265 codec ( FFmpeg ; VideoLAN , b ) . In addition , instance-adaptive finetuning lends itself to a possible reduction in model size , because a smaller network may suffice to model a single instance . We show that in this framework smaller models can still outperform most neural codecs while reducing the computational complexity of the decoder by 72 % . Overall , we demonstrate that unlike other neural video codecs , our method can trade off encoding compute against compression performance , and that this trade-off is more effective than for classical codecs . 2 RELATED WORK . Neural video compression The standard framework used by most neural compression codecs ( either implicitly or explicitly ) is that of the variational ( Kingma & Welling , 2014 ) or compressive ( Theis et al. , 2017 ) autoencoder . An encoder ( or approximate posterior ) qφ ( z|x ) maps a data point x to a latent representation z . This latent is transmitted to the decoding party by means of entropy coding under a latent prior pθ ( z ) . The receiver can then reconstruct the sample with a decoder ( or likelihood ) pθ ( x|z ) . The encoder , prior , and decoder are neural networks , parameterized by weights φ , θ as indicated by the subscripts . These models are trained by minimizing the rate-distortion loss LRD ( φ , θ ) = Ex∼p ( x ) [ β Ez∼qφ ( z|x ) [ − log pθ ( z ) ] ︸ ︷︷ ︸ Rz +Ez∼qφ ( z|x ) [ − log pθ ( x|z ) ] ︸ ︷︷ ︸ D ] , ( 1 ) which combines a distortion metric D and a rate term Rz that approximates the bitrate necessary to transmit the latent code z . Up to a constant entropy term H [ qφ ] , this loss equals the β-weighted VAE loss ( Habibian et al. , 2019 ; Higgins et al. , 2017 ) . Much research has focused on the design of efficient network architectures for neural video codecs , in particular with respect to the temporal structure . While some earlier works were based on 3D convolutions ( Habibian et al. , 2019 ; Pessoa et al. , 2020 ) , recent models use optical flow and residual modeling to exploit similarities between frames . The latter class of approaches can be divided into predictive or P-frame temporal modeling , where the model for each frame xt is conditional on the previous frame ( s ) ( Agustsson et al. , 2020 ; Chen et al. , 2019 ; Liu et al. , 2020 ; Lombardo et al. , 2019 ; Lu et al. , 2019 ; Rippel et al. , 2019 ; Golinski et al. , 2020 ; Rippel et al. , 2021 ; Hu et al. , 2020 ; Yang et al. , 2020b ) , and bidirectional or B-frame modeling , where each frame is conditioned on past as well as future reference frames ( Cheng et al. , 2019 ; Choi & Bajić , 2019 ; Djelouah et al. , 2019 ; Wu et al. , 2018 ; Park & Kim , 2021 ; Pourreza & Cohen , 2021 ; Yang et al. , 2020a ) . Agustsson et al . ( 2020 ) generalize optical flow to scale-space flow by adding dynamic blurring to the warping operation , improving the modeling of uncertainty and leading to state-of-the-art results . Instance-adaptive compression Rate-distortion autoencoders are trained by minimizing the RD loss in Eq . ( 1 ) over a training dataset D. This approach relies on the assumptions that the model is not only able to fit the training data well , but also generalizes to unseen datapoints at inference time . In practice , however , finite training data , limited model capacity , optimization imperfections , and domain shift — differences between train and test distributions — can degrade the performance . This problem can be solved by adapting video codecs to each sequence at test time . Several such methods have been proposed , see Appendix A for an in-depth overview that also covers image compression . These works can be roughly categorized into encoder-only finetuning , limited decoder finetuning , and hybrid classical-neural codecs . Works in the first category ( e. g. Lu et al . ( 2020 ) ) optimize the encoder parameters φ for each data point . Such an update does not have to be communicated to the receiver side . This approach alleviates the generalization problem for the encoder , but does not solve it for the decoder and prior . We will demonstrate later that encoder-only finetuning leads to a limited improvement in the compression performance . The second category of works also adapt parts of the prior and decoder for each video instance . Since these modules need to be known on the receiver side , such an update needs to be signalled in the bitstream . Depending on the implementation , this may increase the bitrate substantially . Aytekin et al . ( 2018 ) ; Zou et al . ( 2020 ) ; Wang et al . ( 2021 ) address this by only allowing limited changes to the decoder-side models , for instance choosing one out of a number of fixed decoder networks , thus limiting the potentialRD gains of these methods . Finally , the third category of works propose a hybrid approach , where a video is first compressed by a classical codec and a finetuned enhancement network is added to the bitstream ( He et al. , 2020 ; 2021 ; Klopp et al. , 2020 ; 2021 ; Liu et al. , 2021 ) . Although these methods have shown great performance at low bitrates , they tend to perform less well at higher bitrates , and do not benefit from developments in end-to-end neural compression . van Rozendaal et al . ( 2021 ) suggest a method to adapt the full model to a single datapoint . The key idea is that the parameter updates δ = θ − θD , where θD are the global model parameters resulting from training onD , can be be transmitted very efficiently . After discretization , the quantized updates δ̄ are entropy-coded under a parameter update prior p [ δ̄ ] that assigns high probability mass ( and thus low transmission cost ) to zero-updates δ̄ = 0 . This procedure enables a neural codec in which the full model — encoder , decoder , and prior — is finetuned on each data sample at test time by minimizing the loss LInstA ( φ , δ ) = LRD ( φ , θD + δ̄ ) + β ( − log p ( δ ) ) ︸ ︷︷ ︸ Rθ . ( 2 ) The Rθ term reflects the increased bitstream length from coding the parameter updates δ̄ . In van Rozendaal et al . ( 2021 ) , this approach was successfully demonstrated on I-frame compression , essentially compressing a set of images with similar content . In this work we adapt it to the compression of full video sequences , as we will describe in the following section . 3 INSTANCE-ADAPTIVE VIDEO COMPRESSION . We now introduce our instance-adaptive compression codec for full video sequences , or InstA for short . The key idea is to optimize the parameters of a rate-distortion VAE for each video sequence to be transmitted , and to send the relevant network parameters in a rate-efficient way to the decoder . This approach can be beneficial whenever a standard rate-distortion autoencoder would not generalize well , for instance because of limited training data or domain shift . In addition , instance adaptation allows us to use smaller models while maintaining most of the compression performance . While encoding the network parameters to the bitstream increases the length of the code , we describe a compression scheme in which this cost is negligible , especially when amortized over many frames of a video sequence . Our approach is agnostic about the base model and we demonstrate it on two different architectures : a scale-space flow model in the low-latency setting and the B-EPIC model in the B-frame setting . In the following , we first describe the base models , before describing how InstA compresses and decompresses video sequences . Low-latency base model : Scale-space flow ( SSF ) We first focus on the low-latency setting , which we define as having only access to current or previous frames when decoding a frame . As a base model for this setting , we use the scale-space-flow architecture introduced by Agustsson et al . ( 2020 ) . Any video sequence is first split into groups of pictures ( GoP ) . The first frame in each GoP is modeled as an image without any dependency on previous frames , i. e. as an I-frame . All other frames are P-frames , modeled as xi = Scale-Space-Warp ( xi−1 , gi ) + ri , ( 3 ) where xi−1 is the reconstructed previous frame , gi is the estimated scale-space flow field , ri is the estimated residual , and the scale-space warping operation performs optical-flow warping of the previous frame with a dynamic , position-dependent amount of Gaussian blur . The I-frame images , P-frame scale-space flow gi , and P-frame residuals ri are compressed with separate hyperprior models ( Ballé et al. , 2018 ) , a hierarchical variational autoencoder architecture . We largely follow the architecture choices of Agustsson et al . ( 2020 ) and refer to this model as SSF18 after its number of decoder-side parameters ( in millions ) . In Appendix C we describe the architecture in detail and highlight the differences to Agustsson et al . ( 2020 ) . Smaller low-latency base models As our instance-adaptive approach is based on finetuning the compression model on a low-entropy “ dataset ” ( a single video sequence ) , we hypothesize that it does not require the full expressivity of the computationally complex SSF18 model . We propose three alternative scale-space flow architectures with a reduced computational complexity . In Tbl . 1 we list the parameter counts and the number of multiply-accumulate ( MAC ) operations required for decoding a sequence , which are reduced by 43–84 % compared to the SSF18 baseline . The architectures of the four SSF models are described in detail in Appendix C. B-frame base model : B-EPIC We also consider a less constrained setting , in which frames can be compressed as B-frames as well , i. e. using both a previous frame and a future frame as reference points . This flexibility allows for even more efficient compression and is used for instance for ondemand video streaming . As a base model we choose the B-EPIC architecture ( Pourreza & Cohen , 2021 ) . Again , the video is split into GoPs . The very first frame of a video is modeled as an Iframe , while the last frame in each GoP is a P-frame , using the last frame of the preceeding GoP as reference ; both I-frame and P-frames are compressed as in the SSF model . Any other frame xi is modeled as a B-frame : it is assigned a past reference frame xj ( j < i ) and a future reference frame xk ( k > i ) ; an off-the-shelf Super-SloMo frame interpolator ( Jiang et al. , 2018 ) is used to interpolate between these two reference frames . The interpolated frame is then used as a basis for scale-space flow warping : xi = Scale-Space-Warp ( Super-SloMo ( xj , xk ) , gi ) + ri . ( 4 ) The interpolated frame provides a more useful starting point for optical-flow warping than the previous frame , e. g. because the combination of past and future knowledge may avoid occlusion effects . Again , I-frames , optical flow , and residuals are modeled with three separate hyperprior models and the model is trained on the RD loss in Eq . ( 1 ) . We only consider a single configuration and use hyperparameters and checkpoints from Pourreza & Cohen ( 2021 ) . This model has 38.5 million parameters , 23.2 million of which are on the decoder side . Encoding a video sequence Our procedure conceptually follows van Rozendaal et al . ( 2021 ) . For completeness we shall describe the full method here . A video sequence x is compressed by : 1 . Finetuning the model parameters ( θ , φ ) of the base model on the sequence x using Eq . ( 2 ) , 2. computing the latent codes z ∼ qφ ( z|x ) , 3. parameterizing the finetuned decoder and prior parameters as updates δ = θ − θD , 4. quantizing latent codes z and network updates δ , and 5. compressing the quantized latents z̄ and updates δ̄ with entropy coding to the bitstream . To finetune a pretrained base compression model , we start with the global model with parameters ( θD , φD ) and minimize the rate-distortion loss given in Eq . ( 2 ) — but only over the single video sequence x . This modified rate-distortion loss explicitly includes the bitrate required to send model updates δ under an update prior p ( δ ) . We compute the regularizing Rθ loss with the unquantized updates δ , but use the quantized parameter updates θD + δ̄ to calculate the D and Rz loss terms , using a straight-trough estimator ( Bengio et al. , 2013 ) in the backward pass . Sending the updated network parameters of course adds to the length of the bitstream . This is even the case when finetuning does not lead to changed parameters ( δ̄ = 0 ) . Given the large size of the neural model we consider , it is therefore essential to choose an update prior that assigns a large probability mass to zero-updates δ̄ = 0 . This allows the network to transmit trivial updates at a negligible rate cost , while giving it the freedom to invest bit cost in non-trivial parameter updates that improve the performance substantially . We use a spike-and-slab prior ( Johnstone & Titterington , 2009 ; van Rozendaal et al. , 2021 ) , a mixture model of a narrow and a wide Gaussian distribution given by p ( δ ) = N ( δ|0 , σ21 ) + αN ( δ ∣∣0 , s21 ) 1 + α , ( 5 ) where the “ slab ” component with variance σ2 keeps the bitrate cost for sizable updates down , and the “ spike ” component with the narrow standard deviation s σ ensures cheap zero-updates . The mixing weight α is a tunable hyperparameter . At the beginning of the finetuning procedure , our neural model is equal to the global model . Because of the spike-slab update prior , the rate cost is only marginally increased and the compression performance essentially equal to the global model . During finetuning , the rate-distortion performance gradually improves , giving us an anytime algorithm that we can stop prematurely to get the best compression performance within a given encoder compute budget . After the model has converged or a compute budget has been exhausted , we use the finetuned encoder qφ to find the latent code z corresponding to the video sequence x . Both z and the updates to prior and decoder δ , which will be necessary to decode the video sequence , are quantized . To discretize the updates δ , we use a fixed grid of n equal-sized bins of width t centered around δ = 0 . We choose n such that p ( −n/t < δ < n/t ) ≈ 1− and clip values at the tails . The quantization of z is analogous , except that we use a bin width of t = 1 and do not clip the values at the tails ( in line with Ballé et al . ( 2018 ) ) . Finally , we write the quantized updates δ̄ and quantized codes z̄ to the bitstream . We use entropy coding under the quantized update prior p ( δ̄ ) and finetuned prior pθD+δ̄ ( z̄ ) , respectively . For latent codes in the tail region we use Exp-Golomb coding ( Wiegand et al. , 2003 ) . Decoding a video sequence The receiver first decodes the prior and decoder updates δ̄ from the bitstream . Once θ̄ = θD + δ̄ is known on the decoder side , the latents z are decoded with the prior pθ̄ ( z ) and the video sequence is reconstructed with the decoder pθ̄ ( x|z ) following standard VAE protocol . The only overhead on the decoder side is therefore the initial decoding of δ̄ . While this results in a small delay before the first frame can be decoded , in practice this delay is very short ( below 0.2 seconds in our experiments ) .

Summary: This paper tried to close the domain gap between the training and testing dataset for the learned video compression. An instance-adaptive video compression approach is proposed by updating the model parameters of the video codecs and the optimal parameters are transmitted to the decoder. Besides, the authors modeled the updated model parameters under a mixture prior model, which further reduces the bit cost. Experimental results on several datasets demonstrated the effectiveness of this approach.