Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set

1 INTRODUCTION . Neural compression methods have enormous potential to improve the efficiency of video coding . With video constituting the large majority of internet traffic , this has significant implications for the internet at large ( Sandvine , 2019 ) . State-of-the-art algorithms model each frame by warping the previous one with a neural estimate of the optical flow ( Lu et al. , 2019 ) or scale-space flow ( Agustsson et al. , 2020 ) and adding residuals modeled by another network . Both the optical flow and the residuals are compressed with variational autoencoders . Such neural codecs have recently achieved results on par with popular classical codecs ( Agustsson et al. , 2020 ; Rippel et al. , 2021 ; Pourreza & Cohen , 2021 ) such as H.264 / AVC ( Wiegand et al. , 2003 ) and H.265 / HEVC ( Sullivan et al. , 2012 ) . However , relatively little research has focused on their computational complexity , and matching the rate-distortion performance of H.266 / VVC ( Bross et al. , 2018 ) is still an open problem . Neural video codecs depend critically on generalization : they are developed assuming that a good performance on training and validation datasets translates to a good performance at test time . However , this assumption does not always hold in practice , for instance because of limited training data or imperfect optimization . Domain shift is a particularly challenging problem given the variety in video content and styles ; for instance , neural video codecs trained on natural scene video data often perform poorly on animated sequences ( Agustsson et al. , 2020 ) . Here we address this problem by optimizing the autoencoder — finetuning its weights — for each single video to be compressed at test time . After training on the test set in this way , the finetuned network parameters need to be made available on the receiver side : they are compressed , quantized , and entropy-coded under a suitable prior to the bitstream along with the latent code . Optimizing the compression network for every video in this way relaxes the requirement of training data that closely matches the test distribution as well as the necessary model capacity . It also allows trading off encoding compute for better compression efficiency . The essential idea of instance-adaptive finetuning was recently proposed by van Rozendaal et al . ( 2021 ) . The authors demonstrated the idea on I-frame compression , i. e. compressing a set of similar images , but did not apply it to video sequences yet . In this work we extend the idea to the compression of full videos , in which only periodic key frames are compressed as images and most frames are encoded relative to one or two reference frames . Our method is general and can be applied to various settings and base models . We first demonstrate it in a low-latency setting with I-frames and P-frames using a scale-space flow architecture ( Agustsson et al. , 2020 ) as base model . Next we show its performance in a B-frame setting , using the base model proposed by Pourreza & Cohen ( 2021 ) . On the UVG-1k , HEVC class-B , and Xiph-5N datasets , our new instance-adaptive video compression codec yields BD-rate savings of 17 to 26 % over the respective base models and 5 to 44 % over the popular ffmpeg ( x265 ) implementation of the H.265 codec ( FFmpeg ; VideoLAN , b ) . In addition , instance-adaptive finetuning lends itself to a possible reduction in model size , because a smaller network may suffice to model a single instance . We show that in this framework smaller models can still outperform most neural codecs while reducing the computational complexity of the decoder by 72 % . Overall , we demonstrate that unlike other neural video codecs , our method can trade off encoding compute against compression performance , and that this trade-off is more effective than for classical codecs . 2 RELATED WORK . Neural video compression The standard framework used by most neural compression codecs ( either implicitly or explicitly ) is that of the variational ( Kingma & Welling , 2014 ) or compressive ( Theis et al. , 2017 ) autoencoder . An encoder ( or approximate posterior ) qφ ( z|x ) maps a data point x to a latent representation z . This latent is transmitted to the decoding party by means of entropy coding under a latent prior pθ ( z ) . The receiver can then reconstruct the sample with a decoder ( or likelihood ) pθ ( x|z ) . The encoder , prior , and decoder are neural networks , parameterized by weights φ , θ as indicated by the subscripts . These models are trained by minimizing the rate-distortion loss LRD ( φ , θ ) = Ex∼p ( x ) [ β Ez∼qφ ( z|x ) [ − log pθ ( z ) ] ︸ ︷︷ ︸ Rz +Ez∼qφ ( z|x ) [ − log pθ ( x|z ) ] ︸ ︷︷ ︸ D ] , ( 1 ) which combines a distortion metric D and a rate term Rz that approximates the bitrate necessary to transmit the latent code z . Up to a constant entropy term H [ qφ ] , this loss equals the β-weighted VAE loss ( Habibian et al. , 2019 ; Higgins et al. , 2017 ) . Much research has focused on the design of efficient network architectures for neural video codecs , in particular with respect to the temporal structure . While some earlier works were based on 3D convolutions ( Habibian et al. , 2019 ; Pessoa et al. , 2020 ) , recent models use optical flow and residual modeling to exploit similarities between frames . The latter class of approaches can be divided into predictive or P-frame temporal modeling , where the model for each frame xt is conditional on the previous frame ( s ) ( Agustsson et al. , 2020 ; Chen et al. , 2019 ; Liu et al. , 2020 ; Lombardo et al. , 2019 ; Lu et al. , 2019 ; Rippel et al. , 2019 ; Golinski et al. , 2020 ; Rippel et al. , 2021 ; Hu et al. , 2020 ; Yang et al. , 2020b ) , and bidirectional or B-frame modeling , where each frame is conditioned on past as well as future reference frames ( Cheng et al. , 2019 ; Choi & Bajić , 2019 ; Djelouah et al. , 2019 ; Wu et al. , 2018 ; Park & Kim , 2021 ; Pourreza & Cohen , 2021 ; Yang et al. , 2020a ) . Agustsson et al . ( 2020 ) generalize optical flow to scale-space flow by adding dynamic blurring to the warping operation , improving the modeling of uncertainty and leading to state-of-the-art results . Instance-adaptive compression Rate-distortion autoencoders are trained by minimizing the RD loss in Eq . ( 1 ) over a training dataset D. This approach relies on the assumptions that the model is not only able to fit the training data well , but also generalizes to unseen datapoints at inference time . In practice , however , finite training data , limited model capacity , optimization imperfections , and domain shift — differences between train and test distributions — can degrade the performance . This problem can be solved by adapting video codecs to each sequence at test time . Several such methods have been proposed , see Appendix A for an in-depth overview that also covers image compression . These works can be roughly categorized into encoder-only finetuning , limited decoder finetuning , and hybrid classical-neural codecs . Works in the first category ( e. g. Lu et al . ( 2020 ) ) optimize the encoder parameters φ for each data point . Such an update does not have to be communicated to the receiver side . This approach alleviates the generalization problem for the encoder , but does not solve it for the decoder and prior . We will demonstrate later that encoder-only finetuning leads to a limited improvement in the compression performance . The second category of works also adapt parts of the prior and decoder for each video instance . Since these modules need to be known on the receiver side , such an update needs to be signalled in the bitstream . Depending on the implementation , this may increase the bitrate substantially . Aytekin et al . ( 2018 ) ; Zou et al . ( 2020 ) ; Wang et al . ( 2021 ) address this by only allowing limited changes to the decoder-side models , for instance choosing one out of a number of fixed decoder networks , thus limiting the potentialRD gains of these methods . Finally , the third category of works propose a hybrid approach , where a video is first compressed by a classical codec and a finetuned enhancement network is added to the bitstream ( He et al. , 2020 ; 2021 ; Klopp et al. , 2020 ; 2021 ; Liu et al. , 2021 ) . Although these methods have shown great performance at low bitrates , they tend to perform less well at higher bitrates , and do not benefit from developments in end-to-end neural compression . van Rozendaal et al . ( 2021 ) suggest a method to adapt the full model to a single datapoint . The key idea is that the parameter updates δ = θ − θD , where θD are the global model parameters resulting from training onD , can be be transmitted very efficiently . After discretization , the quantized updates δ̄ are entropy-coded under a parameter update prior p [ δ̄ ] that assigns high probability mass ( and thus low transmission cost ) to zero-updates δ̄ = 0 . This procedure enables a neural codec in which the full model — encoder , decoder , and prior — is finetuned on each data sample at test time by minimizing the loss LInstA ( φ , δ ) = LRD ( φ , θD + δ̄ ) + β ( − log p ( δ ) ) ︸ ︷︷ ︸ Rθ . ( 2 ) The Rθ term reflects the increased bitstream length from coding the parameter updates δ̄ . In van Rozendaal et al . ( 2021 ) , this approach was successfully demonstrated on I-frame compression , essentially compressing a set of images with similar content . In this work we adapt it to the compression of full video sequences , as we will describe in the following section . 3 INSTANCE-ADAPTIVE VIDEO COMPRESSION . We now introduce our instance-adaptive compression codec for full video sequences , or InstA for short . The key idea is to optimize the parameters of a rate-distortion VAE for each video sequence to be transmitted , and to send the relevant network parameters in a rate-efficient way to the decoder . This approach can be beneficial whenever a standard rate-distortion autoencoder would not generalize well , for instance because of limited training data or domain shift . In addition , instance adaptation allows us to use smaller models while maintaining most of the compression performance . While encoding the network parameters to the bitstream increases the length of the code , we describe a compression scheme in which this cost is negligible , especially when amortized over many frames of a video sequence . Our approach is agnostic about the base model and we demonstrate it on two different architectures : a scale-space flow model in the low-latency setting and the B-EPIC model in the B-frame setting . In the following , we first describe the base models , before describing how InstA compresses and decompresses video sequences . Low-latency base model : Scale-space flow ( SSF ) We first focus on the low-latency setting , which we define as having only access to current or previous frames when decoding a frame . As a base model for this setting , we use the scale-space-flow architecture introduced by Agustsson et al . ( 2020 ) . Any video sequence is first split into groups of pictures ( GoP ) . The first frame in each GoP is modeled as an image without any dependency on previous frames , i. e. as an I-frame . All other frames are P-frames , modeled as xi = Scale-Space-Warp ( xi−1 , gi ) + ri , ( 3 ) where xi−1 is the reconstructed previous frame , gi is the estimated scale-space flow field , ri is the estimated residual , and the scale-space warping operation performs optical-flow warping of the previous frame with a dynamic , position-dependent amount of Gaussian blur . The I-frame images , P-frame scale-space flow gi , and P-frame residuals ri are compressed with separate hyperprior models ( Ballé et al. , 2018 ) , a hierarchical variational autoencoder architecture . We largely follow the architecture choices of Agustsson et al . ( 2020 ) and refer to this model as SSF18 after its number of decoder-side parameters ( in millions ) . In Appendix C we describe the architecture in detail and highlight the differences to Agustsson et al . ( 2020 ) . Smaller low-latency base models As our instance-adaptive approach is based on finetuning the compression model on a low-entropy “ dataset ” ( a single video sequence ) , we hypothesize that it does not require the full expressivity of the computationally complex SSF18 model . We propose three alternative scale-space flow architectures with a reduced computational complexity . In Tbl . 1 we list the parameter counts and the number of multiply-accumulate ( MAC ) operations required for decoding a sequence , which are reduced by 43–84 % compared to the SSF18 baseline . The architectures of the four SSF models are described in detail in Appendix C. B-frame base model : B-EPIC We also consider a less constrained setting , in which frames can be compressed as B-frames as well , i. e. using both a previous frame and a future frame as reference points . This flexibility allows for even more efficient compression and is used for instance for ondemand video streaming . As a base model we choose the B-EPIC architecture ( Pourreza & Cohen , 2021 ) . Again , the video is split into GoPs . The very first frame of a video is modeled as an Iframe , while the last frame in each GoP is a P-frame , using the last frame of the preceeding GoP as reference ; both I-frame and P-frames are compressed as in the SSF model . Any other frame xi is modeled as a B-frame : it is assigned a past reference frame xj ( j < i ) and a future reference frame xk ( k > i ) ; an off-the-shelf Super-SloMo frame interpolator ( Jiang et al. , 2018 ) is used to interpolate between these two reference frames . The interpolated frame is then used as a basis for scale-space flow warping : xi = Scale-Space-Warp ( Super-SloMo ( xj , xk ) , gi ) + ri . ( 4 ) The interpolated frame provides a more useful starting point for optical-flow warping than the previous frame , e. g. because the combination of past and future knowledge may avoid occlusion effects . Again , I-frames , optical flow , and residuals are modeled with three separate hyperprior models and the model is trained on the RD loss in Eq . ( 1 ) . We only consider a single configuration and use hyperparameters and checkpoints from Pourreza & Cohen ( 2021 ) . This model has 38.5 million parameters , 23.2 million of which are on the decoder side . Encoding a video sequence Our procedure conceptually follows van Rozendaal et al . ( 2021 ) . For completeness we shall describe the full method here . A video sequence x is compressed by : 1 . Finetuning the model parameters ( θ , φ ) of the base model on the sequence x using Eq . ( 2 ) , 2. computing the latent codes z ∼ qφ ( z|x ) , 3. parameterizing the finetuned decoder and prior parameters as updates δ = θ − θD , 4. quantizing latent codes z and network updates δ , and 5. compressing the quantized latents z̄ and updates δ̄ with entropy coding to the bitstream . To finetune a pretrained base compression model , we start with the global model with parameters ( θD , φD ) and minimize the rate-distortion loss given in Eq . ( 2 ) — but only over the single video sequence x . This modified rate-distortion loss explicitly includes the bitrate required to send model updates δ under an update prior p ( δ ) . We compute the regularizing Rθ loss with the unquantized updates δ , but use the quantized parameter updates θD + δ̄ to calculate the D and Rz loss terms , using a straight-trough estimator ( Bengio et al. , 2013 ) in the backward pass . Sending the updated network parameters of course adds to the length of the bitstream . This is even the case when finetuning does not lead to changed parameters ( δ̄ = 0 ) . Given the large size of the neural model we consider , it is therefore essential to choose an update prior that assigns a large probability mass to zero-updates δ̄ = 0 . This allows the network to transmit trivial updates at a negligible rate cost , while giving it the freedom to invest bit cost in non-trivial parameter updates that improve the performance substantially . We use a spike-and-slab prior ( Johnstone & Titterington , 2009 ; van Rozendaal et al. , 2021 ) , a mixture model of a narrow and a wide Gaussian distribution given by p ( δ ) = N ( δ|0 , σ21 ) + αN ( δ ∣∣0 , s21 ) 1 + α , ( 5 ) where the “ slab ” component with variance σ2 keeps the bitrate cost for sizable updates down , and the “ spike ” component with the narrow standard deviation s σ ensures cheap zero-updates . The mixing weight α is a tunable hyperparameter . At the beginning of the finetuning procedure , our neural model is equal to the global model . Because of the spike-slab update prior , the rate cost is only marginally increased and the compression performance essentially equal to the global model . During finetuning , the rate-distortion performance gradually improves , giving us an anytime algorithm that we can stop prematurely to get the best compression performance within a given encoder compute budget . After the model has converged or a compute budget has been exhausted , we use the finetuned encoder qφ to find the latent code z corresponding to the video sequence x . Both z and the updates to prior and decoder δ , which will be necessary to decode the video sequence , are quantized . To discretize the updates δ , we use a fixed grid of n equal-sized bins of width t centered around δ = 0 . We choose n such that p ( −n/t < δ < n/t ) ≈ 1− and clip values at the tails . The quantization of z is analogous , except that we use a bin width of t = 1 and do not clip the values at the tails ( in line with Ballé et al . ( 2018 ) ) . Finally , we write the quantized updates δ̄ and quantized codes z̄ to the bitstream . We use entropy coding under the quantized update prior p ( δ̄ ) and finetuned prior pθD+δ̄ ( z̄ ) , respectively . For latent codes in the tail region we use Exp-Golomb coding ( Wiegand et al. , 2003 ) . Decoding a video sequence The receiver first decodes the prior and decoder updates δ̄ from the bitstream . Once θ̄ = θD + δ̄ is known on the decoder side , the latents z are decoded with the prior pθ̄ ( z ) and the video sequence is reconstructed with the decoder pθ̄ ( x|z ) following standard VAE protocol . The only overhead on the decoder side is therefore the initial decoding of δ̄ . While this results in a small delay before the first frame can be decoded , in practice this delay is very short ( below 0.2 seconds in our experiments ) .

Summary: This paper proposes a video compression algorithm based on instance-adaptive learning. On each video sequence to be transmitted, the proposed method finetunes a pretrained compression model. The parameters' change are transmitted to the decoder with little bit-rate overhead. The experiments show the outstanding performance of the proposed method.