Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set
Authors: Ties van Rozendaal
Johann Brehmer
Yunfan Zhang
Reza Pourreza
Auke J. Wiggers
Taco Cohen
Venue: Accepted by TMLR ()

Abstract: We introduce a video compression algorithm based on instance-adaptive learning. On each video sequence to be transmitted, we finetune a pretrained compression model. The optimal parameters are transmitted to the receiver along with the latent code. By entropy-coding the parameter updates under a suitable mixture model prior, we ensure that the network parameters can be encoded efficiently. This instance-adaptive compression algorithm is agnostic about the choice of base model and has the potential to improve any neural video codec. On UVG, HEVC, and Xiph datasets, our codec improves the performance of a scale-space flow model by between 21% and 27% BD-rate savings, and that of a state-of-the-art B-frame model by 17 to 20% BD-rate savings. We also demonstrate that instance-adaptive finetuning improves the robustness to domain shift. Finally, our approach reduces the capacity requirements of compression models. We show that it enables a competitive performance even after reducing the network size by 70%.

Published in Transactions on Machine Learning Research (06/2023)
Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set
Ties van Rozendaal
Johann Brehmer
Yunfan Zhang
Reza Pourreza
Auke Wiggers
Taco S. Cohen
Qualcomm AI Research*
(ties, jbrehmer, yzhang, pourreza, auke, tacos)@oti..uullcom.coo
Reviewed on OpenRevicw:
Abstract
We introduce a video compression algorithm based on instance-adaptive learning, On each video sequence to be transmitted, we finetune pretrained compression model. The optimal parameters are transmitted to the receiver along with the latent code. By entropy-coding the parameter updates under a suitable mixture model prior, we ensure that the network parameters can be encoded efficiently. This anstance-adaptive compression algorithm is agnostic about the choice of base model and has the potential to improve any neural video codec. On UVG, HEVC, and Xiph datasets, scale-space flow model by wetn 21 % and 27 % BD-rate savings, and that of a state-of- the-art B-frame model by 17 to 20 % BD-rate savings. We also demonstrate that instance- adaptive finctuning improves the robustness to domain shift. Finally, our approach reduces the capacity requirements of compression models. We show that enables a competitive performance even after reducing the network size by 70 %.
codec improves the performance of
1 Introduction
Neural compression methods have enormous potential to improve the efficiency of video coding. With video constituting the large majority of internet traffic, this has significant implications for the internet at large (Sandvine 2019). State-of-the-art algorithms model each frame by warping the previous one with neural estimate of the optical flow (Lu et al.) 2019) scale-space flow (Agustsson et al., 2020 and adding residuals modeled by another network. Both the optical flow and the residuals are compressed with variational autoencoders. Such neural codecs have recently achieved results on par with popular classical
codecs [Agustsson et al.) 2020 Rippel et al. 2021 Pourreza & Cohen 2021 such as H.264 / /VC (Wiegand HEVC (Sullivan et al. 2012). However, relatively little research has focused on their computational complexity, and matching the rate-distortion performance of H.266 / VVC (Bross et al. 2018) is still an open problem.
et al. 2003) and H.265
Neural video codecs depend critically on generalization: they are developed assuming that good perfor- mance on training and validation datasets translates to a good performance at test time. However, this assumption does not always hold in practice, for example because of limited training data or imperfect op- timization. Domain shift is a particularly challenging problem given the variety in video content and styles; for instance, neural video codecs trained on natural scene video data often perform poorly on animated soquences (Agustsson et al. 2020
Instead, in this work, we customize the codec for the sequence to compress, relaxing the need for the codec to generalize. We find that this addapivee compression method results in substantially improved compression performance for various neural codecs. Additionally, it allows us to drastically reduce the computational cost of the receiver-side networks.
Qualcomm AT Research is initiative Qualcomm Technologies, Inc.

---

Published in Transactions on Machine Learning Research (06/2023)
We achieve this customization by finetuning the autoencoder on the sequence to compress at test time, and making the finetuned net parameters available on the receiver side. After training on the test set, the parameters are compressed, quantized, and entropy-coded to the bitstream under a suitable prior, along with the latent code. Effectively, this procodure allows trading off encoding compute for better compression efficiency, similar to how standard codecs are able to improve compression performance when given additional encoding time.
The essential idea of instance-adaptive finetuning was recently proposed by van Rozendaal et al. 20221) The authors demonstrated the idea I-frame compression, e. compressing set of similar images, but did not apply it to video sequences yet. In this work, we benchmark this method on the compression of full videos, in which only periodic key frames are compressed as images and most frames are encoded relative to one or two reference frames. Our method is general and can be applied to various settings and base models. We first demonstrate it in a setting with I-frames and P-frames using a scale-space flow architecture (Agustsson et al.) 2020) as base model. Next, we show its performance in B-frame sctting, using the base model proposed by Pourreza & Cohen (2021).
On the UVG-1k, HEVC class-B, and Xiph-5N datasets, new instance-adaptive video codec yields BD- rate savings of 17 to 27 % over the respective base models, and 5 to 44% over the popular ffmpeg (x265) implementation of the H.265 codec (FFmpeg) VideoLAN bb. In addition, instance-adaptive finetuning lends itself to a possible reduction model size, because a smaller network may suffice to model a single instance. We show that this framework, smaller models can still outperform most neural codecs while reducing the computational complexity of the decoder by 70 %0. We demonstrate that, unlike most neural video codecs, our method can trade off encoding compute against compression performance, and that this trade-off is more effective than for standard codecs.
2 Related work
2.1 Neural video compression
The standard framework used by most neural compression codecs (either implicitly or explicitly) is that of the variational (Kingma & Welling 2014) or compressive Thheis et al. 2017) autoencoder. An encoder (or approximate posterior) (2|2)) maps a data point x to a latent reprosentation 2. This latent is transmittod to the decoding party by means entropy coding under a latent prior po(z). The receiver can then reconstruct the sample with a decoder (or likelihood) pe (x]2). The encoder, prior, and decoder are neural networks, parameterized by weights o, A as indicated by the subscripts.
These models are trained by minimizing the rale-distortion loss
CaD(Ø, 0) = Exsp(()) Bengo(( Φ) [- log [o]]] + Eewgs())a) |- log po (1(l2))
(1)
which combines distortion metric D and rate term R2 that approximates the bitrate necessary to transmit the latent code 2. Up to constant entropy term H H(gs), this loss equals the B-weighted VAF loss (Habibian
et al. 2019 Higgins et al. 2017).
Recent research has focused on the design of efficient network architectures for neural video codecs, in particular with respect to the temporal structure. While some carlier works were based on 3D) convolutions (Habibian et al. 2019 Pessoa al. 2020), recent models use optical flow and residual modeling to exploit similarities between frames. The latter class of approaches can be divided into predictive or P-frame temporal modeling, where the model for each frame I; is conditional on the previous frame(s) AAusssson et al. 2020
Chen et al.) 2019 Liu et al. 2020 Lombardo et al.] 2019 Lu et al., 2019 Rippel ct al. 2019: Golinski ct al.
2020 Rippel el al.) 2021
Hu el al.
2020 Yang el al. 2020b), and bidirectional or B-frame modeling, where
each frame is conditioned on past as well as future reference frames (Cheng et al. 2019 Choi & Bajić 2019
Djelouah et al. 2019 Wu et al. 2018 Park Kim] 2021 Pourreza & Cohen 2021 Yang et al. 20200a Lec
et al.) (022)). Agustsson al. (2020) generalize optical flow to scale-space flow by adding dynamic blurring to the warping operation, improving the modeling of uncertainty and leading to state-of-the-art results.
2

---

Published in Transactions on Machine Learning Research (06/2023)
Table 1: Overview of neural instance-adaptive data compression methods. We categorize these works by the data modality and by the adaptation of receiver-side components. The number of adapted network parameters on the decoder side is a proxy for adaptation flexibility, although flexibility is also affected by other measures such the frequency of adaptation, the quantization of updates. *Ballpark estimate based on the reported network architecture.
Decoder-side adaptation
Type
Work
Modality
Method
Parameters
Encoder-only
Campos al. 2019
image
Guo
2020g
image
Lu
020
vidco
Yang
20220c
image
Hybrid
Lam ml.
2019
image
overfit restoration network
0.
He a.. 2020)
video
overfit restoration network
0.2
Klopp A1.
1220
video
overfit restoration network
<0.1 M
Hc al. 02011
vidco
overfit restoration network
0.2 M
Klopp al. (2021
vidco
overfit supcr-resolution network overfit supcr-resolution network
<0.1 M
Liu ct
2021
video
<0.1 M
Limited decoder-side
Aytekin et al.
(0018
image
chaose from decoders
9.6
adaptation
Zou et al (220
image
choose from 255 bias clusters choose from 8 decoders
<0.1 M
Wang et al.
2021
image
0.9
Neural Implicit
Dupont ct al
2021
image
train decoder network from scratch
<0.1 M
Strumpler et A. 220
image
overfit full decoder network
<0.1 M
Chen et al. 2021
video
train decoder network from scratch
12.5 M
Zhang cl al
2022
video
train decoder network from scratch
<0.1 M
Lee cl al.
video
train decoder network from scratch
12.4
Full-model adaptation
van Rozendaal ct al.
2021
I-frames images
overfit full decoder network
4.2M
Mikami ct al. 2021
overfit full docoder network
0.2 M
This work
video
overfit full decoder network
18.0
2.2 Instance-adaptive compression
Rate-distortion autoencoders are trained by minimizing the RD loss in Eq. over training dataset D. This approach relies the assumptions that the model is not only able to fit the training data well, but also generalizes to unseen datapoints at inference time. In practice, however, finite training data, limited model capacity, optimization imperfections, and domain shift differences between train and test distributions can degrade the performance.
This problem can be solved by adapting video codecs to each sequence at test time. Previous work on such approaches can be roughly categorized into encoder-only finetuning, limited decoder finetuning, implicit neural codecs, and hybrid classical-neural codees. We provide an overview of existing methods in Tbl.
Works in the first category (e. g. Lu et al. (0200) optimize the encoder parameters for each data point. Such update does not have to be communicated to the receiver side. This approach alleviates the generalization problem for the encoder, but does not solve it for the decoder and prior. We will demonstrate later that encoder-only finetuning leads to a limited improvement in compression performance.
The second category of works also adapt parts of the prior and decoder for each video instance. Since these modules need to be known on the receiver side, such an update needs to be signaled in the bitstream. Depending the implementation, this may increase the bitrate substantially. Some works (Aytekin et al,,
2018 Zou et al. 2020 Wang et al. 2021) address this by only allowing limited changes to the decoder-side models, for example, choosing one out of number of fixed decoder networks, thus limiting the potential RD gains of these methods. Finally, the third category of works propose a hybrid approach, where a video is first compressed by a classical codec and a finetuned enhancement network is added to the bitstream (He et al. (2020) 2021; Klopp et al. 2020 2021 Liu al.] (2021). Although these methods have shown great performance at low bitrates, they tend to perform less well at higher bitrates and do not benefit from developments in end-to-end neural compression.
3

---

Published in Transactions on Machine Learning Research (06/2023)
The work van Rozendaal et al. (021) introduces method that adapts the full model to single datapoint. The key idea is that the parameter updates 5 = 0 - @D. where OD are the global model parameters resulting from training on D, can be efficiently transmitted. After discretization, the quantized updates are entropy- coded under a parameter update prior p|ö] that assigns high probability mass (and thus low transmission cost) to vero-updates 8 O. This procedure finetunes the full model encoder, decoder, and prior on each data sample test time, by minimizing the following loss:
LiustA (0, - LRD( 0D + ō) + 3 (())
(2)
Re
The Rg term reflects the increased bitstream length from coding the parameter updates 8. It therefore acts as a regularization term that incentivizes the finetuning procedure to produce updates that are cheap to encode under the parameter update prior. In van Rozendaal et al. (2021), this approach was successfully demonstrated on I-frame compression, essentially compressing a set of images with similar content. Since I-frame models camnot exploit temporal redundancy in video sequences it remains an open question how performant this method is when applied to efficient basc codecs for video compression. In this work, we adapt it to the compression of full video sequences, we will describe in the following section.
Related to this technique is class of methods called neural-implicit compression (Dupont et al. 2021 Strümpler ct al. 2021 Zhang et al. 2022 Chen ct al. 2021 Lee ct al. 2022), where a learned function maps coordinates to a point in pixelspace. In contrast to instance-finet finenuninntt,,, these methods typically involve completely fitting the model on the instance without coding any latents in the bitstream. These methods have some advantages over work they do not suffer from data selection bias and might be casier to standardize. However, they also come with major disadvantages. The biggest issue is that for most neural implicit methods the entire bitstream needs to be received before the first frame can be decoded. This makes them impractical for longer videos and for streaming settings. In contrast, our method only requires a very small fraction of the bitstream for the model updates after which the video can be decoded following normal VAE protocol. Furthermore, neural implicit methods typically have a poor initialization and thus need to train for a long time to get up to performance. In contrast, our method benefits from the fact that even after finetuning for 0 steps performance is already as good the baseline.
Finally, our method is also related to the field of model quantization NNagel et al. 2021 and model com- pression (Kuzmin et al.) 2019: Gale et al. 2019) we quantize the model updates to uniform grid and entropy-code them under a prior. Our approach for parameter transmission is relatively simple and might benefit from advances in this field such as non i.id. priors, structured sparsity (Li et al. 2020a), advanced pruning techniques (Liu et al.) 2019), and non-uniform quantization (Li et al. (020b). A key difference with such works is that we compress and quantize model updates instead of model porameters and adapting these methods to the instance-adaptive compression setting remains interesting direction for future research.
3 eestance-adaptive video compression
We now introduce our instanco-adaptive compression codec for full video sequences, or InstA for short. The key idea is to optimize the parameters of a rate-distortion VAE for each video sequence to be transmalted, and to send the relevant network parameters in a rate-cfficient way to the decoder. This approach can be beneficial whenever standard rate-distortion autoencoder would not generalize well, for instance because of limited training data domain shift. In addition, instance adaptation allows to use smaller models while maintaining most of the compression performance. While encoding the network parameters to the bitstream increases the length of the code, we describe a compression scheme in which this cost is negligible, especially when amortized over many frames of a video sequence.
Our approach is agnostic about the base model, and we demonstrate it on two different architectures: a scale-space flow model in the P-frame setting, and the B-EPIC model in the B-frame setting. We first describe the base models, before describing how InstA compresses and decompresses video sequences.
4

---

Published in Transactions on Machine Learning Research (06/2023)
Table 2: Parameter counts and decoding complexity for our various scale-space- flow base models. Compres- sion curves comparing all four models can be seen in the left panel of Fig.
Model
SSF18
SSF8
SSF5
SSF3
Total parameters [10"]
28.9
13.8
8.1
4.9
Decoder-side parameters [10"]
18.0
8.4
5.0
3.1
Decoder-side kMACs/ / pixel
285.8
167.1
84.7
46.0
Decoder kMACs reduction
0%
43 %
70%
84%
3.1 Base model for P-frame setting: Scale-space flow (SSF)
We first focus a P-frame selling, which we define as having only access to the current and previous frame when decoding a frame. As base model for this setting, we use the scale-space-flow architecture introduced by Agustsson et al. 2020). Any video sequence is first split into groups of pictures (GoP). The first frame in each GoP modeled as an image without any dependency on previous frames, i.e. as an I-frame. All other frames are P-frames, modeled as
= Scale-Space- 1a rr.
(3)
where #,-1 the reconstructed previous frame, 9, the estimated scale-space flow field, r, is the estimated residual, and the scale-space warping operation performs optical-flow warping of the previous frame with dynamic, position-dependent amount. of Gaussian blur. The I-frame images, P-frame scale-space flow gi: and P-frame residuals т₃ compressed with separate hyperprior models (Ballé et al. 2018), which have hicrarchical variational autocncoder architecture.
We largely follow the architecture choices of Aggustssnn et al. 2020 and refer to this model as SSF18 after its number of secoder-side parameters (in millions). In Appendix we describe the architecture detail and highlight the differences to Agustsson al. (2020).
3.2 Smaller base models for P-frame setting
As our instance-adaptive approach is based on finetuning the compression model on low-entropy "dataset" (a single video sequence), we hypothesize that it does not require the full exprossivity of the computationally complex SSF18 model. We propose three alternative scale-space flow architectures with reduced computa- tional complexity. In Tbl. 2 we list the parameter counts and the number of multiply-accmmlltte (MAC) operations required for decoding a sequence, which are reduced by 43 84% compared to the SSF18 baseline. The architectures of the four SSF models are described in detail in Appendix B
3.3
Base model for B-frame setting: B-EPIC
We also consider a less constrained setting, in which frames can be compressed as B-frames as well, e. using both a previous frame and a future frame as reference points. This flcxibility allows for even more efficient compression and is used, for example, for on-demand video streaming. As base model we choose the B-IPIC architecture (Pourreza & Cohen 2021 Again, the video is split into GoPs. The very first frame of video is modeled an I-frame, while the last frame in each GoP is a P-frame, using the last frame of the preceding GoP as reference; both I-frame and P-frames are compressed as in the SSF model.
Any other frame z; is modeled as a B-frame: it is assigned a past reference frame Iy (j < i) and future reference frame Tk (A: > i); an off-the-shelf Super-SloMo frame interpolator (Jiang et al. 2018 is used to interpolate between these two reference frames. The interpolated frame is then used basis for scale-space flow warping:
= Scale-Space Seaae-Space-WaappSSppr
+
The interpolated frame provides more useful starting point for optical-flow warping than the previous frame, e. g. because the combination of past and future knowledge may avoid occlusion effects.
5

---

Published in Transactions on Machine Learning Research (06/2023)
Again, I-frames, optical flow, and residuals are modeled with three separate hyperprior models and the model is trained on the RD loss in Eq. (1). We only consider single configuration and use hyperparamet and checkpoints from Pourreza & Cohen (0221). This model has 38.5 million parameters, 23.2 million of which are on the decoder side.
3.4 Encoding video sequence
Our procedure follows van Rozendaal et al. (2021) and mainly differs in the choice of hyper- rrper-parameters and application to video auto-encoder models. For completeness we shall describe the full method here. video sequence x is compressed by:
1. Finetuning the model parameters (0, Φ) of the base model on the sequence æ using Eq. 2
2. computing the latent codes (2(x)),
8p. 4. quantizing latent codes well prior and decoder parameter updates ö, and
parameterizing the finetued decoder and prior parameters as updates
compressing the quantized latents z and updates å with entropy coding to the bitstream.
To finetune a pretrained base compression model, we start with the global model with parameters (0D, Φp) and minimize the rato-distortion loss given in Eq. (2 but only over the single video sequence x. This modified rale-distortion loss explicitly includes the bitrate required to send model updales under an update prior p(0). We compute the regularizing Rg loss with the unquantized updates 8, but use the quantized parameter updates 0D 5 to calculate the D and R2 loss terms, using a straight-through estimator (Bengio al. 2013 in the backward pass.
Sending the updated network parameters of course adds to the length of the bitstream. This is even the case when finetuning does not lead to changed parameters (5 = 0). Given the large size of the neural model we consider, it is therefore essential to choose an update prior that assigns a large probability mass to zero- updates 8 = 0. This allows the network to transmit trivial updates at a negligible rate cost while giving it the freedom to invest bit cost in non-trivial parameter updates that improve the performance substantially. We use spike-ands liike-and-slab prior (Johnstone & Titterington 2009 van Rozendaal et al. 2021). mixture model of narrow and wide Gaussian distribution given by
p(8) NN600, + ++a
(5)
where the "slab" component with variance o² keeps the bitrate cost for sizable updates down, and the "spike" component. with the narrow slandard deviation & a ensures cheap zero-updales. The mixing weight a is tunable hyperparameter.
At the begimning of the finetuning procedure, neural model is equal to the global model. Due to the spike- slab update prior, the rate cost is only marginally increased and the compression performance is essentially equal to the global model. During finetuning, the rate-distortion performance gradually improves, giving us an anytime algorithm that we can stop prematurely to get the best compression performance within a given encoder compute budget.
After the model has convergod or a compute budget has boen exhausted, we use the finetuned encoder ds find the latent code corresponding to the video sequence x. Both and the updates to prior and decoder quantized. To discretize the updates 6, we
8, which will be necessary to decode the video sequence, use fixed grid of equal-sized bins of width t centered around å = 0 and clip values at the tails. The quantization of = is analogous, except that we use a bin width of t = 1 and do not clip the values at the tails (in line with Ballé et al. (2018)).
Finally, we write the quantized updates 8 and quantized codes E to the bitstream. We use entropy coding under the quantized update prior p(5) and finetuned prior po)+ respectively. For latent codes in the tail region we Exp-Golomb coding (Wiegand et al. 2003
6

---

Published in Transactions on Machine Learning Research (06/2023)
3.5 Decoding video sequence
The receiver first decodes the prior and decoder updates ō from the bitstream. Once 0 0D + is known the decoder side, the latents 2 are decoded with the prior Pa(z) and the video sequence is reconstructed with the decoder P5(=|z) following standard VAE protocol. The only overhead on the decoder side is therefore the initial decoding of 8. While this results in a small delay before the first frame can be decoded, in practice this delay very short (below 0.2 seconds our experiments).
4 Experimental setup
4.1 Datasets
We use sequences from five different datascts. The global models are trained on Vimco90k (Xue et al. 2019). We evaluate on the HEVC class-B test sequences (HEVC 2013), on the UVG-1k (Mercat el al. 2020j dataset, and Xiph-5N (van Rozendaal et al., 2021 which entails five Xiph.org test sequences. The performance on out-of-distribution data is tested on two sequences from the animated short film Big Buck Bunny, also part of the Xiph.org collection (Xiph.org). We choose an "casy" (BBB-E) and a "hard" (BBB-H) clip of 10s each, see Appendix A for details. All of these datasets consist of videos in Full-HD resolution (1920 x 1080 pixels). In addition, we evaluate on HEVC class C with a lower resolution of 832 x 480 pixels.
4.2 Global models
The scale-space flow models described in Sec. 3 are trained with the MSE training setup described in Agustsson et al. (2020), except that we use the publicly available Vimco-90k dataset. The D loss is computed with (inleger) quantized latents Σ, while for the Rg loss term we use noisy samples from U(= 1.2 = + 2)) Following Agustsson et al. (2020), we first train for million steps on 256 256 crops with a learning rate of 10-4. We then conduet the MSE "finetune" stage of the training procedure from Agustsson et al. (2020) (not to be confused with cistance-adaptive finetuning) for the SSF18 model, where we train crops of size h x = 256 x 384 with learning rate of 10-5. The models are trained with a GoP size of 3 frames, which means that we split the training video into chunks of 3 frames and randomly sample chunks during training. We finally evaluate the models with a GoP size of 12. Further increasing the GoP size leads to diminishing returns in rate-distortion performance, as we demonstrate in Appendix D
For the B-EPIC model, we use the model trained by Pourreza Cohen (021)) | on Vimco-90k. The setup is similar to that for the SSF models except that B-EPICs more complicated GoP structure requires training with a GoP size of frames. At test time use GoP size of 12, the frame configurations are described in Appendix B
4.3 Instance-adaptive finetuning
On each instance, we finetune the models with the InstA objective in Eq. 2 using the same weight , as used to train the corresponding global model. We finetune for up to two weeks, corresponding to an average of 300 steps.
In the P-frame scenario we use a GoP size of 3 and finetune on full-resolution frames (1920 x 1080 pixels) with a batch size of and a learning rate of 10-5, After finetuning, we transmit sequences with a GoP size of 12. The prior and decoder updates 8 = f 0D are quantizod and coded under the spiko-and-slab network prior described in Sec. 3
While the SSF model follows stationary distribution over frames, the B-EPIC model has more complex frame structure: the distance between a frame and its reference points can vary between 1 to 6 frames. For this reason, we found it beneficial to use a GoP of 12 for InstA finetuning of the B-EPIC model. Due lo memory limitations, we finetune on horizontal crops of size 256x1080 pixels instead of full-resolution frames. For the low-bitrate working points use a learning rate of 10-5, in line with Pourreza & Cohen (2021). For the high-bitrate of 5 10-5
bitrate region (8 < 0.0008), we found that we can get faster convergence by using a learning rate
7

---

Published in Transactions on Machine Learning Research (06/2023)
4.4 Baselines
As our instance-ndaptive video compression method can be applied to any compression architecture, we focus on comparing to the respective base models: we compare InstA-SSF to our reimplementation of the SSF model [Agustsson al., 2020) and InstA-B-EPIC to the B-EPIC model (Pourreza & Cohen 2021). To better understand where the benefits of instance-adaptive finetuning come from, we consider an additional SSF18 baseline in which only the encoder is finetuned on each instance. Unlike our InstA method, encoder- only fineluning does not require sending any model updates in the bitstream. In addition, we show several neural baselines, focusing on those with the strongest published results as well as those that are most closely related to our method.
We also run the popular classical codecs H.264 (AVC) (Wiegand et al. 2003) and H.265 (HEVC) (Sullivan et al, (012)) in the fimpeg (x264/ x265) implementation (FFmpeg VideoLAN () as well as the (slower but more effective) HM reference implementation (HEVC) of H.265 To ensure fair comparison, in the P-frame setting restrict all codees to I-frames and P-frames and fix the GoP size to 12. In the B-frame experiments, we use default settings, allowing the codecs to freely determine frame types and GoP size. For fimpeg, we compare several different encoder presets to study the trade-off between encoding time and RD performance. In Appendix C we provide the exact commands used to generate these baseline results.
4.5 Metrics
We evaluate the fidelity of the reconstructions through the peak signal-to-noise ratio (PSNR) in RGB space. To summarize the rate-distortion performance in single number, we compute the Bjøntegaard Delta bitrate (BD-rate) (Bjontegaard 2001), the relative rate difference between two codecs averaged over distortion range. We evaluate the BD-rate savings of all methods relative to the HM reference implementation of HEVC, using PSNR as the distortion metric, and cubic spline interpolation. BD-rate savings are computed over & certain distortion range. We choose the largest possible range for which we have results for our methods as well as the main baselines, the values of these ranges are shown in Table 0 in the Appendix.
5 Results
5.1 Compression performance
Figures 1 show the rato-distortion curves of our instanco-adaptive video codec (InstA) as well as noural and traditional baselines. Both for SSF in the P-frame and B-EPIC in the B-frame setting, the instance-adaptive models clearly outperform the corresponding base models. Finetuning only the encoder which does not require sending model updates in the bitstream- leads to much more modest improvement over the base models.
In the P-frame setting, InstA-SSF outperforms all other neural models as well as the fimpeg implementations of H.264 and H.265 at all studied bitrates, except for OTU (Klopp et al.) 2021] and ELF-VC (Rippel et al. 2019), which hold an edge in the low-bitrate region, M-LVC (Lin et al. 2020), which is competitive at medium bitrates, as well as C2F (Hu et al.] 2022 , which appeared concurrently with this method and performs strongly. Within the restrictions of the P-frame setting and at medium to high bitrates, it even competitive with the HM reference codec. In the B-frame configuration, InstA-B-EPIC outperforms all baselines from around 0.1 bits per pixel except HM.
In Fig. 2 we show the relative compression performance compared to the HM baseline. In addition to the classical codecs and our base models, we also compare to the neural baselines proposed in Lu el al. (0019
2020 |
Rippel et al. (2021); ; Lin et al. (2020); Klopp et al. (2200) Yang et al. 2020a c); Hu et al. (2021 2020
2022 :
Lee et al. (2022). Relative to classical codees, almost all neural codecs perform better at higher bitratc.
The exception is Klopp et al. (2020), which consists of classical codec followed by neural enhancement.
We HM version 16.19.
8

---

Published in Transactions on Machine Learning Research (06/2023)
1k, setting
B, setting
40
36
PSNR [dB. 1]
38
PSNR (cB. 71
InstA-SSF18 (ours)
InstA-SSF5 (ours)
36
SSF18 (Agustsson 20)
InstA-SSF18 (ours)
SSF5
InstA-S5F5 (ours)
Encoderttuned SSF18
SSF18 (Agustsson 20)
ELF-VC (Rippe) 21)
SSFS
OTU (Klopp 21)
32
Encoder-tuned SSF18
34
C2F (Hu 221
C2F (Hu 22)
H.265 (ffmpeg)
H.265 (ffmpeg)
H.264 (ffmpeg)
H.264 (ffmpeg)
32
30
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.8
Rate [bpp,
Rate [bpp,
UVG-1k, B-frame setting
HEVC class B, B-frame setting
42
38
PSNR [dB, 11
38
PSNR (dB, +]
34
36
B-SLIC (ours)
B B-EPIC (Pourreza 21)
InstA B EPIC (ours)
tuned
(Pourreza 21)
FFNerV
Encoder-tun B-EPIC
HLVC (Yang 20a)
HLVC (Yang 20a)
H.265 (HM)
H.265 (HM)
H.265 (fimpeg)
H.265 (fimpeg)
H.264 (ffmpeg)
H.264 (tmppe)
32
30
0.0
0.2
0.4
0.6
0.0
0.2
0.1
0.6
Rate [bpp.
Rate [bpp. +1
Figure 1: Rate-distortion performance of InstA video codecs. InstA (red) leads to BD-rate savings of 17% to 27% over the corresponding base models (orange) and % to 44 % over H..665 (ffmpeg). For all figures, higher and to the left is better.
5.2 Robustness to model-size reduction
While the full-size InstA-SSF18 model leads to the best RD performance in the P-frame setting, the smaller InstA-SSF5 models hold their own. Despite the 70% reduction dccoder complexity, Insta-SSF5 outper- forms most neural baselines and the fTmpeg codecs, and achieves a performance close to that of InstA-SSF18. In comparison, the gap between the SSF5 and SSF18 baselines is larger. This provides evidence for the hy- pothesis that instanco-adaptive finetuning reduces the capacity requirements on neural compression models.
To further investigate how the performance of our method depends on decoder complexily, we train two additional models, SSF3 and SSFs, as summarized in Tbl. 2 The results are in line with the trends already shown. While the performance of the global models degrades quickly when reducing the model size, instance- adaptive finetuning can compensate for the reduced capacity and offers more "bang for the buck" in terms of (decoding-side) model complexity. For detailed results refer the readers to Appendix D
9

---

Published in Transactions on Machine Learning Research (06/2023)
UVG-1k, P-trame setting
HEVC class B, P-frame setting
-40%
InstA-SSF18 (ours)
(ours)
SSF18 (Agustsson 20)
Rate: relative to H.265 (HM) [4]
Rate relative to H.265 (HM) [4]
0%
0%
Encoder-tuned
DVC 19)
DVC-adaptive 20)
ELF-C (Rippel 21)
40%
40%
M-LVC (Lin 20)
OTU (Klopp 21)
(Yang 20b)
20)
80%-
80%~
(Hu 22)
FVC (Hu 21)
H.265 (HM)
H.265 (fimpeg)
120%
(ffmpeg)
34
36
38
PSNR [dB, 1]
PSNR [dB, 1]
UVG-1k, B-frame setting
HEVC class B-frame setting
-40%
InstA-B-EPIC (ours)
B-EPIC (Pourreza 21)
Encoder-tuned B-EPIC.
Rate relative to H.265 (HM)
0%-
Rate relative to H.265 [4]
HLVC (Yang 20a)
VCII (Wu 18)
40%-
40%
FFNerV (Lee 22)
NeRV (Chen 21)
H.265 (HM)
80%-
H.265 (ffmpeg)
80%
H.264 (ffmpeg)
120%
120%
160%
200%
34
36
38
40
30
32
34 PSNR [dB, 7]
36
38
PSNR (dB, 7]
Figure 2: Bitrate of our instanco-adaptive SSF models (InstA-SSF, red) relative the HM reference imple- mentation of H.265, as function of the quality in dB PSNR. A relative rale of zero indicates a performance par with HM, negative numbers (at the top) indicate fewer bits are needed, and positive numbers (at the bottom) indicate more bits are needed. This means that for all figures, higher is better. We show several neural (orange, blue, green) and traditional (grey) basclines. The interpolated curves are obtained by fitting a function in the (log) bitrate domain, similar to the way Bjøntegaard Delta-rate is computed.
10

---

Published in Transactions on Machine Learning Research (06/2023)
Table 3: Rate-distortion performance of InstA and baselines on several datasets. We report Bjontegaard
Delta-rate relative to HM, the reference implementation of H.265 (lower better).
BD-rate relative H.265 (HM)
Setting
Method
UVG-1k
HEVC-B
HEVC-C
Xiph-5N
BBR-E
BRR-H
P-frames
InstA-SSF18 (ours)
5.0 %
65.0%
10.4 %
13.4 %
28.3 %
InstA-SSF5 (ours)
4..6% %
15.4%
89.3%
77.5 %
31.9 %
43.2
Encoder-finctunee SSF
5.2%
21.7%
80.4%
9.1%
22.7
48.1
SSF18 Agustsson et al
2020
22.5%
38.2%
107.5%
17.5%
32.7%
54.9
SSF5
43.8 %
52.0%
172.4%
41.5%
84.5 %
99.3%
H.265 (ffmpeg)
58.8 %
47.1% %
42.2 %
29.5 %
25.9 %
18.7 %
H.264 (ffmpeg)
113.8% %
9
73.4 %
69.3 %
108.4 %
46.9 %
B-frames
BastA-B-EPIC (ours)
4.6 %
21.9% %
Encoder-finetuned
finetuned B-EPIC
12.6 %
%
B-EPIC (Pourreza & Cohen 2021
30.3 %
47.2%
H.265 (fimpeg)
32.8 %
28.8%
H.264 (fimpeg)
80.4 %
74.2%
5.3 Robustness to dataset variation
Globally trained neural codecs rely the similarity of the test instances to the training data. In comparison, expect our finetuned models to be more robust under domain shift and to perform well across different test sets. To put this to the test, we evaluate the InstA-SSF models and P-frame baselines on different sequences, including two animated scenes from the Big Buck Bunny video. In Tbl. 3 and we report the compression performance as BD-rate savings relative to the HM reference codec.
We find that InstA performs strongly on all datasets. On the animated sequences, the global SSF18 bascline performs poorly relative ffmpeg H.265, showing its susceptibility to domain shift. Our instance-adaptive models are much more robust to this shift and close the gap with classical codecs.
Instanco-adaptive compression is particularly powerful for high-resolution data, where the modcl-rate over- head amortized over more pixels. We test whether InstA still works at a lower resolution by evaluating on the HEVC class C test sequences, which have 81 % less pixels per frame than the Full-HD sequences. Indeed, InstA performs worse relative to the baselines than on the higher-resolution data, but it still is among the best neural codees.
To further quantify robustness to test data characteristics, investigate the effect of the video framerate the model performance. By studying the performance of global and instance-adaptive SSF models on temporally subsampled videos, we confirm that instance-adaptive model is more robust to this variation as well. In Appendix [] we study the performance of global and instance-adaptive SSF models on temporally subsampled videos and again find that the instance-adaptive model is more robust to variations than the global SSF model. While these ablations do not cover all realistic variations, they serve as examples of dataset characteristics that a global neural codec is likely sensitive
5.4 Perceptual quality
We illustrate the performance ant-of-distribution data in Fig. 4 where show original and compressed versions of two frames from the Big Buck Bunny video. Both traditional codecs and the SSF18 baseline lose details and introduce artifacts at the chosen low-bitrate setting. While the InstA-SSF18 compressions still suller from some blurriness, artifacts are less pronounced than in the global model.
In the supplementary matcrials, we attach the corresponding reconstructed vidco sequences. To highlight the differences between the codecs, we again use very low bitrate for all codecs (even though our InstA method is strongest at higher bitrates).
11

---

Published in Transactions on Machine Learning Research (06/2023)
HEVC class P-frame
Xiph-5N. P-frame setting
36
34
38
InstA-SSF18 (ours)
PSNR LdB. 11
32
InstA SSFS (ours)
Encoder-tuned SSF18
PSNR [dB,
SSF18 (Agustsson 20)
SSF5
30
DVC (Lu 19)
InstA-SSF18 (ours)
DVC-adaptive (Lu 20)
34
InstA-SSF5 (ours)
RLVC (Yang 20b)
SSF18 (Agustsson 20)
RaFC (Hu 20)
SSF5
28
FVC (Hu 21)
Encoder-turee SSF18
32
H.265 (HM)
H.265 (HM)
H.265 (ffmpeg)
H.265 (ffmpeg)
H.264 (ffmpeg)
H.264 (ffmpeg)
26
0.0
0.2
0.4
0.8
0.0
0.2
0.6
0.8
Rate [bpp. 41
Rate [bpp. 4]
Big Buck Bunny, easy sequence (BBB-E), E), P-frame setting
Buck Bunny, hard sequence (BBB-H), P-frame setting
48
42
38
PSNR (dB, []
44
PSNR [dB,
InstA-SSF18 (ours)
InstA-SSF18 (ours)
InstA-5SF5 (ours)
34
InstA-SSF5 (ours)
SSF18 (Agustsson 20)
SSF18 (Agustsson 20)
S5F5
55F5
40
Encoder-tuned
Encoder-tuned SSF18
32
H.265 (HM)
H.265 (HM)
H.265 (ffmpeg)
H.265 (ffmpeg)
H.264 (ffmpeg)
(ffmpeg)
0.0
0.1
0.2
0.0
0.2
0.4
0.6
Rate [bpp. 4]
Rate [bpp. 4]
Figure 3: Rate-distortion performance of our instance-adaptive SSF models (InstA-SSF, red) compared to neural (orange) and traditional (grey) baselines. We show results on class of the HEVC test sequences (top left), five Xiph.org test sequences (top right), and the "easy" (left) and "hard" (right) sequences from the Big Buck Bunny video (bottom). For all figures, higher and to the left is better.
12

---

Published in Transactions on Machine Learning Research (06/2023)
Table 4: BD-rate of several codecs (rows) compared to different reference codees (columns) in the P-frame setting. Lower numbers are better, the best results on each dataset are shown in bold.
BD-rate relative to...
Datasel
Method
H.265 (HM)
H.265 (fmmpeg)
H.264 (fmmpeg)
SSF18
UVG
Tnst A-SSF18 (ours) InstA- (ours)
6.7 %
41.9 %
-57.6 %
26.7 %
- 0.3 %
-38.0 %
-54.7 %
-21.7%
SSF18
27.3 %
-20.8 %
-42.1 %
siccoderfinetuune SSF18
8.8 %
-32.3 %
-50.5 %
14.5 %
НEVС-B
InstA-SSF18 (ours)
5.0% %
-28.6 %
-45.5 %
24.0% %
InstA-SSF3 (ours)
15.4 %
-2..6 %
-40.2%
- -16.5 %
SSF18
38.2 %
-6.1 %
-28.3%
Encoder-finetuned SSF18
21.7 %
-17.3 %
-36.9% %
12.0% %
HEVC-C
InstA-SSF18 (ours)
65.0 %
16.0 %
-4.8 %
20.5 %
A-SSF5 (ours)
89.3 %
33.1 %
9.2 %
8.88 %
SSF18
107.5 %
45.9 %
19.7 %
Encoder-finctuned SSF18
80.4 %
26.9 %
4.1 %
13.1 %
xiph5N
InstA-SSF18 (ours)
10.4 %
30.8 %
47.1 %
23.8
A-SSF5 (ours)
-7.5 %
-88.6 %
-45.4 %
21.3%
SSF18
17.5 %
-9.2% %
-00.6% %
fnneuue--inetuned SSF18
9.1 %
-15.7 %
-35.6% %
-7.2%
BBB-E
Tnst A-SSF18 (ours) InstA-SSF5 (ours)
13.4 %
9.9 %
-45.6% %
14.5 %
31.9 %
4.8 %
-36.7%
-0.7%
SSF18
32.7% %
5.4 %
-36.3% %
SSF18
22.7% %
-2. %
-41.1 %
-7.5%
BBB-H
Inst.A-SF118 (ours) InstA-SSF5 (ours)
28.3% %
8.1 %
-12.6%
17.11
43.2%
20.6 %
-2.5%
-7.5%
SSF18
54.9 %
30.5 %
5.4%
Encoder-finetuned SSF18
48.1 %
24.8 %
0.8%
4.4%
Table 5: BD-rate of several codees (rows) compared to different reference codees (columns) in the B-frame setting. Lower numbers are better, the best results on each dataset. are shown in bold.
BD-rate relative to...
Dataset
Method
H.265 (HM)
H.265 (ffmpeg)
H.264 (fimpeg)
R-EPIC
UVG-1k
InstA-B-EPIC (ours)
9.9 %
-16.5 %
-39.4 %
18.7% %
B-EPIC
35.2 %
2.8 %
-25.5 %
Encoder-fintture B-EPIC
15.9 %
-1..9 %
-36.1 %
-1.33%
HEVС-B
InstA-B-ERT B-PIC (ours)
21.9 %
-5.4% %
-30.0 %
-17.2%
B-EPIC
47.2% %
14.3 %
-15.5 %
crrodde-finetued B-EPIC
48.1 %
15.0 %
-15.0 %
0.6%
13

---

Published in Transactions on Machine Learning Research (06/2023)
Original
H64 (ffmeeg) 0,031 bep
HI265 (
0,031 opp
SSF18 (Agustsson 0,030 bpp
InstA-SSF 18 lours) 0,031 bpp
Figure 4: Original and compressed details in frames 7395 (left) and 7276 (right) of the Big Buck Bunny video (Xiph.org), which are compressed as a P-frame (left) and I-frame (right). From top to bottom, we compare the original, compressed frames with the H.264 and H.265 codecs the fimpeg (x264 / x265) implementation, the SSF18 baseline, and our proposed Inst.-SSF188 method. We use scttings with comparable, low bitrates; the achieved bitrates (over the 10s sequence) are given in the left panels. All codecs lose textural details, but our InstA-SSF118 codec reduces the artifacts visible in the SSF18 reconstructions.
5.5 Rate composition
It is remarkable that InstA consistently improves the rate-distortion performance of neural models despite having to send updates for millions of neural network parameters. We investigate this low rate overhead by estimating how the update prior affects the bitrate. We find that the spike-and-slab prior is indeed crucial for the low rale overhead: compared Gaussian coding, it allows us save around 7.5 bits per parameter (see AppendixD) D).
InstA learns to distribute bits model updates and latents. The optimal trade-off generally depends the bitrate, the size of the model and latent code, and on the test data: we expect that for strongly out-of- distribution data, more bits will be spent on model updates. We show the learned bit distribution in Fig. g Indoed we find that in almost all cases, less than 1% of the total rate spent on model updates, while for the animated sequences from the Big Buck Bunny video between 1% and 3% of the total bitrate are invested in model updates.
5.6 Update frequency
For longer videos, the granularity of data on which the model finetuned and the frequency with which updates are transmitted are open questions. Finctuning models on shorter scenes may improve the compres- sion performance. Moreover, each key frame (or random access point) requires the transmission of model update in the bitstream. On the other hand, too frequent update transmissions will increase the bitrate over-
14

---

Published in Transactions on Machine Learning Research (06/2023)
UVG 1k. P-frame setting
class B, P-frame setting
120
100
SSF18 (ours)
InsLA SSF18 (ours)
InstA SSFS (ours)
InstA SSFS (ours)
100
SSF 18 (Agustsson et all)
SSF18 (Agustsson at al.)
SSF5
SSF5
H.265 (HM)
H.265 (HM)
H.265 (tfmpeg)
H.265 (ffmpeg)
80
BD-rate relative 1%.
H.264 (ftmpeg)
BD-rate relative to HM 1%.
H.264 (ffmpeg)
60
40
40-
20
20
0-
0-
10ms
150
15
105
1 min
10min
1h
10ms
100
1s
105
1min
10min
1h
Encoding time / frame
Encoding time / frame
Figure 5: Trade-off between RD performance and encoder-side compute in the P-frame setting, where lower and to the left is better. We summarize the RD performance as BD-rate difference relative to the HM implementation of H.265 (lower better). InstA-SSF (red) is compared to SSF (orange) and traditional (grey) baselines. Note that the runtime difference between neural and traditional codecs strongly depends on the available hardware, since the neural codecs require a GPU to be efficient, while the traditional codecs run on the (multi-core) CPU. Some fluctuations in the measured walltime our computational cluster. Since we can stop the instance finetuning al any time, InstA-SSF allows us to continuously trade off encoder compute for compression performance. t
due to varying loads
head, limiting the improvement in rale-distortion performance from our method. The optimal compromise will depend on the application, video resolution, video content, and model complexity. In our experiments we compressed sequences the 2.5-10s range. Given the small overhead from the model updates, scnding model updates in the bitstream once every second is certainly feasible. In Appendix D we study adaptation at smaller granularity, finding that finetuning model for entire video of 10 minutes provides a slightly worse performance than finetuning scenes of 10 seconds.
5.7 Computational complexity
The improved compression performance of our instance-adaptive video codec comes at the price of increased encoding time. In Fig. 5 we illustrate this trade-off for the P-frame setting. We summarize the rate- distortion performance as BD-rate savings relative to HM and compare to the encoding time"] For our instanco-adaptive methods, this time includes the finetuning in addition to the forward pass of the encoder, the quantization, and the entropy coding (for both the latents and the parameters). By choosing how long to run the finetuning, we can trade off RD performance against the encoding time. Out of the studied codecs, ours can best exploit this trade-off: encoding time and compression performance are fixed in other neural methods, while the existing handles in traditional codees have a limited effect on performance.
We find that instance-adaptive finetuning leads to state-of- the-art RD performance even when finetuning only for few steps. The performance gradually improves over the course of training and eventually flattens out. Care has to be taken in interpreting these results as the different codecs were designed for different hardware and our algorithms have not been optimized for performance.
Instance-adaptive video compression also adds some overhead on the decoder side, as the model parameters need to be decoded before any frame can be reconstructed. For the full-size InstA-SSFF. model and without
We report the walltime on machines with 40-core Intel Xeon Gold 6230 CPUs with 256 GB RAM and NVIDIA Tesla V100-SXM2 GPUs with 32 GR VRAM. We only use a single GPU.
15

---

Published in Transactions on Machine Learning Research (06/2023)
optimizing the code for performance, this initial delay is below 0.2 seconds. The neural network forward pass alone is timed with a speed of around 0.01 seconds per frame TeslaV100 GPU without any optimization. Both the overhead and the network forward pass are thus negligible compared to the time it takes to entropy- decode the latents 2: in our (not optimized) implementation, this takes around 1 second per frame, both for the global SSF models and InstA-SSF. Compared to the neural baselines, InstA-SSF thus yields better performance at comparable complexity. Compared to traditional codecs, there still a gap to bridge, and none of the neural codecs can decode at the framerate of the videos yet.
At the same time, instance-adaptive finetuming allows us to use smaller architectures while maintaining strong performance, as shown above. This potential to reduce the decoder-side compute may outweigh the small overhead from the initial update decoding.
6 Conclusions
We introduced InstA, a method for video compression that finetunes a pretrained neural compression model on each video sequence to be transmitted. The finetuned network parameters are compressed and transmitted along with the latent code. The method is general and can be applied to many different settings and architectures, demonstrated it scale-space flow models in the P-frame setting well as state-of-the- art B-frame architecture.
Instance-adaptive video compression relaxes the requirement of a single global model that needs to be able to generalize to any unseen test sequence. This can be beneficial when the training data is limited, when there is domain shift between training and test distributions, or when the model class is not expressive enough. While comes at the cost. of some overhead to the bitstream, we demonstrated that with a suitable compression scheme this overhead can be kept small, especially since it amortized over the many pixels of video sequence.
Our InstA models clearly outperform the base models in all considered settings, in several cases achieving a state-of-the-art rate-distortion performance. They outperform their corresponding global base models with BD-rate savings of between 17 % and 27 % on the UVG-lk, HEVC class-B, and Xiph-5N datasets, and the popular fimpeg implementation of H.265 with BD-rate savings between 5 % and 44 %. By evaluating models that were trained on natural scene data on animated sequences, we demonstrate that instanco-adaptive finetuning improves the robustness of neural video codecs under domain shift. Finally, instance-adaptive compression reduces the required model expressivily: even after reducing the model size by 70 %, InstAASSSF maintains competitive performance.
The added performance improvement comes with significant added computing cost at the encoder side. However, the ability to trade off encoder-side compute for compression performance is a natural match for one-to-many transmission scenarios, in which a sequence compressed once but transmitted and decoded often. We demonstrated that even just finetuning for seconds per frame can lead to a substantial improvement in compression performance, so instance-adaplive compression may ultimately even be beneficial in real-time applications.
Acknowledgements
We would like to thank the whole Qualcomin AI Research leam, but in particular Amir Said, Guillaume Sautiére, Yang Yang, and Yinhao Zhu for useful discussions and help with the experiments.
We are grateful to the authors and maintainers of fimpeg (FFmpog), Matplotlib (Hunter 2007). Numpy
CCarles R Harris et al. 2020), OpenCV (Bradski 2000), pandas (McKinney) 2010), Python (Python core team 2019). PyTorch (Paszke al. 2017), scipy (SeiPy contributors (020)). and seaborn (Waskom) 2021).
References
Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized video compression. In IEFE Conference on Computer Vision
16

---

Published in Transactions on Machine Learning Research (06/2023)
and Pattern Recognition (CVPR), pp. 8503-8512 2020.
Caglar Aytekin, Francesco Cricri, Antti Hallapuro, Jani Lainema, Emre Aksu, Miska M Hannuksela, and Hatanpaan Valtatie. compression objective and a cycle loss for neural image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2018.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In International Conference on Learning Representations (ICLR), 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013. URL http:://arxi.org/ abs/1308.
Gisle Bjøntegaard. Calculation of average PSNR differences between RD-curves (VCEG-M33). In VCEG Meeting (ITU-T SG16 Q. 6), pp. 2-4, 2001.
G. Bradski. The OpenCV Library. Dr. Dobb's 's Journal of Software Tools, 2000.
Benjamin Bross, Jianle Chen, Shan Liu, and Yo-Kui Wang. Versatile video coding (draft 5). Joint Video Experts Team (JVET) of ITU-T SG, 16:3 12, 2018.
Joaquim Campos, Simon Meierhans, Abdelaziz Djelouah, and Christopher Schroers. Content adaptive optimization for neural image compression. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognition (CVPR) Workshops, pp. 0-0, 2019.
Charles R Harris et al. Array programming with NumPy. Nature 585 (85(7))) 2020. ISSN 1476-4687. URL
doi: 10.1038/s41 (415566002022666
1038 1ttp://1...1188/1116--00000000 020-1588--222--44922
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos, 2021.
Zhibo Chen, Tianyu He, Xin Jin, and Feng Wu. Learning for video compression. IEEE Transactions on Circuils and Systems for Video Technology, PP, 2019. doi: N0.1109/TCSVT.200 2892668.
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learning image and video compres- sion through spatial-temporal energy compaction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Hyomin Choi and Ivan V Bajié. Deep frame prediction for video coding. TEFE Transactions on Circuits and Systems for Video Technology, 30(7):1843 1855, 2019.
Abdelaziz Djelouah, Joaquim Campos, Simone Schaub-Meyer, and Christopher Schroers. Neural inter-frant compression for video coding. In IEFE International Conference on Computer Vision (ICCV), pp. 6420- 6428, 2019. doi: 10.1109 200..006.......
Emilien Dupont, Adam Golinski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. COIN: COmpression with implicit neural representations. In Neural Compression: From Information Theory to Applications Workshop ICLR 2021, 2021. URL httpp:///ppprrreeee...e////aaadd
FFmpeg. FFmpeg. http://ffmpeg: org/
Trevor Gale, Erich Elsen, and Sara Hooker. The state sparsity in deep neural networks, 2019.
Adam Golinski, Reza Pourreza, Yang Yang, Guillaume Sautiere, and Taco S Cohen. Feedback recurrent autocncoder for vidco compression. In IEEE Asian Conference on Computer Vision (ACCV), 2020.
Tianshong Guo, Jing Wang, Ze Cui, Yihui Feng, Yunying Ge, and Bo Bai. Variable rate image compression with content adaptive optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR) Workshops, pp. 122 123, 2020.
17

---

Published in Transactions on Machine Learning Research (06/2023)
Amirhossein Habibian, Ties van Rozendaal, Jakub M Tomezak, and Taco S Cohen. Video compression with rate-distortion autoencoders. In JEEE International Conference on Compuler Vision (ICCV), pp. 2019.
7033-T 7042. ppenaccess.thevii.co
Gang He, Chang Wu, Lei Li, Jinjia Zhou, Xianglin Wang. Yunfci Zheng, Bing Yu, and Weiying Xie. video compression framework using an overfitted restoration neural network. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 148-149, 2020.
Gang He, Chang Wu, Li Xu, Lei Li, Ziyao Xu, Weiying Xie, and Yunsong Li. An efficient video coding system with an adaptive overfitted multi-scale attention network. IEEE Access, 9:64022 (664022-6403,, 2021.
HEVC. HEVC test model (HM). https://vcgit.. www.hiff.auuaofffa..o dew.eeeeeereee/e////eeeeeee
HEVC. Common test conditions and software reference configurations.
ittpnnxxiiittsuee eu/
jct/doc_ /dd__edddd___eedddeeeedddddddddd end /user/currenttddccmeett _cc.mmentt.pppp71d722111 2013.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mo- hamed, and Alexander Lerchner. Deta-VAE: Learning basic visual concepts with constrained variational framework. International Conference on Learning Represenlations (ICLR), 2017.
Zhihao Hu, Zhenghao Chen, Dong Xu, Guo Lu, Wanli Ouyang, and Shuhang Gu. Improving deep video compression by resolution-adaptive flow coding. European Conference on Computer Vision (ECCV), 2020.
Zhihao Hu, Guo Lu, and Dong Xu. FVC: A New Framework Towards Deep Video Compression in Feature Space. In JEEE Conference on Computer Vision and Pattern Recognation (CVPR), pp. 1502-1511, June 2021.
Zhihao Hu, Guo Lu, Jinyang Guo, Shan Liu, Wei Jiang, and Dong Xu. Coarse-to-fine deep video coding with hyperprior-gutee guiddd mode prediction. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 5921-5930, 2022.
John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9(3):90, 2007.
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9000-9008, 2018.
lain Johnstone and Michael Tilteringlon. Statistical challenges of high-dimensional data. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 367:4237 4253, 2009.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. International Conference Learning Representations (ICLR), 2014.
Jan P Klopp, Liang-Gee Chen, and Shao-Yi Yi Chien. Utilising low complexity cnns to lift non-local redun- dancies in video coding. IEEE Tronsactions on Image Processing, 2020.
Jan P Klopp, Keng-Chi Liu, Shao-Yi Chien, and Liang-Gee Chen. Online-trained upsampler for deep low complexity video compression. In Proceedings of the CEE//VEE International Conference on Computer Vision (ICCV), pp. 7929-7938, 2021.
Andrey Kuzmin, Markus Nagel, Saurabh Pitre, Sandeep Pendyam, Tijmen Blankevoort, and Max Welling, Taxonomy and evaluation of structured compression of convolutional neural networks. arXiv preprint arXiv: 192.:1912209800 2019.
Yat Hong Lam, Alireza Zare, Çaglar Aytekin, Francesco Cricri, Jani Lainema, Emre Aksu, and Miska M. Hannukscla. Compressing woight-updates for image artifacts removal neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. Computer Vision Foundation / IEFE, 2019.
18

---

Published in Transactions on Machine Learning Research (06/2023)
Joo Chan Lee, Daniel Rho, Jong Hwan Ko, and Eunbyung Park. Ffnerv: Flow-guided frame-wist neural representations for videos. arXiv preprint arxin; 2022.
Yawei Li, Shuhang Gu, Christoph Mayer, Lue Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp.
2020a.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient. non-uniform discretization for neural networks. In International Conference on Learning Representations, 2020b. URL https://openrevien www.://wwwwev.i..ooooooooaaaaaa. (ttt/foorumm1ddxkg222220
Jianping Lin, Dong Liu, Houqiang Li, and Feng Wu. M-LVC: Multiple Frames Prediction for Learned Video
Compression. TEFE Conference on Computer Vision and Pottern Recognition (CVPR), Jun 2020. doi:
00.1100/cYD:22800220000000
URL http://dx. http::/dx.dd...ooo doi www..o............. 1109///////220000000/ http:///P22200022200000000 2000..003200000000
Haojie Liu, Han Shen, Lichao Huang, Ming Lu, Tong Chen, and Zhan Ma. Learned video compression via joint spatial-temporal correlation exploration. In A AAAT Conference on Arlificial Intelligence, number 07, pp. 11580-1000 11587, 2020.
Jiaming Liu, Ming Lu, Kaixin Chen, Xiaoqi Li, Shizun Wang, Zhaoqing Wang, Enhua Wu, Yurong Chen, Chuang Zhang, and Ming Wu. Overfitting the data: Compact neural video delivery via content-aware feature modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4631 1831-464,,455 2021.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Chong, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3296-3305, 2019.
Salvator Lombardo, Jun Han, Christopher Schrocrs, and Stephan Mandt. Deep generative video compression. In Advances in Neural Information Processing Systems (NeurIPS), pp. 9287 9298. Curran Associates, Inc., 2019.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. DVC: An end-to-end deep video compression framework. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11006-11015, June 2019.
Guo Lu, Chunlei Cai, Xiaoyun Zhang, Li Chen, Wanli Ouyang, Dong Xu, and Zhiyong Gao. Content adaptive and error propagation aware deep vidco compression. European Conference on Computer Vision (ECCV), 2020.
Wes McKinney. Data structures for statistical computing in python. In Stéfan van der Walt and Jarrod Millman (eds.), Python in Science Conference, pp- 51 56, 2010.
Alexandre Mereat, Marko Viitanen, and Jarno Vanne. UVG dataset: 50/120fps 4k sequences for video codec analysis and development. In ACM Multimedia Systems Conference, pp. 297 302, 2020. URL
http://ultravideo http://alrarrddof.f//tttttoooue ftt../ssttttttttuunccssss
Yu Mikami, Chihiro Tsutake, Keita Takahashi, and Toshiaki Fujii. An efficient image compression method based on neural network: An overfitting approach. In 2021 IEEE International Conference on Image Processing (ICIP), pp. 2084-2088, 2021. doi: 10.1109/ICIIPP 229.22.....2195063377
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. white paper on neural network quantization. arXiv preprint Xrx:::200..0m 08295, 2021.
Woonsung Park and Munchurl Kim. Deep predictive video compression using modc-scloctive uni and bi- directional predictions based on multi-frame hypothesis. IEEE Access, 9.72-85, 2021. doi: 10.1109/ ACCFSS.2020.304600
19

---

Published in Transactions on Machine Learning Research (06/2023)
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NTPS Aulodiff' Workshop, 2017.
Jorge Pessoa, Helena Aidos, Pedro Tomás, and Mário AT Figueiredo. End-to-end learning of video compres- sion using spatio-temporal autoencoders. In 2020 IEEE Workshop on Signal Processing Systems (SiPS), pp. 1-6. IEEE, 2020.
Reza Pourreza and Taco S Cohen. Extending neural P-frame codecs for B-frame coding. In IEEE Interna- tional Conference on Computer Vision (ICCV), 2021.
Python core team. Python: dynamic, open source programming language. Python Software Foundation, 2019. URL httts::///vtt http///wwptthh...mmmm python.oho........
Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander G Anderson, and Lubomir Bourdev. Learned video compression. In IEEE International Conference on Computer Vision (ICCV), pp. 3454- 3463, 2019.
Oren Rippel, Alexander G. Anderson, Kedar Tatwawadi, Sanjay Nair, Craig Lytle, and Lubomir D. Bourdev. ELF-VC: efficient learned flexible-rate video coding. CoRR, abs/2104.14335, 2021. URL https: htttss//arxiv, org/ 20g/abs/2104.1420 14335
Sandvine. 2019 Global Internet Phenomena Report. httpgs//wir...daaa..o/////ttttta does oateemmjjoyy------eeeee----tt-t-
2019.
SciPy contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods 2020. doi: http:://bbi.or//1..00/////// 019-066--..
Yannick Strümpler, Janis Postels, Ren Yang, Lue van Gool, and Federico Tombari. Implicit neural repre- sentations for image compression, 2021.
Gary J Sullivan, Jens- Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (HEVC) standard. JEEE Transactions on Circuits and Systems for Video Technology, 22 1126:1649-1668, 2012.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with com- pressive autoencoders. In International Conference on Learning Represenlations (TCLR), 2017.
Ties van Rozendaal, Tris AM Huijben, and Taco S Cohen. Overfitting for fun and profit: Instance-adaptive data compression. In International Conference on Learning Representations (ICLR), 2021.
VideoLAN. x264 library.
a.
VideoLAN. x265 library. http:s//ww...ddeeb...orrrrrrrrrp
b.
Yefei Wang, Dong Liu, Siwei Ma, Feng Wu, and Wen Gao. Ensemble Searning-Basee Rate-Distortion optimization for End-to-End image compression. IEEE Transactions on Circuits and Systems for Video Technology, 31 (3):11933-122)) March 2021.
Michael L. Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6(60):3021, 2021. doi: 10.21105/joss.0002.. URL
Thomas Wiegand, Gary J Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the H.264/AVC video coding standard. IEEE Transactions on Circuits and Systems for Video Technology, 13(7):560-576, 2003.
Chao- Yuan WU, Nayan Singhal, and Philipp Krahenbuhl. Vidco compression through image interpolation. Tn European Conference on Computer Vision (ECCV), pp. 416 431, 2018
Xiph.org. Video test media. https://media. http:://wwdaa....ppp.oo/////////
20

---

Published in Transactions on Machine Learning Research (06/2023)
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision (IJCV), 127(8):1006-112)) 2019. URL
http://toflow http//iafooo.ccailll.o mit.edu/
Ren Yang, Fabian Mentzer, Luc Van Gool, and Radu Timofte. Learning for video compression with hierar- chical quality and recurrent enhancement. IEEE Conference Computer Vision and Pottern Recognation (CVPR), 2020a.
Ren Yang. Fabian Mentzer, Luc Van Gool, and Radu Timofte. Learning for video compression with recurrent aulo-encoder and recurrent probability model. IEEE Journal of Selected Topics in Signal Processing, 15 (2):388-400 -018888411,22 2020b.
Yibo Yang, Robert Bamler, and Stephan Mandt. Improving inference for neural image compression. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020c.
Yunfan Zhang, Ties van Rozendaal, Johann Brehmer, Markus Nagel, and Taco Cohen. Implicit neural video compression. In International Conference on Learning Representations (ICLR) Workshops, 2022.
Nannan Zou, Honglei Zhang, Francesco Cricri, Hamed R Tavakoli, Jani Lainema, Miska Hanmuksela, Emre Aksu, and Esa Rahtu. L Learning to learn to compress. In IEEE International Workshop Mulli- media Signal Processing (MMSP), pp. 1-6. IEEE, 2020.
21

---

Published in Transactions on Machine Learning Research (06/2023)
A Datasets
A.1 UVG-1k
Following Wu ct al. (2018): : Lu et al. (2019); Golinski et al. 2220 Agustsson et al. 2020 report results
on the UVG-1k dataset, which consists of seven test video sequences totaling 3900 frames captured at 120 fps and natively available in 1080p resolution. Note that these sequences are only a subset of the full UVG
dataset (Mercat et al., 2020).
A.2 Xiph-5N
We use the Xiph-5N dataset introduced in van Rozendaal et al. 2021 It consists of the following five sequences from the Xiph.org collection: "in to tree", "aspen", "controlled burn", "sunflower", and "pedes- trian area".
A.3 Big Buck Bunny (BBB-E and BBB-H)
From the Big Buck Bunny vidco, which is also part of the Xiph.org media collection, extract two subso- quences of 10s length. They are chosen based on the frame-wise RD loss achieved by a global SSF18 model (with = 0.0016). We select the 10s window with the lowest RD loss and the 10s window with the highest RD loss, excluding the intro and outro of the vidco. These signify the "casiest" and "hardest" parts of the video to compress and refer to them as "BBB-E" and "BBB-H", respectively. In Fig. P we show the frame-wise RD loss and the selected sequences.
B Model architectures
B.1 Full-size SSF model
We first apply instance-adaptive video compression to the scale-space flow (SSF) architecture proposed by Agustsson et al. (Agustsson et al. 2020) using our own re-implementation of the model. Groups of pictures
0.0015
Loss per GoP
10s rolling average
Intro
BBB-E
BB8-H
Outro
0.0010
RD loss
0.0005
0.0000
2000
4000
6000
8000
10000
12000
14000
Big Buck Bunny frame
Figure Temporal distribution of the RD loss of the global SSF18 model for the Big Buck Bunny video. We use the 10s windows with the lowest and highest loss as benchmarks, excluding the intro and outro of Big Bunny. These subsequences, which call "BBB-F" and "BBB-H" for "easy compress" and "hard to compress", marked in orange and red.
22

---

Published in Transactions on Machine Learning Research (06/2023)
Table PSNR ranges used to compute BD-rate savings for each dataset.
Dataset
PSNR range
UVG-1k
33.5 < PSNR < 40.4dB
HEVC class B
30.3 < PSNR < 37.4dB
HEVC class C
27.2dB < PSNR < 35.4dB
Xiph-5N
31.4 S PSNR S 39.5
BEB-E
29.3 < PSNR S 37.0dB
BBR.H
31.4 S PSNR 39.5
Table 7: Architecture differences between the original SSF model published by Agustsson et al. 2020 and our SSF18 re-implementation. Convolutional layers are indicated as "conv kkerel size| [layer type [stridc] foutput channels]". J is used for convolutional layers, while t is used for transposed convolutions. The hyperdecoder is modeled as a Gaussian p(22/21) = (2(22)((21) a(1)), (2))) where the mean and standard deviation are modeled by two separate neural networks μ(z1) and ((21) with largely identical architectures. These networks differ in their activation functions, activations that are only used in one of the two networks are indicated in the table by their respective symbol. Note that we always clamp (=1) the range [0.11, 00).
Agustsson et al. (Agustsson et al. 2020
Our reimplementation
Hyperencoder (=1
conv 5x5 42 192 ReLU conv 5x5 42 192 ReLU
conv 3x3 $1 192 ReT.U conv 5x5 $2 192 ReLU conv 5x5 42 192
conv 5x5 42 192
Hyperdccoder p(22(21)
conv 5x5 12 192
conv 5x5 12 192 ReL.U conv 5x5 12 192 ReLU
ReLUU a: QReLU
conv 5x5 12 192 ReLU, a: QRLLU conv 5x5 12 192 QReLU
conv 3x3 f1 192 ReLU
(GoPs) consist of 12 frames; the first of them is coded as an I-frame and the remaining 11 as P-frames, each using the previous frame as reference frame. We mostly follow the setup described in Agustsson et al. (2020), but there are some minor differences. In the hyperencoder and hyperdecoder networks we use a slightly different architecture. We list the architectures Tbl. 7 In addition, our implementation of the blur stack and of the scale-space-fiovs warping operation differs from the one used by Agustsson al. (2020). but have verified in experiments that this has negligible effect.
We empirically validated our SSF re-implementation by comparing its performance to the original SSF (using entropy-codings On the UVG-1k dataset, our SSF18 model performed slightly better than the results published by Agustsson et al. 2020
B.2 Smaller SSF models
In addition to the full-size SSF18 model, we consider three smaller scale-space flow architectures. They follow the same structure as SSF18 in Tbl. 7 but differ in the number of latent variables and convolutional channels, as shown in Tbl. g We refer to them as SSF8, SSF5, and SSF3 after the number decoder-side network parameters (in millions..
B.3 B-EPIC model
As second base model we use the B-frame codec developed by Pourreza & Cohen 2021 (B-EPIC). We follow the hierarchical IBP configuration presented in that paper and use the model trained by the authors. During training, GoP consists of 4 frames arranged in IBBP pattern. At test time, GoP size
23

---

Published in Transactions on Machine Learning Research (06/2023)
Table 8: Architecture details for our scale-space- models. In the top, we show the number of channels for the latent and hyperlatent variables, 2, which are the basis for the compressod code; in the bottom, we show the number of channels in the codec and hypercodec networks in the encoder and decoder. We use the same model architecture and size for I-frame, scale-space-flow and residual networks.
Model
SSF18
SSFS
SSF5
SSF3
latent channels
192
192
192
128
hyperlatent channels
192
192
192
192
codec channels
128
hypercodec channels
192
of 12 frames and the following reference structure (where frame refers to the last frame of the previous GoP):
Frame 12:
P-frame referring to frame 0
Frame 6:
B-frame referring to frames 0 and 12
Frame B-frame referring to frames and
Frame 9:
B-frame referring to frames 6 and 12
Frame 1:
B-frame referring to frames and
Frame B-frame referring to frames and
and so on for the remaining frames. The very first frame in a video is encoded as an I-frame. Encoding and decoding this order ensures that the reference frames will always be encoded or decoded before the frames for which they are needed.
B.4 Quantization settings
Ablation studies showed insensitivity to network prior settings in the ranges of 10 < a S 1000, 0.0005 I < 0.005, and 0.01 < S 0.05. For the experiments in this paper we use a bin width 0.001, a 0.05,
t/6, spike-slab ratio a 100, and number of quantization bins of 289.
B.5 Entropy coding
We implement an entropy coder for the latents and model updates in the SSF and InstA-SSF models. We verified that this achieves bitrates in agreement with the probability mass functions of p(2) and p(6). For the B-EPIC and Inst.-BBEEPCC models, we report bitrates based probability mass functions without actually implementing entropy coder.
C Classical codecs
We generate H.265 and H.264 results using version v3.4.8 of ffmpeg (Fmpegg. When comparing neural codecs in the P-frame setting, also choose a low-latency setting that only allows I-frames and P-frames and fix the GoP size to 12. By default, we use the encoder presets "veryslow"; to study the trade-off between encoding compute and compression performance we vary it between "veryslow" and "ultrafast". Example commands are
fimpeg -pix_fm fmt yuv420p -s 1920x1080 -i raw vaw_video.yuv
-c:v 1ibx264 -preset veryslow -crf 23 -tune zerolatency
24

---

Published in Transactions on Machine Learning Research (06/2023)
Architecture ablation on in_ tree
GoP ablation on Xiph-5N
38
40
36
34
PSNR [dB.
PSNR
InstA-SSF18
36
32
InstA-SSF8
InstA-SSF5
InstA-SSF3
SSF18 (Agustsson et al.)
InstA-SSF18 (GoP size 12)
30
SSF8
InstA-SSF18 (GoP size m)
SSF18 (GoP
SSF3
SSF18 (GoP size w)
28
32
0.0
0.2
0.4
0.6
0.8
1.2
1.4
0.2
0.6
0.8
Rate [bpp,
Rate [bpp. 4]
Figure Left: Model size ablation study. We compare several global and instance- finetuued scale-space- flow models, showing the performance the in to tree sequence (Xiph.org). Right: Efffect of the GoP size at test time. We show rate-distortion curves of instance-adaptiv and global SSF18 models on the HEVC Xiph-5N dataset, comparing fixed GoP size of 12 (solid lines) to an infinite GoP size (dotted lines).
-x264-params output.mkv
for H.264 and
ffmpeg "pix_fm fmt 4uv420p -s 1920x1080
120 -i paw_video.yuv
-c:v libx265 -preset veryslow
-crf 23 -tune zerolatency -x265-params htap::/m-..mmm-kkiiit--11....eee output.mkv
for H.265. We also compress the test sequences with the HM reference implementation of HEVC, using the "JowlatencyP" setting with fixed GoP' size of 12.
As baselines for the neural B-frame codees, also relax the GoP and frame-type restrictions for the classical codecs, using the default random-access settings with B-frames and variable GoP size.
D Ablation studies
D.1 Model size ablation
In the left panel of Fig. we compare the compression performance of globally trained SSF18, SSF8, SSF5, and SSF3 models, as well as instance-adaplive versions InstA-SSF18, InstA-SSFs, InstA-SSF5, and InstA- SSF3. Due to the training time requirement we limit this ablation to a single video, the "in to tree sequence from the Xiph.org collection. We find that the larger models lead a better compression perfor- mance, with more pronounced difference in the high-bitrate region. SSF3 and InstA-SSF3 models perform poorly at high bitrates. We attribute this partially to the small size of the latent space compared to all other
25

---

Published in Transactions on Machine Learning Research (06/2023)
Framerate ablation Bosphorus (UVG-1k)
Big Buck Bunny, hard sequence (BBB-H), P-frame setting
43
40
PSNR [dB. t
39
PSNR
37
34
InstA-SSF18, scene-wise finetuning
InstA-SSF18, video-wise finetuning SSF18 (Agustsson 20)
mnstA-SF18-112122
35
InstA-SSF1B-30fps
H.265 (HM)
SSF18-120fps
H.265 (ffmpeg)
SSF18-30fps
H.264 (ffmpeg)
33
30
0.00
0.05
0.10
0.15 0.20 0.25 0.30 0.35 0.40 0.45
0.0
0.2
0.4
0.6
Rate [bpp.
Rate [bpp.
Figure 8: Left: Framerate ablation study. We compare the SSF18 baseline (orange) and InstA-SSF18 (red) the original video at 120 fps (solid) and the temporally subsampled video at 30 fps (dot-dashed). adapiivee compression increases robustness to variation in framerate. Right: Full-video finetuning ablation study. We compare the performance of an SSF model finetuned on the full Big Buck Bunny video (10 minutes) to a SSF model finetuned only on the BBB-H scene (10 seconds) when evaluated on the BBB-H scene.
models. In general, observe that instance-adaptive training improves the performance of models of any complexity. Instance adaption reduces the gap between smaller models and the full-size model, confirming that less expressivity is necessary for good performance on a single instance compared to on the entire data distribution.
D.2 GoP size considerations
Throughout this work, we use a GoP size of 12 at test time. To test the robustnoss of this choice, in the right panel Fig. we compare an infinite GoP size (for the SSF model). In other words, we compress the first frame as I-frame and all remaining frames as P-frames. The overall difference slim, though the gap is larger for some vidcos than others.
D.3 Sensitivity framerate variations
To further quantify our robustness to test data characteristics, we investigate the effect of the video framerate the model performance. We study the performance of global and mnsaacceadaaptve SSF models temporally subsampled videos, e. keeping one out of every four frames. To save computational resources, this ablation only performed on the "Bosphorus" sequence from the UVG-lk dataset. The left panel of Fig. S shows our results. While the SSF18 performance varies greatly between the original and the subsampled video, the difference in performance much smaller for InstA-SSFIs 18. In addition, both models perform better in the 120 fps scenario due to the higher rodundancy across frames.
26

---

Published in Transactions on Machine Learning Research (06/2023)
InstA-
InstA-SSF18, Big Bunny (easy sequence)
100%
80%
80%
Fraction of the ratalrate
Fraction of the total tatal rate
Model updates
60%
Residual latents
60%
dual hyperiatents
Flow latents
Flow hyperiatents
40%
I-rrme latents
80%
Irramne hyper hyperlatents
20%
20%
0%
0.1
0.2
0.3
0.4
0.5
0.1
0.2
Total rate [bits/ / ppaell
ratel [bits pixel]
Figure 9: Bitrate allocation for our InstA-SSF18 model. We show the relative contribution to the overall bitrate from quantized latents for the I-frames, P-frame flow, and P-frame residuals (different shades of blue). In red we show the relative contribution from the quantized model updates. Left: on the UVG-1k dataset, which has similar content to the training data. Right: on the two Big Buck Bunny scenes, which differ substantially from the training data.
D.4 Bitrate allocation
Figure disentangles the various contributions to the total bitrate. We find that the overhead from model updates is generally small (typically well below 1 %), except on videos that differ substantially from the training data: on the animated Big Buck Bunny sequences, Inst learns to adapt more strongly and up to % of the total rate are spent on model updates.
We further break down the model rate in Fig. 10 where we show how the model rate is distributed over the various auto-encoders and submodules of our InstA-SSF18 and InstA-SSF5 model. Tt can be seen that the InstA-SSF5 model has a lot fewer parameters per module by design, but it is making up for its smaller capacity by spending lot more bits/per parameter on model updates compared to InstA-SSF18 (often more than double). This explains why the rate-distortion performance of InstA-SSF5 is close to that of InstA- SSF18. Even though the InstA-SSF5 model needs more bit per parameter to adapt to the data instances, the total bitrate spent on model updates still smaller than that of InstA-SSF18.
Focusing on the distribution of the model bitrate across auto-encoders, we see that more than half of this rate is allocated to the I-frame model. The flow autoencoder is the next biggest consumer of model rate, while only few bits spent to communicate updates to the residual autoencoder. In terms of submodules, it turns out that even though the hyper-decoder has a lot more parameters than the decoder, the decider receives substantially larger update bits per parameter. Together, these findings suggest that instance adaptation is most important for network components that are closer to the data space in the computational graph.
D.5 Update sparsity analysis
The key design for achicving low model rate is in update sparsity. We achieve this by using a spike-and- slab prior around global model parameter values. Training with the spike-and-slab prior is sparsity enforcing (on average around 0.8 % all quantized parameters receive a non-zero update). To quantify savings of our chosen spiko-and-slab prior, we compute the cross-entropy of our prior with the resulting model after training. This is the number of bits we need to transmit the model update. On the other hand, assume we "slab" component, and assuming the resulting 8 follows
had Gaussian prior with the same variance
27

---

Published in Transactions on Machine Learning Research (06/2023)
InstA-SSF18 (136.9 kB)
InstA- SSF5 (105.2 KB)
I-frame
Flow
Residual
I-frame
Flow
Residual
Hyper
108 B
149
193
152
0.0
Hyper
0.1
16.2 kB
12,0 kB
12.5 KB
8.7
4.7 KB
KB
0.0
47.6 KB
0.7
0.6
(bits/parameton)
0.5
Decoder
0.4
31.9 KB
53.6 kB
6.6 KB
0.1
33.9 KB
0.0
1 #params (M)
#params (M)
Figure 10: Model bitrade allocation for our InSLA-SSF18 (left) and InsLA-SSF5 (right) model. We show how the model rate distributed across each autoencoder (columns) and submodule (rows). All plots use grid of the same size, where width corresponds to the number of parameters in that module, height corresponds to the average patssparrmeter used for updating that module, and area corresponds to total number of bits allocated for that module (each block in the grid has an area of 12.5 kB). Results are averaged over all values of B and all datapoints in the UVG-lk and HEVC class dataset.
28

---

Published in Transactions on Machine Learning Research (06/2023)
the same distribution, the entropy of this Gaussian prior gives the expected number of bits to transmit non-sparse model update. We estimate the difference between the two scenarios to be around 7.5 bits per parameter.
D.6 Finetuning strategy for longer videos
For longer videos with multiple scenes, one can consider several different strategies for instance-adaptive finetuning. The simplest approach is finetuning single model on the full video. It may be beneficial to instead finetune the model independently on each scene. Even a hierarchical approach in which a model finetuned on the full video is further finetuned on each scene an option.
The optimal strategy will in general depend on the particular video content and the similarity of the scenes well as the bitrate setting. We perform simple first test by finetuning single model the entire Big Buck Bunny video, 10 min video (14315 frames) with multiple scenes united by common style. The results in the right panel of Fig. 8 show that such single finetuned model leads to clear RD improvements over the global SSF model. However, in the large-bitrate region these improvements are smaller than the gains we achiove by finetuning on the BBB-E and BBB-H scenes individually.
E Broader Impact Statement
Video compression affects the life of millions of people and any algorithm and model should be carefully analyzed for potential ethical issues before deployment. In particular, it important to check for harmful biases, which may be due imbalanced training data, but can also be caused or exacerbated by algorithmic choices. To some extent, the instance-adaptive vidco compression mothod we are proposing may be helpful in combating such biases, as it can improve the compression performance data that is different from typical samples in the training set. However, it cannot solve the problem entirely, more pronounced instance adaptation will incrcase the bitratc.
The environmental impact of our research is also relevant. The energy cost of our experiments directly contributes to the climate catastrophe. In the long term, our work may also have a beneficial aspect, as it allows us to lower the bitrate and thus the energy cost video strcaming, which already now make up the majority of internet traffic.
F Reproducibility statement
Our instance-adaptive video compression algorithm is described in detail in Sec. The most complex components are the base compression models, which are described in more detail in the references we cite well as in appendix With the details provide, instance-adaptive video compression can be implemented and used for any base model with relatively little effort. Finally, in the supplementary material we provide the performance of the various methods on each video sequence as a CSV file to aid comparisons.
29


Reviews:
[{"review_id": "sK0Scsc1hl", "reviewer": "Reviewer_6RHk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Compared with the previous methods, the proposed method achieves quite a decent improvement of the rate-distortion performance.\n2. The method is model-agnostic and could be applied to any codecs.\n\nWeakness:\n1. The writing could be improved. (See requested changes)\n2. The novelty in the method is incremental. Most of the part is based on SSF18.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new instance-adaptive video compression method. The basic assumption of this method is that there could be domain gap between the training data and the test dat such that the model that is well fitted on the training data does not generalize well on the test video. The key contribution is to finetune the codec for each video instance. Then both the latent code of the video, and the parameter updates of the decoder and prior are transmitted. To reduce the bitrates for the parameter updates, a spike-and-slab prior is assumed for the updates. In addition, the updates are also quantized. In the decoder side, the updates of the parameters are first decoded. Then the decoder and the prior network are updated accordingly. After that, the video is decoded with the updated network.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The figures and tables should be placed near the text in the paper. In the current version, it is difficult to relate the figures and the explanation.\n2. The bitrate and decoding overhead should be fully analyzed. In Table 2, there are 4 versions of SSF.  \n3. Figure 2 should be better explained. For example, why are some of the curves not monotomous?  What does percentage on the y-axis mean? Why are most of the curves descreasing? Should a higher relative rate correspond to a higher PSNR?\n4. What is the distribution of the updates? It is better to visualize the distribution.\n5. How is the spike-and-slab prior enforced? Is it used as a regularization term?\n6. What is computational cost to finetune the codecs?\n7. Although this work is about data compression, network compression is closely related. For example, the network updates should be quantized. Thus, there are two questions regarding that. \n1) In the related works, network compression techniques should be discussed a little bit [1-3].\n2) Is it possible to compress the network during the finetuning of them to achieve a reduction of both parameters and computational cost [1-3]?\n\n[1] Li, Y., Dong, X., & Wang, W. (2019). Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks. arXiv preprint arXiv:1909.13144.\n\n[2] Li, Y., Gu, S., Mayer, C., Gool, L. V., & Timofte, R. (2020). Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8018-8027).\n\n[3] Liu, Z., Mu, H., Zhang, X., Guo, Z., Yang, X., Cheng, K. T., & Sun, J. (2019). Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 3296-3305).", "broader_impact_concerns": "The authors discussed two issues about the broader impact. The first one is to reduce the biases due to insufficient training data. The second one is the energy reduction due to the reduced bitrate. Yet, I'm interested in the cost to finetune the models for each instance. Is that overhead acceptable?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TxMv8fd9qh", "reviewer": "Reviewer_DBHN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The proposed method is well understood and the code is easily reproducible.\n2. The performance improvement is significant and the property that the number of neural network model parameters can be saved is valuable.\n\nWeaknesses\n1. This work is migrated from a similar work on image compression, lacking some thought about the optimization of video compression methods.\n2. Fine-tuning the model on the test set is a relatively naive approach at this moment, especially with the advent of implicit neural networks. The introduction of the finetuning time on test set, and extra complexity of both encoder and decoder side makes the method look less friendly. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an instance adaptive video compression method, by finetuning a pretrained neural compression model on the video sequence to be transmitted. The finetuned network parameters are compressed and transmitted along with the latent code under the spike-slab update prior. The proposed method is proved to be able to improve the performance and can also reduce the requirements of the model parameters and MACs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The paper writing should be improved. There are many language organization confusion and typographical problems.\n2. The introduction part is too simple, which should explain clearly the motivation of this work. In addition, the introduction of the concept of instance learning is also missing.\n3. There are many experiments in this article, but a comparison with similar \"fine-tuning\" methods is missing to prove the superiority of the proposed one.\n", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "G6gVomjDeN", "reviewer": "Reviewer_J84y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strengths\n- The paper is well-written.\n- The evaluation has been performed in the B-frame and P-frame settings, showing that the approach is working in both settings. \n- The improvement is not equally large in all settings. But limitations, such as that the model cannot compete with traditional methods in case of difficult out-of-distribution examples have been discussed. It still performs better than the learning-based baselines.\n\n#### Weaknesses\nThe paper could benefit from some clarifications:\n- Abstract claims that the proposed algorithm is model agnostic. However, the method section assumes a VAE structure. This could be more precisely formulated.\n- The method builds upon the image-based approach van Rozendaal et al. (2021). Section 3.4 could be clearer about what is part of the related paper and which modifications were needed to adapt it to the video setting to make the technical contributions clearer.  \n\nBecause the method is instance-adaptive, it requires finetuning for each instance. Section 4.3 mentions that this finetuning is performed for two weeks for the main results, which is very expensive to encode a single 2.5-10s sequence. This makes me wonder about the practicality of the method. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method to improve the bitrate-distortion performance of learning-based video compression algorithms using instance-adaptive learning. The parameters of the variational auto-encoder for compression are finetuned on the specific video sequence to compress and the parameter updates of the decoder are transmitted in an efficient way along the compressed video. The method has been evaluated on various datasets and in comparison to reasonable baselines showing significant BD-rate savings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Some clarifications as metioned in weaknesses.\n- Explaining/discussion the expensive finetuning time.\n- Typo: Caption Figure 8: emphLeft -> \\emph{Left} ?\n- Minor: Adding the used GPU in the caption of Figure 5.\n- Given that the encoding time is one of the main concerns, it would be interesting to add the analysis of the improvement wrt. to the encoding also for the B-frame setting (analog to figure 5).", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three expert reviewers review this paper. The discussions with the authors clarify most of the initial concerns (as outlined by the authors in the "Changes Since Last Submission" section). The revised manuscript has been significantly improved. All three reviewers are satisfied with the revision and recommend learning accept. The AE reads the reviews, the discussions, and agrees with the reviewers to recommend accepting the paper.
Decision: Accept (poster)