Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set
Authors: Ties van Rozendaal
Johann Brehmer
Yunfan Zhang
Reza Pourreza
Taco Cohen
Venue: ICLR 2022 Submitted (2022)
Keywords: Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set

Abstract: We introduce a video compression algorithm based on instance-adaptive learning. On each video sequence to be transmitted, we finetune a pretrained compression model. The optimal parameters are transmitted to the receiver along with the latent code. By entropy-coding the parameter updates under a suitable mixture model prior, we ensure that the network parameters can be encoded efficiently. This instance-adaptive compression algorithm is agnostic about the choice of base model and has the potential to improve any neural video codec. On UVG, HEVC, and Xiph datasets, our codec improves the performance of a low-latency scale-space flow model by between 24% and 26% BD-rate savings, and that of a state-of-the-art B-frame model by 17 to 20% BD-rate savings. We also demonstrate that instance-adaptive finetuning improves the robustness to domain shift. Finally, our approach reduces the capacity requirements on compression models. We show that it enables a state-of-the-art performance even after reducing the network size by 72%.

Under review as a conference paper at ICLR 2022
INSTANCE-ADAPTIVE VIDEO COMPRESSION: IMPROVING NEURAL CODECS BY TRAINING ON THE TEST SET
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce video compression algorithm based on instance-adaptiv learning. On each video sequence to transmitted, we finetune a pretrained compression model. The optimal parameters are transmitted to the receiver along with the latent code. By entropy-coding the parameter updates under a suitable mixture model prior, we ensure that the network parameters can be encoded efficiently. This andapice-adaptive compression algorithm is agnostic about the choice of base model and has the potential to improve any neural video codec. On UVG, HEVC, and Xiph datasets, our codec improves the performance of a low-latency scale- space flow model by between 21 % and 26 % BD-rate savings, and that of state- of-the-art B-frame model by 17 to 20 % BD-rate savings. We also demonstrate that instance-adaptive finetuning improves the robustness domain shift. Finally, our approach reduces the capacity requirements on compression models. We show that enables competitive performance even after reducing the network size by 72 %.
INTRODUCTION
Neural compression methods have enormous potential to improve the efficiency of video coding. With video constituting the large majority of internet traffic, this has significant implications for the internet at large (Sandvine) 2019). State-of-the-art algorithms model each frame by warping the previous one with a neural estimate of the optical flow (Lu al. 2019) or scale-space flow (Agusts-) son et al.] 2020) and adding residuals modeled by another network. Both the optical flow and the residuals are compressed with variational autoencoders. Such neural codecs have recently achieved results par with popular classical codecs (Agustssoon al. 2020 Rippel et al. 2021 PPuureza & Cohen 2021) such as H.264/ AVC (Wiegand et al.) 2003) and H.265/HEVC (Sullivan et al. (012)). However, relatively little research has focused on their computational complexity, and matching the rate-distortion performance of H.266 VVC (Bross et al. 2018)) is still an open problem.
Neural video codecs depend critically on generalization: they are developed assuming that good performance on training and validation datasets translates to good performance at test time. How- ever, this assumption does not always hold in practice, for instance because of limited training data or imperfect optimization. Domain shift is a particularly challenging problem given the varicty in video content and styles; for instance, neural video codecs trained on natural scene video data often perform poorly on animated sequences (Agustsson et al. 2020).
Here we address this problem by optimizing the autoencoder finetuning its weights- for each single video to be compressed at test time. After training on the test set in this way, the finetuned network parameters need to be made available on the receiver side: they are compressed, quantized, and entropy-code under a suitable prior to the bitstream along with the latent code. Optimizing the compression network for every video this way relaxes the requirement of training data that closely matches the test distribution as well as the necessary model capacity. It also allows trading off encoding compute for better compression efficiency.
The essential idea of instance-adaptive finetuning was recently proposed by van Rozendaal et al. (202T). The authors demonstrated the idea on I-frame compression, i.e. compressing a set of sim- ilar images, but did not apply it video sequences yet. In this work extend the idea to the

---

Under review as a conference paper at ICLR 2022
compression of full videos, in which only periodic key frames are compressed images and most frames are encoded relative to one or two reference frames. Our method is general and can be applied to various settings and base models. We first demonstrate it in a low-latency setting with I-frames and P-frames using scale-space flow architecture (Agustsson et al. 2020) as base model. Next show its performance in a B-frame setting, using the base model proposed by PPurrezaa & Cohen (202)).
On the UVG-IK, HEVC class-B, and Xiph-5N datasets, our new instance-adaptive video compres- sion codec yields BD-rate savings of 17 to 26 % over the respective base models and 5 to 44 % over the popular flipeeg (x265) implementation of the H.265 codec (FFmpeg, VideoLAN b). In addi- tion, instance-adaptive finetuning lends itself to possible reduction in model size, because smaller network may suffice model a single instance. We show that this framework smaller models can still outperform most neural codecs while reducing the computational complexity of the decoder by 72%. Overall, we demonstrate that unlike other neural video codecs, our method can trade off encoding compute against compression performance, and that this trade-off is more effective than for classical codecs.
RELATED WORK
Neural video compression The standard framework used by most neural compression codecs (either implicitly or explicitly) is that of the variational (Kingma & Welling) 2014) or compres- sive Theis et al. 2017 autoencoder. An encoder (or approximate posterior) go ( (r) maps a data point 2 latent representation 2. This latent is transmitted to the decoding party by means of entropy coding under a latent prior (2))). The receiver can then reconstruct the sample with a de- coder (or likelihood) po (arlz) The encoder, prior, and decoder are neural networks, parameterized by weights Φ, 0 as indicated by the subseripts.
These models are trained by minimizing the rate-distortion loss
LRD( 0) = Ex-p(x) 3 E Egmgg (21x) log po (=))
Exwgs((12) |- log po (a|2))
(1)
R₂
which combines distortion metric D and a rate term R. that approximates the bitrate necessary to transmit the latent code 2. Up to a constant entropy term H [go]. this loss equals the 3-weighted VAE loss (Habibian et 2019 Higgins et al. 2017).
Much research has focused on the design of efficient network architectures for neural video codecs, in particular with respect to the temporal structure. While some earlier works were based on 3D convolutions (Habibian et al. [201 Pessoa et al. 2020). recent models use optical flow and residual modeling to exploit similarities between frames. The latter class of approaches can be divided into predictive or P-frame temporal modeling, where the model for each frame is conditional on the previous frame(s) (Agustsson et al.] 20200 Chen et al.] 2019 Liu et al., 2020: Lombardo et al,, 2019 Lu et al. 2019 Rippel et al. 2019 Golinski et al. 2020
Rippel et al. [202]] Hu et al. [2020] where each frame is conditioned on
Yang et al. 2020b), and bidirectional or B-frame modeling,
past as well
future reference frames (Cheng al. 2019 Choi & Bajić 2019 Djelouah et al.;
2019 Wu et al., 2018; Park & Kim 2021 Pourreza & Cohen 2021, Yang et al. 2020a).. Agustssoo
al. 2020) generalize optical flow scale-space flow by adding dynamic blurring to the warping operation, improving the modeling of uncertainty and leading to state-of-the-art results.
Instance-adaptive compression Rate-distortion autoencoders are trained by minimizing the RD loss Fq. (T) over a training dataset D. This approach relies on the assumptions that the model not only able to fit the training data well, but also generalizes to unseen datapoints at inference time. In practice, however, finite training data, limited model capacity, optimization imperfections, and domain shift s differences between train and test distributions
degrade the performance.
This problem can be solved by adapting video codecs to each sequence at test time. Several such methods have been proposed, see Appendix A in-depth overview that also covers image com- pression. These works can be roughly categorized into encoder-only finetuning, limited decoder finetuning, and hybrid classical-neural codecs. Works in the first category (e. g. [u et al. (022
2

---

Under review as a conference paper at ICLR 2022
optimize the encoder parameters & for each data point. Such update does not have to be commu- nicated to the receiver side. This approach alleviates the generalization problem for the encoder, but does not solve it for the decoder and prior. We will demonstrate later that encoder-only finetuning leads to a limited improvement in the compression performance.
The second category of works also adapt parts of the prior and decoder for each video instance. Since these modules need to be known on the receiver side, such an update needs to be signalled in the bitstream. Depending on the implementation, this may increase the bitrate substantially, [Aytekin et al. (2018); Zou et al. 2020); Wang et al. (202T) address this by only allowing limited changes the decoder-side models. for instance choosing one out of number of fixed decoder networks, thus limiting the potential RD gains of these methods. Finally, the third category of works propose hybrid approach, where a video first compressed by a classical codec and finetuned enhancement network is added to the bitstream (He et al. 2020 2021 Klopp et al. 2020 2021 Liu et al. 2021 Although these methods have shown great performance at low bitrates, they tend to perform less well at higher bitrates, and do not benefit from developments in end-to-end neural compression.
van Rozendaal et al. (2021) suggest a method to adapt the full model to a single datapoint. The key idea is that the parameter updates from training on D, can be be transmitted very efficiently. After discretization, the quantized updates 8 are entropy-coded under parameter update prior p|o] that assigns high probability mass (and thus low transmission cost) to zero-updates = O. This procedure enables a neural codec in which the full model encoder, decoder, and prior- is finetuned minimizing the loss
0 0D. where Op are the global model parameters resulting
each data sample test time by
log CimuA (ó, 8) = Cuv(d. 0D + 5) + (- po)))
(2)
Ra
The Rg term reflects the increased bitstream length from coding the parameter updates 5. In van Rozendaal et al. (202I), this approach was successfully demonstrated on I-frame compression, es- sentially compressing a set of images with similar content. In this work we adapt it the compres- sion of full video sequences, as we will describe in the following section.
3 INSTANCE-ADAPTIVG VIDEO COMPRESSION
We now introduce short. The key idea is to optimize the parameters a rate-distortion VAE for each video sequence be transmilted, and to send the relevant network parameters in rate-efficient way to the de- coder. This approach can be beneficial whenever a standard rate-distortion autoencoder would not generalize well, for instance because of limited training data domain shift. In addition, instance adaptation allows us to use smaller models while maintaining most of the compression performance. While encoding the network parameters to the bitstream increases the length of the code, we describe a compression scheme in which this cost is negligible, especially when amortized over many frames of a video sequence.
aistance-adaptive compression codec for full video sequences, InstA for
Our approach is agnostic about the base model and we demonstrate it on two different architectures: a scale-space flow model the low-latency setting and the B-EPIC model in the B-frame setting. In the following, we first describe the base models, before describing how InstA compresses and decompresses video sequences.
Low-latency base model: Scale-space flow (SSF) We first focus on the low-latency setting, which we define as having only access to current or previous frames when decoding frame. As base model for this setting, we use the scale-space-flow architecture introduced by Agustsson et al. (2020). Any video sequence is first split into groups of pictures (GoP). The first frame in each GoP is modeled as an image without any dependency on previous frames, e. as an I-frame. All other frames are P-frames, modeled as
Saspppcce-aa((r---.... +r,,
where r;-1 is the reconstructed previous frame, 9i is the estimated scale-space flow field, r; is the estimated residual, and the scale-space warping operation performs optical-flow warping of the previous frame with dynamic, position-dependent amount of Gaussian blur. The I-frame images,
3

---

Under review as a conference paper at ICLR 2022
Table Parameter counts and decoding complexity for our various global scale-space-flow models.
Model
SSF18
SSF8
SSF5
SSF3
Total parameters [10%]
28.9
13.8
8.1
Decoder-sie siooder-side params. [10%]
18.0
8,4
5.0
3.1
Decoder-side kMACs/ pixel
313.4
180.0
88.6
50.8
Decoder kMACs reduction
0%
43 %
72
84 %
P-frame scale-space flow gis and P-frame residuals т; are compressed with separate hyperprior models (Ballé et al., 2018), hierarchical variational autoencoder architecture.
We largely follow the architecture choices of Agustsson al. (2020) and refer to this model as SSF18 after its number of decoder-side parameters (in millions). In Appendix C we describe the architecture detail and highlight the differences to Agustsson et al. (2020).
Smaller low-latency base models As instance-adaptive approach is based finetuning the compression model on a low-entropy "dataser" (a single video sequence), we hypothesize that does require the full expressivity of the computationally complex SSF18 model. We propose three alternative scale-space flow architectures with a reduced computational complexity. In Tbl. list the parameter counts and the number of multiply-accumulate (MAC) operations required for decoding a sequence, which are reduced by 43 43-84% compared to the SSF18 baseline. The architectures of the four SSF models are described detail in Appendix
We also consider less constrained setting, in which frames can be compressed as B-frames as well, i. e. using both a previous frame and a future frame as reference points. This flexibility allows for demand video streaming. As a base model we choose the B-EPIC architecture (Pourreza & Cohen 2021). Again, the video is split into GoPs. The very first frame of video is modeled frame, while the last frame in each GoP a P-frame, using the last frame of the preceeding GoP as reference; both I-frame and P-frames are compressed as in the SSF model.
B-frame base model: B-EPIC
more efficient compression and is used for instance for on-
I-
Any other frame I; is modeled as a B-frame: it is assigned a past reference frame IG Gj < i) and future reference frame Tk i); an off-the-shelf Super-SloMo frame interpolator (Jiang al. 2018) is used to interpolate between these two reference frames. The interpolated frame is then used as basis for scale-space flow warping:
T3 Shale-SpneeWarrpSSpperr
ri-
(4)
The interpolated frame provides a more useful starting point for optical-flow warping than the previ- ous frame, g. because the combination of past and future knowledge may avoid occlusion effects.
Again, I-frames, optical flow, and residuals are modeled with three separate hyperprior models and the model is trained on the RD loss in Eq. (T). We only consider single configuration and use hyperparameters and checkpoints from Pourreza & Cohen (202I). This model has 38.5 million parameters, 23.2 million of which are on the decoder side.
Encoding a video sequence Our procedure conceptually follows yan Rozendaal et al. (2021). For completeness shall describe the full method here. video sequence ær is compressed by:
1. Finetuning the model parameters (0, Φ) of the base model on the sequence x: using Eq.
2. computing the latent codes (=(x))
3. parameterizing the finetuned decoder and prior parameters as updates å = 0 0p.
4. quantizing latent codes and network updates and
5. compressing the quantized latents E and updates with entropy coding to the bitstream.
To finetune a pretrained base compression model, we start with the global model with parameters (Op, ΦD) and minimize the rate-distortion loss given in Eq. 2 but only over the single video sequence x. This modified rate-distortion loss explicitly includes the bitrate required to send model updates 5 under updates 6, but use the quantized parameter updates 0p + to calculate the and Rg loss terms, using a struight-trough estimator (Bengio al., 2013] in the backward pass.
update prior p(6). We compute the regularizing Ro loss with the unquantized
4

---

Under review as a conference paper at ICLR 2022
Sending the updated network parameters of course adds to the length of the bitstream. This is even the case when finetuning does not lead to changed parameters (6 = 0). Given the large size of the neural model we consider, it is therefore essential to choose an update prior that assigns large probability mass to zero-updates 5 = 0. This allows the network to transmit trivial updates at a negligible rate cost, while giving the freedom invest bit cost in non-trivial parameter updates that improve the performance substantially. We use a spike-and-slab prior (Johnstone & Titterington 2009 van Rozendaal et al. 202I), mixture model of narrow and a wide Gaussian distribution given by
p()) _V600, + +
(5)
1+a
where the "slab" component with variance a² keeps the bitrate cost for sizable updates down, and the "spike" component with the narrow standard deviation < a ensures cheap zero-updates. The mixing weight a is tunable hyperparameter.
At the beginning of the finetuning procedure, our neural model is equal to the global model. Because of the spike-slab update prior, the rate cost is only marginally increased and the compression per- formance essentially equal to the global model. During finetuning, the rate- rate-distortion performance gradually improves, giving us an anytime algorithm that we can stop prematurely to get the best compression performance within given encoder compute budget.
After the model has converged or a compute budget has been exhausted, we use the finetuned en- coder 9s to find the latent code corresponding to the video sequence z. Both and the updates to prior and decoder ó, which will be necessary to decode the video sequence, are quantized. To discretize the updates 8, we use fixed grid of equal-sized bins of width i centered around 6 = 0. We choose n such that p(-n/t < < n/t) = - é and clip values at the tails. The quantization of is analogous, except that we use bin width of t = 1 and do not clip the values at the tails (in line with Ballé et al. (2018).
Finally, we write the quantized updates ö and quantized codes the bitstream. We entropy coding under the quantized update prior p(ō) and finetuned prior Pa.. ((3), respectively. For latent codes in the tail region use Exp-Golomb coding (Wiegand et al. 2003
Decoding a video sequence The receiver first decodes the prior and decoder updates from the bitstream. Once 0 = 0p + 8 known on the decoder side, the latents are decoded with the prior Pz(2) and the video sequence is reconstructed with the decoder pa(z|z) following standard VAE protocol. The only overhead on the decoder side therefore the initial decoding of 5. While this results in a small delay before the first frame can be decoded, in practice this delay is very short (below 0.2 seconds in our experiments).
EXPERIMENTAL SETUP
Datasets In our experiments use sequences from five different datasets. The global models trained on Vimeo90k (Xue et al.] 2019). We evaluate on the HEVC class-B test sequences (HEVC) 2013), on the UVG-Ik (Mercat et al. 2020 dataset, and on Xiph-5N (van Rozendaal et al. 202 which entails five Xiph.org test sequences. The performance on out-of-distribatioo data is tested on two sequences from the animated short film Big Buck Bunny, also part of the Xiph.org col- lection (Xiph.org). We choose an "casy" (BBB-E) and a "hard" (BBB-H) clip of 10s each, see Appendix [ for details. All of these datasets consist of videos in Full-HI resolution (1920 x 1080 pixels). In addition, we evaluate on HEVC class C with lower resolution of 832 x 480 pixels.
Global models The scale-space flow models described in Sec. Bare trained with the MSE training setup described in Agustsson et al. (2020), except that we use the publicly available Vimeo-90k dataset. For the B-EPIC model, we use the model trained by Pourreza & Cohen (202I) I) on Vimeo- 90k. We describe the training schemes in detail in Appendix
Instance-adaptive finetuning On each instance, we finetune the models with the InstA objective in Eq. B using the same weight 3 as used to train the corresponding global model. We finetune for up to two weeks, corresponding to an average of 300 000 steps. Our training scheme is described in detail in Appendix D
5

---

Under review as a conference paper at ICLR 2022
Baselines As our instance-adaptive video compression method can be applied to any compres- sion architecture, we focus on comparing to the respective base models: we compare InstA-SSF to reimplementation of the SSF model (Agustsson et al. 2020) and InstA-B-EPIE to the B-EPIC model (Pourreza & Cohen 2021). To better understand where the benefits of instance-adaptive fine- tuning come from, we consider additional SSF18 baseline in which only the encoder is finetuned on each instance. Unlike our InstA method, encoder-only finetuning does not require sending any model updates in the bitstream. In addition, we show the strongest or most closely related neural baselines, with further baselines in Appendix F
We also run the popular classical codecs H.264 (AVC) (Wiegand et al. 2003 and H.265 (HEVC) (Sullivan et al. 2012) in the fimpeg (x264 / x265) implementation (FFmpeg) VideoLAN a b) as well as the (slower but more effective) HM reference implementation (HEVC) of H.265.
To ensure a fair comparison, in the low-latency experiments we restrict all codecs to I-frames and P-frames and fix the GoP size to 12. In the B-frame experiments, we use default settings, allowing the codecs to freely determine frame types and GoP size. For ffimpeg, we compare several different encoder presets to study the trade-off between encoding time and RD performance. In Appendix provide the exact commands used to generate these baseline results.
Metrics We evaluate the fidelity of the reconstructions through the peak signal-to-noise ratio (PSNR) in RGB space. To summarize the rate-distortion performance in a single number, we compute the Bjøntegaard Delta bitrate (BD-rate) (Bjontegaard, [200T) the relative rate difference between two codecs averaged over a distortion range. We evaluate the BD-rate savings of all meth- ods relative to the HM reference implementation of HEVC, using PSNR distortion metric, the intersection of the support of all RD curves as the distortion range, and cubic spline interpolation.
5 RESULTS
Compression performance Figure I shows the rate-distortion curves of our instance-adaptive video codec (InstA) as well as neural and traditional baselines. We compare to more baselines in Appendix F Both for SSF in the low-latency and B-EPIC in the B-frame setting, the instance- adaptive models clearly outperform the corresponding base models. Finetuning only the encoder which does not require sending model updates in the ditstream-een to a much more modest improvement over the base models.
In the low-latency setting, InstA-SSF outperforms all other neural models as well the ffmpeg implementations of H.264 and H.265 at all studied bitrates, except for OTU and ELF-VC, which hold an edge in the low-bitrate region, and M-LVC, which is competitive at medium bitrates. Within the restrictions of the low-latency setting and at medium to high bitrates, it is even competitive with the HM reference codec. In the B-frame configuration, InstA-B-EPHE outperforms all baselines from around 0.1 bits per pixel except HM.
Robustness to model-size reduction While the full-size InstA-SSFI8 model leads to the best RD performance in the low-latency setting, the smaller InstA-SSF5 models hold their own. Despite the 72% reduction in decoder complexity, Insta-SSF5 outperforms most neural baselines and the frimeeg codecs, and achieves performance close to that of InstA-SSF18. between the SSF5 and SSF18 baselines is larger. This provides evidence for the hypothesis that addtaieee finetuning reduces the capacity requirements on neural compression models.
comparison, the gap
To further investigate how the performance of our method depends decoder complexity, we train two additional models, SSF3 and SSF8, as summarized in Tbl. The results are in line with the trends already shown. While the performance of the global models degrades quickly when reducing the model size, instance- addppiv finetuning can compensate for the reduced capacity and offers more "bang for the buck" in terms of (decoding-side model complexity. For detailed results refer the readers to Appendix H
Robustness to dataset variation Globally trained neural codecs rely on the similarity of the test instances to the training data. In comparison, we expect finetuned models to be more robust under domain shift and to perform well across different test sets. To put this to the test, we evaluate the InstA-SSF models and low-latency baselines different sequences, including two animated

---

Under review as a conference paper at ICLR 2022
low latency
latency
42
38
36
PSWA [48, 13
InstA-55F (ours)
PSWR (40)
SSF18 (Agustsion 20)
InstA-SSFS (ours)
34
InStA-SFF5 tours)
InstA-SSF 18 (ours)
S5F5
(Agustsson 20)
36
Encoder-ture deddeer---edddeee SSF18
SSF5
Encoder-tunee
ELF-VC (Rippel 21)
DVC (Lu 19)
ILin
32
DVC.adattvve
CTU (Klopp 21)
MLVC (Un 20)
1:265 (HM)
H,265 (1IM)
H.265 (ftmpeg)
H.265 (ffmpeg)
H.264 (rimpeg)
H.264 (ttmpag)
Rate [app. 11
Rate [bpp, 4.
LVG-1k, B-frames
HEVC class B, B-framess
38
PSWR (4B, 13
p (30)
InstA-B-EPIC (ours)
B-EPIC (Pourreze 211
InstA-B-EPIC fours)
Encoder-tuned B-EPIC
(Pourreza 21)
HLVC (Yang 20a)
32
Encoden-tuned B-EPIC
18)
HLVC (Yang 20a)
H.265 (H84)
H.265 (HM)
H.265 (timpag)
H.265 (ffmpeg)
(ttopag)
(ttmpog)
37
30
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
Rate
Rate
Figure 1: Rate-distortion performance of InstA video codecs. InstA (red) leads BD-rate savings of between 17 and 26 over the corresponding base models (orange) and between 5 % and 44 % over H.265 (ffmpeg).
scenes from the Big Buck Bunny video. In Tbl. 2 report the compression performance BD- rate savings relative to the HM reference codec, the corresponding RD curves can be found in Appendix [F We find that InstA performs strongly
all datasets. On the animated sequences, the global SSF18 baseline performs poorly relative to ffmpeg H.265, showing its susceptibility to domain shift. Our addtaicee-daaiive models are much more robust this shift and close the gap with classical codecs. We illustrate this performance improvement on example frames from the Big Buck Bunny video in Appendix G and provide reconstructed video data in the supplementary material.
Instance Instance-adaptive compression is particularly powerful for high-resolution data, where the model- rate overhead amortized over more pixels. We test whether InstA still works a lower resolution by evaluating on the HEVC class C test sequences, which have 81 % less pixels per frame than the Full-HID sequences. As shown Appendix [ InstA indeed performs worse relative to the baselines than the higher-resolution data. Nevertheless, InstA-SSFI8 still is among the best neural codecs.
To further quantify our robustness to test data characteristics, we investigate the effect of the video framerate the model performance. In Appendix H we study the performance of global and saaunce-adaptive SSF models on temporally subsampled videos and again find that the instance- adaptive model is more robust to variations than the global SSF model. While these ablations do not cover all realistic variations, they serve as examples of dataset characteristics that global neural codec is likely sensitive to.

---

Under review as a conference paper at ICLR 2022
Table Rate-distortion performance InstA and baselines on several datasets. We summarize the rate-distortion performance as BD-rate relative to the HM reference implementation of H.265 in the respective setting (lower is better).
BD-rat rate relative to H.265 (HM) [4]
Setting
Method
UVG- 1k HEVC-B HEVC-C Xiph-5N
BBB-E E BBB-H H
Low latency
InstA-SSF18 (ours)
10.66
5.0%
65.0%
10.4 %
13.4
28.3%
InstA-SSFS (ours)
5..3
14.1% %
89.3%
-7.5% %
31.9 %
43.2 %
Encoder-fine finetuned SSF18
13.6%
21.5%
80.3%
SSF18
22.5%
37.4%
107.5%
17.5% %
32.7% %
54.9%
SSF5
43.8%
52.0%
172.4%
41.5% %
84.5 %
99.3%
H.265 (fimpeg)
58.8% %
47.1%
42.2%
29.5 %
25.9% %
18.7% %
H.264 (fimpeg)
113.8 %
92.9%
73.4% %
69.3 %
108.4 %
46.9%
B-frames
InstA B-EPIC (ours)
26.9 %
45.0%
Encoder- finctte-finetuned B-EPIC
36.7% %
76.2%
B-EPIC
58.2% %
75.2%
H.2 (ffmmeg)
61.2%
53.2%
H.264 (ffmpeg)
119.0
107.3%
Rate composition It is remarkable that InstA consistently improves the rate-distortion perfor- mance of neural models despite having send updates for millions of neural network parameters. We investigate this low rate overhead in Appendix H We find that the spike-and-slab prior is crucial: compared to Gaussian coding, it allows to around 7.5 bits per parameter.
InstA learns to distribute bits on model updates and latents. The optimal trade-off generally depends on the bitrate, the size of the model and latent code, and the test data: we expect that for strongly out-of-distribution data, more bits will be spent on model updates. Indeed we find that in almost all cases, less than 1% of the total rate is spent on model updates, while for the animated sequences from the Big Buck Bunny video between 1 % and 3% of the total bitrate are invested in model updates. We provide more details in Appendix
Computational complexity The improved compression performance of our instance-adaptive video codec comes at the price of increased encoding time. In Fig. we illustrate this trade-off for the low-latency setting. We summarize the rate-distortion performance as BD-rate savings relative HM and compare to the encoding time For our instance-adaptive methods, this time includes the finetuning in addition to the forward pass of the encoder, the quantization, and the entropy coding (for both the latents and the parameters). By choosing how long to run the finetuning, we can trade off RD performance against the encoding time. Out of the studied codecs, ours can best exploit this trade-off: encoding time and compression performance are fixed other neural methods, while the existing handles in traditional codecs have limited effect performance.
We find that instance-and instance-adaptive finetuning leads to state-of-the-as art RD performance even when fine- tuning only for a few steps. The performance gradually improves over the course of training and eventually flattens out. Care has to be taken in interpreting these results as the different codecs were designed for different hardware and our algorithms have not been optimized for performance.
destacee-ddaptve video compression also adds some overhead on the decoder side, as the model parameters need to be decoded before any frame can be reconstructed. For the full-size InstA-SSFI8 model and without optimizing the code for performance, this initial delay is below 0.2 seconds. The neural network forward pass alone is timed with speed of around 0.01 seconds per frame TeslaV100 100 GPU without any optimization. Both the overhead and the network forward pass are thus negligible compared to the time takes entropy-decode the latents 2; in our (not optimized) implementation, this takes around second per frame, both for the global SSF models and InstA- SSF. Compared to the neural baselines, InstA-SSF thus yields a better performance at comparable complexity. Compared to traditional codecs, there is still gap to bridge, and none of the neural codecs can decode at the framerate of the videos yet.
We report the walltime machines with 40-core Intel Xeon Gold 6230 CPUs with 256 GB RAM and NVIDIA Tesla VI0--XX2 GPUs with 32 GB VRAM. We only single GPU.
8

---

Under review as a conference paper at ICLR 2022
low latency
HEVC class latency
120
100
mnstA-SFF1 (ours)
InstA-SSF18 (ours)
ISFSA-SSF5 (ours)
InstA (ours)
100
SSF18 (Agustason et al.
SSF18 (Agustsson et all
80
SSF5
H,265 (HM)
H.265 (HM)
H,265 (tmppg)
H.265 (ttmpeg)
BD-rate relative to HM 1N6
H.264 (fimpeg)
H.264 (ffmpeg)
rD relative to HM
60
0
10ms
100
min
10
100ms
10min
Encoding time / frome
Encoding time frame
Figure 2: Trade-off between RD performance and encoder-side compute in the low-latency setting. We summarize the RD performance BD>rate difference relative the HM implementation of H.265 (lower is better). InstA-SSF (red) is compared to SSF (orange) and traditional (grey) base- lines. Some fluctuations in the measured walltime are due varying load on our computational cluster. Since we can stop the instance finetuning at any time, InstA-SSF allows us to continuously trade off encoder compute for compression performance.
At the same time, instance-adaptive finetuning allows us to use smaller architectures while main- taining a strong performance, as shown above. This potential to reduce the decoder-side compute may outweigh the small overhead from the initial update decoding.
6 CONCLUSIONS
We introduced InstA, a method for video compression that finetunes pretrained neural compression model on each video sequence to be transmitted. The finetuned network parameters are compressed and transmitted along with the latent code. The method is general and can be applied to many different settings and architectures, we demonstrated it on scale-space flow models in the low-latency setting well as state-of-the-arr B-frame architecture.
Instance-adaptio video compression relaxes the requirement of a single global model that needs to be able to generalize to any unseen test sequence. This can be beneficial when the training data is limited, when there is domain shift between training and test distributions, or when the model class is not expressive enough. While it comes the cost of some overhead the bitstream, demonstrated that with a suitable compression scheme this overhead can be kept small, especially since amortized over the many pixels of a video sequence.
Our InstA models clearly outperform the base models in all considered settings, in several cases achieving a state-of-the-art rate-distortion performance. They outperform their corresponding global base models with BD-rate savings of between 17 % and 26 % on the UVG-1K 1k, HEVC class-B, and Xiph-5N datasets, and the popular fimpeg implementation of H.265 with BD-rate savings between 5 % and 44 %. By evaluating models that were trained on natural scene data on animated sequences, we demonstrate that anstance-adaptive finetuning improves the robustness of neural video codecs under domain shift. Finally, instance-adee aadaptcee compression reduces the required model expressivity: even after reducing the model size by 72 %, InstA-SSF maintains a competitive performance.
The ability to trade off encoder-side compute for compression performance is natural match for one-to-many transmission scenarios, in which a sequence is compressed once but transmitted and decoded often. We demonstrated that even just finetuning for seconds per frame can lead to sub- stantial improvement in compression performance, so asstance-adaptive compression may ultimately even be beneficial in real-time applications.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized video compression. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2503-8512, 2020.
Caglar Aytekin, Francesco Cricri, Antti Hallapuro, Jani Lainema, Emre Aksu, Miska M Hannuk- sela, and Hatanpaan Valtatie. A compression objective and a cycle loss for neural image compres- sion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, 2018.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with scale hyperprior. In International Conference Learning Represen- tations (ICLR), 2018.
Yoshua Bengio, Nicholas Léonard. and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.345 2013. URL.
wwtp://gggabb//aaee 3432
Gisle Bjöntegaard. Calculation of average PSNR differences between RD-curves (VCEG-M33). In
VCEG Meeting (ITU-T SG16 Q. 6). pp. 2-4, 2001.
Benjamin Bross, Jianle Chen, Shan Liu, and Ye-Kui Wang. Versatile video coding (draft 5). Joint Video Experts Team (JVET) of ITU-T SG, 16:3-12, 2018.
Joaquim Campos, Simon Meierhans, Abdelaziz Djelouah, and Christopher Schroers. Content adap- tive optimization for neural image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 0-0, 2019.
Zhibo Chen, Tianyu He, Xin Jin, and Feng Wu. Learning for vidco compression. IEEE Transactions Circuits and Systems for Video Technology, PP, 2019. doi: 10.1 10.1109/TCST..201999 2...0......9.......
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learning image and video com- pression through spatial-temporal energy compaction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Hyomin Choi and Ivan V Bajić. Deep frame prediction for video coding. IEEE Transactions on Circuits and Systems for Video Technology, 30(7): 10(7)):843-111 884--185, 2019.
Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autoen- coders. In International Conference on Machine Learning, pp. 1078-1086. PMLR, 2018.
Abdelaziz Djelouah, Joaquim Campos, Simone Schaub-Meyer, and Christopher Schroers. Neural inter-frame compression for video coding. In JEEE International Conference on Computer Vision (ICCV), pp. 6420-6428, 2019, doi: 10. 00.109//CC2.2299000552.
FFmpeg. FFmpeg. http::/
org/
Adam Golinski, Reza Pourreza, Yang Yang, Guillaume Sautiere, and Taco S Cohen. Feedback recurrent autoencoder for video compression. In IEEE Asian Conference on Computer Vision (ACCV), 2020.
Tiansheng Guo, Jing Wang, Ze Cui, Yihui Feng. Yunying Ge, and Bo Bai. Variable rate image compression with content adaptive optimization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 122-123, 2020.
Amirhossein Habibian, Ties van Rozendaal, Jakub M Tomezak, and Taco S Cohen. Video compres- sion with rate-distortion autoencoders. In IEEE International Conference on Computer Vision (ICCV), pp. 7033-7042. preenacces.thhcvf.ccmm 2019.
Gang He, Chang Wu, Lei Li, Jinjia Zhou, Xianglin Wang, Yunfei Zheng, Bing Yu, and Weiying Xie. A video compression framework using overfitted restoration neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 148 148-149 149, 2020.
10

---

Under review as a conference paper at ICLR 2022
Gang He, Chang Wu, Li Xu, Lei Li, Ziyao Xu, Weiying Xie, and Yunsong Li. An efficient video coding system with an adaptive overfitted multi-scale attention network. IEEE Access, 9:64022- 64032, 2021.
HEVC. HEVC test model (HM). https://vcciie hhi. www.nnoofrr..eeeeeeeeeoooo
HEVC. Common test conditions and software reference configurations. ltttp://phenix../ www.eendd_uerrrrrreeeeeee
sudpari s. eu
2013.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with con- strained variational framework. International Conference on Learning Representations (ICLR), 2017.
Zhihao Hu, Zhenghao Chen, Dong Xu, Guo Lu, Wanli Ouyang, and Shubang Gu. Improving deep video compression by inlllutionaddaptiin flow coding. European Conference on Computer Vision (ECCV), 2020.
Zhihao Hu, Guo Lu, and Dong Xu. FVC: A New Framework Towards Deep Video Compression in Feature Space. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 1502-1511, June 2021.
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang. Erik Learned-Miller and Jan Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9000-9008 2018.
lain Johnstone and Michael Titterington. Statistical challenges of high-dimensional data. Philo- sophical Transactions of the Royal Society Mathematical, Physical and Engineering Sciences, 367:4237 - 4253, 2009.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. International Conference Learning Representations (ICLR), 2014.
Jan P Klopp, Liang-Gee Chen, and Shao-Yi Chien. Utilising low complexity cnns to lift non-local redundancies in video coding. IEEE Transactions on Image Processing, 2020.
Jan P Klopp. Keng-Chi Liu, Shao-Yi Yi Chien, and Liang-Gee Chen. Online-trained upsampler for deep low complexity video compression. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), pp. 7929-7938, 2021.
Yat Hong Lam, Alireza Zare, Çaglar Aytekin, Francesco Cricri, Jani Lainema, Emre Aksu, and Miska M. Hannuksela. Compressing weight-updates for image artifacts removal neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Par- tern Recognition (CVPR) Workshops. pp. 0. Computer Vision Foundation IEEE, 2019.
URL
thecvf. com/ content CVPRW
201 9 Lam Bat1s/aaa_Cmmpssst____e Comp ng We i updateuupdesss...emmmm Barrs__fr___________ Tmage_ Arti facts_Removal Removal Neural Networks__Cvee 201 papeer htmll
Jianping Lin, Dong Liu, Houqiang Li, and Feng Wu. M-LVC: Multiple Frames Prediction for Learned Video Compression. IEEE Conference Computer Vision and Pattern Recognition (CVPR). Jun 2020. doi: 10. 00011090/44466000000000000 1109/CVPR42602 101000...000000 2020. 00360
URL http: doi, org/10.
Haojie Liu, Han Shen, Lichao Huang, Ming Lu, Tong Chen, and Zhan Ma. Learned video compres- sion via joint spatial-temporal correlation exploration. In AAAI Conference on Artificial Intelli- gence, volume 34, pp. 1 11580-1 1587, 2020.
Jiaming Liu, Ming Lu, Kaixin Chen, Xiaoqi Li, Shizun Wang, Zhaoqing Wang, Enhua Wu, Yurong Chen, Chuang Zhang, and Ming Wu. Overfitting the data: Compact neural video delivery via content-aware feature modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4631 13440000, 2021.
11

---

Under review as a conference paper at ICLR 2022
Salvator Lombardo, Jun Han, Christopher Schroers, and Stephan Mandt. Deep generative video compression. In Advances in Neural Information Processing Systems (NeurIPS), pp. 9287-9298.
Curran Associates, Inc., 2019.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. DVC: An end- to-end deep video compression framework. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11006-11000 11015, June 2019.
Guo Lu, Chunlei Cai, Xiaoyun Zhang, Li Chen, Wanli Ouyang, Dong Xu, and Zhiyong Gao. Con- tent adaptive and error propagation aware deep video compression. European Conference on Computer Vision (ECCV), 2020.
Alexandre Mercat, Marko Viitanen, and Jarno Vanne. UVG dataset: 50/120fps 4k sequences for video codec analysis and development. ACM Multimedia Systems Conference, pp. 297-302, 2020. URLpattp:/blttraiiiee
fi/ htt.ttstsssuuuenccs.......
Yu Mikami, Chihiro Tsutake, Keita Takahashi, and Toshiaki Fujii. An efficient image compression method based on neural network: An overfitting approach. In 2021 IEEE International Confer- ence on Image Processing (ICIP), pp. 2084-2088, 2021. doi: 10. 109/ICIP42928 2020999992..221..56633677
Woonsung Park and Munchurl Kim. Deep predictive video compression using mode-selective uni- and bi-directional predictions based on multi-frame hypothesis. IEEE Access, 9:72 1772-55,77 2021. doi: 10.1109/ACCESS 2020.30400....
Jorge Pessoa, Helena Aidos, Pedro Tomás, and Mário AT Figueiredo. End-to-end learning of video compression using spatio-temporal autoencoders. In 2020 IEEE Workshop on Signal Processing Systems (SiPS), pp. 1-6. IEEE, 2020.
Reza Pourreza and Taco S Cohen. Extending neural P-frame codecs for B-frame coding. In IEEE International Conference on Computer Vision (ICCV), 2021.
Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander G Anderson, and Lubomir Bour- dev. Learned video compression. In IEEE International Conference on Computer Vision (ICCV), pp. 3454-3463, 2019.
Oren Rippel, Alexander G. Anderson, Kedar Tatwawadi, Sanjay Nair, Craig Lytle, and Lubomir D. Bourdev. ELF-VC: efficient learned flexible-rate video coding. CoRR, abs/2104. 143355, 2021. http:/tss///cxi..o//////////
14335
Sandvine. 2019 Global Phemomemn Report, http:::tttttttttt::::.t
htt///hhttssss new/
2019.
Gary Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions Circuits and Systems for Video Technology, 1649 2012.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. In International Conference Learning Representations (ICLR), 2017.
Ties van Rozendaal, Iris AM Huijben, and Taco Cohen. Overfitting for fun and profit: Instance- adaptive data compression. In International Conference on Learning Representations (ICLR), 2021.
VideoLAN.. x264 htttpp//wwwwww.cccooooooooorrrer
a. b.
VideoLAN, x265 library.
Yefei Wang, Dong Liu, Siwei Ma, Feng Wu, and Wen Gao. Ensemble Leamnin-Basse Based Rate- Distortion optimization for End-to-End image compression. IEEE Transactions on Circuits and Systems for Video Technology, 31(3): 193-1207, March 2021.
Thomas Wiegand, Gary J Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the H.264/AVC video coding standard. IEEE Transactions on Circuits and Systems for Video Technology, 13(7): 560-576, 2003.
12

---

Under review as a conference paper at ICLR 2022
Chao- Chao-Yuan Wu, Nayan Singhal, and Philipp Krahenbuhl. Video compression through image inter- polation. European Conference on Computer Vision (ECCV), pp. 416 431, 2018.
Xiph.org. Video test media.
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision (IJV)), 127(8): 106-1126 1125, 2019.
mit wttp://wsaaaaa.aaaaeooom.mmeeeee
Ren Yang. Fabian Mentzer, Luc Van Gool, and Radu Timofte. Learning for video compression with hierarchical quality and recurrent enhancement. JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020a.
Ren Yang, Fabian Mentzer, Luc Van Gool, and Radu Timofte. Learning for video compression with recurrent auto-encoder and recurrent probability model. IEEE Journal of Selected Topics in Signal Processing, 3582)3388-401, 2020b.
Yibo Yang, Robert Bamler, and Stephan Mandt. Improving inference for neural image compression. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020c.
Nannan Zou, Honglei Zhang, Francesco Cricri, Hamed R Tavakoli, Jani Lainema, Miska Hannuk- sela, Emre Aksu, and Esa Rahtu. L'C - Learning to learn to compress. In IEEE International Workshop Multimedia Signal Processing (MMSP), pp. 1-6. IEEE, 2020.
13

---

Under review as a conference paper at ICLR 2022
A SUMMARY OF INSTANCE-ADAPTIV COMPRESSION METHODS
Table 3: Overview of neural instance-adaptive data compression methods. We categorize these works by the data modality and by the adaptation of receiver-side components. The number of adapted network parameters the decoder side is proxy for adaptation flexibility, although flex- ibility is also affected by other measures such as the frequency adaptation, or the quantization of updates.
Ballpark estimate based on the reported network architecture.
Decoder-side adaptation
Type
Work
Modality
Method
Parameters
Encoder-only
Campos al. 2019)
image image
Guo cl
2020
Lt al.
2020)
video
Yang ct al.
2020c
image
Hybrid
Lam et al. (2019)
image
Overfit restoration network
Ht al. (0220)
video
Overfit restoration network
0.2 M
Klopp et al. (2020)
video
Overfit restoration network
<0.1 M
He et 202T
video
Overfit restoration network
0.2 M
Klopp el al. (202]
video
Overfit super-resolution network
20.1 M
Liu at. 2021
video
Overfit super-resolution network
<0.1 M
Limited dicoder-side
Aytekin al. 2018
image
Choose from decoders
9
Zou et ( (2020
image
Choose from 255 bias clusters
<0.1 M
adaptation
Wang 2202
image
Choose from 8 decoders
0.9
Full-model
van Rozendaal et al.
2021
I-frames
Overfit full decoder network
4.2M
adaptation
Mikami et al. 2021
images
Overfit full decoder network
0.2 M
This work
video
Overfit full decoder network
18.0M
In Tbl. 3 we categorize existing methods for neural instance-adaptive data compression. The cate- gories are based on the extent of adaptation and we shall describe them one-by-one in the following paragraphs.
Encoder-only restance-adaptivy methods only optimize the encoder de Yang et 2020c Lu al.,
2020) or directly optimize the latents (Campos et al. [019 Guo et al. (020) at interence time. These methods can decrease the amortization gap (the problem that the encoder is not good for
the given decoder) (Cremer et al. 2018). However, as they cannot change any components required on the receiver side, their adaptation potential is fundamentally limited.
The next group of methods are the hybrid instance-adaptive compression methods. These consist of a traditional codec which is ran at a low quality setting (Lam al. 2019 He et 2020) Klopp et al,, low resolution (Klopp et al., 2021 Liu et al. 2021), followed by neural
2020 He et al. 2021) enhancement network which is overfitted to the test instance (and signalled in the bitstream). The advantage of this approach is that it can build established, tigh-performing traditional codecs such as the HM reference implementation of H.265. As such, the rate-distortion performance of these methods can be very good, especially at lower bitrates, where neural codecs tend to perform more poorly (see Fig. 4p. The size and computational complexity of these enhancement networks typically very low. However, should be noted that decoding the video still requires running the expensive base codec first. In contrast, we present a smaller neural codec InstA-SSF5 which does not require any compute besides running the neural decoder. Another downside of the hybrid approaches is that end-to-end neural video codecs are continuously improving. While our instance- adaptive method can be applied to any neural base model, not clear how hybrid approaches can benefit from the developments in this field.
The next group we consider consists of fully neural codecs that adapt the decoder and prior networks to each instance, albeit to a limited extent. The works we consider (Aytekin et al. 2018 Zou et al.) 2020 Wang et al. 2021) all show results on image compression, with autoencoders that operate on patches blocks of the image. At training time, these methods train ensemble of different de- coders, which are then frozen as a part of the codec. At test time, the optimal decoder is selected for every patch and communicated the receiver. This is often combined with encoder-side finetuning
14

---

Under review as a conference paper at ICLR 2022
for increased rate-distortion performance. Although this approach allows changing of all decoder- side parameters, the actual degrees of freedom are very limited (up to 255 different possibilities in Zou al. (2200). Furthermore, since the different possible decoders determined during training these approaches is not able to solve the domain gap problem fully.
Finally turn to the full-model iistace-addaptvv methods, in which all components of neural codec (encoder, decoder, and prior) are overfitted at test time, and the changes to the decoder are sig- nalled in the bitstream. Due the large size of neural codecs, for signalling model updates will not dominate the bitrate. This can be achieved by choosing a very small base model (Mikami al. 2021) such that the rate overhead minimized, but this does not scale to the architectures required for SOTA performance. Instead van Rozendaal et al. (2021) pro- pose adapting the codec with a model-rate aware loss function, which includes sparsity-inducing prior. The authors only reported results on I-frame compression task on single dataset. We are the first to show that this type of full-model adaptation is highly efficient for the compression of full video sequences with P-frames B-frames.
has to make sure that the overhead
B DATASETS
UVG-1k
Following Refs. (Wu 2018 Lu al. 2019; Golinski al. 2020 Agustsson al.
2020), we report results the UVG- 1k dataset, which consists of seven test video sequences to- talling 3900 frames captured at 120 fps and natively available in 1080p resolution. Note that these sequences are only subset of the full UVG dataset (Mercat et al. 2020).
Xiph-5N We use the XIph-5N dataset introduced in van Rozendaal et al. (0221) It consists of the following five sequences from the Xiph.org collection: "in to tree", "aspen", "controlled burn"" "sunflower", and "pedestrian area".
Big Buck Bunny (BBB-E and BBB-H) From the Big Buck Bunny video, which is also part of the Xiph.org media collection, we extract subsequences of 10s length. They are chosen based on the frame-wise RD loss achieved by global SSF18 model (with 8 = 0.0016). We select the 10s window with the lowest RD loss and the 10 window with the highest RD loss, excluding the intro and outro of the video. These signify the "easiest" and "hardest" parts of the video to compress and we refer to them as "BBB-E" and "BBB-H", respectively. Fig. 3 we show the frame-wise RD loss and the selected sequences.
0.0015
Loss per GoP
10s ralling average
888-H
0.0010
RD
0.0005
0.0000
2000
4000
6000
8000
10000
12000
14000
Big Buck Bunny frame
Figure 3: Temporal distribution the RD loss of the global SSF18 model for the Big Buck Bunny video. We the 10s windows with the lowest and highest loss as benchmarks, excluding the intro and outro of Big Bunny These subsequences, which we call "BBB-E" and "BBB-H" for "easy to compress" and "hard to compress", marked in orange and red.
15

---

Under review as a conference paper at ICLR 2022
Table 4: Architecture differences between the original SSF model published by Agustsson et al. 2020) and our SSF18 re- re-implementation. Convolutional layers are indicated as "conv [kernel size] [layer type][stride] [output channels]". 4 is used for convolutional layers, while f is used for trans- posed convolutions. The hyperdecoder is modeled where the mean and standard deviation are modeled by two separate neural networks yu(z)) and (21) with largely identical architectures. These network differ in their activation functions, acti- vations that are only used in one of the two networks are indicated in the table by their respective symbol. Note that we always clamp o (21) to the range [0.11,
Gaussian p(>2(z1)
( (z)))
Agustsson et al. Agusssson et al. 2020
Our reimplementation
Hyperencoder ((11
conv 5x5 (2 192 ReLU
conv 3x3 4.1 192 ReLU conv 5x5 (2 192
cony 5x5 (2 192 ReLU
ReLU
cony 5x5 (,2 192
conv 5x55 (2 192
Hyperdecoder p(22|z1)
conv 5x5 t2 192
conv 5x5 12 192 ReL.u conv 5x5 12 192 ReLu conv 3x3 11 192
/:: ReLUU, d: QReLU
cony 5x5 12 192
j:: ReLU, a: QReLU cony 5x5 f2 192
a: QReLU
ReLU
C MODEL ARCHITECTURES
Full-size SSF model We first apply instance-adaptive video compression to the scale-space flow (SSF) architecture proposed by Agustsson et al. (Agustsson et al. 2020), using our own re- implementation of the model. We mostly follow the setup described in Agustsson et al. (2020). but there are some minor differencee. In the hyperencoder and hyperdecoder networks we use a slightly different architecture. We list the architectures in Tbl, 4 In addition, our implementation of the blur stack and the ccale-space-flon warping operation differ from the one used by Ref, 4 but have verified in experiments that this has negligible effect.
We empirically validated our SSF re-implementation by comparing its performance to the original SSF (using entropy-coding). On the UVG-1k dataset, our SSF18 model performed slightly better than the results published by Agustsson et al. (202)).
Smaller SSF models In addition to the full-size SSF18 model, we consider three smaller scale- space flow architectures. They follow the same structure as SSF18 in Tbl, but differ in the number of latent variables and convolutional channels, as shown in Tbl. 5 We refer to them as SSF8, SSF5, and SSF3 after the number of decoder-side network parameters (in millions).
B-EPIC model As second base model we use the B-frame codec developed by Pourreza & Cohen (202T) (B-EPIC). We follow the hierarchical IBP configuration presented in that paper and use the model trained by the authors. During training, GoP consists of 4 frames arranged in an IBBP pattern. At test time, we use a GoP size of 12 frames and the following reference structure (where frame 0 refers to the last frame of the previous GoP):
Frame 12: P-frame referring to frame 0
Frame 6: B-frame referring to frames 0 and 12
Frame 3: B-frame referring to frames 0 and 6
Frame B-frame referring frames 6 and 12
Frame 1:
B-frame referring to frames 0 and 3
Frame 2: B-frame referring frames 1 and
16

---

Under review as a conference paper at ICLR 2022
Table 5: Architecture details for our scale-space-flow models. In the top, we show the number of channels for the latent and hyperlatent variables, 2, which are the basis for the compressed code; in the bottom, we show the number of channels in the codec and hypercodec networks in the encoder and decoder. We use the same model architecture and size for I-frame, scale-space-flow and residual networks.
Model
SSF18
SSIS
SSE5
SSF3
latent channels
192
192
192
128
hyperlatent channels
192
192
192
192
codec channels
128
96
64
48
hypercodec channels
192
and so on for the remaining frames. The very first frame in a video is encoded as an I-frame. Encoding and decoding in this order ensures that the reference frames will always be encoded or decoded before the frames for which they are needed.
Quantization settings Ablation studies showed insensitivity network prior settings in the ranges of 10 I < 1000, 0.0005 < I 0.005, and 0.01 I S 0.05. For the experiments this paper we use bin width 0.001, a 0.05, = 1/6, a spike-slab ratio a 100, and 2
Entropy coding We implement an entropy coder for the latents and model updates in the SSF and InstA-SSF models. We verified that this achieves bitrates in agreement with the probability mass functions of p(z) and p(o). For the B-EPIC and InstA-B-EPIC models, we report bitrates based on probability mass functions without actually implementing an entropy coder.
D TRAINING SCHEME
Global models The scale-space flow models described in Sec. Bare trained with the MSE training setup described in Agusssson et al. (2020). except that we use the publicly available Vimeo-90k dataset. The 1) loss computed with (integer) quantized latents 2, while for the Rg loss term we use noisy samples from U(= 2, 5). Following Agustsson et al. (2020), we first train for million steps on 256 x 256 crops with learning rate of 10 - We then conduct the MSE "finetune" stage of the training procedure from Agusstson et al. (2020) (not to be confused with instance- adaptive finetuning) for the SSF18 model, where we train on crops of size h
256 x 384 with learning rate of 10 The models are trained with a GoP size of 3 frames, which means that we split the training video into chunks of 3 frames and randomly sample chunks during training. We finally evaluate the models with GoP size of 12. This gives similar results to using infinite unrolling at demonstrate in Appendix
test time,
For the B-EPIC model, we use the model trained by Pourreza & Cohen 2021) on Vimeo-900 90k.. The setup is similar to that for the SSF models except that B-EPICs more complicated GoP structure requires training with a GoP size of frames. At test time we use a GoP size of 12, the frame configurations described in Appendix C
Instance-adaptive finetuning In the low-latency scenario we use a GoP size of 3 and finetune on full-resolution frames (1920 1080 pixels) with batch size of and a learning rate of 10-5, After finetuning, we transmit sequences with a GoP size of 12. The prior and decoder updates 5 = 0 - Op are quantized and coded under the spike-and-sla network prior descrihed Sec.
While the SSF model follows a stationary distribution over frames, the B-EPIC model has a more complex frame structure: the distance between frame and its reference points can vary between to 6 frames. For this reason, we found it beneficial use GoP of 12 for InstA finetuning of the B-EPIC model. Due to memory limitations, we finetune on horizontal crops of size 256x1080 pixels instead of full-resolution frames. For the low-bitrate working points line with Pourreza & Cohen (2021). ). For the high- high-bitrate region (8 S 0.0008), we found that we can get faster convergence by using learning rate of 5 10-5
learning rate of 10 5 in
17

---

Under review as a conference paper at ICLR 2022
E CLASSICAL CODECS
We generate H.265 and H.264 results using version v3.4.8 of ffmpeg (FFmpeg) When comparing neural codees the low-latency setting, also choose low-latency setting that only allows frames and P-frames and fix the GoP size to 12. We use various encoder presets. Example commands
ffmpeg -pix fmt yuv420p 1920x1080 \
-r 120 -i zaw_video. yuv \
-civ 1ibx264 -preset medium
-crf 23 -tune zerolatency -xaramsarams "keyint=12 output. .nkv
for H.264 and
ffmpeg -pix fmt yuv420p -s 1920x1080 A
-r 120 -i raw_videss yuv
-e:v 1ibx265 -preset medium
-crf 23 -tune zerolatency -x265-params
output.mku
for H.265. We also compress the test sequences with the HM reference implementation of HEVC, using the "lowlatencyP" setting with a fixed GoP size 12.
As baselines for the neural B-frame codecs, we also relax the GoP and frame-type restrictions for the classical codecs, using the default random-access settings with B-frames and variable GoP size.
F ADDITIONAL CATE-DISTORTIO RESULTS
In Fig. we show the rate-perfonie site-peffomancc performance our models relative the HM baseline. In addition to the classical codecs and our base models, compare to several additional neural baselines: Lu et (0019 2020 : Rippel al. (202T Lin al. (020):: Klopp et al. (2020); Yang al. c)) Hu et al. (2021) and Hu al. (2020). Relative to classical codecs, almost all neural codecs perform better at higher bitrate. The exception is Klopp et al. (2020), which consists of a classical codec followed by neural enhancement. With the exception of this method and in some bitrate regions ELF-VC (Rippel et al. 2019) and M-LVC (Lin et al. 2020), our InstA method achieves state-of tfetthearrt neural compression performance.
Figures S and 6 show rate-distortion curves on the HEVC class C. Xiph-5N, and Big Buck Bunny datasets. The test sequences of HEVC class C have lower resolution of only 832 480. This is a challenging scenario for InstA, as the model rate overhead is amortized over less pixels than in higher-resolution videos. Indeed, classical and some neural baselines as on the higher-resolution datasets. Nevertheless, it still leads sizable improvement over the SSF18 base model.
find that our method does not perform strongly relative to
Finally, in Tbl. we we show the BD-rate savings of the InstA-SSF18, InstA-SSF5, and SSF18 models relative to different H.265 implementations and relative the SSF18 model.
18

---

Under review as a conference paper at ICLR 2022
UVG-1K, low latency
HEVC class latency
120%
InstA-SSF16 (ours)
InstA-SSFS fours)
SSF18 (Agustsson 20)
Rate relalive to H.265 (-M) [4]
Rate relative to H:265 (HM) [4]
80%
Encoden-tuned SSF18
DVC (Lu
DVC-adaptive (Lu 20)
ELF-VC (Rippel 21)
40%-
40%-
M-LVC (Lin
CTU (Klopp 21)
éfang 206)
RaFC C (Hu 20)
0%
FVC
0%
H.265 (HM)
(ffmpeg)
-40%
H.264 (ffmpeg)
40
PSNR [oB, 11
PSNR (dB,
UVG-1K, B-frames
HEVC class 6. B-frames
InstA-B-EPIC (ours)
200%
B-EPIC (Pourreza 21)
200%
Encoder-tuned B-EPIC
Rate relative to H.265 (HM) [4]
HLVC (Yeng 20a)
160%
Rate rolative to 16-255 (HM0
H.265 (HM)
1.60%
H.265 (ffmpeg)
120%
H.264 (ffmpeg)
80%
40%
40%~
0%
-40%
36 PSNR [dB,
30
32
36
[dB,
Figure 4: Bitrate our instance-adaptive SSF models (InstA-SSE, red) relative to the HM reference implementation of H.265 as a function the quality in dB PSNR. We also show several neural (orange, blue, green) and traditional (grey) baselines.
HEVC class C, low latency
Xiph-SN, low latency
42
30
32
InstA-S5F18 (ours)
InstA-S5F5
Encoder-tuned SSF18
PSWR [d6.
36
(Agustason
SSFS
DVC-adeptive (Lu 20)
34
InstA-SSF18 (ours)
(Yng
InstA-5515 (ours)
RaFC (Hu 20)
SSF18 (Agustssinn 201
28
21)
H.265 (HM)
32
H.265 (HM)
(fimpeg)
(ffmpeg)
H.264 (fimpeg)
H.264 (ffmpeg)
0.8
Rabe Tapp. 11
Rate Ibpp. 41
Figure& 5: Rate-distortion performance of our instance-adaptive SSF models (InstA-SSEF SSF, red) com- pared neural (orange) and traditional (grey) baselines class C of the HEVC test sequences (left) and five Xiph.org test sequences (right).
19

---

Under review as a conference paper at ICLR 2022
Table 6: BD-rate of several codecs (rows) compared to different reference codecs (columns) in the low-latency setting. Lower numbers are better, the best results on each dataset are shown in bold.
BD-rate relative
Dataset
Method
H.265 (HM)
H.265 (fimpeg)
H.264 (fimpeg)
SSF18
UVG
InstA-SSF18 (ours)
10.66 %
-43.7 %
-8..2 %
27.11 %
InstA-SSF5 (ours)
-5.3 %
-40.3 %
-55.7 %
-22.7%
SSF18
22.5 %
-2..8 %
42.7 %
Encodder-finttuned SSFI8
-28.1
46.8
-7.2%
HEVC-B
InstA-SSF18 (ours)
5.0%
-28.6
45.5
23.6%
InstA-SSF5 (ours)
14.1
-2241
40.8
17.0%
SSF18
37.4 %
-6.6%
28.77 %
Ennetuer-inetuned SSF18
21.5 %
-17.4%
-37.0 %
11.6%
HEVC-C
InstA-SSF18 (ours)
65.0 %
16.0 %
-4.8 %
%
InstA-SSF5 (ours)
89.3
33.1 %
9.2%
-8.8% %
SSF18
107.5 %
45.9
19.7 %
Encoder-finetuned SSFI8
26.8
4.0%
13.1
Xiph5N
InstA-SSF18 (ours)
10.4 %
30.8
47.1
23.8 %
InstA-SSF5 (ours)
-7.5
-8.66%
-45.4%
-2.33% %
SSF18
17.5 %
-9.2 %
30.6 %
BBB-E
InstA-SSF18 (ours)
13
-9.9
-45.6% %
14.5
InstA-SSF5 (ours)
31.9 %
4.8 %
-36.7 1
-0.7% %
SSF18
32.7 %
5.4 %
-36.3 %
BBB-H
InstA-SSFII (ours)
28.3 %
8.1 %
-2.66 %
17.1%
InstA-SSF5 (ours)
43.2 %
20.6 %
-25 %
-7.5%
SSF18
54.9%
30.5
5.4%
Table 7: BD-rate of several codees (rows) compared different reference codecs (columns) in the B-frame setting. Lower numbers are better, the best results on each dataset are shown in bold.
BD-rate relative to...
Dataset
Method
H.265 (HM)
H.265 (fimpeg)
H.264 (ffmmegg)
B-EPIC
UVG-Ik
InstA-B-EPIC (ours)
26.9 %
21.3 %
42.0 %
19.8 %
B-EPIC
58.2 %
-1.9
-27.8 %
finettred B-EPIC
36.7 %
-15,2
33.66% %
13.6%
HEVC-B
InstA-B-EPIC (ours)
45.0 %
-5.4%
-30.0 %
-17.2%
B-EPIC
75.2
14.3 %
-15.5
B- B-EPIC
76.2 %
15.0 %
-15.0 %
0.6%
20

---

Under review as a conference paper at ICLR 2022
Big Bunny, easy seque
latency
Bunny, sequence (8BB-H), latancy
48
42
40
44
PSWR (d8. F1
InstA: SSF18 (ours)
34
Insti 55F8 (ours)
InstA-SSF5 (ours)
InstA-SSFS (ours)
SSF (Agustsson
SSF18 (Agustsson 20)
SSFS
32
SSFS
H.265 (HM)
H.265 (HM)
H.265 (ftmp-eg)
H.265 (ftmpeg)
H.264 (fimpeg)
H.264 (ffmpea)
30
0.0
0.1
0.2
0.0
0.2
0.6
Rate [bpp. 41
Rate [bpp.
Figure 6: Rate-distortion performance of our instance-adaptive SSF models (InstA-SSF, red) com- pared to neural (orange) and traditional (grey) baselines on the "easy" (left) and "hard" (right) se- quences from the Big Buck Bunny video.
BD-rate savings are computed over certain distortion range. We choose the largest possible range for which we have results for our methods as well as the main baselines. These are:
UVG-1k: 33.5 dB & PSNR S 40.4
HEVC class B: 30.3 dB < PSNR & 3B
HEVC class C: 27.2 dB < PSNR S dB
Xiph-5N: dB < PSNR S 39
BBB E: dB < PSNR < dB
BBB-H: 31.4dB < PSNR < dB
G PERCEPTUAL QUALITY
We illustrate the performance out-of-distribution data in Fig. 7 where we show original and compressed versions of two frames from the Big Buck Bunny video. Both traditional codecs and the SSF18 baseline lose details and introduce artifacts at the chosen low-bitrate setting. While the InstA-SSF18 compressions still suffer from some blurriness, artifacts are less pronounced than in the global model.
In the supplementary materials, we attach the corresponding reconstructed video sequences. To highlight the differences between the codecs, again use very low bitrate for all codecs (even though our InstA method is strongest at higher bitrates).
H ABLATION STUDIES
Model size ablation the left panel of Fig. we compare the compression performance of glob- ally trained SSF18, SSF8, SSF5, and SSF3 models, as well as instance-adaptive versions InstA- SSF18, InstA-SSF8, InstA-SSF5, and InstA-SSF3, Due to the training time requirement we limit this ablation to single video, the "in_to_ tttttreeeeeeeeeeee sequence from the Xiph.org collection. We find that the larger models lead to a better compression performance, with more pronounced difference the high-bitrate region. SSF3 and InstA- models perform poorly at high bitrates. We attribute this partially to the small size the latent space compared to all other models. In general, we observe that instance-adaptive training improves the performance of models of any complexity. Instance
21

---

Under review as a conference paper at ICLR 2022
H:244 (Tmpee)) 0,031 bpp
#1265 (Tmppg) 0,031 hpp
"SFF18 tAgustsson bpp
SSF18 0,031 bpp
Figure 7: Original and compressed details in frames 7395 (left) and 7276 (right) of the Big Buck Bunny video (Xiph.org). From top to bottom, we compare the original, compressed frames with the H.264 and H.265 codees in the fimpeg (x264/ x265) implementation, the SSF18 baseline, and our proposed InstA-SSF18 method. We use settings with comparable, low bitrates; the achieved bitrates (over the 10s sequence) are given in the left panels. All codecs lose textural details, but InstA-SSF18 codec reduces the artifacts visible the SSF18 reconstructions.
adaption reduces the gap between smaller models and the full-size model, confirming that less ex- pressivity is necessary for good performance single instance compared to on the entire data distribution.
GoP size considerations Throughout this work, we use a GoP size of 12 at test time. test the robustness of this choice, in the right panel of Fig. & we compare to an infinite GoP size (for the SSF model). In other words, we compress the first frame as I-frame and all remaining frames as P-frames. The overall difference is slim, though the gap is larger for some videos than others.
Sensitivity to framerate variations To further quantify robustness to test data characteristics, we investigate the effect of the video framerate on the model performance. We study the performance of global and instance-adaptive SSF models on temporally subsampled videos, i.e. keeping one out of every four frames. To save computational resources, this ablation is only performed on the "Bosphorus" sequence from the JV dataset. The left panel of Fig. 9 shows results. While the SSF18 performance varies greatly between the original and the subsampled video, the difference in performance is much smaller for InstA-SSF18. In addition, both models perform better in the 120 fps scenario due to the higher redundancy across frames.
Bitrate allocation Figure 10 disentangl the various contributions to the total bitrate. We find that the overhead from model updates is generally small (typically well below 1 %), except videos that differ substantially from the training data: on the animated Big Buck Bunny sequences, InstA learns to adapt more strongly and up to 3 % of the total rate spent on model updates.
We further break down the model rate in Fig. II where we show how the model rate is distributed over the various auto-encoders and submodules of our InstA-SSF18 and InstA-SSF5 model. It can
22

---

Under review as a conference paper at ICLR 2022
Architecture ablation in_to_notttoooo
GoP ablation Xiph-5N
42
38
38
34
PSWR [dB.
PSNR
InstA, SSF18 InstA-SSF0
InstA-SSF5
InstA SSF3
SSF18 (Agustsson al.)
InstA-SSF18 (GoP 12)
30
InstA-SSF18 (GoP
SSF5
SSF18 (GoP size 12)
SSF3
SSF18 (GoP size s9)
28
32
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.0
0.2
D.4 Rate [bpp, 41
0.6
0.8
Rate [bpp.
Figure 8: Left: Model size ablation study. We compare several global and instance-finetuned scale- space-flow models, showing the performance on the in_te_tree sequence (Xiph.org). Right: Effect of the GoP size test time. We show rate-distortion curves of instance-adaptive and global SSFI8 models on the HEVC Xiph-5N dataset, comparing a fixed GoP size of 12 (solid lines) to an infinite GoP size (dotted lines).
Framerate ablation on Bosphorus (UVG-1K)
Big Buck Bunny, hard sequence (BBB B-) low latency
41
30
39
PSWRR [dB,
36
37
34
InstA-SSF16, scene-wise finetuning
----- InstA-SSF18, video-wise finetuning
35
InsAA-SFFBB-120fp
InSA--5FF8833fps
H. 26.65 (HM)
(Agustsson
SSF18-120fps
(ffmpeg)
SSF18-30fps
H. 264 (ffmpeg)
0.05
0.25 0.30 0.35 0.40
0.6
Rate [bpp. 41
Rate [bpp,
Figure 9: Left: Framerate ablation study. We compare the SSF18 baseline (orange) and InstA- SSF18 (red) the original video at 120 fps (solid) and on the temporally subsampled video at 30 fps (dot-dashed). Instance-adaptive compression increases robustness to variation in framerate. Right: Full-video finetuning ablation study. We compare the performance of SSF model finetuned on the full Big Buck Bunny video (10 minutes) to a SSF model finetuned only on the BBB-H scene (10 seconds) when evaluated on the BBB-H scene.
be seen that the InstA-SSF5 model has fewer parameters per module by design, but is making up for its smaller capacity by spending lot more bits/per parameter model updates compared to InstA-SSF18 (often more than double). This explains why the rate-distortion performance of InstA-SSF5 is close to that of InstA-SSF18. Even though the InstA-SSF5 model needs more bit per parameter to adapt to the data instances, the total bitrate spend on model updates is still smaller than that of InstA-SSF18.
Focusing on the distribution of the model bitrate across auto-en,oders, we see that more than half of this rate is allocated to the I-frame model. The flow autoencoder is the next biggest consumer
23

---

Under review as a conference paper at ICLR 2022
InstA-SSF18, UVG-1k
InstA-SSF16 Bunny (easy hard sequence)
100%
100%
80%
Fraction the total rate
Model updates
60%
Residual latents
60%-
1oock-000 the
Residual hyperletents
Flow: latents
Flow hyperiatents stentss
40%
I-frame latents
50%-
I-rame hyperlatents
0%
0% 0.1
0.1
0.3 Total rate [bits / pixel]
0.5
0.2
Total rate [bits pixel]
Figure 10: Bitrate allocation for our InstA-SSF18 model. We show the relative contribution to the overall bitrate from quantized latents for the I-frames, P-frame flow, and P-frame residuals (different shades of blue). In red we show the relative contribution from the quantized model updates. Left: on the UVG-Ik dataset, which has a similar content to the training data. Right: on the two Big Buck Bunny scenes, which differ substantially from the training data. We find that InstA learns to spend more bits on model updates in this case.
of model rate, while only few bits are spent communicate updates the residual autoencoder. In terms of submodules, it turns out that even though the hyper-decoder has a lot more parameters than the decoder, the decider receives substantially larger update bits per parameter. Together, these findings suggest that instance adaptation is most important for network components that are closer to the data space in the computational graph.
Update sparsity analysis The key design for achieving low model rate is in update sparsity. We achieve this by using a spike-and-slab prior around global model parameter values. Training with the spike-and-slab prior is sparsity enforcing (on average around 10 % of all quantized parameters receive zero update). To quantify savings of our chosen spike-and-slab prior, we compute the cross-entropy of our prior with the resulting model 8 after training. This is the number of bits we need to transmit the model update. On the other hand, assume we had Gaussian prior with same variance as our "slab" component, and assuming the resulting 5 follows the same distribution, the entropy of this Gaussian prior gives the expected number of bits to transmit non-sparse model update. We estimate the difference of the two scenarios to be around 7.5 bits per parameter.
Finetuning strategy for longer videos For longer videos with multiple scenes, one consider several different strategies for instance-adaptive finetuning. The simplest approach is finetuning a single model the full video. It may be beneficial to instead finetune the model independently on each scene. Even hierarchical approach in which model finetuned on the full video further finetuned on each scene is an option.
The optimal strategy will in general depend on the particular video content and the similarity of the scenes as well as on the bitrate setting. We perform a simple first test by finetuning single model on the entire Big Buck Bunny video, 10 min video (14315 frames) with multiple scenes united by a common style. The results in the right panel of Fig. | show that such a single finetuned model leads to clear RD improvements over the global SSF model. However, in the large-bitrate region these improvements are smaller than the gains we achieve by finetuning on the BBB-E and BBB-H scenes individually.
24

---

Under review as a conference paper at ICLR 2022
InstA-SSF18 (136.9
InstA-SSF5 (105.2 kB)
I-frame
Flow
Residual
I-frame
Flow
Residual
0.2
0.1
108 B
149
193
152 B
0.0
0.1
16.2 KB
12/0
12.5
8.7 kB
4.7 kB
5.4
0.0
0.8
47.6 k6
0.7
0.6
(bitspperameeee
0.5
Decoder
0.3
53.6 KB
33.9 KB
8.2
#params
éparams
Figure 11: Model bitrate allocation for our InstA-SSF18 (left) and InstA-SSF5 (right) model. We show how the model rate is distributed across each autoencoder (columns) and submodule (rows). All plots use grid of the same size, where width corresponds to the number of parameters in that module, height corresponds to the average bits/parameter used for updating that module, and area corresponds to total number of bits allocated for that module (each block in the grid has an area of 12.5 kB). Results are averaged over all values of B and all datapoints the UVG- and HEVC class B dataset.
I ETHICS STATEMENT
Video compression affects the life of millions of people and any algorithm and model should be carefully analyzed for potential ethical issues before deployment. In particular, it is important to check for harmful biases, which may be due to imbalanced training data, but can also be caused or exacerbated by algorithmic choices. To some extent, the instance-adaptive video compression method we are proposing may be helpful in combating such biases, as it can improve the compres- sion performance on data that is different from typical samples in the training set. However, it cannot solve the problem entirely, as more pronounced instance adaptation will increase the bitrate.
The environmental impact of our research is also relevant. The energy cost of our experiments directly contributes to the climate catastrophe. In the long term, our work may also have beneficial aspect, it allows to lower the bitrate and thus the energy cost of video streaming, which already now make up the majority of internet traffic.
J REPRODUCIBILITY STATEMENT
Our mustance-adaptinn video compression algorithm is described detail in Sec. 3 The most com- plex components the base compression models, which described in more detail in the ref- erences we cite as well as in appendix C With the details we provide, instance-adaptive video compression be implemented and used for any base model with relatively little effort. Finally, in the supplementary material we provide the performance of the various methods on each video sequence CSV file to aid comparisons.
25


Reviews:
[{"review_id": "z3fBb9dR3G", "reviewer": "Reviewer_TVKU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThis paper is an extention of instance-adaptive image compression to video compression. Main advantage of the proposed method is that it's compatible with many existing learned video compression frameworks and the increase of computational complexity at the decoder side is negligible. Besides, the experiments are sufficient and the performance gain is convincing. \n\nWeakness:\n\nThe overall novelty seems limited since the instance-adaptive method is from existing work with no primary changes. Here are some main questions and concerns:\n\n1). How many optimization steps are used to produce the final reported performance in Figure.1 as well as in some other figs and tables? \n\n2). The proposed method looks stronger at high bitrate but close to the baselines at low bitrate. What is the precise bitrate range used for BD-rate comparison?\n\nBesides, a related work about implementing content adaptive algorithm in learned video compression is suggested for discussion or comparison:\n\nGuo Lu, et al., \"Content Adaptive and Error Propagation Aware Deep Video Compression.\" ECCV 2020.\n", "summary_of_the_paper": "This paper proposed a video compression algorithm based on instance-adaptive learning. Experiments show that this method can significantly improve the performance of existing learned video compression codec with acceptable decoding complexity and latency.", "summary_of_the_review": "The paper is well written in general. However, the overall contribution may not be sufficient enough to reach the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kjz86jYMbH", "reviewer": "Reviewer_8iiZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-Strengths\n1. Good motivations. The domain gap between the training and testing datasets existed in the practical applications and it is important to solve this problem. \n2. The proposed approach is straightforward and neat. Transmitting the model parameters significantly improves the compression performance. \n3. Comprehensive analysis. A lot of results are provided in the ablation study and it is very helpful. \n\n-Weaknesses\n1. Novelty. This paper extended the previous work Rozendaal et al. (2021) to video compression and may have limited novelty. The extension from image compression to video compression is expected. The major modification is the prior model for entropy coding, which is not enough. \n2. Another concern is computational complexity. This new approach increases a lot of computational burdens, which is another issue. This method is even slower than HM in some cases. \n3. The authors only evaluate the proposed method on the UVG and Class B datasets, which are two high-resolution(1920x1080) videos. How about the experimental results on other datasets, like Class c or D?\n4. There are also some related works that should be discussed.   (a) Content Adaptive and Error Propagation Aware Deep Video Compression. Lu et al. (b) Efficient Video Compression via Content-Adaptive Super-Resolution. Khani et al.  (c) Online-Trained Upsampler for Deep Low Complexity Video Compression. Klopp et al. \n\n", "summary_of_the_paper": "This paper tried to close the domain gap between the training and testing dataset for the learned video compression. An instance-adaptive video compression approach is proposed by updating the model parameters of the video codecs and the optimal parameters are transmitted to the decoder. Besides, the authors modeled the updated model parameters under a mixture prior model, which further reduces the bit cost. Experimental results on several datasets demonstrated the effectiveness of this approach. ", "summary_of_the_review": "The main concern is the novelty as a similar approach has been used in image compression. Basically, it is a borderline paper and I am open to other reviews. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dSZEuSlRptt", "reviewer": "Reviewer_TzFr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces an interesting on-line parameter updating scheme into video compression. The experiments show good RD performance and the analyses on the trade-off between encoding time and RD performance show the flexibility of encoding time.\n\nIn my point of view, the paper may have the following weakness:\n\n1. Limited novelty and contribution.  The video compression architecture is borrowed from scale-space flow (SSF), and the parameter updating scheme follows the similar idea on image compression (Rozendaal et al., 2021), and other detailed technics are also from previous works, e.g., spike-and-slab prior (Johnstone & Titterington, 2009; Rozendaal et al., 2021). Although it is new and interesting to explore the effectiveness of these technics in learned video compression, the reviewer thinks that the novelty and contribution are limited. \n\n2. The ablation studies are missed. How many overhead bits are consumed due to the parameters' update? What is the proportion of the bits used for encoding quantized updates \\bar{\\theta}, quantized codes \\bar{z} and and the prior p(\\bar{\\theta})? These are necessary to analyze the proposed method.\n\n2. Some representative existing works are missed from related works and experimental comparisons. For instance,\n\nCheng, Zhengxue, et al. \"Learning image and video compression through spatial-temporal energy compaction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\nHabibian, Amirhossein, et al. \"Video compression with rate-distortion autoencoders.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n\nLin, Jianping, et al. \"M-LVC: multiple frames prediction for learned video compression.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\nYang, Ren, et al. \"Learning for video compression with hierarchical quality and recurrent enhancement.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\nLiu, Haojie, et al. \"Learned video compression via joint spatial-temporal correlation exploration.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020.\n\nLu, Guo, et al. \"Content adaptive and error propagation aware deep video compression.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\nHu, Zhihao, et al. \"Improving deep video compression by resolution-adaptive flow coding.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\nYang, Ren, et al. \"Learning for video compression with recurrent auto-encoder and recurrent probability model.\" IEEE Journal of Selected Topics in Signal Processing 15.2 (2021): 388-401.\n\nGolinski, Adam, et al. \"Feedback recurrent autoencoder for video compression.\" Proceedings of the Asian Conference on Computer Vision. 2020.\n\n", "summary_of_the_paper": "This paper proposes a video compression algorithm based on instance-adaptive learning. On each video sequence to be transmitted, the proposed method finetunes a pretrained compression model. The parameters' change are transmitted to the decoder with little bit-rate overhead. The experiments show the outstanding performance of the proposed method.", "summary_of_the_review": "+ The method is interesting and effective.\n\n- The contribution and novelty are limited.\n\n- The experiments (and ablations) are incremental.\n\nOverall speaking, the reviewer thinks that this paper stands at the borderline, but marginally above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_Vy1qIkqRH0", "reviewer": "Reviewer_pT9R", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantages:\n\nThis paper introduces an instance-adaptive video compression method, which determines a pretrained neural compression model on each video sequence to be transmitted, adjusts the network parameters, and is transmitted together with the latent coding. It is verified by the latest scale-space flow model and B-frame compression model, and some bitstream savings are achieved.\n\nDisadvantages:\n\na The key idea of this paper comes from the instance-adaptive data compression, which is only applied to the scale-space flow model and B-frame compression architecture. The method of this paper is lack of innovation.\nb The difference between the model used and the scale-space flow described in the appendix: the size of the first layer convolution kernel is changed from 5 * 5 to 3 * 3. No reasonable explanation was given.\nc In the experiment, only UVG and class B of the standard test sequences were compared. How does it compare with FVC on the HEVC Class C and HEVC Class D ?\nd There are some spelling and grammatical problems in the paper, such as page 3, the third line of the paragraph above formula 2, parameter “can be be transmitted very effectively”. In the references on page11, the abbreviation of the conference “SiPS” should be capitalized, just as “Dvc” should be expressed by “DVC”.\ne The reference format of the paper is not uniform. For example, on page10, the author of the seventh is inconsistent with other formats. Make sure that some references have been officially published, such as the last one on page 11, which has been published on ICLR. Please check others. On page 12, the conference name of the third reference should be capitalized in accordance with other references. On page 11 there is a problem with the format of the 10th reference. Please check other parts carefully.", "summary_of_the_paper": "This paper mainly introduces a video compression algorithm. The main feature is that it can simultaneously optimize the codec model parameters while compressing the transmitted video, and dynamically adapt to different video styles. The author's main contributions are:\n\na. In this paper, the instance-adaptive data compression method is applied to the compression of complete video sequences. During the test of each video, by fine tuning the model parameters of the codec, the robustness to deal with various fields of video is improved. This method can reduce the model capacity, because a smaller network may suffice to model a single instance, and relax the assumption that the distribution of training set and test set always match. It can be applied to any neural network video codec model.\n\nb. This paper demonstrates the effectiveness of the method through experiments on two different architectures: scale-space flow with low delay setting and B-frame based model.", "summary_of_the_review": "This paper applies the instance-adaptive image compression method to video compression, and proves the effectiveness of the instance-adaptive data compression method in video through the implementation on two existing models, but I want to weak accept it because of the following problems:\n\na The method of this paper is lack of innovation. In fact, most of the work is done by combining other people's methods. There is no doubt that this paper applies the method of I-frame adaptive compression to video and combines it with other better models to get a certain effect. Authors need to highlight their innovative contributions.\n\nb In addition, there are many errors in the references of the manuscript, such as inconsistent format, author abbreviations, etc.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, this paper receives negative reviews due to limited technical novelty and contributions. The authors' rebuttal does not address all the raised concerns. As such, the area chair agrees with the reviewers and does not recommend it be accepted at this conference
Decision: Reject