VCT: A Video Compression Transformer
Authors: Fabian Mentzer
George Toderici
David Minnen
Sergi Caelles
Sung Jin Hwang
Mario Lucic
Eirikur Agustsson
Venue: NeurIPS 2022 Accept (2022)
Keywords: Video compression
transformers

Abstract: We show how transformers can be used to vastly simplify neural video compression. Previous methods have been relying on an increasing number of architectural biases and priors, including motion prediction and warping operations, resulting in complex models. Instead, we independently map input frames to representations and use a transformer to model their dependencies, letting it predict the distribution of future representations given the past. The resulting video compression transformer outperforms previous methods on standard video compression data sets. Experiments on synthetic data show that our model learns to handle complex motion patterns such as panning, blurring and fading purely from data. Our approach is easy to implement, and we release code to facilitate future research.

VCT: A Video Compression Transformer
Fabian Mentzer
George Toderici
David Minnen Google Research dminnen@google.. com
Google Research
Google Research
mentzer@google.co com
gtoderici@google... com
Sung Jin Hwang Google Research
Sergi Caelles
Mario Lucic Google Research
Eirikur Agustsson Google Research eirikur@google..
Google Research
sjhnangdgoogle.com
scaelles@google.co
lucicogoogle.
Abstract
We show how transformers can be used to vastly simplify neural video compression. Previous methods have been relying on an increasing number of architectural biases and priors, including motion prediction and warping operations, resulting in com- plex models. Instead, we independently map input frames to representations and transformer model their dependencies, letting it predict the distribution of future representations given the past. The resulting video compression transformer outperforms previous methods on standard video compression data sets. Exper- iments on synthetic data show that our model learns to handle complex motion patterns such as panning, blurring and fading purely from data. Our approach is easy to implement, and we release code to facilitate future research.
Introduction
Neural network based video compression techniques have recently emerged rival their non-neural counter parts in rate-distortion performance [e.g.. 17 30 42). These novel methods tend to incor- porate various architectural biases and priors inspired by the classic, non-neural approaches. While many authors tend to draw a line between "hand-crafted" classical codecs and neural approaches, the neural approaches themselves are increasingly "hand-crafted", with authors introducing complex connections between the many sub-components. The resulting methods are complicated, challenging implement, and constrain themselves work well only on data that matches the architectural biases. In particular, many methods rely on some form of motion prediction followed by warping operation [e.g..
23 42 41). These methods warp previous reconstructions with the predicted flow, and calculate a residual.
In this paper, replace flow prediction, warping, and residual compensation, with elegantly simple but powerful munsformer-based temporal entropy model. The resulting video compression transformer (VCT) outperforms previous methods standard video compression data sets, while being free from their architectural biases and priors. Furthermore, we create synthetic data to explore the effect of architectural biases, and show that compare favourably previous approaches on the types videos that the architectural components were designed for (panning on static frames, or blurring), despite our transformer not relying on any of these components. More crucially, we outperform previous approaches on videos that have no obvious matching architectural component (sharpening, fading between scenes), showing the benefit of removing hand-crafted elements and letting a transformer learn everything from data.
We transformers compress videos in two steps (see Fig. 1 First, using lossy transform coding (3) we map frames from image space to quantized representations y:, independently for each frame. From y/ we can recover reconstruction Second, let transformer leverage
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1
D
D
Bottleneck
P( Veet, 2: 1)
Vi-2
Yi-1
Masked Block- Autoregressive Transformer
Transformer
1/
E
E
E
y1
Must
Already transmitted
To be transmitted
2
Figure I: We independently and lossily map input frames æ into quantized representations y. From can recover reconstruction ê. To store y, with few bits, we use transformers to model temporal dependencies and to predict a distribution for yi given previously transmitted representations. We use P to losslessly compress the quantized Yi using entropy coding. The better the transformer predicts P. the fewer bits are required to store yi- We note that we have no hard-coded components such as motion prediction warping.
temporal redundancies to model the distributions of the representations. We these predicted distributions to losslessly compress the quantized Yi using entropy coding (43] Sec 2.2.1J. The better the transformer predicts the distributions, the fewer bits required to store the representations.
This setup avoids complex state transitions or warping operations by letting the transformer learn to leverage arbitrary relationships between frames. As a bonus, get rid of temporal error propagation by construction since the reconstruction i; does not depend on previous reconstructions. Contrast with warping-based based approaches, where 2, is a function of the warped Φ;-1 meaning that any visual errors in 2; will be propagated forward and require additional bits to correct with residuals.
VCT is based on the original language translation transformer 35 We can view our problem as "translating" two previous representations y:-2: V/-1 to y,- However, there are various challenges in the way of directly applying the NLP formulation. Consider a 1080p video frame; using typical neural image compression encoder (4) that downscales by a factor 16 and has 192 output channels, (1080, 1920, 3) 3)-dimensional input frame is mapped to (68, 120, 192)-dimensional feature representation leading to approximately 1.6 million symbols. Naively correlating all of these symbols to all symbols in previous representation would yield 1.6M 1. 6M-diimesionnal attention matrix. To address this computationally impractical problem, we introduce independence assumptions to shrink the attention matrix and enable parallel exccution subsets of the symbols.
Our model is easy to implement with contemporary machine learning frameworks, and we provide an extensive code and model release to allow future work to build on this direction
2 Related Work
Transformers were initially proposed for machine translation 35 was used to obtain state-of-the-an results. This led to wide range follow-up research, and state- of-the-art natural language processing (NLP) models are still based on transformers [e.g.. g IT]. Motivated by these advancements, Dosovitski et al. (10) replaced CNNs with a transformer based architecture to achieve state-of-the-at results in image classification, which in turn spurred more exploration of transformers in the computer vision community including both image tasks [e.g.. 22
where encoder-decoder structure
37 45
well video analysis [e.g..
28
32
Recently, transformers were incorporated into neural image compression models. Qian et al. replaced the autoregressive hyperprior (26) with self-attention stack, and Zhu et al. (46) replaced all
convolutions in the standard approach [4
27 with Swin Transformer [22] blocks.
Neural video compression remains CNN-based. After initial work used frame interpolation (39
9). Lu et al. (23) followed the more traditional approach of predicting optical flow between the previous reconstruction and the input, transmitting compressed representation of the flow, and also transmitting residual image to correct visual errors after warping. Many papers extended this
httpe://goo..leevcctppp

---

Tokens
Blocks
Representations
Already Transmitted
To be transmitted
Figure 2: From representations to tokens. We essentially use a sliding window to split the current representation into non-overlapping we x we blocks, and previous representations Vi-2, Ui-1 into overlapping wp x up blocks with stride we (W'p > we). We flatten blocks spatially (raster-sean order, see left arrows) to obtain tokens for the transformer, which remain de-dimensional since they are just a different view of yi- We show w.=3, 5p=5, dc= but we use wt 44 - 192 in practice.
approach, for example Agustsson et al. (T] introduced the notion of flow predictor that also supports
blurring called "Scale Space Flow" (SSF), which became building block for other approaches [42
30). Rippel et al. [30] achieved sffe-offthe-ar results by using SSF and more context to predict flow.
RNNs and ConvLSTMs were used to build recurrent decoders [13 or entropy models [41].
Some work does not rely on pixel-space flow: Habibian el al. used 3D autoregressive entropy model, FVC (17) predicted flow in a 2x downscaled feature space, and Liu el al. (20) used a
ConvLSTM to predict representations which are transmitted using an iterative quantization scheme. DCVC [19] estimated motion in pixel space but performed residual compensation in a feature space. Liu et al. 2T] also losslessly encoded frame-level representations, but rely on CNNs for temporal
modelling. Finally, recent work employed GAN losses to increase realism 24
40
3 Method
3.1
Overview and Background
Frame encoding and decoding A high-level overview of our approach is shown in Fig. | We split video coding into two parts. First, we independently encode each frame x; into a quantized representation y/ y;= [==](xx,)) using a CNN-based image encoder E' followed by quantization to an integer grid. The encoder downscales spatially and increases the channel dimension, resulting in ye being (H, W, -de)-mmemiional feature map, where H, W are 16x smaller than the input image resolution. From Yt. we can recover a reconstruction f;; using the decoder D. We train E, D using standard neural image compression techniques to lossy transforms reaching nearly any desired distortion d(z,, i;:) by varying how large the range of each element in y: is. For now, let us assume we have pair F,, D reaching a fixed distortion.
Naive approach After having lossily converted the sequence of input frames x, sequence of representations y E<x,), one can naively store all y to disk losslessly. To see why this is sub- optimal, let each element yi.j of y, be symbol in S = {- .., L.). Assuming that all [S] symbols appear with equal probability, Le., P(y..) = 1 1/|S|, one can transmit y, using H W de log, [S] bits. Using a realistic T.=32, this implies that would need 9 MB, or 2GGpp at 30fps, to encode a single HD frame (where H W 1 1.6M, see Introduction). While arguably inefficient, this is valid compression scheme which will result in the desired distortion. The aim of this work is to improve this scheme by approximately two orders of magnitude.
An efficient coding scheme Given a probability mass function (PMF) P estimating the true distribution Q of symbols in yt. we can use entropy coding (EC) to transmit y, with H W de- By using EC, we can encode more frequently occurring values with fewer bits, and hence improve the efficiency. Note that the expectation term representing the average bit count corresponds to the cross-entropy of Q with respect to P. Our main idea is to parameterize P
(oog
bits Fl
Conssistent with neural compression literature but in contrast Information Theory, we use P for the model.
3

---

Masked
Transformer Layer
3pint
T
MLP
Transformer Layer
4x
Transformer Layer
P(4)...)
Noma
e+
Temporal Pos
T
MDSA
T
T
Necon
Maskod
the
Transformer Layer 6x 6x Transformer Layer
5xx Transformer Layer
MUSA
Project + Pos
Project + Pos
Pro Project Pos
Norm
Miwiond MHSA
b
Noma
Already Transmitted
To be transmitted
Figure 3: The transformer operates on the pink set of blocks/tokens b4-2, b by (obtained as shown in Fig. 2). We first extract temporal information Sjoint from already transmitted blocks. Tour is shown predicting P(talts, t1, 22, ajoint) where ty is a learned start token.
as conditional distribution using very flexible transformer models, and to minimize the cross-entropy and thus maximize coding efficiency. We emphasize that we use P for lossless EC, we do not sample from the model to transmit data. Even if the resulting model of P is sub-optimal, y still be stored losslessly, albeit inefficiently.
Why would one hope to do better than the uniform distribution over y:? In principle, the model should be able to exploit the temporal redundancy across frames, and the spatial consistency within frames.
3.2 Transformer-basee Temporal Entropy Model
To transmit a video of F frames, IF,, we first map E over each frame obtaining quantized representations V/,,-- ... yF- Let's assume we already transmitted y/..-.., M/-1- To transmit yt+ we use the transformer to predict (/ 11 Using this distribution, we entropy code y, to create 91-2:
a compressed, binary representation that can be transmitted. To compress the full video, we simply apply this procedure iteratively, letting the transformer predict ((y padding with zeros when predicting distributions for y1, V2- The receiver follows the same procedure to recover all yj. i.e., iteratively calculates P(y; 9j 1)-1) to entropy decode each yj- After obtaining each representation, f1, U2. yF, the receiver generates reconstructions.
for j {I..... F).
Tokens When processing the current representation yı, we split it spatially into non-ovene non-overlapping blocks with size w. x we shown in Fig. 2 Previous representations 3/-2, V/-1 become correspond- ing overlapping wp x wp blocks (where Wp >> wa) to provide both temporal and spatial context for 11 Intuitively, the larger spatial extent provides useful context to predict the distribution of the current block. Note that all blocks span a relatively large spatial region in image space due to the downscaling convolutional encoder E. We flatten each block spatially (see Fig. 2 to obtain tokens for the transformers. The transformers then run independently on corresponding blocks/tokens, i.e., tokens of the same color in Fig. get processed together, trading reduced spatial context for parallel execution
predicting P(y 22 ((y:(y
This independence assumption allows us to focus single set of blocks, e.g., the pink blocks in Fig. 2 In the following text and in Fig. 3 we thus show how we predict distributions for the =²=16 tokens f1, 12,22..... t in block 6, given the 2w tokens from the previous blocks b,- b
Step 1: Temporal Mixer We use two transformers to extract temporal information from b;-2, b first transformer Teep operates separately on each previous block. Then, we concatenate the outputs in the token dimension and run the second transformer, T Tjoint. on the result to mix information across time. The output 2joint is 2n² features, containing everything the model "knows" about the past.
Step 2: Within-Block-ee Autor--outorrgression The second part of our method is the masked transformer Teur, which predicts PMFs for each token using auto-regression within the block. We obtain a
3 As side benefit, the number of tokens for the transformers is not function of image resolution, unlike VIT- based approaches I0]

---

PSNR on MCL-JCV
on
VCT (Ours)
38
38
PLF-V--Oocva
DEVE (NeutPS211)
37
37
FVC (CVPRC21)
36
36
SST (CVPR 20)
(ASTSEA))
35
et (BCCV20)
DVC (CVPR19)
HEVC veryslow
33
HEvC medium
32
52
11.664 medium
0.10
0.20
0.40
MS-SSIM on MCL-JCV
MS-SSIM on UVG
0.990
0.985
0.98
0.500
0.975
0.970
0.96
0.960
0.95
110ps
577
11.4
17.0
24.5
49.8
74.6
Figure 4: Comparing rate-distortion on MCL-JCV (*=2FPSS and UVG (120FPS), We report bits per pixel (bpp) and megabits per second (mbps). For MS-SSIM, we only show methods optimized for (using tune ssim for HEVC/H264). App. A.5 shows large version of these plots.
powerful model by conditioning Teur on Zjoint as well as already transmitted tokens within the block. For entropy coding, both the sender and the receiver must be able to obtain exactly the same PMFs, i.e. Tour must be causal and start from a known initialization point. For the latter, we learn a start token ts.
To send the tokens, we first obtain Pjoint- After that, we feed [ts] Teurs obtain P(t Its; 2join). and use entropy coding to store the de symbols in token 4 into bitstream using P(4 Fjoim): ). Then, we feed [4s, 41). obtain P((2 (1 ls; Zjaim); store l2 in the bitstream, and so on. The receiver gets the resulting bitstream and can obtain the same distributions, and thereby the tokens, by first feeding [ts] to T cur. obtaining P(t, [t:; 2joint) ) entropy decoding 4 from the bitstream, then feeding (ts, 41] to Fig. visualizes this for P(tz
obtain (t tsi 2joint), and
We run this procedure in parallel over all blocks, and thereby send/receive by running Tewr 2 16 times. Each run produces [//w]] [W/w.] de distributions. To ensure causality of T during training, we mask the self-attention blocks similar to 35
Independence Apart from assuming blocks in Yi are independent, we emphasize that each token is vector and that we assume the symbols within each token are conditionally independent given previous tokens, i.e., Teur predicts the de distributions required for token at once. One could instead predict joint distribution over all possible [S dd realisations, use channel-autoreeression (27) or use vector quantization on tokens. The latter two are interesting directions for future work. Finally, we note that we do not rely on additional side information, in contrast e.g., autoregressive image compression entropy models [26 27
3.3 Architectures
Transformers As visualized in Fig. all of our transformers are based standard architec- tures [35] TOJ. We start by projecting the dimdiessii tokens dd-dimensional space (dy =7y in our model) using a single fully connected layer, and adding a learned positional embedding. While both Teep and jo are stacks of multi-head self-attention (MHSA) layers, Taw uses masked
"conditionall" transformer layers, similar to Vaswani el al. (35): These alternate between masked
MHSA layers and MHSA layers that use 2join as keys (K) and values (V), as shown Fig. 3 We
transformer layers for Tepp. 4 for Tjonn and 5 masked transformer layers for Tear- We use 16
5

---

a) Shift
b) SharpenOrBlur
c) Fade
T= 10
3=30
2=2
2=2.0
0010
m=0020
0.9
VCT (O
Averge Lxolatcan Los
0.8
SSF
1IEVC
0.35
0.7
0.30
0.6
0.6
0.25
0.5
0.5
0.20
0.4
0.4
0.3
0.10
0.00
Amount of motion videos
sharpen
bir
Fade speed
Figure 5: To understand what types of temporal patterns our transformer has learned to exploit, we synthesize videos representing commonly seen patterns. We compare to HEVC, which has built-in support for motion, and SSF, which has built-in support for motion and blurrying. VCT learns to handle all patterns purely from data. We refer to the text for a discussion.
attention heads everywhere. We learn separate temporal positional embedding to add to the input of Joint-
Image encoder E, decoder D The image encoder and decoder E, D are the focus of this paper, so we use architectures based on standard image compression approaches [26] 27). For the encoder, we 4 strided convolutional layers, downscaling by factor 16x in total. For the decoder, we use transposed convolutions and additionally add residual blocks at the low resolutions. We use dgD=192 filters for all layers. See App. A.1 for details and an exploration of architecture variants.
3.4 Loss and Training Process
The modeling choices introduced the previous section allow for an efficient training procedure where we decompose the training into three stages, which enables rapid experimentation (Tab. [ In Stage we train the per-frame encoder E and decoder D by minimizing the rate-distortion
trf (3 Sec 3.1.1). Lf denote uniform distribution in [-0.5, 0.5). We minimize
L = Bespx wnll log plū + u) +A -AMSE(I, R
T=E(z), t= D(roundsTE (ū)). (I)
bit-rate
distortion
using i to refer to the unquantized representation, and æ Px are frames drawn from the training set. Intuitively, we want to minimize the reconstruction error under the constraint that we can effectively quantize the encoder output, with À controlling the tradeoff. For Stage I, we thus employ the mean- scale hyperprior (26) approach to estimate p. the de facto standard in neural image compression, which we discard for later stages To enable end-to-end training, we also follow (26). adding i.i.d.
"In short, the hyperprior estimates the PMF of y using VAE 18 by predicting p(y) z)), where is side information transmitted first. We refer the paper details 26
Components trained D.
Loss
B
Np
LR
Steps
steps/s
Stage Stage II
r + Xd
16
1E=4 4
2M
100
Teep: Tjoints Tour Teep Tjoint, Tours E, D
32
3
1E
1M
10
Stage III
r + Xd
32
3
2.5
250k
5
Table We split training in three stages for training efficiency (note the steps/s column). À controls the rate-distortion trade-off, r is bitrate, d is distortion, B is batch size, Nr the number of frames.

---

21
R1-1
Decode:
0 tokans
block
tokins
black
13
tokkens
block
OkB
0.13kB
0.78kB
Figure 6: Visualizing the sample mean from the block-autoregresssvve distribution predicted by the transformer, as decode more and more tokens (see Sec. [5) We show the kilobytes (kB) required to transmit the decoded (gray) tokens. On the left, we see the two previous reconstructions 2; - In the middle, any information (OkB). The next twO images shows that as we decode more tokens, the model gets more certain. and the image obtained from the sample mean sharpens. Note that we never sample from the model for actual video coding.
what the transformer expects the current frame, before decoding
uniform noise u to ÿ when calculating r. and using straight-through estimation (STE) gradients when rounding ÿ to feed it to D.
33 27 for
For Stage II, we train the transformer to obtain p. and only minimize rate:
Lu Bo
p(ūs + u/y1, v2)]
D Ú--E(x), y: ro-nound(j.).
where (21, 22, Ta) ~ Px1s are triplets of adjacent video frames. We assume each of the de unquantized elements in each token follow Gaussian distribution, p ~ NV, and let the transformer predict de means and de scales per token. Finally, we finetune everything jointly in Stage III, adding the distortion loss d from Eq. Io Eq.
We note that is also possibe to train the model from scratch and obtain even better performance, App.
To obtain a discrete PMF P for the quantized symbols (for entropy coding), we again follow standard practice [4). convolving p with unit-width box and evaluating it at discrete points, P(y) Sueu P(y+ (y+-u)da y E Z [see, e.g., 43. Sec. 3.3.3, for details]. train, we use random spatio-temporal crops of LB , 256, 256, 3) pixels, where B is the batch size, and NF the number of frames (values given in Tab. T. We use the linearly decaying learning rate (LR) schedule with warmup, where we warmup for 10k steps and then linearly decay from the LR shown in the table to lE Stage I is trained using 0-0011. To navigate the rate-distottion trade-off and obtain results for multiple rates, we fine-tune 9 models in Stage !II, using - 2', 22. Google Cloud TPUv4 chips.
.3....,. 5). We train all models on
3.5 Latent Residual Predictor (LRP)
To further leverage the powerful representation that the transformer learns, we adapt the "latent residual predictor" (LRP) from recent work image compression [27]: The final features žeur from Teur have the same spatial dimensions as Yi, and contain everything the transformer knows about the current and previous representations. Since we have to compute them compute P. they constitute "free" extra features that are helpful to reconstruct ft, We thus use Seur by feeding vo = ys fLue to (we enable this Stage II/), where fLRP consists of a 1 convolution mapping from dr to dED followed by residual block. We note that this implies that é; D(v() indirectly depends on U:-2; V/-1, Vi- Since this is a bounded window into the past and y does not depend on Hzcis we remain free from temporal error propagation.
7

---

Context
LRP
bpp /
PSNR f
No previous frames (image codec)
0
0.218
36.1
previous frame
1
0.0907 (-58%)
36.1 1
2 previous frames
2
0.0775 (-64%)
36.1
2 previous frames and LRP (VCT (Ours)
2
v
0.0775 (-64%)
36.8 (+0.7dB)
Table 2: Ablating how many previous frames we feed to the transformer ("Context"), and whether we use latent-residual prediction (LRP).
4 Experiments
4.1 Data sets
We train on one million Internet video clips, where each clip has nine frames. We obtained high- resolution videos which we downscale with a random factor (removing previous compression arti- facts), from which get a central 256 crop. Training batches are made up of randomly selected triplets of adjacent frames. We evaluate on two common benchmark data sets: (1) MCL-JCV (36) MIT Licence] made up of thirty 1080p videos captured at either 25 30FPS and averaging 137 frames per video, and (2) UVG [25 CC-BY-NC Licence] containing twelve 1080p 120FPS videos with either 300 or 600 frames each.
Synthetic videos We explore three parameterized synthetic data sets that we build by generating videos from still images from the CLIC2020 test set [34] Unsplash licence], (see Fig. 5). Each data set has a parameter x that we vary, and we create 100 videos for each value of x. Each video is 12 frames of 512x512px. We explore: Shift, where we pan from the center of the image towards the lower right, shifting by pixels in each step. SharpenOrBlur, where if x20, we apply Gaussian blurring with sigma x 1 at time step f. If x<0, create videos that get sharper over time by playing a video blurred with |x]] in reverse. Fade, where we linearly transition between two unrelated images using alpha blending (as in scene cut). We release the code synthesize these videos.
4.2 Models
We refer to our video compression transformer as VCT. We the widely used, non-neural, standard codec HEVC [31] (a.k.a. H.265) using the ffmpeg x265 codec in the medium and veryslow settings, as well H.264 using x264 in the medium setting. For fair comparison to method, we follow previous work [T 24 30 in disabling B-Frames, but do not constrain the codecs in any other way. We the public DVC 23 code, and additionally obtain numbers from the following papers: SSF (T), which introduced scale-space-flovw an architectural component to support warping and blurring, commonly used in follow-up work, ELF-VC (30), to the best of our knowledge the state-of the-art neural method in terms of PSNR on MCL-JCV, which extends the motion compensation of SSF with more motion priors, FVC [17] and DCVC [19]. both strong models based warping plus residual coding in a representation space, RLVC 41]. using ConvLSTMMs as a sequence model, and Liu al. (21), who study losslessly transmitting representations using CNNs for temporal entropy modelling. To explore how architectural biases behave on synthetic data, we reproduce SSF, using exactly the same training data as for VCT.
4.3 Metrics
We evaluate the common PSNR and MS-SSIM (38 in RGB. We train all models using MSE as distortion and use 200 (1- MS-SSIM &)) as the training objective Stage III (Tab. Tp obtain MS-SSIM models.
8

---

5 Results
5.1 Comparison to State of the Art
In Fig. we depict rate distortion graphs for our method and the neural video compression methods introduced in Sec. 4 on MCL-JCV and UVG. Despite the simplicity of our approach, and the fact that we use no motion or warping components, we outperform all methods in both PSNR and MS-SSIM.
5.2 Synthetic data
In Fig. 5 we show how the transfomer learns to exploit various types of temporal patterns by applying to the synthetic data sets introduced in Sec. [ and reporting the evaluation R-D loss 5
We compare to HEVC and SSF, which both have explicit support for shifting motion, while SSF also has explicit support for blurring. We expect them to perform well on temporal patterns for which they have corresponding architectural priors. In contrast, VCT has no such priors. For each data set, explore different values for the parameter æ (see Sec. loss over the 100 videos created with x.
point the plot represents the average evaluation
We observe: a) On videos with shifting based motion, VCT obtains =45% lower R-D loss compared to SSF, which saturates at about x = 10, presumably due to the shallow CNN used for flow estimation. Since HEVC supports motion compensating with arbitrary shifts of previous frames, it excels on these kinds of videos. For shifts that are a multiple of 16, the representations shifts by exactly 1 symbol in each step, and VCT matches HEVC. The reason for this is that our encoder so it is only chitt-equivariant for shifts which are multiples of the stride (16). Any shift in [1, 15] pixels causes the representation to change in a complex way (cf. [44)). b) For blurring/hhareningg. we outperform both HEVC and SSF; despite the latter having explicit support for blurring. Note that the curve for SSF is asymmetric: since it has built-in support for blurring, it gets a 2:20% lower RD loss on blurring compared to sharpening. loss as we increase r, in contrast to the baselines, neither of which has a explicit support for fading. SSF is =20% better than HEVC, possibly due to its blurrying capabilities. For completely static videos x=0, we observe that VCT is at a slight disadvantage compared to the previous approaches. Overall, we believe that synthetic data give better insight into the strengths and weaknesses of methods, and hope that future work can compare on these data sets.
CNN,
VCT learns to handle fading, exhibiting
near-constant RD
5.3 Visualizing certainty during decoding
After having seen i tokens in each block, the transformer predicts a PMF P(tx+1 ((gg, Syoim). This induces a joint distribution P(t>x|_... over all unseen (not yet decoded) tokens. Intuitively, if the transformer is certain about the future, this distribution will be concentrated on the actual future tokens we will decode. In Fig. 6 we visualize the sample mean of this distribution by feeding it the
through D, i.e., we sample N realisations of the unseen tokens in each block, conditioned already decoded ones, for k E {0, 2, 13}. In the middle image in Fig. 6 we show what the transformer expects at the current frame, before decoding any information (k- i.e., bits). We observe that the model is able-to some degree-to learn second order motion implicitly. The next two images shows that as we decode more tokens, the model gets more certain, and the image sharpens.
5.4 Ablations
In Tab. 2 we explore the importance of temporal context from previous frames and latent residual prediction (LRP) on MCL.-JCV. We start from a baseline that does not use any previous frames, iee, an image model, used to independently code each frame. Conditioning on one previous frame reduces bitrate by- 58%. Using two previous frames yields an additional improvement of 6%% In the final configuration (our model, VCT), which adds LRP, we observe an increase in PSNR of 0. at the same bitrate. We did not observe further gains from more context.
'C=r+ + Ad. calculate C for HEVC, we find the quality factor y matching our À via q=arg min,r(g) + Ad(g). which yields yields 222 for 0

---

sop and Join
Tour
EC
D
FPS estimate
Ours
1080p
168 ms
326 ms
30.5 ms
168 ms
1.. FPS
720p
37.6 ms
44.8 ms
17.0 ms
49.5 ms
66. FPS
480p
18.1 ms
23.1 ms
9.02 ms
23.3 ms
113. FPS
360p
7.3 ms
14.9 ms
4.24 ms
10.1 ms
*=7.3 FPS
Table Runtimes of our components. For ours, we use Google Cloud TPU v4 to run transformers and D. Entropy Coding (EC) is run on CPU.
5.5
Runtime
To obtain runtimes of the transformers (Tepp. Toints Teur) and the decoder (D), we employ a Google Cloud TPU v4 (single core) using Flax (16). which has an efficient implementation for autoregressive transformers. We use Tensorflow Compression to measure time spent entropy coding (EC), on an Intel Skylake CPU core. In Tab. [] we report numbers for 1280 1280x720px, 852x480px, and x60x3 Since this benchmark is not fully end-to-end, we only report an FPS estimate by calculating 1000/(s (suu/ of individual runtimes in ms). Note that running Teur at 720p once only takes 22,8ms, but we run it 11 times to decode a frame. To run Joint. we only have to run Tap once representation, since we can re-use the output of running Tep the previous representation.
Many neural compression methods do not detail inference time and do have code available, but we copy the results from DCVC [9]. FVC [17], and ELF-VC (30). in Table 4
6 Conclusion and Future Work
We presented an elegantly simple transformer-bant approach to neural video compression, outper- forming previous methods without relying on architectural priors such explicit motion prediction or warping. Notably, our results are achieved by conditioning the transformer only on a 2-frame window into the past. For some types of videos, it would be interesting to scale this up, or to introduce notion of more long-term memory, possibly leveraging arbitrary reference frames.
As mentioned towards the end of Sec. 3.2 various different ways to factorize the distributions could be explored, including vector quantization, channel- autoregression or changing the independence assumptions around how we split representations into blocks.
We hope our method can serve as the foundation for a new generation of video codecs. This could have net net-positive impact society by reducing the bandwidth needed for video conferencing and video streaming and to better utilize storage space, therefore increasing the capacity of knowledge preservation.
Societal Impact
We thank Basil Mustafa, Ashok Popat, Huiwen Chang, Phil Chou, Johannes Ballé, and Nick Johnston for the insightful discussions and feedback.
Acknowledgements
Resolution
FPS estimate
Ours
1080p
11.4 FPS
720p
<6.7 FPS
480p
133.6 FPS
360p
*=27.3 FPS
DCVC 119
1080p
x1.1 FPS
FVC [17]
1080p
#1.8 FPS
ELE-VC (30
1080p
118 FPS
720p
<55 FPS
Table 4: Comparing decoding speed to other methods. We directly copy reported results from the respective papers, so platforms are not comparable.
10

---

References
[1] Eirikur Agustsson al. "Scale-space flow for end-to-end optimized video compression". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, pp. 8512.
[2] Anurag Amab et al. "Vivit: A video vision transformer", In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 6836-6846.
[3] Johannes Ballé et al. "Nonlinear transform coding". In: IEEE Journal of Selected Topics in Signal Processing 15.2 (2020), pp. 339-353.
[4] Johannes Ballé et al. "Variational image compression with a scale hyperprior". In: International Conference on Learning Representations (ICI.R.. 2018.
[5] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. "Is Space- Space-Time Attention All You Need for Video Understanding?" In: International Conference Machine Learning. PMLR. 2021, pp. 813-824.
[6] Tom Brown et al. "Language models are few-shot learners". In: Advances in neural information processing systems 33 (2020), pp. 1877-1901.
[7] Aakanksha Chowdhery et al. "Palm: Scaling language modeling with pathways". In: arXiv preprint arXiv:2204.00 02311 (2022).
[8] Jacob Devlin et al. "Bert: Pre-training of deep bidirectional transformers for language under- standing". In: arXiv preprint arXiv: 1810.04803 (2018).
[9] Abdelaziz. Djelouah et al. "Neural inter-frame compression for video coding". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019, pp. 6421-6429
[10] Alexey Dosovitskiy et al." "An image is worth 16x16 words: Transformers for image recognition
at scale". In: arXiv preprint arXiv:2010.1 1929 (2020).
[11] Sergey Edunov arXiv::808. 0031888880033/// (2018).
al. "Understanding back-translation at scale". In: arXiv preprint
[12] Haoqi Fan et al. "Multiscale vision transformers". In: Proceedings of the IEEF/CVF Interna- tional Conference on Computer Vision. 2021, pp. 6824-6835.
[13] Adam Golinski et al. "Feedback recurrent autoencoder for video compression". In: Proceedings of the Asian Conference on Computer Vision. 2020.
[14] Amirhossein Habibian et "Video compression with rate-distortion autoencoders". Pro- ceedings of the IEEE/CVF International Conference on Computer Vision. 2019, pp. 7033- 7042.
[15] Dailan He et al. "Elic: Efficient learned image compression with unevenly grouped space- channel contextual adaptive coding". In: Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition. 2022, pp. 5718-5727.
[16] Jonathan Heek et al. Flax: A neural network library and ecosystem for JAX. Version 0.4.2.
2020. URL: http:://// / htt///gitttu com/ google/flax
[17] Zhihao Hu, Guo Lu, and Dong Xu. "FVC: A new framework towards deep video compression in feature space". In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2021, pp. 1502-1511.
[18) Diederik P Kingma and Max Welling. "Auto-encoding variational bayes". In: arXiv preprint arXiv:1312.6114 (2013).
[19] Jiahao Li, Bin Li, and Yan Lu. "Deep contextual video compression", In: Advances Neural Information Processing Systems 34 (2021).
[20] Bowen Liu et al. "Deep learning in latent space for video prediction and compression". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 701-710.
[21] Jerry Liu et al. "Conditional entropy coding for efficient video compression". In: European Conference on Computer Vision. Springer. 2020, pp. 453-468.
[22] Ze Liu et al. "Swin transformer: Hierarchical vision transformer using shifted windows". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 10012- 10022.
[23] Guo Lu al. "Dve: An end-to-end deep video compression framework". In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, pp. 11006-11015.

---

[24]
Fabian Mentzer et al. "Neural Video Compression using GANs for Detail Synthesis and Propagation". In: arXiv preprint vxXi:2207.12038 (2021). Alexandre Mercat, Marko Viitanen, and Jarno Vanne. "UVG dataset: 50/120fps 4K sequences for video codec analysis and development". In: Proceedings of the 11th ACM Multimedia Systems Conference. 2020, pp. 297-302.
[25]
[26] David Minnen, Johannes Ballé, and George D Toderici. "Joint autoregressive and hierarchical priors for learned image compression". In: Advances in Neural Information Processing Systems. 2018, pp. 10771-10780.
[27] David Minnen and Saurabh Singh. "Channel-wise autoregressive entropy models for learned image compression". In: arXiv preprint arXiv:2007.0873: (2020).
[28] Daniel Neimark et al. "Video transformer network". In: Proceedings of the IEEEJCVF Interna- tional Conference on Computer Vision. 2021, pp. 3163-3172.
[29] Yichen Qian et al. "Entroformer: A Transformer-basel Entropy Model for Learned Image Compression". In: arXiv preprint arXiv:2202.05442 (2022).
[30] Oren Rippel al. "Elf-ve: Efficient learned flexible-rate video coding". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 14479 14498.
[31] Gary J Sullivan et al. "Overview of the high efficiency video coding (HEVC) standard". In: IEEE Transactions on circuits and systems for video technology 22.12 (2012), pp. 1649-1668.
[32] Chen Sun et al. "Videobert: A joint model for video and language representation learning". In: Proceedings of the IEEF/CVF International Conference on Computer Vision. 2019, pp. 7464- 7473.
[33] Lucas Theis et al. "Lossy image compression with compressive autoencoders". In: Interna- tional Conference on Learning Representations (ICLR). 2017.
[34] George Toderici et al. CLIC 2020: Challenge on Learned Image Compression. http:
compression cc. 2020.
[35] Ashish Vaswani al. "Attention is all you need". In: Advances in neural information process- ing systems 30 (2017).
[36]
Haiqiang Wang et al. "MCL-JCV: a JND-based H.264/AVC video quality assessment dataset".
In: 2016 IEEE International Conference on Image Processing (ICIP). IEEE. 2016, pp. 1509- 1513.
[37]
Huiyu Wang et al. "Maxxdeeplab: End-to-end panoptic segmentation with mask transformers". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 5463-5474.
[38] Zhou Wang, Eero P Simoncelli, and Alan C Bovik. "Multiscale structural similarity for image quality assessment". In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003. Vol. 2. leee. 2003, pp. 1398-1402.
[39] Chao- Yuan Wu, Nayan Singhal, and Philipp Krahenbuhl. "Video compression through image interpolation". In: Proceedings of the European Conference on Computer Vision (ECCV). 2018, pp. 416-431.
[40] Ren Yang, Luc Van Gool, and Radu Timofte. "Perceptual Learned Video Compression with Recurrent Conditional GAN". In: arXiv preprint 2 009003082 (2021).
[41] Ren Yang et al. "Learning for Video Compression with Recurrent
and Recurrent Probability Model". In: IEEE Journal of Selected Topics in Signal Processing 15.2 (2021),
pp. 388-401.
[42] Ruihan Yang et al. "Hierarchical autoregressive modeling for neural video compression". In: arXiv preprint arXiv:2010. 10258 (2020).
[43] Y. Yang, S. Mandt, and I.. Theis. " "An Introduction to Neural Data Compression", preprint.
2022. URL: https: //axiivaaiivv///aa org/ a20/ 06533
[44] Richard Zhang. "Making convolutional networks ihift-invariant again". In: International conference on machine learning. PMLR. 2019. pp. 7324-7334.
[45] Sixiao Zheng et al. "Rethinking semantic segmentation from a
perspec-
tive with transformers" In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021, pp. 6881-6890.
[46] Yinhao Zhu, Yang Yang, and Taco Cohen. "Transformer-based Transform Coding". In: Inter- national Conference on Learning Representations. 2021.
12

---

NeurIPS Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] see Sec
(c) Did you discuss any potential negative societal impacts of your work? [Yes] see Societal Impact in S
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [No] We cannot release training data but will release code the paper is published.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] However, we find in most experiments, multiple runs end at similar final losses.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We specify training platform and training times in 3.4 as well as how many models we train.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Sec
(b) Did you mention the license of the assets? [Yes] See Sec 4
(c) Did you include any new assets either in the supplementary material or as a URL?
YYs] We will release Github URL to code upon publication.
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A] We don't 't release new data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We don't 't release new data.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A] No crowdsourcing or human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] No crowdsourcing or human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] No crowdsourcing human subjects.
13


Reviews:
[{"review_id": "wA3dKxtJEvU", "reviewer": "Reviewer_7MPC", "summary": "The authors tackle the problem of video compression using transformers. The proposed architecture doesn't include specific biases and priors like motion prediction, blurring, etc. Transformers are used to predict the distribution of the future representation given the earlier video frames. The idea is to exploit the temporal redundancy across frames and the spatial consistency within frames which can help in entropy coding. They show good results using transformers on standard video compression data sets (MCL-JCV, UVG, synthesized videos from CLIC2020) as compared to previous approaches.", "questions": "It may help if the authors can tell how the entropy coding is done in the current context (see weaknesses above). Also if the authors can address any other weaknesses mentioned above then it can help in better understanding of the approach.", "limitations": "Yes the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* There is no explicit biases or priors, which may help the proposed architecture to generalize better across datasets\n* The authors introduce independence assumptions among the blocks of each video frame which shrinks the attention matrix and enables parallel execution on subsets of the video frame.\n\nWeaknesses:\n* The paper is not easily readable as there are several inconsistencies as pointed below\n* In Fig 1, it seems that y_i is fed to the Masked Block-Autoregressive Transformer, but from the later text, it seems it only previous tokens of y_i are fed to get the probability of the current token of y_i.\n* line 207: Even though it is a bounded window, if z_cur is used to decide y'_i then y'_i does indirectly depend on all previous encodings which can propagate errors. This counters the claim made in line 39 that the architecture doesn't propagate temporal errors.\n* Entropy coding: The distribution of the next token can help identify the number of bits to encode the token. But it is not clear how one of the S symbols can be arbitrarily encoded into that many bits.\n* Availability of code or at least pseudo-code in the appendix could have helped understand how entropy coding is done in the current approach\n* Table 3: To evaluate better, the runtime of the baselines should also be presented\n\nMinor:\n* Figure 4: bits per pixel is not cited\n* line 189: what is distortion loss?\n* line 280: Even though no further gain is observed from more context, it may help if it is shown in the Table 2.\n\nTypos:\n* line 36: '2.1.1' instead of '2.2.1'\n* line 56: 'for' is redundant\n* line 108: The output of T_cur should be z_cur, but the figure has z_joint.\n* Eqn 2: round(y_i) should be round(\\tilde(y)_i) ?\n* line 192: integration is done over du ?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "r_6zlwehFpe", "reviewer": "Reviewer_yzCS", "summary": "The paper proposes a transformer architecture for video compression and demonstrates its favorable performance over prior methods. The method is explicit into two stages: firstly lossily encode each frame independently into a quantized latent space, and secondly use a transformer model to find a probability mass function under which the quantized latent sequence can be efficiently transmitted. The proposed model relies on data-driven prior instead of architectural biases as used in many prior methods. ", "questions": "1. In table 2, the PSNR of model variants with 0, 1 and 2 previous frames are the same. Does the author have an intuition why this is the case? \n2. The paper suggests that more than 2 context frames do not provide further gains in line 280. Is this a result of the type of motion presented in the testing datasets? With larger temporal motions in the testing data, will more frames provide additional benefits, or the it's the capacity of the current model architecture that limits more context information from being efficiently used? \n2. A potential extension of the work is to incorporate temporal-aware quantization in the first-stage training instead of processing each frame individually. ", "limitations": "The limitations and societal impact are addressed. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is clearly written and easy to follow.\n2. The method achieves favorable empirical performance on standard benchmarks. \n2. The paper discusses the the type of temporal patterns that the model learns to exploit by examining synthetic videos. The discussion provides a good understanding of the model's capability when applied to data sets with different motion patterns. \n\nWeaknesses:\n1. The comparisons of model size and inference time with baseline methods are not listed.\n2. The paper provides an intuitive explanation in line 108-110 of why the second stage is advantageous compared to the naive solution. The hypothesis could be tested on synthetic datasets, which will further strengthens this claim and verifies if this is indeed what the type of inductive bias that the model learns from the data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M2HxUoy6CLC", "reviewer": "Reviewer_xvyj", "summary": "This paper uses transformer to capture the relevance between the latent representations of different frames. By using the latent representations of two previous frames as the entropy model input, a good probability estimation can be obtained for that of the current frame. In addition, an improved auto-regressive model is used, where 16 times are needed to decode a frame. ", "questions": "- The authors use TPU to measure the training and testing time. However, TPU is not available for most researchers. Profiling in NVidia GPU is recommended, and it is easy for the comparison with other related works. \n- This paper manually uses the latent representations of 2 previous frames, why not 3 or 4 frames?\n-  Each block is divided into 16 parts, why not 4, 9, 25,or 36 parts?\n", "limitations": "No potential negative societal impact ", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrength\n\n- This paper investigates the conditional entropy coding-based framework rather than following the mainstream residual-coding based framework where the motion estimation and motion compensation are needed. This is an encouraging attempt. \n- Investigate different kinds of synthetic videos including shifting, sharpen/blur, and fading. Some analyses are conducted.\n\nWeakness\n\n- The most issue of this paper is that the novelty is limited. The core contributions have been investigated in other papers.\n  - Using the latent representations of previous frames to estimate the probability for that of the current frame has been proposed in [1]. The difference is that one previous frame is used in [1] and two previous frames are used in this paper.\n  - Using transformer as the basic unit has been investigated in many neural image codecs, like [2,3,4,5]. Directly applying transformer into neural video codec is straightforward and the novelty is limited.\n  - The block-based auto-regressive entropy model is a trade-off between the normal auto-regressive model and checkboard prior model [6]. Actually, the checkboard prior can be regarded as a kind of block-based auto-regressive entropy model, where the block contains two parts. In addition, [7] has investigated a similar idea where each block contains 4 parts. The difference of this paper is that each block contains 16 parts. \n - This paper claims that the motion prediction and warping operations are complex and hand-crafted in mainstream residual-coding based framework, and this paper does not need these designs. The block-based auto-regressive entropy model, LRP, and  3-stage training are also complex and hand-crafted designs. \n - The compression ratio improvement over previous SOTA ELF-VC is limited. The RD-curves are very close. The BD-rate numbers are best presented. \n - Actually, HEVC and H.264 only specify the decoding part and exclude the encoder part. They do not have “veryslow” or “medium” settings. Please use x264 and x265 in the experiment description. In addition, outperforming x264 and x265 is very easy in 2022. Comparisons with JM, HM, VTM are recommended because they represent the best encoders of H.264, HEVC, VVC. The work [8] in 2021 has already compared with VTM. \n\n[1]. Conditional Entropy Coding for Efficient Video Compression, ECCV 2020\n\n[2]. The Devil Is in the Details: Window-based Attention for Image Compression, CVPR 2022\n\n[3]. Entroformer: a transformer-based entropy model for learned image compression, ICLR 2022\n\n[4]. Transformer-based transform coding, ICLR 2022\n\n[5]. Joint Global and Local Hierarchical Priors for Learned Image Compression, CVPR 2022\n\n[6] Checkerboard Context Model for Efficient Learned Image Compression, CVPR 2021\n\n[7] High-Efficiency Lossy Image Coding Through Adaptive Neighborhood Information Aggregation, arXiv 2022.\n\n[8] Versatile Learned Video Compression, arXiv 2021\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7an1olKB5w_", "reviewer": "Reviewer_w64c", "summary": "The authors proposed a video compression transformer (VCT) for neural video compression. Input frames are mapped to representations, and a transformer is used to model the dependencies. The VCT model achieves SOTA performance on MCL-JCV and UVG datasets. Both theoretical and empirical results are promising and will benefit future research.", "questions": "1. What’s the HEVC setting in Figure 5? Since SSF is not using a pretrained optical flow model, I wonder what other models (e.g., DVCPro or DCVC that builds on SPyNet) would behave on the shift sequences. It seems that learning-based models achieve superior benchmark performance than HEVC by handling distortions such as sharpen/blur/fade better, but not as well to shifts. It’s unclear what’s the cost of not using motion compensation in the VCT framework, whether the cost is negligible or if the transformer predicts good enough distributions such that an ME module is unnecessary.\n\n2. It seems that the VCT handles shifts better when the shift is the same size as the patch size. Are the authors aware of/considering any approach to improve this problem?\n\n3. VCT and SSF are trained on different data from most other works. Although larger-scale training has been proved necessary on various language/vision tasks, is there an ablation study on the size of the training data for VCT? This would help to understand the limitation of the model, its scalability, and its potential for future work.\n\n4. A less important question: have the authors considered the HEVC dataset and other video compression metrics (e.g., AVQT or perceptual-based metrics)? As VCT is a novel architecture, it would help to understand how it compares with previous works as well as decide future directions.", "limitations": "In this work the authors proposed VCT for neural video compression. The model design is novel and elegant and the model architecture follows most common designs. It helps to show the efficacy of the proposed framework and many improvements are also possible (as summarized in Section 6). Some possible limitations in terms of motion compensation and training data are mentioned in Question 1 and 3.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1. The proposed framework is simple and effective. The frames are decoded from the quantized representation, which prevents the problem of error propagation for residual coding-based video compression methods.\n\n2. The initial frames are decoded with zero padding for y_i-1 and y_i-2 so the I-frame and P-frame models in residual coding-based framework are unified into one model. This design large simplifies the video compression framework and will benefit future research.\n\n3. In order to speed up the execution, the authors assumed independence between blocks and only overlapping tokens from i-1 and i-2 are used. This seems a bold assumption at the beginning but proved effective.\n\n**Weaknesses**:\n\n1. Most modules follow the most common designs and are not curated for this specific model. Some weaknesses/improvements are considered by the authors in Section 3.2-Independence and Section 6.\n\n2. Other weaknesses and concerns are detailed in the ‘Question’ section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept