MIMT: Masked Image Modeling Transformer for Video Compression
Authors: Jinxi Xiang
Kuan Tian
Jun Zhang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: video compression
masked image modeling
transformer
entropy model

Abstract: Deep learning video compression outperforms its hand-craft counterparts with enhanced flexibility and capacity. One key component of the learned video codec is the autoregressive entropy model conditioned on spatial and temporal priors. Operating autoregressive on raster scanning order naively treats the context as unidirectional. This is neither efficient nor optimal, considering that conditional information probably locates at the end of the sequence. We thus introduce an entropy model based on a masked image modeling transformer (MIMT) to learn the spatial-temporal dependencies. Video frames are first encoded into sequences of tokens and then processed with the transformer encoder as priors.   The transformer decoder learns the probability mass functions (PMFs) \emph{conditioned} on the  priors and masked inputs. Then it is capable of selecting optimal decoding orders without a fixed direction.  During training, MIMT aims to predict the PMFs of randomly masked tokens by attending to tokens in all directions. This allows MIMT to capture the temporal dependencies from encoded priors and the spatial dependencies from the unmasked tokens, i.e., decoded tokens. At inference time, the model begins with generating  PMFs of all masked tokens in parallel and then decodes the frame iteratively from the previously-selected decoded tokens (i.e., with high confidence). In addition, we improve the overall performance with more techniques, e.g.,  manifold conditional priors accumulating a long range of information,  shifted window attention to reduce complexity. Extensive experiments demonstrate the proposed MIMT framework equipped with the new transformer entropy model achieves state-of-the-art performance on HEVC, UVG, and MCL-JCV datasets, generally outperforming the VVC in terms of PSNR and SSIM. 

Published as a conference paper at ICLR 2023
MIMT: MASKED IMAGE MODELING TRANSFORMER FOR VIDEO COMPRESSION
Jinxi Xiang', Kuan Tian", Jun Zhang', Xiao Han, Wei Yang
Tencent AI Lab, Shenzhen
{ [jinxixang) kuantian, june jzhang, haroldhan, eeeey
ABSTRACT
Deep learning video compression outperforms hand-craft counterparts with en- hanced flexibility and capacity. One key component of the learned video codec is the autoregressive entropy model conditioned on spatial and temporal priors. Operating autoregressive raster scanning order naively treats the context unidirectional. This is neither efficient nor optimal considering that conditional information probably locates at the end of the sequence. We thus introduce entropy model based on a masked image modeling transformer (MIMT) to learn the spatial-temporal dependencies. Video frames are first encoded into sequences of tokens and then processed with the transformer encoder as priors. The trans- former decoder learns the probability mass functions (PMFs) conditioned on the priors and masked inputs, and then it is capable of selecting optimal decoding or- ders without a fixed direction. During training, MIMT aims to predict the PMFs of randomly masked tokens by attending to tokens in all directions. This allows MIMT to capture the temporal dependencies from encoded priors and the spatial dependencies from the unmasked tokens, i.e., decoded tokens. At inference time, the model begins with generating PMFs of masked tokens in parallel and then decodes the frame iteratively from the revioousl-sseecteed decoded tokens (i.e., with high confidence). In addition, we improve the overall performance with more techniques, e.g., manifold conditional priors accumulating long range of infor- mation, shifted window attention to reduce complexity. Extensive experiments demonstrate the proposed MIMT framework equipped with the new transformer entropy model achieves state-of-the-arr performance on HEVC, UVG, and MCL- JCV datasets, generally outperforming the VVC in terms PSNR and SSIM.
INTRODUCTION
Videos continue to grow exponentially as demand for various video applications increases on so- cial media platforms and mobile devices. Traditional video compression codecs, such as HEVC and VVC, are still moving toward more efficient, hardware-friendl and versatile. However, their framework still followed hybrid coding framework that remained unchanged decades ago: spatial- temporal prediction coding plus transformation-based residual coding.
Neural video compression surged to outperform handcraft codecs by optimizing the rate-distortion loss in an end-to-end manner. One line of earlier work replaces traditional coding modules, in- cluding motion estimation, optical-flow-bee networks. Recently, residual coding has been proved to be suboptimal compared with context cod- ing. Moreover, a pixel in frame D; is related to all pixels in the previously decoded frames ø=t and pixels already decoded at ø;. Due the huge space, it is impossible for traditional video codecs to explore the correlation between all rules using handcrafted rules explicitly.
warping, and residual coding modules with neural
Using the entropy model to exploit the spatial-temporal dependencies from the current and past decoded frames can vastly reduce data redundancies. The transformer is rising for computer vi- sion tasks, including low-level image analysis. Inspired by the tranlaiotranslatioon model, VCT
"qual contribution correspoondng author

---

Video Compression. Lu et al. (2119) developed the DVC model with all modules in the traditional hybrid video codec replaced by the network. DVC-Pro is proposed with a more advanced entropy sioccallffow-bbsees Published as a conference paper at ICLR 2023
(Mentzer et al. 2022) for the first time uses transformer the conditional entropy model predict the probability mass function (PMF) from the previous frames. VCT uses the estimated probability to losslessly compress the quantized latent feature map D. without direct warping residual coding modules. The better the transformer predicts the PMFs, the fewer bits are required for the video frames.
For VCT, the transformer decoder is an autoregressive model which regards video frames naively as sequences of tokens and decodes the current frame yr sequentially in raster scanning order (i.e.. token-by-token). We find this strategy neither optimal nor efficient, and thus, we propose a masked image modeling transformer (MIMT) using bidirectional attention. During training. MIMT aims to optimize a proxy task similar to the mask prediction in BERT (Devlin et al., (0118) and BEIT (Bao et al. 2 to predict the PMFs of masked tokens. At inference, MIMT adopts a novel non- sequential autoregressive decoding method to predict the image in few steps. Each step keeps the most confident (smallest entropy) token for the next iteration.
Our contributions are summarized follows: (1) We design entropy model based on bi- directional transformer MIMT to compress the spatial-temporal redundancy in video frames. MIMT is trained on masked image modeling tasks. It can capture temporal information from past frames and spatial information from the decoded tokens at inference time. (2) More techniques are in- troduced to make video compression model versatile. We employ manifold priors, including the recurrent latent prior, to accumulate an extended range of decoded frames. To further reduce introduce alternating transformer layers with non-overlapping shifted win- dow attention. (3) The proposed MIMT achieves state-of-the-art compression results with all these improvements. It generally outperforms the last H.266 (VTM) in terms of PSNR and SSIM. The bitrate saving over H.266 (VTM) is 29.6% on the UVG dataset in terms of PSNR.
MIMT complexity,
2 RELATED WORK
model and deeper network (Lu et al., 2020). Agustsson et al.) (2020) extended
estimation to a 3D transformation by adding scale dimension. Hu et al. (2020) considered rate- distortion optimization when encoding motion vectors. Li al. (2020), single reference frame is extended to multiple reference frames. Yang et al. (2020) proposed a residual encoder and decoder based on RNN exploit accumulated temporal information.
Deviated from the residual coding, DCVC (Li et al. 2021) employed contextual coding to compen- sate for the shortness of the residual coding scheme. Mentzer et al. (2022) proposed simplifying the "hand-craft" video compression network of explicit motion estimation, warp, and residual coding with a transformer- based temporal model. Contemporary work from Li al. (2022) uses multiple modules, e.g., learnable quantization and parallel entropy model, improve significantly the com- pression performance, which surpasses the latest VVC codec. ACphaVC (Shi et al. (022)) introduced several techniques, e.g., conditional I-frame and fixel-to-feature motion prediction, effectively improve the rite-distortion performance.
Masked Image Modeling, Masked language modeling, first proposed in BERT (Devlin et al. 2018), has revolutionized the field of natural language processing, significantly when scaling to large datasets and huge models (Brown et al. (2020). The success in NLP has also been replicated in vision tasks by masking patches of pixels (He et al. 2022) or masking tokens generated by pretrained dVAE (Bao al. 2022]] Xie al.) 2022). Recently, these works have also been extended other domains to learn good representations for action recognition (Tong et al. 20022 Feichtenhofer et al.
2022 video prediction (Gupta et al. 2022), and image generation (Chang et al. 2022] Wu al. 2022
3 METHOD
As shown in Fig. 1 we encade sequence of (RGB) video frames {=t) into latent tokens {yr) using a CNN-based image encoder. Next, we get a temporal sequence ....1..... & from the decoded frames buffer. We decoded sequence compress y, with the transformer
2

---

Published as a conference paper at ICLR 2023
MIMT Entingy Modkl
Yr-ts 1--
Component
Symbol
decosical batant
y+
Impor
3;
Input-rune frune
(R, 02)
Reconstraction
x;
Image Encesler
AD
Image Decoder
Image Encoder ((x))
y:
Laseuts
y:
Quant. Latent
9%
Mp-2
Image D(9))
Dattetion loos
92-2
ConLLSTM
Hyper-latents
Dacoded
Side Information
Seenng Comna
At-1
Entropy Purass.
(Ms, 0z)
Moli-cak
Hidden State
Saree Nit
(Ar. 0:)
Fe-1
Entropy Model
Optical
Tx
Masked Latent
y'
Figure 1: The core the proposed video compression framework a MIMT entropy model used to fully exploit the spatial-temporal correlation among video frames. The estimated parameters generated from the side information prior st. recurrent prior 1 1. and the last decoded latent -.-1-
entropy model for non-sequential autoregressive. At the receiver side, recover ÿe and recon- struct frames { with CNN-based image decoder by feeding with the quantized latent tokens
(
3.1 PRELIMINARY
Given the true PMF (y.) and the estimated PMF (3t),, arithmetic coding (AC) (Langdon
1984 to transmit {yr ) with expected bit-rate H (p, q) which is expressed as the cross entropy:
(pp 9) Eyypp loga g (ye c)],
with e as the conditional information of yı. e.g.. hyper-prior, spatial autoregressive context.
The expected bit-rate be reduced if Y/ has higher certainty. We encode more frequently occurring values with fewer bits, and hence improve the efficiency. For this purpose, video coding exploits temporal relationship from previous frames {3:-1,.... 313 reduce the estimated bit-rate:
H (p. q) = Eyg-pp [- logg (yr 1t y1, c)].
Our main idea is to parameterize q (y), y:-1...., yı c) as a conditional distribution using a MIMT to minimize the cross-entropy H (p.q). q).
3.2 CNN-BASED IMAGE ENCODER/DECODEE
Video is transformed with a CNN-based image encoder to reduce dimensionality. Each frame æ; E R ff Xx3 is represented
grid of tokens y: E dxxwxde The image encoder downsamples the raw image spatially and increases the channel dimension, yielding (h, w. de) dimensional tokens where (h, w) are 16x smaller than (H, W'. We reconstruct image ã; with decoder D(Gt).
One straightforward way is to train E, D using standard neural image compression. But this in- dependent transformation without any temporal hints would lay heavy burdens on the conditional coding transformer, considering that the previously decoded frame -;- loses rich information as it only contain 3 channels. Thus, is also optimal learn temporal contexts merely from é; Following the contextual decooer (Sheng et al., 2021 Li et al.) 2022), we also build the image encoder and decoder using motion v, with little bit-stream overhead:
y: E(x, Senneed (ύ.-1 0.)), e. D(D contex(( 0.))
where Vi estimated from optical flow network fav (I,, x1-1), and feontexl (U--1, i.)) generates multi-scale warped feature map aligned in the i-frame space.
3.3 MASKED IMAGE MODELING TRANSFORMER ENTROPY MODEL
Compressing the current frame can be formulated as sequential autoregressive. Let ye
[ i=1 denote the latent tokens obtained by feeding the image æ; the encoder, where N is the
3

---

Published as a conference paper at ICLR 2023
Hyper-prior $
AC
P2
A4
Last frame 91-1
potor Tt-1
Convolution Parameter Network
Swin Trunsformer
Encooder
K.
Swin Transformer Decoder
embalding
y
M
Clttten
M
M
embbeldingg
Figure 2: MIMT overview. During training, some of the tokens are masked, and the task is to predict the PMFs of masked tokens. At inference, the model begins with predicting all masked tokens and keeping the most certain ones, which are put back in sequence for the next prediction. We do this iteratively until all tokens are decoded.
length the flattened token matrix. Given already transmitted tokens y ن the transformer learns to predict the probability distribution of next index, i.e., ((vi). This allows to directly minimize:
N
(AR(Y:) = Eyyrp
log2 (vf y1-1.-... U1
(4)
As illustrated in Fig. sequential autoregressive model can only make predictions based on the observed pixels (left upper part of the target pixel) due to the inductive bias caused by the strict adherence to the unidirectional scanning order. The fixed scanning order may achieve suboptimal performance, For example, if the informative condition is located at the end of the autoregressive sequence, it is difficult for the model to take full advantage of such relevant information.
We propose a bi-directional non-sequential transformer entropy model. The training task of the transformer is masked image modeling. The strategy is mask-then-predie which straightfor- ward: during training, we randomly mask some proportion image tokens and feed the cor- rupted sequence the transformer. Fig. shows the overview of our transformer entropy model. M [m,])"" is the corresponding mask. We randomly sample a subset (0% 100%) of tokens and replace them with a special learnable [M] token. The token y is replaced with [M] if m, 1, otherwise, if m, 0, y will be left intact.
Denote TM the masked sequence by applying M ye- The optimization objective is minimize the cross entropy of the predicted PMFs and the true distribution, i.c., the expected bit rate:
Rmusk (y:) Enyc- logz a (y: yr 9r-1, ... y1 y.
We aim to fully exploit the temporal correlations from video frames ( feed manifold inputs to the MIM transformer and let the model learn complementary information from the rich temporal/spatial inputs which have different characteristics. Side information is decoded from &, which a hierarchical spatial estimation of y:; Φ:-1 is most correlated latent with y: in the past; Tz-1 is from a recurrent network with convolution layers and ConvLSTM cells. Due the recurrent structure, r is generated based on all previous latents (Ū:-2..... 01).
yı ). In concrete, we
After transformer encoding, we feed the rich encoded priors into a multi-layer bidirectional trans- former to predict the PMFs of masked tokens, where the cross-entropy between the ground-truth distribution and the predicted distribution is
R (yr)
Ey
log, q (y. s:, 1.-1, T, - y
(6)
Notice the key difference to autoregressive modeling: the conditional dependency in MIMT is bidi- rectional that allows PMF prediction utilize richer contexts in the current frame. For decoding.
4

---

Published as a conference paper at ICLR 2023
2
decoded
6
be decoded
unidirectional context
decoded
Mailee
bi-directional context
Left: Compared to the autoregressive model with a unidirectional context, MIM trans- former benefits from the bi-directional context information. Right: With pre-trained MIM trans- former, we can decode yr in few steps. Each step recovers portion of tokens based on the decoded ones. Only the most certain tokens with the largest entropy are kept for the next iteration.
Figure 3:
PMFs of tokens are iteratively predicted with the MIM transformer following decoded ratio sched- uler at each step as shown below.
3.4 ITERATIVE DECODING SCHEDULER
At receiver side, first decode the optical flow 6, and the hyper-prior 2, before decoding the latent dt. Since we do not make any assump- tions about the distribution of Ú, and 2, non- parametric, fully factorized entropy model is used for decoding (Ballé et al. 2018).
Algorithm 1: MIMT Iterative Decoding
Input: 9:-1, Tt-1, 7(), n, Bs // Bs is the bitstream. a step. Output: â; // reconstructed image /s Initialization
With the pre- praited MIMT, we can decode the current frame Y/: in one step assuming all to- kens in yM are [M] token. But the PMFs ob- tained from all [M] tokens are not optimal esti- mations because the spatial context information in the current frame totally ignored. Thus it would introduce great bit-rate cost for arith- metic coding. Instead, we decompose the one- step decoding into several iterations in Fig.
Decode 6, from Bs using arithmetic decoder
Decode &, from using arithmetic decoder: Get 8 by pass 2, to hyper-prior decoder; i t 0,M +1;
/. iterative decoding while i E n do
(er, δ.) MIMT TT-1, D
i s+i+1;
Select Ly(1) -ii 1).7 tokens by 77:
Let 7(1) = sin(i #), where {o. 1o........... 11. be decoded ratio scheduler that determines the number of tokens to be recovered at each step. 8 by default. The scheduler is monotonically increasing and (()) + 0, 7(1 + 1. At O. we start with all tokens in Mr are [M]. Only most certain 17(1)- N7 tokens with the smallest entropy
E
Decode the selected tokens from Bs with (f4, ōt)::
Replace the masked token [M] y with the decoded tokens;
T ((1:-1,/1-1)) // requrrent net 9: y.M.
11 = 0
L. +- D(gr Scanexx Voomex((:-11 (0r-1,
are kept for the next iteration. The entropy of token is accumulated along the channel:
de
H H = log II p(u)), p(m)2) ot (4 + + & (14) )
1=1
where de is dimension of tokens; (-) is the cumulative function Gaussian distribution N (1)) a. 2.22)
We iteratively replenish y,M with the decoded tokens until all tokens being decoded. The scheduler consistent with the intuition that video decoding follows flow of information from coarse to fine. In the beginning. the model decodes the "sketch" of the image with small entropy. The "sketch" is
5

---

Published as a conference paper at ICLR 2023
VSW--
W
N7
windowed attention
Transformer encoder
Transformer decoder
shifted windowed attention
Figure 4: Transformer encoder and decoder with chffted-windon attention. W-MSA: windowed multi-head self-attention. SW-MSA: shifted windowed multi-head self-attention. W-MCA: win- dowed multi-hean cross attention. SW-MCA: shifted windowed multi-head cross attention.
easy to predict from frame redundancies. Gradually, with more decoded context, the model takes small steps tweak the details considering the slow slop of sine around 1.
3.5 IMPLEMENTATION DETAILS
Shifted window attention. The global self-attention of the transformer is computationally unaf- fordable for video compression with long sequences. We use shifted window attention Liu et al. 202T) in both transformer encoder and decoder to alleviate this problem, as visualized in Fig. 4 The transformer encoder makes use of interchange of shifted windows and thus, the computa- tional cost is greatly reduced since self-attention is considered within local windows. In contrast to SW-MSA which uses the same input as the key, query and value, SW-MCA uses the encoded priors of (st 9t-1, r(-1) as the key and value, while using y. as the query.
Optimization loss. Rate-distortion loss is optimized throughout the training process:
CRD
R(ge)
R(2,)
R(d:)
+X d(xx tt):
(8)
rate (latents)
rate (hypor-latents)
(optical flow)
distortion
where d(-) represents the mean square rate-distortion trade-off. We train four models with different values {256, 512, 1024, 2048). By default, we train models with MSE loss. When using the MS-SSIM metric, the model is fine-tuned with the MS-SSIM loss. More training details are described in the Appendix.
MS-SSIM; À is hyperparameter used control the
EXPERIMENTS
Dataset. We Vimeo-90k (Xue et al. 2019) for training. The videos randomly cropped into 256 256 patches. The test videos include HEVC Class B, UVG (Mercat et al. 2020). and MCL-JCV (Wang al. 2016) datasets. All three datasets have resolution of 1920 1080.
Baselines. We compare MIMT with widely acknowledged basclines, e.g., x265, HM, VTM codecs. x265 is implemented with the commercial FFmpeg. whereas HM and VTM standard versions that can achieve much better performance but are extremely slow. We further obtains reported results
from the following learned codecs: DVCPro (Lu et al., 2020). FVC (Hu et al. 2021). DCVC LLi
et al. 2021. C2F (Hu et al.) 2022), VCT (Mentzer et al. 2022) and DMC (Li et al. 2022).
Metric. The common PSNR and MS SS-M evaluated for all models (ang et al. 2004 in RGB.
we set the GoP size as 32 for all datasets and use learned model (Cheng et al. 2020) for frame compression. We center crop 1080p image into 1790 1024 to make it divisible by 256.

---

Published as a conference paper at ICLR 2023
UVG dataset
MCL:JCV dataset
HEVC-B dataset
36
37
DMC
37
DMC
MIMT
36
36
HM
33
DCVC
DCVC
HM
DVCPTo
DVCPto
X265
33
CFF
C2F
DVCPro
33
32
VCT
VCT
30
FVC
0.000
0.025 0,050 0.075 0.100 0.125 0.150
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
0.25
UVG dalaset
MCL-JCV dataset
HEVC 8 dalaset
0.99
0.990
0.985
0.98
0.985
0.980
DMC
0.480
DMC
0.975
MIMT
0.97
VTM
0.975
VTM
0.370
VTM
DCVC
DEVC
HIM
0.96
0.970
DVCPro
DVCPro
X265
C2F
0.965
C2F
DVCPro
VCT
VCT
FVC
0.0
0.1
0,2
0.3
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.0
0.1
0.2
0.3
0.4
Figure RD performance on UVG, MCL-JCV, and HEVC-B dataset.
DVCPro
FVC
DCVC
HM
C2F
DMC
MIMT
UVG
+227.0%
+008.6%
+95.8
+40.5%
+16.0%
-18,2%
-29.6%
MCL-JCV
+180.8%
+71.0%
+66.1%
+45.4%
+19.6%
-6.4%
-1.0%
HEVC-B
+209.8%
+108.3%
+49.8%
+40.4%
+14.2%
55.%
-25.5%
Average
+205.8%
+95.9%
+70.5%
+42.1%
+ +16.6%
-9.9%
-22.7%
Table 1: BD-rate calculated by PSNR with respect to the anchor VTM.
RESULTS
5.1 CATE-DISTORTION PERFORMANCE
plot the rate-distortion of method and the baseline methods introduced in Section We see that our MIMT outperforms most of these baselines across different datasets in terms of PSNR and SSIM. DMC is the best-performed competing method with hybrid spatial-temporal entropy model. For MIMT, higher quality can be obtained with less bit rate cost under the same quality. This verifies the effectiveness of entropy model in exploiting the temporal redundancies among video frames as well as spatial correlations among intra frarr--frmee tokens.
In Fig.
As shown in Table and we make quantitative metric with BD-rate Bjontegaard 2001 computed from PSP and SSIM-BPP, respectively. The best traditional codec VTM is used as the anchor. From Table 1, we can find that DMC Li al. (2022) the only baseline method that surpasses VTM with 9.9% bite-rate saving. Our MIMT improves this performance further and achieves an average of 22.7% bitrate saving over VTM on all datasets. From the perspective of SSIM, the improvement of MIMT is even larger, which is 56.3% bit rate saving over VTM. We notice that C2F and DMC also outperform VTM in terms MS-SSIM.
5.2 ABLATION STUDIES
We conduct series of ablation studies demonstrate the effectiveness of different components.
Entropy coding. The MIMT uses a dynamic strategy to determine the encoding/decoding order for auto-regressive. Other strategies would be raster scan order autoregressive entropy model like

---

Published as a conference paper at ICLR 2023
DVCPro
FVC
DCVC
HM
C2F
DMC
MIMT
UVG
+68.1%
+10.1%
+33.6%
+36.9%
20.1%
33.1%
40.0%
MCL-JCV
+37.8%
+15.0%
+4.7%
+43.7%
-9.1%
-6.8%
64.7%
HEVC-B
+61.7%
+60.9%
+31.0%
+36.7%
+12.7%
-48.1%
-64.2%
Average
+55.8%
+28.6%
+23.1%
+39.1%
-5.5%
-43.3%
56.3%
Table 2: BD-rate calculated by SSIM with respect to the anchor VTM.
Modules
Ablation Option
UVG
MCL.JCV
HEVC-B
Average
MIMT MIMT
auto-regressive (full)
+5.2%
+6.9%
+7.1%
+6.4%
auto-regressive (block)
+ +16.6%
+19.3%
+22.8%
+19.5%
MIMT
checkerboard
+15.1%
+17.7%
+19.2%
+17.3%
contextual
non-contextual
+13.0%
+18.4%
+19.5%
+16.9%
encoder/decoder
encoder/decoder
Table 3: Ablations the entropy model and the contextual encoder/decoder. The first column is the original design and the second column is the optional modules. The last four columns show the BD rate increase the anchor of the complete MIMT model.
VCT (Mentzer et al. 2022), or parallel-friendly checkerboard entropy model (He et al. 2021). For fair comparisons, both auto-regressive and checkerboard models use the same transformer encoder- decoder network, but they train and predict differently. For the auto-regressive model, we train the transformer with self-attention mask to ensure causality similar to Vaswani et al. (2017). and then predict each token one by in raster scan order. We make two variants of auto-regressive: the full variant means operating on all 16 x tokens, and the block variant means splitting the tokens into 4 non-overlapping blocks similar VCT. No windowed attention mask is applied for autoregressive (block). As for the checkerboard, we predict the first half tokens with all input tokens masked, and then use the decoded tokens predict the rest.
In Table 3 we calculate the BD-rate increase after ablations. In the first row, the MIMT entropy performs better than the full auto-regressive with +6.4% bitrate savings. The time-cost of auto- regressive (full) decoding is prohibitive for practical usage. As an alternative, outo-regressive (block) partitions the tokens into sub- blockloks and decodes them parallel. Although it can speed up greatly, it builds on the assumption that all blocks are independent, which is not necessarily true as concluded by VCT (Mentzer al. 2022). this way, the performance of autoregressive (block) degrades obviously with + 19.5% bitrate increase. The parallel checkerboard entropy model is slightly better but still has +7.3% increase. This is intuitively reasonable because the checkerboard model could be considered simplified case of MIMT with fixed encoding/decoding order.
We follow good practice to use flow-based transformation coding (Sheng et al., 2021 Li al. 2021 2022). In the last row of Table 3 conduct ablation study on the image encoder by removing the temporal context. The results show the contextual encoder provides significant bitrate savings of +16.9% over the non-contextual encoder with independent image compression.
Decoding scheduler. The masked scheduler non-trivial design for encoding/decoding. The default scheduler is s sine function. We further explore two alternatives, i.e., linear, and exponential functions. The sine is concave, whereas the exponential is convex. A sine scheduler is in line with such intu- itions: due to the temporal redundancy, the cur- rent frame has high similarities with previous frames. In the beginning, we can take a large step with high certainty. In the last few steps, we have to cope with pixels of large entropy where we should take small steps make more precise estimations. We adjust the number of steps
Figure 6: Decoding scheduler.
from to 12. As shown in Fig. 6 the results demonstrate that the sine performs consistently better than the others. The 1 ineer function provides decent results but is inferior the s ne.
8

---

Published as a conference paper at ICLR 2023
The exponential performs poorly. We provided further visualization in the Appendix, Section Increasing the number of steps can improve the performance, but for steps larger than 8, the gains marginal and it will introduce computational overhead for training and inference.
hyper-prior
last frame
recurrent prior
UVG
MCL-JCV
HEVC-B
Average
St
D1
Tt-1
+8.6%
+11.3%
+13.5%
+11.1%
X
+9.2%
+14.9%
+15.6%
+13.2%
+11.7%
+17.3%
+16.6%
+15.2%
0%
0%
0%
0%
Table 4:
Ablations on the manifold inputs of MIMT, The figure in the table denotes the BDDrate increase over the full model used as the anchor.
Manifold inputs. MIMT takes three different kinds of inputs: side information 8, from hyper prior, temporal prior from the last decoded frame Út-1, and temporal recurrent prior Ti-1- We conduct ablation studies to verify these priors for entropy modeling. As shown in Table 4 hyper- prior 8 providing hierarchical information brings the most significant BD-rate improvements of +1..%. D is the most temporally correlated frame in the past and it achieves +3.2%% gains. By accumulating long range of temporal information, the recurrent prior Tt-1 effectively compensate for the overall performance with + +11.1% BD-rate.
5.3 DECODING EFFICIENCY
[oplee]compact the model complexity in the number of parameters, MACs (multiply-accummllae operations), encoding time, and decoding time with learned codecs. The test video frames are 1080p and we use the publicly available source code of DVC, SSF, DCVC, and DMC. We deploy all models on a server with an NVIDIA P40 GPU. The time consumption includes bit-stream writing and reading with arithmetic coding. Note that all models are not optimized for coding efficiency.
In Table 5 we find that the decoding speed of DVC is moderate even without sequential/recureet network. DCVC is high of high complexity for encoding/decodigg because employs a sequential autoregressive context model. DMC splits the latents along the mhannel-wise and spatial-wise direc- tions to accelerate the autoregressive. The proposed MIMT equipped with shifted window attention and non-sequential decoding strategy is comparable with DMC, in terms of model complexity and encoding and decoding time. For the other three variants MIMT, the number parameters the same but they are trained differently, thus leading to varying MACs requirements. Not surprisingly, the parallel- parallel-friendl friendly checkerboard is the most efficient consuming only 798 ms for decoding. The full autoregressive model is prohibitive slow at decoding, and the block auto-regressive reduces the time-cost remarkably to 969 ms by partitioning the latent into small blocks for parallel decoding.
CONCLUSIONS
In this work, we propose masked image modeling transformer for deep video compression. Fol- lowing the proxy task in pretrained language/image model, the transformer is trained to fully exploit the temporal correlation among frames and spatial tokens in few autoregressive steps. The MIMT differs from the conventional autoregressive (raster scanning order) in two ways: more flexible bidi- rectional attention and more efficient parallel decoding. Visualization and qualitative metric on standard datasets demonstrate our method outperforms VTM in terms of both PSNR and SSIM. Our work provides new insights into the use or extension of the transformer in video compression.
REFERENCES
Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8503- 8512, 2020.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with scale hyperprior. arXiv preprint TrXivv/8002..0 01120 2018.

---

Published as a conference paper at ICLR 2023
Params
MACs
Encoding time
Decoding time
DVC (Lu al. 2019
4.727 MB
1.721
3,522 ms
3,296 ms
SSF (Agusisson et al. 2020
22.991 MB
1.424
572 ms
392 ms
DCVC
6.526 MB
2.411
17,856
97,642
DMC (Li et al. 2022
17.520 MB
3.261 T
1,297 ms
912 ms
MIMT (auto-regressive, full)
38.471 MB
15.407
1,237 ms
117,476 ms
MIMT (auto-regressive, block)
38.471 MB
T
1,007 ms
969 ms
MIMT (checkerboard)
38.471 MB
3.365
964 ms
798 ms
MIMT (proposed)
38.471 MB
T
1,901 ms
1,372 ms
Table 5: Model complexity. All models are tested with 1080p video on an NVIDIA P40 GPU.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 2021.
Gisle Bjontegaard. Calculation of average psnr differences between rd-curves. VCEG-M33, 2001.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-1133 11325, 2022.
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7939-7948, 2020.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders spa- tiotemporal learners. arXiv preprint 2rXiv:2205.09) 13, 2022.
Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Marthn and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206. 1894, 2022.
Dailan He, Yaoyan Zheng. Baochen Sun, Yan Wang, and Hongwei Qin. Checkerboard context model for efficient learned image compression. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14766-14775, 2021.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Zhihao Hu, Zhenghao Chen, Dong Xu, Guo Lu, Wanli Ouyang, and Shuhang Gu. Improving deep video compression by dasolution-adaptii flow coding. In European Conference on Computer Vision, pp. 193-209. Springer, 2020.
Zhihao Hu, Guo Lu, and Dong Xu. Fve: A new framework towards deep video compression in feature space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1502-1511, 2021.
Zhihao Hu, Guo Lu, Jinyang Guo, Shan Liu, Wei Jiang, and Dong Xu. Coarse-to-fine deep video coding with esperprior-guided mode prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5921-5930, 2022.
Glen Langdon. An introduction to arithmetic coding. IBM Journal of Research and Development, 28(2):135-149, 1984.
10

---

Published as a conference paper at ICLR 2023
Jiahao Li, Bin Li, and Yan Lu. Deep contextual video compression. Advances in Neural Information Processing Systems, 11118114-18111 2021.
Jiahao Li, Bin Li, and Yan Lu. Hybrid mentialltemporll entropy modelling for neural video com- pression. arXiv preprint arXh::200.005894, 2022.
Jianping Lin, Dong Liu, Houqiang Li, and Feng Wu. M-Ive: Multiple frames prediction for learned video compression. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3546-3554, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision pp. 10012-10022, 2021.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dve: An end-to-end deep video compression framework. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 11006-11015, 2019.
Guo Lu, Xiaoyun Zhang, Wanli Ouyang, Li Chen, Zhiyong Gao, and Dong Xu. An end-to-end learning framework for video compression. IEEE transactions on pattern analysis and machine intelligence, 43(10):3292-3300 2020.
Siwei Ma. Xinfeng Zhang, Chuanmin Jia. Zhenghui Zhao, Shiqi Wang. and Shanshe Wang. Image and video compression with neural networks: A review. IEEE Transactions on Circuits and Systems for Video Technology, 3066):1683-1695 2019.
Fabian Mentzer, George Toderici, David Minnen, Sung-Jin Hwang, Sergi Caelles, Mario Lucic, and Eirikur Agustsson. Vet: video compression transformer. arXiv preprint 2rXX::2206.07307, 2022.
Alexandre Mercat, Marko Viitanen, and Jarno Vanne. Uvg dataset: 50/120fps 4k sequences for video codec analysis and development. In Proceedings of the 11th ACM Multimedia Systems Conference, pp. 297-302, 2020.
Xihua Sheng, Jiahao Li, Bin Li, Li, Dong Liu, and Yan Lu. Temporal context mining for learned video compression. arXiv preprint 1rXiv:2111.1333 2021.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Din Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: machine learning approach for precipitation nowcasting. Ad- vances in neural information processing systems, 28, 2015.
Yibo Shi, Yunying Ge, Jing Wang, and Jue Mao, Alphave: High- perfforrrfomm and efficient learned video compression. arXiv preprint arXiv:2207. 14678, 2022.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training, arXiv preprint arXiv: 2022.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv,
2017.
Haiqiang Wang, Weihao Gan, Sudeng Hu, Joe Yuchich Lin, Lina Jin, Longguang Song, Ping Wang, loannis Katsavounidis, Anne Aaron, and C-C Jay Kuo. Mcl-jev: jnd-based h. 264/ave video quality assessment dataset. 2016 IEEE international conference on image processing (ICIP), pp. 15139 IEEE, 2016.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
Chenfei Wu, Jian Liang, Xiaowei Hu, Zhe Gan, Jianfeng Wang, Lijuan Wang, Zicheng Liu, Yuejian Fang, and Nan Duan. Nuwa-infinity: Autoregressive over autoregressive generation for infinite visual synthesis. arXiv preprint arXiv:2207.098 2022.

---

Published as a conference paper at ICLR 2023
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653-96633 2022.
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision, 127(8):1 125, 2019.
Ren Yang, Fabian Mentzer, Lue Van Gool, and Radu Timofte. Learning for video compression with hierarchical quality and recurrent enhancement. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 6628-6637, 2020.
A NETWORK
Image encoder E, decoder D. The image encoder and decoder are not the main contributions of our work. We develop our model heavily dependent on the contextual encoding backbone from (Sheng 2021 Li et al. 2022). We use 4 strided convolutional layers for the encoder, resulting in total factor of 16x downsampling. For the decoder, we use transposed convolutions and add residual blocks at low resolutions. Another critical module is the context network, called the temporal context mining (TCM) module in (Sheng et al. 2021). It generates 1x, 2x, and 4x downsampled context by warping the feature the last frame and the predicted optical flow. These contexts are concatenated with the image encoder/decoder at the corresponding scale for temporal reference.
et al.
Transformer Encoder, Decoder. We map the input frame of (256, 256, 3) (16, 16, 192) latent with image encoder. We flatten the 2D grid latent in to sequence (1, 16 x 16, 192). Thus, the dimen- sion of input Ú:-1 is (1, 256, 192). The output of hyper-prior decoder &; is flatten into (1, 256, 96), and the output of ConvI is (1, 256, 192). We project priors into dc = 768 dimensional with linear projection layer before feeding to the transformer. These priors are separately processed with a transformer encoder and then concatenated together for temporal mixture with joint transformer encoder. W-MS A and SW-MSA blocks have another input from the output of the joint encoder. As shown in Table 6 show the architecture of the transformer encoder and decoder. We use cross attention, ie., W-MCA and SW-MCA layers, to encourage information mixing of spatial-tempont context and the current frame. The key and value in the cross-altention block are from the output of the joint encoder, where the continuing output of the decoder is the query.
The basic building layer of the transformer is the shifted window attention. The decoder alternate between the W-MSA layer and SW-MSA layer using the output of the joint encoder as the key and value. The window size of W-MSA and SW-MSA 4. The number of attention heads is 8 for every attention layer. We use two layers of MLP with input size, hidden size, and output size all 768.
Hyper-prior Encoder, Decoder. The hyper-prior network compresses and stores a bitstream used as the hierarchical prior (Ballé et al. 2018). The encoder downsamples the latent y:. At the receiver side, the decoder recovers by upsampling. as shown in Table
ConYLSTM. We use a ConvLSTM module (Shi et al. 2015) to aggregate temporal priors of all previous frames before y:-1. We employ convolution with one-step LSTM, as described in Table 7
Multi-frame training. We can apply multi-framm (up to 7 frames) and patch-size (256 x 256) for training. Using long video sequences for training can alleviate the accumulated errors propagating throughout inter-frame coding. In the first stage, we use two consecutive frames, including one frame and one P frame, to train our model for M steps using the hyper-prior entropy model without MIMT. This stage targets a good image coder-decoder that can faithfully reconstruct the image with less emphasis the bit rate cost. Then we add the MIMT entropy model make better bit-rate estimation and use two consecutive frames to minimize the rate-distortion loss for M steps. Finally, extend the length of the training video sequence to 7 frames for 300 K steps. The learning rate is set as 5e-5. We set the batch size as 8, using the Adam optimizer on a single V100 GPU.
12

---

Published as a conference paper at ICLR 2023
Encoder for 81 out: (1, 256, 768)
Encoder for 9t-1
Encoder for Pt-1
Joint Encoder out: (1, X 256, 768)
Decoder out: (1, 256, 2 x 192)
out: (1, 256, 768)
out: (1, 256, 768)
MLP
MLP
MLP
MLP
Conv(768, 2 x 192, 1, 0)
Layer Norm SW-MSA
Layer Norm SW-MSA
Layer Norm SW-MSA
Layer Norm SW-MSA
MLP
Layer Norm + SW-MCA
Layer Norm
Layer Norm
Layer Norm
Layer Norm
MLP
MLP
MLP
MLP
Layer Norm SW-MSA
W-MSA
W-MSA
W-MSA
W-MSA
Layer Norm Linear(192, 768) in: (1, 256, 192)
Layer Norm Linear(192 192, 768) in: (1, 256, 192)
Layer Norm L192ar(192, 768) in: (1, 256, 192)
Layer Norm
Layer Norm
MLP
in: (1,3 x 256, 768)
Layer Norm
+ W-MCA
Layer Norm W-MSA Layer Norm Linean( 192, 768) in: (1, 256, 192)
Table 6: Transformer architecture. Each column is transformer model, and each row corresponds to layer in the module. Note that omit the skip connection in the table. We only display one attention block in the table. A learnable positional embedding is added at the input of each module. In practice, encoder $t. Dt-1, and r; have 2 blocks; joint encoder has blocks; decoder has 4 blocks.
Hyper-prior Encoder
Hyper-prior Decoder
ConvLSTM
ResBlock
out: 2; (1, (1,96,4 4)
out: 8, (1, 192, 16, 16)
out: P/-1 (1, 256, 192)
out: (1, 192, 16, 16)
Conv(96, 96, 5, 2, 2)
Conv ConnTrrasppooo996 192, 3, 1,
h + LSTM(112, 192, 1)
skip connection
ReLU()
ReLU(
reshape
ReLUO
Conv(96, 96, 2, 2)
ConTTranspose((99 96, 5, 2, ReLU()
ResBlock
Conv(192, 192, 3, 0)
ReLU()
ReLUU
ReLUO Conv(192, 192, 3, 0)
Conv(192, 96, 3, 1,
Con/Transposee 96, 5, 2, in: &, (1, 96, 4, 4)
Conv(192, 192, 3, 1)
in: yı (1, 192, 16, 16)
in: V:-2 2 (1, 192, 16, 16)
in: (1, 192, 16, 16)
Table 7: Hyper-prior encoder/decoder and ConvLSTM architecture. cony(iiuutcchaneee out- put_chape chaachnnsss, kernel size, stride, padding). ConTTranspose(inpu. chhaeeceeesipputcccchelssss output_chamme ker- sizesize, stride, padding, paddut-padding), LSTM(input.size hidden sizee, num layers)
B RECURRENT PRIOR
Exploiting inter-frame redundancy has always been critical consideration for video compression.
Generally, deep learning model can use two consecutive frames {y,, y (Lu et al. 2019 Li
al., 2021). three consecutive frames {3r Yt- Mc-2) (Mentzer et al., 2022). four consecutive
frames yt. y 11.-1.1 91-2. 3 (Hu et al. 2021), or all previous frames using recurrent network (Ma et al. 2019 Yang et al. 2020). We use a one-step ConvLSTM aggregate all previous frames before yt 1- From Table it can be discovered that recurrent prior r bring 11.1% bitrate savings. One alternative simplify the recurrent prior to remove the ConvLSTM and use ÿi-2 instead. An ablation study is conducted in Table g The second row shows that ÿt-2 help to save significant bitrate. But there are still 3.9% bitrate saving improvements. This result implies that ÿ:- 1, D:-2 contains most temporal context need to compress ye. but there is still room for improvement if we exploit a longer range of frames. This observation is consistent with the conclusion from FVC (Hu al. 2021) and RLVC (Yan al. 2020).
C VISUALIZATION OF DECODING
In Fig we visualize the intermediate results of the iterative decoding process. At the k-th step, the transformer entropy model predicts the PMFs of undecoded tokens, and we directly fill the non- decoded tokens with the predicted mean μ4 of PMFs. The quality the intermediate image can intuitively reflect the entropy prediction ability of the MIMT model.
In the first column, zero tokens of y, are decoded. i..., 0 bpp for y. The bit cost on U, and x, is 0.006 and 0.007, respectively. MIMT can predict coarse estimation of the from encoded
13

---

Published as a conference paper at ICLR 2023
Inputs for Entropy Model
UVG
MCL-JCV
HEVC-B
Average
St. Vt-1
+8.6%
+11.3%
+13.5%
-11.1%
St. Y1-1, y1-2
+2.2%
+4.1%
+5.4% 0%
+3.9%
St, D 1, Pt-1
0%
0%
0%
Table 8: Ablations the recurrent prior r--1 of MIMT.
decade step-O
droode stpp-
decado stope5
decede stepoó bpp-0 O5B PSNR-300542
droede step-
0.1-0.013 PSNR-22.199
hpp-2015 PSNR-22.079
hyp- 0.051
PSNR-36. 129
9pp-0.062 PSNR-355244
Figure 7: Illustration of decoding process Jockey in UVG dataset. The first row is the recon- structed images at different steps. The second row shows zoom-in regions in the reconstructed images. The third row plots the pixel-to-pixce to--eell loss with respect to the ground-truth image. Brighter red represents a higher loss. The fourth row shows the decoded tokens position, where at the begin- ning tokens are decoded (blue) and last, all tokens are decoded (yellow).
priors Út-1, 2,, Although the image is blurred without details, we can be aware of the rough background, the main instances of image, etc. The follow-up images show that as we decode more tokens, the model gets more context, and the current frame is refined progressively.
We can find that the model decodes images following an easy-to-hard process. Considering the high temporal redundancy of frame sequences, large portion of pixels, mostly background, is easy to predict. The model starts from the tokens with the smallest entropy (c.g., sky and ground) to get context information. In the beginning, bitrate. Gradually, when it "knows" more about the image, refines the large-entropy tokens (c.g., the fast-moving billboard and horseshoe) which are hard to predict. We should take small step painter who starts with a
take large steps because these tokens consume a little
make more accurate estimations. This is analogous to the behaviors sketch and then progressively refines by filling in tweaking the details. This dynamic autore- gressive strategy differs from the raster scanning order might encounter hard-to-easy dilemma and result in more bit-stream consumption.
D DECODING EFFECIENCY
We obtain the opensource code DVC SSF comparison.
DCVC and DMC for decoding efficiency
14

---

Published as a conference paper at ICLR 2023
E
FULL-SIZED RATE-DISTORTION PLOTS
In the following, we show full-size versions of Fig. for better readability.
UVG dataset
39
38
37
MIMT*
DMC
36
VTM
PSNR
HM
35
DCVC
X265
34
DVCPro
C2F
33
FVC
32
VCT
0.000
0.025 0.050 0.075 0.100 0.125 0.150 Bpp
Figure 8: RD performance on UVG dataset in terms PSNR.
MCL-JCV dataset
39
38
MIMT*
37
DMC
VTM
PSNR
36
HM
DCVC
35
x265
DVCPro
34
C2F
33
FVC
VCT
32
0.05
0.10
0.15
0.20
Bpp
Figure 9; RD performance MCL-JCV dataset terms of PSNR.
15

---

Published as a conference paper at ICLR 2023
HEVC-B dataset
36
35
34
MIMT*
DMC
PSNR
33
VTM
HM
32
DCVC
X265
31
DVCPro
C2F
30
FVC
0.00
0.05
0.10
0.15
0.20
0.25
Bpp
Figure 10: RD performance on HEVC-B dataset in terms of PSNR.
UVG dataset
0.99
0.98
MIMT*
DMC
VTM
SSiM
0.97
HM
DCVC
x265
0.96
DVCPro
C2F
FVC
0.95
VCT
0.0
0.1
0.2
0.3
Bpp
Figure 11: RD performance on UVG dataset in terms of SSIM.
16

---

Published as a conference paper at ICLR 2023
MCL-JCV dataset
0.990
0.985
MIMT*
0.980
DMC
VTM
SSIM
0.975
HM
DCVC
0.970
X265
DVCPro
0.965
C2F
FVC
VCT
0.960
0.0
0.1
0.2
0.3
Bpp
Figure 12: RD performance on MCL-JCV dataset in terms of SSIM.
HEVC-B dataset
0.985
0.980
0.975
M
DMC
SSiM
0.970
VTM
HM
0.965
DCVC
x265
0.960
DVCPro
0.955
C2F
FVC
0.950
0.0
0.1
0.2
0.3
0.4
Bpp
Figure 13; RD performance HEVC-B dataset in terms of SSIM.
17


Reviews:
[{"review_id": "SOFZctXGP3", "reviewer": "Reviewer_Gx3J", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a neural video compression method which leverages masked image modeling for entropy coding. The advantage of masked image modeling in the context of image generation, where it was previously used, is that it provides better quality-complexity trade offs than fully autoregressive modeling. When combined with other techniques such as motion compensation, hyper-prior modeling, and a recurrent prior the proposed method achieves state-of-the-art BD-rate performance compared to recent engineered and neural video compression algorithms, on UVG, MCL-JCV, and HEVC-B, in PSNR and SSIM.", "summary_of_the_review": "The paper proposes to use masked image modeling for entropy coding in the context of neural video compression and presents a corresponding system which outperforms recent state-of-the-art engineered and neural codecs. While this is a strong result, the paper lacks an in-depth evaluation of the masked image modeling component - its main technical contribution.", "strength_and_weaknesses": "*Strengths:*\n\nThe proposed method obtains state-of-the-art results, outperforming several recent engineered and neural codecs across a variety of established benchmark data sets and metrics. Moreover, the proposed method seems to be the first one to use masked image modeling in the context of entropy coding for neural compression. This intuitively makes sense as masked image modeling was recently successfully deployed for likelihood-based generative image modeling (Chang et al. 2022) and where it allows for good quality-speed tradeoffs.\n\n*Weaknesses:*\n\nBesides masked image modeling, the method employs a large set of known modules/tricks (optical flow with multi-scale motion, hyper-prior, recurrent prior, windowed self-attention,...) some of which are ablated and shown to be effective. However, the paper’s main technical novelty - masked image modeling - is not ablated. How does this compare to (potentially structured) autoregressive modeling? By performing autoregression e.g. over spatial blocks or channels, similar rate/compute tradeoffs might be attainable similar to those realized by masked image modeling. Indeed, Table 4 shows that some of the modules/tricks lead to BD rate increases of 10-15%, so they might be to a large part be responsible for the superiority of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PEsN_NVF0c", "reviewer": "Reviewer_gTZB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use masked transformers in the representation space of a compressive autoencoder to do video coding. The masked transformer predicts the distribution of the representation auto-regressively, where more and more tokens are uncovered, similar to MaskGIT (Chang et al 2022). In contrast to other recent transformer work for video compression (VCT, Mentzer et al 2022), the transformer   does not just see see the last two representations (y_t-1, y_t-2), but also a hyper latent z_t, as well as the output of a ConvLSTM. Furthermore, the encoder and decoder transforms are conditioned on motion between adjacent frames, introducing again a predictive coding mode (in contrast to VCT): the frame at time step t depends on all reconstructions since the last I-frame.", "summary_of_the_review": "I am intrigued by the rate distortion performance and using masking is elegant, but for acceptance we need more ablation studies to actually learn something from this paper besides \"many components lead to good performance\".\n\nI am happy to change my rating if we get an ablation on the flow in the Encoder/Decoder (_b_ above), my concern about the ConvLSTM is less important (_a_ above) but that ablation would be nice to have.", "strength_and_weaknesses": "## Strengths\n\n- Great PSNR performance on common benchmark datasets, outperforming VTM in PSNR/RGB.\n- Using Masked transformers makes sense from a compute perspective and is an interesting direction.\n- Paper reports runtime numbers, important for this growing field.\n\n## Weaknesses\n\nAblation Study and Clarity of Results. While the results are very convincing, I left the paper wondering what we actually learned. Which of the introduced components is how cruicial? In contrast to VCT, we have a) hyperprior, b) recurrenct prior, c) flow conditioning on E/D. While Table 4 answers some questions, in light of VCT I would have really like a row that is just \"last two frames\":\n\na) I am unconvinced that the ConvLSTM provides any benefit over just feeding the last two representations (i.e., why not feed y_{t-1}, y_{t-2} instead of y_{t-1}, r_{t-1}). The authors present no evidence that the (complicated and annoying to train) ConvLSTM provides any benefit.\n\nb) Furthermore, and *more cruicially*, no ablation on c) flow conditioning is performed. I would be really interested in seeing how that affects PSNR.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LSuHKZ0UxH-", "reviewer": "Reviewer_DWzw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes MIMT which is a transformer-based model that can predict the PMFs of tokens in arbitrary order, not necessarily in raster-scanning order (e.g. line-by-line). After training the MIMT model, they propose an “iterative decoding scheduler” which decodes tokens from the current frame from smallest entropy to largest. ", "summary_of_the_review": "In summary, I think MIMT is a promising direction and a cool idea - I like the intuition of decoding tokens in non-raster order. But it needs more fleshing out from an experimental and potentially a theoretical perspective. The authors provide relative comprehensive experimental results on overall bitrate/distortion curves, but I would want to see more results that highlight the benefit of MIMT itself. It is also unclear to me how much stuff like the hyperprior, last frame, recurrent prior (none of which are novel ideas) are contributing to the boost of results vs. this entropy model. And a bonus would be to see a larger mathematical justification. \n", "strength_and_weaknesses": "Strengths:\n- The authors have relatively comprehensive evaluations over UVG/MCL/HEVC-B and show that their work is state of the art.\n- The authors provide runtime comparisons, which I appreciate \n- The intuition of decoding the image in an “easy-to-hard” process is interesting, I appreciate the diagram shown in Fig. 6.\n\n\nWeaknesses:\n- I think the iterative decoding scheduler is interesting but I’d like more theoretical and/or empirical analysis as to why it makes sense to 1) go from low entropy to high-entropy, and 2) using a sine function. Did the authors try out other functions / decoding orders? Why was this function and ordering the one that the authors decided to settle on? \n- The recurrent prior + Optical flow predictor isn’t really explained. It doesn’t seem to be a main contribution (the main contribution seems to be MIMT), but it also seems to provide a fairly significant ~10% bitrate savings to all datasets. And this goes into a broader framing of the paper itself. Are the authors proposing the entire e2e system as the novel idea? Or the entropy model as the novel idea? To me, the entropy model seems to be the main idea, and I would love to see an experimental setup that helps me understand what the benefits of that are. \n- I don’t entirely get the purpose of section 5.4 (Ablation Studies), since none of these items are central to the author’s core contributions, which is using MIMT as the entropy model. The paper is also missing ablation studies that would help validate the claims that the authors make regarding MIMT. For instance, the benefits of decoding token by token instead of decoding the current frame all in one step. Or the benefits of going low entropy to high entropy vs. say a raster ordering. Or the decision to use a sine function. \n- I don’t think this is never explicitly stated in the paper but conceptually you would need to use the decoding scheduler when first encoding each frame into tokens, as well as during decoding time right? Because you would need to determine the order in which to encode the tokens for each frame in order to decode the tokens using the same sequence. It’s not clear from the paper itself, and I was initially confused as to how the decoding scheduler would work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K9JD2I3SuX", "reviewer": "Reviewer_1F8U", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends the notion of conditional coding in DMC and Sheng et al., 2021 by additionally introducing a masked image transformer-based entropy model (MIMT) for learned video compression. The MIMT appears to be inspired by VCT and improves on VCT by a scheduling-based transmission. The gain of the proposed method looks very promising.   ", "summary_of_the_review": "This work combines the conditional coding scheme in DMC and an extended version of VCT. The MIMT is inspired by VCT and is novel. While the idea of MIMT is interesting, how it compares with the group-based auto-regressive model needs further clarification. Also, more ablation experiments are needed to understand the complexity-performance trade-off of MIMT. The proposed scheme shows good coding results at the cost of high MAC. It is to be noted that DMC (the existing work) contribute partly (likely considerably) to the overall gain presented in this paper. It is unclear how the coding results may look like if contextual encoding and decoding is disabled and how such a stripped scheme compares to VCT.   ", "strength_and_weaknesses": "Strengths:\n(1) This work introduces a transformer-based, bi-directional entropy model with a masked and incremental coding order. This idea is interesting and novel. \n(2) The gain presented looks very promising. \n\nWeaknesses:\n(1) In my opinion, MIMT presents a different approach to the group-based auto-regressive model, e.g. Checkerboard Context Model for Efficient Learned Image Compression, CVPR 2021. Similar to the group-based model, MIMT encodes the tokens in an incremental manner. However, the tokens to be encoded in each iteration are identified based on their predicted entropy rates, instead of using a fixed pattern as with the group-based auto-regressive model. The dynamic determination of tokens to be encoded comes at the cost of additional compute for sorting. Moreover, the number of tokens to be coded in each iteration crucially determines the final coding performance. Currently, this is done in a heuristic manner by choosing the scheduling scheme r(i) as sin (i * pi/2). The choice of this scheduling scheme needs justification. I believe a different choice of r(i) would lead to different coding performance. Also, the gain of this dynamic approach over the fixed-pattern approach, e.g. checkerboard context model, needs to be clarified.\n(2) The impact of the number of iterations, e.g. the value of n, together with the scheduling scheme r(i) on the final coding result should be studied in an ablation experiment. Moreover, this study should look into how these design choices may affect the complexity in terms of MAC and encoding/decoding runtimes on both CPU and GPU.  \n(3) The entropy model proposed in [1] uses TWO previously reconstructed latents as inputs to estimate the probability distributions of the latents in the current frame. The proposed method replaces one of the previously reconstructed latents with the Recurrent Prior. It is unclear how much additional gain the Recurrent Prior can bring. Note that the Recurrent Prior may cause error propagation in the error-prone transmission.  \n(4) While the proposed scheme shows very promising coding gain, its MAC (4.4T) is very high. It also inherits the disadvantage of DMC of high memory requirements due to the need to buffer contextual information. I wonder how the coding result would look like if contextual encoding and decoding is disabled. Note that without contextual encoding, the proposed method would reduce to a scheme similar to [1], which will not suffer from temporal cascading errors.   \n\n [1] Mentzer et. al.  “Vct: A video compression transformer,“ arXiv 2206.07307, 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AktOMSOAc0-", "reviewer": "Reviewer_33TR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use an iterative decoding scheme for entropy model based on masked image modeling to improve video compression. This scheme has the advantage that a fixed autoregressive order is not used and thus leading to a better modeling of the latents. Further, SWIN transformer based model is used to improve the efficiency of the method.", "summary_of_the_review": "Overall, despite the marginal novelty, I am positive about this paper. Existing techniques are used effectively to improve compression performance. I think these improvements are substantial enough to warrant acceptance.", "strength_and_weaknesses": "Strengths\n- The paper is clearly written and easy to understand.\n- Improved performance over previous strong baselines.\n- A good set of ablation experiments that study some of the method choices and provide insights into them (e.g. visualization and effect of prior information).\n\nWeaknesses\n- Novelty: Incremental. Two main contributions are use of SWIN transformer inspired architecture for efficiency and Masked autoencoding based entropy model. Individually both are incremental. However, the improvements resulting from this combination make up for the incremental novelty in my view.\n- Missing ablation of decoding schedule: Paper uses a sine based schedule to decide patches to decode. It would be useful to understand how critical this choice is to the method's performance. For example, a simple comparison would be simple linear schedule.\n- Training details are missing, leading to poor reproducibility. Model details of how prior information is generated is missing too. Please clarify in appendix at the very least.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%