Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors tackle the problem of video compression using transformers. The proposed architecture doesn't include specific biases and priors like motion prediction, blurring, etc. Transformers are used to predict the distribution of the future representation given the earlier video frames. The idea is to exploit the temporal redundancy across frames and the spatial consistency within frames which can help in entropy coding. They show good results using transformers on standard video compression data sets (MCL-JCV, UVG, synthesized videos from CLIC2020) as compared to previous approaches.
strengths_and_weaknesses: Strengths:
* There is no explicit biases or priors, which may help the proposed architecture to generalize better across datasets
* The authors introduce independence assumptions among the blocks of each video frame which shrinks the attention matrix and enables parallel execution on subsets of the video frame.

Weaknesses:
* The paper is not easily readable as there are several inconsistencies as pointed below
* In Fig 1, it seems that y_i is fed to the Masked Block-Autoregressive Transformer, but from the later text, it seems it only previous tokens of y_i are fed to get the probability of the current token of y_i.
* line 207: Even though it is a bounded window, if z_cur is used to decide y'_i then y'_i does indirectly depend on all previous encodings which can propagate errors. This counters the claim made in line 39 that the architecture doesn't propagate temporal errors.
* Entropy coding: The distribution of the next token can help identify the number of bits to encode the token. But it is not clear how one of the S symbols can be arbitrarily encoded into that many bits.
* Availability of code or at least pseudo-code in the appendix could have helped understand how entropy coding is done in the current approach
* Table 3: To evaluate better, the runtime of the baselines should also be presented

Minor:
* Figure 4: bits per pixel is not cited
* line 189: what is distortion loss?
* line 280: Even though no further gain is observed from more context, it may help if it is shown in the Table 2.

Typos:
* line 36: '2.1.1' instead of '2.2.1'
* line 56: 'for' is redundant
* line 108: The output of T_cur should be z_cur, but the figure has z_joint.
* Eqn 2: round(y_i) should be round(\tilde(y)_i) ?
* line 192: integration is done over du ?


questions: It may help if the authors can tell how the entropy coding is done in the current context (see weaknesses above). Also if the authors can address any other weaknesses mentioned above then it can help in better understanding of the approach.
limitations: Yes the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors proposed a video compression transformer (VCT) for neural video compression. Input frames are mapped to representations, and a transformer is used to model the dependencies. The VCT model achieves SOTA performance on MCL-JCV and UVG datasets. Both theoretical and empirical results are promising and will benefit future research.
strengths_and_weaknesses: **Strengths**:

1. The proposed framework is simple and effective. The frames are decoded from the quantized representation, which prevents the problem of error propagation for residual coding-based video compression methods.

2. The initial frames are decoded with zero padding for y_i-1 and y_i-2 so the I-frame and P-frame models in residual coding-based framework are unified into one model. This design large simplifies the video compression framework and will benefit future research.

3. In order to speed up the execution, the authors assumed independence between blocks and only overlapping tokens from i-1 and i-2 are used. This seems a bold assumption at the beginning but proved effective.

**Weaknesses**:

1. Most modules follow the most common designs and are not curated for this specific model. Some weaknesses/improvements are considered by the authors in Section 3.2-Independence and Section 6.

2. Other weaknesses and concerns are detailed in the ‘Question’ section.
questions: 1. What’s the HEVC setting in Figure 5? Since SSF is not using a pretrained optical flow model, I wonder what other models (e.g., DVCPro or DCVC that builds on SPyNet) would behave on the shift sequences. It seems that learning-based models achieve superior benchmark performance than HEVC by handling distortions such as sharpen/blur/fade better, but not as well to shifts. It’s unclear what’s the cost of not using motion compensation in the VCT framework, whether the cost is negligible or if the transformer predicts good enough distributions such that an ME module is unnecessary.

2. It seems that the VCT handles shifts better when the shift is the same size as the patch size. Are the authors aware of/considering any approach to improve this problem?

3. VCT and SSF are trained on different data from most other works. Although larger-scale training has been proved necessary on various language/vision tasks, is there an ablation study on the size of the training data for VCT? This would help to understand the limitation of the model, its scalability, and its potential for future work.

4. A less important question: have the authors considered the HEVC dataset and other video compression metrics (e.g., AVQT or perceptual-based metrics)? As VCT is a novel architecture, it would help to understand how it compares with previous works as well as decide future directions.
limitations: In this work the authors proposed VCT for neural video compression. The model design is novel and elegant and the model architecture follows most common designs. It helps to show the efficacy of the proposed framework and many improvements are also possible (as summarized in Section 6). Some possible limitations in terms of motion compensation and training data are mentioned in Question 1 and 3.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a transformer architecture for video compression and demonstrates its favorable performance over prior methods. The method is explicit into two stages: firstly lossily encode each frame independently into a quantized latent space, and secondly use a transformer model to find a probability mass function under which the quantized latent sequence can be efficiently transmitted. The proposed model relies on data-driven prior instead of architectural biases as used in many prior methods. 
strengths_and_weaknesses: Strengths:
1. The paper is clearly written and easy to follow.
2. The method achieves favorable empirical performance on standard benchmarks. 
2. The paper discusses the the type of temporal patterns that the model learns to exploit by examining synthetic videos. The discussion provides a good understanding of the model's capability when applied to data sets with different motion patterns. 

Weaknesses:
1. The comparisons of model size and inference time with baseline methods are not listed.
2. The paper provides an intuitive explanation in line 108-110 of why the second stage is advantageous compared to the naive solution. The hypothesis could be tested on synthetic datasets, which will further strengthens this claim and verifies if this is indeed what the type of inductive bias that the model learns from the data.
questions: 1. In table 2, the PSNR of model variants with 0, 1 and 2 previous frames are the same. Does the author have an intuition why this is the case? 
2. The paper suggests that more than 2 context frames do not provide further gains in line 280. Is this a result of the type of motion presented in the testing datasets? With larger temporal motions in the testing data, will more frames provide additional benefits, or the it's the capacity of the current model architecture that limits more context information from being efficiently used? 
2. A potential extension of the work is to incorporate temporal-aware quantization in the first-stage training instead of processing each frame individually. 
limitations: The limitations and societal impact are addressed.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper uses transformer to capture the relevance between the latent representations of different frames. By using the latent representations of two previous frames as the entropy model input, a good probability estimation can be obtained for that of the current frame. In addition, an improved auto-regressive model is used, where 16 times are needed to decode a frame. 
strengths_and_weaknesses: 
Strength

- This paper investigates the conditional entropy coding-based framework rather than following the mainstream residual-coding based framework where the motion estimation and motion compensation are needed. This is an encouraging attempt. 
- Investigate different kinds of synthetic videos including shifting, sharpen/blur, and fading. Some analyses are conducted.

Weakness

- The most issue of this paper is that the novelty is limited. The core contributions have been investigated in other papers.
  - Using the latent representations of previous frames to estimate the probability for that of the current frame has been proposed in [1]. The difference is that one previous frame is used in [1] and two previous frames are used in this paper.
  - Using transformer as the basic unit has been investigated in many neural image codecs, like [2,3,4,5]. Directly applying transformer into neural video codec is straightforward and the novelty is limited.
  - The block-based auto-regressive entropy model is a trade-off between the normal auto-regressive model and checkboard prior model [6]. Actually, the checkboard prior can be regarded as a kind of block-based auto-regressive entropy model, where the block contains two parts. In addition, [7] has investigated a similar idea where each block contains 4 parts. The difference of this paper is that each block contains 16 parts. 
 - This paper claims that the motion prediction and warping operations are complex and hand-crafted in mainstream residual-coding based framework, and this paper does not need these designs. The block-based auto-regressive entropy model, LRP, and  3-stage training are also complex and hand-crafted designs. 
 - The compression ratio improvement over previous SOTA ELF-VC is limited. The RD-curves are very close. The BD-rate numbers are best presented. 
 - Actually, HEVC and H.264 only specify the decoding part and exclude the encoder part. They do not have “veryslow” or “medium” settings. Please use x264 and x265 in the experiment description. In addition, outperforming x264 and x265 is very easy in 2022. Comparisons with JM, HM, VTM are recommended because they represent the best encoders of H.264, HEVC, VVC. The work [8] in 2021 has already compared with VTM. 

[1]. Conditional Entropy Coding for Efficient Video Compression, ECCV 2020

[2]. The Devil Is in the Details: Window-based Attention for Image Compression, CVPR 2022

[3]. Entroformer: a transformer-based entropy model for learned image compression, ICLR 2022

[4]. Transformer-based transform coding, ICLR 2022

[5]. Joint Global and Local Hierarchical Priors for Learned Image Compression, CVPR 2022

[6] Checkerboard Context Model for Efficient Learned Image Compression, CVPR 2021

[7] High-Efficiency Lossy Image Coding Through Adaptive Neighborhood Information Aggregation, arXiv 2022.

[8] Versatile Learned Video Compression, arXiv 2021

questions: - The authors use TPU to measure the training and testing time. However, TPU is not available for most researchers. Profiling in NVidia GPU is recommended, and it is easy for the comparison with other related works. 
- This paper manually uses the latent representations of 2 previous frames, why not 3 or 4 frames?
-  Each block is divided into 16 parts, why not 4, 9, 25,or 36 parts?

limitations: No potential negative societal impact
review_title: null

Meta-review: metareview: This paper uses transformers for video compression, using less components compared to competing methods. Video compression is an important application in machine learning, and the use of transformers is well-timed w.r.t. generally strong interest in the architecture. There were some concerns over clarity of presentation, as well as issues with some of the experimentation, which the reviewers and authors seem to mostly seem to have been able to work out. There is one exception in one reviewer: the authors seem to have worked very hard to address all of this reviewer's concerns, but said reviewer did not adjust their score at all. In addition there was disagreement on some of that reviewers more prominent points.

So I recommend acceptance of this paper.

Overall, w64c stood out as an exceptional reviewer, as they engaged beyond their own review and actively both with the authors and the other reviewers. xvyj brought up some good points, but they were almost tyrannical towards the authors and never gave back in terms of score when the authors clearly satisfied their concerns.
confidence: Certain
Decision: Accept