Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors focus on the problem of lossy video compression. To this end they propose the application of latent variable sequential generative models, specifically autoregressive flows to compress video streams. They evaluate variations of these models quantitatively including their own proposed version of scale space flow. They also introduce a new dataset named Youtube-NT and show promising quantitative performance.

Pros:

1. The proposed problem of video compression has direct societal applications.

2. The application of generative models, especially flow based models, to video compression is a novel and relatively underexplored topic in the community.

3. Quantitative performance is promising.

4. New dataset released will be useful for future work.

Cons:

1. There ideally should be more qualitative evaluation. How do the reconstructions of all these approaches look side by side?

2. The new dataset is fairly small by the standards of video datasets. For example, Kinetics and Youtube 8m have hundreds of thousands or even millions of examples.

In summary, the paper explores an interesting and novel application of flow based models to video compression. While I think the paper could be strengthened by more qualitative examples and a larger dataset, I think this could be a good contribution to the conference.
review_title: Interesting Application of Contemporary Generative Models

final_score: rating: 7: Good paper, accept
review_content: This paper proposes improved version of Scale-Space Flow (SSF) model. Authors add two enhancements to SSF for improve Rate-Distortion performance: (i) learnable scale transform and (ii) structured prior (SP). The paper also introduce a dataset for neural video compression that collected from youtube.com.

Pros: 
- Proposal seems better results than conventional SSF in some bit-rate and dataset conditions (Figure 3).
- Publishing a new dataset may help the community.

Cons: 
- I think the introduction of scale parameter is the most important point of this paper. Authors says it acts as a gating mechanism. Figure 2 shows the case of an image, but no comparison with traditional SSF has been made. Figure 4 shows final RD-curve comparison from traditional SSF but I don't know how the gating mechanism works. For example, showing some kind of residual noise amount of conventional SSF (Agustsson et al., 2020) and show a reduce effect by scale parameter will make the claim of the paper credible.
- There needs to be a more specific and clear explanation of how the SP is processed.
- Figure 4 shows the effect of SP is that the image quality is better than SSF at high bitrates, but the image quality is worse at low bitrates. I would like to see a discussion on the reason for this.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a lossy video coding scheme using autoregressive generative models. 
+ It is good to see that the paper includes comprehensive reviews regarding neural network based compression schemes and autoregressive models.

- The contribution has been depicted in aspects of 1) a new framework, 2) a new model, and 3) a new dataset. However, beside to 3), it is hard to see the other two aspects have significant novelty both in a neural video coding architecture (Agustsson et al. 2020) and in temporal autoregressive models, etc. 

- Furthermore, current video codecs (i.e. HEVC) are developed for YUV4:2:0 because the current broadcasting system has supported the video format. The evaluation should be conducted in YUV 4:2:0 domain for fair and reasonable comparisons.  


review_title: Regarding major contributions and experiments

final_score: rating: 7: Good paper, accept
review_content: #### Summary
In this paper, the authors provide a new interpretation of existing video compression models. Their perspective is that a video decoder is a stochastic temporal autoregressive model with latent variables. The introduced latent variables could be either used for providing more expressive power for 1) motion estimation&compensation modeling and 2) residual noise modeling, which are two key components of traditional video codecs. The proposed method shows favorable results when the bitrate is higher than 0.12 bits per pixel on the public benchmarks.

#### Strength
The proposed method has the following strengths.
* The new interpretation of the video compression model gives a unified framework and insights. Although I am not entirely familiar with the learned video compression field, the interpretation looks quite interesting. It would be helpful for other researchers in the field.
* New moderately high-resolution dataset is provided for training learned video compression models. This is a good contribution to the community.
* The proposed model improves performance over the baselines. They propose two new components: 1) a learnable scaling transform for modeling motion estimation/compensation, and 2) structured priors for modeling the entropy.

To make the work stronger, I have a few suggestions as follows.
* Study on the effect of using a pre-trained optical flow model. It seems that the parameters for motion estimation (g_w, f_w, w_t) are learned end-to-end. However, it might be challenging to learn motion estimation parameters end-to-end with other parameters. I am curious how much gain we can get if we plug in the pre-trained optical flow estimator. Intuitively it should give some boost, as one major component of the video codec is the motion estimator.
* More failure mode analyses. The proposed models, STAT and STAT-SSF, show favorable results when the bitrate is high but underperforms baselines in low bitrate. It would be better to have some discussion about why the case is. 
* High latency extension. It would be interesting to see how the proposed framework can be extended for high-latency compression. In traditional video codecs, high latency compressions are also widely used to achieve high rate-distortion performance. However, in this work, only the frame t and t-1 are considered for compression. 


#### Minor comments
* I am not sure the authors can cite papers in the abstract.
review_title: This work brings a new interpretation of learned video compression and shows promising performance.

Meta-review: All reviewers recommend acceptance. The authors have addressed several of the reviewers' concerns in their comments, conducted additional experiments, and updated the manuscript accordingly.

A concern was raised regarding the size of the dataset introduced and used by the authors for this work. However, I agree with the authors that it doesn't necessarily make sense to compare this to datasets designed for training video classification and/or generation models; In the compression setting, the quality of individual data points matters much more than their quantity, as the authors argue.

Reviewer 2 was curious about the potential of a pre-trained optical flow module. I believe the authors have convincingly argued that end-to-end learning is likely to be more effective and practical (and indeed, there is plenty of evidence for this in other ML contexts where training data is not scarce). I agree that a direct comparison in the paper would have been interesting, but this would constitute a significant investment of time and effort on the authors' part (as they also point out, training such a module separately could actually be more difficult), and I think it would be unreasonable to make this a condition for acceptance.
Decision: Accept (Poster)