Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed a scalable implicit neural network for reconstructing video signals in a parameter-efficient and computationally efficient manner. In order to do this, they proposed learnable positional features inspired by the existing work in the literature (Instant-ngp). In general, implicit neural network maps from input co-ordinates to RGB pixels directly using one mapping function, here the authors maps the pixel co-ordinates to intermediate latent space, and from latent space to RGB pixels using another mapping function with modulation function. In the latent space, the authors learn the key frames of the given video in the latent grids, for xy, xt, yt, xyt. These are compressed using existing standard codecs HEVC and JPEG, and interpolation is used to find the latent vectors for any given (x,y,t), and then latent to RGB mapping function. The authors show their proposed method trains faster, and reconstructs the video better in a parameter efficient way.
strengths_and_weaknesses: Strengths:
The paper proposes the parameter and computational efficient implicit neural network to reconstruct videos from pixel coordinates. The method has the advantage of learning keyframes automatically from the videos, whereas the other compression methods do not have the capability to learn the keyframes.
Learns the grid representations in the latent space for different directions (xy, xt, yt, xyt). The authors claim that it improves parameter efficiency of Neural video representation (NVR)

Weakness:
For the compression of the latent spatial grids, the HEVC and JPEG compression methods are used. For the video compression applications, the combination of traditional codecs with learning-based compression is not interesting in my opinion. The traditional codecs are bound to have artifacts that the proposed method incorporates into the proposed method 
The performance of the proposed method are below the HEVC and H264, and the datasets used to evaluate the proposed method is not a good benchmark. I see that in the honeybee video, the video is almost static and it is difficult to conclude whether the proposed method can capture the dynamic motions in the temporal dimensions.
The proposed method is not compared with other state-of-the-art methods in learning-based video compression methods.

questions: 1) In learning latent keyframes, what kind of information do the keyframes contains, and in terms of visualization how does it look like?. does it learn image like structure?
2) how the BPP is calculated? whether the weights of the network are used in full precision or half precision to compute the BPP. In the total filesize, how much information is from latent grids, and how much information is from the latent to RGB mapping function.
3) I visualized a few videos in the dataset used in the paper, the temporal motion in the videos is not much dynamic, it's almost static. Maybe this is the reason, the method is able to have good reconstruction results. I suspect whether the proposed might have a similar performance with high temporal variations.
 
limitations: the limitations are discussed in the paper
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a coordinate-based architecture for video representation. The key element of the architecture is the introduction of keyframes (positional encodings of the features). The approach is validated on the UVG benchmark showing benefits over prior art.
strengths_and_weaknesses: The paper is in general well-structured and well written. The model design choices are motivated, and the reported results looks solid. The idea is moderately original and the quality of the work is good enough to recommend the acceptance. However,  he clarity of the presentation could be improved and the authors could highlight better the significance of the work. See detailed comments and questions below.

**Abstract:**
- Abstract could be strengthened to reflect better the paper contribution and its importance.

**Introduction:**
- The introductory section positions well the work w.r.t prior art.
- Introduction does not motivate why research on CNRs for video representation is an important research avenue. Adding such motivation would make the paper stronger.
- The connection between learnable positional features and keyframes is now well exposed in the intro and in the abstract. It is hard to understand why keyframes could play the role of learnable positional features.
- Fig 2 could be a bit more informative; the key element of the paper seems to be the construction of the keyframes however, the figure does not give details on how the keyframes are constructed.
- L24 “… numerous appealing properties …“- only one property is mentioned.
- L36 “… while enjoying lots of intriguing properties …” – could the authors enumerate the properties
- L71 “… 34.43 in 5 minutes …” – can the authors add details here? What is the hardware? How is the time measured?



**Methodology:**
- In general methodology section is well written and easy to follow. One exception is the construction of the keyframes. This part could benefit from some re-writing. Also, adding a figure explaining the process could ease the understanding. Moreover, it is unclear why U_l is represented as a set.
- Did the authors consider different types of or designs of keyframes?
- Modulation, in L174 the authors write: “… we found such a simple MLP architecture lacks an expressive power and fails to capture the complex dynamics… “. However, based on ablation study the benefit of modulation seems marginal (in terms of PSNR). Could the authors comment on this?

**Experiments:**
- The ablation table is missing std, could the std information be added to the ablation table?
- Adding additional dataset would make the validation and observations stronger.

questions: How does the method scale with video size in terms of both the spatial resolution of the frame as well as the number of frames?
limitations: The limitations are discussed but are centered around the general idea of CNR for video representation. The discussion of the possible limitation of the model design could be extended. For example, one might expect that the keyframe idea (amortizing over the video) would not work well for long videos composed of multiple possibly dynamic scenes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new CNR framework by introducing “learnable positional features” and decomposing the video into 2D and 3D latent representations. Besides, a compute- and memory-efficient compression procedure is introduced to further reduce the parameters and training cost. Experiments are conducted on UVG-HD benchmark for evaluating video encodings.
strengths_and_weaknesses: Strengths: 
This work proposes a new CNR framework to solve the dilemma of quality and efficiency in video encoding. The experimental results have been improved on both PSNR metrics and parameter-efficiency. The overall clarity of the paper is good.

Weakness: 
This work is mainly built based on the existing CNR framework, but the overall originality is not high. The experimental part is not sufficient, and some experimental settings are also unfair to some extent.
questions: Here are some concerns to be addressed:

1. My major concern about this work is the experiment results shown in Table. 1, in which the experimental setting is different from NeRV. Maybe the results on “Big Buck Bunny” sequence should be given to present a more intuitive comparison.
2. Also in Table. 1, why is instant-NGP not compared after encoding time > 15hours?
3. The results of video compression evaluation should be more detailed. Figure. 4 only shows the results on ShakeNDry and Beauty. More comprehensive experimental results should be presented.
4. Lack of comparison of decoding time.
5. Why is the grid size selected as H×W×S? It is better to show the influence of different grid sizes.
6. More experiments comparing with traditional coding frameworks such as VVC and HEVC are suggested.

limitations: Yes.
review_title: null

Meta-review: metareview: The paper proposes a coordinate-based architecture for representing a video in a parameter-efficient and computationally efficient manner. Such architecture can be used for video compression, inpainting and frame interpolation.

The initial concerns were addressed during the rebuttal period and all the reviewers had a positive opinion of the paper. I therefore recommend acceptance.
confidence: Less certain
Decision: Accept