Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes to model videos as a function mapping time to image, and uses a neural network to model this function. It trains a neural network on one video or a concatenation of several videos, by overfitting the neural network on the videos. Since the motivation is to represent a video in full, the network is not trained for generalization, rather it is trained to overfit. Then, the paper tries to do video compression by compressing the weights of the neural network. It uses 3 model compression techniques in a cumulative fashion : pruning, weight quantization, entropy encoding. It is found that this gets the compressed representation close to standard video compression techniques in terms of PSNR vs BPP (distortion ratio).
main_review: The paper introduces a simple method for video representation : use the weights of a neural network to represent the video itself. Although there have been other techniques that have tried this, they have been clearly mentioned in Section 2. However, the paper's idea is rather more direct : overfit the weights of a neural network to the video pixels as a function of a Fourier representation of the time index.

This simple idea seems to lead to a representation that is comparable to standard video compression techniques. Ablation studies is provided to explore different aspects of the neural network: 1) PixelShuffle is better for image upsampling, 2) normalization harms the performance (probably because the goal is to overfit), 3) LeakyReLU is better than ReLU, 4) Fourier embedding of time is better than using the time directly. Although each of these ablations studies could be more extensive, the overall network is already comparable to standard techniques in terms of performance, as shown in Figures 5 and 6.

Figure 4 is an insightful plot into how each of the model compression techniques contributes to the method's performance. The conclusions drawn from this plot are reasonable, the significant impact of each compression technique is clearly illustrated.

Apart from video compression, the paper shows the robustness of this method to noisy videos in Table 9. It would be great if it is clarified whether the same noise is added at every batch or whether new noise samples are added per batch. If it is the former, it is indeed impressive that the model is robust to such noise. However, if it is the latter, then it is reasonable that the neural network is learning to ignore the noise for the content.

There is only 1 qualitative example provided in the supplementary. It would be good if more qualitative samples are provided to illustrate the effectiveness of this method.

Another clarification is why the PSNR of videos with random noise (0.3) added is quite high (39.07), even higher than denoised videos.

The paper also makes a case for temporal interpolation. However, only 1 example has been shown (Figure 7). It would be good if more examples are provided.

The paper does mention some of the above as limitations.

Code has been provided to train the model and reproduce results.



Line 189 : Table 3*
Line 220 : Table 3*
Line 233 : bilin*ear
Line 261 : fix*, avoiding*
limitations_and_societal_impact: The limitations of the method are clearly described at every stage. The major limitation is that although the performance is comparable to standard techniques, the training time is much higher. Moreover, inference time (time taken to translate from time value to pixel) has not been mentioned. This is an important metric to compare with standard techniques.

Code has been provided to train the model and reproduce results.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: Video is encoded as a neural network that takes frame index and outputs the video frame corresponding to the index. Video compression problem is cast as a model compression problem, and the proposed neural representation achieves performance comparable to standard frame-based compression method. The proposed neural representation is demonstrated for temporal interpolation and video denoising tasks. 
main_review: (+) The paper is relatively well organized and easy to follow. 
(+) The paper also introduces an interesting idea for representing a video but the benefit/significance is not very clear.

(-) The task mentioned in the paper seems very similar to the copy task introduced in the paper Neural Turing Machine. It would be interesting to compare how the proposed network would perform in the copy task. There is an obvious difference in input and output between the proposed architecture and the NTM. Is it possible to have a neural architecture that is not dedicated to a video but a set of videos would be an interesting question.
Graves, Alex; Wayne, Greg; Danihelka, Ivo (2014). "Neural Turing Machines". arXiv:1410.5401.
(-) The paper does not discuss the philosophy behind architecture. It is difficult for the reader to understand why the network was designed the way it was. For example, there is very little explanation about why the input is encoded as a Fourier feature (is it the only high embedding space to consider) and what the role of pixel shuffle is in the architecture.
(-) Conventional Model compression described in Section 3.1 seems as complex as conventional video compression methods (H264, HEVC). How is the audio of the video encoded?  It would seem that the compression performance will depend on many factors (weight quantization and entropy encoding) which are not fully discussed.
(-) The proposed algorithm does not achieve SOTA and does not present a clear benefit for representing the video in the manner the paper is proposing. 
(-) Coordinate-based neural representation is not a novel idea and applying it to video does not seem to give any benefits in terms of performance. 
(-) There are various components of the architecture which the authors have compared  (table 4-8) but it is not clear why one would perform better than others. A deeper understanding and intuition would be appreciated. 
[1] Readers will appreciate any insights the authors can provide as to how high-frequency variations of the timestamp provides a better result than just the timestamp. Line 128-129 (.. input to high dimensional space seems ... neural network can better fit...) to be suggesting inputting vector representation of scaler values [0, 0.1,0.2,....1] gives better results. The authors do provide additional experimental results using different values of $b$ and $l$ but it does explain why the result would depend on the different representations of deterministic values. 
[2] The reader would be interested in the author(s)' elaboration on "burdensome parameters" which is the motivation of upscaling using pixel-shuffle. 



limitations_and_societal_impact: NA
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes  a novel neural representation for videos. The key idea is to represent a video as a function conditioned on the time index input. Given a video, it uses a MLP to fit such a function. Two applications are conducted: video compression and video denoising. 
main_review: I cannot see any strengths from this paper. My concerns include:
1) The key idea, to use time index as an condition input is really a common-used strategy in many existing works, like video generation. It lacks insights. 
2) Compared with traditional video compression methods, the proposed algorithm requires a video-specific training and a model-pruning post-processing, which needs 3-4 days. This is also stated by the authors in the limitation. In this sense, it lacks applicability. 
3) For the experiment on video denoising, training a model to fit a noisy video can finally produce denoised video. This is actually the key insight of "deep image prior" and has no relation to the proposed idea. 

limitations_and_societal_impact: The limitation about the time-consuming is very difficult to handle, given the proposed key idea.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper describes a method for representing videos using neural networks. This is inspired by recent advancements in implicit neural representations, which encode various types of signals (audio, images, video, shapes, radiance fields) in the weights of a MLP overfit to the signal. The paper accomplishes this by contributing a new network architecture utilizing convolutional layers which maps each frame index t to the **entire image frame**. Since this network memorizes each frame of the video, compression methods can then be applied on the network instead of the frames themselves to yield a comparable compression versus video quality trade-off. The paper demonstrates that this alternative compression method is comparable to existing standard video compression methods, and that it is robust to noise in the ground truth video.
main_review: The paper has a number of strengths and weaknesses. In my opinion, the strengths of the paper are:
1. **Clarity**. The paper is written very clearly. The method described is concise, and the paper does a good job of explaining why this sort of representation makes sense for video compression, denoising, and frame interpolation. 
2. The quantitative results described are exhaustive, and show that the method does work in memorizing video data using the network architecture described. The rate distortion plots are clear and demonstrate that the method is comparable to existing standard video compression methods, and the tables show that the method has the ability to filter out noise from frames of the video.

In my opinion, the weaknesses are:
1. The related work section misses a few very related works. 
    - One major omission is that there is no mention of [SIREN](https://vsitzmann.github.io/siren/) which performs the same experiment of video representation using an implicit neural representation. While I agree that the network architecture used in this case is significantly different (NeRV utilizes convolutional layers to output discrete frames as a function of a continuous frame coordinate, while SIREN also model the pixels in each frame continuously), I think that this work needs to be at the very least mentioned, and ideally compared to.
    - Another major omission is [Deep Image Prior](https://dmitryulyanov.github.io/deep_image_prior), which shows that denoising on images can be done with no supervision, by only training a CNN to represent the image and using early stopping to prevent overfitting. This is highly related to the fact that NeRV is robust to ground truth video noise, and it would be nice to see an ablation, on perhaps a smaller video, to see if given enough training time, the NeRV representation would still learn to memorize the noisy video. This would provide significantly more understanding about *why* this representation is robust to noisy video. A comparison to this method for denoising could also help quantify the difference between NeRV, and the DIP effect.
2. My major concern is with the **significance** of the contribution. I do think the **originality** of the novel network architecture and idea of framing video compression as network compression is worth publishing, in the current state of the paper the results seem not quite complete yet:
    - The limitation of long training time to learn the representation makes this method for video compression not very attractive, especially when considering that quantitatively it only performs on-par with other video compression methods.
    - It’s unclear how the method scales to shorter or longer videos. Would it even be feasible to compress a video which is hours long? No ablations are provided on this aspect, which is very important for evaluating whether this could make an impact in video compression.
    - Getting video frame interpolation for free by using a continuous representation seems appealing, but based on the qualitative results shown in the paper, it does not work quite yet. If this worked well, I think this could significantly improve the quality of the paper since it can also be compared to existing frame interpolation methods.
3. The paper provides limited qualitative comparisons. While seeing the average PSNR or SSIM of the video frames gives some idea of the quality of the estimates, it would make sense to include some sort of qualitative visualization of the result, to make sure there are not artifacts which may induce very small quantitative error, but be very noticeable by human observers. For example, if every other frame was consistently biased away from the GT frame in opposite directions, this could manifest as a very annoying flickering effect in the video which is not caught by the metrics which are used to evaluate quality.

Overall, I like the idea of the paper and think that it has done a good job of making the method simple and clear, and justifying why the modified network architecture and alternative video compression method via representation network compression are valuable and original. However, I think that the significance or impact of the method is limited, since the results are not better than any existing standard compression methods in rate versus distortion, or compression time - the two axes which are most important for video compression. Additionally, the frame interpolation using the continuous representation doesn’t work well enough yet to compare along the frame interpolation axis, and the denoising isn’t studied in depth enough to be sure that NeRV provides novel insight or contribution here. Without these improved results, I am not sure if the ideas of the method alone are ready to be published yet. With improved results, the potential impact of the paper could certainly be improved significantly and make this a much stronger submission in my opinion.

Other comments and questions on the submission:
- typo: On line 189, should reference Table 3 instead of Table 9.
- The section on using a single network to represent multiple videos is not described thoroughly, which makes it hard to understand whether or not some kind of generalization / feature re-use is being done here. Is each video represented by a latent code, which is also input into the network? Or is each video fit in a different part of the input domain? This should be clarified, because I think that this generalization aspect could be one way to improve the rate distortion results, and also provide very interesting insight into feature generalization within implicit neural representations. In my opinion, this explanation and emphasis here could make the submission a lot stronger as well.
- The ablations show that the BatchNorm significantly decreases the quality, but all the experiments are done using a batch size of 1. Does BatchNorm even make sense when the batch size is 1, or does it throw away some information by just normalizing the entire output feature in each layer? Could this be why it does so poorly?
- minor: The references section is inaccurate. For example, [43] was published at NeurIPS 2020, [11] was published at NeurIPS 2019.
- It may be worth to consider changing the name of the method in the title, since NeRV already exists as a publication at CVPR 2021: [NeRV](https://pratulsrinivasan.github.io/nerv/).
- The checklist answers are not completely accurate:
    - 4(b)-(e) are not correct (no new material was included in the submission, and the content of the data used was not discussed). It’s not an issue to answer no or N/A to these questions if no license could be found, for example...
    - 3(d) is not included in the supplementary material

**Changes after Author Responses**:

The response has provided additional comparisons with SIREN architectures, which I think when included in the manuscript should make the contribution of their architecture more impactful. The additional clarifications and comparisons have also cleared up a number of my concerns regarding denoising capabilities, longer videos, and batch size. Thus, I have increased my score. The paper proposes an architecture change, ablates it well, and shows that it gets reasonable results.

However, my concerns remain regarding the limitation of long training time and compression rate compared to existing methods, leading to my borderline assessment of the paper. Without either focusing in depth on *why* the architecture change is an immensely impactful shift in coordinate-based network architectures for compression which will lead to countless future improvements, or providing some kind of utility over baseline methods, I think that the significance of the method is limited.

**Additional Changes**:

The authors have responded to me changing my score by providing more comparisons and an argument related to decoding time being more important than encoding time. I certainly agree with them on this point and appreciate the comparison, and see this as a significant benefit of NeRV. However, shifting the paper to one which tells a story about maximizing a trade-off between compression and decoding time may require a significant amount of additional experiments and ablations (ex: trade-off between PSNR, compression rate, and decoding time). Since there are additional variables here, I don't think I can provide a conditional acceptance in this version of the paper.

As the current version of the paper stands, I think that performance compared to the baselines is too similar to justify the architecture contribution alone. Without either this, or some insight about the architecture change which shows why this is an impactful shift in coordinate-based network architectures for compression which will lead to future improvement, I think the significance of the contribution is not as high as it could be.

I certainly think that there are the makings of a good paper here with some additional experiments and insight, but in its current state with some minor modifications, I will retain my borderline score.

limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations of their work. They have also addressed the potential negative societal impact of their work.
review_title: null

Meta-review: While there was only one reviewer who recommended accepting this work, this positive reviewer made a strong case for why this paper should be accepted.  While there is some concerns about the practical applicability of the proposed method as a real-world compression method, the reviewers have pointed out that a video need only be encoded once in many situations. As such arguments about the amont of time required to encode a video are less strong.  The issue of not handling audio with this work is not relevant. Video compression research papers rarely also address the audio component of a video. 

The authors provided a solid rebuttal addressing key points raised by reviewers. In particular, the author response and additional comparisons with SIREN architectures should be integrated into the final manuscript. Reviewers believe that this will help make the contribution of the architecture here more impactful. Additional clarifications and comparisons provided during the discussion phase also cleared up a number of reviewer concerns regarding the denoising capabilities, longer videos, and batch size. In the end the most engaged, but still somewhat negative leaning reviewer asserted that "The paper proposes an architecture change, ablates it well, and shows that it gets reasonable results.". 

The positioning of the paper with respect to prior art and contributions was sufficient and by acknowledging the points of the reviewers properly in the final version of this manuscript, I think this work will serve as a good contribution to the conference. The lack of SOTA results is not a reason for rejecting this work. The general idea here is quite interesting and novel enough to warrant acceptance. The idea proposed here is sufficiently interesting and well explored in this work that it would be an interesting contribution to the conference.  This paper is likely to stimulate further interesting discussion at NeurIPS and has potential for further impact.

The AC recommends accepting this paper.

Decision: Accept (Poster)