NeRV: Neural Representations for Videos
Authors: Hao Chen
Bo He
Hanyu Wang
Yixuan Ren
Ser-Nam Lim
Abhinav Shrivastava
Venue: NeurIPS 2021 Poster (2021)
Keywords: neural representation
implicit representation
video compression
video denoising

Abstract: We propose a novel neural representation for videos (NeRV) which encodes videos in neural networks. Unlike conventional representations that treat videos as frame sequences, we represent videos as neural networks taking frame index as input. Given a frame index, NeRV  outputs the corresponding RGB image. Video encoding in NeRV is simply fitting a neural network to video frames and decoding process is a simple feedforward operation.  As an image-wise implicit representation, NeRV output the whole image and shows great efficiency compared to pixel-wise implicit representation, improving the encoding speed by $\textbf{25}\times$ to $\textbf{70}\times$, the decoding speed by $\textbf{38}\times$ to $\textbf{132}\times$, while achieving better video quality.  With such a representation, we can treat videos as neural networks, simplifying several video-related tasks. For example, conventional video compression methods are restricted by a long and complex pipeline, specifically designed for the task. In contrast, with NeRV, we can use any neural network compression method as a proxy for video compression, and achieve comparable performance to traditional frame-based video compression approaches (H.264, HEVC \etc). Besides compression, we demonstrate the generalization of NeRV for video denoising. The source code and pre-trained model can be found at https://github.com/haochen-rye/NeRV.git.

NeRV: Neural Representations for Videos
Hao Chen', Bo He', Hanyu Wang', Yixuan Ren', Ser-Nam Lim", Abhinav Shrivastava
University of Maryland, College Park, Faacebook Al
(chenh, bohe, hywang66, yxren, abhinav) @umd edu, sernamling fb. com
Abstract
We propose novel neural representation for videos (NeRV) which encodes videos in neural networks. Unlike conventional representations that treat videos frame sequences, we represent videos as neural networks taking frame index as input. Given a frame index, NeRV outputs the corresponding RGB image. Video encoding in NeRV is simply fitting a neural network to video frames and decoding process is simple feedforward operation. As image-wist implicit representation, NeRV output the whole image and shows great efficiency compared to pixel- wise implicit representation, improving the encoding speed by 25 to 70x, the decoding speed by 38x to 132x, while achieving better video quality. With such a representation,
treat videos as neural networks, simplifying several video- related tasks. For example. conventional video compression methods are restricted by long and complex pipeline, specifically designed for the task. contrast, with NeRV, we can use any neural network compression method as a proxy for video compression, and achieve comparable performance video compression approaches (H.264, HEVC etc.). Besides compression, we demonstrate the generalization NeRV for video denoising. The source code and pre-trained model can be found at
traditional frame-based
Introduction
What is video? Typically, video captures dynamic visual scene using sequence of frames. A schematic interpretation this is a curve in 2D space, where each point can be characterized with a (x, v) pair representing the spatial state. If we have model for all (Ir, y) pairs, then, given any can easily find the corresponding y state. Similarly, we can interpret a video as a recording of the visual world, where can find corresponding RGB state for every single timestamp. This leads to our main claim: can we represent a video as a function of time?
More formally, can represent video V as V {rr)T= where v fo(t), i.e., frame at times- tamp f, represented as function f parameterized by 6. Given their remarkable representational capacity [1]. we choose deep neural networks as the function in our work. Given these intuitions, we propose NeRV, novel representation that represents videos as implicit functions and encodes them into neural networks. Specifically, with fairly simple deep neural network design, NeRV can reconstruct the corresponding video frames with high quality, given the frame index. Once video is encoded into neural network, this network can be used as a proxy for video, where directly extract all video information from the representation. Therefore, unlike traditional video representations which treat videos sequences of frames, shown in Figure 1 (a), proposed NeRV considers a video as a unified neural network with all information embedded within its architecture and parameters, shown in Figure 1 (b).
As an image-win -isee implicit representation, NeRV shares lots of similarities with pixel-wise implicit visual representations [5, 6] which takes spatial-temporal coordinates as inputs. The main differences between our work and image-wise implicit representation are the output space and architecture
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Vidco
Vidco
ft
fo
Network layers
(a) Explicit representations for videos (e.g.. IIEVC)
(b) Neural implicit representations for videos (e.g., NeRV)
Figure I: (a) Conventional video representation as frame sequences. (b) NeRV, representing vidco as neural networks, which consists of multiple convolutional layers, taking the normalized frame index as the input and output the corresponding RGB frame.
Table 1:
Comparison of different video representations. Although explicit representations outperform implicit ones in encoding speed and compression ratio now, NeRV shows great advantage in decoding speed. And NeRV outperforms pixel-wise implicit representations in all metrics.
Explicit (frame-based)
Implicit (unified)
Learning- -ased (e.g.. HEVC (2]) (e.g., DVC (3)
Hand-crafted
Pixel-wise
Image-wise (Ours)
(e.g.. NeRF [4]
Encoding speed Decoding speed
Fast
Medium
Very slow
Slow
Medium
Slow
Very slow
Fast
Compression ratio
Medium
High
Low
Medium
designs. Pixel-wise representations output the RGB value for each pixel, while NeRV outputs a whole image, demonstrated in Figure 2. Given a video with size of x x W, pixel-wise representations need to sample the video T H W times while NeRV only need sample T times. Considering the huge pixel number, especially for high resolution videos, NeRV shows great advantage for both encoding time and decoding speed. Different output space also leads to different architecture designs, NeRV utilizes a MLP + ConvNets architecture to output an image while pixel-wise representation uses simple MLP output the RGB value of the pixel. Sampling efficiency of NeRV also simplify the optimization problem, which leads to better reconstruction quality compared to pixel-wise representations.
We also demonstrate the flexibility of NeRV by exploring several applications it affords. Most notably, examine the suitability of NeRV for video compression. Traditional video compression frameworks are quite involved, such as specifying key frames and inter frames, estimating the residual information, block-size the video frames, applying discrete cosine transform the resulting image blocks and so on. Such a long pipeline makes the decoding process very complex as well. In contrast, given neural network that encodes video in NeRV, can simply cast the video compression task as a model compression problem, and trivially leverage any well-established or cutting edge model compression algorithm to achieve good compression ratios. Specifically, model compression pipeline: model pruning, model quantization, and weight encoding, and show the contributions of each step for the compression task. We conduct extensive experiments on popular video compression datasets, such as UVG [7], and show the applicability of model compression techniques on NeRV for video compression. We briefly compare different video representations in Table and NeRV shows great advantage in decoding speed.
explore three-step
Besides video compression, also explore other applications of the NeRV representation for the video denoising task. Since NeRV a learnt implicit function, we can demonstrate its robustness to noise and perturbations. Given noisy video input, NeRV generates high-quality denoised output, without any additional operation, and even outperforms conventional denoising methods.
The contribution of this paper can be summarized into four parts:

---

We propose NeRV, a novel image-wist wise implicit representation for videos, representating a video as neural network, converting video encoding model fitting and video decoding simple feedforward operation.
Compared to pixel- wise implicit representation. NeRV output the whole image and shows great efficiency, improving the encoding speed by x5x to 70x, the decoding speed by 38x to 132x, while achieving better video quality.
NeRV allows us to convert the video compression problem to a model compression problem, allowing us leverage standard model compression tools and reach comparable performance with conventional video compression methods, e.g., H.264 [8], and HEVC [2].
As a general representation for videos, NeRV also shows promising results in other tasks, e.g., video denoising. Without any special denoisng design, NeRV outperforms traditional hand-crafted denoising algorithms (medium filter etc.) and ConyNets-based denoisng methods.
2
Related Work
Implicit Neural Representation. Implicit neural representation is a novel way to parameterize a variety of signals. The key idea is to represent object function approximated via neural network, which maps the coordinate to its corresponding value (e.g., .. pixel coordinate for an image and RGB value of the pixel). It has been widely applied in many 3D vision tasks, such as 3D shapes [9, 10], 3D scenes [11-14], and appearance of the 3D structure [4, 15, 16]. Comparing to explicit 3D representations, such voxel, point cloud, and mesh, the continuous implicit neural representation can compactly encode high-resolution signals in a memory seemor--eficieent way. Most recently, [17] demonstrated the feasibility of using implicit neural representation for image compression tasks. Although it is not yet competitive with the state-of-the-art compression methods, it shows promising and attractive proprieties. In previous methods, MLPs are often used to approximate the implicit coordinate as the input and output the neural representations, which take the spatial or density). In contrast, our NeRV representation, signals that single point (e.g., RGB value, volume trains a purposefully designed neural network composed of MLPs and convolution layers, and takes the frame index as input and directly outputs all the RGB values of that frame.
Video Compression. As a fundamental task of computer vision and image processing, visual data compression has been studied for several decades. Before the resurgence of deep networks, handcrafted image compression techniques, like JPEG [18] and JPEG2000 [19], were widely used. Building upon them, many traditional video compression algorithms, such as MPEG [20], H.264 [8], and HEVC [2], have achieved great success. These methods are generally based on transform coding like Discrete Cosine Transform (DCT) [21] or wavelet transform [22]. which are well-engineered and tuned to be fast and efficient. More recently, deep lcarning- based visual compression approaches have been gaining popularity. For video compression, the most common practice is to utilize neural networks for certain components while using the traditional video compression pipeline. For example, [23] proposed
effective image compression approach and generalized it into video compression by adding interpolation loop modules. Similarly, [24] converted the video compression problem into an image interpolation problem and proposed an interpolation network, resulting in competitive compression quality. Furthermore, [25] generalized optical flow to scale-space flow to better model uncertainty in compression. Later, [26] employed temporal hierarchical structure, and trained neural networks for most components including key frame compression, motion estimation, motions compression, and residual compression. However, all of these works still follow the overall pipeline of traditional compression, arguably limiting their capabilities.
Model Compression. The goal of model compression is to simplify an original model by reducing the number of parameters while maintaining its accuracy. Current research on model compression research can be divided into four groups: parameter pruning and quantization [27-32]; low-rank factorization [33-35]; transferred and compact convolutional filters [36-39]; and knowledge distilla- tion [40-43]. Our proposed NeRV enables us to reformulate the video compression problem into model compression, and utilize standard model compression techniques. Specifically, we use model pruning and quantization to reduce the model size without significantly deteriorating the performance.
3

---

(R,0,
Ceordinati: (
Enboadiigg
Ourput
Commotann
(C*
(a) Pixel-wise implicil representation (e.g. SIREN)
Adtivation Layer
Froms
ODO
(C, (0,5
OuIDut
(b) NeRV: Imago-wise implicit representation (ours)
NeRV block
Figure 2: (a) Pixel-wise implicit representation taking pixel coordinates as input and use simple MLP to output pixel RGB value (b) NeRV: Image- wise implicit representation taking frame index imput and use MLP + ConvNets output the whole image. (c) NeRV block architccture, upscale the feature map by S here.
Neural Representations for Videos
We first present the NeRV representation in Section 3.1, including the input embedding, the network architecture, and the loss objective. Then, we present model compression techniques on NeRV in Section 3.2 for video compression.
3.1 NeRV Architecture
In NeRV, each video V [ R x3 represented by a function fo R Rlx where the input is frame index t and the output is the corresponding RGB image i E #x The encoding function is parameterized with deep neural network 8, 1t = ft). Therefore, video encoding done by fitting neural network to given video, such that it can map each input timestamp to the corresponding RGB frame.
Input Embedding. Although deep neural networks can be used as universal function approxima- tors [1], directly training the network So with input timestamp results in poor results, which is also observed by [4, 44]. By mapping the inputs high embedding space, the neural network can better variations. Specifically, in NeRV, we use Positional Encoding [4, 6, 45] our embedding function
fit data with
T(t)= (sin (b)T)) cos (bOTl) ,...- sin (()) cos xt))
where and I are hyper- parameters of the networks. Given input timestamp t, normalized between (0, 1], the output of embedding function r(.) then fed to the following neural network.
Network Architecture. NeRV architecture is illustrated in Figure (b). NeRV takes the time embedding as input and outputs the corresponding RGB Frame. Leveraging MLPs directly output all pixel values of the frames can lead to huge parameters, especially when the images resolutions are large. Therefore, we stack multiple NeRV blocks following MI.P layers so that pixels different locations can share convolutional kernels, leading to efficient and effective network. Inspired by the super-resolution networks, we design the NeRV block, illustrated in Figure 2 (c), adopting PixelShuffle technique [46] for upscaling method. Convolution and activation layers are also inserted to enhance the expressibilty. The detailed architecture can found in the supplementary material.
Loss Objective. For NeRV, we adopt combination of LI and SSIM loss as our loss function for network optimization, which calculates the loss over all pixel locations of the predicted image and the ground-trus image as following
1 T Σ
L
(lo vlll + SSIM( v.))
where the frame number, fo(1) E Hx the NeRV prediction, e, E Hxx33 the frame ground truth, is hyper- parameee to balance the weight for each loss component.

---

Video
Model
Model Quantization
Weight Encoding
Overfit
Pruning
Figure 3: NeRV-based video compression pipeline.
3.2 Model Compression
In this section, we briefly revisit model compression techniques used for video compression with NeRV. Our model compression composes of four standard sequential steps: video overfit, model pruning, weight quantization, and weight encoding as shown in Figure 3.
Model Pruning. Given a neural network fit video, use global unstructured pruning to reduce the model size first. Based on the magnitude of weight values, we set weights below a threshold as zero,
10. {0 Os, if 0; 2 0g
0,
otherwise,
where fg is the 9 percentile value for all parameters in A. As a normal practice, we fine-tune the model to regain the representation, after the pruning operation.
Model Quantization. After model pruning, we apply model quantization to network parameters. Note that different from many recent works [31, 47-49] that utilize quantization during training, NeRV only quantized post-hoc (after the training process). Given a parameter tensor j
N4 round
(Li -
P'mn
scale + /mmin= scale
Mmmax
Mmmn
-2hit
2
where 'round' is rounding value to the closest integer, 'bit' the bit length for quantized model, Mmmux and Mmin the max and min value for the parameter tensor 'scale" the scaling factor. Through Equation 4, each parameter can be mapped Mmin can be ignored given the large parameter number of μu, e.g., they account for only 0.005% in small 3 Conv with 64 input channels and 64 output channels (37k parameters in total).
'bit" length value. The overhead store 'scale' and
Entropy Encoding, Finally, we use entropy encoding to further compress the model size. By taking advantage of character frequency, entropy encoding can represent the data with more efficient codec. Specifically, we employ Huffman Coding [50] after model quantization. Since Huffman Coding is lossless, it is guaranteed that decent compression can be achieved without any impact the reconstruction quality. Empirically, this further reduces the model size by around 10%.
Experiments
4.1 Datasets and Implementation Details
We perform experiments on "Big Buck Bunny" sequence from scikit-video to compare NeRV with pixel-wise implicit representations, which has 132 frames 720 x 1080 resolution. To compare with state-of-the-arss methods video compression task, we do experiments on the widely used UVG [7], consisting of 7 videos and 3900 frames with 1920 x 1080 in total.
In our experiments, we train the network using Adam optimizer [51] with learning rate of 5e-4. For ablation study on UVG, we use cosine annealing learning rate schedule [52], batchsize of 1, training epochs of 1 150, and warmup epochs of 30 unless otherwise denoted. When compare with state-of-the-arts, we run the model for 1500 epochs, with batchsize 6. For experiments on "Big Buck Bunny", we train NeRV for 1200 epochs unless otherwise denoted. For fine-tune process after pruning. we use 50 epochs for both UVG and "Big Buck Bunny".
For NeRV architecture, there are 5 NeRV blocks, with up-scale factor 5, 3, 2, 2, 2 respectively for 1080p videos, and 5, 2, 2, 2, 2 respectively for 720p videos. By changing the hidden dimension of MLP and channel dimension of NeRV blocks, we can build NeRV model with different sizes. For input embedding
Equation 1, we use b 1.25 and objective in Equation 2, a is set to 0.7. We evaluate the video quality with two metrics: PSNR and MS-SSIM [53]. pirs-eer- (BPP) is adopted to indicate the compression ratio. We implement our model in PyTorch [54] and train it in full precision (FP32). All experiments are run with NVIDIA RTX2080ti. Please refer the supplementary material for more experimental details, results, and visualizations (e.g., MCL-JCV [55] results)
80 as our default setting. For loss
5

---

Table 2: Compare with pixel-wise implicit representations. Training
Table 3: PSNR vs. epochs. Since vidco encoding NeRV an over- fit process, the reconstructed video quality keeps increasing with more training epochs. NeRV-S/M/L.
speed means time/epoch, while encoding time the total training time.
Training Speed
Encoding Time
Decoding FPS
Methods
Purameters
PSNR
SIREN [5]
3.2M
1x
2.5x
31.39
1.4
NeRF [4]
3.2M
1x
2.5x
33.31
1.4
models with different sizes.
NeRV-S (ours)
3.2M
25x
1x
34.21
54,5
Epoch
NeRV-S
NeRV-M
NeRV-L
SIREN [5]
6,4M
1x
5x
31.37
0.8
NeRF [4]
6,4M
1x
5x
35.17
0.8
300
32.21
36,05
39.75
NcRV-M
6.3M
50x
Ix
38.14
53.8
600
33.56
37,47
40.84
1.2k
34.21
38.14
41.29
SIREN [S]
12.7M
1x
7x
25.06
0,4
18k
34.33
38.32
41.68
NeRF [4]
12.7M
1x
7X
37.94
0.4
2.4k
34,86
38.7
41.99
NeRV-L (ours)
12.5M
70
1x
41.29
52.9
40
40
335
PSNR
35
PSNR
38
30
NeRV-5
Original
Pruned
30
+
NeRV-S
NeRV-M
Quantized
NeRV-M
25
NeRVIL
NeRV-L
0.4
4
8
DR
16
32
0.1
Bits per pixel (8PP)
0.3
Sparsity
Figure 6: Compression pipeline to show how much cach step con- tribute compression ratio.
Figure 5: Model quantization. Bit
Figure 4: Model pruning. Sparsity is the ratio of parameters pruned.
is the bit length used to represent pa- rameter value.
4.2 Main Results
We compare NeRV with pixel-wise implicit representations on 'Big Buck Bunny" video. We take SIREN [5] and NeRF [4] as the baseline, where SIREN [5] takes the original pixel coordinates as input and uses sine activations, while NeRF [4] adds one positional embedding layer encode the pixel coordinates and uses ReLU activations. Both SIREN and FFN use a 3-layer perceptron and we change the hidden dimension to build model of diffferent sizes. For fair comparison, we train SIREN and FFN for 120 epochs to make encoding time comparable. And we change the filter width build NeRV model of comparable sizes, named as NeRV-S. NeRV-M, and NeRV-L.. In Table NeRV outperforms them greatly in both encoding speed, decoding quality, and decoding speed. Note that NeRV can improve the training speed by 25x 70x, and speedup the decoding FPS by 38x to 132x. We also conduct experiments with different training epochs in Table 3, which clearly shows that longer training time can lead to much better overfit results the video and we notice that the final performances have not saturated as long as it trains for more epochs.
4.3 Video Compression
Compression ablation. We first conduct ablation study on video "Big Buck Bunny". Figure shows the results of different pruning ratios, where model of 40% sparsity still reach comparable performance with the full model. As for model quantization step in Figure 5, 8-bit model still remains the video quality compared to the original one (32-bit). Figure 6 shows the full compression pipeline with NeRV. The compression performance quite robust to NeRV models of different sizes, and each step shows consistent contribution to our final results. Please note that we only explore these three common compression techniques here, and we believe that other well-established and cutting edge model compression algorithm can be applied to further improve the final performances of video compression task, which left for future research.
Compare with state-of-the-arts methods. We then compare with state-of -ie---ffthhhrrree methods on UVG dataset. First, we concatenate videos into one single video along the time dimension and train NeRV on all the frames from different videos, which we found to be more beneficial than training single model for each video. After training the network, we apply model pruning, quantization, and weight encoding as described in Section3.2. Figure 7 and Figure 8 show the rate-distortion curves. We compare with H.264 [8], HEVC [2], STAT-SSF-SP [56], HLVC [26], Scale-space [25], and Wu al. [24]. H.264 and HEVC DEVCam performed with medium preset mode. As the first image- wise neural

---

0.98
0.96
36
PSNR
H.264 HEVC
MS (Slss
0.94
H.264
34
HLVC
0.92
HEVC
STAT-SSF-SP Scale-space
HLVC
Scale-space
et al.
0.88
Wu et al.
NeRV (Ours)
NeRV (Ours)
30
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.86
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Bits per pixel (BPP)
Bits per pixel (BPP)
Figure 7: PSNR vy. BPP on UVG dataset.
Figure 8: MS-SSIM vs. BPP on UVG dataset.
(a) Ground Truth Frame
(b) NeRV Decoded Frame (0.077 BPP)
(c) HEVC Decoded Frame (0.075 BPP)
Figure 9; Vidco compression visualization. At similar BPP, NeRV reconstructs videos with better details.
representation, NeRV generally achieves comparable performance with traditional video compression techniques and other learning- based video compression approaches. is worth noting that when BPP small, NeRV can match the performance of the state-of oftt--of-the-an method, showing its great potential in high-rate video compression. When BPP becomes large, the performance gap is mostly because of the lack of full training due to GPU resources limitations. As shown Table 3, the decoding video quality keeps increasing when the training epochs are longer. Figure shows visualizations for decoding frames. At similar memory budget, NeRV shows image details with better quality.
Decoding time We compare with other methods for decoding time under similar memory budget. Note that HEVC is run on CPU, while all other learning-basse methods are run on a single GPU, including our NeRV. We speedup NeRV by running it in half precision (FP16). Due to the simple decoding process (feedforward operation), NeRV shows great advantage, even for H.264. And lots of speepup can be expected by running quantizaed model special hardware. All the other video compression methods have two types of frames: key and interval frames. Key frame be reconstructed by its encoded feature only while the interval frame reconstruction is also based on the reconstructed key frames. Since most video frames are interval frames, their decoding needs to be done in sequential manner after the reconstruction of the respective key frames. On the contrary, our NeRV can output frames at any random time index independently. thus making parallel decoding much simpler. This can be viewed distinct advantage over other methods.
4.4
Video Denoising
We apply several common noise patterns on the original video and train the model on the perturbed ones. During training, no masks noise locations are provided to the model, i.e., the target of the model the noisy frames while the model has no extra signal whether the input noisy or not. Surprisingly, model tries to avoid the influence of the noise and regularizes them implicitly with
7

---

Table 4: Decoding speed with BPP 0.2 1080p videos
Table 5: PSNR results for video denoising. "baseline" refers the noisy frames before any denoising
Methods
FPS
noise
white f
black t
pepper t
salt &
random ?
Average
Habibian ct al. [14]
10-3.7 3.7
Wu et al. [24]
10-3
Baseline
27.85
28.29
27.95
30.95
28.74
Rippel et al. [57]
Gaussian
30.27
30.14
30.23
30.99
30.41
DVC [3]
1.8
Uniform
29.11
29.06
29.10
29.63
29.22
Liu et al [58]
3
Median
33.89
33.84
33.87
33.89
33.87
H.264 [8]
9.2
Minimum
20.55
16.60
18.09
18.20
18.36
NeRV (FP32)
5.6
Maximum
16.16
20.26
17.69
17.83
17.99
NeRV (FP16)
12.5
NeRV
33.31
34.20
34.17
34.80
34.12
(b) Noisy Frame
(c) DIP Output (30.13 PSNRY-Iteration 1800
(a) Ground Truth Frame
(d) NeRV Output (39.27 PSNR)
(e) Outpat (32.30 IteRa-tteration
Denoising visualization. (c) and (e) denoising output for DIP [59]. Data generalization of NeRV leads robust and better denoising performance since all frames share the same representation, while DIP model overfits one model to one image only.
Figure 10:
little harm to the compression task simultaneously, which can serve well for most partially distorted videos in practice.
The results are compared with some standard denoising methods including Gaussian, uniform, and median filtering. These can be viewed as denoising upper bound for any additional compression process. As listed in Table the PSNR NeRV output usually much higher than the noisy frames although it's trained on the noisy target in fully supervised manner, and has reached an acceptable level for general denoising purpose. Specifically, median filtering has the best performance among the traditional denoising techniques, while NeRV outperforms it in most cases or is at least comparable without any extra denoising design in both architecture design and training strategy.
We also compare NeRV with another neural-network-batee based denoising method, Deep Image Prior (DIP) [59]. Although its main target is image denoising, NeRV outperforms in both qualitative and quantitative metrics, demonstrated in Figure 10. The main difference between them is that denoising of DIP only comes form architecture prior, while the denoising ability of NeRV comes from both architecture prior and data prior. DIP emphasizes that its image prior is only captured by the network structure of Convolution operations because only feeds on a single image. But the training data of NeRV contain many video frames, sharing lots of visual contents and consistences. As result, image prior captured by both the network structure and the training data statistics for NeRV. DIP relies significantly on good early stopping strategy to prevent it from overfitting to the noise. Without noise prior, has be used with fixed iterations settings, which is not easy to generalize to any random kind of noises as mentioned above. By contrast, NeRV is able to handle this naturally by keeping training because the full set of consecutive video frames provides strong regularization on image content over noisc.

---

Table 6: Input embedding ablation. PE means positional encoding
Table 7: Upscale layer ablation
Table 8: Norm layer ablation
PSNR
MS-SSIM
PSNR
MS-SSIM
PSNR
MS-SSIM
Bilinear Pooling
29.56
0.873
BatchNorm
36.71
0.971
None
24.93
0.769
Transpose Conv
36.63
0.967
InstanceNorm
35.5
0.963
PE
37.26
0.970
PixclShuffle
37.26
0,970
None
37.26
0.970
Table 9; Activation function ablation
Table 10: Loss objective ablation
PSNR
MS-SSIM
L1
SSIM
PSNR
MS-SSIM
35.64
0.956
ReLU
35.89
0.963
35.77
0.959
Leaky ReLU
36.76
0.968
35.69
0.971
Swish
37.08
0.969
35.95
0.960
GELU
37.26
0,970
37.26
0.970
4.5
Ablation Studies
Finally, we provide ablation studies on the UVG dataset. PSNR and MS-SSIM are adopted for evaluation of the reconstructed videos.
Input embedding. In Table 6, PE means positional encoding as in Equation 1, which greatly improves the baseline, None means taking the frame index as input directly. Similar findings can be found in [4], without any input embedding, the model can not learn high-frequency information, resulting in much lower performance.
Upscale layer. In Table 7, we show results of three different upscale methods. i.e., Bilinear Pooling, Transpose Convolution, and PixelShuffle [46]. With similar model sizes, PixelShuffle shows best results. Please note that although Transpose convolution [60] reach comparable results, it greatly slowdown the training speed compared to the PixelShuffle.
Normalization layer. In Table 8, apply common normalization layers in NeRV block. The default setup, without normalization layer, reaches the best performance and runs slightly faster. We hypothesize that the normalization layer reduces the over-fitting capability of the neural network, which is contradictory to our training objective.
Activation layer. Table 9 shows results for common activation layers. The GELU [61] activation function achieve the highest performances, which is adopted as our default design. Loss objective. We show loss objective ablation in Table 10. We shows performance results of different combinations of L2, LI, and SSIM loss. Although adopting SSIM alone can produce the highest MS SSIM score, but the combination of L1 loss and SSIM loss can achieve the best trade-off between the PSNR performance and MS-SSIM score.
5 Discussion
Conclusion. In this work, we present novel neural representation for videos, NeRV, which encodes videos into neural networks. Our key sight is that by directly training a neural network with video frame index and output correspondigg RGB image, we can the weights of the model to represent the videos, which is totally different from conventional representations that treat videos as consecutive frame sequences. With such a representation, model compression techniques, NeRV can match the performances of traditional video compression approaches for the video compression task, without the need design long and complex pipeline. We also show that NeRV can outperform standard denoising methods. We hope that this paper can inspire further research works to design novel class of methods for video representations.
show that by simply applying general
Limitations and Future Work. There are some limitations with the proposed NeRV. First, to achieve the comparable PSNR and MS-SSIM performances, the training time of our proposed approach is longer than the encoding time of traditional video compression methods. Second, the architecture design of NeRV is still not optimal yet, believe more exploration the neural architecture design can achieve higher performances. Finally, more advanced and cutting the edge model compression methods can be applied to NeRV and obtain higher compression ratios.
Acknowledgement. This project was partially funded by the DARPA SAIL-ON (W91 INF2020009) program, independent grant from Facebook AI, and Amazon Research Award to AS.

---

References
[1] Kurt Homnik, Maxwell Stincheombe, and Halbert White. Multilayer feedforward networks universal approximators. Neural networks, 2(5):359-366, 1989. 1,
[2] Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding standaad,,
standard. IEEE Transactions on circuits and systems for video technology. 22(11)) 1668, 2012. 2, 3,6
[3] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chumlei Cai, and Zhiyong Gao. Dve: An end-to-end deep vidco compression framework. In CVPR, 2019. 2,8
[4] Ben Mildenhall. Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405-421. Springer, 2020. 2, 3, 4, 6,9
[5] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020. 1,6
[6] Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Ken Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurlPS, 2020. 1,4
[7] Alexandre Mercat, Marko Viitanen, and Jarno Vanne. Uvg dataset: 50/120fps 4k sequences for video codec analysis and development. In Proceedings of the 11th ACM Multimedia Systems Conference, pages 297-302, 2020. 2,5
[8] Thomas Wiegand, Gary Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview the 264/ave video coding standard. IEEE Transactions on circuits and systems for video technology, 13(7):560-5)) 576, 2003. 3, 6,
[9] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Samn,, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7154-7164, 2019. 3
[10] Kyle Genova, Forrester Cole, Avncesh Sud, Aaron Sama,, and Thomas A Funkhouser. Deep structured implicit functions. 2019.
[11] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-struuctuee-aware neural scene representations. arXiv preprint arXiv: veXiv:/906.0f/ 1906.01618, 2019. 3
[12] Chiyu Jiang, Avneesh Sud, Amecsh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6001-600 1001-001,,20 2020.
[13] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. arXiv preprint arXiv:2003.046//8 2, 2020.
[14] Rohan Chabra, Jan E Lenssen. Eddy Ilg. Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe, Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In European Conference on Computer Vision, pages 608-625. Springer, 2020. 3.8
[15] Michacl Niemeyer, Lars Mescheder, Michael Occhslc, and Andreas Geiger. Diffcrentiable volumetric rendering: Leamingg implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504-3515, 2020. 3
[16] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: lecrming texture representations in function space. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 4531-4540, 2019. 3
[17] Emilien Dupont, Adam Goliáski, Milad Alizadeh, Yee whye Teh, and Arnad Doucet, Coin: Compression with implicit neural representations. arXiv preprint arXiv: 103.03/0 2021. 3
[18] Gregory K Wallace. The jpeg still picture compression standard. IEEE transactions on consumer
electronics, xviixxxvi,, 1992. 3
[19] Athanassios Skodras, Charilaos Christopoulos, and Touradj Ebrahimi. The jpeg 2000 still image compres- sion standard. IEEE Signal processing magazine, 18(5):36 58, 2001. 3
10

---

[20] Didier Le Gall. Mpeg: A video compression standard for multimedia applications. Communications of the ACM, 34(4):46-58, 1991. 3
[21] Nasir Ahmed, T_ Natarajan, and Kamisetty R Rao. Discrete cosine transform. IEEE transactions on Computers. 100(1):90 93, 1974. 3
[22] Mare Antonini, Michel Barlaud, Pierre Mathieu, and Ingrid Daubechies, Image coding using wavelet transform. IEEE Transactions on image processing, 12):2005-22, 1992. 3
[23] Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learning image and video compression through spatial-temporal energy compaction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10071-10080. 2019.
[24] Chao- Wu, Nayan Singhal, and Philipp Krahenbuhl. Video compression through image interpolation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 416-431, 2018. 3, 6, 8
[25] Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8503-8512, 2020. 3,
[26] Ren Yang, Fabian Mentzer, Luc Van Gool, and Radu Timofte. Learning for video compression with hierarchical quality and recurrent enhancement. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pages 6628-6637, 2020. 3, 6
[27] Vincent Vanhoucke, Andrew Senior, and Mark Z Mao. Improving the speed of neural networks on epus. 2011. 3
[28] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International conference machine learning, pages 1737-1746. PMLR, 2015.
[29] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 10.00149, 10.0049,, 2015.
[30] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. arXiv preprint arXiv: (al.:::6000066666 2016.
[31] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2704-2713, 2018.
[32] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper.
arXiv preprint arXiv: 1806.0834, 2018. 3
[33] Roberto Rigamonti, Amos Sironi, Vincent Lepetit, and Pascal Fua. Learning separable filters. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2754-2761, 2013.
[34] Emily Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. arXiv preprint arXiv: arXiv:1404.077 1404.0736, 2014.
[35] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv: 400_38666, 2014. 3
[36] Taco Cohen and Max Welling. Group equivariant convolutional networks. International conference on machine learning, pages 2990-2999, PMLR, 2016. 3
[37] Shuangfei Zhai, Yu Cheng. Weining Lu, and Zhongfei Zhang. Doubly convolutional neural networks. arXiv preprint arXiv: 1610,0971 16, 2016.
[38] Wenling Shang. Kihyuk Sohn, Diogo Almeida, and Honglak Lec. Understanding and improving convo- lutional neural networks via concatenated rectified lincar units. In international conference on machine learning, pages 2217-2225. PMLR, 2016.
[39] Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symmetry in convolutional neural networks. In International conference on machine learning, pages 1889-1898. PMLR, 2016.
[40] Lei Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? arXiv preprint arXiv: 1312.6184. 2013. 3
11

---

[41] Geoffrey Hinton. Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini wwXiv:1503.0253:3 2015.
[42] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Leaming efficient object detection models with knowledge distillation. In Proceedings of the 31 International Conference on Newral Information Processing Systems, pages 742-751, 2017.
[43] Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization.
arXiv preprint arXiv:1802. 05668. 2018. 3
[44] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville, On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019. 4
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: 03762, 2017, 4
[46] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874-1883, 2016. 4,
[47] Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. arXiv preprint arXiv: 10066, 2018.
[48] Fartash Faghri, Iman Tabrizian, Ila Markov, Dan Alistarh, Daniel Roy, and Ali Ramezani Senezani-Keebya, Adaptive gradient quantization for data-parallel sgd. arXiv preprint arXiv:2010. /2460, 2020.
[49] Naigang Wang, Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. arXiv preprint arXiv: 1812.0801.. 2018. 5
[50] David Huffman. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 109 1952.
[51] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69980 2014. 5
[52] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint 16Xiv:1608.0 03983, 2016. 5
[53] Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pages 1398-1402. Ieee, 2003. 5
[54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- tegh-performacc deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 0024-8035. Curran Associates, Inc., 2019. 5
[55] Haiqiang Wang. Weihao Gan, Sudeng Hu, Joe Yuchich Lin, Lina Jin, Longguang Song. Ping Wang, loannis Katsavounidis, Anne Aaron, and C-C Jay Kuo. Mcl-jev: a jnd-based h. 264/ave video quality assessment dataset. In 2016 IEEE International Conference on Image Processing (ICIP), pages 1509-1512 IEEE, 2016. 5
[56] Ruihan Yang, Yibo Yang, Joseph Marino, and Stephan Mandt. Hierarchical autoregressive modeling for neural video compression. arXiv preprint arXiv:2010.10258 2020. 6
[57] Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander G Anderson, and Lubomir Bourdev. Learned video compression. In ICCV, 2019. 8
[58] Jerry Liu, Shenlong Wang, Wei-Chiu Ma, Meet Shah, Rui Hu, Pranaab Dhawan, and Raquel Urtasun. Conditional entropy coding for efficient video compression. In ECCV, 2020. 8
[59] Dmitry Ulyanvv, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 9446-9454, 2018. 8
the
[60] Vincent Dumoulin and Francesco Visin. A guide to convolution arithmetic for deep learning. arXiv preprint McXiv:1603007288 2016. 9
[61] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1804.1500..8415 2016. 9
12


Reviews:
[{"review_id": "umEckUSHQKR", "reviewer": "Reviewer_sHtY", "summary": "This paper proposes  a novel neural representation for videos. The key idea is to represent a video as a function conditioned on the time index input. Given a video, it uses a MLP to fit such a function. Two applications are conducted: video compression and video denoising. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I cannot see any strengths from this paper. My concerns include:\n1) The key idea, to use time index as an condition input is really a common-used strategy in many existing works, like video generation. It lacks insights. \n2) Compared with traditional video compression methods, the proposed algorithm requires a video-specific training and a model-pruning post-processing, which needs 3-4 days. This is also stated by the authors in the limitation. In this sense, it lacks applicability. \n3) For the experiment on video denoising, training a model to fit a noisy video can finally produce denoised video. This is actually the key insight of \"deep image prior\" and has no relation to the proposed idea. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation about the time-consuming is very difficult to handle, given the proposed key idea. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "NLaYjoV2lv", "reviewer": "Reviewer_mzHf", "summary": "Video is encoded as a neural network that takes frame index and outputs the video frame corresponding to the index. Video compression problem is cast as a model compression problem, and the proposed neural representation achieves performance comparable to standard frame-based compression method. The proposed neural representation is demonstrated for temporal interpolation and video denoising tasks. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(+) The paper is relatively well organized and easy to follow. \n(+) The paper also introduces an interesting idea for representing a video but the benefit/significance is not very clear.\n\n(-) The task mentioned in the paper seems very similar to the copy task introduced in the paper Neural Turing Machine. It would be interesting to compare how the proposed network would perform in the copy task. There is an obvious difference in input and output between the proposed architecture and the NTM. Is it possible to have a neural architecture that is not dedicated to a video but a set of videos would be an interesting question.\nGraves, Alex; Wayne, Greg; Danihelka, Ivo (2014). \"Neural Turing Machines\". arXiv:1410.5401.\n(-) The paper does not discuss the philosophy behind architecture. It is difficult for the reader to understand why the network was designed the way it was. For example, there is very little explanation about why the input is encoded as a Fourier feature (is it the only high embedding space to consider) and what the role of pixel shuffle is in the architecture.\n(-) Conventional Model compression described in Section 3.1 seems as complex as conventional video compression methods (H264, HEVC). How is the audio of the video encoded?  It would seem that the compression performance will depend on many factors (weight quantization and entropy encoding) which are not fully discussed.\n(-) The proposed algorithm does not achieve SOTA and does not present a clear benefit for representing the video in the manner the paper is proposing. \n(-) Coordinate-based neural representation is not a novel idea and applying it to video does not seem to give any benefits in terms of performance. \n(-) There are various components of the architecture which the authors have compared  (table 4-8) but it is not clear why one would perform better than others. A deeper understanding and intuition would be appreciated. \n[1] Readers will appreciate any insights the authors can provide as to how high-frequency variations of the timestamp provides a better result than just the timestamp. Line 128-129 (.. input to high dimensional space seems ... neural network can better fit...) to be suggesting inputting vector representation of scaler values [0, 0.1,0.2,....1] gives better results. The authors do provide additional experimental results using different values of $b$ and $l$ but it does explain why the result would depend on the different representations of deterministic values. \n[2] The reader would be interested in the author(s)' elaboration on \"burdensome parameters\" which is the motivation of upscaling using pixel-shuffle. \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "HAKPn5D38Qm", "reviewer": "Reviewer_RbcY", "summary": "The paper proposes to model videos as a function mapping time to image, and uses a neural network to model this function. It trains a neural network on one video or a concatenation of several videos, by overfitting the neural network on the videos. Since the motivation is to represent a video in full, the network is not trained for generalization, rather it is trained to overfit. Then, the paper tries to do video compression by compressing the weights of the neural network. It uses 3 model compression techniques in a cumulative fashion : pruning, weight quantization, entropy encoding. It is found that this gets the compressed representation close to standard video compression techniques in terms of PSNR vs BPP (distortion ratio).", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces a simple method for video representation : use the weights of a neural network to represent the video itself. Although there have been other techniques that have tried this, they have been clearly mentioned in Section 2. However, the paper's idea is rather more direct : overfit the weights of a neural network to the video pixels as a function of a Fourier representation of the time index.\n\nThis simple idea seems to lead to a representation that is comparable to standard video compression techniques. Ablation studies is provided to explore different aspects of the neural network: 1) PixelShuffle is better for image upsampling, 2) normalization harms the performance (probably because the goal is to overfit), 3) LeakyReLU is better than ReLU, 4) Fourier embedding of time is better than using the time directly. Although each of these ablations studies could be more extensive, the overall network is already comparable to standard techniques in terms of performance, as shown in Figures 5 and 6.\n\nFigure 4 is an insightful plot into how each of the model compression techniques contributes to the method's performance. The conclusions drawn from this plot are reasonable, the significant impact of each compression technique is clearly illustrated.\n\nApart from video compression, the paper shows the robustness of this method to noisy videos in Table 9. It would be great if it is clarified whether the same noise is added at every batch or whether new noise samples are added per batch. If it is the former, it is indeed impressive that the model is robust to such noise. However, if it is the latter, then it is reasonable that the neural network is learning to ignore the noise for the content.\n\nThere is only 1 qualitative example provided in the supplementary. It would be good if more qualitative samples are provided to illustrate the effectiveness of this method.\n\nAnother clarification is why the PSNR of videos with random noise (0.3) added is quite high (39.07), even higher than denoised videos.\n\nThe paper also makes a case for temporal interpolation. However, only 1 example has been shown (Figure 7). It would be good if more examples are provided.\n\nThe paper does mention some of the above as limitations.\n\nCode has been provided to train the model and reproduce results.\n\n\n\nLine 189 : Table 3*\nLine 220 : Table 3*\nLine 233 : bilin*ear\nLine 261 : fix*, avoiding*", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the method are clearly described at every stage. The major limitation is that although the performance is comparable to standard techniques, the training time is much higher. Moreover, inference time (time taken to translate from time value to pixel) has not been mentioned. This is an important metric to compare with standard techniques.\n\nCode has been provided to train the model and reproduce results.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Dj_IX66kY-h", "reviewer": "Reviewer_4zsT", "summary": "The paper describes a method for representing videos using neural networks. This is inspired by recent advancements in implicit neural representations, which encode various types of signals (audio, images, video, shapes, radiance fields) in the weights of a MLP overfit to the signal. The paper accomplishes this by contributing a new network architecture utilizing convolutional layers which maps each frame index t to the **entire image frame**. Since this network memorizes each frame of the video, compression methods can then be applied on the network instead of the frames themselves to yield a comparable compression versus video quality trade-off. The paper demonstrates that this alternative compression method is comparable to existing standard video compression methods, and that it is robust to noise in the ground truth video.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has a number of strengths and weaknesses. In my opinion, the strengths of the paper are:\n1. **Clarity**. The paper is written very clearly. The method described is concise, and the paper does a good job of explaining why this sort of representation makes sense for video compression, denoising, and frame interpolation. \n2. The quantitative results described are exhaustive, and show that the method does work in memorizing video data using the network architecture described. The rate distortion plots are clear and demonstrate that the method is comparable to existing standard video compression methods, and the tables show that the method has the ability to filter out noise from frames of the video.\n\nIn my opinion, the weaknesses are:\n1. The related work section misses a few very related works. \n    - One major omission is that there is no mention of [SIREN](https://vsitzmann.github.io/siren/) which performs the same experiment of video representation using an implicit neural representation. While I agree that the network architecture used in this case is significantly different (NeRV utilizes convolutional layers to output discrete frames as a function of a continuous frame coordinate, while SIREN also model the pixels in each frame continuously), I think that this work needs to be at the very least mentioned, and ideally compared to.\n    - Another major omission is [Deep Image Prior](https://dmitryulyanov.github.io/deep_image_prior), which shows that denoising on images can be done with no supervision, by only training a CNN to represent the image and using early stopping to prevent overfitting. This is highly related to the fact that NeRV is robust to ground truth video noise, and it would be nice to see an ablation, on perhaps a smaller video, to see if given enough training time, the NeRV representation would still learn to memorize the noisy video. This would provide significantly more understanding about *why* this representation is robust to noisy video. A comparison to this method for denoising could also help quantify the difference between NeRV, and the DIP effect.\n2. My major concern is with the **significance** of the contribution. I do think the **originality** of the novel network architecture and idea of framing video compression as network compression is worth publishing, in the current state of the paper the results seem not quite complete yet:\n    - The limitation of long training time to learn the representation makes this method for video compression not very attractive, especially when considering that quantitatively it only performs on-par with other video compression methods.\n    - It’s unclear how the method scales to shorter or longer videos. Would it even be feasible to compress a video which is hours long? No ablations are provided on this aspect, which is very important for evaluating whether this could make an impact in video compression.\n    - Getting video frame interpolation for free by using a continuous representation seems appealing, but based on the qualitative results shown in the paper, it does not work quite yet. If this worked well, I think this could significantly improve the quality of the paper since it can also be compared to existing frame interpolation methods.\n3. The paper provides limited qualitative comparisons. While seeing the average PSNR or SSIM of the video frames gives some idea of the quality of the estimates, it would make sense to include some sort of qualitative visualization of the result, to make sure there are not artifacts which may induce very small quantitative error, but be very noticeable by human observers. For example, if every other frame was consistently biased away from the GT frame in opposite directions, this could manifest as a very annoying flickering effect in the video which is not caught by the metrics which are used to evaluate quality.\n\nOverall, I like the idea of the paper and think that it has done a good job of making the method simple and clear, and justifying why the modified network architecture and alternative video compression method via representation network compression are valuable and original. However, I think that the significance or impact of the method is limited, since the results are not better than any existing standard compression methods in rate versus distortion, or compression time - the two axes which are most important for video compression. Additionally, the frame interpolation using the continuous representation doesn’t work well enough yet to compare along the frame interpolation axis, and the denoising isn’t studied in depth enough to be sure that NeRV provides novel insight or contribution here. Without these improved results, I am not sure if the ideas of the method alone are ready to be published yet. With improved results, the potential impact of the paper could certainly be improved significantly and make this a much stronger submission in my opinion.\n\nOther comments and questions on the submission:\n- typo: On line 189, should reference Table 3 instead of Table 9.\n- The section on using a single network to represent multiple videos is not described thoroughly, which makes it hard to understand whether or not some kind of generalization / feature re-use is being done here. Is each video represented by a latent code, which is also input into the network? Or is each video fit in a different part of the input domain? This should be clarified, because I think that this generalization aspect could be one way to improve the rate distortion results, and also provide very interesting insight into feature generalization within implicit neural representations. In my opinion, this explanation and emphasis here could make the submission a lot stronger as well.\n- The ablations show that the BatchNorm significantly decreases the quality, but all the experiments are done using a batch size of 1. Does BatchNorm even make sense when the batch size is 1, or does it throw away some information by just normalizing the entire output feature in each layer? Could this be why it does so poorly?\n- minor: The references section is inaccurate. For example, [43] was published at NeurIPS 2020, [11] was published at NeurIPS 2019.\n- It may be worth to consider changing the name of the method in the title, since NeRV already exists as a publication at CVPR 2021: [NeRV](https://pratulsrinivasan.github.io/nerv/).\n- The checklist answers are not completely accurate:\n    - 4(b)-(e) are not correct (no new material was included in the submission, and the content of the data used was not discussed). It’s not an issue to answer no or N/A to these questions if no license could be found, for example...\n    - 3(d) is not included in the supplementary material\n\n**Changes after Author Responses**:\n\nThe response has provided additional comparisons with SIREN architectures, which I think when included in the manuscript should make the contribution of their architecture more impactful. The additional clarifications and comparisons have also cleared up a number of my concerns regarding denoising capabilities, longer videos, and batch size. Thus, I have increased my score. The paper proposes an architecture change, ablates it well, and shows that it gets reasonable results.\n\nHowever, my concerns remain regarding the limitation of long training time and compression rate compared to existing methods, leading to my borderline assessment of the paper. Without either focusing in depth on *why* the architecture change is an immensely impactful shift in coordinate-based network architectures for compression which will lead to countless future improvements, or providing some kind of utility over baseline methods, I think that the significance of the method is limited.\n\n**Additional Changes**:\n\nThe authors have responded to me changing my score by providing more comparisons and an argument related to decoding time being more important than encoding time. I certainly agree with them on this point and appreciate the comparison, and see this as a significant benefit of NeRV. However, shifting the paper to one which tells a story about maximizing a trade-off between compression and decoding time may require a significant amount of additional experiments and ablations (ex: trade-off between PSNR, compression rate, and decoding time). Since there are additional variables here, I don't think I can provide a conditional acceptance in this version of the paper.\n\nAs the current version of the paper stands, I think that performance compared to the baselines is too similar to justify the architecture contribution alone. Without either this, or some insight about the architecture change which shows why this is an impactful shift in coordinate-based network architectures for compression which will lead to future improvement, I think the significance of the contribution is not as high as it could be.\n\nI certainly think that there are the makings of a good paper here with some additional experiments and insight, but in its current state with some minor modifications, I will retain my borderline score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations of their work. They have also addressed the potential negative societal impact of their work. ", "review_text": "", "extra_scores": {"ethical_concerns": "I do not think there are any ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: While there was only one reviewer who recommended accepting this work, this positive reviewer made a strong case for why this paper should be accepted.  While there is some concerns about the practical applicability of the proposed method as a real-world compression method, the reviewers have pointed out that a video need only be encoded once in many situations. As such arguments about the amont of time required to encode a video are less strong.  The issue of not handling audio with this work is not relevant. Video compression research papers rarely also address the audio component of a video. 

The authors provided a solid rebuttal addressing key points raised by reviewers. In particular, the author response and additional comparisons with SIREN architectures should be integrated into the final manuscript. Reviewers believe that this will help make the contribution of the architecture here more impactful. Additional clarifications and comparisons provided during the discussion phase also cleared up a number of reviewer concerns regarding the denoising capabilities, longer videos, and batch size. In the end the most engaged, but still somewhat negative leaning reviewer asserted that "The paper proposes an architecture change, ablates it well, and shows that it gets reasonable results.". 

The positioning of the paper with respect to prior art and contributions was sufficient and by acknowledging the points of the reviewers properly in the final version of this manuscript, I think this work will serve as a good contribution to the conference. The lack of SOTA results is not a reason for rejecting this work. The general idea here is quite interesting and novel enough to warrant acceptance. The idea proposed here is sufficiently interesting and well explored in this work that it would be an interesting contribution to the conference.  This paper is likely to stimulate further interesting discussion at NeurIPS and has potential for further impact.

The AC recommends accepting this paper.

Decision: Accept (Poster)