NeRV: Neural Representations for Videos

1 Introduction . What is a video ? Typically , a video captures a dynamic visual scene using a sequence of frames . A schematic interpretation of this is a curve in 2D space , where each point can be characterized with a ( x , y ) pair representing the spatial state . If we have a model for all ( x , y ) pairs , then , given any x , we can easily find the corresponding y state . Similarly , we can interpret a video as a recording of the visual world , where we can find a corresponding RGB state for every single timestamp . This leads to our main claim : can we represent a video as a function of time ? More formally , can we represent a video V as V = { vt } Tt=1 , where vt = fθ ( t ) , i.e. , a frame at timestamp t , is represented as a function f parameterized by θ . Given their remarkable representational capacity [ 1 ] , we choose deep neural networks as the function in our work . Given these intuitions , we propose NeRV , a novel representation that represents videos as implicit functions and encodes them into neural networks . Specifically , with a fairly simple deep neural network design , NeRV can reconstruct the corresponding video frames with high quality , given the frame index . Once the video is encoded into a neural network , this network can be used as a proxy for video , where we can directly extract all video information from the representation . Therefore , unlike traditional video representations which treat videos as sequences of frames , shown in Figure 1 ( a ) , our proposed NeRV considers a video as a unified neural network with all information embedded within its architecture and parameters , shown in Figure 1 ( b ) . As an image-wise implicit representation , NeRV shares lots of similarities with pixel-wise implicit visual representations [ 5 , 6 ] which takes spatial-temporal coordinates as inputs . The main differences between our work and image-wise implicit representation are the output space and architecture 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . designs . Pixel-wise representations output the RGB value for each pixel , while NeRV outputs a whole image , demonstrated in Figure 2 . Given a video with size of T × H × W , pixel-wise representations need to sample the video T ∗ H ∗W times while NeRV only need to sample T times . Considering the huge pixel number , especially for high resolution videos , NeRV shows great advantage for both encoding time and decoding speed . Different output space also leads to different architecture designs , NeRV utilizes a MLP + ConvNets architecture to output an image while pixel-wise representation uses a simple MLP to output the RGB value of the pixel . Sampling efficiency of NeRV also simplify the optimization problem , which leads to better reconstruction quality compared to pixel-wise representations . We also demonstrate the flexibility of NeRV by exploring several applications it affords . Most notably , we examine the suitability of NeRV for video compression . Traditional video compression frameworks are quite involved , such as specifying key frames and inter frames , estimating the residual information , block-size the video frames , applying discrete cosine transform on the resulting image blocks and so on . Such a long pipeline makes the decoding process very complex as well . In contrast , given a neural network that encodes a video in NeRV , we can simply cast the video compression task as a model compression problem , and trivially leverage any well-established or cutting edge model compression algorithm to achieve good compression ratios . Specifically , we explore a three-step model compression pipeline : model pruning , model quantization , and weight encoding , and show the contributions of each step for the compression task . We conduct extensive experiments on popular video compression datasets , such as UVG [ 7 ] , and show the applicability of model compression techniques on NeRV for video compression . We briefly compare different video representations in Table 1 and NeRV shows great advantage in decoding speed . Besides video compression , we also explore other applications of the NeRV representation for the video denoising task . Since NeRV is a learnt implicit function , we can demonstrate its robustness to noise and perturbations . Given a noisy video as input , NeRV generates a high-quality denoised output , without any additional operation , and even outperforms conventional denoising methods . The contribution of this paper can be summarized into four parts : • We propose NeRV , a novel image-wise implicit representation for videos , representating a video as a neural network , converting video encoding to model fitting and video decoding as a simple feedforward operation . • Compared to pixel-wise implicit representation , NeRV output the whole image and shows great efficiency , improving the encoding speed by 25× to 70× , the decoding speed by 38× to 132× , while achieving better video quality . • NeRV allows us to convert the video compression problem to a model compression problem , allowing us to leverage standard model compression tools and reach comparable performance with conventional video compression methods , e.g. , H.264 [ 8 ] , and HEVC [ 2 ] . • As a general representation for videos , NeRV also shows promising results in other tasks , e.g. , video denoising . Without any special denoisng design , NeRV outperforms traditional hand-crafted denoising algorithms ( medium filter etc . ) and ConvNets-based denoisng methods . 2 Related Work . Implicit Neural Representation . Implicit neural representation is a novel way to parameterize a variety of signals . The key idea is to represent an object as a function approximated via a neural network , which maps the coordinate to its corresponding value ( e.g. , pixel coordinate for an image and RGB value of the pixel ) . It has been widely applied in many 3D vision tasks , such as 3D shapes [ 9 , 10 ] , 3D scenes [ 11–14 ] , and appearance of the 3D structure [ 4 , 15 , 16 ] . Comparing to explicit 3D representations , such as voxel , point cloud , and mesh , the continuous implicit neural representation can compactly encode high-resolution signals in a memory-efficient way . Most recently , [ 17 ] demonstrated the feasibility of using implicit neural representation for image compression tasks . Although it is not yet competitive with the state-of-the-art compression methods , it shows promising and attractive proprieties . In previous methods , MLPs are often used to approximate the implicit neural representations , which take the spatial or spatio-temporal coordinate as the input and output the signals at that single point ( e.g. , RGB value , volume density ) . In contrast , our NeRV representation , trains a purposefully designed neural network composed of MLPs and convolution layers , and takes the frame index as input and directly outputs all the RGB values of that frame . Video Compression . As a fundamental task of computer vision and image processing , visual data compression has been studied for several decades . Before the resurgence of deep networks , handcrafted image compression techniques , like JPEG [ 18 ] and JPEG2000 [ 19 ] , were widely used . Building upon them , many traditional video compression algorithms , such as MPEG [ 20 ] , H.264 [ 8 ] , and HEVC [ 2 ] , have achieved great success . These methods are generally based on transform coding like Discrete Cosine Transform ( DCT ) [ 21 ] or wavelet transform [ 22 ] , which are well-engineered and tuned to be fast and efficient . More recently , deep learning-based visual compression approaches have been gaining popularity . For video compression , the most common practice is to utilize neural networks for certain components while using the traditional video compression pipeline . For example , [ 23 ] proposed an effective image compression approach and generalized it into video compression by adding interpolation loop modules . Similarly , [ 24 ] converted the video compression problem into an image interpolation problem and proposed an interpolation network , resulting in competitive compression quality . Furthermore , [ 25 ] generalized optical flow to scale-space flow to better model uncertainty in compression . Later , [ 26 ] employed a temporal hierarchical structure , and trained neural networks for most components including key frame compression , motion estimation , motions compression , and residual compression . However , all of these works still follow the overall pipeline of traditional compression , arguably limiting their capabilities . Model Compression . The goal of model compression is to simplify an original model by reducing the number of parameters while maintaining its accuracy . Current research on model compression research can be divided into four groups : parameter pruning and quantization [ 27–32 ] ; low-rank factorization [ 33–35 ] ; transferred and compact convolutional filters [ 36–39 ] ; and knowledge distillation [ 40–43 ] . Our proposed NeRV enables us to reformulate the video compression problem into model compression , and utilize standard model compression techniques . Specifically , we use model pruning and quantization to reduce the model size without significantly deteriorating the performance . 3 Neural Representations for Videos . We first present the NeRV representation in Section 3.1 , including the input embedding , the network architecture , and the loss objective . Then , we present model compression techniques on NeRV in Section 3.2 for video compression . 3.1 NeRV Architecture . In NeRV , each video V = { vt } Tt=1 ∈ RT×H×W×3 is represented by a function fθ : R→ RH×W×3 , where the input is a frame index t and the output is the corresponding RGB image vt ∈ RH×W×3 . The encoding function is parameterized with a deep neural network θ , vt = fθ ( t ) . Therefore , video encoding is done by fitting a neural network fθ to a given video , such that it can map each input timestamp to the corresponding RGB frame . Input Embedding . Although deep neural networks can be used as universal function approximators [ 1 ] , directly training the network fθ with input timestamp t results in poor results , which is also observed by [ 4 , 44 ] . By mapping the inputs to a high embedding space , the neural network can better fit data with high-frequency variations . Specifically , in NeRV , we use Positional Encoding [ 4 , 6 , 45 ] as our embedding function Γ ( t ) = ( sin ( b0πt ) , cos ( b0πt ) , . . . , sin ( bl−1πt ) , cos ( bl−1πt ) ) ( 1 ) where b and l are hyper-parameters of the networks . Given an input timestamp t , normalized between ( 0 , 1 ] , the output of embedding function Γ ( · ) is then fed to the following neural network . Network Architecture . NeRV architecture is illustrated in Figure 2 ( b ) . NeRV takes the time embedding as input and outputs the corresponding RGB Frame . Leveraging MLPs to directly output all pixel values of the frames can lead to huge parameters , especially when the images resolutions are large . Therefore , we stack multiple NeRV blocks following the MLP layers so that pixels at different locations can share convolutional kernels , leading to an efficient and effective network . Inspired by the super-resolution networks , we design the NeRV block , illustrated in Figure 2 ( c ) , adopting PixelShuffle technique [ 46 ] for upscaling method . Convolution and activation layers are also inserted to enhance the expressibilty . The detailed architecture can be found in the supplementary material . Loss Objective . For NeRV , we adopt combination of L1 and SSIM loss as our loss function for network optimization , which calculates the loss over all pixel locations of the predicted image and the ground-truth image as following L = 1 T T∑ t=1 α ‖fθ ( t ) − vt‖1 + ( 1− α ) ( 1− SSIM ( fθ ( t ) , vt ) ) ( 2 ) where T is the frame number , fθ ( t ) ∈ RH×W×3 the NeRV prediction , vt ∈ RH×W×3 the frame ground truth , α is hyper-parameter to balance the weight for each loss component .

Summary: The paper describes a method for representing videos using neural networks. This is inspired by recent advancements in implicit neural representations, which encode various types of signals (audio, images, video, shapes, radiance fields) in the weights of a MLP overfit to the signal. The paper accomplishes this by contributing a new network architecture utilizing convolutional layers which maps each frame index t to the **entire image frame**. Since this network memorizes each frame of the video, compression methods can then be applied on the network instead of the frames themselves to yield a comparable compression versus video quality trade-off. The paper demonstrates that this alternative compression method is comparable to existing standard video compression methods, and that it is robust to noise in the ground truth video.