NeRV: Neural Representations for Videos

1 Introduction . What is a video ? Typically , a video captures a dynamic visual scene using a sequence of frames . A schematic interpretation of this is a curve in 2D space , where each point can be characterized with a ( x , y ) pair representing the spatial state . If we have a model for all ( x , y ) pairs , then , given any x , we can easily find the corresponding y state . Similarly , we can interpret a video as a recording of the visual world , where we can find a corresponding RGB state for every single timestamp . This leads to our main claim : can we represent a video as a function of time ? More formally , can we represent a video V as V = { vt } Tt=1 , where vt = fθ ( t ) , i.e. , a frame at timestamp t , is represented as a function f parameterized by θ . Given their remarkable representational capacity [ 1 ] , we choose deep neural networks as the function in our work . Given these intuitions , we propose NeRV , a novel representation that represents videos as implicit functions and encodes them into neural networks . Specifically , with a fairly simple deep neural network design , NeRV can reconstruct the corresponding video frames with high quality , given the frame index . Once the video is encoded into a neural network , this network can be used as a proxy for video , where we can directly extract all video information from the representation . Therefore , unlike traditional video representations which treat videos as sequences of frames , shown in Figure 1 ( a ) , our proposed NeRV considers a video as a unified neural network with all information embedded within its architecture and parameters , shown in Figure 1 ( b ) . As an image-wise implicit representation , NeRV shares lots of similarities with pixel-wise implicit visual representations [ 5 , 6 ] which takes spatial-temporal coordinates as inputs . The main differences between our work and image-wise implicit representation are the output space and architecture 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . designs . Pixel-wise representations output the RGB value for each pixel , while NeRV outputs a whole image , demonstrated in Figure 2 . Given a video with size of T × H × W , pixel-wise representations need to sample the video T ∗ H ∗W times while NeRV only need to sample T times . Considering the huge pixel number , especially for high resolution videos , NeRV shows great advantage for both encoding time and decoding speed . Different output space also leads to different architecture designs , NeRV utilizes a MLP + ConvNets architecture to output an image while pixel-wise representation uses a simple MLP to output the RGB value of the pixel . Sampling efficiency of NeRV also simplify the optimization problem , which leads to better reconstruction quality compared to pixel-wise representations . We also demonstrate the flexibility of NeRV by exploring several applications it affords . Most notably , we examine the suitability of NeRV for video compression . Traditional video compression frameworks are quite involved , such as specifying key frames and inter frames , estimating the residual information , block-size the video frames , applying discrete cosine transform on the resulting image blocks and so on . Such a long pipeline makes the decoding process very complex as well . In contrast , given a neural network that encodes a video in NeRV , we can simply cast the video compression task as a model compression problem , and trivially leverage any well-established or cutting edge model compression algorithm to achieve good compression ratios . Specifically , we explore a three-step model compression pipeline : model pruning , model quantization , and weight encoding , and show the contributions of each step for the compression task . We conduct extensive experiments on popular video compression datasets , such as UVG [ 7 ] , and show the applicability of model compression techniques on NeRV for video compression . We briefly compare different video representations in Table 1 and NeRV shows great advantage in decoding speed . Besides video compression , we also explore other applications of the NeRV representation for the video denoising task . Since NeRV is a learnt implicit function , we can demonstrate its robustness to noise and perturbations . Given a noisy video as input , NeRV generates a high-quality denoised output , without any additional operation , and even outperforms conventional denoising methods . The contribution of this paper can be summarized into four parts : • We propose NeRV , a novel image-wise implicit representation for videos , representating a video as a neural network , converting video encoding to model fitting and video decoding as a simple feedforward operation . • Compared to pixel-wise implicit representation , NeRV output the whole image and shows great efficiency , improving the encoding speed by 25× to 70× , the decoding speed by 38× to 132× , while achieving better video quality . • NeRV allows us to convert the video compression problem to a model compression problem , allowing us to leverage standard model compression tools and reach comparable performance with conventional video compression methods , e.g. , H.264 [ 8 ] , and HEVC [ 2 ] . • As a general representation for videos , NeRV also shows promising results in other tasks , e.g. , video denoising . Without any special denoisng design , NeRV outperforms traditional hand-crafted denoising algorithms ( medium filter etc . ) and ConvNets-based denoisng methods . 2 Related Work . Implicit Neural Representation . Implicit neural representation is a novel way to parameterize a variety of signals . The key idea is to represent an object as a function approximated via a neural network , which maps the coordinate to its corresponding value ( e.g. , pixel coordinate for an image and RGB value of the pixel ) . It has been widely applied in many 3D vision tasks , such as 3D shapes [ 9 , 10 ] , 3D scenes [ 11–14 ] , and appearance of the 3D structure [ 4 , 15 , 16 ] . Comparing to explicit 3D representations , such as voxel , point cloud , and mesh , the continuous implicit neural representation can compactly encode high-resolution signals in a memory-efficient way . Most recently , [ 17 ] demonstrated the feasibility of using implicit neural representation for image compression tasks . Although it is not yet competitive with the state-of-the-art compression methods , it shows promising and attractive proprieties . In previous methods , MLPs are often used to approximate the implicit neural representations , which take the spatial or spatio-temporal coordinate as the input and output the signals at that single point ( e.g. , RGB value , volume density ) . In contrast , our NeRV representation , trains a purposefully designed neural network composed of MLPs and convolution layers , and takes the frame index as input and directly outputs all the RGB values of that frame . Video Compression . As a fundamental task of computer vision and image processing , visual data compression has been studied for several decades . Before the resurgence of deep networks , handcrafted image compression techniques , like JPEG [ 18 ] and JPEG2000 [ 19 ] , were widely used . Building upon them , many traditional video compression algorithms , such as MPEG [ 20 ] , H.264 [ 8 ] , and HEVC [ 2 ] , have achieved great success . These methods are generally based on transform coding like Discrete Cosine Transform ( DCT ) [ 21 ] or wavelet transform [ 22 ] , which are well-engineered and tuned to be fast and efficient . More recently , deep learning-based visual compression approaches have been gaining popularity . For video compression , the most common practice is to utilize neural networks for certain components while using the traditional video compression pipeline . For example , [ 23 ] proposed an effective image compression approach and generalized it into video compression by adding interpolation loop modules . Similarly , [ 24 ] converted the video compression problem into an image interpolation problem and proposed an interpolation network , resulting in competitive compression quality . Furthermore , [ 25 ] generalized optical flow to scale-space flow to better model uncertainty in compression . Later , [ 26 ] employed a temporal hierarchical structure , and trained neural networks for most components including key frame compression , motion estimation , motions compression , and residual compression . However , all of these works still follow the overall pipeline of traditional compression , arguably limiting their capabilities . Model Compression . The goal of model compression is to simplify an original model by reducing the number of parameters while maintaining its accuracy . Current research on model compression research can be divided into four groups : parameter pruning and quantization [ 27–32 ] ; low-rank factorization [ 33–35 ] ; transferred and compact convolutional filters [ 36–39 ] ; and knowledge distillation [ 40–43 ] . Our proposed NeRV enables us to reformulate the video compression problem into model compression , and utilize standard model compression techniques . Specifically , we use model pruning and quantization to reduce the model size without significantly deteriorating the performance . 3 Neural Representations for Videos . We first present the NeRV representation in Section 3.1 , including the input embedding , the network architecture , and the loss objective . Then , we present model compression techniques on NeRV in Section 3.2 for video compression . 3.1 NeRV Architecture . In NeRV , each video V = { vt } Tt=1 ∈ RT×H×W×3 is represented by a function fθ : R→ RH×W×3 , where the input is a frame index t and the output is the corresponding RGB image vt ∈ RH×W×3 . The encoding function is parameterized with a deep neural network θ , vt = fθ ( t ) . Therefore , video encoding is done by fitting a neural network fθ to a given video , such that it can map each input timestamp to the corresponding RGB frame . Input Embedding . Although deep neural networks can be used as universal function approximators [ 1 ] , directly training the network fθ with input timestamp t results in poor results , which is also observed by [ 4 , 44 ] . By mapping the inputs to a high embedding space , the neural network can better fit data with high-frequency variations . Specifically , in NeRV , we use Positional Encoding [ 4 , 6 , 45 ] as our embedding function Γ ( t ) = ( sin ( b0πt ) , cos ( b0πt ) , . . . , sin ( bl−1πt ) , cos ( bl−1πt ) ) ( 1 ) where b and l are hyper-parameters of the networks . Given an input timestamp t , normalized between ( 0 , 1 ] , the output of embedding function Γ ( · ) is then fed to the following neural network . Network Architecture . NeRV architecture is illustrated in Figure 2 ( b ) . NeRV takes the time embedding as input and outputs the corresponding RGB Frame . Leveraging MLPs to directly output all pixel values of the frames can lead to huge parameters , especially when the images resolutions are large . Therefore , we stack multiple NeRV blocks following the MLP layers so that pixels at different locations can share convolutional kernels , leading to an efficient and effective network . Inspired by the super-resolution networks , we design the NeRV block , illustrated in Figure 2 ( c ) , adopting PixelShuffle technique [ 46 ] for upscaling method . Convolution and activation layers are also inserted to enhance the expressibilty . The detailed architecture can be found in the supplementary material . Loss Objective . For NeRV , we adopt combination of L1 and SSIM loss as our loss function for network optimization , which calculates the loss over all pixel locations of the predicted image and the ground-truth image as following L = 1 T T∑ t=1 α ‖fθ ( t ) − vt‖1 + ( 1− α ) ( 1− SSIM ( fθ ( t ) , vt ) ) ( 2 ) where T is the frame number , fθ ( t ) ∈ RH×W×3 the NeRV prediction , vt ∈ RH×W×3 the frame ground truth , α is hyper-parameter to balance the weight for each loss component .

Summary: This paper proposes  a novel neural representation for videos. The key idea is to represent a video as a function conditioned on the time index input. Given a video, it uses a MLP to fit such a function. Two applications are conducted: video compression and video denoising.