3D Scene Compression through Entropy  Penalized Neural Representation Functions

1 INTRODUCTION . The ability to render 3D scenes from arbitrary viewpoints can be seen as a big step in the evolution of digital multimedia , and has applications such as mixed reality media , graphic effects , design , and simulations . Often such renderings are based on a number of high resolution images taken of some original scene , and it is clear that to enable many applications , the data will need to be stored and transmitted efficiently over low-bandwidth channels ( e.g. , to a mobile phone for augmented reality ) . Traditionally , the need to compress this data is viewed as a separate need from rendering . For example , light field images ( LFI ) consist of a set of images taken from multiple viewpoints . To compress the original views , often standard video compression methods such as HEVC ( Sullivan et al. , 2012 ) are repurposed ( Jiang et al. , 2017 ; Barina et al. , 2019 ) . Since the range of views is narrow , light field images can be effectively reconstructed by “ blending ” a smaller set of representative views ( Astola & Tabus , 2018 ; Jiang et al. , 2017 ; Zhao et al. , 2018 ; Bakir et al. , 2018 ; Jia et al. , 2019 ) . Blending based approaches , however , may not be suitable for the more general case of arbitrary-viewpoint 3D scenes , where a very diverse set of original views may increase the severity of occlusions , and thus would require storage of a prohibitively large number of views to be effective . A promising avenue for representing more complete 3D scenes is through neural representation functions , which have shown a remarkable improvement in rendering quality ( Mildenhall et al. , 2020 ; Sitzmann et al. , 2019 ; Liu et al. , 2020 ; Schwarz et al. , 2020 ) . In such approaches , views from a scene are rendered by evaluating the representation function at sampled spatial coordinates and then applying a differentiable rendering process . Such methods are often referred to as implicit representations , since they do not specify explicitly the surface locations and properties within the scene , which would be required to apply some conventional rendering techniques like rasterization ( Akenine-Möller et al. , 2019 ) . However , finding the representation function for a given scene requires training a neural network . This makes this class of methods difficult to use as a rendering method in the existing framework , since it is computationally infeasible on a low-powered end device like a mobile phone , which are often on the receiving side . Due to the data processing inequality , it may also be inefficient to compress the original views ( the training data ) rather than the trained Entropy Poses Views of 3D scene Neural representation function Loss Radiance field Sender Receiver Images Pose 11000010 ... + length ( 11000010 ... ) Reconstruction error Rate Novel rendering Radiance field ? Figure 1 : Overview of cNeRF . The sender trains an entropy penalized neural representation function on a set of views from a scene , minimizing a combination of rate and distortion . The receiver can use the compressed model to render novel views . representation itself , because the training process may discard some information that is ultimately not necessary for rendering ( such as redundancy in the original views , noise , etc. ) . In this work , we propose to apply neural representation functions to the scene compression problem by compressing the representation function itself . We use the NeRF model ( Mildenhall et al. , 2020 ) , a method which has demonstrated the ability to produce high-quality renders of novel views , as our representation function . To reduce redundancy of information in the model , we build upon the model compression approach of Oktay et al . ( 2020 ) , applying an entropy penalty to the set of discrete reparameterized neural network weights . The compressed NeRF ( cNeRF ) describes a radiance field , which is used in conjunction with a differentiable neural renderer to obtain novel views ( see Fig . 1 ) . To verify the proposed method , we construct a strong baseline method based on the approaches seen in the field of light field image compression . cNeRF consistently outperforms the baseline method , producing simultaneously superior renders and lower bitrates . We further show that cNeRF can be improved in the low bitrate regime when compressing multiple scenes at once . To achieve this , we introduce a novel parameterization which shares parameters across models and optimize jointly across scenes . 2 BACKGROUND . We define a multi-view image dataset as a set of tuplesD = { ( Vn , Xn ) } Nn=1 , where Vn is the camera pose and Xn is the corresponding image from this pose . We refer to the 3D ground truth that the views capture as the scene . In what follows , we first provide a brief review of the neural rendering and the model compression approaches that we build upon while introducing the necessary notation . Neural Radiance Fields ( NeRF ) The neural rendering approach of Mildenhall et al . ( 2020 ) uses a neural network to model a radiance field . The radiance field itself is a learned function gθ : R5 → ( R3 , R+ ) , mapping a 3D spatial coordinate and a 2D viewing direction to a RGB value and a corresponding density element . To render a view , the RGB values are sampled along the relevant rays and accumulated according to their density elements . The learned radiance field mapping gθ is parameterized with two multilayer perceptrons ( MLPs ) , which Mildenhall et al . ( 2020 ) refer to as the “ coarse ” and “ fine ” networks , with parameters θc and θf respectively . The input locations to the coarse network are obtained by sampling regularly along the rays , whereas the input locations to the fine network are sampled conditioned on the radiance field of the coarse network . The networks are trained by minimizing the distance from their renderings to the ground truth image : L = N∑ n=1 ∥∥X̂cn ( θc ; Vn ) −Xn∥∥22︸ ︷︷ ︸ Lc ( θc ) + N∑ n=1 ∥∥X̂fn ( θf ; Vn , θc ) −Xn∥∥22︸ ︷︷ ︸ Lf ( θf ; θc ) ( 1 ) Where || · ||2 is the Euclidean norm and the X̂n are the rendered views . Note that the rendered view from the fine network X̂fn relies on both the camera pose Vn and the coarse network to determine the spatial locations to query the radiance field . We drop the explicit dependence of Lf on θc in the rest of the paper to avoid cluttering the notation . During training , we render only a minibatch of pixels rather than the full image . We give a more detailed description of the NeRF model and the rendering process in Appendix Sec . A . Model Compression through Entropy Penalized Reparameterization The model compression work of Oktay et al . ( 2020 ) reparameterizes the model weights Θ into a latent space as Φ . The latent weights are decoded by a learned function F , i.e . Θ = F ( Φ ) . The latent weights Φ are modeled as samples from a learned prior q , such that they can be entropy coded according to this prior . To minimize the rate , i.e . length of the bit string resulting from entropy coding these latent weights , a differentiable approximation of the self-information I ( φ ) = − log2 ( q ( φ ) ) of the latent weights is penalized . The continuous Φ are quantized before being applied in the model , with the straight-through estimator ( Bengio et al. , 2013 ) used to obtain surrogate gradients of the loss function . Following Ballé et al . ( 2017 ) , uniform noise is added when learning the continuous prior q ( φ + u ) where ui ∼ U ( − 12 , 1 2 ) ∀ i . This uniform noise is a stand-in for the quantization , and results in a good approximation for the self-information through the negative log-likelihood of the noised continuous latent weights . After training , the quantized weights Φ̃ are obtained by rounding , Φ̃ = bΦe , and transmitted along with discrete probability tables obtained by integrating the density over the quantization intervals . The continuous weights Φ and any parameters in q itself can then be discarded . 3 METHOD . To achieve a compressed representation of a scene , we propose to compress the neural scene representation function itself . In this paper we use the NeRF model as our representation function . To compress the NeRF model , we build upon the model compression approach of Oktay et al . ( 2020 ) and jointly train for rendering as well as compression in an end-to-end trainable manner . We subsequently refer to this approach as cNeRF . The full objective that we seek to minimize is : L ( Φ , Ψ ) = Lc ( Fc ( Φ̃c ) ) + Lf ( Ff ( Φ̃f ) ) ︸ ︷︷ ︸ Distortion +λ ∑ φ∈Φ I ( φ ) ︸ ︷︷ ︸ Rate ( 2 ) where Ψ denotes the parameters of F as well any parameters in the prior distribution q , and we have explicitly split Φ into the coarse Φc and fine Φf components such that Φ = { Φc , Φf } . λ is a trade-off parameter that balances between rate and distortion . A rate–distortion ( RD ) plot can be traced by varying λ to explore the performance of the compressed model at different bitrates . Compressing a single scene When training cNeRF to render a single scene , we have to choose how to parameterize and structure F and the prior distribution q over the network weights . Since the networks are MLPs , the model parameters for a layer l consist of the kernel weights and biases { Wl , bl } . We compress only the kernel weights Wl , leaving the bias uncompressed since it is much smaller in size . The quantized kernel weights W̃l are mapped to the model weights by Fl , i.e . Wl = Fl ( W̃l ) . Fl is constructed as an affine scalar transformation , which is applied elementwise to W̃l : Fl ( W̃l , ij ) = αlW̃l , ij + βl ( 3 ) We take the prior to be factored over the layers , such that we learn a prior per linear kernel ql . Within each kernel , we take the weights in W̃l to be i.i.d . from the univariate distribution ql , parameterized by a small MLP , as per the approach of Ballé et al . ( 2017 ) . Note that the parameters of this MLP can be discarded after training ( once the probability mass functions have been built ) . Compressing multiple scenes While the original NeRF model is trained for a single scene , we hypothesize that better rate–distortion performance can be achieved for multiple scenes , especially if they share information , by training a joint model . For a dataset of M scenes , we parameterize the kernel weights of model m , layer l as : Wml = Fml ( W̃ml , S̃l ) = αml W̃ m l + β m l + γlS̃l ( 4 ) Compared to Eqn . 3 , we have added a shift , parameterized as a scalar linear transformation of a discrete shift S̃l , that is shared across all models m ∈ { 1 , ... , M } . S̃l has the same dimensions as the kernel Wml , and as with the discrete latent kernels , S̃l is coded by a learned probability distribution . The objective for the multi-scene model becomes : L ( Φ , Ψ ) = M∑ m=1 [ Lmc ( Fmc ( Φ̃mc , Φ̃sc ) ) + Lmf ( Fmf ( Φ̃mf , Φ̃sf ) ) + λ ∑ φ∈Φm I ( φ ) ] + λ ∑ φ∈Φs I ( φ ) ( 5 ) where Φs is the set of all discrete shift S̃ parameters , and the losses , latent weights and affine transforms are indexed by scene and model m. Note that this parameterization has more parameters than the total of the M single scene models , which at first appears counter-intuitive , since we wish to reduce the overall model size . It is constructed as such so that the multi-scene parameterization contains the M single scene parameterizations - they can be recovered by setting the shared shifts to zero . If the shifts are set to zero then their associated probability distributions can collapse to place all their mass at zero . So we expect that if there is little benefit to using the shared shifts then they can be effectively ignored , but if there is a benefit to using them then they can be utilized . As such , we can interpret this parameterization as inducing a soft form of parameter sharing .

Summary: This paper proposes to compress nerf models with entropy loss, where instead of directly training nerf model parameters, it trains a new function F which takes some compressed information and decodes to the nerf models. Then it did the same things as nerf, which render scenes in novel views. The authors show that the function F could largely compress the original nerf models while keeping similar PSNR.