SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aarush Gupta $^{1}$  Junli Cao $^{2,\dagger}$  Chaoyang Wang $^{2,\dagger}$  Ju Hu $^{2}$  Sergey Tulyakov $^{2}$   
Jian Ren $^{2}$  László A Jeni $^{1}$ $^{1}$ Robotics Institute, Carnegie Mellon University  $^{2}$ Snap Inc.  
Project page: https://lightspeed-r2l.github.io
Real-time novel-view image synthesis on mobile devices is prohibitive due to the limited computational power and storage. Using volumetric rendering methods, such as NeRF and its derivatives, on mobile devices is not suitable due to the high computational cost of volumetric rendering. On the other hand, recent advances in neural light field representations have shown promising real-time view synthesis results on mobile devices. Neural light field methods learn a direct mapping from a ray representation to the pixel color. The current choice of ray representation is either stratified ray sampling or Plücker coordinates, overlooking the classic light slab (two-plane) representation, the preferred representation to interpolate between light field views. In this work, we find that using the light slab representation is an efficient representation for learning a neural light field. More importantly, it is a lower-dimensional ray representation enabling us to learn the 4D ray space using feature grids which are significantly faster to train and render. Although mostly designed for frontal views, we show that the light-slab representation can be further extended to non-frontal scenes using a divide-and-conquer strategy. Our method offers superior rendering quality compared to previous light field methods and achieves a significantly improved trade-off between rendering quality and speed.
Real-time rendering of photo-realistic 3D content on mobile devices such as phones is crucial for mixed-reality applications. However, this presents a challenge due to the limited computational power and memory of mobile devices. The current graphics pipeline requires storing tens of thousands of meshes for complex scenes and performing ray tracing for realistic lighting effects, which demands powerful graphics processing power that is not feasible on current mobile devices. Recently, neural radiance field (NeRF [23]) has been the next popular choice for photo-realistic view synthesis, which offers a simplified rendering pipeline. However, the computational cost of integrating the radiance field remains a bottleneck for real-time implementation on mobile devices. There have been several attempts to reduce the computational cost of this integration step, such as using more efficient radiance representations [13, 40, 28, 17, 5, 10] or distilling meshes from radiance field [34, 6, 39, 35, 27, 29]. Among these approaches, only a handful of mesh-based methods [6, 29] have demonstrated real-time rendering capabilities on mobile phones, but with a significant sacrifice in rendering fidelity. Moreover, all aforementioned methods require significant storage space (over 200MB), which is undesirable for mobile devices with limited onboard storage.
These authors contributed equally.
(a) Rendering latency v/s fidelity.
(b) Faster training speed.
Figure 1: Our LightSpeed approach demonstrates a superior trade-off between on-device rendering quality and latency while maintaining a significantly reduced training time and boosted rendering quality. (a) rendering quality and latency on the  $400 \times 400$  Lego scene [23] running on an iPhone 13. (b) training curves for the  $756 \times 1008$  Fern scene [22].
Alternatively, researchers have used 4D light field<sup>1</sup> (or lumigraph) to represent radiance along rays in empty space [11, 24, 12, 19], rather than attempting to model the 5D plenoptic function as in NeRF-based approaches. Essentially, the light field provides a direct mapping from rays to pixel values since the radiance is constant along rays in empty space. This makes the light field suitable for view synthesis, as long as the cameras are placed outside the convex hull of the object of interest. Compared to integrating radiance fields, rendering with light fields is more computationally efficient. However, designing a representation of light field that compresses its storage while maintaining high view-interpolation fidelity remains challenging. Previous methods, such as image quilts [38] or multiplane images (MPI) [41, 16, 32, 9], suffer from poor trade-offs between fidelity and storage due to the high number of views or image planes required for reconstructing the complex light field signal. Recent works [36, 4, 2, 31] have proposed training neural networks to represent light fields, achieving realistic rendering with a relatively small memory footprint. Among those, MobileR2L [4] uses less than 10MB of storage per scene, and it is currently the only method that demonstrates real-time performance on mobile phones.
However, prior neural light field (NeLF) representations, including MobileR2L, suffer from inefficiencies in learning due to the high number of layers (over 60 layers), and consequently, a long training time is required to capture fine scene details. One promising strategy to address this issue is utilizing grid-based representations, which have proven to be effective in the context of training NeRFs [30, 25, 17, 10]. Nonetheless, incorporating such grid-based representation directly to prior NeLFs is problematic due to the chosen ray parameterization. R2L [36] and MobileR2L [4] parameterize light rays using a large number of stratified 3D points along the rays, which were initially motivated by the discrete formulation of integrating radiance. However, this motivation is unnecessary and undermines the simplicity of 4D light fields because stratified sampling is redundant for rays with constant radiance. This becomes problematic when attempting to incorporate grid-based representations for more efficient learning, as the high-dimensional stratified-point representation is not feasible for grid-based discretization. Similarly, the 6-dimensional Plücker coordinate used by Sitzmann et al. [31] also presents issues for discretization due to the fact that Plücker coordinates exist in a projective 5-space, rather than Euclidean space.
In this paper, we present LightSpeed, the first NeLF method designed for mobile devices that uses a grid-based representation. As shown in Fig. 1, our method achieves a significantly better trade-off between rendering quality and speed compared to prior NeLF methods, while also being faster to train. These advantages make it well-suited for real-time applications on mobile devices. To achieve these results, we propose the following design choices:
First, we revisit the classic 4D light-slab (or two-plane) representation [12, 19] that has been largely overlooked by previous NeLF methods. This lower-dimensional parameterization allows us to compactly represent the rays and efficiently represent the light field using grids. To our knowledge,
For the rest of the paper, we will use the term 'light field' to refer to the 4D light field, without explicitly stating the dimensionality.
Attal et al. [2] is the only other NeLF method that has experimented with the light-slab representation. However, they did not take advantage of the grid-based representation, and their method is not designed for real-time rendering. Second, to address the heavy storage consumption of 4D light field grids, we take inspiration from k-planes [10] and propose decomposing the 4D grids into six 2D feature grids. This ensures that our method remains competitive for storage consumption compared to prior NeLF methods. Third, we apply the super-resolution network proposed by MobileR2L [4], which significantly reduces the computational cost when rendering high-resolution images. Finally, the light-slab representation was originally designed for frontal-view scenes, but we demonstrate that it can be extended to represent non-frontal scenes using a divide-and-conquer strategy.
Our contributions pave the way for efficient and scalable light field representation and synthesis, making it feasible to generate high-quality images of real-world objects and scenes. Our method achieves the highest PSNR and among the highest frame rates (55 FPS on iPhone 14) on LLFF (frontal-view), Blender  $(360^{\circ})$ , and unbounded  $360^{\circ}$  scenes, proving the effectiveness of our approach.
Light Field. Light field representations have been studied extensively in the computer graphics and computer vision communities [38]. Traditionally, light fields have been represented using the 4D light slab representation, which parameterizes the light field by two planes in 4D space [12, 19]. More recently, neural-based approaches have been developed to synthesize novel views from the light field, leading to new light field representations being proposed.
One popular representation is the multi-plane image (MPI) representation, which discretizes the light field into a set of 2D planes. The MPI representation has been used in several recent works, including [41, 16, 32, 9, 7]. However, the MPI representation can require a large amount of memory, especially for high-resolution light fields. Another recent approach that has gained substantial attention is NeRF [23] (Neural Radiance Fields), which can synthesize novel views with high accuracy, but is computationally expensive to render and train due to the need to integrate radiance along viewing rays. There has been a substantial amount of works [37, 26, 28, 21, 13, 40, 28, 17, 5, 10, 34, 6, 39, 35, 27, 29, 36, 4, 2, 31] studying how to accelerate training and rendering of NeRF, but in the following, we focus on recent methods that achieve real-time rendering with or without mobile devices.
Grid Representation of Radiance Field. The first group of methods trade speed with space, by precomputing and caching radiance values using grid or voxel-like data structures such as sparse voxels [30, 13], octrees [40], and hash tables [25]. Despite the efficient data structures, the memory consumption for these methods is still high, and several approaches have been proposed to address this issue. First, Chen et al. [5] and Fridovich-Keil et al. [10] decompose voxels into matrices that are cheaper to store. Takikawa et al. [33] performs quantization to compress feature grids. These approaches have enabled real-time applications on desktop or server-class GPUs, but they still require significant computational resources and are not suitable for resource-constrained devices such as mobile or edge devices.
Baking High Resolution Mesh. Another group of methods adopts the approach of extracting high-resolution meshes from the learned radiance field [6, 29, 35]. The texture of the mesh stores the plenoptic function to account for view-dependent rendering. While these approaches have been demonstrated to run in real-time on mobile devices, they sacrifice rendering quality, especially for semi-transparent objects, due to the mesh-based representation. Additionally, storing high-resolution meshes with features is memory-intensive, which limits the resolution and complexity of the mesh that can be used for rendering.
Neural Light Fields. Recent works such as R2L [36], LFNS [31] and NeuLF [20] have framed the view-synthesis problem as directly predicting pixel colors from camera rays, making these approaches fast at inference time without the need for multiple network passes to generate a pixel color. However, due to the complexity of the 4D light field signal, the light field network requires sufficient expressibility to be able to memorize the signal. As a result, Wang et al. [36] end up using as many as 88 network layers, which takes three seconds to render one  $200 \times 200$  image on iPhone 13. In this regard, Cao et al. [4] introduce a novel network architecture that dramatically reduces R2L's computation through super-resolution. The deep networks are only evaluated on a low-resolution ray bundle and then upsampled to the full image resolution. This approach, termed MobileR2L, achieves real-time rendering on mobile phones. NeuLF [20] also proposes to directly regress pixel colors
using a light slab ray representation but is unable to capture fine-level details due to lack of any sort of high-dimensional input encoding and is limited to frontal scenes. Another notable work, SIGNET [8], utilizes neural methods to compress a light field by using a ultra spherical input encoding to the light slab representation. However, SIGNET doesn't guarantee photorealistic reconstruction and hence deviates from task at hand. Throughout the paper, we will mainly compare our method to MobileR2L [4], which is currently the state-of-the-art method for real-time rendering on mobile devices and achieves the highest PSNR among existing methods.
It is important to note that training NeLFs requires densely sampled camera poses in the training images and may not generalize well if the training images are sparse, as NeLFs do not explicitly model geometry. While there have been works, such as those by Attal et al. [2], that propose a mixture of NeRF and local NeLFs, allowing learning from sparse inputs, we do not consider this to be a drawback since NeLFs focus on photo-realistic rendering rather than reconstructing the light field from sparse inputs, and they can leverage state-of-the-art reconstruction methods like NeRF to create dense training images. However, it is a drawback for prior NeLFs [36, 4] that they train extremely slowly, often taking more than two days to converge for a single scene. This is where our new method comes into play, as it offers improvements in terms of training efficiency and convergence speed.
4D Light Fields or Lumigraphs are a representation of light fields that capture the radiance information along rays in empty space. They can be seen as a reduction of the higher-dimensional plenoptic functions. While plenoptic functions describe the amount of light (radiance) flowing in every direction through every point in space, which typically has five degrees of freedom, 4D light fields assume that the radiance is constant along the rays. Therefore, a 4D light field is a vector function that takes a ray as input (with four degrees of freedom) and outputs the corresponding radiance value. Specifically, assuming that the radiance  $\mathbf{c}$  is represented in the RGB space, a 4D light field is mathematical defined as a function, i.e.:
$$
\mathcal {F}: \mathbf {r} \in \mathbb {R} ^ {M} \mapsto \mathbf {c} \in \mathbb {R} ^ {3}, \tag {1}
$$
where  $\mathbf{r}$  is  $M$ -dimensional coordinates of the ray depending how it is parameterized.
Generating images from the 4D light field is a straightforward process. For each pixel on the image plane, we calculate the corresponding viewing ray  $\mathbf{r}$  that passes through the pixel, and the pixel value is obtained by evaluating the light field function  $\mathcal{F}(\mathbf{r})$ . In this paper, our goal is to identify a suitable representation for  $\mathcal{F}(\mathbf{r})$  that minimizes the number of parameters required for learning and facilitates faster evaluation and training.
MobileR2L. We adopt the problem setup introduced by MobileR2L [6] and its predecessor R2L [36], where the light field  $\mathcal{F}(\mathbf{r})$  is modeled using neural networks. The training of the light field network is framed as distillation, leveraging a large dataset that includes both real images and images generated by a pre-trained NeRF. Both R2L and MobileR2L represent  $\mathbf{r}$  using stratified points, which involves concatenating the 3D positions of points along the ray through stratified sampling. In addition, the 3D positions are encoded using sinusoidal positional encoding [23]. Due to the complexity of the light field, the network requires a high level of expressiveness to capture fine details in the target scene. This leads to the use of very deep networks, with over 88 layers in the case of R2L. While this allows for detailed rendering, it negatively impacts the rendering speed since the network needs to be evaluated for every pixel in the image.
To address this issue, MobileR2L proposes an alternative approach. Instead of directly using deep networks to generate high-resolution pixels, they employ deep networks to generate a low-resolution feature map, which is subsequently up-sampled to obtain high-resolution images using shallow super-resolution modules. This approach greatly reduces the computational requirements and enables real-time rendering on mobile devices. In our work, we adopt a similar architecture, with a specific focus on improving the efficiency of generating the low-resolution feature map.
Figure 2: LightSpeed Model for Frontal Scenes. Taking a low-resolution ray bundle as input, our approach formulates rays in two-plane ray representation. This enables us to encode each ray using multi-scale feature grids, as shown. The encoded ray bundle is fed into a decoder network consisting of convolutions and super-resolution modules yielding the high-resolution image.
3.2 LightSpeed
We first describe the light-slab ray representation for both frontal and non-frontal scenes in Sec. 3.2.1. Next, we detail our grid representation for the light-slab in Sec. 3.2.2 and explain the procedure for synthesizing images from this grid representation in Sec. 3.3. Refer to Fig. 2 for a visual overview.
3.2.1 Ray Parameterization
Light Slab (two-plane representation). Instead of utilizing stratified points or Plücker coordinates, we represent each directed light ray using the classic two-plane parameterization[19] as an ordered pair of intersection points with two fixed planes. Formally,
$$
\mathbf {r} = (x, y, u, v), \tag {2}
$$
where  $(x,y)\in \mathbb{R}^2$  and  $(u,v)\in \mathbb{R}^2$  are ray intersection points with fixed planes  $P_{1}$  and  $P_{2}$  in their respective coordinate systems. We refer to these four numbers as the ray coordinates in the 4D ray space. To accommodate unbounded scenes, we utilize normalized device coordinates (NDC) and select the planes  $P_{1}$  and  $P_{2}$  as the near and far planes (at infinity) defined in NDC.
Divided Light Slabs for Non-frontal Scenes. A single light slab is only suitable for modeling a frontal scene and cannot capture light rays that are parallel to the planes. To model non-frontal scenes, we employ a divide-and-conquer strategy by using a composition of multiple light slab representations to learn the full light field. We partition the light fields into subsets, and each subset is learned using a separate NeLF model. The partitions ensure sufficient overlap between sub-scenes, resulting in a continuous light field representation without additional losses while maintaining the frontal scene assumption. To perform view synthesis, we identify the scene subset of the viewing ray and query the corresponding NeLF to generate pixel values. Unlike Attal et al. [2], we do not perform alpha blending of multiple local light fields because our division is based on ray space rather than partitioning 3D space.
For object-centric  $360^{\circ}$  scenes, we propose to partition the scene into 5 parts using surfaces of a near-isometric trapezoidal prism and approximate each sub-scene as frontal (as illustrated in Fig. 3). For unbounded  $360^{\circ}$  scenes, we perform partitioning using k-means clustering based on camera orientation and position. We refer the reader to the supplementary material for more details on our choice of space partitioning.
3.2.2 Feature Grids for Light Field Representation
Storing the 4D light-slab directly using a high-resolution grid is impractical in terms of storage and inefficient for learning due to the excessive number of parameters to optimize. The primary concern arises from the fact that the 4D grid size increases quartically with respect to resolutions. To address this, we suggest the following design choices to achieve a compact representation of the light-slab without exponentially increasing the parameter count.
Figure 3: Space Partitioning for Non-frontal scenes. We partition object-centric  $360^{\circ}$  scenes into 5 parts as shown. Each colored face of the trapezoidal prism corresponds to a partitioning plane. Each scene subset is subsequently learned as a separate NeLF
Lower Resolution Feature Grids. Instead of storing grids at full resolution, we choose to utilize low-resolution feature grids to take advantage of the quartic reduction in storage achieved through resolution reduction. We anticipate that the decrease in resolution can be compensated by employing high-dimensional features. In our implementation, we have determined that feature grids of size  $128^4$  are suitable for synthesizing full HD images. Additionally, we adopt the approach from Instant-NGP [25] to incorporate multi-resolution grids, which enables an efficient representation of both global and local scene structures.
Decompose 4D Grids into 2D Grids. Taking inspiration from k-planes [10], we propose to decompose the 4D feature grid using  $\binom{4}{2}=6$  number of 2D grids, with each 2D grid representing a sub-space of the 4D ray space. This results in a storage complexity of  $\mathcal{O}(6N^2)$ , greatly reducing the storage required to deploy our grid-based approach to mobile devices.
3.3 View Synthesis using Feature Grids
Similar to MobileR2L [4], LightSpeed takes two steps to render a high resolution image (see Fig. 2).
Encoding Low-Resolution Ray Bundles. The first step is to render a low-resolution  $(H_{L} \times W_{L})$  feature map from the feature grids. This is accomplished by generating ray bundles at a reduced resolution, where each ray corresponds to a pixel in a downsampled image. We project each ray's 4D coordinates  $\mathbf{r} = (x, y, u, v)$  onto 6 2D feature grids  $\mathbf{G}_{xy}, \mathbf{G}_{xu}, \mathbf{G}_{xv}, \mathbf{G}_{yu}, \mathbf{G}_{yv}, \mathbf{G}_{uv}$  to obtain feature vectors from corresponding sub-spaces. The feature values undergo bilinear interpolation from the 2D grids, resulting in six interpolated  $F$ -dimensional features. These features are subsequently concatenated to form a  $6F$ -dimensional feature vector. As the feature grids are multi-resolutional with  $L$  levels, features  $g_{l}(\mathbf{r}) \in \mathbb{R}^{6F}$  from different levels (indexed by  $l$ ) are concatenated together to create a single feature  $g(\mathbf{r}) \in \mathbb{R}^{6LF}$ . Combining the features from all rays generates a low-resolution 2D feature map  $\tilde{\mathbf{G}} \in \mathbb{R}^{H_{L} \times W_{L} \times 6LF}$ , which is then processed further in the subsequent step.
Decoding High-Resolution Image. To mitigate the approximation introduced by decomposing 4D grids into 2D grids, the features  $g(\mathbf{r})$  undergo additional processing through a MLP. This is implemented by applying a series of  $1 \times 1$  convolutional layers to the low-resolution feature map. Subsequently, the processed feature map is passed through a sequence of upsampling layers (similar to MobileR2L [4]) to generate a high-resolution image.
Datasets. We benchmark our approach on the real-world forward-facing [22] [23], the realistic synthetic  $360^{\circ}$  datasets [23] and unbounded  $360^{\circ}$  scenes [3]. The forward-facing dataset consists of 8 real-world scenes captured using cellphones, with 20-60 images per scene and 1/8th of the images used for testing. The synthetic  $360^{\circ}$  dataset has 8 scenes, each having 100 training views and 200 testing views. The unbounded  $360^{\circ}$  dataset consists of 5 outdoor and 4 indoor scenes with a central object and a detailed background. Each scene has between 100 to 300 images, with 1 in 8 images used for testing. We use  $756 \times 1008$  LLFF dataset images,  $800 \times 800$  resolution for the  $360^{\circ}$  scenes, and 1/4th of the original resolution for the unbounded  $360^{\circ}$  scenes.
(a) Scene
(b) Ground truth
(c) NeRF
Figure 4: Qualitative Results on frontal and non-frontal scenes. Zoomed-in comparison between NeRF [23], MobileR2L [4] and our LightSpeed approach.
(d) MobileR2L
(e) LightSpeed