LightSpeed: Light and Fast Neural Light Fields on Mobile Devices
Authors: Aarush Gupta
Junli Cao
Chaoyang Wang
Ju Hu
Sergey Tulyakov
Jian Ren
Laszlo Attila Jeni
Venue: NeurIPS 2023 poster (2023)
Keywords: light field
neural radiance field
novel view synthesis
TL;DR: A new neural representation of light field that enables real-time synthesis of novel view images on mobile devices.

Abstract: Real-time novel-view image synthesis on mobile devices is prohibitive due to the limited computational power and storage. Using volumetric rendering methods, such as NeRF and its derivatives, on mobile devices is not suitable due to the high computational cost of volumetric rendering. On the other hand, recent advances in neural light field representations have shown promising real-time view synthesis results on mobile devices. Neural light field methods learn a direct mapping from a ray representation to the pixel color. The current choice of ray representation is either stratified ray sampling or Plücker coordinates, overlooking the classic light slab (two-plane) representation, the preferred representation to interpolate between light field views. In this work, we find that using the light slab representation is an efficient representation for learning a neural light field. More importantly, it is a lower-dimensional ray representation enabling us to learn the 4D ray space using feature grids which are significantly faster to train and render. Although mostly designed for frontal views, we show that the light-slab representation can be further extended to non-frontal scenes using a divide-and-conquer strategy. Our method provides better rendering quality than prior light field methods and a significantly better trade-off between rendering quality and speed than prior light field methods.

LightSpeed: Light and Fast Neural Light Fields on Mobile Devices
Aarush Gupta'
Junli Cao
Chaoyang Wang2tt Ju Hu²
Sergey Tulyakov²
Jian Ren László A Jeni"
Robotics Institute, Carnegie Mellon University 2Snap Inc.
Project page: https://ighhseeed-2.ggghbbbbbb
Abstract
Real-time novel-view image synthesis on mobile devices prohibitive due to the limited computational power and storage. Using volumetric rendering methods, such as NeRF and derivatives, on mobile devices is not suitable due the high computational cost of volumetric rendering. On the other hand, recent advances in neural light field representations have shown promising real-time view synthesis results mobile devices. Neural light field methods learn direct mapping from a ray representation to the pixel color. The current choice of ray representation is either stratified ray sampling Plücker coordinates, overlooking the classic light slab (two-plane) representation, the preferred representation to interpolate between light field views. In this work, find that using the light slab representation is efficient representation for learning a neural light field. More importantly, is a nower-dimensional ray representation enabling to learn the 4D ray space using feature grids which are significantly faster to train and render. Although mostly designed for frontal views, show that the light-slab representation extended to non-frontal scenes using divide-and-conquu strategy. Our method offers superior rendering quality compared previous light field methods and achieves significantly improved trade-off between rendering quality and speed.
be further
Introduction
Real-time rendering of photo-realistic 3D content mobile devices such phones is erucial for mixed-reality applications. However, this presents a challenge due to the limited computational power and memory of mobile devices. The current graphics pipeline requires storing tens of thousands of meshes for complex scenes and performing ray tracing for realistic lighting effects, which demands powerful graphics processing power that not feasible on current mobile devices. Recently, neural radiance field (NeRF [23]) has been the next popular choice for photo-realistic view synthesis, which offers simplified rendering pipeline. However, the computational cost of integrating the radiance field remains bottleneck for real-time implementation on mobile devices. There have been several attempts to reduce the computational cost of this integration step, such as using more efficient radiance representations [13, 40, 28, 17, 17,5, 5, 10] or distilling meshes from radiance field [34, 6, 39, 35, 27, 29]. Among these approaches, only a handful of mesh-based methods [6, 29] have demonstrated real- time rendering capabilities on mobile phones, but with significant sacrifice in rendering fidelity. Moreover, aforementioned methods require significant storage space (over 200MB), which undesirable for mobile devices with limited onboard storage.
These authors contributed equally.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

30-L
Rendering Quality (PSNR)
31.0
15-L
60:1
Rendering Quality (PSNR)
24.0
30.5
60-1.
30.0
23.5
4X
29.5
30-1
LightSpeed (Ours)
23.0
MobilcRZL
LightSpeed (Ours)
29.0
15-L
MobileNeRF
22.5
MobileR21.
7.5
10.0
12.5
15.0
17.5
100000
200000
300000
400000
Rendering latency (in ms)
Training iterations
(a) Rendering latency v/s fidelity.
(b) Faster training speed.
Our LightSpeed approach demonstrates a superior trade-off between on-device rendering quality and latency while maintaining significantly reduced training time and boosted rendering quality. (a) rendering quality and latency on the 400 400 Lego scene [23] running on an iPhone 13. (b) training curves for the 756 x 1008 Fern scene [22].
Figure I:
Alternatielly, researchers have used 4D light field' (or lumigraph) represent radiance along rays in empty space [11, 24, 12, 19], rather than attempting to model the 5D plenoptic function as in NeRF-based approaches. Essentially, the light field provides direct mapping from rays pixel values since the radiance constant along rays in empty space. This makes the light field suitable for view synthesis, long as the cameras are placed outside the convex hull of the object of interest. Compared to integrating radiance fields, rendering with light fields is more computationally efficient. However, designing representation of light field that compresses its storage while maintaining high view-interpolation fidelity remains challenging. Previous methods, such as image quilts [38] or multiplane images (MPI) [41, 16, 32, 9], suffer from poor trade-offs between fidelity and storage due to the high number of views or image planes required for reconstructing the complex light field signal. Recent works [36, 2, 31] have proposed training neural networks represent light fields, achieving realistic rendering with relatively small memory footprint. Among those, MobileR21, [4] uses less than 10MB of storage per scene, and currently the only method that demonstrates real-time performance mobile phones.
However, prior neural light field (NeLF) representations, including MobileR2L, suffer from inefficiencies in learning due to the high number of layers (over 60 layers), and consequently, capture fine scene details. One promising strategy address
long training time is required this issue is utilizing grid- based representations, which have proven to be effective in the context of training NeRFs [30, 25, 17, 10]. Nonetheless, incorporating such grid-based representation directly to prior NeLFs is problematic due to the chosen ray parameterization. R2L [36] and MobileR2L [4] parameterize light rays using a large number of stratified 3D points along the rays, which were initially motivated by the discrete formulation of integrating radiance. However, this motivation is unnecessary and undermines the simplicity of 4D light fields because stratified sampling is redundant for rays with constant radiance. This becomes problematic when attempting to incorporate grid-based representations for more efficient learning, as the high-dimensional stratified-point representation not feasible for grid-based discretization. Similarly, the 6-dimensional Plücker coordinate used by Sitzmann et al. [31] also presents issues for discretization due to the fact that Plücker coordinates exist in projective 5-space, rather than Euclidean space.
In this paper, we present LightSpeed, the first NeLF method designed for mobile devices that uses grid-based representation. As shown in Fig. 1, our method achieves significantly better trade-off between rendering quality and speed compared prior NeLF methods, while also being faster to train. These advantages make it well-suited for real-time applications on mobile devices. To achieve these results, we propose the following design choices:
First, we revisit the classic 4D light-slab (or two-plane) representation [12, 19] that has been largely overlooked by previous NeLF methods. This lower-dinenssonal parameterization allows us to compactly represent the rays and efficiently represent the light field using grids. To our knowledge.
For the rest of the paper, we will use the term 'light field' to refer to the 4D light field, without explicitly stating the dimensionality.

---

Attal et al. [2] is the only other NeLF method that has experimented with the light-slab representation. However, they did not take advantage of the grid-based representation, and their method is not designed for real-time rendering. Second, to address the heavy storage consumption of 4D light field grids, we take inspiration from k-planes [10] and propose decomposing the 4D grids into six 2D) feature grids. This ensures that our method remains competitive for storage consumption compared to prior NeLF methods. Third, we apply the super-resolution network proposed by MobileR2L [4], which significantly reduces the computational cost when rendering high-resolution images. Finally, the light-slab representation was originally designed for frontal-view scenes, but we demonstrate that it can be extended to represent non-frontal scenes using dwvide-and-oonuue strategy.
Our contributions pave the way for efficient and scalable light field representation and synthesis. making it feasible to generate high- high-quality images of real-world objects and scenes. Our method achieves the highest PSNR and among the highest frame rates (55 FPS on iPhone 14) on L.LFF (frontal-view), Blender (360"), and unbounded 360 scenes, proving the effectiveness of our approach.
Related work
Light Field. Light field representations have been studied extensively in the computer graphics and computer vision communities [38]. Traditionally, light fields have been represented using the 4D light slab representation, which parameterizes the light field by two planes in 4D space [12, 19]. More recently, neural- based approaches have been developed to synthesize novel views from the light field, Ieading to new light field representations being proposed.
One popular representation is the multi-plane image (MPI) representation, which discretizes the light field into a set of 2D planes. The MPI representation has been used in several recent works, including [41, 16, 32, 9, 7]. However, the MPI representation can require a large amount of memory, especially for high-resolution light fields. Another recent approach that has gained substantial attention is NeRF [23] (Neural Radiance Fields), which can synthesize novel views with high accuracy, but is computationally expensive to render and train due to the need to integrate radiance along Viewing rays. There has been a substantial amount of works [37, 26, 28, 1,, 13, 40, 28, 17, 5, 10, 34, 6, 39, 35, 27, 29,2,, 36, 4, 2, 31] studying how to accelerate training and rendering of NeRF, but in the following, we focus recent methods that achieve real-time rendering with without mobile devices.
Grid Representation of Radiance Field. The first group of methods trade speed with space, by precomputing and caching radiance values using grid or voxel-like data structures such as sparse voxels [30, 13], octrees [40], and hash tables [25]. Despite the efficient data structures, the memory consumption for these methods is still high, and several approaches have been proposed to address this issue. First, Chen el al. [5] and Fridovich- Keil et al. [10] decompose voxels into matrices that are cheaper to store. Takikawa el al. [33] performs quantization to compress feature grids. These approaches have enabled real-time applications on desktop or server-class GPUs, but they still require significant computational resources and are not suitable for resource-constrained devices such as mobile or edge devices.
Baking High Resolution Mesh. Another group of methods adopts the approach of extracting high-resolution meshes from the learned radiance field [6, 29, 35]. The texture of the mesh stores the plenoptic function to account for view-dependent rendering. While these approaches have been demonstrated to run in real-time on mobile devices, they sacrifice rendering quality, especially for semi semi-transparent objects, due to the mesh-based representation. Additionally, storing high-resolution meshes with features memory-intensive which limits the resolution and complexity of the mesh that can be used for rendering.
Neural Light Fields. Recent works such as R2L [36]. LFNS [31] and NeuLF [20] have framed the view-synthesis problem as directly predicting pixel colors from camera rays, making these approaches fast at inference time without the need for multiple network passes to generate pixel color. However, due to the complexity of the 4D light field signal, the light field network requires sufficient expressibility to be able memorize the signal. As result, Wang et al. [36] end up using as many as 88 network layers, which takes three seconds to render one 200 x 200 image on iPhone 13, In this regard, Cao el al. [4] introduce a novel network architecture that dramatically reduces R2L's computation through super-resolution. The deep networks are only evaluated on a low-resolution ray bundle and then upsampled to the full image resolution. This approach, termed MobileR2I, achieves real-time rendering mobile phones. NeuLF [20] also proposes to directly regress pixel colors
3

---

using a light slab ray representation but is unable to capture fine-level details due to lack of any sort of digeeiiiensional input encoding and is limited to frontal scenes. Another notable work, SIGNET [8], utilizes neural methods to compress light field by using ultra spherical input encoding to the light slab representation. However, SIGNET doesn't I guarantee photorealistic reconstruction and hence deviates from task at hand. Throughout the paper, we will mainly compare our method to MobileR2L [4], which is currently the state-of-the-art method for real-time rendering on mobile devices and achieves the highest PSNR among existing methods.
important to note that training NeLFs requires densely sampled camera poses in the training images and may not generalize well if the training images are sparse, as NeLFs do not explicitly model geometry. While there have been works, such as those by Attal et al. [2], that propose mixture of NeRF and local NeLFs, allowing learning from sparse inputs, we do not consider this to be a drawback since NeI.Fs focus on photo-realistic rendering rather than reconstructing the light field from sparse inputs, and they can leverage state-of-the-art reconstruction methods like NeRF to create dense training images. However, it is drawback for prior NeLFs [36, 4] that they train extremely slowly, often taking more than two days to converge for single scene. This is where our new method comes into play, as offers improvements in terms of training efficiency and convergence speed.
3 Methodology
3.1 Prerequisites
4D Light Fields or Lumigraphs are representation of light fields that capture the radiance information along rays in empty space. They can be seen as a reduction of the higher-dimensional plenoptic functions. While plenoptic functions describe the amount of light (radiance) flowing in every direction through every point in space, which typically has five degrees of freedom, 4D light fields assume that the radiance is constant along the rays. Therefore, a 4D light field is vector function that takes a ray as input (with four degrees of freedom) and outputs the corresponding radiance value. Specifically, assuming that the radiance is represented in the RGB space, 4D light field is mathematical defined as a function, i.e.:
FirER +CR
(I)
where r is --dimensional coordinates of the ray depending how it is parameterized.
Generating images from the 4D light field is a straightforward process. For each pixel on the image plane, we calculate the corresponding viewing ray that passes through the pixel, and the pixel value is obtained by evaluating the light field function F(r). In this paper, our goal is identify a suitable representation for F(r) that minimizes the number of parameters required for learning and facilitates faster evaluation and training.
MobileR2I. We adopt the problem setup introduced by MobileR2L [6] and its predecessor R2L [36], where the light field F(r) is modeled using neural networks. The training of the light field network is framed distillation, leveraging a large dataset that includes both real images and images generated by pre-trained NeRF. Both R2L and MobileR2L represent using stratified points, which involves concatenating the 3D positions of points along the ray through stratified sampling. In addition, the 3D positions are encoded using sinusoidal positional encoding [23]. Due to the complexity of the light field, the network requires a high level of expressiveness to capture fine details in the target scene. This leads to the use of very deep networks, with over 88 layers in the case of R2L. While this allows for detailed rendering, it negatively impacts the rendering speed since the network needs to be evaluated for every pixel in the image.
To address this issue, MobileR2L proposes an alternative approach. Instead of directly using deep networks to generate high-resolution pixels. they employ deep networks to generate low-resolution feature map, which subsequently up-sampled to obtain high-resolution images using shallow super-resolution modules. This approach greatly reduces the computational requirements and enables real-time rendering on mobile devices. In our work, we adopt similar architecture, with a specific focus on improving the efficiency of generating the low-resolution feature map.

---

UV
Super-RRcccution
Conn Ix
Yu
yV
Two-Plane Light Field
Ercoded Ray
Low-resolution Ray Bundle
Decoder Network
Grids
Ray-space Grid Encoder
Figure 2: LightSpeed Model for Frontal Scenes. Taking a low-resolution ray bundle as input, our approach formulates rays in two-plane ray representation. This enables encode each ray using multi-scale feature grids, as shown. The encoded ray bundle is fed into a decoder network consisting of convolutions and super-resolution modules yielding the high-resolution image.
3.2 LightSpeed
We first descrihe the light-slab ray representation for both frontal and non-frontal scenes in Sec. 3.2.1. Next, detail our grid representation for the light-slab in Sec. 3.2.2 and explain the procedure for synthesizing images from this grid representation in Sec. 3.3. Refer to Fig. 2 for a visual overview.
3.2.1 Ray Parameterization
Light Slab (two-plane representation). Instead of utilizing stratified points or Plücker coordinates, represent each directed light ray using the classic two-plane parameterization| 19] ordered pair of intersection points with two fixed planes. Formally,
r (2,, y, u, v),
where (z, E R² and (u, E R are ray intersection points with fixed planes P1 and P2 in their respective coordinate systems. We refer these four numbers the ray coordinates in the 4D ray space. To accommodate unbounded scenes, we utilize normalized device coordinates (NDC) and select the planes P1 and P2 the near and far planes (at infinity) defined in NDC.
Divided Light Slabs for Non-frontal Scenes. A single light slab is only suitable for modeling a frontal scene and cannot capture light rays that are parallel the planes. To model non-frontal scenes, we employ divide-and-conge representations to learn the full light field. We partition the light fields into subsets, and each subset learned using separate NeLF model. The partitions ensure sufficient overlap between sub-scenes, resulting in a continuous light field representation without additional losses while maintaining the frontal scene assumption. To perform view synthesis, identify the scene subset of the viewing ray and query the corresponding NeLF to generate pixel values. Unlike Attal et al. [2], we do not perform alpha blending of multiple local light fields because our division is based on ray space rather than partitioning 3D space.
strategy by using a composition of multiple light slab
For object-centric 360 scenes, we propose to partition the scene into parts using surfaces of near-isometric trapezoidal prism and approximate each sub-scene as frontal (as illustrated Fig. 3). For unbounded 360 scenes, perform partitioning using k-means clustering based camera orientation and position. We refer the reader to the supplementary material for more details on our choice of space partitioning.
3.2.2 Feature Grids for Light Field Representation
Storing the 4D light-slab directly using reghuursoottion grid is impractical in terms of storage and inefficient for learning due to the excessive number of parameters to optimize. The primary concern arises from the fact that the 4D grid size increases quartically with respect to resolutions. To address this, we suggest the following design choices achieve a compact representation of the light-slab without exponentially increasing the parameter count.
5

---

Divided Object-Centric Scene
Scene Subset Frontal Scene
Two-Plane Light Field
Figure 3: Space Partitioning for Non-frontal scenes. We partition object-centric 360 scenes into parts shown. Each colored face of the trapezoidal prism corresponds partitioning plane. Each scene subset is subsequently learned as a separate NeLF
Lower Resolution Feature Grids. Instead of storing grids at full resolution, we choose to utilize low-resolution feature grids to take advantage of the quartic reduction storage achieved through resolution reduction. We anticipate that the decrease in resolution can be compensated by employing high-dimensional features. In our implementation, we have determined that feature grids of size 128 are suitable for synthesizing full HD images. Additionally, we adopt the approach from Instant- NGP [25] incorporate multi-resolution grids, which enables an efficient representation of both global and local scene structures.
Decompose 4D Grids into 2D Grids. Taking inspiration from k-planes [10], we propose to decompose the 4D feature grid using (*) 6 number of 2D grids, with each 2D grid representing sub-space of the 4D ray space. This results in storage complexity of (66N2), greatly reducing the storage required to deploy our grid-based approach to mobile devices.
3.3 View Synthesis using Feature Grids
Similar to MobileR21. [4], LightSpeed takes two steps to render high resolution image (see Fig. 2).
Encoding Low-Resolution Ray Bundles. The first step is to render low-resolution (H₁, WT) feature map from the feature grids. This is accomplished by generating ray bundles resolution, where each ray corresponds to pixel in downsampled image. We project each ray's 4D coordinates r = (z, y, u, v) onto 2D feature grids Gzy: Gaus Gzvs Gyn, Gye Guv to obtain feature vectors from corresponding sub-spaces. The feature values undergo bilinear interpolation from the 2D grids, resulting in six interpolated F-dimensional features. These features are subsequently concatenated to form 6F -i-deensiooaal feature vector. As the feature grids are multi-resolutional with L levels, features (r) E R from different levels (indexed by I) are concatenated together to create single feature g(r) R6LF., Combining the features from all rays generates low-resolution 2D feature map G , which is then processed further the subsequent step.
reduced
Decoding High-Resolutee Resolution Image. To mitigate the approximation introduced by decomposing 4D grids into 2D grids, the features g(r) undergo additional processing through MLP. This is implemented by applying a series
x convolutional layers to the low-resolution feature map Subsequently, the processed feature map is passed through sequence of upsampling layers (similar to MobileR21. [4]) to generate a high-resolution image.
4 Experiments
Datasets. We benchmark our approach on the real-world forward-facing [22] [23], the realistic synthetic 360 datasets [23] and unbounded 360 scenes [3]. The forward-facing dataset consists of real-world scenes captured using cellphones, with 20-60 images per scene and 1/8th of the images used for testing. The synthetic 360 dataset has scenes, each having 100 training views and 200 testing views. The unbounded 360" dataset consists of 5 outdoor and indoor scenes with a central object and detailed background. Each scene has between 100 to 300 images, with in 8 images used for testing. We use 756 1008 LLFF dataset images, 800 800 resolution for the 360 scenes, and 1/4th of the original resolution for the unbounded 360 scenes.

---

(a) Scene
(b) Ground truth
(c) NeRF
(d) MobileR2L
(e) LightSpeed
Figure 4: Qualitative Results frontal and non-frontal scenes. Zoomed-in comparison between
NeRF [23], MobileR21. [4] and our LightSpeed approach.
Training Details. We follow similar training scheme as MobileR2I.: train the LightSpeed model using pseudo-data mined from a pre-trained NeRF teacher. We specifically train MipNeRF teachers to sample 10k pseudo-data points for the LLFFF dataset. For synthetic and unbounded 360 scenes, we mine 30k samples per scene using Instant-NGP [25] teachers. Following this, we fine-tune the model on the original data. We optimize for the mean-squared error between generated and ground truth images. We refer the reader to the supplementary material for more training details.
We use 63 84 (12x downsampled from the desired 756 x 1008 resolution) input ray bundles for the forward-facing scenes. For 360 scenes, we use 100 100 (8x downsampled from the desired 800 x 800 image resolution) ray bundles. For unbounded scenes, we use ray bundles 12x downsampled from the image resolution we use. We train our frontal LightSpeed models as well as each sub-scene model in non-frontal scenes for 200k iterations.
Baselines and Metries. We compare method's performance on bounded scenes with MobileR2L[6]. MobileNeRF[6] and SNeRG[13]. We evaluate our method for rendering quality using three metries: PSNR, LPIPS, and SSIM. For unbounded scenes, we report the PSNR metric on 6 scenes and compare it with MobileNeRF [6] and NeRFMeshing [27]. To further demonstrate the effectiveness of our approach. we compare our approach with others on two other criteria: (a) On-device Rendering Speed: We report and compare average inference times per rendered frame on various mobile chips, including Apple A15, Apple MI Pro and Snapdragon SM8450 chips; and (b) Efficient Training: We compare the number of iterations LightSpeed and MobileR2I. require to reach a target PSNR. We pick Lego scene from 360 scenes and Fern from forward-facing scenes as representative scenes to compare. We also report the storage requirements of our method per frontal scene and compare with baselines.
4.1 Results and Analysis
Rendering Quality, As in Tab. 1, we obtain better results on all rendering fidelity metrics on the two bounded datasets. We also outperform MobileNeRF and NeRFMeshing 4 out of 6 unbounded 360 scenes. We refer the reader to Fig. 4 for a visual comparison of our approach with MobileR2L and NeRF. Our method has much better rendering quality, capturing fine-level details where MobileR2L, and in some cases, even the original NeRF model, fails. Note that we use Instant-NGP teachers for 360 scenes, which have slightly inferior performance to MipNeRF teachers used by MobileR21.. This further shows the robustness of our approach to inferior NeRF teachers.
Storage Cost. We report storage requirements in Tab. 1. Our approach has competitive on-device storage to the MobileR2L model. Specifically, we require a total 16.3 MB of storage per frontal scene. The increase in storage is expected since we'r re using grids to encode our light field. We also report storage values for lighter LightSpeed networks in the ablation study (see Tab. 5), all of which have similar better rendering quality than the full-sized MobileR2L network.
Training Speed. We benchmark the training times and the number of iterations required for LightSpeed and MobileR21. in Tab. with target PSNR of 24 for Fern scene and 32 for the Lego scene, Our approach demonstrates training speed-up of 2.5x on both scenes. Since we are train them in parallel (which is not
modeling 360 scenes composition of 5 light fields,
7

---

Table 1: Quantitative Comparison on Forward- faccing Synthetic 360 and Unbounded 360 Datasets. LighSpeed achieves the best rendering quality with competitive storage. We use an out-of-the-box Instant-NGP [25] implementation [1] (as teachers for 360 scenes) which dose not report SSIM and LPIPS values. We omit storage for NeRF-based methods since they are not comparable.
Method
Synthetic 360
Forward- Facing
PSNR
SSIM 1
LPIPS +
PSNR
SSIM
LPIPS +
Storage +
NeRF [23]
31.01
0.947
0.081
26.50
0.811
0.250
NeRF-PyTorch
30.92
0.991
0.045
26.26
0.965
0.153
SNeRG [13]
30.38
0.950
0.050
25.63
0.818
0.183
337.3 MB
MobileNeRF [6]
30.90
0,947
0.062
25.91
0.825
0.183
201.5 MB
MobileR2I. [4]
31.34
0.993
0.051
26.15
0.966
0.187
8.2 MB
LightSpeed (Ours)
32.23
0.994
0.038
26.50
0.968
0.173
16.3 MB
Our Teacher
32.96
26.85
0.827
0.226
Unbounded 360
Method
Bicycle
Garden
Stump
Bonsai
Counter
Kitchen
MobileNeRF [6]
21.70
23.54
23.95
NeRFMeshing [27]
21.15
22.91
22.66
25.58
20.00
23.59
LightSpeed (Ours)
22.51
24.54
22.22
28.24
25.46
27.82
Instant-NGP (Our teacher) [25]
21.70
23.40
23.20
27.4
25.80
27.50
Table Training Time for Lego and Fern scenes with 32 and 24 target PSNRs. LightSpeed trains significantly faster than MobilcR2L. It achieves even greater speedup when trained in parallel for 360 scenes (parallel training is not applicable for frontal scenes).
Forward Forward-Facing: Fern
Synthetic 360"; Lego
Method
Duration | Iterations
Duration
Iterations
MobileR21.
12.5 hours
70k
192 hours
860k
LightSpeed
4 hours
27k
75 hours
425k
LightSpeed (Parallelized)
15 hours
85k
possible for MobileR2L), further trimming down the training time. Moreover, the training speedup reaches ~ 4x when networks are trained beyond the mentioned target PSNR (see Fig. 1).
Inference Speed. Tab. 3 shows our method's inference time as compared MobileR2L and MobileNeRF. We maintain comparable runtime as MobileR2L while having better rendering fidelity. Since on-device inference crucial to our problem setting, we also report rendering times of a smaller 30-layered decoder network that has similar rendering quality as the MobileR2L. model (see Tab. 5).
Table 3: Rendering Latency Analysis. LightSpeed maintains a competitive rendering latency (ms) N in table) due to to prior works. MobileNeRF is not able render out of real-world scenes J memory constraints, and no numbers are reported for A13, M1 Pro and Snapdragon chips.
Forward-Facing
Synthetic 360
Chip
MohileNeRF
MnbileR21.
Ours
Ours (30-L.
MohileNeRF
MobileR21.
Ours
Ours (30-1)
Apple A13 (Low-end)
40.23
41.06
32.29
65.54
66.10
$3.89
Apple A15(Low-end)
27.15
18.04
19.05
15.28
17.54
26.21
27.10
20.15
Apple AI5(High-end)
20.98
16.48
16.67
22.65
26.47
20.35
Apple MI Pro
17.65
17.08
13.86
27.37
27.14
20.13
Snapdragon SM8450
39.14
45.65
32.89
40.86
41.26
33.87
4.2 Ablations
Data Requirements. We use 10k samples as used by MobileR2I. to train LightField models for frontal scenes. However, for non-fronta scenes, resort to using 30k pseudo-data samples per

---

12
Resteng Wusterngg Collity (1500)
Readi Oaloo Oally (DSNR))
-0
GGah D15
-6-
LS (Gri) DH
LS (Gri)
L5 D15
L5 (Frg DM
(Fegg)
Padier (Prog) D15
Packer
Puzke (Prc
19303)
Thiling
(a) 15 15-Layer Networks
(b) 30-Layer Networks
(c) 60-Layer Networks
Figure 5: Test PSNR v/s Training Iterations. We compare test set PSNR obtained by LightSpeed (orid)ourrs), LightSpeed (frequency encoded), and Plücker--asse neural light field as the training progresses for 3 different network configurations.
scene. Dividing 10k samples amongst sub-scenes assigns too few samplers per sub-e, which is detrimental to grid learning. We experimentally validate data requirements by comparing MobileR2L and LightSpeed trained for different amounts of pseudo-data. We train one 400 x 400 sub-scene from the Lego scene for 200k iterations with 1/5th of 10k and 30k samples, i.e. 2k and 6k samples. Tab. exhibits significantly decreased rendering quality for the LightSpeed network as compared to MobileR2L when provided with less pseudo-data.
Table 4: Pseudo-Data Requirement for
Scenes. We analyze the importance of mining more pseudo-data for non-frontal scenes. Using 1/5th of 10k and 30k sampled pseudo-data points, we find more pseudo-data is crucial for the boosted performance of the LightSpeed model.
2k Samples
6k Samples
Method
PSNR f
SSIM f
LPIPS
PSNR 1
SSIM 1
LPIPS
MobileR21.
30.19
0.9894
0.0354
30.56
0.9898
0.0336
LightSpeed (Ours)
30.44
0,9899
0.0299
31.2
0.9906
0.0284
Decoder Network Size. We further analyze the trade-off between inference speed and rendering quality of our method and MobileR2L. To this end, we experiment with decoders of different depths and widths. Each network is trained for 200k iterations and benchmarked on iPhone 13. Tab. 5 shows that a 30-layered LightSpeed model has better inference speed and rendering quality as compared the 60-layered MobileR2L model. This 30-layered variant further occupies less storage as compared to its full-sized counterpart. Furthermore, lighter LightSpeed networks obtain comparable performance as the 60-layered MobileR2L. Note that reducing the network capacity of MobileR2L results in significant drops in performance. This means that we can get the same rendering quality as MobileR2I. with considerably reduced on-device resources, paving the way for much better trade-off between rendering quality and on-device inference speed.
Table 5: Decoder Network Size. Our approach maintains much better tradcoff between inference speeds v/s rendering quality, with our smallest network achieving comparable quality the MobileR21. Benchmarking done on an iPhone 13. L is network depth, and W is network width.
Method
PSNR f
Latency
Storage
FLOPS
15-L. W-256 MobileR21.
27.69
14.54 ms
2.4
12626M
30-L W-128 MobileR2I.
27.54
14.47 ms
1.4 MB
8950M
30-I. W-256 MobileR21.
29.21
18.59
4.5 MB
23112M
60-L W-256 MobileR2L
30.34
22.65
8.2 MB
42772M
15-L W-256 LightSpeed
30.37
14.94
10.5 MB
12833M
30-L. W-128 LightSpeee
30.13
14.86
9.5 MB
9065M
30-L W-256 LightSpeed
31.70
20.35 ms
12.6 MB
23319M
60-L W-256 LightSpeed
32,34
26.47 ms
16.3 MB
42980M
Ray-Space Grid Encoding. We provide an ablation Tab. below on how the proposed ray-space grid encoder helps as compared to just using the light-slab representation with traditional frequency encoder. We compare different LightSpeed configurations with grid-encoder and frequency encoders. Networks are trained for 200k iterations on full-resolution 800 800 Lego sub-scene from Synthetic

---

360 dataset. Further, we show the training dynamics of all the trained variants in Fig. 5 (red and green plots). As claimed, our approach offers better visual fidelity and training dynamics (iterations to reach target PSNR) for both computationally cheaper small networks as well as full sized networks.
Table 6: Effect of using a Ray-Space Grid Encoder. We demonstrate the effect of using grid-based LightSpeed by comparing with a frequency encoded variant (no grid). L is network depth, and W is network width.
Method
PSNR
15-L W-256 LS (PE)
28.84
30-L W-256 LS (PE)
30.63
60-I. W-256 L$ (PE)
32.16
15-L W-256 LS (Grid)
30.37
30-L W-256 LS (Grid)
31.70
60-L W-256 LS (Grid)
32.34
Comparison with Plücker Representation. Given the challenges of discretizing Plücker representation, we compare between using positionally encoded Plücker coordinates and our grid- based light-slab approach in Tab. below for different network sizes to demonstrate the effectiveness of our approach. We train all models for 200k iterations on one 800 800 Lego sub-scene. We also share training curves for the variants in question Fig. (red and blue curves). As claimed, our integrated approach performs better in terms of training time and test-time visual fidelity for large and small models (having less computational costs) alike whereas the Plücker-based network shows a sharp decline in visual fidelity and increased training times to reach target test PSNR network size is reduced.
Table 7: Light-Slab Grid Representation vs. Plücker Coordinates. We compare the light-slab based LightSpeed (LS) with a positionally encoded variant of the Plücker ray representation. is network depth, and W is network width.
Method
PSNR
15-L W-256 Plücker
28.65
30-L W-256 Plücker
30.84
60-L W-256 Plücker
32.14
15-L W-256 LS
30.37
30-L W-256 LS
31.70
60-L W-256 LS
32.34
5 Discussion and Conclusion
In this paper, we propose an efficient method, LightSpeed, to learn neural light fields using the classic two-plane ray representation. Our approach leverages grid-based light field representations to accelerate light field training and boost rendering quality. We demonstrate the advantages of our approach not only frontal scenes but also non-frontal scenes by following divide-and-conques strategy and modeling them as frontal sub-scenes. Our method achieves SOTA rendering quality amongst prior works same time providing significantly better trade-off between rendering fidelity and latency, paving the way for real-time view synthesis on assource-constrain mobile devices.
Limitations. While LightSpeed excels at efficiently modeling frontal and 360 light fields, it currently lacks the capability to handle free camera trajectories. The current implementation does not support refocusing, anti- alii-aliasing, and is limited to static scenes without the ability to model deformable objects such as humans. We plan to explore these directions future work.
Broader Impact. Focused on finding efficiencies in novel view synthesis, our study could significantly reduce costs, enabling wider access to this technology. However, potential misuse, like unsolicited impersonations, must be mitigated.
10

---

References
[1] ngp pl. httpp:///gtaabb..oom/wwe////////
[2] Benjamin Attal, Jia-Bin Huang, Michael Zollhöfer, Johannes Kopf, and Changil Kim. Learning neural light fields with ray-space embedding. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19819-19829, 2022. 2,3, 4,5
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hodman. Mip-nerf 360:
Unbounded anti-aliased neural radiance fields. CVPR, 2022. 6, 16
[4] Junli Cao, Huan Wang, Pavlo Chemerys, Vladislav Shakhrai. Ju Hu, Yun Fu, Denys Makoviichuk, Sergey Tulyakov, and Jian Ren, Real-time neural light field on mobile devices. arXiv preprint arXiv:2212.080577 2022. 2, 3, 4, 6, 7, 8, 15,
[5] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, 2022. 1,3
[6] Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Taglissacchi. Mohilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. arXiv preprint arXiv:2208. 2000222080002277 00277, 2022. 1, 4, 7, 15
[7] Inchang Choi, Orazio Gallo, Alejandro Troccoli, Min H Kim, and Jan Kautz. Extreme view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7781-7790, 2019. 3
[8] Brandon Yushan Feng and Amitabh Varshney. Signet: Efficient neural representation for light fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14224-1423,, October 2021.
[9] John Flynn, Michael Broxton, Paul Dehevec, Matthew DuVall, Graham Fyffe, Ryan Overheck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2367-2397 2019. 2,3
[10] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes for radiance fields in space, time, and appearance, 2023. 1, 2, 6
[11] Andreï Gershun. The light field. Journal of Mathematics and Physics, 18(144)51--51, 1939. 2
[12] Steven Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F Cohen. The lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 43 54, 1996. 2,
[13] Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. ICCV, 2021. 7,8,
[14] Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian linear units. CoRR, abs/1606.08415, 2016. URL http://arxii http:///aaarivv../aaaaaaaa www./oo.//////.....oo//... http//1666084440 14
[15] Sergey Loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1500200 167, 2015. URL battt::/aaate /wwpiv..vvivv////// org/ watb.////..... 03167. 14
[16] Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics (TOG), 35(6):1-10, 2016. 2,
[17] Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Mitra. Relu fields: The little non-linearity that could. Transactions on Graphics (Proceedings of SIGGRAPH), volume 41, number = 4, year 2022, month = july, pages = 1-::8,, doi = 10. 00001144//52823.35530000
1, 2,
[18] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014. 14
[19] Mare Levoy and Pat Hanrahan. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 31-42, 1996. 2, 3.5
[20] Zhong Li, Liangchen Song, Celong Liu, Junsong Yuan, and Yi Xu. Neulf: Efficient novel view synthesis with neural 4d light field. In Earographics Symposium on Rendering, 2022. 3
[21] D. B. Lindell, J. N. P.* Martel, and G. Wetzstein. Autoint: Automatic integration for fast neural volume rendering, In Proc. CVPR, 2021.

---

[22] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon Nima Khademi Kalantari, Ravi Ramamoorthi. Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines, ACM Transactions on Graphicx (TOG), 2019. 2, 6, 14
[23] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65 (1):99 106, 2021. 1, 3, 6, 7, 8, 15, 16
[24] Parry Moon and Domina Eberle Speneer. Theory of the photic field. Journal of the Franklin Institute, 255 (1):33-50, 1953. 2
[25] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Gruph,, (1(44:100:11102::15 July 2022. doi: 10.1145/
3528223.3530127, URL https: http///i..o////////////// 1145/35282223 3530127. 3, 6, 8, 16
[26] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg Mueller, Chakravarty Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger. Donerf: Towards real-tine time rendering of compact neural radiance fields using depth oracle networks. In Computer Graphics Forum, volume 40, pages 45-59, Wiley Online Library, 2021. 3
[27] Muric-Julie Rakotosaona, Fabian Manhardt, Diego Martin Arroyo, Michael Nicmeyer, Abhijit Kundu, and Federico Tombari. Nerfmeshing: Distilling neural radiance fields into geometrically 3d meshes. arXiv preprint
2023. 3, 7,8
[28] Christian Reiser, Songyou Peng. Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mips. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 14335- 2021. 1,3
[29] Sara Rojas, Jesus Zarzar, Juan Camilo Perez. Artsiom Sanakoyeu, Ali Thabet, Albert Pumarola, and Bernard Ghanem. Re-rend: Real-time rendering of nerfs across devices. arXiv preprint arXiv:2303.0877 08711, 2023. 3,
[30] Sara Fridovich-Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022. 2,3
[31] Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Proc. NeurlPS. 2021. 2,
[32] Pratul P Srinivasan, Tongzhou Wang, Ashwin Sreeial, Ravi Ramamoorthi, and Ren Ng. Learning to synthesize a 4d rgbd light field from a single image. In Proceedings of the IEEE International Conference Computer Vision, pages 2243-2251, 2017. 2.3
[33] Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. Variable bitrate neural fields. ACM SIGGRAPH 2022 Conference Proceedings, pages 1-9, 2022. 3
[34] Jiaxiang Tang, Hang Zhou, Xiaokang Chen, Tianshu Hu, Errui Ding, Jingdong Wang. and Gang Zeng. Delicate textured mesh recovery from nerf via adaptive surface refinement. arXiv preprint arXiv:2303.0209) 2022. 3,3
[35] Ziyu Wan, Christian Richardt, Aljaž Božič, Chao Li, Vijay Rengarajan, Sconghyeon Nam, Xiaoyu Xiang, Tuotuo Li, Bo Zhu, Rakesh Ranjan, et al. Learning neural duplex radiance fields for wall-tmme view synthesis. arXiv preprint arXiv:2304. 10537, 2023. 1,3
[36] Huan Wang, Jian Ren, Zeng Huang, Kyle Olszewski, Menglei Chai, Yun Fu, and Sergey Tulyakov. R21: Distilling neural radiance field to neural light field for efficient novel view synthesis. ECCV, 2022. 2, 3,
[37] Peng Wang. Yuan Liu, Guying Lin, Jiatao Gu, Lingjie Liu, Taku Komura, and Wenping Wang. Progressively-connett light field network for efficient view synthesis. arXiv preprint arXiv:2207.04466 2022. 3
[38] Gaochang Wu, Belen Masia, Adrian Jarabo, Yuchen Zhang, Liangyong Wang, Qionghai Dai, Tianyou Chai, and Yebin Liu. Light field image processing: An overview. IEEE Journal of Selected Topics in Signal Processing, 11(7):926-95) 954, 2017. 2.3
12

---

[39] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P Srinivasan, Richard Szeliski, Jonathan T Barron, and Ben Mildenhall. Bakedsdf: Meshing neural sdfs for real-time view synthesis. arXiv preprint arXiv:2302. 14859, 2023. 1,3
[40] Alex Yu, Ruilong Li. Matthew Tancik, Hao Li. Ren Ng. and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. ICCV, 2021. 1.3
[41] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018. 2,
13

---

A Training Details
Network architecture. Our multi-scale feature grids have 16 levels, with resolutions exponentially growing from 16 256, and 4-D features every grid. Our LightSpeed network follows a similar architecture to MobileR2L: 60 point-wise residual convolutions with 256 channels and BatchNorm [15]and GeLU [14] activation interleaved. The convolutions are followed by 3 super-resolution modules to upsample the low-resolution input to the desired resolution. The first two super-resolution modules upsample the input by 2x and consist of transposed convolution layers with 4 x kernel size followed by 2 residual convolution layers cach. The third super-resolution module consists of transposed kernel size with 4 x kernel size (upsample by 2x) for 360 scenes (both bounded and unbounded) and 3 x 3 kernel size (upsample by 3x) for forward-facing [22] scenes.
Training details. We use Adam [18] optimizer with batch size of 32 to train the feature grids and decoder network. We use an initial learning rate of le-5 with 100 warmup steps taking the learning rate to 5e-4. Beyond that, the learning rate decays linearly until the training finishes. All our experiments are conducted on Nvidia V100s and A A100s.
B More Ablation Analysis
We discuss two aspects of dividing non-frontal scenes into separate light fields: the number of parts to divide the scene into and the placement of the splitting planes. We find the optimal number of splits for 360 scenes to be 5 since more number of splits would mean increased storage cost, which is detrimental to mobile deployment. We also want the scene splits to be collectively exhaustive (but mutually exclusive to maintain continuity while switching from one light field to another) in the poses sampled around the object. Consequently, fewer planes would mean placing the splitting planes near the scene origin cover the entire scene, which starts to violate the frontal assumption for each sub-scene.
Choice of Splitting Planes.
Given poses distributed on the surface of sphere with radius propose assigning each pose to (possibly multiple) sub-scenes based on the camera origin satisfying one or more of the following criteria:
0
V2
2
0
V2
-
22
V2
Z
T
2
22 -
V
V2 1
These five hyperplanes form the surface of a near-isometric trapezoidal prism, as shown in Fig. 3 (main paper). We experimentally show the effect of the choice of splitting plane by training LightSpeed models on Lego sub-scene with different plane placements and compare with the corresponding MobileR21. models trained on the same data. Specifically, we choose two axis-aligned planes at a distance of radius and radius from the scene origin and train models with 6k pscudo data points sampled independently from the two resulting sub-scenes. As shown in Tab. 8, placing the splitting plane at distance of radius results in inferior performance as compared to placing
v33
the splitting plane at a distance of radsus from the origin. This suggests that frontal sub-scene /2
approximation starts to break down as we move the splitting plane closer to the origin.
Table 8: Choice of Splitting Planes. We experiment with two planes parallel to the x-y sub-space at different distances. Splitting planes further from the origin work better empirically maintaining the frontal sub-scene assumption.
LF Representation
PSNR
SSIM f
LPIPS $
radius 1/2
30.44
0.9903
0.028
radius //3
30.23
0.9899
0.031
14

---

C Per-Scene Quantitative Results
We provide per-scene quantitative comparison between LightSpeed. MobileR2L [6] and NeRF [23] on the synthetic 360 dataset (Tab. 9, Tab. 10, and Tab. 11) and forward-facing dataset (Tab. 12, Tab. 13, and Tab. 14). We use PSNR, LPIPS, and SSIM as comparison metries. As can be seen from the comparisons, LighSpeed (our approach) outperforms MobileR2L [4] on almost all the metrics. Further, LightSpeed performs comparably or even better than NeRF [23]. We also provide additional zoom-in comparisons between LightSpeed and MobileR2L in Fig. 7.
Table 9: Per-scene PSNR comparison on the Synthetic 360 dataset between NeRF [23]. MobileR2L
[4], and our approach.
Method
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Average
NeRF[23]
33.00
25.01
30.13
36.18
32.54
29.62
32.91
28.65
31.01
MobileR2L [4]
33.66
25.05
29.80
36.84
32.18
30.54
34.37
28.75
31.34
LightSpeed (Ours)
34.21
25.63
32.82
36.77
34.35
29.51
35.65
28.90
32.23
Table 10:
Per-scene SSIM comparison on the Synthetic 360 dataset between NeRF [23].
MobileR2L [4], and our approach.
Method
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Average
NcRF[23]
0.967
0.925
0.964
0.974
0.961
0.949
0.980
0.856
0.947
MobileR2L [4]
0.998
0.986
0.996
0.998
0.992
0.992
0.997
0.982
0.993
LightSpeed (Ours)
0.998
0.988
0.998
0.998
0.994
0.990
0.998
0.984
0.994
Table 11: Per-scene LPIPS | comparison the Synthetic 360 MobileR2I. [4], and our approach.
dataset between NeRF [23],
Method
Chair
Drums
Fieus
Hotdog
Lego
Materials
Mic
Ship
Average
NeRF[23]
0.046
0.091
0.044
0.121
0.050
0.063
0.028
0.206
0.081
MobileR2L [4]
0,027
0.083
0.025
0.026
0.043
0.029
0.012
0.162
0.051
LightSpeed (Ours)
0.017
0.061
0.016
0.023
0.019
0.030
0.007
0.138
0.039
Table 12:
Per-scene PSNR t comparison the forward-facing dataset between NeRF [23].
MobileR2L [4], and our approach.
Method
Room
Femn
Leaves
Fortress
Orchids
Flower
T-Rex
Horns
Average
NeRF[23]
32.70
25.17
20.92
31.16
20.36
27.40
26.80
27.45
26.50
MobileR2L [4]
32.09
24.39
20.52
30.81
20.06
27.61
26.71
27.01
26.15
LightSpeed (Ours)
32.32
25.05
21.01
31.45
20.33
27.88
26.93
27.04
26.50
Table 13: Per-scene SSIM comparison the forward-facing dataset between NeRF [23].
MobileR2L. [4], and our approach.
Method
Room
Femn
Leaves
Fortress
Orchids
Flower
T-Rex
Horns
Average
NeRF[23]
0.948
0.792
0.690
0.881
0.641
0.827
0.880
0.828
0.811
MobileR2I.
0.995
0.973
0.923
0.995
0.916
0.971
0.973
0.982
0.966
LightSpeed (Ours)
0.991
0.976
0.931
0.996
0.921
0.972
0.975
0.983
0.968
15

---

Table 14: Per-scene LPIPS 4 comparison on the forward-facing dataset between NeRF [23].
MobileR2I. [4], and our approach.
Method
Room
Fern
Leaves
Fortress
Orchids
Flower
T-Rex
Horns
Average
NeRF[23]
0.178
0.280
0.316
0.171
0.321
0.219
0.249
0.268
0.250
MobilcR2L, [4]
0.088
0.239
0.280
0.103
0.296
0.150
0.121
0.217
0.187
Legd (Ours)
0.085
0.211
0.255
0.093
0.272
0.145
0.119
0.209
0.173
D Limitations
Results on Unbounded Scenes. The rendering fidelity of LightSpeed is closely tied to the performance of the corresponding NeRF teacher. LightSpeed uses Instant NGP [25] teachers for both bounded and unbounded scenes to maintain experimental consistency. We would like to highlight that Instant-NGP introduces the artifacts to unbounded scenes, which are carried forward LightSpeed via the mined pseudo-data. We share some of the pseudo-data images from Instant-NGP in Fig. 6. MipNeRF360 [3] specifically uses space contraction techniques model the unbounded nature the scene and deal with blurriness in the renderings. It further introduces a distortion iisttrtion-bbaed regularizer to remove floater artifacts and prevent background collapse. The techniques introduced by MipNeRF360 tackle the same type of artifacts pointed out in Fig. 6. Hence, using MipNeRF360 teachers will mitigate both these issues and could boost the visual fidelity on unbounded scenes for LightSpeed.
Figure 6: Instant NGP Failure Cases for Unbounded Scenes. Such artifacts carry over to LightSpeed, affecting visual fidelity on unbounded scenes.
16

---

misi
4
Ppeeer
(a) Scene
(b) Ground truth (c) MobileR2L
(d) LightSpeed
Figure 7: Qualitative Results frontal and non-frontal scenes. Zoomed-in comparison between
M1 and our LightSpeed approach.
17


Reviews:
[{"review_id": "jKxJuUvmI5", "reviewer": "Reviewer_bQpx", "summary": "The paper introduces LightSpeed, a method aimed at simplifying real-time novel-view image synthesis on mobile devices, which typically face constraints related to computational power and storage. By adopting the traditionally underutilized 4D light-slab (two-plane) representation for learning a neural light field, LightSpeed offers a more compact and efficient ray representation. While the light-slab representation has its limitations, mainly being designed for frontal views, the paper demonstrates a way to extend it for non-frontal scenes using a divide-and-conquer strategy. ", "questions": "-In line 175, it is unclear how the authors decided on the locations for the two planes, P1 and P2. Could the authors provide clarification on their choice of plane locations?\n\n-It is also not mentioned whether positional encoding was utilized before feeding data into the network. Could the authors specify if this step was incorporated in their method?", "limitations": "The paper presents a noteworthy approach by merging neural light field representation with grid representation. However, it primarily rehashes existing methods, with limited novel contribution. Experimental results exhibit limitations, and concerns about the feasibility of real-time rendering on mobile devices persist. Therefore, considering these constraints, I suggest a borderline reject for this paper in its current state.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes a promising approach that integrates neural light field representation with grid representation.", "weaknesses": "- The Contribution Context: While the methodology primarily integrates existing techniques like the neural light field method and grid representation from k-plane[9] and tensorf[5], it lacks a direct comparison or detailed discussion on the light-slab representation versus the Plücker coordinate representation[26] for non-frontal scenes. Given that the Plücker coordinate representation might achieve similar results, an experimental comparison would provide a more definitive understanding of the actual performance improvements, if any, achieved by their proposed method. Although such an integrated approach has its merits, the overall performance improvement appears to be marginal without these comparative analyses.\n\n- Coverage of Related Work: The paper mentions another neural light field method using light-slab representation. However, other related and potentially influential works such as \"Neulf: Efficient novel view synthesis with neural 4D light field, EGSR 2022\" and \"Signet: Efficient neural representation for light fields, ICCV 2021\" have been overlooked.\n\n- Limited Results: The experimental results presented have some limitations. Notably, the occurrence of the 'jelly effect', particularly in unbounded scenes, implies that the proposed method could benefit from further optimization. Furthermore, the synthetic scene experiments were conducted at a resolution of 400 x 400, as highlighted in Figure 1 and the ablation study. It remains uncertain how the proposed method would perform at the dataset's original resolution of 800 x 800. This higher-resolution evaluation could provide a more thorough understanding of the method's capabilities.\n\n- Feasibility for Real-Time Rendering: The paper claims real-time rendering feasibility on mobile devices, but it does not provide sufficient evidence such as measured MACs or a real-time rendering video demonstration. The computational cost of the 30-layered decoder plus 6-plane feature query might prove too expensive for the intended mobile applications. Further investigations should be conducted to substantiate these claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**We thank ther reviewer for the valuable feedback. We appreciate that the reviewer finds our approach of  integrating light fields with grid-based representations noteworty and promising. We address the concerns in the following. We hope our response can further demonstrate the strengths and real-time feasibility of our method.**\n\n\n***\n**Q1. About issues for Plücker representation discretization.** \n\nPlücker representation lies in the projective 5D space, presenting challenges for discretization and grid-based learning. Even if we ignore the projective nature, discretization results in a 5D ray space which (in both original and decomposed form) has a bigger storage cost as compared to its light-slab counterpart. Given the target devices are mobile, storage shall be as limited as possible.\n\n***\n**Q2. Comparison with Plücker representation.** \n\nGiven the challenges of discretizing Plücker representation, we show a comparison between using positionally encoded Plücker coordinates and our grid-based light-slab approach in Tab. A below for different network sizes to demonstrate the effectiveness of our approach. We train all models for 200k iterations on one Lego sub-scene at the *full 800x800 resolution*. We also share training curves for the variants in question in Fig. 2 of rebuttal PDF (red and blue curves). As claimed, our integrated approach performs better in terms of training time and test-time visual fidelity for large and small models (having less computational costs) alike whereas the Plücker-based network shows a sharp decline in visual fidelity and increased training times to reach a target test PSNR as network size is reduced. \n\n>Table A: **Light-Slab Grid Representation vs. Plücker Coordinates:** We compare the light-slab based LightSpeed (LS)  with a positionally encoded variant of the Plücker ray representation.\n>| Method  | PSNR $\\uparrow$|\n>| :------------ | :-----------: | \n>| 15-L W-256 Plücker | 28.65  | \n>| 30-L W-256 Plücker | 30.84  | \n>| 60-L W-256 Plücker | 32.14  | \n>| 15-L W-256 LS | 30.37 |\n>| 30-L W-256 LS | 31.70 |\n>| 60-L W-256 LS | 32.34|\n>\n\n***\n**Q3. Full-resolution ablation.** \n\nOur evaluations in Tab. 1 of the main paper (cropped version as Tab. B below) are conducted at *full resolution*. We further show visual fidelity and on-device latency tradeoff at full-resolution in Tab. C below. LightSpeed maintains a significantly better tradeoff as compared to MobileR2L on full resolutions scenes as well.\n\n\n\n>Table B: **Quantitative Comparison**: on Forward Facing and Synthetic $360^\\circ$ scenes. \n>| Method  | Synthetic $360^\\circ$ PSNR $\\uparrow$| LLFF PSNR $\\uparrow$ |\n>| :------------ | :-----------: | :-----------: | \n>| NeRF               | 31.01 |26.50| \n>| NeRF-PyTorch       | 30.92 |26.26| \n>| SNeRG              | 30.38 |25.63| \n>| MobileNeRF         | 30.90 |25.91|\n>| MobileR2L          | 31.34 |26.15|\n>| LightSpeed (Ours)  | **32.23** | **26.50**|\n\n\n>Table C: **Full-Resolution Fidelity-Latency Tradeoff**: LightSpeed (LS) maintains a much better tradeoff than MobileR2L (MR2L). Benchmarking done on an iPhone 13 with full-resolution images. L is network depth, and W is network width.\n>| Method  |  PSNR $\\uparrow$| Latency $\\downarrow$| FLOPs $\\downarrow$|\n>| :------------ | :-----------: | :-----------: | :-----------: | \n>| 15-L W-256 MR2L | 27.69 | 14.54 ms | 12626M |\n>| 30-L W-128 MR2L | 27.54 | 14.47 ms | 8950M |\n>| 30-L W-256 MR2L | 29.21 | 18.59 ms | 23112M |\n>| 60-L W-256 MR2L |30.34 | 22.65 ms | 42772M |\n>| 15-L W-256 LS | 30.37 | 14.94 ms | 12833M |\n>| 30-L W-128 LS | 30.13 | 14.86 ms | 9065M |\n>| 30-L W-256 LS | 31.70 | 20.35 ms | 23319M |\n>| 60-L W-256 LS | 32.34 | 26.47 ms | 42980M |\n\n\n***\n**Q4. On-device feasibility.** \n\nWe report the FLOPs for our method and MobileR2L in Tab. C. We demonstrate *real-time latency numbers* obtained from benchmarking the LightSpeed framework *directly on mobile devices* using the Xcode from Apple. Real-time demo to be released with the code later. Specifically, using a 30-layered decoder plus a six-plane feature query has almost half the operations of full-sized network from MobileR2L, both of which run on mobile devices.\n\n***\n**Q5. Location for planes P1 and P2.** \n\nWe use the same NDC trick as leveraged by NeRF to project rays to the NDC space: project the scene's near plane to z = -1 and the plane at infinity to z = 1. We use these projected planes (z=+-1) as planes P1 and P2.\n\n***\n**Q6. Use of positional encoding.** \n\n*No positional encoding* is utilized before feeding the grid encodings to the decoder network. Grid-based representations and positional encodings offer alternative ways to provide interpolation capabilities to the network, and hence, using one of them is sufficient.\n"}, {"review_id": "s0Pr66MpQi", "reviewer": "Reviewer_XtjZ", "summary": "Real-time novel-view image synthesis on mobile devices is challenging due to limited computational power and storage. Volumetric rendering methods are unsuitable due to their high computational cost. The authors propose using the efficient light slab representation for learning a neural light field, which achieves better rendering quality and a favorable trade-off between quality and speed compared to prior light field methods.", "questions": "N/A", "limitations": "Yes", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes to use the light slab representation for learning a neural light field, which has not been used significantly in the literature before. The proposed method using light slab presentation is shown to perform better than the SOTA while providing a computational advantage over the existing methods.", "weaknesses": "NA", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**We thank the reviewer for their feedback and strong rating. As summarized in the review, we propose a novel real-time view-synthesis method that is based on light fields. We leverage previously overlooked 4D light-slab representation for easy discretization and grid-based representations for neural light field learning. Our grid-based neural light field obtains a significant boost in training speed and performs better than exsiting works. Our approach further provides a computational advantage over existing methods in the form of an excellent tradeoff between rendering fidelity and on-device latency, paving the way for easy deployment to mobile devices.**"}, {"review_id": "eYKOfu8qjR", "reviewer": "Reviewer_BeXQ", "summary": "This paper introduces LightSpeed, which uses traditional 4D light-slab representation and merges the super-resolution network proposed by MobileR2L. LightSpeed uses the NeLF method and will be primarily implemented on mobile.", "questions": "I think the storage problem is not the most significant in the rendering problem in mobile. The mobile device can upload data to the cloud to solve this problem. RealityScan adopts this method and can obtain similar results based on NeRF method. So what is the most significant strength of LightSpeed for a user who can use cloud storage? Moreover, since MobileNeRF can achieve real-time manipulation, can LightSpeed achieve similar effects? Besides, demonstrating more data will make this paper more convincing, like experimenting on more different chips.", "limitations": "please see weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Originality: Utilize the overlooked method of 4D light-slab representation and merge the good method of saving memory and time from other NeLF methods (The super-resolution network proposed by MobileR2L). Extend the application scenes of light-slab representation to non-frontal scenes using the divide-and-conquer strategy.\n\nQuality: Greatly saves the storage compared with other methods and balances the reconstruction quality and storage well. \n\nClarity: Clearly show the results and the advantages of LightSpeed.\n\nSignificance: Advance the general application of the NeLF method on mobile.\n", "weaknesses": "Since LightSpeed was proposed to solve the reconstruction problem in mobile, this paper does not show enough data for different mobile platforms. This paper will be more convincing with more data in different chips or multi-platform.\n\nThis paper only shows the examples of unbounded datasets in Fig.4, and more comparisons of other datasets should be shown to be more convincing. The data demonstrated now keeps me in doubt about the effects.\n\nTable 3 should be placed in section 4.2 rather than 4.3.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank the reviewer for their positive feedback and valuable suggestions. We appreciate that the reviewer finds our work original, explained clearly and of significance towards light field methods for mobile devices. We address concerns via more visual and on-device results and hope our response demonstrates the strengths of our approach.**\n\n***\n**Q1. Latency on different chipsets.** \n\nWe compare the latency of our approach with MobileR2L on 6 different chipsets as shown in Tabs. D-1 and D-2 below. We obtain competitive latency numbers for our full-sized LightSpeed network, and much better latency for our 30-layered network (on all devices), which has better rendering fidelity than full-sized MobileR2L as shown in Tab. D-3 below.\n\n\n> Table D-1: **Rendering Latency Analysis on LLFF Scenes**: LightSpeed maintains a competitive rendering latency (ms) to prior works. \n>| Chip  |  MobileR2L| Ours | Ours (30-L) |\n>| :------------ | :-----------: | :-----------: | :-----------: | \n>| Apple A13 (Low-end)  | 40.23 | 41.06 |  32.29|\n>| Apple A15(Low-end)  | 18.04 |  19.05 | 15.28|\n>| Apple A15(High-end) |  16.48 | 17.68 | 15.03|\n>| Apple A16  | 17.84 | 18.15 |  14.83 |\n>| Apple M1 Pro  | 17.65 | 17.08 | 13.86 |\n>| Snapdragon SM8450  | 39.14 | 45.65 | 32.89|\n\n\n>Table D-2: **Rendering Latency Analysis on Synthetic $360^\\circ$ Scenes**: LightSpeed maintains a competitive rendering latency (ms) to prior works. \n>| Chip  |  MobileR2L| Ours | Ours (30-L) |\n>| :------------ | :-----------: | :-----------: | :-----------: | \n>| Apple A13 (Low-end)  | 65.54 | 66.10 |  53.89|\n>| Apple A15(Low-end)  | 26.21 | 27.10|  20.15 |\n>| Apple A15(High-end) |  22.65 | 26.47 | 20.35|\n>| Apple A16  | 25.98 | 26.44 |  20.46 |\n>| Apple M1 Pro  | 27.37 | 27.14 | 20.13 |\n>| Snapdragon SM8450  | 40.86 | 41.26 | 33.87|\n\n\n\n>Table D-3: **Full-Resolution Fidelity-Latency Tradeoff**: LightSpeed (LS) maintains a much better tradeoff than MobileR2L (MR2L). Benchmarking done on an iPhone 13 with full-resolution images. L is network depth, and W is network width.\n>| Method  |  PSNR $\\uparrow$| Latency $\\downarrow$| FLOPs $\\downarrow$|\n>| :------------ | :-----------: | :-----------: | :-----------: | \n>| 15-L W-256 MR2L | 27.69 | 14.54 ms | 12626M |\n>| 30-L W-128 MR2L | 27.54 | 14.47 ms | 8950M |\n>| 30-L W-256 MR2L | 29.21 | 18.59 ms | 23112M |\n>| 60-L W-256 MR2L |30.34 | 22.65 ms | 42772M |\n>| 15-L W-256 LS | 30.37 | 14.94 ms | 12833M |\n>| 30-L W-128 LS | 30.13 | 14.86 ms | 9065M |\n>| 30-L W-256 LS | 31.70 | 20.35 ms | 23319M |\n>| 60-L W-256 LS | 32.34 | 26.47 ms | 42980M |\n\n***\n\n**Q2. Visual results.** \n\nWe show all video results in the supplementary material and comparison results on 4 different scenes in Fig. 4 and Fig. 1 of the main paper and supplementary material respectively. We further share results on more scenes in Fig. 1 of the rebuttal PDF to strenghten our claims. As claimed, LightSpeed is able to capture fine-level details better than previous state-of-the-art MobileR2L [a]. Competitive on-device runtimes and significantly better visual fidelity-latency tradeoff further demonstrate the strengths of our work.\n\n***\n**Q3. About the placement of the table.** \n\nThank you for pointing this out. We will make sure Table 3 is placed in Sec. 4.2 in the revised paper. \n\n***\n**Q4. Using cloud storage.** \n\nUploading LightSpeed models to the cloud would add additional latency to the test-time view-synthesis process, destroying the purpose of using a real-time rendering framework like LightSpeed. To use cloud storage for unnloading models off the mobile devices, we would still want the models to be small enough for fast on-loading and off-loading (bigger models would introduce more overhead). Additionally, privacy concerns and monetary costs also arise if models are stored in the cloud. \n\n***\n**Q5. Real-time scene manipulation.** \n\nSince light fields inherently do not model the scene geometry, scene manipulation is not currently possible with LightSpeed. There are possibilities for composing two light fields for scene manipulation, and we plan to explore this in future work. MobileNeRF uses explicit scene representation in the form of a mesh and hence can manipulate scenes.  However, MobileNeRF does not run on mobile devices for all scenes: it runs out of memory for complex scenes and hence presents a crucial drawback by not supporting different kinds of scenes. On the contrary, LightSpeed does not suffer from any such drawbacks and can easily handle complex scenes as well within on-device computational limits.\n\n***\n**References:**\n\n[a] Cao, Junli, et al. \"Real-Time Neural Light Field on Mobile Devices.\" CVPR. 2023.\n"}, {"review_id": "JSnc9CKhlY", "reviewer": "Reviewer_8eAH", "summary": "This paper presents the LightSpeed for real-time rendering on mobile devices. The approach involves replacing the commonly used Plücker coordinates with a light slab representation and implementing multi-level grids similar to Instant-NGP and k-planes. Additionally, It introduced a divide-and-conquer strategy to address the issue of light slab's inability to represent 360-degree objects effectively.  ", "questions": "Please address my questions in the weakness section.  ", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The proposed approach effectively improves the efficiency with several interesting design. Firstly, it uses light-slab to parameterize ray, which shows efficiency compared to the commonly used Plücker coordinate.  Besides, the practice of compressing 4D space through six sub-planes can also improve efficiency.  Finally, the divide-and-conquer strategy to solve the problem that traditional light-slab method fail to represent 360 degree objects. ", "weaknesses": "1. In the abstract and introduction sections, the authors repeatedly claim that existing methods overlook the light-slab parameterization. However, research has already been conducted on novel view synthesis based on light-slab parameterization. Besides Attal et al.  mentioned in line 69, there are some other literatures that are related but ignored, such as \nNeuLF: Efficient Novel View Synthesis with Neural 4D Light Field\nNeLF: Practical Novel View Synthesis with Neural Light Field\nProgressively-connected Light Field Network for Efficient View Synthesist\nAll these papers all based on light-slab parametrization and its variants, and thus should be carefully discussed or compared with.\n\n2. Some claims are hard to understand. For example, in lines 51-52, current ray parametrization will encounter issues with the introduction of grid-based representations. However, no specific problem is identified, and the subsequent description only explains that the existing method is \"redundant.\" Also, in line 57,  “the high-dimensional stratified-point representation is not feasible for grid-based discretization.”, it is unclear why it was not feasible. In my understanding, nerf is also a neural representation that needs sampling, and it can use the grid-based representation. \n\n3. In the ablation section, the authors only ablate the data requirements and the decoder network size, which is not sufficient. More in-depth discussions are needed to demonstrate the effectiveness of each part of the proposed method. For example, it would be great if the authors could make comparison between the grid-based approach and the pure MLP approach within their own framework, such as replacing the Ray-space Grid Encoder with a traditional frequency encoder or so on. Please carefully check the claimed contributions and ensure that all of them are supported by the experiments. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**We thank the reviewer for their positive review and feedback. We appreciate that the reviewer acknowledges the efficiency of our method and interesting design choices of using a light-slab ray paramterization with 4D grid compression via decomposition. We address the feedback provided by the reviewer in the following.**\n\n***\n**Q1. About the compatibility of other ray parameterizations with grid-based representations, L51-52, 57.** \n\n**(a). Plücker representation.** \nPlücker representation lies in the projective 5D space, presenting challenges for discretization and grid-based learning. Even if we ignore the projective nature, discretization results in a 5D ray space which (in both original and decomposed form) has a bigger storage cost as compared to its light-slab counterpart. Given the target devices are mobile, storage must be as limited as possible.\n\n**(b). Stratified ray representation.** \nFurther, the stratified ray representation used by R2L [a] and MobileR2L [b] can be potentially discretized in two ways: (1) defining three new dimensions for every point sampled along the ray, which results in an extremely overparameterized and *redundant* ray space with unusable storage costs, and (2) querying a spatial 3D grid for points sampled along the ray. Multiple queries of the 3D grid per ray as done by NeRF-based counterparts *increases run-time per pixel*, prohibiting real-time inference on mobile devices. Alternatively, the light-slab ray-space grid is compact and allows a *single grid query per ray/pixel* conducive to real-time inference. Hence, we find light-slab paramterization to be most effective and others presenting issues towards grid-based learning.\n\n***\n**Q2. Ablation on ray-space grid encoding.** \n\nWe provide an ablation in Tab. E below on how the proposed Ray-Space Grid Encoder helps as compared to just using the light-slab representation with a traditional frequency encoder. For the purpose of this ablation, we train LightSpeed with grid-encoder and frequency encoders for 200k iterations with different network sizes and compare results on a full-resolution 800X800 Lego sub-scene from Synthetic $360^\\circ$ dataset. Further, we show the training dynamics for all the trained variants in Fig. 2 of rebuttal PDF (red and green plots). As claimed, our approach offers better visual fidelity and training dynamics (iterations to reach a target PSNR) for both computationlly cheaper small networks as well as full sized networks. \n\n\n>Table E: **Effect of using a Grid Encoder**: We demonstrate the effect of using a grid-based LightSpeed by comparing with a frequency encoded variant (no grid). L and W refer to network depth and width respectively.\n>| Method  | PSNR $\\uparrow$|\n>| :------------ | :-----------: | \n>| 15-L W-256 LS (PE) | 28.84          | \n>| 30-L W-256 LS (PE) | 30.63          | \n>| 60-L W-256 LS (PE) | 32.16          | \n>| 15-L W-256 LS (Grid) |30.37 |\n>| 30-L W-256 LS (Grid) |31.70 |\n>| 60-L W-256 LS (Grid) | 32.34|\n\n***\n**Q3. Full-resolution ablation.** \n\nWe show visual fidelity and on-device latency tradeoff at *full-resolution* in Tab. F below. We also report FLOP values as an indicator of computational resources required at run-time. LightSpeed maintains a significantly better tradeoff as compared to MobileR2L on full resolutions scenes as well.\n\n\n>Table F: **Full-Resolution Fidelity-Latency Tradeoff**: LightSpeed (LS) maintains a much better tradeoff than MobileR2L (MR2L). Benchmarking done on an iPhone 13 with full-resolution images. L is network depth, and W is network width.\n>| Method  |  PSNR $\\uparrow$| Latency $\\downarrow$| FLOPs $\\downarrow$|\n>| :------------ | :-----------: | :-----------: | :-----------: | \n>| 15-L W-256 MR2L | 27.69 | 14.54 ms | 12626M |\n>| 30-L W-128 MR2L | 27.54 | 14.47 ms | 8950M |\n>| 30-L W-256 MR2L | 29.21 | 18.59 ms | 23112M |\n>| 60-L W-256 MR2L |30.34 | 22.65 ms | 42772M |\n>| 15-L W-256 LS | 30.37 | 14.94 ms | 12833M |\n>| 30-L W-128 LS | 30.13 | 14.86 ms | 9065M |\n>| 30-L W-256 LS | 31.70 | 20.35 ms | 23319M |\n>| 60-L W-256 LS | 32.34 | 26.47 ms | 42980M |\n\n\n\n***\n**References:**\n\n[a] Wang, Huan, et al. \"R2l: Distilling neural radiance field to neural light field for efficient novel view synthesis.\" ECCV. 2022.\n\n[b] Cao, Junli, et al. \"Real-Time Neural Light Field on Mobile Devices.\" CVPR. 2023.\n\n"}, {"review_id": "uBJMyIS7To", "reviewer": "Reviewer_CSSm", "summary": "This paper describes a novel representation for learning view synthesis from a set of input images with known camera poses.  They parameterize a classical two-slab 4D light field using a K-Planes representation (using 6 feature planes).  Feature queries are processed through many layers of 1x1 convolutions before being post-processed by a super-resolution network to produce the output image.  Similar to previous neural light field methods they augment the training data using virtual views rendered using a trained NeRF model.  In experiments on standard benchmark datasets they achieve good rendering quality compared to previous methods capable of rendering on mobile devices, and also achieve a fast training time and compact representation.", "questions": "How do the results compare to state-of-the-art (regardless of rendering speed)?  \n\nWhat is the effect of directly rendering a full-resolution image (rather than using the super-resolution network)?", "limitations": "They do describe limitations but I think more discussion of the 360 unbounded results would strengthen the paper.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The proposed approach of parameterizing a neural lightfield using the K-Planes concept is reasonable and novel to the best of my knowledge.  They show convincingly that this approach leads to faster convergence and a more compact representation compared to MobileR2L.\n\nThey also achieve high-quality rendering with fast rendering speeds even on mobile devices.  In some cases they actually outperform the teacher model.\n\nThey present an extensive evaluation using several benchmark datasets, including both synthetic and real data.  They also include an ablation study to consider the effect of virtual views and decoder network size.  They also compare implementations on different mobile processors in terms of latency.\n\nThe presentation is clear and easily understandable.", "weaknesses": "The results on unbounded scenes especially are lower quality than state-of-the-art (non-real-time) methods such as Mip-NeRF 360.  Artifacts such as blurriness and inconsistent shape are clearly evident in the result videos for unbounded scenes (but not for the object-centric scenes).  These limitations deserve discussion.\n\nThe approach of rendering a low-resolution image and then upsampling would seem to be a limiting factor in terms of rendering quality.  It would be informative to see what quality is possible with this representation when directly rendering the full-resolution image (without the super-resolution network).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**We thank the reviewer for their positive comments and insightful feedback. We appreciate that the reviwer acknowledges our approach to be novel with fast and high-quality renderings. We further note the reviewer finds evaluations extensive and the paper's presentation clear and easily understandable. In the following, we address the feedback and questions presented by the reviewer.**\n\n***\n**Q1. Regressing full-resolution image directly.** \n\nWe thank the reviewer for raising an insightful point! In our preliminary experiments, we found a *slight* increase in visual fidelity if we regress the full-resolution image directly. However, since rendering full-resolution images directly is not feasible on mobile devices in real-time (as pointed out in MobileR2L [a]) and the visual fidelity gains are marginal, we did not pursue this direction of experiments. \n\n\n***\n**Q2. About more comparisons.** \n\nOur method is tailored specifically for mobile devices, and achieves state-of-the-art rendering fidelity on both LLFF and Synthetic $360^\\circ$ scenes compared to prior methods in this domain (Tab. G below). Given the use of teacher NeRF methods to generate pseudo-data for light field training, we can potentially improve the rendering fidelity of LightSpeed by leveraging newer NeRF-based methods.\n\n\n>Table G: **Quantitative Comparison** on Forward Facing and Synthetic $360^\\circ$ scenes. \n>| Method  | Synthetic $360^\\circ$ PSNR $\\uparrow$| LLFF PSNR $\\uparrow$ |\n>| :------------ | :-----------: | :-----------: | \n>| NeRF               | 31.01 |26.50| \n>| NeRF-PyTorch       | 30.92 |26.26| \n>| SNeRG              | 30.38 |25.63| \n>| MobileNeRF         | 30.90 |25.91|\n>| MobileR2L          | 31.34 |26.15|\n>| LightSpeed (Ours)  | **32.23** | **26.50**|\n\n\n***\n**References:**\n\n[a] Cao, Junli, et al. \"Real-Time Neural Light Field on Mobile Devices.\" CVPR. 2023.\n"}]

Author rebuttal:
We sincerely thank all the reviewers for their thoughtful comments and appreciate their findings that our novel method simplifies real-time novel view synthesis on mobile devices while performing better than exisiting works with high-quality and fast rendering even on mobile devices (bQpx, XtjZ, CSSm, BeXQ). 


Specifically, our method is `noteworthy` in using a `traditionally underutilized light-slab representation` (bQpx) to formulate a grid-based ray-space for light field learning. This yields a`more compact and efficient ray representation` (bQpx) and replaces `the commonly used Plücker coordinates` (8eAH). We also `compress the ray space using 2D planes for efficienty` (8eAH). Our method further overcomes the limitations of light-slab representation by extending it to non-frontal $360^\circ$ scenes using a divide-and-conquer strategy (bQpx,BeXQ,8eAH). Our study provides `an extensive evaluation` (CSSm), demonstrating `clear results and advantages` (BeXQ) over prior methods. Further, our paper has `clear and easily understandable presentation` (CSSm). 

We will incorporate all suggestions and sincerely hope this will help reviewers finalize their judgements.


***
We address common questions from reviewers in the following and answer other questions in the individual responses. We also upload the author response PDF to include more comparison figures.

**Q1. To bQpx, 8eAH. About comparison with SIGNET, NeuLF, and ProLiF.** 

We thank the reviewers for pointing out these works. We propose a novel method to utilize grid-based representations for light field learning. To this end, we find the light-slab representation to be compact and easy to discretize for learning a grid-based ray space. To limit storage requirements, we decompose the 4D ray-space into six 2D planes. Further, we propose a divide-and-conquer strategy to utilize the light-slab representation, originally designed for frontal scenes, for modelling non-frontal scenes.

While all SIGNET, NeuLF, and ProLiFF use the light-slab representation, they differ significantly from LightSpeed. 

- SIGNET[a] explores a different problem of compressing a given light field using ultra spherical network input encoding, with no guarantee for photo-realistic view synthesis, while our method targets and *achieves* real-time photorealistic view synthesis, specifically on mobile devices. 
- NeuLF[b] fails to capture fine-level details of scenes due to the absence of any network input encoding and takes 70ms to render a 567x1008 frame on an RTX 3090. On the other hand, LightSpeed has no issues *learning fine-level detail*s, as shown in Fig. 4 in the main paper and *runs in real-time* on mobile devices.
- ProLiF[c] uses volumetric rendering to generate pixel values prohibiting real-time inference speeds. In contrast, LightSpeed runs on mobile-devices in *real-tim*e. 
 
We will discuss these works in detail in the revised paper.

***
**Q2. To bQpx, CSSm. About results on unbounded scenes.** 


The rendering fidelity of our approach (LightSpeed) is closely tied to the performance of the corresponding NeRF teacher. LightSpeed uses Instant NGP [d] teachers for both bounded and unbounded $360^\circ$ scenes to maintain experimental consistency. We would like to highlight that Instant NGP introduces the artifacts to unbounded scenes, which are carried forward to LightSpeed via the mined pseudo-data. We share some of the pseudo-data images from Instant-NGP in Fig. 3 of the rebuttal PDF. MipNeRF360 [e] specifically uses space contraction techniques to model the unbounded nature of the scene and deal with blurriness in the renderings. It further introduces a distortion-based regularizer to remove floater artifacts and prevent background collapse. The techniques introduced by MipNeRF360 tackle the same type of artifacts pointed out in the reviews. Hence, using MipNeRF360 teachers will mitigate both these issues and could boost the visual fidelity on unbounded scenes for LightSpeed.


***
**References:**

[a] Feng, Brandon Yushan, and Amitabh Varshney. "Signet: Efficient neural representation for light fields." ICCV. 2021.

[b] Li, Zhong, et al. "Neulf: Efficient novel view synthesis with neural 4d light field." arXiv. 2021.

[c] Wang, Peng, et al. "Progressively-connected Light Field Network for Efficient View Synthesis." arXiv. 2022.

[d] Müller, Thomas, et al. "Instant neural graphics primitives with a multiresolution hash encoding." ToG. 2022.

[e] Barron, Jonathan T., et al. "Mip-nerf 360: Unbounded anti-aliased neural radiance fields." CVPR. 2022.


Meta-review: The manuscript received a range of evaluations from the reviewers, including one strong accept, two weak accepts, one borderline accept, and one borderline reject. It introduces an approach to parameterizing a neural light-field using the K-Planes concept. Initial concerns were raised regarding the ablation comparisons with Plücker coordinates and performance across various mobile platforms. The author addressed these concerns by presenting additional experiments in the rebuttal. The AC recommends acceptance of the manuscript. However, the AC agrees with the sentiment that a video demonstration showcasing real-time performance would significantly enhance the paper's impact. Therefore, the lack of such a demonstration hinders the recommendation for spotlight or oral presentation.
Decision: Accept (poster)