SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training Details. We follow a similar training scheme as MobileR2L: train the LightSpeed model using pseudo-data mined from a pre-trained NeRF teacher. We specifically train MipNeRF teachers to sample 10k pseudo-data points for the LLFF dataset. For synthetic and unbounded  $360^{\circ}$  scenes, we mine 30k samples per scene using Instant-NGP [25] teachers. Following this, we fine-tune the model on the original data. We optimize for the mean-squared error between generated and ground truth images. We refer the reader to the supplementary material for more training details.
We use  $63 \times 84$  ( $12 \times$  downsampled from the desired  $756 \times 1008$  resolution) input ray bundles for the forward-facing scenes. For  $360^{\circ}$  scenes, we use  $100 \times 100$  ( $8 \times$  downsampled from the desired  $800 \times 800$  image resolution) ray bundles. For unbounded scenes, we use ray bundles  $12 \times$  downsampled from the image resolution we use. We train our frontal LightSpeed models as well as each sub-scene model in non-frontal scenes for  $200\mathrm{k}$  iterations.
Baselines and Metrics. We compare our method's performance on bounded scenes with MobileR2L[6], MobileNeRF[6] and SNeRG[13]. We evaluate our method for rendering quality using three metrics: PSNR, LPIPS, and SSIM. For unbounded scenes, we report the PSNR metric on 6 scenes and compare it with MobileNeRF [6] and NeRFMeshing [27]. To further demonstrate the effectiveness of our approach, we compare our approach with others on two other criteria: (a) On-device Rendering Speed: We report and compare average inference times per rendered frame on various mobile chips, including Apple A15, Apple M1 Pro and Snapdragon SM8450 chips; and (b) Efficient Training: We compare the number of iterations LightSpeed and MobileR2L require to reach a target PSNR. We pick Lego scene from  $360^{\circ}$  scenes and Fern from forward-facing scenes as representative scenes to compare. We also report the storage requirements of our method per frontal scene and compare it with baselines.
4.1 Results and Analysis
Rendering Quality. As in Tab. 1, we obtain better results on all rendering fidelity metrics on the two bounded datasets. We also outperform MobileNeRF and NeRFMeshing on 4 out of 6 unbounded  $360^{\circ}$  scenes. We refer the reader to Fig. 4 for a visual comparison of our approach with MobileR2L and NeRF. Our method has much better rendering quality, capturing fine-level details where MobileR2L, and in some cases, even the original NeRF model, fails. Note that we use Instant-NGP teachers for  $360^{\circ}$  scenes, which have slightly inferior performance to MipNeRF teachers used by MobileR2L. This further shows the robustness of our approach to inferior NeRF teachers.
Storage Cost. We report storage requirements in Tab. 1. Our approach has a competitive on-device storage to the MobileR2L model. Specifically, we require a total of  $16.3\mathrm{MB}$  of storage per frontal scene. The increase in storage is expected since we're using grids to encode our light field. We also report storage values for lighter LightSpeed networks in the ablation study (see Tab. 5), all of which have similar or better rendering quality than the full-sized MobileR2L network.
Training Speed. We benchmark the training times and the number of iterations required for LightSpeed and MobileR2L in Tab. 2 with a target PSNR of 24 for Fern scene and 32 for the Lego scene. Our approach demonstrates a training speed-up of  $2.5 \times$  on both scenes. Since we are modeling  $360^{\circ}$  scenes as a composition of 5 light fields, we can train them in parallel (which is not
Table 1: Quantitative Comparison on Forward-facing, Synthetic  $360^{\circ}$  and Unbounded  $360^{\circ}$  Datasets. LighSpeed achieves the best rendering quality with competitive storage. We use an out-of-the-box Instant-NGP [25] implementation [1] (as teachers for  $360^{\circ}$  scenes) which dose not report SSIM and LPIPS values. We omit storage for NeRF-based methods since they are not comparable.
Method	Synthetic 360°	Forward-Facing
PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓	Storage ↓
NeRF [23]	31.01	0.947	0.081	26.50	0.811	0.250	-
NeRF-PyTorch	30.92	0.991	0.045	26.26	0.965	0.153	-
SNeRG [13]	30.38	0.950	0.050	25.63	0.818	0.183	337.3 MB
MobileNeRF [6]	30.90	0.947	0.062	25.91	0.825	0.183	201.5 MB
MobileR2L [4]	31.34	0.993	0.051	26.15	0.966	0.187	8.2 MB
LightSpeed (Ours)	32.23	0.994	0.038	26.50	0.968	0.173	16.3 MB
Our Teacher	32.96	-	-	26.85	0.827	0.226	-
Unbounded 360°
Method	Bicycle	Garden	Stump	Bonsai	Counter	Kitchen
MobileNeRF [6]	21.70	23.54	23.95	-	-	-
NeRFMeshing [27]	21.15	22.91	22.66	25.58	20.00	23.59
LightSpeed (Ours)	22.51	24.54	22.22	28.24	25.46	27.82
Instant-NGP (Our teacher) [25]	21.70	23.40	23.20	27.4	25.80	27.50
Table 2: Training Time for Lego and Fern scenes with 32 and 24 target PSNRs. LightSpeed trains significantly faster than MobileR2L. It achieves even greater speedup when trained in parallel for  $360^{\circ}$  scenes (parallel training is not applicable for frontal scenes).
Method	Forward-Facing: Fern	Synthetic 360°: Lego
Duration ↓	Iterations ↓	Duration ↓	Iterations ↓
MobileR2L	12.5 hours	70k	192 hours	860k
LightSpeed	4 hours	27k	75 hours	425k
LightSpeed (Parallelized)	-	-	15 hours	85k
possible for MobileR2L), further trimming down the training time. Moreover, the training speedup reaches  $\sim 4\times$  when networks are trained beyond the mentioned target PSNR (see Fig. 1).
Inference Speed. Tab. 3 shows our method's inference time as compared to MobileR2L and MobileNeRF. We maintain a comparable runtime as MobileR2L while having better rendering fidelity. Since on-device inference is crucial to our problem setting, we also report rendering times of a smaller 30-layered decoder network that has similar rendering quality as the MobileR2L model (see Tab. 5).
Table 3: Rendering Latency Analysis. LightSpeed maintains a competitive rendering latency (ms) to prior works. MobileNeRF is not able to render 2 out of 8 real-world scenes ( $\frac{N}{M}$  in table) due to memory constraints, and no numbers are reported for A13, M1 Pro and Snapdragon chips.
Chip	Forward-Facing	Synthetic 360°
MobileNeRF	MobileR2L	Ours	Ours (30-L)	MobileNeRF	MobileR2L	Ours	Ours (30-L)
Apple A13 (Low-end)	-	40.23	41.06	32.29	-	65.54	66.10	53.89
Apple A15(Low-end)	27.15	18.04	19.05	15.28	17.54	26.21	27.10	20.15
Apple A15(High-end)	20.98	16.48	17.68	15.03	16.67	22.65	26.47	20.35
Apple M1 Pro	-	17.65	17.08	13.86	-	27.37	27.14	20.13
Snapdragon SM8450	-	39.14	45.65	32.89	-	40.86	41.26	33.87
Data Requirements. We use 10k samples as used by MobileR2L to train LightField models for frontal scenes. However, for non-frontal scenes, we resort to using 30k pseudo-data samples per
(a) 15-Layer Networks
(b) 30-Layer Networks
Figure 5: Test PSNR v/s Training Iterations. We compare test set PSNR obtained by LightSpeed (Grid)(ours), LightSpeed (frequency encoded), and Plücker-based neural light field as the training progresses for 3 different network configurations.
(c) 60-Layer Networks
scene. Dividing 10k samples amongst 5 sub-scenes assigns too few samplers per sub-scene, which is detrimental to grid learning. We experimentally validate data requirements by comparing MobileR2L and LightSpeed trained for different amounts of pseudo-data. We train one  $400 \times 400$  sub-scene from the Lego scene for 200k iterations with 1/5th of 10k and 30k samples, i.e., 2k and 6k samples. Tab. 4 exhibits significantly decreased rendering quality for the LightSpeed network as compared to MobileR2L when provided with less pseudo-data.
Table 4: Pseudo-Data Requirement for Non-Frontal Scenes. We analyze the importance of mining more pseudo-data for non-frontal scenes. Using 1/5th of  $10\mathrm{k}$  and  $30\mathrm{k}$  sampled pseudo-data points, we find more pseudo-data is crucial for the boosted performance of the LightSpeed model.
Method	2k Samples	6k Samples
PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓
MobileR2L	30.19	0.9894	0.0354	30.56	0.9898	0.0336
LightSpeed (Ours)	30.44	0.9899	0.0299	31.2	0.9906	0.0284
Decoder Network Size. We further analyze the trade-off between inference speed and rendering quality of our method and MobileR2L. To this end, we experiment with decoders of different depths and widths. Each network is trained for 200k iterations and benchmarked on an iPhone 13. Tab. 5 shows that a 30-layered LightSpeed model has a better inference speed and rendering quality as compared to the 60-layered MobileR2L model. This 30-layered variant further occupies less storage as compared to its full-sized counterpart. Furthermore, lighter LightSpeed networks obtain a comparable performance as the 60-layered MobileR2L. Note that reducing the network capacity of MobileR2L results in significant drops in performance. This means that we can get the same rendering quality as MobileR2L with considerably reduced on-device resources, paving the way for a much better trade-off between rendering quality and on-device inference speed.
Table 5: Decoder Network Size. Our approach maintains a much better tradeoff between inference speeds v/s rendering quality, with our smallest network achieving comparable quality to the MobileR2L. Benchmarking done on an iPhone 13. L is network depth, and W is network width.
Method	PSNR ↑	Latency ↓	Storage ↓	FLOPs ↓
15-L W-256 MobileR2L	27.69	14.54 ms	2.4 MB	12626M
30-L W-128 MobileR2L	27.54	14.47 ms	1.4 MB	8950M
30-L W-256 MobileR2L	29.21	18.59 ms	4.5 MB	23112M
60-L W-256 MobileR2L	30.34	22.65 ms	8.2 MB	42772M
15-L W-256 LightSpeed	30.37	14.94 ms	10.5 MB	12833M
30-L W-128 LightSpeed	30.13	14.86 ms	9.5 MB	9065M
30-L W-256 LightSpeed	31.70	20.35 ms	12.6 MB	23319M
60-L W-256 LightSpeed	32.34	26.47 ms	16.3 MB	42980M
Ray-Space Grid Encoding. We provide an ablation in Tab. 6 below on how the proposed ray-space grid encoder helps as compared to just using the light-slab representation with a traditional frequency encoder. We compare different LightSpeed configurations with grid-encoder and frequency encoders. Networks are trained for 200k iterations on a full-resolution  $800 \times 800$  Lego sub-scene from Synthetic
$360^{\circ}$  dataset. Further, we show the training dynamics of all the trained variants in Fig. 5 (red and green plots). As claimed, our approach offers better visual fidelity and training dynamics (iterations to reach a target PSNR) for both computationally cheaper small networks as well as full sized networks.
Table 6: Effect of using a Ray-Space Grid Encoder. We demonstrate the effect of using a grid-based LightSpeed by comparing with a frequency encoded variant (no grid). L is network depth, and W is network width.
Method	PSNR ↑
15-L W-256 LS (PE)	28.84
30-L W-256 LS (PE)	30.63
60-L W-256 LS (PE)	32.16
15-L W-256 LS (Grid)	30.37
30-L W-256 LS (Grid)	31.70
60-L W-256 LS (Grid)	32.34
Comparison with Plücker Representation. Given the challenges of discretizing Plücker representation, we compare between using positionally encoded Plücker coordinates and our grid-based light-slab approach in Tab. 7 below for different network sizes to demonstrate the effectiveness of our approach. We train all models for 200k iterations on one  $800 \times 800$  Lego sub-scene. We also share training curves for the variants in question in Fig. 5 (red and blue curves). As claimed, our integrated approach performs better in terms of training time and test-time visual fidelity for large and small models (having less computational costs) alike whereas the Plücker-based network shows a sharp decline in visual fidelity and increased training times to reach a target test PSNR as network size is reduced.
Table 7: Light-Slab Grid Representation vs. Plücker Coordinates. We compare the light-slab based LightSpeed (LS) with a positionally encoded variant of the Plücker ray representation. L is network depth, and W is network width.
Method	PSNR ↑
15-L W-256 Plücker	28.65
30-L W-256 Plücker	30.84
60-L W-256 Plücker	32.14
15-L W-256 LS	30.37
30-L W-256 LS	31.70
60-L W-256 LS	32.34
In this paper, we propose an efficient method, LightSpeed, to learn neural light fields using the classic two-plane ray representation. Our approach leverages grid-based light field representations to accelerate light field training and boost rendering quality. We demonstrate the advantages of our approach not only on frontal scenes but also on non-frontal scenes by following a divide-and-conquer strategy and modeling them as frontal sub-scenes. Our method achieves SOTA rendering quality amongst prior works at same time providing a significantly better trade-off between rendering fidelity and latency, paving the way for real-time view synthesis on resource-constrained mobile devices.
Limitations. While LightSpeed excels at efficiently modeling frontal and  $360^{\circ}$  light fields, it currently lacks the capability to handle free camera trajectories. The current implementation does not support refocusing, anti-aliasing, and is limited to static scenes without the ability to model deformable objects such as humans. We plan to explore these directions in future work.
Broader Impact. Focused on finding efficiencies in novel view synthesis, our study could significantly reduce costs, enabling wider access to this technology. However, potential misuse, like unsolicited impersonations, must be mitigated.
[39] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P Srinivasan, Richard Szeliski, Jonathan T Barron, and Ben Mildenhall. Bakedsdf: Meshing neural sdfs for real-time view synthesis. arXiv preprint arXiv:2302.14859, 2023. 1, 3
[40] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021. 1, 3
[41] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snively. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018. 2, 3
Network architecture. Our multi-scale feature grids have 16 levels, with resolutions exponentially growing from 16 to 256, and 4-D features in every grid. Our LightSpeed network follows a similar architecture to MobileR2L: 60 point-wise residual convolutions with 256 channels and BatchNorm [15] and GeLU [14] activation interleaved. The convolutions are followed by 3 super-resolution modules to upsample the low-resolution input to the desired resolution. The first two super-resolution modules upsample the input by  $2 \times$  and consist of transposed convolution layers with  $4 \times 4$  kernel size followed by 2 residual convolution layers each. The third super-resolution module consists of transposed kernel size with  $4 \times 4$  kernel size (upsample by  $2 \times$ ) for  $360^{\circ}$  scenes (both bounded and unbounded) and  $3 \times 3$  kernel size (upsample by  $3 \times$ ) for forward-facing [22] scenes.
Training details. We use Adam [18] optimizer with a batch size of 32 to train the feature grids and decoder network. We use an initial learning rate of 1e-5 with 100 warmup steps taking the learning rate to 5e-4. Beyond that, the learning rate decays linearly until the training finishes. All our experiments are conducted on Nvidia V100s and A100s.
B More Ablation Analysis
Choice of Splitting Planes. We discuss two aspects of dividing non-frontal scenes into separate light fields: the number of parts to divide the scene into and the placement of the splitting planes. We find the optimal number of splits for  $360^{\circ}$  scenes to be 5 since more number of splits would mean increased storage cost, which is detrimental to mobile deployment. We also want the scene splits to be collectively exhaustive (but not mutually exclusive to maintain continuity while switching from one light field to another) in the poses sampled around the object. Consequently, fewer planes would mean placing the splitting planes near the scene origin to cover the entire scene, which starts to violate the frontal assumption for each sub-scene.
Given poses distributed on the surface of a sphere with radius  $r$ , we propose assigning each pose to (possibly multiple) sub-scenes based on the camera origin satisfying one or more of the 5 following criteria:
$$
\left[ \begin{array}{c c c} 0 & 0 & \sqrt {2} \\ \sqrt {2} & 0 & \sqrt {2} - 1 \\ - \sqrt {2} & 0 & \sqrt {2} - 1 \\ 0 & \sqrt {2} & \sqrt {2} - 1 \\ 0 & - \sqrt {2} & \sqrt {2} - 1 \end{array} \right] \left[ \begin{array}{l} x \\ y \\ z \end{array} \right] \geq \left[ \begin{array}{l} r \\ r \\ r \\ r \\ r \end{array} \right] \tag {3}
$$
These five hyperplanes form the surface of a near-isometric trapezoidal prism, as shown in Fig. 3 (main paper). We experimentally show the effect of the choice of splitting plane by training LightSpeed models on a Lego sub-scene with different plane placements and compare with the corresponding MobileR2L models trained on the same data. Specifically, we choose two axis-aligned planes at a distance of  $\frac{radius}{\sqrt{2}}$  and  $\frac{radius}{\sqrt{3}}$  from the scene origin and train models with 6k pseudo data points sampled independently from the two resulting sub-scenes. As shown in Tab. 8, placing the splitting plane at a distance of  $\frac{radius}{\sqrt{3}}$  results in inferior performance as compared to placing the splitting plane at a distance of  $\frac{radius}{\sqrt{2}}$  from the origin. This suggests that frontal sub-scene approximation starts to break down as we move the splitting plane closer to the origin.
Table 8: Choice of Splitting Planes. We experiment with two planes parallel to the x-y sub-space at different distances. Splitting planes further from the origin work better empirically maintaining the frontal sub-scene assumption.
LF Representation	PSNR ↑	SSIM ↑	LPIPS ↓
radius /√2	30.44	0.9903	0.028
radius /√3	30.23	0.9899	0.031
C Per-Scene Quantitative Results
We provide a per-scene quantitative comparison between LightSpeed, MobileR2L [6] and NeRF [23] on the synthetic  $360^{\circ}$  dataset (Tab. 9, Tab. 10, and Tab. 11) and forward-facing dataset (Tab. 12, Tab. 13, and Tab. 14). We use PSNR, LPIPS, and SSIM as comparison metrics. As can be seen from the comparisons, LighSpeed (our approach) outperforms MobileR2L [4] on almost all the metrics. Further, LightSpeed performs comparably or even better than NeRF [23]. We also provide additional zoom-in comparisons between LightSpeed and MobileR2L in Fig. 7.
Table 9: Per-scene PSNR  $\uparrow$  comparison on the Synthetic  ${360}^{ \circ  }$  dataset between NeRF [23],MobileR2L [4], and our approach.
Method	Chair	Drums	Ficus	Hotdog	Lego	Materials	Mic	Ship	Average
NeRF[23]	33.00	25.01	30.13	36.18	32.54	29.62	32.91	28.65	31.01
MobileR2L [4]	33.66	25.05	29.80	36.84	32.18	30.54	34.37	28.75	31.34
LightSpeed (Ours)	34.21	25.63	32.82	36.77	34.35	29.51	35.65	28.90	32.23
Table 10: Per-scene SSIM  $\uparrow$  comparison on the Synthetic  $360^{\circ}$  dataset between NeRF [23], MobileR2L [4], and our approach.
Method	Chair	Drums	Ficus	Hotdog	Lego	Materials	Mic	Ship	Average
NeRF[23]	0.967	0.925	0.964	0.974	0.961	0.949	0.980	0.856	0.947
MobileR2L [4]	0.998	0.986	0.996	0.998	0.992	0.992	0.997	0.982	0.993
LightSpeed (Ours)	0.998	0.988	0.998	0.998	0.994	0.990	0.998	0.984	0.994
Table 11: Per-scene LPIPS  $\downarrow$  comparison on the Synthetic  $360^{\circ}$  dataset between NeRF [23], MobileR2L [4], and our approach.
Method	Chair	Drums	Ficus	Hotdog	Lego	Materials	Mic	Ship	Average
NeRF[23]	0.046	0.091	0.044	0.121	0.050	0.063	0.028	0.206	0.081
MobileR2L [4]	0.027	0.083	0.025	0.026	0.043	0.029	0.012	0.162	0.051
LightSpeed (Ours)	0.017	0.061	0.016	0.023	0.019	0.030	0.007	0.138	0.039
Table 12: Per-scene PSNR  $\uparrow$  comparison on the forward-facing dataset between NeRF [23], MobileR2L [4], and our approach.
Method	Room	Fern	Leaves	Fortress	Orchids	Flower	T-Rex	Horns	Average
NeRF[23]	32.70	25.17	20.92	31.16	20.36	27.40	26.80	27.45	26.50
MobileR2L [4]	32.09	24.39	20.52	30.81	20.06	27.61	26.71	27.01	26.15
LightSpeed (Ours)	32.32	25.05	21.01	31.45	20.33	27.88	26.93	27.04	26.50
Table 13: Per-scene SSIM  $\uparrow$  comparison on the forward-facing dataset between NeRF [23], MobileR2L [4], and our approach.
Method	Room	Fern	Leaves	Fortress	Orchids	Flower	T-Rex	Horns	Average
NeRF[23]	0.948	0.792	0.690	0.881	0.641	0.827	0.880	0.828	0.811
MobileR2L [4]	0.995	0.973	0.923	0.995	0.916	0.971	0.973	0.982	0.966
LightSpeed (Ours)	0.991	0.976	0.931	0.996	0.921	0.972	0.975	0.983	0.968
Table 14: Per-scene LPIPS  $\downarrow$  comparison on the forward-facing dataset between NeRF [23], MobileR2L [4], and our approach.
Method	Room	Fern	Leaves	Fortress	Orchids	Flower	T-Rex	Horns	Average
NeRF[23]	0.178	0.280	0.316	0.171	0.321	0.219	0.249	0.268	0.250
MobileR2L [4]	0.088	0.239	0.280	0.103	0.296	0.150	0.121	0.217	0.187
LightSpeed (Ours)	0.085	0.211	0.255	0.093	0.272	0.145	0.119	0.209	0.173
Results on Unbounded Scenes. The rendering fidelity of LightSpeed is closely tied to the performance of the corresponding NeRF teacher. LightSpeed uses Instant NGP [25] teachers for both bounded and unbounded scenes to maintain experimental consistency. We would like to highlight that Instant-NGP introduces the artifacts to unbounded scenes, which are carried forward to LightSpeed via the mined pseudo-data. We share some of the pseudo-data images from Instant-NGP in Fig. 6. MipNeRF360 [3] specifically uses space contraction techniques to model the unbounded nature of the scene and deal with blurriness in the renderings. It further introduces a distortion-based regularizer to remove floater artifacts and prevent background collapse. The techniques introduced by MipNeRF360 tackle the same type of artifacts pointed out in Fig. 6. Hence, using MipNeRF360 teachers will mitigate both these issues and could boost the visual fidelity on unbounded scenes for LightSpeed.
Figure 6: Instant NGP Failure Cases for Unbounded Scenes. Such artifacts carry over to LightSpeed, affecting its visual fidelity on unbounded scenes.
Figure 7: Qualitative Results on frontal and non-frontal scenes. Zoomed-in comparison between MobileR2L and our LightSpeed approach.