SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Impact of the Sample Count In Figure 3, we evaluate the tradeoff between frame latency and PSNR and compare it to previous methods for various sample counts. Notice that only adaptive methods can use non-integer spp budgets because there is no procedure to choose how to sample non uniformly for non-adaptive methods. Previous adaptive methods approach uniform sampling as their spp budget approaches 1.0 because they cannot sample less than 1 spp for any pixel. Indeed, the gradient approximation (Equation 1) diverges in case the spp recommendation is 0. This imposes a hard lower limit on the frame latency. Contrapositively, only our method can use a spp budget smaller than 1.0 because we train the sampling importance network with reinforcement learning to overcome the diverging gradient limitation. We observe that the visual quality collapses for very low spp counts (0.01 spp), greatly increases from 0.01 to 2 spp, and then increases slower until 4 spp count.
We present a variant of our method called "ours-small" that corresponds to our method with smaller sampling importance, latent encoder, and denoiser networks (see the network variational study in the Appendix). This variant has a smaller inference time which allows reaching even lower frame latencies. We observe that this variant is Pareto-optimal until 38 ms/frame. We also notice that this variant really starts improving after reaching a 0.1 spp count, whereas our main method already improves between 0.01 spp and 0.1 spp counts. This indicates that our variant with small networks is not able to fully leverage the potential of 0.1 spp count.
Comparison of our Method with other Related Work We observe that our method is Pareto-optimal compared to previous methods for any spp budget. Not only does our method outperform any other baseline for equal spp counts, but also for higher spp counts: our method with 2 spp budget outperforms every baseline with 4 spp budget. Given that some of the current trends in rendering aim
Figure 4: Ground truth using 1000 spp on the left, ours using 4.0 spp on the middle, previous adaptive sampling state-of-the-art (Neural Temporal Adaptive Sampling [23]) with 4.0 spp on the right. Image rendered on the Ripple Dream dataset. 720x720 frame on top, and 200x200 zoom to a particular area on the bottom. The PSNR for our rendered image is 29.1 dB and is 27.7 dB for the previous state-of-the-art rendered image.
at minimizing the latency time (no adaptive sampling, supersampling,...), the fact that our method has a relatively large latency but manages to outperform quality-wise very fast baselines for an equal latency is an interesting result [27, 24, 25]. Given an equal budget of 4 spp, our method renders outputs with a  $0.6\mathrm{dB}$  higher PSNR compared to the strongest baseline for a negligible latency increase (3 ms latency increase which corresponds to  $5.5\%$ ). Our approach with 2 spp reaches a higher visual quality (PSNR improvement of  $0.2\mathrm{dB}$ ) for a  $1.6\mathrm{x}$  latency reduction compared to the strongest baseline at 4 spp.
Ablation Study of our Method We observe on Table 1 that our method is among the slowest in terms of ms/frames, but generally outperforms all other baselines. Our method and three of our variants (ours-A1, ours-B1, ours-C) always outperform all previous works except ReSTIR+OIDN [25, 9] for an equal spp count. On average, our method outperforms the variant with the gradient approximation (ours-A1) by  $0.3\mathrm{dB}$ , and the variant with uniform sampling (ours-A2) by  $0.8\mathrm{dB}$ . This demonstrates both the utility of adaptive sampling and the superiority of training the sampling importance network with RL instead of the gradient approximation. We include a study of the sampling heatmap and error distribution for several scenes in the supplementary material to improve the understanding of the effect of RL-based sampling vs gradient-based sampling.
Comparing our method to ours-B1, we see that using the latent space encoder and having as temporal feedback the warped latent space instead of the sampled pixel values (ours-B1) allows increasing the PSNR by 0.7 dB, and increases the PSNR by 1.3 dB in the case with no temporal feedback (Ours-B2). Finally using the sampled pixel values instead of the averaged values (ours-C) increases the PSNR by 0.5 dB.
Comparison with Adaptive Sampling SOTA Our method using RL for the adaptive sampling outperforms our method using the gradient approximation (Ours-A1) for the adaptive sampling. Furthermore, our method with the gradient approximation outperforms NTAS [23] by a large margin
(0.7 dB) showing the importance of using the individual sampled pixel values instead of the averaged values, and of using a latent space encoder.
Comparison with Hybrid Latent State SOTA Our method with uniform sampling (Ours-A2) has as high visual quality as IMCD [31]. This is not surprising because both approaches use temporal feedback and use sampled pixel values instead of only the average. The main difference in the design is our state encoder that creates a meaningful state, whereas IMCD only uses simple heuristics like an exponential average of the previous latent spaces. This difference allows our variant to reach equal visual quality despite the 2.1x smaller inference time.
Comparison with Probabilistic Latent Space SOTA ReSTIR+OIDN [25, 9] uses probabilistic heuristics and the non-averaged sampled pixel values to recommend from which light to sample, and to choose how to update the spatiotemporal latent space. Our method conceptually has the same objectives. The fact that our method outperforms ReSTIR+OIDN by  $0.5\mathrm{dB}$  shows that using networks that learn how to do the sampling and how to update the spatiotemporal latent state instead of using theoretical heuristics allows reaching higher results.
Visual Quality Comparison Looking at Figure 4, we observe that our method generally looks more similar to the ground-truth image. For example, large uniform surfaces present less noise and advanced details such as edges, small bubbles, reflection, and refraction effects look more realistic with our method. In the supplementary material, we compare rendered animations and observe that our method manages to maintain temporally more coherent results as well.
Limitations The current method does not consider the application of after-effects such as motion blur, which could allow the reduction of samples collected on fast-moving objects, while in the current setting we would implicitly focus the ray tracing samples on such an object to minimize the error. Further, this method is applicable to entertainment applications and can potentially generate artifacts not present in the real scene, which could be an obstacle in application scenarios such as VR/AR medical devices. Additionally, it requires a system capable of performing both path tracing and DNN inference. While current graphics cards provide this capability and we assess the frame rendering time considering both components, the underlying workloads are significantly different: DNN inference is generally a very structured and high arithmetic intensity workload whereas path tracing is branching-intensive and performs random look-ups into memory. As these are fundamentally different, we see dedicated ray tracing units on modern GPUs. In future systems, it is conceivable that dedicated devices are used for each step, which would enforce a fixed capability for each type of compute and limit a free trade-off between the two as we make use in this work. Additionally, applying a DNN adds a memory overhead, although it can remain minimal compared to other components such as textures that commonly fill most the GPU's memory. Specifically, the model requires 110 MB more memory to store the input and latent space data,  $<50$  MB to store the model, and a few MB of working memory for intermediate feature maps that can be processed in tiles.
Memory Overhead We store the following data in memory. 1) the model weights (<50MB), 2) the sampled pixel values (24 channels; RGB values for up to 8 non-averaged samples), 3) 7 additional input channels (3 for surface normals, 3 for albedo, 1 for depth), 4) 32 channels for the state (warped latent space), and 5) a small working memory for intermediate feature data during inference. The components 2-4 add up to 53 more channels than previous works (which use 3 channels for input channels and 7 for additional data); hence 212 Byte/pixel; 110MB in total for 720x720 pixel data; and the required working memory is minimal as the inference can be done on tiles. With a total memory footprint in the order of  $200\mathrm{MB}$ , the overhead is insignificant ( $<2\%$ ) compared to  $>12\mathrm{GB}$  of textures loaded for current games on high-end GPUs where such capacity is available.
We explored three different ideas to improve sampling and denoising for low sampling count and showed through a leave-one-contribution-out that they are all important: The first one is to adaptively sample leveraging RL to make better sampling decisions; using the previously recommended gradient approximation leads to a 0.3 dB PSNR decrease. The second one is to let a deep learning latent state
encoder decide which pixel information to keep given the previous state to optimize the denoising and further rendering; removing this network leads to a 0.7 dB PSNR decrease. The third one is to not aggregate the pixel values by averaging, but to keep all information for the state encoder network; averaging the sampled pixel values leads to a 0.5 dB PSNR decrease. We discovered that combining those ideas allows outperforming different state-of-the-arts by at least 0.7 dB for an equal spp budget, or by a 1.6x latency reduction while keeping a marginal (0.2 dB) visual quality improvement. Finally, we are the first adaptive sampling method that can accept a lower spp budget than 1 spp, allowing using adaptive sampling for unprecedentedly high frame rates. Finally, we release our implementation and datasets.
[1] J. Arvo and D. Kirk, "Particle transport and image synthesis," in Proc. Conf. on Computer Graphics and Interactive Techniques, 1990, pp. 63-66.
[2] M. F. Cohen, J. R. Wallace, and P. Hanrahan, Radiosity and realistic image synthesis. Morgan Kaufmann, 1993.
[3] R. L. Cook and K. E. Torrance, "A reflectance model for computer graphics," ACM Siggraph Computer Graphics, vol. 15, no. 3, pp. 307-316, 1981.
[4] A. S. Glassner, An introduction to ray tracing. Morgan Kaufmann, 1989.
[5] Y. Huo and S.-e. Yoon, "A survey on deep learning-based monte carlo denoising," Computational visual media, vol. 7, pp. 169-185, 2021.
[6] H. Dammertz, D. Sewtz, J. Hanika, and H. P. Lensch, "Edge-avoiding a-trous wavelet transform for fast global illumination filtering," in Proceedings of the Conference on High Performance Graphics, 2010, pp. 67-75.
[7] S. Bako, T. Vogels, B. McWilliams, M. Meyer, J. Novák, A. Harvill, P. Sen, T. Derose, and F. Rousselle, "Kernel-predicting convolutional networks for denoising monte carlo renderings." ACM Trans. Graph., vol. 36, no. 4, pp. 97-1, 2017.
[8] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in Proc. Int. Conf. on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Springer, 2015, pp. 234-241.
[9] A. T. Áfra, "Intel® open image denoise," 2019.
[10] C. R. A. Chaitanya, A. S. Kaplanyan, C. Schied, M. Salvi, A. Lefohn, D. Nowrouzezahrai, and T. Aila, "Interactive reconstruction of monte carlo image sequences using a recurrent denoising autoencoder," ACM Transactions on Graphics (TOG), vol. 36, no. 4, 2017.
[11] M. E. Lee, R. A. Redner, and S. P. Uselton, "Statistically optimized sampling for distributed ray tracing," in Proc. Conference on Computer Graphics and Interactive Techniques, 1985, pp. 61-68.
[12] R. S. Overbeck, C. Donner, and R. Ramamoorthi, “Adaptive wavelet rendering,” ACM Trans. Graph., vol. 28, no. 5, p. 140, 2009.
[13] K. Egan, Y.-T. Tseng, N. Holzschuch, F. Durand, and R. Ramamoorthi, "Frequency analysis and sheared reconstruction for rendering motion blur," in Proc. ACM SIGGRAPH, 2009.
[14] N. K. Kalantari and P. Sen, “Removing the noise in monte carlo rendering with general image denoising algorithms,” in Computer Graphics Forum, vol. 32, no. 2pt1. Wiley Online Library, 2013, pp. 93–102.
[15] F. Rousselle, C. Knaus, and M. Zwicker, "Adaptive sampling and reconstruction using greedy error minimization," ACM Transactions on Graphics (TOG), vol. 30, no. 6, 2011.
[16] B. Moon, N. Carr, and S.-E. Yoon, "Adaptive rendering based on weighted local regression," ACM Transactions on Graphics (TOG), vol. 33, no. 5, 2014.
[17] B. Moon, S. McDonagh, K. Mitchell, and M. Gross, "Adaptive polynomial rendering," ACM Transactions on Graphics (TOG), vol. 35, no. 4, 2016.
[18] T. Viitanen, M. Koskela, K. Immonen, M. J. Makitalo, P. Jaaskelainen, and J. Takala, "Sparse sampling for real-time ray tracing," in VISIGRAPP (1: GRAPP), 2018, pp. 295-302.
[39] E. Liang, R. Liaw, R. Nishihara, P. Moritz, R. Fox, K. Goldberg, J. Gonzalez, M. Jordan, and I. Stoica, “Rlib: Abstractions for distributed reinforcement learning,” in International Conference on Machine Learning. PMLR, 2018, pp. 3053–3062.
[40] K. A. Nicholas Hull and N. Benty, "Nvidia emerald square, open research content archive (orca)," July 2017, http://developer.nvidia.com/orca/nvidia-emerald-square.
[41] E. Games, "Unreal engine sun temple, open research content archive (orca)," October 2017, http://developer.nvidia.com/orca/epic-games-sun-temple.
[42] M. Winkelmann, “Zero-day, open research content archive (orca),” November 2019, https://developer.nvidia.com/orca/beeple-zero-day.
A Individual Results Per Scene
The analysis in Section 3.3 averages the latency and PSNR across different scenes. This could potentially hide differences in the scene complexity where the cost of sampling a ray might vary significantly. We thus provide a more fine-grained per-scene evaluations in Figure 5.
Figure 5: PSNR quality as a function of time per frame on the cross-validated scenes EmeraldPlace (top left), SunTemple (top right), ZeroDay (lower left) and Ripple Dream (lower right). Magenta color corresponds to a 0.01 spp count, cyan color corresponds to a 0.1 spp count, black color corresponds to 0.3 spp count, green color corresponds to a 0.5 spp count, red color corresponds to a 1.0 spp count, yellow color corresponds to a 2.0 spp count and blue color corresponds to a 4.0 spp count. Upper-left is better.
B Architecture and Variational Network Study
The basic architecture of our networks is visualized in Figure 7. We justify the choice of size of the different different models used within our solution using a variational study: we fix a time budget of  $100\mathrm{ms}$  per frame and select a single scene such that the sampling time for  $1.0\mathrm{spp}$  is fixed. We then study how varying the size of each of our 3 networks affects the quality of the results, taking into account that changing the size of the network will change the total sampling budget. We evaluate our method on the EmeraldPlace dataset.
For the small variant of the denoiser we scale down the architecture shown in Figure 7 by reducing the number of channels by  $2 \times$  every convolution layer, and its large variant is identical except we scale up the number of channels by  $2 \times$ . The small sampling importance network is a simple CNN with three  $3 \times 3$  convolutional blocks using 1 latent channel, and the large sampling importance network is the original UNET from [8]. The small state encoder architecture corresponds to the architecture in Figure 7 except we replaced the  $3 \times 3$  convolutions with  $1 \times 1$  convolutions and the big state encoder architecture corresponds to the architecture in Figure 7 except that we added two additional  $3 \times 3$  convolutions with a ReLU layer in between. Ours-small mentioned in the paper corresponds to using both the small sampling importance network, the small state encoder, and the small denoiser network.
Sampling Importance Network This variational study shows that choosing a very fast sampling importance network is beneficial compared to using a slower architecture for our fixed time budget experiment. This result probably does not hold when the time budget increases because the inference time of the network will become negligible. Nevertheless, we focus on real-time applications which means that the time budget has to be smaller or equal to  $30\mathrm{ms}$ . This architecture design contrasts with previous designs in adaptive sampling [23, 20], where the sampling importance network and the
Table 3: Inference time in ms and corresponding spp count for every network. Spp count is computed as follows: given x the total inference time, spp count  $= (100 - x) / 69.0$ , where 69.0 is the sampling time in ms for 1 spp on EmeraldPlace.
Variation	Sampling importance	State encoder	Denoiser	Total	Spp count
Normal	2.5	3.7	16.3	22.5	1.12
Small Sampling importance	2.0	3.7	16.3	22.0	1.13
Large Sampling importance	4.5	3.7	16.3	24.5	1.09
Small state encoder	2.5	2.2	16.3	21.0	1.14
Large state encoder	2.5	9.4	16.3	28.2	1.04
Small denoiser	2.5	3.7	10.2	16.4	1.21
Large denoiser	2.5	3.7	25.1	31.3	0.99
Figure 6: Comparison of PSNR when varying the size of every network for equal latency constraints. Higher is better.
denoiser network had almost the same architecture and inference time. This difference in design can be explained because we take the latent space as input which is pre-processed information, whereas previous approaches give row information as input, and the sampling importance network typically had to guess which were the high variance areas, for example by learning to segment edges, whereas our method can more easily extract more meaningful data like sample count, or variance from the latent space.
State Encoder Network We also observe that the state encoder does not need to be a UNET, which implies that local information is sufficient to update the latent state.
Denoiser Network Finally, the denoiser network takes the majority of the inference time of our approach. Our architecture is almost identical to the denoising architecture of [9]. We observe in the ablation study that trying to shrink this architecture strongly deteriorates the results.
Figure 7: Architecture of the sampling importance network (top left), the latent state encoder network (top right), and the denoiser network (bottom). Max Pooling  $2 \times 2$  refers to pooling with kernel size 2 and stride 2. Typically, given an input image with a resolution of  $720 \times 720$ , the resolution after the first max pooling is  $360 \times 360$ , after the second is  $180 \times 180$ , after the third is  $90 \times 90$ , after the fourth is  $45 \times 45$ , and the symmetrical opposite resolutions for the upsampling way. The number above the convolutional blocks represents the number of output channels for every layer in the block. Upper arrows represent residual connections. The concatenation layers are not shown but happen after every upsampling layer.
C Sampling Strategy Comparison
Identifying the differences between the sampling heatmaps when using reinforcement learning and when using the gradient approximation in non-trivial (Figure 10). We attribute this to the sampling importance network learning to adaptively sample by taking into account the impact of the denoiser, and not simply by sampling in high variance areas like edges. We found it more informative to compare the MSE per pixel between the RL adaptive sampling and gradient approximation adaptive sampling method (Figure 8, Figure 9). In the additional material, we include similar data for moving animations. The per-pixel difference in MSE between the methods remains challenging to interpret directly (Figure 8). To otherwise visualize the per-pixel MSE gains, we generated histograms (Figure 9) clearly showing that the RL-based method leads to generally better results as the mode and the mean of the distributions are always negative. In particular, we observe from the lower histogram that our RL-based method particularly avoids large errors.
Sparse Sampling A common strategy that we observe in both our method using RL and using the gradient approximation, but that we did not observe in previous adaptive sampling methods is sparse sampling. Given a uniform area, sparse sampling consists of a pattern such that few pixels get high spp count recommendation, and surrounding pixels get low spp count recommendation. This can be an efficient sampling strategy assuming that locally close pixel values are not independent (cf. Figure 10).
Figure 8: Comparison of the binary MSE difference between our method when trained with RL (top left), and the gradient approximation (top right). White colors mean that the MSE of the method with the gradient approximation is bigger for this pixel and inversely for black colors.
Comparison of the extreme binary MSE difference between our method when trained with RL (center left), and the gradient approximation (center right). White colors mean that the difference in MSE of the method with the gradient approximation is bigger than the threshold for this pixel, and inversely for black colors.
Ground truth images included for reference (bottom left and bottom right).
Figure 9: Comparison of the distribution of MSE differences between our method when trained with RL and with the gradient approximation. Negative values mean that the MSE of the method with the gradient approximation is bigger and inversely for positive values. The top image corresponds to the distribution for the left image in Figure 8, and the bottom image to the distribution for the right image in Figure 8.
Figure 10: Comparison of the sampling recommendation from our sampling importance network when trained with RL (left), and the gradient approximation (center). Lighter colors mean higher sampling recommendations. Identical color scale. We observe sparse sampling and recurrent sampling patterns in both sampling heatmaps. Ground truth image included for reference (right).
D Supplementary Material
Together with the paper, we provide several short video sequences to verify that our solution does not introduce any flickering or other artifacts and generally enable a qualitative comparison of methods. GD-ours-NTAS.gif extends Figure 4 in the main manuscript. We include FIG4a.gif, FIG4b.gif, FIG4c.gif, FIG4d.gif, and FIG4e.gif that extend Figure 8 of this manuscript. We include FIGupper.gif and FIGlower.gif that extend Figure 9 from this manuscript. The GIFs have been compressed to max. 100MB.