RL-based Stateful Neural Adaptive Sampling and Denoising for Real-Time Path Tracing
Authors: Antoine Scardigli
Lukas Cavigelli
Lorenz K Muller
Venue: NeurIPS 2023 poster (2023)
Keywords: computer graphics
rendering
ray tracing
GPU acceleration
RL
spatiotemporal latent space
TL;DR: We make ray tracing rendering faster: we encode more efficiently previously observed pixel values using neural networks instead of heuristics; we use RL to choose where to sample light rays.

Abstract: Monte-Carlo path tracing is a powerful technique for realistic image synthesis but suffers from high levels of noise at low sample counts, limiting its use in real-time applications. To address this, we propose a framework with end-to-end training of a sampling importance network, a latent space encoder network, and a denoiser network. Our approach uses reinforcement learning to optimize the sampling importance network, thus avoiding explicit numerically approximated gradients. Our method does not aggregate the sampled values per pixel by averaging but keeps all sampled values which are then fed into the latent space encoder. The encoder replaces handcrafted spatiotemporal heuristics by learned representations in a latent space. Finally, a neural denoiser is trained to refine the output image. Our approach increases visual quality on several challenging datasets and reduces rendering times for equal quality by a factor of 1.6x compared to the previous state-of-the-art, making it a promising solution for real-time applications.

RL-based Stateful Neural A Adaptive Sampling and Denoising for Real-Time Path Tracing
Antoine Scardigli, Lukas Cavigelli, Lorenz K. Müller Computing Systems Lab, Huawei Zurich Research Center, Switzerland scardigglanttohne@@maal com, (lukas. cavigelli, lorenz. mrteenz.mellerrhhnaaae com
Abstract
Monte-Carlo path tracing is a powerful technique for realistic image synthesis but suffers from high levels of noise at low sample counts, limiting its use in real-time applications. To address this, we propose a framework with end-to-end training sampling importance network, a latent space encoder network, and denoiser network. Our approach uses reinforcement learning to optimize the sampling importance network, thus avoiding explicit numerically approximated gradients. Our method does not aggregate the sampled values per pixel by averaging but keeps all sampled values which are then fed into the latent space encoder. The encoder replaces handerafted spatiotemporal heuristics by learned representations in latent space. Finally, neural denoiser is trained to refine the output image. Our approach increases visual quality on several challenging datasets and reduces rendering times for equal quality by a factor of 1.6x compared the previous state-of-the-art, making it promising solution for real-time applications.
Introduction
Monte-Carlo path tracing relies on repeatedly sending discrete rays that bounce in random directions to approximate the real-life (continuous) light scattering. The number of rays sent per pixel is called the Sample Per Pixel (spp) count. The Monte-Carle method guarantees that the estimated pixel value is unbiased and that the standard error decreases at rate proportional to va count [1, 2, 3, 4].
where is the spp
Denoising Offline rendering can afford high spp counts. Nevertheless, it is very common to spatial denoising as postprocessing to reduce the computational budget by several orders of magnitude for equivalent quality outputs [5]. First, denoising approaches were heuristic-based [6], but learned methods recently imposed themselves [7]. In particular, encoder-decoder neural networks like
UNETs [8] the current state of the art terms of denoising [9]. Training the denoiser specifically for low sample counts (4 spp) allows still reaching high-quality results [10].
Adaptive Sampling Following the intuition that low spp counts are likely sufficient for some parts of the frames like large uniform areas, and that some areas with bigger variance like edges or highly reflective surfaces might require a higher sample count, heuristic-based methods for sampling a non-uniform sample count within frame have first been proposed. Some approaches use specific metrics like variance [11], contrast [12], frequency content [13], or mean absolute deviation [14] to assign more samples in challenging areas. Further work noticed that the denoiser should use information from the sampling heatmap or vice versa. They perform joint adaptive sampling and denoising by estimating the most problematic regions for the denoiser using Gaussians [15]. linear regression [16], or polynomials [17]. As these metrics have to be estimated to sufficient precision, high spp count (4 more) is required, making them unsuitable for real-time applications: It is
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

considered that real-time ray tracing corresponds to rendering process with a latency smaller than 30 assuming that roughly 30 frames are rendered per second. Such a computational budget generally corresponds to an overall spp count between 0 and 4 for million pixels using a single recent+ GPU, depending on the scene to be rendered [18, 19].
Gradient- Adaptive Sampling Deep Adaptive Sampling for Low Count Rendering [20] is the current state-of-the the-ahtt art in adaptive sampling and the first to use learned approach: The authors propose to first sample uniformly 1 spp and then use UNET to generate the sampling heatmap for the remaining budget (3 spp). Finally, another UNET denoises the averaged sampled pixel values. In order to train the networks gud-to-end, the gradient of the rendered image needs to be computed with respect to the sampling map for every pixel, which is not analytically possible. The authors estimate the gradient numerically instead:
θT. To - 1.
Øs
8
with So the ground truth pixel value, I, the average of the sampled pixel and s the sample count.
We identify 3 issues with this derivation: First, the numerical approximation converges the real gradient in the limit, but no guarantees are offered for lower sample counts. Second, by design, the formula uses L (the average of the sampled pixel) which enforces that sampled pixel values are aggregated by averaging them in the rest of the framework, hence surrendering information like higher-order statisties. Finally, the method does not allow sampling spp for any pixel as this would make the gradient unstable.
Deep Adaptive Sampling and Reconstruction using Analytic Distributions [21] approximates the effect of the path-sampler using an analytical (gamma) distribution which allows backpropagating the gradient to the sampling importance network. This method is not suitable for real-time applications, but is very efficient training-wise because it only uses the ground truth estimate's 's mean and variance and does not use additional sampled frames.
Adaptive Incident Radiance Field Sampling and Reconstruction Using Deep Reinforcement Learn- ing [22] leverages reinforcement learning to adaptively sample in the first bounce incidence radiance field by iteratively partitioning a tile into several tiles, or doubling the sample count of tile. The first bounce incidence radiance field is then reconstructed and integrated with the BRDF for rendering. This method is unfortunately too slow for real-time applications.
Spatiotemporal Reuse
Assuming that the goal is to render animation and not single frame, reusing information from previously rendered frames or from samples collected during the rendering of the previous frames can increase the quality of the result as it can increase the implicit sample count. Motion vectors: pixel-coordinate mappings of backward backward-warping, are used to warp the saved information from the previous frame so that it can be reused for the next frame.
Some related works only save the averaged pixel values of the previous frame [23], some save the
averaged pixel values as well as some higher order statistics like the variance [24], and some store subset of the sampled pixel values (non-averaged) into a sputiotemporal reservoir [25, 26] (a 3D grid that is used to store list of path traced color samples for every pixel of the 2D image) to directly store an estimation of the true distribution. For example, ReSTIR [25] outperforms the state of the art in scenes with thousands of lights, and ReSTIR GI [27] in scenes where lights are seldomly visible through shadow rays. Those two methods interact with the path-samples instead of considering it as a black box like other compared work.
Neural Temporal Adaptive Sampling [23] is the current state-of-the-are in real-time Monte-Carlo rendering using adaptive sampling. The authors reuse the gradient approximation of [20] to train the sampling importance network, but add temporal feedback that is the previously obtained denoised averaged sampled pixel values. This temporal feedback is the main input of the sampling importance network, and one of the main inputs of the denoiser.
Hybrid Latent Space Using samples rather than pixel averages increases the computational cost but can improve denoising [28, 29, 30j. Some hybrid approaches try to get both benefits: [31, 32] are learned denoising (with uniform sampling) methods that propose hybrid approach between keeping sampled pixel values and directly averaging them: Every individual sampled pixel value and its additional features go through fully connected network, and only then are all latent frames an

---

aggregated average. This allows extracting some additional statistics than the average. Nevertheless, the numerical approximation of the gradient (Equation requires use the averaged pixels for backpropagating the gradient through the path-s,mpler, which prevents current adaptive sampling methods to use information other than the averaged sampled values.
We notice two opportunities along which the previous state-of-the-art in adaptive denoising [23] can be improved:
1.
The spatiotemporal reuse only stores the previous denoised averaged frame and hence does not extract any other higher-order statistics than the average from the distribution of path-traced samples. This is problematic because information like variance or confidence is not used. For example, the sampling importance network has to guess the areas with high variance by learning to detect edges highly reflective surfaces whereas the variance information could have been stored in the previous frame. One of our motivations is to increase the spatiotemporal -infrrrmatinooooiiioonno reuse.
2.
We notice that the numerical approximation of the gradient (Equation 1) that is derived in [20] has some shortcomings: such as being unstable for spp counts, and being a rough numerical approximation that only converges to the true gradient in the limit. Our second motivation is to use Reinforcement Learning (RL) for adaptive sampling because it can learn an implicit gradient that works better than the approximated numerical one given our quantized problem.
2 Method
2.1 Spatiotemporal Latent Space
We mentioned that spatiotemporal reuse could consist of storing the average, some higher-order statistics, or directly the sampled pixel values to get the most insight of the sampled distribution. An example of this most efficient reusing is ReSTIR [25, 27] which updates list of sampled colours for every pixel through probabilistic heuristics. To minimize the spatiotemporal loss of information, we use a spatiotemporal reservoir too (that we will call spatiotemporal latent space). However, instead of updating the spatiotemporal latent space using resampled importance sampling [33], we train a CNN network that learns to update the latent space an optimal way.
The output of the latent state encoder network the new spatiotemporal latent space is the sole input of the denoiser and the main input of the sampling importance network. In comparison, previous work [23] gives the output of the denoiser to the sampling importance network. This is a waste of information because all the inputs of the denoiser are compressed into only 3 channels. simple way to see the problem of this approach is that the denoiser network and the sampling importance network have no information in their inputs to get an insight about confidence: They do have any input that stores the variance or the sample count. Finally, the fact that the latent space is the sole input of the denoiser leads to our method being very temporally stable without needing an explicit temporal loss as in [23].
2.2 Reinforcement Learning- based Adaptive Sampling
We described in the introduction section that the current state of the art for real-time adaptive sampling is based deep learning using a numerical gradient approximation (Equation 1) for the sampling pass. We identified some problems with this numerical gradient approximation that could be mitigated using RL-based sampling recommendations:
1.
The gradients are not analytically available: DASR [20] and NTAS [23] use an approximation of the real (inaccessible gradient) with the only guarantee that it converges to the true one in the limit, which is the opposite case than for real-time applications that use low spp counts. Instead of this explicit approximated gradient, it is possible that the RL algorithm will learn a better implicit gradient, allowing higher-quality learning and outputs.
2. Another limitation of the approximated gradient is that it does not allow to sample 0 samples per pixel because the numerical approximation is not stable in this case. For this reason, prior work [23, 20] cannot scale down to 0 spp. With the average spp budget for real-time
3

---

sumple
State
Importanse
Encoce
lampond
Oear
woaping
Figure 1:
An overview of our Approach. The red temporal delay indicates that the warped state is delayed for the next time iteration. Warping takes as additional omitted input the motion vectors.
path tracing being in the range [0; 4], this first means that those methods do not work when the spp budget is smaller than 1, and also that their sampling recommendations very constrained in the remaining case. Using reinforcement learning allows us to not use this explicit gradient and hence avoid this issue.
3. Furthermore, the numerical approximation uses the averaged sampled pixel values for backpropagating the gradient through the path-sampler, which constrains the denoiser into using the averaged sampled pixel values instcad of more complete information like higher order statisties or directly the sampled pixel values. Using RL removes this constraint.
2.3 Algorithms
Our approach uses 3 models:
The Sampling Importance Network is UNET [8] that takes as input the additional features and the warped latent space, and outputs sample map. The additional features have 7 channels (3 for the normals, for the albedo, and for the depth), and the latent space has 32 channels. Therefore, the sampling importance network takes 39 channels input and one channel output. The choice of the UNET architecture has two main motivations: First the upsampling will allow the output frame to contain sparse recommendations, and second, it makes sense that the recommendation values need to exploit information at different scales. The architecture inspired by [8] because of computational efficiency. The actual recommendations given the network recommendations H are follows:
Vi.j round (spp_budgel. E... mmin(r)) We found that the final recommendations would
Z., min=))
were not bounded because some of the values of would be dominantly larger than others. We thus bound the network recommendations by adding a Tanh activation layer [34] as final layer. The number output channels of the first convolution (4) is much smaller than the input number of channels (39), following the rationale that most of the encoded information in the latent space is mostly useful for the denoiser network but not for the sampling importance network.
become unstable the network recommendations
The Latent Encoder Network is CNN [35] that takes the warped latent space and the new samples as input, and outputs a new latent space. New samples can contain up to 8 values per pixel. For this reason, the new-samples input contains 24 channels, where non-attributed pixel values assigned the value (-1) with the rationale that the path-sampler only outputs non-negative values. The 8 frames such that if a pixel is assigned sample count of n, the first frames will contain sampled pixel value, and the last coordinate. This architecture not permutation invariant, which allows all sampled values of the same pixel and from spatially neighboring pixels unlike previous work in hybrid latent space [23, 31, 32]. We added Tanh layer the end of our network to bound the state values. The number of state channels (32 in our case) is chosen considering the performance over time trade-off.
n frames will contain the anomalous value "-1" for this pixel
extract information from
The Denoiser a UNET that takes the new latent space as input and outputs a denoised image. Using UNET for ray tracing denoising is recommended design [10], and we reuse the architecture from [9] as one of the most widely used solutions in terms fast ray tracing denoising. We do not use kernel prediction [7] because [23] found it was not significantly useful.

---

In the Appendix, present network variational study that motivates the size every network should have maximize PSNR at given latency, and we include diagrams of the network architectures.
2.4 Loss, RL Algorithm, and Reward
We train the latent encoder state network and the denoiser network using gradient backpropagation, and the sampling importance network using RL-based policy optimization leveraging APPO [36]. Given the ground truth image, we backpropagate the gradients for the denoiser and the latent space using the mixed 21MMS-SSIM importance network SSIM loss [37] with ratio of O. 11-to--0.84 The reward for the sampling be ten the power of one minus the loss. Our RL framework can be represented by (S, A, P. R. 7). with S the observation space, A is the action space, P is the transition probability function from action and old observation to new observation, and y is the discount factor. Given that is the denoiser, is the latent state encoder, h is the ray sampler, and is the time warper: P(s'ls, a) follows the distribution of (w o ()((), s). R(s, a, s') is 101-lossee (oss((fog)(t((a)))))d) where gd is the corresponding ground truth image, and loss is the mixed & MMSSSSM loss. 0.99.
the reward function,
the space [-1; [-1:1]=20-700--0
A is in the space [-1: 72000720
2.5 Training
All networks are trained in a closed loop 100 epochs with batch size of 4. We train all networks end-to-end, such that the sampling importance network can insert more samples where the denoiser does poor job, and vice versa. The learning rate schedule follows the same pattern as the learning rate used to train the OIDN Denoiser [9]: the learning rate has maximum value 0.1 and minimum value of 10-$, with a warmup phase of 15%, and then an exponential decay phase. We use Adam optimizer [38], Pytorch and Ray-RLlib [39]. We transform the images by using vertical and/or horizontal flips, by randomly cropping and rescaling the images. We randomly permute the order of input images to teach state encoder to be permutation invariant.
Baselines We present in Table 1 the description and inference time of every approach. Unlike our work, some of the presented baselines did not release their implementation [23, 24, 25, 31], we therefore reimplemented them as described by the authors or used unofficial implementations, and trained all algorithms using the same amount of data and epochs.
Table Description, visual quality evaluation at 4.0 spp budget, and inference time in ms.
Algorithm
Description
spp PSNR (dB)
Inference time (ms)
name
Our method
28.7
22.5
ours-A1
Ours, no RL, but gradient approx.
28.4
22.5
ours-A2
Ours with uniform sampling
27.9
20.0
ours-B1
Ours, no latent space encoder
28.0
19.0
ours-B2
Ours-B1, no temporal feedback
27.4
18.8
ours-C
Ours with averaged ray samples
28.2
22.1
ours-small
Ours scaled-down version
27.3
14.4
IMCD
[31]
27.9
46.4
NTAS
[23]
27.7
17.4
DASR
[20]
27.2
21.9
ReSTIR+OIDN
ReSTIR denoised by OIDN [25, 9]
28.0
18.3
SVGF
[24]
22.8
3.9
OIDN
[9]
26.2
16.3
ReSTIR
[25]
22.1
2.0
MC
Monte-Carle path tracing average
17.0
0.0
5

---

Figure 2: A ground truth image from the Emerald scene, the Ripple Dreams scene, the ZeroDay scene, and the Suntemple scene (going left right)
3 Results
3.1 Experimental Setup
As in related work [23, 20, 24), we precompute dataset such that live communication with the path-sampler is not necessary. We use the path-sampler Cycles using the GPU's RTX acceleration. We use three 3D scenes Emerald Square [40], SunTemple [41], and Zero-Day [42] released as part of the Open Research Content Archive (ORCA) under CC 4.0 license. We also use the scene Ripple Dreams released by James Redmond. The scenes we used from the ORCA project are the biggest available subset of the scenes used in the most related work [23]. Unfortunately, the ORCA scenes have been further modified and not shared by the authors of [23], which makes an exact usage of their scenes impossible. We release our dataset scenes and code implementation' to facilitate the usage and comparison of our method.
For every 3D scene, we extract sequence of frames that forms continuous video animation. For the Emerald scene we extract 1400 frames, for the ZeroDay scene we extract 400 frames, for the SunTemmle scene we extract 1200 frames, and for the Ripple Dream scene we extract 500 frames. We further split those animations into independent clips of 20 frames. For every frame, we compute the ground truth image which is unbiased Monte-Carlo estimate using 1000 the additional data which includes normals, albedo, depth, motion vectors (a pixel-to-pixel mapping from one frame to the next one). and the inputs which are 8 path-traces images with 1 spp each. Previous work [23, 20] instead stored input images with spp count 2 for i = 0 to 5, such that they could compute the average pixel values in the range [0: Σ 2 = [0; 63) using only 5 input frames, whereas our method gives access to between O and 8 pixel values instead of only the average. The limit of 8 samples has been chosen as we have seen that less than 0.6% of the pixels get higher spp count recommendations in 4 spp average count scenarios. For scenarios with higher spp rendering counts than 4 (which would not be realistic in practice for real-time usage), increasing this number could be required at the cost of increasing GPU memory usage.
We perform our measurements on a Nvidia GeForce RTX 2080 Ti GPU. The sampling times for 1 spp frames with resolution of 720 x 720 shown in Table 2.
3.2 Quantitative Results
In the result section, will present the cross-validated results for each scene: For every test scene, we train on all the other scenes, and output the average PSNR over the whole test scene. We use PSNR metric because this is the most widely used metric used in the relevant literature [23, 27, 20]. Instead of presenting the PSNR over latency time per frame for a single spp count as in the related literature, we include several different spp counts to get insight into the quality over time tradcoff for every method. in this section we present the averaged results, but the cross-validated results per dataset are available in the Appendix. We measure a standard deviation of 0.06 dB for our method based on 5 trainings with different seeds, which makes barplots or confidence intervals unnecessary.
3.3 Qualitative Results
We present in Figure 4 a qualitative example of the visual quality of our method's 's rendering compared to the previous state of the art. Please look at the additional material to observe more qualitative results including videos.
Source code available httpp:://giiiiii com/ AJSVB/JVVVBBBB BBSSVB/BB_PPP TAACIICIII

---

p
x
ours
24
PSNR (dB)
ours-small
X DASR
22
IMCD
T SVGF
20
MC
ReSTIR
OIDN
ReSTIR+OIDN
20
100
120
140
frame latency (ms/frame)
Figure 3: PSNR quality function frame latency for the overall cross-validated averaged results among all datasets. Magenta color corresponds to 0.01 spp count, cyan color corresponds to 0.1 spp count, black color corresponds to 0.3 spp count, green color corresponds to 0.5 spp count, red color corresponds to spp count, yellow color corresponds to 2.0 spp count and blue color corresponds to a 4.0 spp count. Upper-left better.
Table 2: Sampling time for 1 spp given our datasets and hardware.
Dataset
1 spp sampling time (ms)
Suntemple
16.6
Ripple Dream
17.6
Emerald
34.5
ZeroDay
36.6
Average
26.3
4 Discussion
Impact of the Sample Count In Figure 3, we evaluate the tradeoff between frame latency and PSNR and compare to previous methods for various sample counts. Notice that only adaptive methods can use non-integer spp budgets because there is no procedure to choose how to sample non uniformly for non-adaptive methods. Previous adaptive methods approach uniform sampling as their spp budget approaches 1.0 because they cannot sample less than I spp for any pixel. Indeed, the gradient approximation (Equation 1) diverges case the spp recommendation 0. This imposes a hard lower limit the frame latency. Contrapositively, only method than 1.0 because we train the sampling importance network with reinforcement learning to overcome the diverging gradient limitation. We observe that the visual quality collapses for very low spp counts (0.01 spp), greatly increases from 0.01 to 2 spp, and then increases slower until 4 spp count.
spp budget smaller
We present a variant of our method called "ours-small" that corresponds to our method with smaller sampling importance, latent encoder, and denoiser networks (see the network variational study in the Appendix). This variant has smaller inference time which allows reaching even lower frame latencies. We observe that this variant is Pareto-optima until 38 ms/frame. We also notice that this variant really starts improving after reaching a 0.1 spp count, whereas our main method already improves between 0.01 spp and 0.1 spp counts. This indicates that our variant with small networks is not able to fully leverage the potential 0. spp count.
Comparison of our Method with other Related Work observe that our method Pareto- optimal compared to previous methods for any spp budget. Not only does our method outperform any other baseline for equal spp counts, but also for higher spp counts: our method with spp budget outperforms every baseline with 4 spp budget. Given that some of the current trends in rendering aim

---

Figure 4: Ground truth using 1000 spp on the left, ours using 4.0 spp on the middle, previous adaptive sampling state-of-the-art (Neural Temporal Adaptive Sampling [23]) with spp on the right. Image rendered the Ripple Dream dataset. 720x720 frame top, and 200x200 zoom particular area on the bottom. The PSNR for our rendered image is 29.1 dB and is 27.7 dB for the previous state-of-the-arr rendered image.
at minimizing the latency time (no adaptive sampling, supersampling....) the fact that method has a relatively large latency but manages to outperform quality-wise very fast baselines for an equal latency is interesting result [27, 24, 25]. Given equal budget of 4 spp, our method renders outputs with a OB higher PSNR compared to the strongest baseline for a negligible latency increase ms latency increase which corresponds to 5.5%). Our approach with spp reaches higher visual quality (PSNR improvement of 0.2 dB) for a 1.6x latency reduction compared to the strongest baseline spp.
Ablation Study of our Method We observe Table that method is among the slowest in terms of ms/frames, but generally outperforms all other baselines. Our method and three of our variants (ours-A1, ours-B1, ours-C) always outperform all previous works except ReSTIR+OIDN [25, 9] for an equal spp count. On average, our method outperforms the variant with the gradient approximation (ours-Al) by 0.3 dB, and the variant with uniform sampling (ours-A2) by 0.8 dB. This demonstrates both the utility of adaptive sampling and the superiority of training the sampling importance network with RL instead of the gradient approximation. We include study of the sampling heatmap and error distribution for several scenes in the supplementary material to improve the understanding of the effect of RL-based sampling vs gradient-basee sampling.
Comparing our method to ours-BI. B1, we see that using the latent space encoder and having as temporal feedback the warped latent space instead of the sampled pixel values (ours-B1) allows increasing the PSNR by 0.7 dB, and increases the PSNR by 1.3 dB in the case with no temporal feedback (Ours-B2) Finally using the sampled pixel values instead of the averaged values (ours-C) CC) increases the PSNR by 0.5 dB.
Comparison with Adaptive Sampling SOTA Our method using RL for the adaptive sampling outperforms our method using the gradient approximation (Qurs-A1) for the adaptive sampling. Furthermore, our method with the gradient approximation outperforms NTAS [23] by large margin

---

(0.7 dB) showing the importance of using the individual sampled pixel values instead of the averaged values, and of using a latent space encoder.
Comparison with Hybrid Latent State SOTA Our method with uniform sampling (Ours-A2) has as high visual quality as IMCD [31]. This is not surprising because both approaches use temporal feedback and use sampled pixel values instead of only the average. The main difference in the design our state encoder that creates a meaningful state, whereas IMCD only uses simple heuristics like an exponential average of the previous latent spaces. This difference allows our variant to reach equal visual quality despite the 2.1x smaller inference time.
Comparison with Probabilistic Latent Space SOTA ReSTIR+OIDN [25, 9] uses probabilistic heuristics and the non-averaged sampled pixel values to recommend from which light to sample, and choose how to update the spatiotemporal latent space. Our method conceptually has the same objectives. The fact that our method outperforms ReSTIR+OIDN by 0.5 dB shows that using networks that learn how to do the sampling and how to update the spatiotemporal latent state instead of using theoretical heuristics allows reaching higher results.
Visual Quality Comparison Looking at Figure 4, we observe that our method generally looks more similar to the ground-truth image. For example, large uniform surfaces present less noise and advanced details such as edges, small bubbles, reflection, and refraction effects look more realistic with our method. In the supplementary material, we compare rendered animations and observe that our method manages to maintain temporally more coherent results as well.
The current method does not consider the application of after-effects such as motion blur, which could allow the reduction of samples collected on fast-moving objects, while in the current setting we would implicitly focus the ray tracing samples such an object to minimize the error. Further, this method is applicable to entertainment applications and can potentially generate artifacts not present in the real scene, which could be obstacle in application scenarios such as VR/AR medical devices. Additionally. it requires a system capable of performing both path tracing and DNN inference. While current graphics cards provide this capability and assess the frame rendering time considering both components, the underlying workloads are significantly different: DNN inference is generally very structured and high arithmetic intensity workload whereas path tracing is -inttiiingi and performs random look-ups into memory. As these are fundamentally different, we see dedicated ray tracing units on modern GPUs. In future systems, it is conceivable that dedicated devices are used for each step, which would enforce fixed capability for each type of compute and limit a free trade-off between the two as we make use in this work. Additionally, applying a DNN adds a memory overhead, although it can remain minimal compared to other components such as textures that commonly fill most the GPU's memory. Specifically, the model requires 110 MB more memory to store the input and latent space data, <50 MB to store the model, and a few MB of working memory for intermediate feature maps that can be processed in tiles.
Limitations
Memory Overhead We store the following data in memory. 1) the model weights (<50MB), 2) the sampled pixel values (24 channels; RGB values for up to 8 non-averaged samples), 3) 7 additional input channels (3 for surface normals, 3 for albedo, for depth), 4) 32 channels for the state (warped latent space), and 5) small working memory for intermediate feature data during inference. The components 2-4 add up to 53 more channels than previous works (which use 3 channels for input channels and 7 for additional data); hence 212 Byte/pixel: 10MB in total for 720x720 pixel data; and the required working memory is minimal as the inference can be done on tiles. With a total memory footprint in the order of 200 MB, the overhead is insignificant (<2%) compared to > >12GB of textures loaded for current games on high-end GPUs where such capacity is available.
5 Conclusion
We explored three different ideas to improve sampling and denoising for low sampling count and showed through a siine-onecoontribttionooo that they are all important: The first one is to adaptively sample leveraging RL. to make better sampling decisions; using the previously recommended gradient approximation leads to 0.3 dB PSNR decrease. The second one is to let a deep learning latent state

---

encoder decide which pixel information to keep given the previous state to optimize the denoising and further rendering; removing this network leads to a 0.7 dB PSNR decrease. The third one is to not aggregate the pixel values by averaging, but to keep all information for the state encoder network; averaging the sampled pixel values leads to 0.5 dB PSNR decrease. We discovered that combining those ideas allows outperforming different state-of- ttate-of-the-arts by at least 0.7 dB for an equal spp budget, or by a 1.6x latency reduction while keeping a marginal (0.2 dB) visual quality improvement. Finally, we are the first adaptive sampling method that can accept a lower spp budget than spp, allowing using adaptive sampling for unprecedentedly high frame rates, Finally, we release our implementation and datasets.
References
[1] J. Arvo and D. Kirk, "Particle transport and image synthesis," in Proc. Conf. on Computer Graphics and Interactive Techniques, 1990. pp. 63-66.
[2] M. F. Cohen, R. Wallace, and Hanrahan, Radiosity and realistic image synthesis. Morgan Kaufmann, 1993.
[3] R. L. Cook and K. E. Torrance, "A reflectance model for computer graphics," ACM Siggraph Computer Graphies, vol. 15, no. 3, pp. 307-316, 1981.
[4] A. S. Glassner, An introduction to ray tracing. Morgan Kaufmann, 1989.
[5] Y. Huo and S.-e. Yoon, "A survey on deep learning-based monte carlo denoising," Computational visual media, vol. 7, pp. 169-185, 2021.
[6] H. Dammertz, D. Sewtz, J. Hanika, and H. P. Lensch, "Edge-avoiding a-trous wavelet transform for fast global illumination filtering," in Proceedings of the Conference on High Performance Graphics, 2010, pp. 67-75.
[7] S. Bako, T. Vogels, B. McWilliams, M. Meyer, J. Novák, A. Harvill, P. Sen, T. Derose, and Rousselle, "Kernci-predicting convolutional networks for denoising monte carlo renderings." " ACM Trans. Graph., vol. 36, no. 4, pp. 97-1, 2017.
[8] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in Proc. Int. Conf. on Medical Image Computing and Computer-Asisted Intervention (MICCAI). Springer, 2015, pp. 234-241.
[9] A. T. Áfra, "Intel@ open image denoise," 2019.
[10] C. R. A. Chaitanya, A. S. Kaplanyan, C. Schied, M. Salvi, A. Lefohn, D. Nowrouzezahrai, and
T. Aila, "Interactive reconstruction of monte carlo image sequences using a recurrent denoising autoencoder," ACM Transactions Graphics (TOG), vol. 36, no. 4, 2017.
[11] M. E. Lee, R. A. Redner, and S. P. Uselton, "Statistically optimized sampling for distributed ray tracing," in Proc. Conference on Computer Graphics and Interactive Techniques, 1985, pp. 61-68.
[12] R. S. Overbeck, C. Donner, and R. Ramamoorthi, "Adaptive wavelet rendering." ACM Trans. Graph., vol. 28, no. 5, p. 140, 2009.
[13] K. Egan, Y.-T. Tseng, N. Holzschuch, F. Durand, and R. Ramamoorthi, "Frequency analysis and sheared reconstruction for rendering motion blur," in Proc. ACM SIGGRAPH, 2009.
[14] N. K. Kalantari and P. Sen, "Removing the noise in monte carlo rendering with general image denoising algorithms," " in Computer Graphics Forum, vol. 32, no. 2pt1. Wiley Online Library, 2013, pp. 93-102.
[15] F. Rousselle, C. Knaus, and M. Zwicker, "Adaptive sampling and reconstruction using greedy error minimization," " ACM Transactions on Graphics (TOG), vol. 30, no. 6, 2011.
[16] B. Moon, N. Carr, and S.-E. Yoon, "Adaptive rendering based on weighted local regression," ACM Transactions Graphics (TOG), vol. 33, no. 5, 2014.
[17] B. Moon, S. McDonagh, K. Mitchell, and M. Gross, "Adaptive polynomial rendering," ACM Transactions on Graphics (TOG), vol. 35, no. 4, 2016.
[18] Viitanen, M. Koskela, K. Immonen, M. Mākitalo, P. Jääskeläinen, and J. Takala, "Sparse sampling for real-time ray tracing." in VISIGRAPP (1: GRAPP), 2018, pp. 295-302.
10

---

[19] M. Zwicker, W. Jarosz, J. Lehtinen, B. Moon, R. Ramamoorthi. F. Rousselle, P. Sen, C. Soler, and S.-E. Yoon, "Recent advances in adaptive sampling and reconstruction for monte carlo rendering," in Proc. Computer Graphics Forum, vol. 34, no. 2. Wiley Online Library, 2015, pp. 667-681.
[20] A. Kuznetsov, N. K. Kalantari, and Ramamoorthi, "Deep adaptive sampling for low sample count rendering," in Computer Graphics Forum, vol. 37, no. 4. Wiley Online Library, 2018, 88 pp. 35-44,
[21] Salchi, M. Manzi, G. Roethlin, R. Weber, C. Schroers, and M. Papas, "Deep adaptive sampling and reconstruction using analytic distributions," ACM Transactions on Graphics (TOG), vol. 41, no. 6, 2022.
[22] Y. Huo, Wang, R. Zheng, H. Xu, H. Bao, and S.-E. Yoon, "Adaptive incident radiance field sampling and reconstruction using deep reinforcement learning," ACM Transactions on Graphics (TOG), vol. 39, no. 1, 2020.
[23] Hasselgren, J. Munkberg, M. Salvi, A. Patney, and A. Lefohn, "Neural temporal adaptive sampling and denoising," in Computer Graphics Forum, vol. 39, no. 2. Wiley Online Library, 2020, pp. 147-155.
[24] C. Schied, A. Kaplanyan, C. Wyman, A. Patney, C. R. A. Chaitanya, Burgess, S. Liu, C. Dachsbacher, A. Lefohn, and M. Salvi, "Spatiotemporal variance-guided filtering: real- time reconstruction for path-traced global illumination," in Proceedings of High Performance Graphics, 2017.
[25] B. Bitterli, C. Wyman, M. Pharr, P. Shirley, A. Lefohn, and W. Jarosz, "Spatiotemporal reservoir resampling for real-time ray tracing with dynamic direct lighting." " ACM Transactions on Graphics (TOG), vol. 39, no. 4, pp. 148-1, 2020.
[26] C. Wyman and A. Panteleev, "Rearchitecting spatiotemporal resampling for production," " in Proceedings of the Conference
Hfigh-Performant Graphics, 2021, pp. 23-41.
[27] Y. Ouyang. S. Liu, M. Kettunen, M. Pharr, and J. Pantaleoni, "Restir gi: Path resampling for real-time path tracing," in Computer Graphics Forum, vol. 40, no. 8. Wiley Online Library, 2021, pp. 17-29.
[28] T. Hachisuka, W. Jarosz, R. P. Weistroffer, K. Dale, G. Humphreys, M. Zwicker, and H. W. Jensen, "Multidimensional adaptive sampling and reconstruction for ray tracing," " in Proc. ACM SIGGRAPH, 2008.
[29] J. Lehtinen, T. Aila, J. Chen, S. Laine, and F. Durand, "Temporal light field reconstruction for rendering distribution effects," " in Proc. ACM SIGGRAPII, 2011.
[30] J. Lehtinen, Aila, S. Laine, and Durand, "Reconstructing the indirect light field for global illumination," ACM Transactions Graphics (TOG), vol. 31, no. 4, 2012.
[31] M. Işık, K. Mullia, M. Fisher, J. Eisemmann, and M. Gharbi, "Interactive monte carlo denoising using affinity of neural features," ACM Transactions on Graphics (TOG), vol. 40, no. 4, 2021.
[32] M. Gharbi, T.-M. Li, M. Aittala, J. Lehtinen, and F. Durand, "Sample- based monte carlo denoising using a kernel-splatting network," ACM Transactions on Graphics (TOG), vol. 38, no. 4, 2019.
[33] S. Liu, R. Chen, and T. Logvinenko. "A theoretical framework for sequential importance sampling with resampling," Sequential Monte Carlo methods in practice, pp. 225-246, 2001.
[34] C. Nwankpa, W. Ijomah, A. Gachagan, and S. Manshall, "Activation functions: Comparison of trends in practice and research for deep learning." arXiv preprint arXiv: 1eee:811133333 1.03378, 2018.
[35] Y. LeCun, L.. Bottou, Y. Bengio, and P. Haffner, "Gradient- -aased learning applied to document recognition," Proceedings of the IEEF, vol. 86, no. 11, pp. 2278-2324, 1998.
[36] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint 1.XXv:1707.06332 2017.
[37] H. Zhao, O. Gallo, I. Frosio. and J. Kautz, "Loss functions for image restoration with neural networks." " IEEE Transactions on computational imaging, vol. 3, no. 1, pp. 47-57, 2016.
[38] D. P. Kingma and Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014.
11

---

[39] E. Liang, R. Liaw, R. Nishihara, P. Moritz, R. Fox, K. Goldberg, J. Gonzalez, M. Jordan, and I. Stoica, "Rllib: Abstractions for distributed reinforcement learning," in International Conference on Machine Learning. PMLR, 2018, pp. 3053-3062.
[40] K. A. Nicholas Hull and N. Benty, "Nvidia emerald square, open research content archive (orca)," " July 2017, http://davellpprrr avidia, com/ -t
[41] E. Games, "Unreal engine temple, open research content archive (orca)," October 2017,
http://deeeelper..viiie wwwp:ppe...iiii...o.oooooo
[42] M. Winkelmann, "Zero-day, open research content archive (orca)"" November 2019.
https://developer.nut nvidia cww cotporra/ebppppeeeerrroooooo
12

---

A Individual Results Per Scene
The analysis in Section 3.3 averages the latency and PSNR across different scenes. This could potentially hide differences in the scene complexity where the cost of sampling ray might vary significantly, We thus provide a more fine-grained per-scene evalautions in Figure 5.
nabural
BCD
1971
KDG
HING
MC
ODM
OGM
Iarn Mancy Irethaad
marss Warr inotsse
30
3755
Firs
bick
MC
HINS
SUsE
5007
Resta
15.
18
AJSTT8-C.OM CM
152
37%
120
Inaters!
Figure 5: PSNR quality as a function of time per frame on the cross- cross- validated scenes EmeraldPlace (top left), Sun SunTemple (top right). ZeroDay (lower left) and Ripple Dream (lower right). Magenta color corresponds to a 0.01 spp count, cyan color corresponds to O. spp count, black color corresponds to 0.3 spp count, green color corresponds yellow color corresponds to a 2.0 spp count and blue color corresponds to a 4.0 spp count. Upper-left better.
0.5 spp count, red color corresponds to sp0 count,
B Architecture and Variational Network Study
The basic architecture of our networks is visualized in Figure 7. We justify the choice of size of the different different models used within our solution using variational study: we fix time budget of 100 ms per frame and select a single scene such that the sampling time for 1.0 spp is fixed. We then study how varying the size of each of our networks affects the quality of the results, taking into account that changing the size of the network will change the total sampling budget. We evaluate our method the EmeraldPlace dataset.
For the small variant of the denoiser we scale down the architecture shown in Figure 7 by reducing the number of channels by 2 every convolution layer, and its large variant is identical except scale up the number of channels by 2 x. The small sampling importance network is a simple CNN with three 3 x 3 convolutional blocks using latent channel, and the large sampling importance network is the original UNET from [8]. The small state encoder architecture corresponds to the architecture in Figure 7 except replaced the 3 x 3 convolutions with 1 x convolutions and the big state encoder architecture corresponds to the architecture in Figure except that we added two additional 3 x 3 convolutions with ReLU layer in between. Ours-small mentioned in the paper corresponds using both the small sampling importance network, the small state encoder, and the small denoiser network.
Sampling Importance Network This variational study shows that choosing a very fast sampling importance network is beneficial compared to using a slower architecture for our fixed time budget experiment. This result probably does not hold when the time budget increases because the inference time of the network will become negligible. Nevertheless, we focus real-time applications which means that the time budget has to be smaller or equal to 30 ms. This architecture design contrasts with previous designs in adaptive sampling [23, 20], where the sampling importance network and the
13

---

Table 3: Inference time in ms and corresponding spp count for every network. Spp count is computed as follows: given the total inference time, spp count = (100-x)/69.0, 00, where 69.0 the sampling time in ms for spp on EmeraldPlace.
Variation
Sampling importance
State encoder
Denoiser
Total
Spp count
Normal
2.5
3.7
16.3
22.5
1.12
Small Sampling importance
2.0
3.7
16.3
22.0
1.13
Large Sampling importance
4.5
3.7
16.3
24.5
1.09
Small state encoder
2.5
2.2
16.3
21.0
1.14
Large state encoder
2.5
9.4
16.3
28.2
1.04
Small denoiser
2.5
3.7
10.2
16.4
1.21
Large denoiser
2.5
3.7
25.1
31.3
0.99
large denoiser
large encoder
26.30
large recommender normal
26.25
recommender small state encoder
26.20
PSNR (ci)
26.15
26.05
25.95
1.00
1.05
1.10
1.15
1.20
Sampling budget (spp)
Figure 6: Comparison of PSNR when varying the size of every network for equal latency constraints.
Higher is better.
denoiser network had almost the same architecture and inference time. This difference in design can be explained because take the latent space input which pre-processed information, whereas previous approaches give row information as input, and the sampling importance network typically had to guess which were the high variance areas, for example by learning to segment edges, whereas our method can more easily extract more meaningful data like sample count, or variance from the latent space.
State Encoder Network We also observe that the state encoder does not need to be UNET, which implies that local information is sufficient update the latent state.
Denoiser Network Finally, the denoiser network takes the majority of the inference time of approach. Our architecture almost identical to the denoising architecture of [9]. We observe the ablation study that trying shrink this architecture strongly deteriorates the results.
14

---

Ine44
3x3 Convolutionn
Mox Pooling 2x2
Upsompling
RELU
3x3 Coom
Tarth
64
96
96
Ove
Pool
Figure 7: Architecture of the sampling importance network (top left), the latent state encoder network (top right), and the denoiser network (bottom). Max Pooling 2 x 2 refers pooling with kernel size 2 and stride 2. Typically, given an input image with resolution of 720 x 720, the resolution after the first pooling is 360 360, after the second is 180 180, after the third is 90 x 90, after the fourth 45 x 45, and the symmetrical opposite resolutions for the upsampling way. The number above the convolutional blocks represents the number of output channels for every layer in the block. Upper arrows represent residual connections. The concatenation layers are not shown but happen after every upsampling layer.
C Sampling Strategy Comparison
Identifying the differences between the sampling heatmaps when using reinforcement learning and when using the gradient approximation in non-trivial (Figure 10). We attribute this to the sampling importance network learning adaptively sample by taking into account the impact of the denoiser, and not simply by sampling in high variance areas like edges. We found it more informative to compare the MSE per pixel between the RL adaptive sampling and gradient approximation adaptive sampling method (Figure 8, Figue 9). In the additional material, we include similar data for moving animations. The per-pixel difference in MSE between the methods remains challenging interpret directly (Figure 8). To otherwise visualize the per-pixel MSE gains, we generated histograms (Figure 9) clearly showing that the RL-based method leads generally better results as the mode and the mean of the distributions are always negative. In particular, we observe from the lower histogram that our RL-based method particularly avoids large errors.
Sparse Sampling A common strategy that we observe in both our method using RL and using the gradient approximation, but that we did observe in previous adaptive sampling methods is sparse sampling. Given uniform area, sparse sampling consists of a pattern such that few pixels get high spp count recommendation, and surrounding pixels get low spp count recommendation. This can be an efficient sampling strategy assuming that locally close pixel values are not independent (cf. Figure 10).
15

---

Figure 8: Comparison of the binary MSE difference between our method when trained with RL (top left), and the gradient approximation (top right). White colors mean that the MSE of the method with the gradient approximation is bigger for this pixel and inversely for black colors.
Comparison of the extreme binary MSE difference between our method when trained with RL (center left), and the gradient approximation (center right). White colors mean that the difference in MSE of the method with the gradient approximation is bigger than the threshold for this pixel, and inversely for black colors.
Ground truth images included for reference (bottom left and bottom right).
16

---

12000
-N
420
-Wb
3600
1000
1500
1900
36
15000
3080
Figure 9: Comparison of the distribution of MSE differences between our method when trained with RI. and with the gradient approximation. Negative values mean that the MSE of the method with the gradient approximation is bigger and inversely for positive values. The top image corresponds to the distribution for the left image in Figure and the bottom image to the distribution for the right image in Figure 8.
Figure 10: Comparison of the sampling recommendation from sampling importance network when trained with RL (left), and the gradient approximation (center). Lighter colors mean higher sampling recommendations. Identical color scale. We observe sparse sampling and recurrent sampling patterns in both sampling heatmaps. Ground truth image included for reference (right).
17

---

D Supplementary Material
Together with the paper, we provide several short video sequences to verify that our solution does not introduce any flickering other artifacts and generally enable a qualitative comparison of methods. GD-ours-NZA -f extends Figure 4 in the main manuscript. We include FlG4a.gif. FIG4b.gif FIG4c.gif. FIG4d.gif. and FIG4e.gif that extend Figure 8 of this manuscript. We include FIGupper.gf and FIGlowergif that extend Figure from this manuscript. The GIFs have been compressed to max. 100MB.
18


Reviews:
[{"review_id": "tDSjCnBu1L", "reviewer": "Reviewer_CLYz", "summary": "This paper proposes to use reinforcement learning (RL) to improve adaptive sampling effectiveness in Monte Carlo ray tracing.  Another key contribution they claim is the use of a latent space representation to encode temporal information, which improves the reuse of spatiotemporal pixel information across frames.  The impact of this work is to improve quality of real-time path tracing results with respect to noise and temporal stability.", "questions": "In Figure 2 and through the paper, some of the ground truth rendered images appear very dark.  Is this intentional and if so, why?\n\nHow sensitive is the inference performance to the choice of training data?  Are there any challenges generalizing to new rendered scenes?\n\nThis paper reviews the approach for approximating gradients from \"Deep Adaptive Sampling for Low Sample Count Rendering (Kuznetsov et al.)\", and proposes RL as an alternative to their formulation.  Out of curiosity, would integrating a differentiable renderer (e.g. Mitsuba) into the pipeline from Kuznetsov et al. provide any useful additional gradients?\n\nCould you elaborate on why RL specifically is a good learning paradigm for learning the importance sampling module?  Does using RL ", "limitations": "The paper does not identify any limitations, and I do not see any clear limitations.  ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written, and provides a clear preliminary of the necessary background on adaptive sampling and denoising, as well as reviewing current SOTA approaches which are compared with their method.  The usage of the \"spatiotemporal\" latent \nspace, which is claimed to encode temporal information more effectively than previous works, is an insightful idea.  The RL-based, learned importance sampler is also an interesting approach that addresses limitations of prior works.  Qualitatively and quantitatively, their method outperforms other methods by a significant margin.", "weaknesses": "Evaluation of the method is limited to only a small number of scenes.  It would be nice to see a larger scale evaluation, though it is understandable that training/evaluation data is laborious to assemble.\n\nPossible lack of theoretical novelty -- the method is validated with empirical results which are quite promising, but there is little discussion or analysis of the stability / robustness of the method.\n\nAdditional images for qualitative comparison could be helpful for visualizing some of the key concepts discussed in the paper.  For example, visualizing qualitative differences between various sample counts, or dedicated visualizations of the estimated sample maps and/or RL state.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer CLYz, thank you for your time and interesting remarks.\n\n> Additional images for qualitative comparison could be helpful for visualizing some of the key concepts discussed in the paper. For example, visualizing qualitative differences between various sample counts, or dedicated visualizations of the estimated sample maps and/or RL state.\n\nWe provide qualitative comparisons and estimated sample maps for various sample counts in the supplementary material as well as some additional visualizations: state (stt.png), sample maps (hitmap.png, 540hitmap.png) and comparisons for various sample counts (0.1spp.gif, 1.0spp.gif, 4.0spp.gif) in the following anonymous link: https://drive.google.com/drive/folders/1zrKGelIwSxcGkww6u3-MYDRyVTO45SJ3?usp=sharing . We will include them all in the final supplementary materials. \n\n> How sensitive is the inference performance to the choice of training data? Are there any challenges generalizing to new rendered scenes?\n\nOur main results are obtained by performing a Leave One Scene Out Validation (See line 235). This means that given one test scene, we train on all scenes except the test scene, and then evaluate on the test scene. We repeat the same procedure such that every scene becomes the test scene once and finally average results. Our results show low sensitivity to the choice of training data (see appendix for individual Leave One Scene Out results) and good generalization to the unseen data from the test scene. \n\n> In Figure 2 and through the paper, some of the ground truth rendered images appear very dark. Is this intentional and if so, why?\n\nThe ground truth images selected for visualization in the paper are not outliers in terms of brightness. We agree that two of the four images appear dark in the paper. However, zooming in on them in full-screen without any of the paper's white background visible as to mimick an in-game experience will change the perception to this being a rather \"normal\" brightness. \nWe do not feel comfortable performing range compression or otherwise modifying the data for an improved visual appearance of the ground truth frames in the figures in the paper. \n\n> This paper reviews the approach for approximating gradients from \"Deep Adaptive Sampling for Low Sample Count Rendering (Kuznetsov et al.)\", and proposes RL as an alternative to their formulation. Out of curiosity, would integrating a differentiable renderer (e.g. Mitsuba) into the pipeline from Kuznetsov et al. provide any useful additional gradients?\n\nWhile the renderer would become differentiable, the hard/non-differentiable decision on how many samples to collect would remain. It would open up the possibility to learn a continuous-valued sampling recommendation which could be quantized to integer values with a straight-through estimator, stochastic rounding, or similar way during the forward pass and inaccurate gradients in the backward pass. However, it is not immediately obvious how this would impact the results, i.e., if there would be a benefit to dropping the RL component in favor of inaccurate gradients (straight-through estimator, stochastic rounding). In any case, it would be impractical, as artists develop scenes for a specific game engine that is then the only program that can render the desired output, which generally does not support differentiable rendering. \n\n> Could you elaborate on why RL specifically is a good learning paradigm for learning the importance sampling module? \n\nQuantized problems, such as choosing an integer number of samples to collect per pixel, are commonly addressed by using hard decisions or stochastic rounding in the forward step and computing (very) approximate gradients for the backward pass (e.g., straight-through estimator). This negatively impacts the learning process and quality of results. We can avoid such an approximation and instead formulate an exact end-to-end optimization criterion using reinforcement learning. Using RL further allows collecting less than 1spp, which is a common limitation in related work that strongly impacts the capabilities in low-latency/real-time rendering scenarios. Permitting to collect 0 samples at some pixels with gradient-based learning implies that there is no longer a gradient to the sampling recommendation network and it can thus not be learned appropriately for this scenario. \n\nWe hope that we provided satisfactory answers to your questions and comments, and that any remaining concerns could be addressed.\n\nThe authors\n\n"}, {"review_id": "r8a171mX08", "reviewer": "Reviewer_WTeK", "summary": "This paper tackles the issue of Monte-Carlo path tracing which is an important field for computer graphics and rendering. The paper first analyzes the current state of the art and identifies mainly two flaws which are addressed thereafter: First, the authors introduce a spatio temporal latent space serving as input to the denoising autoencoder which outputs the final image frames as well as (as a feedback) to the importance sampling network which outputs a distribution for rays per pixel. Second, as opposed to previous work, this distribution is not sampled from a sampling heatmap as in previous works but since this is problematic during training due to a coarse numerical estimation of the output gradient with respect to the gradient. Instead an RL based method for importance sampling is used during training which removes this need for a numerical estimation of the gradient. The resulting model seems to yield higher PSNR values than previous works.   However I am very unfamiliar with this topic so I cannot really estimate how thorough the evaluation is done.", "questions": "* Is PSNR the only important metric in this field? SSIM should be a comparable measure in my experience. Why is SSIM not reported?\n* How does the method generalize to unseen data (not from validation/test sets). Did you try to apply it to some other data?", "limitations": "no limitations section is present. It would be imoportant to add such a section in my opinion.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The paper seems to identify limitations and flaws in current methods which are then overcome by simple solutions which can furthermore be trained in combination in and end-to-end manner. However since I don't know all the related work, I could've missed sth.\n* The proposed results seem to yield better results (measured by PSNR) at the cost of slightly higher inference times ", "weaknesses": "* For Table 1, there is no information regarding the dataset and resolution of the images/videos used to assess the presented values. This is confusing, because the PSNR values are shown for 4 spp, resulting in an inference time of 22.5 ms which is faster than base. However, in Fig. 3, the average inference time for the proposed method  is much higher . Can the authors clarify this?\n* The results are not presented as a function of image resolution (or number of pixels in the image). It would be interesting and important (e.g. to estimate the method's rea-world applicability) to report that, since nowadays most images/video to be rendered are of high spatial resolution.\n* training datasets are nowhere listed? Is the method trained on all three presented datasets jointly or independently for each of them?\n* Writing down the final objective function for the overall model would be good, since it would give the reader a final overview and summarize all the different aspects of the method. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer WTeK, thank you for your time and interesting remarks.\n\n> For Table 1, there is no information regarding the dataset and resolution of the images/videos used to assess the presented values. This is confusing, because the PSNR values are shown for 4 spp, resulting in an inference time of 22.5 ms. However, in Fig. 3, the average inference time for the proposed method is much higher. \n\nIn Table 1, we report the aggregate inference time of the neural networks or the method in general for related work. We exclude the pixel sampling time (thus also the 0.0 inference time for Monte-Carlo sampling). Differently, in Table 3, we report the entire frame latency of the whole frameworks including the pixel sampling, showing that our method is effectively faster. In terms of evaluation data, we use the procedure outlined in Section 3.1: Experimental Setup, i.e., the 4 mentioned scenes with leave-one-scene-out cross-validation. \n\n> The results are not presented as a function of image resolution\n\nWe trained and evaluated using resolutions of 720x720 pixels (cf. Line 233), as in OIDN, and more than in other related work (DASR: 512x512). Our method scales linearly in terms of inference time with the number of pixels for higher resolutions. For high-resolution rendering, the outputs of ray-tracing methods such as ours as well as related work are commonly applied at a below-native resolution and upsampled/interpolated using methods such as Nvidia's DLSS or AMD's FidelityFX Super Resolution as a post-processing step. An accurate estimate for your preferred resolution can thus be obtained by scaling the reported time proportionally to the ratio between the desired pixel count (after considering the preferred upscaling factor) and $720^2$. The relative latency to related work will remain unchanged. \n\n> Training datasets are nowhere listed? Is the method trained on all three presented datasets jointly or independently for each of them?\n\nThere are no commonly used datasets available for evaluating ray tracing methods. However, several scenes are available with licenses compatible to perform such research. We list the 4 scenes used in this work in Section 3.1 Experimental Setup (cf lines 211-213), and further release the code to reproduce the results in the supplementary materials. For the quantitative evaluations, we perform 4-fold cross-validation across the scenes (cf. lines 235-236), i.e., we pick one scene for testing and train on the remaining 3 scenes, then rotate to the next scenes for evaluation while averaging the results. \n\n> Writing down the final objective function for the overall model would be good, since it would give the reader a final overview and summarize all the different aspects of the method.\n\nIf we understand the remark correctly, this is presented in Line 192.\n\n> Is PSNR the only important metric in this field? SSIM should be a comparable measure in my experience. Why is SSIM not reported?\n\nWe agree that SSIM is the better metric as it is closer to visual perception, and thus we use its multi-scale variant in our loss function (Line 186). We report PSNR as the metric for comparability as all related works (DASR, NTAS, ReSTIR, OIDN, ...) evaluate with PSNR. \n\n> How does the method generalize to unseen data (not from validation/test sets). Did you try to apply it to some other data?\n\nThe test data is unseen. While we would like to expand both the training and test sets, the available data with a license permitting research use is very limited and most related work trains and evaluates on non-public data, making the results non-reproducible. We aim to improve on this by releasing code of not just our own method but also re-implementations of related work, with consistent evaluations on the same data. \n\n> No limitations section is present. It would be important to add such a section in my opinion.\n\nIf permitted, we will add the following section to the paper or at least to the appendix. \n\n```\nLimitations: The current method does not consider the application of after-effects such as motion blur, which could allow the reduction of samples collected on fast-moving objects, while in the current setting we would implicitly focus the ray tracing samples on such an object to minimize the error. Further, this method is applicable to entertainment applications and can potentially generate artifacts not present in the real scene, which could be an obstacle in application scenarios such as VR/AR medical devices. Additionally, it requires a system capable of performing both path tracing and DNN inference. While current graphics cards provide this capability and we assess the frame rendering time considering both components, the underlying workloads are significantly different: DNN inference is generally a very structured and high arithmetic intensity workload whereas path tracing is branching-intensive and performs random look-ups into memory.  In future systems, it is conceivable that dedicated devices are used for each step, which would enforce a fixed capability for each type of compute and limit a free trade-off between the two as we make use in this work. Additionally, applying a DNN adds a memory overhead, although it can remain minimal compared to other components such as textures that commonly fill most the GPU's memory. Specifically, the model requires 110 MB to store the input and latent space data, <50 MB to store the model, and a few MB of working memory for intermediate feature maps that can be processed in tiles.  This overhead is insignificant (<2%) compared to >12GB of textures loaded for current games on high-end GPUs where such capacity is available. \n```\n\nWe appreciate your dedication to thoroughly review NeurIPS papers including ours. We hope to have cleared out all or most of your concerns and would be delighted to see a revision of the rating. If any questions or doubts remain, we will gladly answer any further questions. \n\nThe authors"}, {"review_id": "97GvmrrxAD", "reviewer": "Reviewer_1MpK", "summary": "This paper proposes two techniques to improve the performance of Monte-Carlo patch tracing on real-time image rendering: 1) keep all previously sampled values to improve spatial-temporal information reuse; 2) use reinforcement learning to optimize the sampling importance network, avoiding the explicit numerically approximated gradients.", "questions": "See the weakness.", "limitations": "There are no potential negative societal impacts of this work.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem is clearly illustrated and the motivation is easy to understand. The experiments are adequate to demonstrate the arguments of this paper.", "weaknesses": "1. Line71 describes some methods that improve the spatiotemporal reuse by storing not only the averaged pxiel values. But there are no further discussion about their difference to the proposed method.\n\n2. Although a well-trained RL framework performs good during inference, the training of a RL framework is usually unstable. The authors are suggested to add more details about the training techniques.\n\n\n\n------------Post Rebutal-----------\n\nI have carefully read other reviewers' comments and the authors' responses. Although I think this paper proposed a good method for the defined problem, other reviewers argue that most of the techniques used in this paper have been discussed in previous papers, which heavily weakens the novelty of this paper. Since I am not familiar with this topic, I adjust my score for the concern of the novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer 1MpK, thank you for your time and interesting remarks.\n\n> Line 71 describes some methods that improve the spatiotemporal reuse. But there are no further discussion about their difference to the proposed method.\n\nThe mentioned methods are ReSTIR and derivative work of ReSTIR. Those methods do not use neural networks and are very different from ours: \n Their latent space consists of storing a buffer of lights for every pixel. They update the buffer by sampling from new visible lights or reusing spatiotemporally nearby pixels, leveraging simple statistical heuristics (importance sampling). This latent space is directly used only for denoising. \nDifferently, our latent space consists of storing a learned encoding given all previously available information to guide future sampling and denoising. It is updated using a dedicated neural network given new information and the previous state.  \nWe further discuss their conceptual and qualitative differences to our method in Figure 3 and lines [108-123, 266-276, 302-307].\n\n\n\n> Although a well-trained RL framework performs good during inference, the training of a RL framework is usually unstable. The authors are suggested to add more details about the training techniques.\n\nRL training is indeed often unstable. This is linked to several factors such as delayed rewards, non-stationarity, and off-policy learning. In our case, these factors are not as present as in other applications: rewards are delayed for very few frames/time steps as opposed to robotics applications; non-stationarity is present through the latent space, however, an effective latent space representation (i.e., our only system state) is not required to produce a first meaningful output and thus allows for a smooth instead of a steep learning curve; off-policy learning is commonly used as on-policy learning requires too many expensive simulation steps, however, as our environment is not as complex to simulate (in the order of tens of milliseconds; producing over 1M results/output values per step), we can apply on-policy learning. We further train only the sampling importance network using RL to overcome the hard/non-differentiable decision on how many samples to collect and use standard backpropagation for all other components. For those reasons, our training was relatively stable and converged with insignificant variance to the results presented. Here are other training parameters we tuned: buffer size of size 100, kl_coeff of weight 5e-7, kl_target of weight 5e-8, lambda of weight 0.2, vf_loss_coeff of 0.5, and an entropy_coeff of 1e-5.   \n\nWe hope that we provided satisfactory answers to your questions and comments, and that any remaining concerns could be addressed.\n\nThe authors\n"}, {"review_id": "3RsRioItsZ", "reviewer": "Reviewer_YEyS", "summary": "Ray tracing faces the difficulties of being applied practical real-time applications due to high levels of noise when sample counts are low. Sample counts are often limited to as low as 4 when considering real-time applications (~30ms latency). As such, the paper proposes an end-to-end training of a RL-based sampling importance network, a latent space encoder network, and a denoiser network. As a result, the proposed framework achieves strong performance on several benchmarks when considering performance and latency trade off.", "questions": "Please refer to the weakness section.", "limitations": "The paper does not include the limitation section. One limitation would be slightly higher inference time compared to previous works. Another possible limitation would be larger memory consumption, which cannot be said for certain until the authors provide comparisons.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow\n- The proposed framework demonstrates strong performance in terms of performance and latency trade off.", "weaknesses": "- What is the exact novelty in comparison to previous works? The proposed framework seems to be combination of RL-based adaptive sampling [22], the use of sampling importance network [23], spatiotemporal reservoir (spatiotemporal latent space) [25,27], and the use of denoisers [10], except for minute details, such as not feeding the output of a denoiser to the sampling importance network?\n\n- Without detailed discussions on the major differences between the proposed method and previous works and/or combination of previous works, it is difficult to assess the novelty and contribution of the proposed method.\n\n- How does the memory consumption compare between the proposed method and previous works?\n\n- Is the inference time in Table referring to the latency of the whole framework, including the inference of networks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer YEyS, thank you for your time and interesting remarks.\n\n> What is the exact novelty in comparison to previous works? The proposed framework seems to be combination of RL-based adaptive sampling [22], the use of sampling importance network [23], spatiotemporal reservoir (spatiotemporal latent space) [25,27], and the use of denoisers [10], except for minute details, such as not feeding the output of a denoiser to the sampling importance network? Without detailed discussions on the major differences between the proposed method and previous works and/or combination of previous works, it is difficult to assess the novelty and contribution of the proposed method.\n\nWe have 2 major novelties compared to previous work (cf lines 108-144):\n\n1) Learned latent space representation: The only previous works using a latent space are ReSTIR and derivative works. Their latent space consists of storing a buffer of lights for every pixel. They update the buffer by sampling from new visible lights or reusing spatiotemporally nearby pixels, leveraging simple statistical heuristics (importance sampling). This latent space is directly used only for denoising. \nDifferently, our latent space consists of storing a learned encoding given all previously available information to guide future sampling and denoising. It is updated using a dedicated neural network given new information and the previous state.  \n\n2) RL-based learning of the sampling recommendation model:\nThe only related work using RL is [22], which is not suitable for real-time and does not deal with adaptive path tracing but with the incompatible task of incidence radiance field.\nPrevious work targeting sampling recommendation (DASR [20], NTAS [23]) used approximated gradients instead of RL. We avoided such an approximation and instead formulated an exact end-to-end optimization criterion using reinforcement learning. It also allowed us reaching unprecedented low sample counts and hence inference times as non-RL methods naturally cannot be scaled below 1spp as there is no longer a gradient when no samples are collected. \n\nTo identify the impact of each component, we perform an ablation study (Lines 277ff. and Table 1) showing that each contribution is essential to achieve the final performance. \n\n> How does the memory consumption compare between the proposed method and previous works?   \n\nThe memory usage from neural networks weights is minimal (<50MB). We store more input and state information than previous work: besides the sampled pixel values (24 channels; RGB values for up to 8 non-averaged samples) and 7 additional input channels (3 for surface normals, 3 for albedo, 1 for depth), we store 32 channels for the state (warped latent space)---a total of 53 channels more than previous work (using 7 channels for additional data and 3 channels for pixel values) or 212 Byte/pixel which amounts to 110 MB at 720x720 pixels. As working memory during inference, we can operate with a minimal buffer size as the high-resolution images can be tiled. With a total memory footprint in the order of 200 MB, the overhead is insignificant (<2%) compared to >12GB of textures loaded for current games on high-end GPUs where such capacity is available. \n\n> Is the inference time in Table referring to the latency of the whole framework, including the inference of networks?\n\nIn Table 1, we report the execution time of the method without the cost of collecting ray tracing samples, i.e., the neural networks and warping for our method or the best implementation we could find of related work such as ReSTIR with Monte-Carlo path tracing setting the baseline at 0 inference time. We report these as they are largely independent of the scene. In Figure 3, we compare the latencies of the entire systems including networks and collecting the pixel samples, evaluated on the test datasets.\n\n> The paper does not include the limitation section. One limitation would be slightly higher inference time compared to previous works. Another possible limitation would be larger memory consumption, which cannot be said for certain until the authors provide comparisons.\n\nThe main purpose of our paper is to deliver a better PSNR/inference-time trade-off. Our main results (Figure 3) show the relationship between frame-latency (which is the inference time of the whole framework including the pixel sampling) and the quality for our method (including variants) and relevant related work. Our method is Pareto optimal (always on the upper left), which means it has a lower inference time for equal visual quality compared to all other methods. Our method has a marginally larger memory consumption.  \nBased on the feedback of Reviewer WTeK, and if permitted, we will add a \"limitation\" section to the paper or at least to the appendix (see official comment). \n\n\nThank you for dedicating time to evaluate our paper and provide thoughtful feedback. We appreciate your acknowledgment of its strengths in soundness, presentation, and contribution. We understand that some reservations led to a borderline reject rating, and hope to have dispelled any misgivings through this rebuttal, demonstrating the merit of our work and addressing any lingering doubts. If there are still concerns, we'll promptly respond to further queries.\n\nThe authors"}]

Author rebuttal:
Based on the feedback of Reviewers WTeK and YEyS, and if permitted, we will add the following section to the paper or at least to the appendix. Please look at individual rebuttals for all other comments and answers.

```
Limitations: The current method does not consider the application of after-effects such as motion blur, which could allow the reduction of samples collected on fast-moving objects, while in the current setting we would implicitly focus the ray tracing samples on such an object to minimize the error. Further, this method is applicable to entertainment applications and can potentially generate artifacts not present in the real scene, which could be an obstacle in application scenarios such as VR/AR medical devices. Additionally, it requires a system capable of performing both path tracing and DNN inference. While current graphics cards provide this capability and we assess the frame rendering time considering both components, the underlying workloads are significantly different: DNN inference is generally a very structured and high arithmetic intensity workload whereas path tracing is branching-intensive and performs random look-ups into memory. As these are fundamentally different, we see dedicated ray tracing units on modern GPUs. In future systems, it is conceivable that dedicated devices are used for each step, which would enforce a fixed capability for each type of compute and limit a free trade-off between the two as we make use in this work. Additionally, applying a DNN adds a memory overhead, although it can remain minimal compared to other components such as textures that commonly fill most the GPU's memory. Specifically, the model requires 110 MB more memory to store the input and latent space data, <50 MB to store the model, and a few MB of working memory for intermediate feature maps that can be processed in tiles. 

Memory overhead: We store the following data in memory. 1) the model weights (<50MB), 2) the sampled pixel values (24 channels; RGB values for up to 8 non-averaged samples), 3) 7 additional input channels (3 for surface normals, 3 for albedo, 1 for depth), 4) 32 channels for the state (warped latent space), and 5) a small working memory for intermediate feature data during inference. The components 2-4 add up to 53 more channels than previous works (which use 3 channels for input channels and 7 for additional data); hence 212 Byte/pixel; 110MB in total for 720x720 pixel data; and the required working memory is minimal as the inference can be done on tiles. With a total memory footprint in the order of 200 MB, the overhead is insignificant (<2%) compared to >12GB of textures loaded for current games on high-end GPUs where such capacity is available. 
```
We hope that any remaining concerns could be addressed.

The authors

Meta-review: After considering the rebuttal and internal discussions, all reviewers are in favor of accepting this submission. The paper is well-written, it identifies open issues in previous work, and presents simple solutions that are empirically shown to be effective (though experiments could be extended)
-> Accept. Congratulations!
Please take the reviewer comments into account when working on the camera-ready.
Decision: Accept (poster)