summarize the text: Hey there, I'm Thomas and I'll be presenting joint work with Fabrice Roussel, Jan Nowak and Alex Keller. This talk is going to be about a fully fused neural network that we can use for radiance caching in real-time rendering. More specifically, our ultimate goal is photorealistic rendering in real-time. And one crucial aspect of photorealism that we are going to focus on is global illumination, which means that we let light bounce around as many times as would be physically plausible in order to get realistic inter-reflections and color bleeding. In order to get accurate global illumination, you'd use an algorithm like path tracing, where the indirect reflections are simulated by tracing long paths of light until they hit a light source. But that's expensive, and on top of that, tracing long paths also leads to noisy images. So instead of tracing long, noisy paths, we want to make them short and terminate them by asking a radiance cache how much light should be injected into the ends of these short paths. Now this is not a new idea at all. Video games and movie rendering systems have been doing this for ages. I think the most well-known approach is that of irradiance probes, where you create a grid that spans your scene, and you precompute the diffuse irradiance arriving at each of the grid cells. But having to do such a pre-computation is for one annoying, and there are also other limitations like only supporting diffuse materials and getting issues like light leakage when the grid doesn't align well with geometry. Now there has been active research into improving irradiance probes, but without wanting to discount that, in this talk here I want to go for a much more radical paradigm shift, namely what about a neural network? Well, for one, such a network would be agnostic to the scene's lighting, materials and geometry, because it's a black box that just takes a position and direction as input and then spits out the radiance, leaving that position. But on the other hand, neural networks have this reputation of not only needing very elaborate pre-training but also being expensive to use. And perhaps the most important thing to take away from this talk is that that's not necessarily the case. More specifically, we're going to train our neural network while rendering, and on top of that, we'll see that it's surprisingly cheap to use. We start by showing you our end result, actually. Here's a view of a typical living room that's illuminated by environment lighting through some open windows. Now, one thing that's important to realize is that not all of this noise that you see is caused by long light paths. Some of it is just because shadow rays are not traced optimally, and we do actually have a remedy for that. The ReSTIR algorithm from Bitterly and colleagues. But as you can see, even with ReSTIR enabled, there is still lots of noise, and that's the noise that comes from long paths that the neural radiance cache attacks. With the neural radiance cache enabled, you get an additional complementary noise reduction on top of ReStore, and this noise reduction is now kind of good enough that denoising starts to become a real option, and I'll show you some results with that later on. But what about performance? Well, path tracing alone is, of course, the fastest, but the noise makes it pretty much useless. I think the better baseline would be to use ReStar, which gets 121 frames per second. Now, compared to that, adding the neural radiance cache doesn't actually slow things down much. We still get 111 frames per second, and I'm showing you this number mostly to reinforce the point that neural networks, even if you evaluate them for every path that you trace, they don't have to be slow. With that, let's dive into the rendering algorithm, and I'll be focusing more on the neural network side later on in the talk. Suppose we want to render an image of this room here. We have a path tracer represented by the camera and a single neural network that acts as our radiance cache. We trace a bunch of paths and terminate them after very few bounces. For those paths that didn't already exit the scene, we take the hit information at the endpoints, stuff like 3D coordinate, view direction, surface normal, material properties, etc. And then we pass that information to the neural network that will predict how much light to inject into the paths. And that's how you render with the Neural Radiance Cache. Pretty simple. But now comes the question of how you actually train the cache. Well, we already traced all these paths for rendering, one for each pixel, and what you're looking at is the image that might come out of this. Since we don't want to waste any computation, we will reuse a sparse subset of these paths for training. And when I say sparse, I mean really sparse. We subsample the image with a stride that's chosen automatically to match a fixed performance budget for training. And in this scene, the automatic stride selects 2-3% of the paths. The reason we go so sparse is that against all intuition about neural network training, in our case, training the network is actually really data efficient. And one way to look at it is if you have 60 frames per second, even a small number of paths per frame is going to create a gigantic training dataset over time. And of course, another benefit of going so sparse is that compared to actually querying the network for rendering, training on just these 2-3% is really, really cheap. Okay, so back to the illustration from before. Suppose that the bottom path falls onto the sparse grid that we've just seen and therefore is being selected for training. We will extend that bottom path by just a few bounces and then query the neural radiance cache again at the new end of that path. The reason we terminate into the neural radiance cache, even in training, is that just like with rendering, it saves computation and it also reduces noise. And in this case, it reduces noise in the training data. To generate that training data, we simply propagate the predicted radiance backwards along the training path, such that at each vertex of that path we get an estimate of reflected light, and that's what we're going to train on by simply feeding it to a standard neural network optimizer. And then we can start rendering the next frame and rinse and repeat. There's just one more important thing about training I want to mention. Since we're doing online training on a sparse set of paths, there's going to be lots of noise in our training data. And even more challenging, we want to be able to adapt to animated content, so we have to use a very large learning rate. The result is that our trained network weights oscillate like crazy around the local optima, and if we just used these weights as is, we would get severe flickering in our rendered image. But there's a surprisingly simple fix that doesn't compromise the speed of training. We create a second set of weights that's a low-pass filtered version of the trained weights, and we use that low-pass filtered set of weights when we are rendering. And since the low-pass filter doesn't feed back into the training loop, it doesn't mess with the fast training, but it does keep the radiance cache flicker-free when it's used for rendering, as you can see in the right video. Okay, so now that you know how we train and use the Neural Radiance Cache, I want to show you some results of what it actually learns. What I'm going to show you are images where we just trace primary rays from the camera and then immediately query the cache. So the colors you're going to see are completely synthesized by the neural network. No paths are actually traced. I'll start with an indoor scene that has lots of geometric complexity and that's indirectly lit through a window. I want to call special attention to two things. I'll start with an indoor scene that has lots of geometric complexity and that's indirectly lit through a window. I want to call special attention to two things. First, the cache is good at avoiding light leaks. The arrow points to thin geometry where both the illumination outside and inside has the correct brightness. That's something that's difficult to achieve with traditional radiance and irradiance caches. And second, the cache can learn sharp directionally resolved effects like this reflection of the window in the mirror. And this is possible because the dual radiance cache learns the directional radiance rather than irradiance, not making any assumptions about the underlying material model. Let me now switch to a reference image of the scene so that we can compare how close the cache is to the path-traced ground truth. Now, this is the reference image, and it does have much more detail than the cache, there's no question about that, but it's not too far off. And the important thing is that the overall colors match really well, which will be important when we use this cache for indirect illumination later on. Let me just flip back and forth one or two more times to let you compare these images. Here's a visualization of the radiance cache. Here's the reference image. Here's again a visualization of the radiance cache. And here's the reference image again. Now let me show you another scene. This time an outdoor environment to show that the radiance cache can learn detail at larger scales. I want to call special attention to the sharp shadows that the cache can learn. You can look pretty much anywhere in this scene, and you will see that the shadows look accurate and detailed. And just like before, let me flip back and forth with the path traced reference image to let you gauge the quality for yourself. Here's the reference image, and here's a visualization of the cache. And here's the reference image again, visualization of the cache again, and one last time, the reference image. Now, of the cache again, and one last time the reference image. Now, lastly, let's look at a scene which consists of complicated specular and glossy interreflections. On the floor and ceiling, you see reflections of the orange and blue emissive surfaces. This is again something that an irradiance cache wouldn't give you. But admittedly, the new irradiance cache also has room for improvement here. Switching to the reference, you can see there's much more detail than was captured in the cache. But to be fair to the cache, all this detail is difficult to learn on a millisecond's budget because none of the detail is contained in textures or other pre-existing surface information that the cache could have used. So it's kind of a worst-case situation. Okay, I think with these last few images, you now have a pretty good idea of the quality of the cache and now let's look at how quickly it actually trains and how well it adapts to animated content. Here you can see the same scene again, but now everything is moving around. Have a look at how the reflections on the ground keep up with the camera movement and only disappear when the light sources spin rapidly. And even when the light sources spin rapidly, the cache doesn't completely break, but simply misses the reflections on the floor. They kind of get blurred out. Here's another very difficult situation. Only the adjacent room is illuminated and light enters the room with the camera through a constantly opening and closing door. This is a really extreme change in lighting, since you're going from complete darkness to light almost instantaneously. If you look at the floor where the door passes over, you can get a good idea of the temporal delay of training the cache. That delay is just about 100 milliseconds, which for indirect illumination is actually almost unnoticeable according to recent studies by MacGyver and colleagues. So let's summarize the characteristics of this Neural Radiance Cache. It learns relatively quickly, which we are happy with. It captures sharp details in the illumination, whether these details are shadows or glossy reflections. And the cache matches the overall color of the reference image quite well. However, the main downside are these striped artifacts and blurry shapes when there is no texture information. So in other words, at a small scale, the cache doesn't always look visually pleasing. But let me remind you that we don't actually render by visualizing the cache in this way, this was just to let you gauge how well the cache learns and adapts to animated content, and in reality we query the cache after a few bounces along a light path, in which case the artifacts of it get mostly blurred away. In order to guarantee that the path and directions indeed blur the artifacts of the cache, we need to only query the cache when the footprint of the path is sufficiently spread out. For one, the footprint is determined by the material that the path interacts with. If the material is specular, there isn't any blurring. Think of looking through a mirror or through glass. If the material is glossy, there's a little bit of blurring. And if the material is diffuse, that's pretty much maximal blurring. And the footprint doesn't just depend on the material, but also the distance that the path travels since the last interaction. We use this in our renderer and we terminate paths when that radius becomes bigger than the user-chosen threshold. Now, usually, user-chosen thresholds are terrible and you want to avoid them like the pest, but here, I think this parameter is actually really useful, because it lets the user dial in their personal preference of whether they'd rather have more noise or more bias. Small thresholds result in shorter paths, which are less noisy and cheaper, but also show more of the cache's artifacts, and conversely, large thresholds result in less biased but more noisy paths. And that's our rendering algorithm. I think we can look at one more result before shifting gears and then talking about how we made the neural networks fast. On this particular scene, path tracing looks like this, and as expected, a restore improves the noise quite a bit. The cool thing is, in this scene, adding the neural radiance cache not only improves the noise further, but it also makes rendering much faster, so in this case, the savings from tracing shorter paths are actually much greater than the cost of the neural networks. The key to getting that kind of speed out of a neural network lies in our fully fused implementation. What we mean by fully fused is that the entire neural network is implemented in a single CUDA kernel. This has the benefit of being able to store the intermediate results of evaluating the network, so the hidden layers, in the fast on-chip memory, so shared memory, caches, registers, instead of slow GPU RAM. If you've ever heard of operator fusion, that's where the name fully fused comes from. And it pays off. In terms of inference, we are much faster than an equivalent implementation in TensorFlow. On this graph here, you can see how many elements the fully fused approach in TensorFlow can each process per second. The x-axis goes from very small batch sizes to very large batch sizes. As you can see, the speedup is largest for small batch sizes on the left of the graph. But in practice, at least for us, if we render an image, we have like 2 million pixels in full HD. So that would be on the large side of batches where we are like 8.6 times faster. For training, the difference between TensorFlow and the fully fused approach is much less pronounced because it turns out that not all components of training easily fit this fully fused paradigm. But in training, unlike inference, we are actually interested in the small batch sizes. And that's because we'd rather have many small training iterations than a single big one. Because remember, we want to learn quickly to adapt to animated content. So in practice, our training speedup over TensorFlow is on the left hand side of that graph, which is similar to inference around nine times faster. So where exactly does the speedup come from? First of all, we have an extremely simple architecture which makes handcrafted optimization easier. We have a standard multilayer perception or MLP, which consists of a bunch of fully connected layers that each have a small amount of neurons. And then there are pointwise activations, in our case, relu's and no biases to keep things simple. Evaluating such a network on a bunch of inputs is just a sequence of alternated matrix multiplications and activation functions. And the important thing about such networks is that the compute cost scales by the square of the number of neurons and the memory traffic scales linearly. So for big neural networks that have lots of neurons, you don't really need to focus on memory because their cost will be overshadowed by the quadratic compute requirements. However, in our special case, what we want is a really, really fast neural network. So we use few neurons, so the constant factors start to matter. And you probably all know, GPUs and modern processors in general are far better at compute than memory traffic. So the constant factors tell us that the memory cost dominates here. That's where the fully fused approach shines. So how does it work? Well, as mentioned before, if you have few neurons, all the intermediate results of the computation as well as the weight matrices can fit into on-chip memory. In order to enable this, the evaluation of the network has to be broken down into thread blocks that operate completely independently from each other. And this is in contrast with the typical approach of parallelizing the individual operations of the network. Once this is in place, the only RAM accesses that are left are reading the network inputs, writing the outputs, and most importantly, the weight matrices which need to be loaded redundantly into each thread block. So what we want to do lastly to minimize this redundant loading is to make the thread blocks operate on as large chunks of the batch as fits into their on-chip memory to minimize the total number of them. Of course, I'm omitting many more low-level implementation details, like we could zoom into one of those per-thread-block matrix multiplications, and what you would see there is an inner loop that's tailored to getting the most out of hardware matrix multipliers, like NVIDIA's Tensor cores or so. Okay, but now let's break down the average cost of the Neural Radiance cacheache across all our test scenes. Our baseline is an optimized path tracer, where a restore is used for direct illumination. This takes around 11 milliseconds to render a full HD frame. Using our Neural Radiance Cache, the ray tracing cost is quite a bit lower because the paths are terminated early, but there's extra cost for querying and for online training of the neural network. But as you can see, that extra cost is relatively small. So overall, this means that using the Neural Radiance Cache doesn't actually increase the total cost of rendering. It's basically free noise reduction. Let's also quickly look at the memory footprint, because the network has so few neurons. The weight matrices are just 50 kilobytes in size, and this is almost laughably small for a light field representation, right? It's small enough that you could even think about streaming the weights over the internet. So for example training the neural network on the cloud. Next, it's worth pointing out that querying the network, so inference, needs no extra memory because remember all the intermediate activations of the neural network remain in on-chip memory because it's fully fused. So all the memory you need is the input buffer and the output buffer and that's comparable to using something like G buffers and deferred shading. And then that said, there's a non-trivial memory cost associated with training because in that case, we can't easily get around storing the intermediate activations in RAM. With all that said, let's look at some interesting future work. Firstly, volumetric scattering. Perhaps you already noticed, nothing about the Neural Radiance Cache is specific to surfaces or surface light transport. It could just as well learn scattered radiance throughout a volume. And we actually have some preliminary results that confirm this. Here's a comparison of path tracing with and without the neural radiance cache. And the noise reduction is quite good, but since we didn't make a detailed investigation, I don't want to jump to any conclusions yet. So take this with a grain of salt. since we didn't make a detailed investigation, I don't want to jump to any conclusions yet. So take this with a grain of salt. Next, something that we think is a promising avenue is to use ReSTIR to directly import and sample the Neural Radiance Cache. Basically, you can think of the Neural Radiance Cache as a global light source that injects indirect illumination into the scene. And under this view, the Neural Radiance Cache can be sampled using ReSTIR like any other emitter. So we hope it'll be basically free noise reduction. Then denoising. What we're interested in is a denoiser that can deal well with the remaining noise that's left over after we use the new irradiance cache. And we got really good preliminary results by using a recent real-time denoiser from Munkberg and Hasselgren, even though that denoiser was actually trained on completely different data, so we are hopeful it can be improved further. Here's a video that shows this denoising on a path tracer, then a path tracer with Ristor, and then with our Radiance cache. And you can really see that the flickering becomes best with the Radiance cache enabled. This quality is actually really close to usable, just needs a little bit of improvement. And the last point I wanna mention is the Stripe artifacts that we've seen before. These artifacts arise from the input encoding that we use. What's an input encoding? Well, so far I've talked about the radiance cache as if it looked like this, query that gets fed through a network, which in turn produces reflected radiance. But in truth, there's an intermediate step that previous work showed improves the quality of the neural network by a lot. And I mean a lot, a lot. So we really want to use it. But the main downside of even the best encodings that we have are the striped artifacts. So we are searching for better encodings that give us similar quality benefits without introducing any unwanted artifacts. Actually, we already made some progress in that direction. I have to admit that some of the results that I've shown you in this presentation already had these improvements incorporated. So some of this presentation actually looked better than the results we have in the paper and we're planning to release more details on these soon. Now let's wrap up and summarize again what this has been all about. We've seen that neural networks are perhaps surprisingly efficient and accurate gradients cache. With this cache we can terminate paths early and reap a pretty significant reduction in noise, sometimes even in cost. And here's a rendered image that shows this, where both the noise and the frame rate is improved. This is possible because our networks are around nine times faster than an equivalent implementation in TensorFlow. The crucial thing that enables this efficiency is the fully fused implementation that I think has many other applications than just radiance caching or even graphics in general. The cool thing about the neural network is that it's an oblivious data structure that doesn't make any assumptions about the underlying scene, geometry, materials, lighting, anything. It follows that the neural radiance cache is intrinsically adaptive to detail, like there's no need for manual adaptation, which also means that the neural radiance cache tends to just work, at least on all the test scenes that we tried so far. And lastly, the cache is trained online while rendering using a pretty cheap procedure which makes it adapt to animated content relatively well. And with that summary, I thank you all for watching this recording. The source code for the neural networks can be found here, and feel free to send me an email or drop by a Q&A session soon. Bye bye and thank you!