Computational Visual Media
https://doi.org/10.1007/s41095-0xx-xxxx-x
Research/Review Article

Neural Scene Baking for Permutation Invariant Transparency Rendering with
Real-time Global Illumination
Ziyang Zhang(B)1 and Edgar Simo-Serra1© The Author(s)Abstract Neural rendering provides a fundamentally new
way to render photorealistic images. Similar to traditional
light-baking methods, neural rendering utilizes neural networks to bake representations of scenes, materials, and lights
into latent vectors learned from path-tracing ground truths.
However, existing neural rendering algorithms typically use
G-buffers to provide position, normal, and texture informationof scenes, which are prone to occlusion by transparent surfaces, leading to distortions and loss of detail in the rendered
images. To address this limitation, we propose a novel neuralrendering pipeline that accurately renders the scene behindtransparent surfaces with global illumination and variable
scenes. Our method separates the G-buffers of opaque andtransparent objects, retaining G-buffer information behindtransparent objects. Additionally, to render the transparent
objects with permutation invariance, we designed a new
permutation-invariant neural blending function. We integrate
our algorithm into an efficient custom renderer to achieve
real-time performance. Our results show that our method
is capable of rendering photorealistic images with variable
scenes and viewpoints, accurately capturing complex transparent structures along with global illumination. Our renderer
can achieve real-time performance (256 × 256 at 63 FPS
and 512 × 512 at 32 FPS) on scenes with multiple variabletransparent objects.
Keywords global illumination; neural rendering; real-timerendering; transparency rendering

1

Introduction

Global illumination rendering has been a fundamental challenge in computer graphics due to the complexity of thephysical phenomena involved. Traditionally, path tracing
1 Faculty of Science and Engineering, Waseda University, Tokyo,
Japan. E-mail: Z. Zhang(B), ziyangz5@toki.waseda.jp; E. SimoSerra, ess@waseda.jp

Fig. 1 Transparency quality improvement. We introduce ourcustom neural rendering framework to address transparency rendering issues in existing neural rendering approaches. Comparedto Active Exploration, our method is capable of rendering objects
behind transparent surfaces with minimum loss of details with global
illumination effects.

algorithms [1, 2] have been the main method to render photorealistic images with high-quality global illumination effects.
However, such methods are known to be expensive and timeconsuming; rendering one image may take minutes. Baked
global illumination techniques, on the other hand, provide
efficient solutions for interactive rendering by pre-computing
lighting information. Methods such as lightmap baking andlight probes [3] have been commonly used in game engines
to generate real-time global illumination, but they have limitations on dynamic scenes.
Recently, neural rendering approaches have emerged aspromising alternative for realistic image synthesis. Usingneural networks to represent scenes, materials, and lighting
can be seen as one type of pre-computation method. In the2

Z. Zhang, E. Simo-Serra

Fig. 2 Overview of our neural scene rendering framework. First, our rasterization-based renderer will first render the G-buffer, direct
lighting, and transparency buffers. Then, a neural network, which we denote as GlassNet, will use those buffers and rendering results as
inputs to synthesize high quality images with global illumination and accurate transparency. Details of GlassNet can be found in Sec. 3.4 andFig. 3.

early stage, neural rendering methods were only capable ofrendering fixed scenes under limited types of lighting withcomplex spatial inputs [4]. More recently, by using G-buffer
as inputs, newer methods avoid using complex spatial datasuch as point clouds, and are able to render scenes with variable geometries, materials, and lighting [5–9]. However, theusages of G-buffer re-introduces the classic problem of transparency rendering. When generating G-buffers, the shading
information will be overwritten by transparent surfaces in
the front, as shown in Fig. 4. Without such crucial shading
information, rendering models can only hard code all theinformation into the neural network based on ground truths,
which is unrealistic and ineffective, leading to poor quality.
To address this limitation, we propose a novel neuralrendering framework that separately renders opaque objects
and semi-transparent surfaces into two groups of G-buffers.
By doing this, all necessary shading information of both
opaque and transparent surfaces are preserved and can beused for further rendering. Furthermore, we design a neuralblending function to encode the transparency buffer into alatent representation with permutation invariance to achieve
Order-Independent-Transparency (OIT). Finally, we use adeep CNN (Convolutional Neural Network) to encode scenes
and blend them with transparency representations, rendering
the resulting image with global illumination.
Our results show that we significantly improve the transparency quality, exceeding the most recent neural approaches,
Active Exploration [8] and Neural Global Illumination [9].
Our method can accurately render the geometry and texture
details under transparent surfaces, whereas methods withouttransparency buffers converge to low-quality results.
In summary, our main contributions are:

(1) A novel neural rendering framework preserves both
opaque and transparent information simultaneously on
buffers, providing accurate shading information on rendering transparency.
(2) A neural blending function that encodes transparent
surfaces with permutation invariant, achieving OIT.
(3) A real-time renderer capable of rendering images withhigh performance enabled by the above algorithms.

2

Related Work

2.1 Traditional Transparency Rendering
In the traditional forward rendering pipeline, various techniques have been used to handle transparency. One widelyused algorithm is alpha blending [10], which utilizes a separate alpha channel to control the color composition of semitransparent objects with opaque objects on each fragment.
However, alpha blending is order-dependent and requires
sorting of objects by depth, which can be impractical. The
high computational complexity is not the only problem of
those sorting-based algorithms. When transparent objects
intersect with each other, it is impossible to sort the objects
globally unless the sorting is done by fragments, which again
greatly increases the computational complexity. Another algorithm called the A-Buffer algorithm [11] stores a list of
relevant fragments in each pixel, but sorting of all fragments
is required with unbounded memory usage. The Z 3 algorithm [12] is a similar algorithm with bounded memory usage
but comes with a trade-off in image quality. In contrast, ourproposed method does not require any sorting, and does not
generate a large amount of frame buffers during rendering
procedure.
To avoid sorting, several techniques have been introduced.

Neural Scene Baking for Permutation Invariant Transparency Rendering with Real-time Global Illumination

3

...

...

Fig. 3 Architecture of GlassNet. Our proposed method, GlassNet, contains four building blocks including the scene encoder, F; thepermutation invariant transparency buffer blending function, T ; the final blending network, B; and the rendering network, R. All parts aretrained jointly.

Depth peeling [13] renders objects in multiple passes, peeling transparent objects based on the z-buffer. However, thenumber of passes required by depth peeling increases thecomputational cost.
These algorithms often encounter challenges in the modern deferred rendering pipeline. Deferred rendering [14] isa technique that separates the geometric stage and lighting
stage into two passes. In the geometry pass, all the geometry
information is rendered into G-buffers to be used by thelighting pass for shading. While deferred rendering significantly improves rendering performance with multiple lightsources, it introduces complexity to transparency rendering.
Rendering geometry information into the G-buffer causes theobjects behind transparent surfaces to be occluded, making
it impossible to retain all geometry information without aprohibitively large number of buffers. Several algorithms
solve this problem with worsened image quality. [15, 16].
By applying a permutation-invariant blending algorithm, ourmethod can combine generated transparency information
during inference, while only requiring two frame buffers
regardless of the number of transparent objects, significantly
lowering the memory usage.
Several neural rendering approaches utilize the G-buffers as
input for their neural networks [6, 8, 9, 17], resulting in thesemethods also encountering transparency occlusion issues
with the G-buffers, causing bad image quality. In contrast,

our method is not affected by such issues and supports OIT
rendering.

2.2 Baked Global Illumination
Different from expensive path tracing methods, which are
the main approach to render photorealistic images in themodern film industry [18], baked or precomputed global
illumination methods are more often used in interactive orreal-time settings [19].
Lightmap baking is a popular technique used widely inmodern game engines [20]. It precomputes the diffuse global
illumination and uses interpolation to generate global illumination during rendering. Several methods have been proposedto generate the lightmap offline, such as by radiosity [21] or
path-tracing. Naively, baked lightmap only supports static
scenes with diffuse objects. Light probes technique was introduced to enable global illumination for dynamic objects [3].
McGuire et al. [22] extended light probes to support glossy
surfaces. However, such methods are limited to either static
lighting or a limited amount of light transportation types.
Several neural approaches use neural networks to encode
the scene representation into neural vectors and synthesize
global illumination such as Compositional Neural Scene
Representations (CNSR) [6], Active Exploration and Neural
Global Illumination. Similar to such neural approaches, ourmethod can render global illumination with dynamic scenes

4

Z. Zhang, E. Simo-Serra

with various light transportation types, with support fortransparent objects.

3

2.3 Neural Rendering

Given a scene, the goal of our approach is to bake a neuralscene representation into a neural network that can correctly
approximate global illumination with variable parameters ofmaterial, geometry, and lighting without being affected bytransparent surfaces. For any shading position, x, on surfaces,
the outgoing radiance, Lo , can be calculated by the Rendering
Equation[1]:Z
Lo (x, ωo ) =Li (x, ωi )fx (x, ωo , ωi )|nx · ωi | dωi(1)Ω+ Le (x, ωo )Using neural networks for rendering is a rapidly developing
research topic that provides fundamentally new approaches tophotorealistic rendering. We will introduce the closest related
paper to this work. We refer to [23] for a more detailed neuralrendering survey.
Initially, Ren et al. [4] used neural networks to learn indirect
lights from dynamic lighting and roughness with fixed geometry, complex spatial data, and only point lights. Later, Eslami
et al. [5] proposed a method that trains an encoder-decoder
neural network to represent scenes as latent vectors. CNSR
by Granskog et al. further improved the interpretability of theneural scene representation by partitioning the scene representation vectors into sections of lighting, materials, geometry,etc. Similar to our method, most recent solutions [6–9] use
G-buffers as the input to the neural networks to produce
indirect lighting and global illumination. Using G-buffers
instead of point clouds or voxel data simplifies the input data.
Such algorithms can usually be seen as precomputed lighttransportation and a form of scene baking. Using neural networks to precompute wavelet function has also shown strong
improvement on glossy global illumination [24]. Attempts onhigh performance real-time neural rendering are also madeby Xin et al. [17], but only limited on single-bounce diffuse
indirect illumination
However, using G-buffers as inputs brings back the classic
issue caused by transparent objects. The G-buffers cannot
provide information on objects behind transparent surfaces,
ending with poor rendering quality by such methods. In ourframework, we use independent G-buffers for the transparent
objects and a permutation invariant neural network to achieve
OIT. Note that the recent studies in neural rendering are mainly
orthogonal to our proposed method with different goals. Our
method can be directly combined with those methods toimprove the rendering quality further.

2.4 Permutation Invariant Machine Learning
PointNet [25] uses a multi-layer perceptron network to transform the input into latent vectors and uses a symmetricfunction to compose such vectors, which is proved to be asimple but strong method. PointNet inspires how our rendering neural network can process each transparency input tocancel the permutation order dependency. We refer a moredetailed review of permutation-invariant neural network to
Section 6 of [26].

Proposed Approach

3.1 Overview

where fx and nx represent the BSDF and normal; ωo indicatesoutgoing or viewing direction, ωi indicates the incident direction; Li and Le denote the incident radiance and emission
radiance. Ω denotes a hemisphere.
The Rendering Equation can also be written into the following form such as the direct lighting and the indirect lighting
are separated:
Lo (x, ωo ) = Le (x, ωo ) + Ld (x, ωo ) + Lg (x, ωi )(2)

where Ld is the direct lighting, and Lg is the indirect lighting
which contributes mainly to the global illumination. Current
methods use high-dimension complex non-linear functions,
i.e., neural networks, to predict Lo (x, ωo ) given all the necessary geometry, lighting, and material parameters as inputs.
In contrast, our approach consists of a neural network, which
we denote as GlassNet, to predict the indirect lighting, Lg , asshown in Fig. 2. We use Linearly Transformed Cosines (LTCs)[27] as a sufficient approximation to the direct illumination
term Ld as an input to our neural network to achieve efficiency.
Notice that as an input to the neural network, Ld is not necessarily to be analytically accurate because the neural network
will learn the complexity of the lighting transportation fromthe path tracing ground truths.
Most recent methods provide necessary shading information such as ωo , position, texture, and material parameters
through G-buffers. However, the naive approach using Gbuffers as input brings a significant problem: as shown in
Fig. 4, important shading information is hidden by transparent
surfaces. We propose a new neural rendering framework that
separately draws opaque G-buffers and transparency buffers,
preserving all the information, including the transparent surfaces and the objects behind those. Then, we introduce asymmetric neural function to cancel the permutation dependency in the transparency buffers and generate a neuraltransparency representations as shown in Sec. 3.3.

Neural Scene Baking for Permutation Invariant Transparency Rendering with Real-time Global Illumination

5transparent objects can be described as[10, 30, 31]:t X
T (C1 , α1 ), (C2 , α2 ), . . . , (Ct , αt ) =αi Ci zi
i=1

Fig. 4 Our G-buffer scene representation approach. Insteadof naively rendering the entire scene to a single set of G-buffers,
we use separate buffers for the transparent objects, allowing us torepresent complex transparency visibility in scenes accurately.

3.2 Transparency Buffers
For a given scene, we rasterize each transparent surface intoa list of G-buffers containing all the information requiredby a traditional path tracer to render Lo , including position,
surface normal, albedo, ωo , and material parameters as shownin Fig. 5. We generate a set of buffers with a size of (w, h, c ∗(t + 1)), where w, h and c denote the width, height andchannel of images. t is the number of transparent objects inthe scene. Finally, our rasterization-based renderer creates
the opaque G-buffers, and use LTCs to compute the direct
lighting Ld . By doing so, we can preserve all the necessary
information on both normal surfaces and transparent surfaces.
However, further processing is necessary for the naivetransparency buffer to be a good representation as a neural network input. Generally, neural networks are known
to depend on the order of inputs[25]. Inputting the naivetransparency buffer will cause our rendering network to beorder-dependent on transparency. With modern renderingengine optimization features such as batch rendering[28] andocclusion culling[29], the order of rendered objects cannot be
guaranteed. The surfaces can even be culled during rendering.
It is also infeasible to sort the rendered buffers after rendering.
In order to solve this problem, we designed a symmetric neural network invariant to permutation to blend the transparency
buffer.

3.3 Permutation Invariant Neural Representation
We design a blending invariant algorithm to cancel any sensitivity to the order of the inputs while significantly lowering
the memory usage during inference as shown in Sec. 4.4.1,
inspired by traditional transparency algorithms such as alphablending. In alpha blending, the function, T , of compositing(3)
where Ci and αi respectively are the color and the alpha valueof each transparent object. zi is the product of all alpha values
in front of the object:(1
if i = tzi = Qt−i
(4)
(1
−
α
)
otherwisejj=1
This function is clearly input-order-dependent, but it can berevised into a function with permutation invariant to inputs.
Based on the alpha blending function with the inspirationfrom the PointNet, we design our permutation invariant
transparency buffer blending function, T , to generate theneural representation, τ :T {b1 , b2 , . . . , bt }, σ≈ g h(b1 , σ), h(b2 , σ), . . . , h(bt , σ)=tXh(bi , σ) = τ

(5)

i=1

where b is the transparency buffer, g is a symmetric function,
σ is a scene representation, and h is a neural network shared by
all the transparency buffers generating neural representations.
We use addition as g to match the alpha blending function,
which clearly makes this function mathematically permutation
invariant to inputs. This allows the neural network to blend
all the transparency buffers into a single latent vector that is
independent of the input order.

3.4 Rendering Model
We introduce our rendering model, GlassNet as shown in
Fig. 2. GlassNet has four components: the scene encoder,
F; the permutation invariant transparency buffer blending
function, T ; the final blending network, B; and the rendering
network, R. Both F and B use U-Net [32] as the network
backend, whereas other networks use multi-layer CNN without down-sampling. All four components are trained jointly
as one neural network. Full details of the model are shown in
Fig. 3.
The scene encoding network, F, accepts tensors containing the position, normal, texture, material parameters, ωo ,depth buffer, and direct lighting. It then generates a neuralscene representation, σ. Similarly, each transparency buffer
is processed by T , generating the neural representation oftransparent surfaces ρ. All the inputs use positional encoding

6

Z. Zhang, E. Simo-Serra
NormalPosition

AlbedoRoughness

ωo

Depth

Ld
Ground Truth

G-Buffer
NormalPosition

AlbedoRoughness

ωo

Depth

Relative Depth

Transparency Buffer
Fig. 5G-buffer, transparency buffer, and ground truth. Our method can overcome the noise in the ground truth.

by Mildenhall et al. [33] to capture high-frequency information. Afterward, the blending network B blends the scene
representation σ and transparent object representations ρ andprocesses visibility. By doing so, B generates the final neuralbuffer, ϕ.
Finally, the rendering network R takes the neural buffer
ϕ with additional direct lighting as input to render the final
image. Although previous works, such as Neural Global Illumination, report using convectional neural network structureon the rendering network can lead to distortion of shapes, wefind that this problem can usually be solved by using kernels
with the size of 1 × 1 at the end of the network, and adding
larger kernel can help with global effects such as blurring.
For balancing the render quality and performance, the GlassNet is trained with lower resolution (256 × 256) textures,
G-buffers, and ground truths. A multi-layer CNN is used asa super-sampling network that super-samples the result at
×2 the original resolution. Given that we can cheaply gethigh-resolution texture and G-buffer from the rasterizer, thesuper-sampling network can be simple to achieve real-time
performance.

3.5 Training
We train the model in an end-to-end fashion to bake a specificscene by randomly sampling camera angles and positions
within certain ranges of each scene following uniform distributions to render ground truth images. In particular, foreach scene, we generate output image and G-buffer pairs fortraining. The ground truth is rendered by the modern pathtracer, Mitsuba 3 [34]. For the G-buffer and direct lighting
generation, we implemented our custom rasterization rendererusing OpenGL in C++, with CUDA support to support neuralnetwork inference. As a loss function, we choose is the com-

Fig. 6 Randomly selected images from datasets. The transparent
areas are highlighted by the blue masks in the second row.

bination of Structural Similarity Index Measure (SSIM) [35]and L1 loss.

4

Experiments

4.1 Experimental Settings
The input and output resolution used for training are both
256 × 256 px. The super-sampling network uses 512 × 512
textures to super-sample the 256 × 256 GlassNet renderedresults to 512 × 512 px, improving the visual quality. In
addition, all inputs are positional encoded using the method
proposed by Mildenhall et al.. All types of inputs and groundtruth images are shown in Figure 5. We used a relatively low
number of path tracing samples and did not pre-process thedata with a denoiser.
We used Adam optimizer [36] to perform gradient descentwith different hyperparameters on each scene. The dataset
details and hyperparameters such as train/validation/test splits,
learning rate and regularization for each scene can be foundin Appendix A. The training time varies from 5 to 10 hours
using 8 RTX 2080 Ti GPUs, depending on the scene.
As evaluation metrics, we use Mean Absolute Error (MAE),
LPIPS [37], DSSIM [38], and Peak Signal-to-Noise RatioNeural Scene Baking for Permutation Invariant Transparency Rendering with Real-time Global Illumination

Ours

Active Exploration

CORNELLBOXDESKBATHROOM

Ground Truth

7

Fig. 7 Qualitative comparison. We compare with Active Exploration in several challenging scenes. Note that our method can preserve
details of objects behind transparent surfaces with the synthesis of global illumination. All scenes are rendered on test sets with different
camera views from the training set. The transparent areas are indicated by yellow dashed lines in the column of ground truths.

8Z. Zhang, E. Simo-Serra

Scene
BATHROOMCORNELLBOXDESKCORNELLBOX8MAE ↓LIPIPS ↓Metrics
DSSIM ↓PSNR ↑

T.MAE ↓

T.PSNR ↑Active Exp.
Neural G.I.
Naive
Ours
Active Exp.
Neural G.I.
Naive
Ours
Active Exp.
Neural G.I.
Naive
Ours

0.01475
0.00934
0.021310.00808
0.032370.040930.04988
0.021420.017390.01733
0.03274
0.01658

0.02995
0.009610.04409
0.00686
0.18584
0.12171
0.209160.01595
0.05523
0.04327
0.078450.011500.048400.02280
0.056220.01887
0.20583
0.192130.232960.05436
0.061170.049850.07987
0.01782

30.85680
35.03145
29.17893
36.81351
25.21868
24.13943
22.90424
29.75453
27.87549
29.34936
26.16205
34.82217

0.061700.03888
0.082080.02738
0.10491
0.131410.154710.06087
0.15792
0.12360
0.19205
0.05487

23.72751
28.31231
22.41524
32.4614119.7812517.84513
17.80871
25.44725
16.31566
17.96721
15.75951
28.24681

Active Exp.
Neural G.I.
Naive
Ours

0.029110.043070.050930.020640.177190.19579
0.28167
0.04152

0.20583
0.27657
0.319390.0731126.16428
22.83434
21.84069
28.56370

0.089240.130610.151470.0572121.14504
17.90722
17.09835
24.25974

MethodTable 1 Quantitative comparison. We compare our approach with the naive baseline, Active Exploration and Neural Global Illumination
on several scenes with diverse metrics. T.MAE and T.PSNR denote the MAE loss and PSNR on the areas of the image with transparent
objects. Best result is denoted in bold.

(PSNR). For transparency quality evaluation, we use PSNR
and MAE computed on pixels rendered with transparency(T.MAE and T.PSNR).

4.2 Comparison with the State-of-the-Art
We compared with Active Exploration and Neural Global
Illumination, as they are the closest recent work. Comparisonswere carried out in 4 challenging scenes: BATHROOM,CORNELLBOX, DESK, and CORNELLBOX8 as shownin Fig. 6. The BATHROOM was modified from the Bitterli
dataset [39], while DESK and CORNELLBOX were creating
using existing models [40], to focus on transparency effects.
In CORNELLBOX8, We added 4 more transparent objects
to CORNELLBOX to explore the performance under thecondition of high number of transparent objects. Qualitative
results are shown in Fig. 1 and Fig. 7, while quantitative
results are shown in Tab. 1. A comparison with real-time pathtracing method is also demonstrated in Fig. 8
From the qualitative comparison in Fig. 7 we can seethat although Active Exploration performed well in many
areas, without transparency buffers, the neural network could
not synthesize the area behind the semi-transparent surfaces
without losing fundamental details. In the DESK and CORNELLBOX scenes, the text texture was almost impossible
to render without provided texture information. Instead, ourmethod could synthesize all the texture accurately. Furthermore, in the BATHROOM scene, all the detailed shapes
behind the glass such as taps were missing in the Active
Exploration, where our method preserved them well. The

Metrics
DSSIM ↓ T.MAE ↓ T.PSNR ↑CNSR 0.1502
0.097320.4854
Ours 0.0451
0.045726.2100

Scene Method
CORNELLBOX
DESK

CNSR
Ours

0.05390.01740.11150.053318.5453
28.6498

Table 2 Quantitative comparison with CNSR. Best result ishighlighted in bold.

quantitative comparison in Tab. 1 corroborates the visual
improvement of our method, which achieves better values onmost metrics, with more significant improvements on parts of
the image that need transparency. In addition, the renderingresults and comparison video are available in Video S1 in theElectronic Supplementary Material.
Overall, our method preserves the details of texture andgeometry of shape well. The global illumination effects suchas mirror reflection are also well synthesized. Our approachshows strong performance on preserve all the high frequency
texture, which is also shown on the wood texture of the
BATHROOM scene. Such details are extremely hard to besynthesized behind nearly transparent surfaces without transparency buffers.
Our experiments show that the transparency buffer is essential for neural network to render objects behind transparent
surfaces accurately. Although the global illumination effects
are not the main focus of our method, we also demonstrate
good quality on indirect diffuse lighting, mirror reflection,and soft shadows on par with existing approaches.
We also compared our method to CNSR to further verify theNeural Scene Baking for Permutation Invariant Transparency Rendering with Real-time Global Illumination

w/o P.E.
L1 Loss
Ours(a) Ground truth(b) Ours(c) Denoised Optix

Fig. 8 Comparison to real-time ray tracing. The real-time pathtracing denoiser failed to reconstruct the high frequency texture andsmall objects behind the glass.

effectiveness of our approach to transparency. We generated
the dataset in the format CNSR requires and used the identical
dataset to train the CNSR model and our model. The results
are shown in Fig. 9 and Tab. 2. The transparency quality
rendered by our approach is significantly better. Note that thiscomparison experiment is only supplementary since the main
focus of the two methods is different.
In addition, we compared our approach to the real-time
path tracing method in Fig. 8. We used 32 samples per
pixel to generate the path tracing image with a standard
Optix denoiser [41]. The Optix denoiser failed to reconstruct
the wood texture and the tape behind the glass, while ouralgorithm rendered such objects accurately. Due to the glass,the denoiser can no longer rely on albedo and normal AOV togain extra information, causing the loss of details.
Size

Type

256 × 256

Framerate (FPS)
Frametime (ms)# of Trans. Obj.
12
4
8
100 77 63 45
10
13 16 221024 × 1024†CARN

Framerate (FPS)
Frametime (ms)Framerate (FPS)
Frametime (ms)

50
20
24
41

37
27
21
47

32
31
20
50

23
43
18
55

1024 × 1024†NSRR

Framerate (FPS)
Frametime (ms)

17
56

16
62

15
66

12
78

512 × 512

Table 3 Performance evaluation. We can interactively render all
test cases. † 1024 × 1024 achieves by an external supersampling network, CARN or NSRR with temporal stability, as a post-processing
procedure.

4.3 Ablation Study
Although various previous works such as Active Exploration and CNSR reported that the PixelGenerator has betterup-sampling performance than CNN, we found that PixelGenerator is not able to effectively synthesize the transparent
structure given the same amount of training time as CNN.
We implemented the same rendering network structure with

9

MAE↓

LIPIPS↓DSSIM↓

PSNR↑

0.023711
0.023923
0.021424

0.026580
0.033820
0.015953

0.059718
0.072169
0.054358

28.937529
25.218679
29.754528

Table 4 Impact of positional encoding and loss functions.
Our experiments show the combination of positional encoding and
L1SSIM loss function achieves the best performance of the network.
Best result is highlighted in bold.

BATHROOMDESKCORNELLBOX

MAE

MAE (inv. order)

Difference

0.00805
0.01658
0.02125

0.00805
0.01658
0.021250.00000
0.00000
0.00000

Table 5 MAE on different transparency buffer order. We invert
the order of transparency buffer, and achieve identical renderingresults.

PixelGenerator as the backbone network. Section 4.2 showsthat PixelGenerator failed to reconstruct the glossy effects ontransparent objects given 400 training iterations.
To overcome the bad up-sampling performance of CNN,
an up-sampling network is solely trained to super-sample therendered images at two times the resolution. Our experimentshows fine-tuning the model with a smaller high-resolution
dataset cannot achieve good visual effects. Figure 13 showsthat the image generated by our up-sampling network has
better visual quality, especially on high-roughness transparent
objects, than retraining the CNN.
We studied the influence of positional encoding on our
method. Mildenhall et al.show that positional encoding canhelp neural networks learn high-frequency information; therefore, all the inputs to our neural network are positional encoded. We also demonstrate that using the combination of
MAE loss and DSSIM loss can achieve the best quality, withquantitative results are shown in Tab. 4. Furthermore, we
verified the effectiveness of our permutation-invariant algorithm. Table 5 shows that shuffled transparency buffer causes
on effect on our model performance.

4.4 Performance Evaluation
4.4.1 Real-time Performance
We tested our renderer on one RTX 4090, and the performanceis shown in Tab. 3. The computation time of our algorithm does
not significantly grow w.r.t. the complexity of meshes because
the transparent buffers only separately record each transparent
object instead of each primitive. The CORNELLBOX andDESK (30,344 and 38,400 triangles of transparent objects)
show that our method can efficiently render complex meshes.
Furthermore, the neural blending algorithm shown in
Eq. (5) also grows linearly w.r.t. the number of transparent objects given a specific resolution because h is n O(1)

10

Z. Zhang, E. Simo-Serra
Ours

CNSR

CORNELLBOX

Ground Truth

Fig. 9 Rendering results of ours and Granskog et al. Our method preserves the transparency quality better. The transparent areas areindicated by yellow dashed lines.

(a) PixelGenerator(b) CNN

Fig. 10 PixelGenerator v.s. CNN. Under the same training iterations, PixelGenerator completely failed to capture the glossy effects
on the glass bunny.

function under specific input size. With the resolution of
512 × 512, one extra transparent object costs 3.3 ms moretime on average. We believe this cost can be further optimized
by pruning the model.
Our approach can be further supersampled by other existing
fast supersampling networks. As examples, we demonstratethat by using CARN[42] and NSRR[43] as post-processing
models, our method can reach 1024 × 1024 resolution with
anti-aliasing effects. The results can be found in Tab. 3, Fig. 11,
and Video S1 in the Electronic Supplementary Material.

4.4.2 Memory Efficiency
The permutation invariant blending algorithm also has amemory efficiency advantage over existing approaches. With
traditional methods, the model has to receive all transparency
buffers in order to make inference, so that the memory
costs grow linearly w.r.t. the number of transparent objects.
However, with our proposed algorithm, there is no additionalframebuffer costs because the model can operate on a single
buffer of each transparent objects, and then add it to thesummed latent representation. The memory usage comparison

Fig. 11 Result in combining with other supersampling models.
We demonstrate that using external supersampling models such as
NSRR can further supersample our results to 1024 × 1024.

Active Exploration
Ours

RAM (GB)
2.50
0.68

VRAM (GB)11
4

Table 6 Runtime memory usage comparison. We compare ourmethod with Active Exploration on 512 × 512 with three transparent
objects. Least memory usage is highlighted in bold.

is plotted in Fig. 12 approximated by framebuffer usage. We
can see that our permutation invariant blending algorithm
has constant space growth w.r.t. the number of transparent
objects.
We also compare the actual runtime RAM and VRAM
usage of our method with that of Active Exploration. The
experiment uses the DESK scene with the resolution of
512 × 512 to test the memory performance. Table 6 shows our
implementation significantly reduces the required runtime
memory.

5Limitations and Discussion

Although our method shows significant improvements inneural rendering of scenes with transparent objects with full
OIT, our method relies on two fundamental assumptions
about scenes: the presence of transparent surfaces with alow refraction index and the absence of participating media.

Required memory (Megabytes)

Neural Scene Baking for Permutation Invariant Transparency Rendering with Real-time Global Illumination
36
34
32
30
28
26
24
22
20
18
16
14
12
10
8
6
4
2

11

Ours
Traditional(a) Ground truth1

2
4
Number of transparent objects

8Fig. 12 Buffer memory usage comparison. Under 256×256, one
transparent object buffer causes around 4.5 megabytes of memory.
Our blending algorithm is summation based and thus do not requirespace to store each transparency buffer.

(a) Ground truth(b) Supersampling(c) Fine-tuned

Fig. 13 Fine-tuning v.s. Supersampling. Fine-tuning network on
512 × 512 data has bad quality of roughness effects. The window
frame should be blurred out as in the ground truth.

However, when refraction occurs, the shading information
provided by the G-buffer is no longer accurate, which becomesan even more significant issue on spherical surfaces. This
limitation is illustrated in Fig. 14. Our rendering network
currently struggles to produce correct results when strong
refraction effects are present. Additionally, due to the use ofG-buffers, our framework is unable to represent participating
media, leading to the ineffective synthesis of volumetriceffects.
As a problem of neural rendering in general, current neuralrendering approaches perform relatively poorly on highresolution images. Although our method uses CARN or NSRR
as a post-processing supersampling procedure to boost theresolution further to 1024 × 1024 as described in Sec. 4.4.1,
further attempts to raise resolution will significantly affectthe performance, causing low framerates. Further research
may focus on the performance of neural rendering in highresolution settings.
Another potential direction for future research is incorporat-(b) Ours

Fig. 14 Failed case. When the refraction effects dominate, ourmethod can no longer accurately render surfaces under transparent
objects because G-buffer cannot provide useful information.

ing path prediction techniques, such as the method proposedby Li et al. [44], into scene representation. Furthermore,
exploring efficient approaches for representing participating
media on both the input and neural network sides presents an
intriguing research topic.
Several orthogonal methods can be combined with ourwork. For example, the Monte Carlo sampling techniqueproposed in Active Exploration can be utilized to improve theeffects such as perfect mirror reflection and caustic effects.
Neural Global Illumination uses radiance cues as inputs tothe model, which improves the scene representation learning.

Appendix
Table 7 records the key scene settings and hyperparameters
during the dataset generation, training, and evaluation.

Declaration of competing interest
The authors have no competing interests to declare that are
relevant to the content of this article. No funding was received
to assist with the preparation of this manuscript.

Electronic Supplementary Material
The rendering video recorded in real-time is available in theelectronic supplementary material of this article. The code ispublicly available on https://github.com/ziyangz5/NeuralBa
kingTransparency/.

References
[1]

[2]

Kajiya JT. The rendering equation. In Proceedings of the
13th annual conference on Computer graphics and interactive
techniques, 1986, 143–150. 1, 4
Lafortune EP, Willems YD. Rendering Participating Media
with Bidirectional Path Tracing. In X Pueyo, P Schröder,editors, Rendering Techniques ’96, 1996, 91–100. 1

12

Z. Zhang, E. Simo-Serra
Scene

Train

Validation

Test

S.P.P.

L.R.

W.D.

# of Trans. Obj.

Variable

BATHROOMCORNELLBOXDESKCORNELLBOX8

1800
1800
1800
900

200
200
200
100

128
128
128
128

8192
4096
4096
4096

0.00010.00010.00010.00010.000750.00010.00010.000114
3
8

1 (Roughness)
4 (Position and Lighting)3 (Color)
7 (Position and Lighting)

Table 7 Scene settings and hyperparameter choices. S.P.P. indicates the number of samples per pixel using during the ground truth
generation. L.R. refers to learning rate and W.D. refers to weight decay.

Greger G, Shirley P, Hubbard PM, Greenberg DP. The irradiance volume. IEEE Computer Graphics and Applications,
1998, 18(2): 32–43. 1, 3
[4] Ren P, Wang J, Gong M, Lin S, Tong X, Guo B. Global
illumination with radiance regression functions. ACM Trans.
Graph., 2013, 32(4): 130–1. 2, 4
[5] Eslami SA, Jimenez Rezende D, Besse F, Viola F, Morcos
AS, Garnelo M, Ruderman A, Rusu AA, Danihelka I, Gregor
K, et al.. Neural scene representation and rendering. Science,
2018, 360(6394): 1204–1210. 2, 4
[6] Granskog J, Rousselle F, Papas M, Novák J. Compositional
neural scene representations for shading inference. ACM Transactions on Graphics (TOG), 2020, 39(4): 135–1. 3, 4
[7] Rainer G, Bousseau A, Ritschel T, Drettakis G. Neural Precomputed Radiance Transfer. Computer Graphics Forum (Proceedings of the Eurographics conference), 2022, 41(2).
[8] Diolatzis S, Philip J, Drettakis G. Active exploration for neuralglobal illumination of variable scenes. ACM Transactions on
Graphics (TOG), 2022, 41(5): 1–18. 2, 3
[9] Gao D, Mu H, Xu K. Neural Global Illumination: Interactive
Indirect Illumination Prediction under Dynamic Area Lights.
IEEE Transactions on Visualization and Computer Graphics,
2022. 2, 3, 4[10] Porter T, Duff T. Compositing digital images. In Proceedingsof the 11th annual conference on Computer graphics andinteractive techniques, 1984, 253–259. 2, 5
[11] Carpenter L. The A-buffer, an antialiased hidden surface
method. In Proceedings of the 11th annual conference on
Computer graphics and interactive techniques, 1984, 103–108.
2[12] Jouppi NP, Chang CF. Z 3: an economical hardware techniquefor high-quality antialiasing and transparency. In Proceedings of the ACM SIGGRAPH/EUROGRAPHICS workshop on
Graphics hardware, 1999, 85–93. 2[13] Everitt C. Interactive order-independent transparency. White
paper, nVIDIA, 2001, 2(6): 7. 3
[14] Saito T, Takahashi T. Comprehensible rendering of 3-D shapes.
In Proceedings of the 17th annual conference on Computer
graphics and interactive techniques, 1990, 197–206. 3[15] Pangerl D. Deferred rendering transparency. ShaderX7: Advanced Rendering Techniques, ShaderX series, 2009: 217–224.
3[16] Mara M, McGuire M, Luebke D. Lighting deep g-buffers:
Single-pass, layered depth images with minimum separation[3]

applied to indirect illumination. NVIDIA Corporation, 2013. 3[17] Xin H, Zheng S, Xu K, Yan LQ. Lightweight bilateral convolutional neural networks for interactive single-bounce diffuse
indirect illumination. IEEE Transactions on Visualization andComputer Graphics, 2020, 28(4): 1824–1834. 3, 4[18] Keller A, Fascione L, Fajardo M, Georgiev I, Christensen
P, Hanika J, Eisenacher C, Nichols G. The Path Tracing
Revolution in the Movie Industry. In ACM SIGGRAPH 2015
Courses, 2015, 1–7, doi:10.1145/2776880.2792699. 3[19] Ritschel T, Dachsbacher C, Grosch T, Kautz J. The state of theart in interactive global illumination. In Computer graphics
forum, volume 31, 2012, 160–188. 3[20] O’Donnell Y. Precomputed Global Illumination in Frostbite,2018. 3[21] Cohen MF, Wallace JR, Hanrahan P. Radiosity and realistic
image synthesis, 1993. 3[22] McGuire M, Mara M, Nowrouzezahrai D, Luebke D. Realtime global illumination using precomputed light field probes.
In Proceedings of the 21st ACM SIGGRAPH symposium oninteractive 3D graphics and games, 2017, 1–11. 3[23] Tewari A, Fried O, Thies J, Sitzmann V, Lombardi S, Sunkavalli
K, Martin-Brualla R, Simon T, Saragih J, Nießner M, et al..
State of the art on neural rendering. In Computer Graphics
Forum, volume 39, 2020, 701–727. 4[24] Raghavan N, Xiao Y, Lin KE, Sun T, Bi S, Xu Z, Li TM, Ramamoorthi R. Neural Free-Viewpoint Relighting for Glossy Indirect Illumination. In Computer Graphics Forum, volume 42,2023, e14885. 4[25] Qi CR, Su H, Mo K, Guibas LJ. Pointnet: Deep learning on
point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and patternrecognition, 2017, 652–660. 4, 5[26] Bloem-Reddy B, Teh YW. Probabilistic symmetries and invariant neural networks. The Journal of Machine Learning
Research, 2020, 21(1): 3535–3595. 4[27] Heitz E, Dupuy J, Hill S, Neubelt D. Real-time polygonal-lightshading with linearly transformed cosines. ACM Transactions
on Graphics (TOG), 2016, 35(4): 1–8. 4[28] Wloka M. Batch, batch, batch: What does it really mean, 2003.
5[29] Pantazopoulos I, Tzafestas S. Occlusion culling algorithms:
A comprehensive survey. Journal of Intelligent and Robotic
Systems, 2002, 35: 123–156. 5[30] Meshkin H. Sort-independent alpha blending. GDC Talk, 2007,
2(4). 5Neural Scene Baking for Permutation Invariant Transparency Rendering with Real-time Global Illumination[31] Salvi M, Vaidyanathan K. Multi-layer alpha blending. In Proceedings of the 18th meeting of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, 2014, 151–158.
5[32] Isola P, Zhu JY, Zhou T, Efros AA. Image-to-image translation
with conditional adversarial networks. In Proceedings of theIEEE conference on computer vision and pattern recognition,
2017, 1125–1134. 5[33] Mildenhall B, Srinivasan PP, Tancik M, Barron JT, Ramamoorthi R, Ng R. Nerf: Representing scenes as neural radiance fields
for view synthesis. Communications of the ACM, 2021, 65(1):99–106. 6[34] Jakob W, Speierer S, Roussel N, Vicini D. Dr.Jit: A JustIn-Time Compiler for Differentiable Rendering. Transactions
on Graphics (Proceedings of SIGGRAPH), 2022, 41(4), doi:
10.1145/3528223.3530099. 6[35] Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality
assessment: from error visibility to structural similarity. IEEE
transactions on image processing, 2004, 13(4): 600–612. 6[36] Kingma DP, Ba J. Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980, 2014. 6[37] Zhang R, Isola P, Efros AA, Shechtman E, Wang O. The
unreasonable effectiveness of deep features as a perceptual
metric. In Proceedings of the IEEE conference on computer
vision and pattern recognition, 2018, 586–595. 6[38] Loza A, Mihaylova L, Canagarajah N, Bull D. Structuralsimilarity-based object tracking in video sequences. In 2006
9th International Conference on Information Fusion, 2006,
1–6. 6[39] Bitterli B. Rendering resources, 2016, https://benediktbitterli.me/resources/. 8[40] Zhou K, Wang X, Tong Y, Desbrun M, Guo B, Shum HY.
Texturemontage: Seamless Texturing of Arbitrary Surfaces
From Multiple Images. ACM Transactions on Graphics, 2005,
24(3): 1148–1155. 8[41] Chaitanya CRA, Kaplanyan AS, Schied C, Salvi M, Lefohn
A, Nowrouzezahrai D, Aila T. Interactive reconstruction ofMonte Carlo image sequences using a recurrent denoising
autoencoder. ACM Transactions on Graphics (TOG), 2017,
36(4): 1–12. 9[42] Ahn N, Kang B, Sohn KA. Fast, accurate, and lightweightsuper-resolution with cascading residual network. In Proceedings of the European conference on computer vision (ECCV),
2018, 252–268. 10[43] Xiao L, Nouri S, Chapman M, Fix A, Lanman D, Kaplanyan
A. Neural supersampling for real-time rendering. ACM Transactions on Graphics (TOG), 2020, 39(4): 142–1. 10[44] Li Z, Yeh YY, Chandraker M. Through the looking glass:
Neural 3d reconstruction of transparent shapes. In Proceedingsof the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, 2020, 1262–1271. 11

13

Author biography
Ziyang Zhang is currently a Ph.D. candidate in Faculty of Science and Engineering
at Waseda University. He received his bachelor of science degree in computer sciencefrom University of California, Irvine in 2020,and his master of science degree in computerscience from University of California, San
Diego in 2022. His research interests include
photorealistic rendering, real-time rendering and machine learningwith computer graphics.
Edgar Simo-Serra is currently an associate professor at Waseda University. He obtained his Industrial Engineering degree from
BarcelonaTech in 2011 and his Ph.D. in 2015
from the same university. From 2015 to 2018he was at Waseda University as a junior researcher (assistant professor), and during 2018he was a JST Presto Researcher, before rejoining Waseda. His general research interests are in the intersectionof computer vision, computer graphics, and machine learning withapplications to large-scale real world problems.