COMPUTER GRAPHICS forum

Pacific Graphics 2024
R. Chen, T. Ritschel, and E. Whiting(Guest Editors)

Volume 43 (2024), Number 7

CrystalNet: Texture-Aware Neural Refraction Baking for Global
Illumination
Z. Zhang1

and E. Simo-Serra1

1 Waseda University, Japan

Rendered Images

Ground Truth

Ours

Refracted Texture Reconstruction

GlassNet [ZSS24]

Object Indices

Refracted UV

Refracted Texture

Figure 1: Rendered images by our proposed appoarch. We introduce our novel neural rendering framework, CrystalNet, for high quality
refraction rendering. We use a neural network to bake UV coordinates and object information from refracted rays to reconstruct high frequency
color information on refractive surfaces. Compared to previous baking approaches such as GlassNet [ZSS24], our method captures refraction
effects more accurate with less loss of details.
Abstract
Neural rendering bakes global illumination and other computationally costly effects into the weights of a neural network, allowing
to efficiently synthesize photorealistic images without relying on path tracing. In neural rendering approaches, G-buffers obtained
from rasterization through direct rendering provide information regarding the scene such as position, normal, and textures tothe neural network, achieving accurate and stable rendering quality in real-time. However, due to the use of G-buffers, existing
methods struggle to accurately render transparency and refraction effects, as G-buffers do not capture any ray information frommultiple light ray bounces. This limitation results in blurriness, distortions, and loss of detail in rendered images that containtransparency and refraction, and is particularly notable in scenes with refracted objects that have high-frequency textures. In thiswork, we propose a neural network architecture to encode critical rendering information, including texture coordinates fromrefracted rays, and enable reconstruction of high-frequency textures in areas with refraction. Our approach is able to achieves
accurate refraction rendering in challenging scenes with a diversity of overlapping transparent objects. Experimental results
demonstrate that our method can interactively render high quality refraction effects with global illumination, unlike existingneural rendering approaches. Our code can be found in https://github.com/ziyangz5/CrystalNet
CCS Concepts• Computing methodologies → Rendering; Neural networks;© 2024 Eurographics - The European Association
for Computer Graphics and John Wiley & Sons Ltd.

2 of 9Z. Zhang & E. Simo-Serra / CrystalNet: Texture-Aware Neural Refraction Baking

1. Introduction

2.1. Traditional Refraction Rendering

Using neural networks to “bake” or precompute global illumination and render photorealistic images has provided a new way tosolve the key challenge of real-time global illumination in computer
graphics. Traditionally, global illumination is achieved by using
expensive path tracing [Kaj86], which is hard to achieve real-time
performance, or cheap precomputed methods [GSHG98] with limited light transportation type supported. Neural rendering instead
uses end-to-end deep models to learn and bake global illuminationand scene representation into the weights, supervised by path-tracing
images.

Specular refraction rendering has been a challenging problem incomputer graphics. Early attempts focus on refraction effects on arbitrary shapes [Wym05a] under distant lighting. Wyman [Wym05b]
extended this method to handle nearby geometries. Later, Oliveiraand Brauwers [OB07] improved it to handle more complex geometries. Génevaux et al. [GLD06] used pre-computed rays to approximate light paths, but such baking procedure is usually high-cost.
To avoid expensive precomputation and increase the capability torender deformable objects, the ray-object intersection in 2D texture space methods are proposed [ED06]. Despite the efficiency
and stability of traditional refraction rendering approaches, they
typically suffer from limited lighting types, constrained geometries’
complicity, unsupported overlapping transmissive objects, and theincapability of combining with other global illumination. In thispaper, we propose a neural rendering method that takes advantageof the flexibility of neural rendering methods to solve the problems
above and simultaneously handle refraction effects.

Recent neural rendering methods utilize G-buffers, which simplify input handling by avoiding spatial data [GRPN20, DPD22,
GMX22]. However, the usage of G-buffers has introduced a problem with transmissive object rendering. In such cases, transmissive
surfaces obscure crucial shading information in the G-buffers, complicating the rendering process. Zhang and Simo-Serra [ZSS24]
proposed a method that separates transmissive surfaces in G-buffer
to improve the transparency quality but assumes a low index of refraction. Currently, no existing method is capable of baking specularrefraction effects because the G-buffer of the transmissive surface
do not provide information on incoming rays, thus losing highfrequency information from refracted images. To address this limitation, we introduce a neural rendering framework, which we denote
CrystalNet, which consists of a refraction buffer (R-buffer) generator and rendering neural network. First, the R-buffer generator
predicts UV coordinates and object indices of the incoming rays ontransmissive surfaces from the G-buffers, which allows for texture
lookups, allowing for recovering high frequency texture information.
Afterwards, the neural renderer combines all the information to synthesize the scene. The entire framework is optimized for a specificscene by baking all the necessary information for rendering into
the weights of the neural networks, and can generate high quality
specular refraction effects such as caustic effects.
Our research demonstrates the practicality of using low-frequency
information baked by neural models to reconstruct high-frequency
textures. In comparison to directly representing texture in neuralnetworks, our proposed method, CrystalNet, significantly enhances
the quality of specular refraction rendering. This advancement has
the potential to enable real-time global illumination in computer
graphics without ray computations, making it more accessible andefficient for a wide range of applications.
Our main contributions in this paper include:
1. A novel neural rendering framework, CrystalNet, for renderingof scenes with high quality specular refraction effects with fullglobal illumination including soft shadows and caustics.
2. Refraction buffers (R-buffers) to bake information into data thatis usable by neural networks based on refracted UV coordinates
on transmissive surfaces.
3. In-depth evaluation and comparison with existing approaches.
2. Related WorkIn this section, we review the most related works in traditional
refraction rendering and neural rendering.

Refraction with roughness and BTDF is also a major research
topic in refraction rendering [ED06, DRBS∗ 11, GP16]. However, it
is out of the scope of this paper as neural rendering methods areoften highly capable of capturing low-frequency global illumination.
We include an analysis on rough surface refraction in Sec. 4.3.
2.2. Neural Rendering
As a group of data-driven rendering methods, neural rendering cantheoretically bake various light transportation types under area lighting and effectively and efficiently unify refraction rendering withother global illumination.
Ren et al. [RWG∗ 13] initiated the usage of neural networks tobake relightable global illumination with fixed objects and limited
types of lighting. Encoding scenes to latent vectors was introducedby Eslami et al. [EJRB∗ 18]. Later, Granskog et al. [GRPN20] expands the encoding into explainable compositional latent vectors.
Several later works empathize with the improvement of inputs tothe neural rendering models. Gao et al. [GMX22] explored the usageof radiance cues to improve the rendering quality. Active Exploration
proposed by Diolatzis et al. [DPD22] improved the sampling procedure of the training images. Zhang and Simo-Serra [ZSS24] recognized the limitation of using G-buffer as inputs to neural renderingmodels caused by the occlusion by transparent objects and proposed
GlassNet to improve the transparency rendering quality. However,
not all previous works have addressed the effects of refraction ontransmissive objects. Even the closet work, GlassNet, only considers
transmissive objects with refraction index close to 1. Instead, our proposed method, CrystalNet, is able to capture the refraction effects
even on smooth refractive objects with high-frequency refracted
information. To losslessly preserve all information of transparent
objects, our method utilizes the permutation-invariant network structure proposed in GlassNet.
Real-time path tracing and denoising, aided by neural networks,
are rapidly developing areas. Although these methods are generally orthogonal to neural rendering techniques, they still havesome limitations that neural rendering can address. Neural radiance caching [MRNK21] achieves real-time path tracing but has© 2024 Eurographics - The European Association
for Computer Graphics and John Wiley & Sons Ltd.

Z. Zhang & E. Simo-Serra / CrystalNet: Texture-Aware Neural Refraction Baking

3 of 9Figure 2: Overview of CrystalNet. Initially, our rasterization-based renderer generates the G-buffer, direct lighting, and transparency buffers.
Then, a refraction buffer generator will predict UV coordinates and object indices of refracted objects, which allow for bilinear texture lookups.
Finally, a neural network will combine all of the provided information and synthesize high quality images with global illumination and accuratetransparency. The full details of the network structure can be found in the supplementary materials.

limitations on complex light paths due to the low sampling number. Recurrent Denoising Autoencoder [CKS∗ 17] can effectively
remove noise even with extremely low sample per pixel. However, it
still generates specular refractive effects poorly for lack of arbitrary
output variables under such cases [ZSS24].
Recently, Neural Radiance Fields (NeRF) based methods have significantly contributed to the neural rendering field. We refer the development of NeRF to a comprehensive survey [GGH∗ 22]. Various
latest approaches [BMRF∗ 22,PSH∗ 22,LLW∗ 23,CLZ∗ 23,DCS∗ 24]
attempt to extend the ability of NeRF into refractive objects. However, such methods often restricted to static objects. Furthermore,
NeRF-based methods also require ray marching during rendering,which increases the computational burden. Compared to NeRFbased approaches, our method does not require ray marching, andsupports dynamic scenes.

3.1. Overview
The general objective of neural rendering is to use a neural network
to bake the outgoing radiance, Lo at position p. Lo can be computedby the rendering equation [Kaj86]:
Z

Lo (p, ωo ) =Li (p, ωi ) f (m, ωo , ωi )|n · ωi | dωi + Le (p, ωo ) (1)ΩTo compute Lo , instead of doing expensive path tracing, neuralrendering methods use a neural network F to represent the global
illumination in latent space:

Lo (p, ωo ) ≈ F(p, n, m, ωi , σ)(2)

To more accurately capture the refraction effects, we explicitly
separate the reflection scattering over the upper hemisphere Ω+ , andthe refraction under the lower hemisphere Ω− :3. Proposed Approach
We introduce our neural rendering framework, CrystalNet. Notations
used in this paper are summarized in Tab. 1. Our approach consistsof two major components: a refraction buffer (R-buffer) generator,
and a rendering model. The R-buffer generator predicts UV coordinates and object indices of refracted objects from direct lighting
information, which allow for texture lookups. Afterwards, all thecomputed information is used by the rendering model to synthesize
a high quality image of the scene. Our network main utilizes the
structure from UNet [RFB15] and GlassNet, and the implementation
can be found in the code and the Fig. 1 of supplementary documents.
© 2024 Eurographics - The European Association
for Computer Graphics and John Wiley & Sons Ltd.

Z

Lo (p, ωo ) =
ZΩ

+

Ω−

Li (p, ωi ) fr (mr , ωo , ωi )|n · ωi | dωi +(3)Li (p, ωi ) ft (mt , ωo , ωi )|n · ωi | dωi

In the case of baking reflection scattering, it is possible to leverageG-buffers from direct rendering, as neural networks do not need to
encode high-frequency information such as textures, but only needto encode global illumination.
In contrast, the refraction component, especially on smooth surfaces, often consists of distorted images of the objects behind the4 of 9Z. Zhang & E. Simo-Serra / CrystalNet: Texture-Aware Neural Refraction Baking

Notation

Description

Lo
Li
Le
ωo
ωi
n
m
Ω
Ω+ Ω−
ι
p
puv pιuv
PUV
FF
TT
σ
γ
c cp
I[x,y]
uv
wh
L
λtv
numt
ˆ·
·r
·t

Outgoing radiance
Incoming radiance
Emission radiance
Outgoing direction
Incoming direction
NormalMaterial parameters
Hemisphere
Upper and lower hemisphere
Object index
Position in 3D space
UV coordinates with ι indicating specific object
UV maps as 2D tensors with channels
Neural networks predicting radiance
Neural networks predicting refraction information
Neural scene representation in latent space
Refraction buffer
Color with subscripts indicating specific location
Color on texture or image at coordinate (x, y)
UV coordinates
Image width and height
Loss function
Weight of total variation loss
Total number of transmissive objects
All hat notations indicate symbols are predictions
Subscripts indicate symbols are for reflective objects
Subscripts indicate symbols are for transmissive objects

neural networks to directly bake the high-frequency texture andachieve high-quality refraction through texture look-ups instead.
Similar to recent neural rendering approaches, we also use aG-buffer as input to our approach and path tracing images as theground truth. To maximally retain the information occluded by
the transmissive objects, we apply the permutation invariant Gbuffer encoding method used in [ZSS24]. By doing so, the G-buffer
provided to the rendering model contains all the information behind
the refractive models.
3.2. Permutation Invariant Network Structure
We address the occlusion problem by transparent objects on Gbuffers by separating the G-buffers of transparent objects into individual buffers. As previous works proposed [QSMG17, ZSS24], apermutation invariant neural network, f , can be approximated bya deep function g combined with a symmetric operator, and thenpost-processed by an outer neural network k. As described below,
we choose addition as the symmetric operator:numt

f ({x1 , x2 , . . . , xnumt } , γ̂) = k( ∑ h(xi , σ, γ̂))

(5)

i=1

By applying this structure, our model can process the transparent objects without losing input information, and does not require
specific input order of the transparent objects.

Table 1: Glossary of notations used in this paper.

3.3. Refraction Baking: R-Buffer
refracting surface, with high frequency texture information. Suchrefraction can not be rendered simply using G-buffers, and it is necessary to either provide high-frequency information of the refracted
objects into the rendering neural network through expensive ray
tracing, or encode it into the network itself. Specifically, in addition
to all the rendering information regarding the transmissive object
itself, a neural network, Ft , requires information regarding the highfrequency information of the refracted objects γ. Accordingly, mostof the neural rendering model bake γ directly by using an implicit
deep model T :

Z
ZΩ

+

Ω−

Our R-buffer generator, T , takes a G-buffer as an input and generates
a tuple of normal and UV coordinates for a refracted objects, ι:T (p, n, m, ωi , σ, γ) = {n̂t , p̂ιuv , ι̂}(6)

We take advantage of the fact that modern rendering engines use
UV coordinates and bilinear interpolation to reconstruct 2D textureof color, normal, etc., to 3D objects. We used the predicted refraction
UV coordinates, p̂ιuv to reconstruct the color information, ĉ, fromtextures of objects by bilinear interpretation:

Li (p, ωi ) fr (ωo , ωi )|n · ωi | dωi ≈ Fr (p, nr , mr , ωi )Li (p, ωi ) ft (ωo , ωi )|n · ωi | dωi ≈ Ft (p, nt , mt , ωi , γ)(4)

≈ Ft (p, nt , mt , ωi , T (σ))
Note that T can only rely on the neural scene latent representation
to bake the high-frequency information of the refraction, leading toblurry and loss of details on high-frequency color or normal textures.
Instead of directly baking the high-frequency information, ourmethod, CrystalNet, bakes the lower frequency UV coordinates andthen reconstructs the textures directly. Specifically, our refraction
baking model, T , reconstructs γ by generating the R-buffer containing UV coordinates and normals. The overall CrystalNet structureis shown in Fig. 2. By applying such an approach, we avoid usingĉιpuv ≈I[x, y] · (1 − x′ ) · (1 − y′ )+I[x + 1, y] · x′ · (1 − y′ )+I[x, y + 1] · (1 − x′ ) · y′ +(7)I[x + 1, y + 1] · x′ · y′where x′ = ⌊w · u + 1⌋ − w · u and y′ = ⌊h · v + 1⌋ − h · v. Then, thepredicted R-buffer, γ̂, can be used in the rendering model.
Given that UV mapping is highly sensitive to the stability of thegenerated UV coordinates, we use the total variation loss [ROF92]
to enforce the smoothness of generated UV maps. Specifically, theloss function is the combination of L1 reconstruction loss and thetotal variation loss with a balance coefficient λtv :© 2024 Eurographics - The European Association
for Computer Graphics and John Wiley & Sons Ltd.

5 of 9Z. Zhang & E. Simo-Serra / CrystalNet: Texture-Aware Neural Refraction Baking

LT PUV , P̂UV = PUV − P̂UV

1+ λtv Ltv P̂UV

(8)

where the total variation loss, Ltv , is defined as:


UV
UV
UV
Ltv P̂UV = ∑ P̂UV
i+1, j − P̂i, j + P̂i, j+1 − P̂i, j

(9)

i, j

The R-buffer generator is implemented using a permutation invariant multi-layer convolutional neural network described in Sec. 3.2.
3.4. Rendering Model
We next introduce our neural rendering model.It contains three building blocks: a reflection baking network, Fr , a transmissive baking
network, Ft , and final rendering network R. For the network architectures, Fr and R uses a U-Net model [IZZE17], and Ft uses
the permutation invariant multi-layer convolutional neural network
described in [ZSS24]. We use G-buffers with separated transmissive objects as inputs to prevent the occlusion to the G-buffer bytransmissive objects and maximally retain inputting information asshown in Fig. 2.
The reflection baking network Fr accepts all information required by rendering equation. As with most other neural rendering
approaches, besides p, nr , mr , and ωi , we take advantage of a rasterization engine to also directly generate the depth buffer, zr , anddirect lighting, dr , as network inputs. We use Linearly Transformed
Cosines [HDHN16] to approximate the direct lighting. Fr also generates a neural scene representation latent vector, σ used in Ft .

Figure 3: Random selection of images from the dataset.
Parameter
Training set size
Validation set size
Test set size
Samples per pixel
Learning rate of the renderer
Learning rate of the R-buffer generator
Weight decay
Total variation weight, λtv

Value
1380
120
128
4096
1 × 10−4
7.5 × 10−5
1 × 10−4
1.5 × 10−8

Table 2: Dataset settings and hyperparameter choices.

The transmissive baking network Ft takes the rendering information of only refraction objects (p, nt , mt , zt , and ωi ) and, as describedin Sec. 3.3, the R-buffer γ̂ generated by T the inputs. As we use
separated groups of G-buffers for transmissive objects, we need apermutation-invariant network structure mentioned in Sec. 3.2 tohandle the G-buffer groups with uncertain ordering.

of L1 loss and DSSIM [LMCB06], and the loss function for T is
described in Eq. (8). We trained all our models end-to-end and used
Adam [KB14] as the optimizer.

Finally, the final rendering network uses the outputs from Fr
and Ft to approximate Lo . We use positional encoding to process
all the input data, which has proven to be beneficial for neuralnetworks to learn high-frequency information [MST∗ 21]. The full
implementation details of the network structure can be found in thereleased code and the Fig. 1 of supplementary documents.

4.1. Experiment Settings

3.5. Training
We randomly choose the camera position and targeting center in abounding box with sizes depending on the scenes. Each of the 3D
coordinates of the position of movable objects follows a uniform
distribution with a given range. We generated our ground truth byusing Mitsuba 3 [JSRV22] path tracer. The ground truth for refracted
UV maps is generated by our custom renderer using a ray-castingalgorithm. Fr , Ft and R are trained as one neural network. To
achieve efficiency, T is separately trained because UV coordinates
ground truth is much easier to get, and, therefore, T can be trainedwith much more data than the networks using path tracing images as
ground truth. The loss function of Fr , Ft and R is the combination© 2024 Eurographics - The European Association
for Computer Graphics and John Wiley & Sons Ltd.

4. Experiments

We choose 256 × 256 as the input and output resolution for theexperiments. Note that resolution is not the main focus of thispaper. As indicated by previous works, it is feasible to enhance
resolution by placing a supersampler [ZSS24] or by using the PixelGenerator as the network backbone to support multi-resolution[GRPN20]. We test our models under three scenes: CORNELLBOX, DESK, and BEDROOM. BEDROOM is created from the
Bitterli datasets [Bit16]. CORNELLBOX and DESK are createdusing existing models and textures in public domain. The settingof datasets and hyperparameter choose during training are shownin Fig. 3 and Tab. 2. The training time of the neural renderer andR-buffer generator T are around 6 hours and 16 hours.
In this section, we use L1 reconstruction error, LPIPS [ZIE∗ 18],
SSIM [LMCB06], and Peak Signal-to-Noise Ratio (PSNR) as theevaluation metrics. We evaluate the models on transmissive only
area to further investigate the performance of our method focusing
on the refraction quality. Such areas are evaluated by L1 error and
PSNR (T.L1 and T.PSNR). As our method focuses on reconstruct6 of 9Z. Zhang & E. Simo-Serra / CrystalNet: Texture-Aware Neural Refraction Baking

Ours

GlassNet [ZSS24]

Active Exploration [DPD22]

BEDROOMDESK

CORNELLBOX

Ground Truth

Figure 4: Qualitative comparison. We compare with GlassNet and Active Exploration [DPD22] in three challenging scenes. Our method
achieves the best visual refraction rendering quality among all compared methods.

© 2024 Eurographics - The European Association
for Computer Graphics and John Wiley & Sons Ltd.

7 of 9Z. Zhang & E. Simo-Serra / CrystalNet: Texture-Aware Neural Refraction Baking

Scene

CORNELLBOXBEDROOM

DESK

Metrics

MethodL1 ↓LPIPS ↓

SSIM ↑PSNR ↑

T.L1 ↓

T.PSNR ↑T.Amp. ↓GlassNet
Active Exp.
Ours

0.029880.07726
0.028360.044440.179640.033810.891890.75572
0.90364

23.85371
17.59048
24.63194

0.22757
0.391709
0.21195

13.52789
10.20211
14.30030

0.115640.168400.11139

GlassNet
Active Exp.
Ours

0.03279
0.037210.02709

0.07125
0.139400.05951

0.881290.860130.89872

27.04549
24.30689
28.60590

0.188370.31550
0.16426

17.22410
12.35947
18.00709

0.085310.13601
0.08389

GlassNet
Active Exp.
Ours

0.03691
0.03515
0.02330

0.08818
0.07266
0.053440.881150.906370.93148

25.0712521.61073
27.69721

0.291590.264360.20789

12.78175
14.85235
15.25814

0.239170.353300.23184

Table 3: Quantitative comparison. Results show that our method, CrystalNet, has better performance on scenes with refraction objects
compared to GlassNet and Active Exploration. Best result is denoted in bold.

Variable Lighting(c) Denoiser

Figure 5: Comparison with a path tracing denoiser. The refraction
rendering quality of real-time path tracing denoiser is unsatisfied
even with 32 samples per pixel.

high frequency texture, for transmissive parts, we also include theL2 error of the amplitude spectrum on frequency domain (T.Amp.):ˆ =LAmp. (I, I)w,hˆ i, j∑i, j ( FFT I)i, j − FFT(I)
w·h

2
(10)

We convert the rendered images into frequency domain by Fast
Fourier Transform (FFT).
4.2. Comparison with the State-of-the-Art
We compare our method to other state-of-the-art methods: GlassNet [ZSS24], which only focuses on transparent objects with low
refraction index, and Active Exploration [DPD22] which bakes
scene global illumination based on active sampling processes. The
qualitative and quantitative results can be found in Fig. 4 and Tab. 3.
In summary, our proposed methods achieve the best specularrefraction rendering quality among the compared methods and testing scenes. In CORNELLBOX, other methods completely failed tocapture the refracted texture of the background wall, whereas our
method reconstructed it successfully. For smaller refraction objects
in DESK, GlassNet failed to render the checkerboard texture on
the teapot. Although Active Exploration was able to reconstruct thecheckerboard texture, the geometric structure of the refraction was
inaccurate. Our method achieves both better texture rendering quality and the precision of the refracted geometry. In BEDROOM, our© 2024 Eurographics - The European Association
for Computer Graphics and John Wiley & Sons Ltd.

Metrics

Ours

GlassNet

Ours

GlassNet

L1 ↓LPIPS ↓
T.L1 ↓
T.Amp. ↓0.02432
0.02203
0.16413
0.092400.03495
0.05687
0.25086
0.11860

0.03784
0.05375
0.17037
0.116880.039640.110110.210480.14551

Table 4: Quantitative comparison under more scene parameters.
Our experiments show that our method still performs strong undervariable lighting and IoR. Best result is highlighted in bold.

1.25

R-Generator Training Ended(b) Ours

Rendering loss (validation)(a) Ground truth

Variable IoR

1
0.75
0.5
0.25
0

5

10
15
Training time (hour)

CrystalNet
GlassNet

20

25

Figure 6: Evolution of the validation loss w.r.t. training time.

method also generates the refracted texture much better than othermethods. Furthermore, we compare the performance of the models
under the exact training time in Fig. 6 under CORNELLBOX. The
experiment shows GlassNet is unable to accurate render the scene
regardless the length of the training time.
To further demonstrate the advantage of using G-buffers as inputs, we test the ability of our model under variable lighting andindex of refraction (IoR) conditions. As shown in Fig. 7 and Tab. 4,
CrystalNet still significantly outperforms the GlassNet even under8 of 9Z. Zhang & E. Simo-Serra / CrystalNet: Texture-Aware Neural Refraction Baking

L1 Refraction Rendering Error
L1UV
L1UV + L1Tex
L2UV + L1Tex(a) Ground truth(b) Ours(c) GlassNet

Figure 7: Qualitative results on three refractive objects withvariable IoR. All three refractive objects have different IoR, andthe IoR of the glass plane varies from 1.6 to 2.

γ̂ = {n̂}γ̂ = {ĉ}γ̂ = {n̂, ĉ}

L1 ↓SSIM ↑

T.L1 ↓

T.Amp. ↓0.032200.025200.024390.884090.927360.933710.228620.174140.16471

0.12305 0.10046
0.09735

0.16490.16570.1710

Table 6: Impact of the texture reconstruction loss. Our experiments show the combination of texture reconstruction loss and UV
space loss does not improve the performance of the network. Best
result is highlighted in bold.

Table 5: Impact of compositions of the R-buffer. Results show
including refracted normal, n̂, and refracted texture, ĉ, in R-buffers
increases rendering quality. Best result is highlighted in bold.
(a) Ground truth

such challenging conditions. Additionally, our video demonstration
can be found the supplementary documents.
We also compared to the real-time path tracing denoising algorithm proposed by Chaitanya et al. [CKS∗ 17]. As shown in Fig. 5,
we achieve a much better refraction quality than the path tracing
denoiser under 32 sample-per-pixel under the similar frame timearound 18 to 20 milliseconds.
4.3. Ablation Study
One important decision is what types of information should be
included in the R-buffer. We explore whether including refracted
normal in R-buffer or not can improve the rendering quality. As
shown in Tab. 5, using predicted normals can positively affect therendering quality. We also investigated the effectiveness of combining texture reconstruction loss and UV coordinates loss whentraining the R-buffer generator. As shown in Tab. 6, using a combination of color space loss and UV space loss does not improve
but worsens the rendering quality. Therefore, we choose not to use
color space loss, which also decreases the required training time asconverting to during training is expensive.
We further demonstrate that our method is capable of handlingrefraction with roughness in Fig. 8. As shown in various previous works [XZX19, MST∗ 21], neural networks tend to learn lowfrequency information well, whereas ignores high-frequency information. Refraction on rough surface contains much lower frequency,
and thus can be baked well in neural rendering models.
4.4. Performance Analysis
We implement our method into a custom CUDA/OpenGL rendering
framework. Under this framework, our renderer is able to render
256 × 256 images with two refraction objects at 55 FPS using a
RTX 4090 GPU, achieving real-time performance. We also demonstrate the performance advantage to other methods. Under the same(b) Ours

Figure 8: Refraction with roughness. Our method is able to capture
the roughness effect.

hardware condition, Active Exploration requires averagely 40 milliseconds frame time, whereas our method is significantly more
efficient, requiring only averagely 18 milliseconds frame time. Furthermore, by utilizing a supersampler, CARN [AKS18], our method
can render 1024 × 1024 with two refraction objects at 25 FPS.
Additionally, as we are using the permutation-invariant structure
as described in Eq. (5), the memory consumption does not growwith the number of transmissive objects because the transmissive
objects are handled one by one.
5. Limitations and Discussion
Due to computation time scaling with resolution size, most neuralrendering approaches are limited to lower resolutions. In this work,
we have focused on improving the quality of the results, without
significant improvements in computational speed. To increase therendering resolution while maintaining similar performance, it isnecessary to employ supersampling-based approaches. We believethat through optimization and breakthroughs in supersampling willallow our proposed approach to run in real-time at acceptable resolutions, allowing for general usage.
Our approach is able to accurately reconstruct high frequency
textures through refraction thanks to the R-buffer, greatly improving
results over existing approaches. However, in one special situation
where total reflection occurs, the approximation error may become
significant due to R-buffer only capturing refraction rays. Although
it is not possible to directly extend the R-buffer concept to handle
total reflection, it should be possible to use a similar approach toapproximate the reflective component through baking it into anotherneural network component.
© 2024 Eurographics - The European Association
for Computer Graphics and John Wiley & Sons Ltd.

Z. Zhang & E. Simo-Serra / CrystalNet: Texture-Aware Neural Refraction Baking

References[AKS18] A HN N., K ANG B., S OHN K.-A.: Fast, accurate, andlightweight super-resolution with cascading residual network. In Proceedings of the European conference on computer vision (ECCV) (2018),
pp. 252–268. 8[Bit16] B ITTERLI B.: Rendering resources, 2016.
bitterli.me/resources/. 5

https://benedikt-[BMRF∗ 22] B EMANA M., M YSZKOWSKI K., R EVALL F RISVAD J., S EI DEL H.-P., R ITSCHEL T.: Eikonal fields for refractive novel-view synthesis. In ACM SIGGRAPH 2022 Conference Proceedings (2022), pp. 1–9.
3
[CKS∗ 17] C HAITANYA C. R. A., K APLANYAN A. S., S CHIED C.,S ALVI M., L EFOHN A., N OWROUZEZAHRAI D., A ILA T.: Interactive reconstruction of monte carlo image sequences using a recurrent
denoising autoencoder. ACM Transactions on Graphics (TOG) 36, 4(2017), 1–12. 3, 8
[CLZ∗ 23] C HEN X., L IU J., Z HAO H., Z HOU G., Z HANG Y.-Q.: Nerrf:
3d reconstruction and view synthesis for transparent and specular objects
with neural refractive-reflective fields. arXiv preprint arXiv:2309.13039(2023). 3

9 of 9[JSRV22] JAKOB W., S PEIERER S., ROUSSEL N., V ICINI D.: Dr.jit:
A just-in-time compiler for differentiable rendering. Transactions on
Graphics (Proceedings of SIGGRAPH) 41, 4 (July 2022). doi:10.
1145/3528223.3530099. 5[Kaj86] K AJIYA J. T.: The rendering equation. In Proceedings of the
13th annual conference on Computer graphics and interactive techniques(1986), pp. 143–150. 2, 3
[KB14] K INGMA D. P., BA J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). 5[LLW∗ 23] L I Z., L ONG X., WANG Y., C AO T., WANG W., L UO F.,
X IAO C.: Neto: neural reconstruction of transparent objects with selfocclusion aware refraction-tracing. In Proceedings of the IEEE/CVF
International Conference on Computer Vision (2023), pp. 18547–18557.
3[LMCB06] L OZA A., M IHAYLOVA L., C ANAGARAJAH N., B ULL D.:
Structural similarity-based object tracking in video sequences. In 2006
9th International Conference on Information Fusion (2006), pp. 1–6. 5[MRNK21] M ÜLLER T., ROUSSELLE F., N OVÁK J., K ELLER A.:
Real-time neural radiance caching for path tracing. arXiv preprint
arXiv:2106.12372 (2021). 2[DCS∗ 24] D ENG W., C AMPBELL D., S UN C., K ANITKAR S., S HAFFER
M., G OULD S.: Ray deformation networks for novel view synthesis ofrefractive objects. In Proceedings of the IEEE/CVF Winter Conference
on Applications of Computer Vision (2024), pp. 3118–3128. 3[MST∗ 21] M ILDENHALL B., S RINIVASAN P. P., TANCIK M., BARRON
J. T., R AMAMOORTHI R., N G R.: Nerf: Representing scenes as neuralradiance fields for view synthesis. Communications of the ACM 65, 1(2021), 99–106. 5, 8

[DPD22] D IOLATZIS S., P HILIP J., D RETTAKIS G.: Active exploration
for neural global illumination of variable scenes. ACM Transactions on
Graphics (TOG) 41, 5 (2022), 1–18. 2, 6, 7

[OB07] O LIVEIRA M. M., B RAUWERS M.: Real-time refraction throughdeformable objects. In Proceedings of the 2007 symposium on Interactive
3D graphics and games (2007), pp. 89–96. 2[DRBS∗ 11] D E ROUSIERS C., B OUSSEAU A., S UBR K., H OLZSCHUCH
N., R AMAMOORTHI R.: Real-time rendering of rough refraction. IEEE
Transactions on Visualization and Computer Graphics 18, 10 (2011),
1591–1602. 2[PSH∗ 22] PAN J.-I., S U J.-W., H SIAO K.-W., Y EN T.-Y., C HU H.-K.:
Sampling neural radiance fields for refractive objects. In SIGGRAPH
Asia 2022 Technical Communications (2022), pp. 1–4. 3[ED06] E ISEMANN E., D ÉCORET X.: Fast scene voxelization and applications. In Proceedings of the 2006 symposium on Interactive 3Dgraphics and games (2006), pp. 71–78. 2[EJRB∗ 18] E SLAMI S. A., J IMENEZ R EZENDE D., B ESSE F., V IOLA F.,
M ORCOS A. S., G ARNELO M., RUDERMAN A., RUSU A. A., DANI HELKA I., G REGOR K., ET AL .: Neural scene representation and rendering. Science 360, 6394 (2018), 1204–1210. 2[GGH∗ 22] G AO K., G AO Y., H E H., L U D., X U L., L I J.: Nerf: Neuralradiance field in 3d vision, a comprehensive review. arXiv preprint
arXiv:2210.00379 (2022). 3[QSMG17] Q I C. R., S U H., M O K., G UIBAS L. J.: Pointnet: Deeplearning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition(2017), pp. 652–660. 4[RFB15] RONNEBERGER O., F ISCHER P., B ROX T.: U-net: Convolutional networks for biomedical image segmentation. In Medical image
computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings,
part III 18 (2015), Springer, pp. 234–241. 3[ROF92] RUDIN L. I., O SHER S., FATEMI E.: Nonlinear total variation
based noise removal algorithms. Physica D: nonlinear phenomena 60,
1-4 (1992), 259–268. 4[GLD06] G ÉNEVAUX O., L ARUE F., D ISCHLER J.-M.: Interactive refraction on complex static geometry using spherical harmonics. In Proceedings of the 2006 symposium on Interactive 3D graphics and games(2006), pp. 145–152. 2[RWG∗ 13] R EN P., WANG J., G ONG M., L IN S., T ONG X., G UO B.:
Global illumination with radiance regression functions. ACM Trans.
Graph. 32, 4 (2013), 130–1. 2[GMX22] G AO D., M U H., X U K.: Neural global illumination: Interactive indirect illumination prediction under dynamic area lights. IEEE
Transactions on Visualization and Computer Graphics (2022). 2[Wym05a] W YMAN C.: An approximate image-space approach for interactive refraction. ACM transactions on graphics (TOG) 24, 3 (2005),
1050–1053. 2[GP16] G UO J., PAN J.-G.: Real-time rendering of refracting transmissive
objects with multi-scale rough surfaces. The Visual Computer 32 (2016),
1579–1592. 2[Wym05b] W YMAN C.: Interactive image-space refraction of nearby
geometry. In Proceedings of the 3rd international conference on Computer
graphics and interactive techniques in Australasia and South East Asia
(2005), pp. 205–211. 2[GRPN20] G RANSKOG J., ROUSSELLE F., PAPAS M., N OVÁK J.: Compositional neural scene representations for shading inference. ACM Transactions on Graphics (TOG) 39, 4 (2020), 135–1. 2, 5
[GSHG98] G REGER G., S HIRLEY P., H UBBARD P. M., G REENBERG
D. P.: The irradiance volume. IEEE Computer Graphics and Applications
18, 2 (1998), 32–43. 2[HDHN16] H EITZ E., D UPUY J., H ILL S., N EUBELT D.: Real-time
polygonal-light shading with linearly transformed cosines. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1–8. 5[IZZE17] I SOLA P., Z HU J.-Y., Z HOU T., E FROS A. A.: Image-to-imagetranslation with conditional adversarial networks. In Proceedings of theIEEE conference on computer vision and pattern recognition (2017),
pp. 1125–1134. 5© 2024 Eurographics - The European Association
for Computer Graphics and John Wiley & Sons Ltd.

[XZX19] X U Z.-Q. J., Z HANG Y., X IAO Y.: Training behavior of deepneural network in frequency domain. In Neural Information Processing:
26th International Conference, ICONIP 2019, Sydney, NSW, Australia,
December 12–15, 2019, Proceedings, Part I 26 (2019), Springer, pp. 264–274. 8[ZIE∗ 18] Z HANG R., I SOLA P., E FROS A. A., S HECHTMAN E., WANG
O.: The unreasonable effectiveness of deep features as a perceptual metric.
In Proceedings of the IEEE conference on computer vision and patternrecognition (2018), pp. 586–595. 5[ZSS24] Z HANG Z., S IMO -S ERRA E.: Neural scene baking for permutation invariant transparency rendering with real-time global illumination.
arXiv preprint arXiv:2405.19056 (2024). 1, 2, 3, 4, 5, 6, 7