Plateau-Reduced Differentiable Path Tracing

Michael Fischer 1 Tobias Ritschel 1

Abstract
Differentiable renderers provide gradients w.r.t.
arbitrary scene parameters, but the mere existenceof these gradients does not guarantee useful update steps in an optimization. Instead, inverse
rendering might not converge due to plateaus, i.e.,
regions of zero gradient, in the objective function. We propose to alleviate this by convolving the rendering equation with an additional kernel that blurs the parameter space. We describe
two Monte Carlo estimators to compute plateaureduced gradients efficiently, i.e., with low variance, and show that these translate into net-gains
in optimization error and runtime. Our approachis a straightforward extension to both black-box
and differentiable renderers and enables optimization of problems with intricate light transport,such as caustics or global illumination, that existing differentiable renderers do not converge
on. Our code is at https://github.com/
mfischer-ucl/prdpt.

InitialReference

Path Tracer

Our Method

Figure 1. Optimization results with a differentiable path tracer (weuse Mitsuba 3 (Jakob et al., 2022)) and our proposed method. The
task is to rotate the cup around its z-axis to match the reference.
Due to a plateau in the objective function (when the handle isoccluded by the cup), regular methods do not converge.

To alleviate this, previous work uses re-sampling of silhouette edges and integrand reparametrizations (Li et al.,2018; Loubet et al., 2019), which enable the optimization ofprimitive- or light positions. For rasterization, differentiability is achieved by replacing discontinuous edge- and z-tests
with hand-crafted derivatives (Loper & Black, 2014; Rhodin
et al., 2015; Liu et al., 2019; Le Lidec et al., 2021). Unfortunately, rasterization by design does not capture complex
light transport effects, e.g., global illumination or caustics.
However, Metz et al. (2021) show that the mere existence of
gradients is no guarantee for an optimization’s convergence.
In fact, there are surprisingly many cases where they do notlead to a successful optimization, due to a plateau in theobjective function. An example is the optimization of the
mug’s rotation in Fig. 1: As soon as the handle disappears
behind the cup, no infinitesimally small rotation change will
result in a reduced loss. We have hence reached a plateau in
the objective function, i.e., a region of zero gradients.

1. Introduction
Regressing scene parameters from 2D observations is a taskof significant importance in graphics and vision, but also a
hard, ill-posed problem. When all rendering steps are differentiable, we can derive gradients of the final image w.r.t.
the scene parameters. However, differentiating through thediscontinuous rendering operator is not straightforward due
to, e.g., occlusion. The two prevalent rendering approaches
are path tracing and rasterization.

To alleviate this, we take inspiration from differentiable
rasterization, where discontinuities are replaced by smoothapproximations (Liu et al., 2019). This makes the edge- andz-tests continuous and hence differentiable, and in passing(and much less studied) removes plateaus. In this work, we
aim to find a way to apply the same concept to complex
light transport. We thus path-trace an alternative, smoothversion of the Rendering Equation (RE), which we achieve
by convolving the original RE with a smoothing kernel. Our
proposed method is a lightweight extension to (differentiable) path tracers that extends the infinitely-dimensional
path integral to the product space of paths and scene parameters. We further show that the resulting double integral
can be MC-solved efficiently through our derived variancereduction techniques, importance- and antithetic sampling.

Path-tracing aims to solve the rendering equation by computing a Monte Carlo (MC) estimate for each pixel. Unfortunately, MC is only compatible with modern Automatic Differentiation (AD) frameworks under continuous integrands,e.g., color, but not for spatial derivatives.
1

Department of Computer Science, University College London, United Kingdom. Correspondence to: Michael Fischer<m.fischer@cs.ucl.ac.uk>.
Published at the Differentiable Almost Everything Workshop of the
40 th International Conference on Machine Learning, Honolulu,
Hawaii, USA. July 2023. Copyright 2023 by the author(s).

1Plateau-Reduced Differentiable Path Tracing

2. Background2.3. Rasterization

2.1. Rendering equation

Rasterization is often used in practical applications due toits simplicity and efficiency, but lacks the ability to readily
compute complex light transport phenomena, as it solves a
simplified version of the RE, where for every pixel, the lightpath length is limited to one.

The RE (Kajiya, 1986) defines a pixel P asZ
P (θ) =f (x, θ)dx ,(1)Ω

an integral of the scene function f (x, θ), that depends onscene parameters θ ∈ Θ, over all light paths x ∈ Ω. In
inverse rendering, we aim to find the parameters θ∗ that best
explain the pixels Pi in the reference image withθ∗ = arg min
θXi

L (Pi (θ) − Pi (θref )) ,A rasterizer projects the a scene’s primitives to screen space
and then resolves occlusion. Both these operations are inherently non-differentiable due to jump discontinuities, which
therefore – in order to backpropagate gradients throughthem – need to be replaced with smooth approximations,e.g., a Sigmoid. For a survey on differentiable rasterizationand the used employed smoothing approximations, we refer
to (Kato et al., 2020) and (Petersen et al., 2022).

(2)

where L is the objective function and Pi (θref ) are the target
pixels created by the (unknown) parameters θref .

Choosing smoothing functions with infinite support (forinstance, the Sigmoid), implicitly resolves the plateau problem as well. Our method (Sec. 3) draws inspiration fromthis concept of “differentiating through blurring”. However,most differentiable rasterizers make simplifying assumptions, e.g., constant colors, the absence of shadows or reflections, and no illumination interaction between objects. We
will see in later examples that this leads to non-convergencein multi-bounce light transport scenarios, e.g., the optimization from a scene’s global illumination. Our formulation, in
contrast, does not make such assumptions.

We will now discuss the two predominant ways to differentiably solve the RE: path tracing, which can simulate allforms of complex light transport but suffers from plateaus(Sec. 2.2), and rasterization, which removes plateaus but
is limited to simple once-bounce light transport (Sec. 2.3).
Other renderers, e.g., volumetric rendering (Henzler et al.,2019; Mildenhall et al., 2021) or neural rendering (Nalbach
et al., 2017; Sitzmann et al., 2021) are mostly limited tosimple, approximate light transport or come with their own
drawbacks (e.g., static scene constraints) and thus are notwithin the scope of this work.

3. Plateau-reduced GradientsAs differentiable rasterization (cf. Sec. 2.3) has established,
the blurring of primitive edges is a viable means for differentiation. But what if there is no “primitive edge” in the firstplace, as we deal with general light paths instead of simple
rasterization? The edge of a shadow, for instance, is not
optimizable itself, but the result of a combination of light
position, reflection, occlusion, etc. Therefore, to achieve aneffect similar to that of differentiable rasterizers, we would
need a method that blurs the entire light path (instead of justprimitive edges) over the parameter space θ. If this method
used a blur kernel with infinite support (e.g., a Gaussian
distribution), the plateau in the objective would vanish, asa small parameter change in any direction would induce achange in the objective function.

2.2. Path tracing
As there is no closed-form solution to Eq. 1, path tracing
uses MC to estimate the integral by sampling the integrandat random paths xi :1 X
Pb ≈f (xi , θ)(3)
N i

We are interested in the partial derivatives of P with respect
to the scene parameters θ, i.e.,
Z
Z
∂
∂
∂P
=
f (x, θ)dx . (4)f (x, θ)dx =
∂θ
∂θ Ω
∂θ
Ω

In order to make Eq. 4 compatible with automatic differentiation, Li et al. (2018) propose a re-sampling of silhouetteedges and Loubet et al. (2019) suggest a re-parametrizationof the integrand. Both approaches allow to MC-estimate thegradient as
N
d
∂P
1 X ∂
≈
f (xi , θ) .
∂θN i ∂θIn Fig. 2, we again aim to optimize the cup’s rotation aroundits z-axis to have the handle point to the right, a 1D problem.
As we have seen previously, using an image-based objectivefunction leads to a plateau in the “hard” optimization setting, i.e., without blur (the blue line in the plot). Blurring
the cup’s rotation parameter, on the other hand, leads toθ continuously influencing the value of the objective andtherefore resolves the plateau (orange line in the plot). Naturally, it is easy to descend along the gradient of the orange
curve, while the gradient is zero on the plateau of the blue
curve.

(5)This is now standard practice in modern differentiable rendering packages (Nimier-David et al., 2019; Li et al., 2018;
Zeltner et al., 2021; Zhang et al., 2021; 2020), none of whichattempt to actively resolve plateaus.
2Plateau-Reduced Differentiable Path Tracing

b)

c)

d)

Image MSEa)y=κ(τ)
Mug Rotationy=∇κ(τ)y=|∇κ(τ)|

Figure 3. Our kernel κ (a), its gradient ∇κ (b), the positivized
gradient (c) and samples drawn proportional thereto (d).

Figure 2. Optimizing the cup’s rotation in the hard (left, blue) andsmooth (right, orange) setting (note the blurred handle). The imagespace loss is displayed on the right: blurring removes the plateau.

rendering features. Our derivation also supports this case,as we can rewrite Eq. 7 as

3.1. The Plateau-reduced Rendering Equation

Nd∂Q
1 X ∂κ≈(τi ) P (θ − τi ) ,∂θ
N i=1 |∂θ{z } | {z }We realize our blurring operation as a convolution of the
RE (Eq. 1) with a blur kernel κ over the parameter space Θ:Diff. Kernel

Q(θ) = κ ⋆ P (θ) =
=

Z

κ(τ )

ZΘ

Θ×Ω

Z

Ω

κ(τ )f (x, θ − τ ) dxdτ . (6)While κ(τ ) could be any symmetric monotonous decreasing
function, we here choose a Gaussian kernel. The kernel
acts as a weighting function that weights the contributionof parameters θ that were offset by τ ∈ Θ. This means
that, in addition to integrating all light paths x over Ω, we
now also integrate over all parameter offsets τ in Θ. Notethat we do not convolve across the path space Ω but acrossthe parameter space θ, e.g., the cup’s rotation in Fig. 2.
Analogous to Eq. 3 and Eq. 4, we can estimate the integral
in Eq. 6 through an MC estimator and take its derivative as

Renderer

3.2. Variance Reduction
Drawing uniform samples from Θ × Ω will result in a sample distribution that is not proportional to the integrandand hence lead to high-variance gradient estimates and ultimately slow convergence for inverse rendering. In our case,the integrand is the product the kernel κ and the scene function f , which Veach (Veach, 1998) showed how to optimallysample for. As we generally consider the rendering operatora black box, we can only reduce variance by sampling according to the remaining function, the (differentiated) kernelκ (Fig. 3b).

(7)

Due to the linearity of differentiation and convolution, there
are two ways of computing Eq. 7: one for having a differentiable renderer, and one for a renderer that is not differentiable. We discuss both options next.

While importance-sampling for a Gaussian (τi ∼ κ,
required to reduce variance of Eq. 8) is easily done,
importance-sampling for the gradient of a Gaussian (τi ∼∂κ/∂θ, to be applied to Eq. 9) is more involved, as thegradient of our kernel
 2
∂κ
−τ
−τ
(10)
(τ ) = √ exp
3
∂θ
2σ 2
σ 2π

Plateau-reduced gradients if P is differentiable With
access to a differentiable renderer (i.e., access to ∂P /∂θ),
we can rewrite Eq. 7 as
N
d
∂Q
1 X
∂P
=
(θ − τi ) .
κ(τi )∂θ
N i=1}|∂θ {z(9)which equals sampling a non-differentiable renderer andweighting the result by the gradient of the blur kernel. Thisis possible due to the additional convolution we introduce:
prior work (Li et al., 2018; Loubet et al., 2019) must takespecial care to compute derivatives (Eq. 5), as in their case,
optimizing θ might discontinuously change the pixel integral. We circumvent this problem through the convolution
with κ, which ensures that, in expectation, θ continuously
influences the pixel integral.

f (x, θ − τ ) dxdτ

N
d
∂ 1 X
∂Q
κ(τi )f (xi , θ − τi ) .
=∂θ∂θ N i=1y~|∇κ(τ)|(8)

is negative for τ > 0. We enable sampling proportional to
its Probability Density Function (PDF) by “positivization”(Owen & Zhou, 2000), and hence sample for | ∂κ∂θ (τ )| instead(Fig. 3c). The kernel’s closed-form PDF allows us to use theinverse Cumulative Distribution Function (ICDF) method
for importance sampling. The ICDF of Eq. 10 isp
F −1 (ξ) = −2σ 2 log(ξ) ,Diff. Renderer

Therefore, all that that needs to be done is to classicallycompute the gradients of a randomly perturbed rendering
and weight them by the blur kernel.
Plateau-reduced gradients if P is not differentiable In
several situations, we might not have access to a differentiable renderer, or a non-differentiable renderer might haveadvantages, such as computational efficiency or advanced

into which we feed uniform random numbers ξ ∈ (0, 1) thatgenerate samples proportional to | ∂κ∂θ (τ )| (Fig. 3d).

3Plateau-Reduced Differentiable Path Tracing
CUP
Img. MSE

GLOBAL ILLUMINATIONSHADOWS

Param. MSE

Img. MSE

Param. MSE

Img. MSE

Param. MSE

InitialReference

InitialReference

InitialReference

Diff. Path Tracer

Ours

Diff. Path Tracer

Ours

Diff. Path Tracer

Ours

Figure 4. We show the optimization tasks and results for Our∂κP (“Ours”, Eq. 9, orange) and our baseline Mitsuba 3 (“Diff. Path Tracer”,blue). The right upper subfigure in each column depicts the scene setup, with the optimizable parameters colored in red. All methods
operate in image space only. We additionally plot parameter-space error as a quality control metric.

Timing Additionally, we have found that our approach’s
runtime on average is 8× faster than differentiable renderingwith Mitsuba, as our stochastic gradient estimation throughthe derivative-kernel allows us to skip the gradient computation step of the renderer, which leads to significant savingsin computation time. Moreover, as we do not need special
treatment of discontinuities (e.g., re-parametrization), we
can use Mitsuba’s regular forward path tracer at no additional runtime cost.

In order to obtain a zero-variance estimator, we allocate anequal number of samples to the positivized and regular parts(Owen & Zhou, 2000). As the function is point symmetric around the origin, we use antithetic sampling to do so(Hammersley & Mauldon, 1956), i.e., for each sample τ ,we additionally generate its negated counterpart −τ .
Moreover, we decrease the amount of smoothing, the kernel’s bandwidth σ, over time. For more information on
implementation as well as an outline of our plateau-reduced
gradient computation, please cf. Appendix A.

5. Discussion and Conclusion
Relation to Variational Optimization Indeed, the formalism developed in Sec. 3.1 can be interpreted as a form ofvariational optimization (Staines & Barber, 2012; 2013),
where one would descend along the (smooth) variational
objective instead of the true underlying function. As such,
Eq. 9 can be seen as an instance of a score-based gradientestimator (Sutton et al., 1999), while Eq. 8 can be interpreted as reparametrization gradient (Kingma et al., 2015;
Schulman et al., 2015). Suh et al. (2022) provide intuitionon each estimator’s performance and align with our findingsof the score-based estimator’s superiority under a discontinuous objective. It is one of the contributions of this work toconnect these variational approaches with inverse rendering.

4. Results
We analyze our method and its variants on a variety oftasks that feature advanced light transport, plateaus andambiguities (for detailed task descriptions, cf. Appendix B,
for task visualizations, cf. Fig. 4). We compare our method(Eq. 9) against Mitsuba3 (Jakob et al., 2022) and show
numerical comparisons against other methods in Appendix
B. We run all methods for the same number of iterations
and with the same settings.
Performance As is evident from Fig. 4, our method converges reliably on tasks where regular differentiation does
not converge due to plateaus in the objective. This is due toour stochastic smoothing of the parameter space, which effectively allows the optimizer to descend a smoother versionof the loss landscape. Importantly, we effectively handle
complex light transport, as is evident from the rightmost column in Fig. 4, where out-of-view objects in the scene scene
are optimized solely from the global illumination scatteredfrom the walls. For a more detailed discussion of our resultson each task, cf. Appendix B.

Conclusion In summary, we have proposed a method forinverse rendering that convolves the rendering equation witha smoothing kernel, which allows straight-forward differentiation and removes plateaus. The idea combines strengthsof differentiable rasterization and differentiable path tracing. Our approach is simple to implement, efficient, hastheoretical justification and optimizes tasks that existing
differentiable renderers so far have diverged upon.
4

Plateau-Reduced Differentiable Path Tracing

References

Nimier-David, M., Vicini, D., Zeltner, T., and Jakob, W.
Mitsuba 2: A retargetable forward and inverse renderer.
ACM Trans. Graph., 38(6), 2019.

Hammersley, J. M. and Mauldon, J. General principles ofantithetic variates. In Mathematical proceedings of theCambridge philosophical society, volume 52, 1956.

Owen, A. and Zhou, Y. Safe and effective importancesampling. J of the American Statistical Association, 95(449), 2000.

Henzler, P., Mitra, N., and Ritschel, T. Escaping plato’s caveusing adversarial training: 3d shape from unstructured 2d
image collections. In Proc. ICCV, 2019.

Petersen, F., Goldluecke, B., Borgelt, C., and Deussen, O.
Gendr: A generalized differentiable renderer. In Proc.
CVPR, 2022.

Jakob, W., Speierer, S., Roussel, N., Nimier-David, M.,
Vicini, D., Zeltner, T., Nicolet, B., Crespo, M., Leroy, V.,
and Zhang, Z. Mitsuba 3 renderer, 2022. https://mitsubarenderer.org.

Rhodin, H., Robertini, N., Richardt, C., Seidel, H.-P., andTheobalt, C. A versatile scene model with differentiablevisibility applied to generative pose estimation. In Proc.
ICCV, 2015.

Kajiya, J. T. The rendering equation. In Proc. SIGGRAPH,
1986.

Schulman, J., Heess, N., Weber, T., and Abbeel, P. Gradient estimation using stochastic computation graphs.
Advances in neural information processing systems, 28,
2015.

Kato, H., Beker, D., Morariu, M., Ando, T., Matsuoka, T.,
Kehl, W., and Gaidon, A. Differentiable rendering: A
survey. arXiv preprint arXiv:2006.12057, 2020.
Kingma, D. P., Salimans, T., and Welling, M. Variational
dropout and the local reparameterization trick. Advancesin neural information processing systems, 28, 2015.

Sitzmann, V., Rezchikov, S., Freeman, B., Tenenbaum, J.,
and Durand, F. Light field networks: Neural scene representations with single-evaluation rendering. NeurIPS, 34,
2021.

Le Lidec, Q., Laptev, I., Schmid, C., and Carpentier, J. Differentiable rendering with perturbed optimizers. NeurIPS,
34, 2021.

Staines, J. and Barber, D. Variational optimization. arXiv
preprint arXiv:1212.4507, 2012.
Staines, J. and Barber, D. Optimization by variational bounding. In ESANN, 2013.

Li, T.-M., Aittala, M., Durand, F., and Lehtinen, J. Differentiable Monte Carlo ray tracing through edge sampling.
ACM Trans Graph., 37(6), 2018.

Suh, H. J., Simchowitz, M., Zhang, K., and Tedrake, R.
Do differentiable simulators give better policy gradients?
In International Conference on Machine Learning, pp.
20668–20696. PMLR, 2022.

Liu, S., Li, T., Chen, W., and Li, H. Soft rasterizer: A
differentiable renderer for image-based 3d reasoning. In
Proc. ICCV, 2019.

Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y.
Policy gradient methods for reinforcement learning withfunction approximation. Advances in neural information
processing systems, 12, 1999.

Loper, M. M. and Black, M. J. Opendr: An approximate
differentiable renderer. In Proc. ECCV, 2014.
Loubet, G., Holzschuch, N., and Jakob, W. Reparameterizing discontinuous integrands for differentiable rendering.
ACM Tran Graph., 38(6), 2019.

Veach, E. Robust Monte Carlo methods for light transport
simulation. Stanford University, 1998.

Metz, L., Freeman, C. D., Schoenholz, S. S., and Kachman, T. Gradients are not all you need. arXiv preprint
arXiv:2111.05803, 2021.

Zeltner, T., Speierer, S., Georgiev, I., and Jakob, W. Monte
carlo estimators for differential light transport. ACM
Trans Graph, 40(4), 2021.

Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T.,
Ramamoorthi, R., and Ng, R. Nerf: Representing scenes
as neural radiance fields for view synthesis. Communications of the ACM, 65(1), 2021.

Zhang, C., Miller, B., Yan, K., Gkioulekas, I., and Zhao, S.
Path-space differentiable rendering. ACM Trans. Graph.,39(4), 2020.
Zhang, C., Dong, Z., Doggett, M., and Zhao, S. Antithetic
sampling for monte carlo differentiable rendering. ACM
Trans. Graph., 40(4), 2021.

Nalbach, O., Arabadzhiyska, E., Mehta, D., Seidel, H.-P.,
and Ritschel, T. Deep shading: convolutional neuralnetworks for screen space shading. Comp. Graph. Forum,
36(4), 2017.
5

Plateau-Reduced Differentiable Path Tracing

A. Implementation Details

this task, as they a) do not implement shadows, and b)
cannot differentiate what they do not rasterize. Again, ourmethod matches the reference position very well. Mitsuba
first moves the sphere away from the plane (in negative
z-direction), as this reduces the footprint of the sphere’s
shadow on the plane and thus leads to a lower error, andthen finally moves the sphere out of the image, where aplateau is hit and the optimization can not recover. The blueline in the image-space plot in Fig. 4 illustrates this problem,as the parameter-error keeps changing very slightly, but theimage-space error stays constant.

A.1. Adaptive Bandwidth
Adjusting σ, the kernel’s spread, controls over how far fromthe current parameter θ our samples will be spaced out. A
high σ may be useful in the early stages of the optimization,
when there still is a considerable difference between θ andθref , whereas we want a low σ towards the end of the optimization to zero-in on θref . Throughout the optimization,
we hence decay the initial σ0 according to a linear schedule,i.e., σt+1 = σ0 −t(σ0 −σm ), where σm is a fixed minimum
value we choose to avoid numerical instabilities that would
otherwise arise from σ → 0 in, e.g., Eq. 10.

G LOBAL I LLUMINATION We here show that our method
is compatible with the ambiguities encountered in advanced
light transport scenarios. The goal of this optimization task
is to simultaneously move the top-light to match the scene’s
illumination, change the left wall’s color to create the color
bleeding onto the box, and also to rotate the large box to anorientation where the wall’s reflected light is actually visible.
The optimization only sees an inset of the scene (as shownin Fig. 4) and hence only ever sees the scattered light, butnever the wall’s color or light itself. This task therefore is notsolvable for differentiable rasterizers, as they do not modelsuch advanced light transport. The differentiable path tracer,on the other hand, cannot resolve the ambiguity between
the box’s rotation, the light position and the wall’s color, as
it is operating in a non-smoothed space. Our method finds
the correct combination of rotation, light position and wallcolor.

A.2. Implementation
We outline our gradient estimator in pseudo-code in Alg. 1.
We importance-sample for our kernel with zero variance,
use antithetic sampling and adapt the smoothing strength
via σ. As Alg. 1 shows, our method is simple to implement
and can be incorporated into existing frameworks with only
a few lines of code. We implement our method in PyTorch,with Mitsuba as rendering backbone, and use Adam as our
optimizer.
Algorithm 1 Gradient estimation of the scene function f atparameters θ with perturbations τ ∼ N (0, σ) at N samples.
1: # Equation 9
2: procedure E STIMATE G RADIENT(P , θ, σ, N )3:
G := 04:
for i ∈ (1, N/2) do
5:
ξ←U
pNIFORM(0, 1)
6:τ ← −2σ 2 log(ξ)
7:G ← G + P (θ + τ ) − P (θ − τ )8:end for9:
return G / N
10: end procedure

We will now detail three additional tasks that we did notinclude in the main text for brevity. Their qualitative results
are displayed in Fig. 5.
O CCLUSION Here, a sphere translates along the viewing
axis to match the reference. The challenge is that the sphere
initially is occluded by another sphere, i.e., we are on aplateau as long as the occluder is closer to the camera than
the sphere we are optimizing. The baseline path tracer initially pushes the red sphere towards the back of the box,
as this a) reduces the error in the reflection on the bottom
glass plane, and b) lets the shadow of the red sphere (visible underneath the blue sphere in the initial configuration)
shrink, which again leads to a lower image-space error. Our
method, in contrast, successfully differentiates through boththe plateau (the red sphere has a negligible effect on the objective) and the discontinuity that arises when the red spherefirst moves closer to the camera than the blue occluder.

B. Tasks and Additional Results
The following will detail the task setup and our method’s
optimization performance on our evaluation tasks.
B.1. Tasks
C UP A mug is rotated around its vertical axis and as its handle gets occluded, the optimization has reached a plateau.
Our method differentiates through the plateau. The differentiable path tracer gets stuck in the local minimum afterslightly reducing the loss by turning the handle towards theleft, due to the direction of the incoming light.

S ORT: In this task, we need to sort a randomly perturbed
assortment of 75 colored primitives into disjoint sets. We
optimize the x- and y-coordinates of each cube, which leads
to a 150-dimensional setting, with a plateau in each dimension, as most of the cubes are initially not overlapping theirreference. Mitsuba cannot find the correct position of nonoverlapping primitives and moves them around to minimize

S HADOWS An object outside of the view frustum is castinga shadow onto a plane. Our goal is to optimize the hidden
object’s position. Differentiable rasterizers can not solve
6Plateau-Reduced Differentiable Path Tracing
SORT

OCCLUSION
Img. MSE

Param. MSE

Img. MSE

CAUSTIC

Param. MSE

Img. MSE

Param. MSE

InitialReference

InitialReference

InitialReference

Diff. Path Tracer

Ours

Diff. Path Tracer

Ours

Diff. Path Tracer

Ours

Figure 5. We show the optimization tasks and results for Our∂κP (Eq. 9, orange) and our baseline Mitsuba 3 (“Diff. Path Tracer”, blue).
Table 1. Image- and parameter-space MSE of different methods (columns) on different tasks (rows). Lower is better for both metrics.

Rasterizer
SoftRasImg.

C UP
S HADOWSO CCL .
G LOBAL I LL .
S ORTC AUSTIC

Para.

Path Tracer
MitsubaImg.

3.66×10−1 2.72×10−2 5.49×10−3
1.64×10−3 1.42×10−1 1.64×10−3
5.33×10−2 7.18×10−3 5.85×10−2–– 3.78×10−2
1.85×10−2 1.57×10−0 1.18×10−2–– 3.12×10−1

Our∂κP

Ourκ∂P

Para.

Img.

Para.

Img.

Para.

0.75×10−1
5.06×10−0
5.23×10+1
3.87×10−1
6.64×10−0
8.50×10−0

4.92×10−6
1.74×10−5
2.34×10−4
5.07×10−5
3.81×10−3
1.89×10−5

4.18×10−7
1.81×10−3
3.29×10−3
8.71×10−4
4.19×10−1
9.76×10−5

4.75×10−4
5.12×10−4
5.37×10−2
5.88×10−2
1.02×10−2
2.42×10−1

2.77×10−1
1.28×10−0
1.87×10+1
2.55×10−1
2.24×10−0
4.03×10−0

B.2. Quantitative Results

the image error, which is ultimately achieved by moving
them outside of the view frustum. Our method, admittedly
not perfect on this task, finds more correct positions, a result
more similar to the reference.

Tab. 1 reports image- and parameter-space MSE and confirms what Fig. 4 conveyed visually: regular gradient-based
path tracers that operate on non-smooth loss landscapes failcatastrophically on our tasks. SoftRas manages to overcome some plateaus, but struggles with achieving accurate
results, as it blurs in image space but must compare to thenon-blurred reference, which leads to a notable difference
between the final state and the reference parameters. Our
method Our∂κP , in contrast, achieves errors of as low as
10−7 , and consistently outperforms its competitors on alltasks by several orders of magnitude. Interestingly, Ourκ∂P(i.e., using the gradients from the differentiable renderer)
works notably worse than Our∂κP , which we attribute to
the fact that we cannot importance-sample for the gradienthere, as we do not know its PDF. Instead, we can only draw
samples proportional to κ(τ ), which places many samples
where the kernel is high, i.e., at the current parameter. As we
can see from the rigid optimization by Mitsuba, the gradientat the current position is not very informative, so placing
samples there is not very helpful.

C AUSTIC Lastly, the C AUSTIC task features a light source
outside the view frustum illuminating a glass sphere, whichcasts a caustic onto the ground. The goal is to optimize thelight’s position in order to match a reference caustic. As
the sphere does not change its appearance with the light’s
movement, the optimization has to solely rely on the caustic’s position to find the correct parameters. Similar to the
GI task, this is not solvable for rasterizers. Our method
reaches the optimum position with high accuracy. For thebaseline path tracer, we see a failure mode that is similar
to the S HADOW task. In this case, the image space error
can be reduced by moving the light source far away, as mostof the error comes from the caustic not being at the correct
position. Moving the light source far away reduces thiserror, but also leads into a local minimum where there is no
illumination at all, resulting in the gray image in Fig. 5.

7