Transforming a Non-Differentiable Rasterizer into a Differentiable One
with Stochastic Gradient Estimation
Thomas Deliot 1 Eric Heitz 1 Laurent Belcour 1Abstract

Slang.D [7] is not an option. Using existing differentiable
rasterizers such as NV D IFF R AST [8] requires externalizing
the workflow and relying on external (sometimes vendorspecific) dependencies, which is also problematic. This is
why we aim at transforming an existing non-differentiable
rasterizer into a differentiable one.

We show how to transform a non-differentiable
rasterizer into a differentiable one with minimalengineering efforts and no automatic differentiation. To do so, we improve on Stochastic Gradient Estimation by using a Per-Pixel Loss whichleverage the fact that only a few primitives contribute to a given pixel. Estimating gradients on
a per-pixel basis bounds the dimensionality ofthe optimization problem and makes the method
scalable. To track parameters contributing to apixel, we use ID- and UV-buffers, which are often
already available or trivial to obtain. With theseminor modifications, we obtain an in-engine optimizer for 3D assets with millions of geometry
and texture parameters.

Contribution. Our method is based on the concept ofStochastic Gradient Estimation [9], a stochastic variant of finite differentiation that allows for estimating gradients without a differentiable framework. However, akin to finite differentiation, this method does not scale to high-dimensional
problems: the more dimensions, the noisier the gradient estimates, the more optimization steps are required. Our idea isto cut down the dimensionality by estimating gradients on aper-pixel basis rather than on the whole image. Indeed, thenumber of parameter contributing to a given rasterized pixel
is of tractable dimensionality, regardless of the total numberof parameters in the scene. This idea yields a method tomake an existing rasterizer differentiable. Namely:

1. Introduction
Motivation for differentiable rendering. A differentiable renderer is a rendering engine that computes a 2D
image for a given 3D scene and has, in addition, the ability to provide gradients for the 3D scene parameters via
backpropagation through the rendering calculations. The
benefits of having these gradients is that it makes possibleto optimize the 3D scene parameters to obtain a target 2D
image via gradient descent. This allows for many applications such as object placement [1], object reconstruction [2],[3], model simplification [4], material estimation [5], etc.

• It is simple to implement. Our base differential rasterization component consists of adding ID/UV-buffers to
the existing raster targets and two compute shaders.
• It keeps the workflow self-contained by bringing thebenefits of differentiable rasterization to an existing
conventional rasterizer without requiring external dependencies.
• It is cross-platform since it uses only conventionalgraphics API functionalities. This is a significant bonus
point for adoption given that existing differential rendering solutions are bound to vendor-specific hardware
and/or software.

Objective. We assume that a rasterization engine is available and we wish to use differentiable rendering to optimize
assets for their final in-engine rendering. Ideally, the solution should keep the workflow simple and self-contained, i.e.
without using other tools and dependencies than the engine
itself. In this context, implementing a renderer from scratch
within a differentiable frameworks such as Dr.JIT [6] or• It is efficient and scales well in scene complexity. We
optimize scenes with 1M+ parameters in seconds on acustomer GPU.
• It covers multiple use cases. We estimate gradients formeshes, displacement mapping, Catmull-Clark subdivision surfaces [10], semi-transparent geometry, physically based materials, 3D volumetric data and 3D Gaussian Splats [11].

1Intel Labs, Grenoble, FRANCE. Correspondence to: Thomas
Deliot <thomas.deliot@intel.com>.
Published at the 2 nd Differentiable Almost Everything Workshop
at the 41 st International Conference on Machine Learning, Vienna, Austria. July 2024. Copyright 2024 by the author(s).

1Differentiable Raterizer using Stochastic Gradient Estimation
R+ ϵ1parameters

θ2
θ3
θ4
θ5
θ6
θ7− ϵ2− ϵ3+ ϵ4+ ϵ5+ ϵ6− ϵ7

θ8+ ϵ8θs⊙ϵ2perturbed parameters

random perturbation

θ1

θ±s⊙ϵ

stochastic gradients

PG

rasterized pixel
Iw,h (θ ± s ⊙ ϵ)target pixel
Iw,h∂fw,h∂θpixel errorfw,h (θ ± s ⊙ ϵ)Figure 1: Overview of our differentiable rasterizer. The first compute shader (P) perturbs the scene parameters before
they are rasterized (R). The second compute shader (G) accumulates the error differences, which provide a gradient estimate.
The key point of our approach is that it accumulates the contribution of a pixel (in red in the images) only in its contributing
parameters (in red in the vectors).

2. Per-Pixel Stochastic Gradient Estimation

Derivation. We assume that the loss function we use (thel2 error in our applications), is a sum of per-pixel errors:
Xf (θ) =fw,h (θ),(2)

Our per-pixel method builds on Stochastic Gradient Estimation to differential rasterization. There, the objective is tooptimize a 3D scene such that a rasterized 2D image fromthis scene matches a target image. To do this, we need toestimate the gradients of the rasterization computations.

(w,h)∈W ×H

and the gradient can be defined in the same way:∂f=∂θi

2.1. Stochastic Gradient EstimationdIn this context, the vector θ ∈ R represents a 3D scene
defined by a set of parameters, typically geometry, textures,etc. A rasterizer computes a 2D image I(θ) using this3D scene. Finally, the objective function f (θ) is the error
between the rasterized image I(θ) and a target image I.

=f (θ + s ⊙ ϵ) − f (θ − s ⊙ ϵ).
2 si ϵi(w,h)∈W ×H∂fw,h.
∂θi(3)

Note that if the parameter θi is not implicated in the compu∂f= 0. We can thus rewrite
tation of pixel (w, h) then ∂θw,hi
the gradient with a sparse sum where only impacted pixels
contribute:

Our goal is to compute the gradient of the loss function w.r.t.
∂fthe input parameters: ∂θ. Instead of relying on automaticidifferentiation, we estimate the gradient stochasticallyd∂f
∂θiX

∂f=
∂θiX

(w,h) impacted by θi∂fw,h.
∂θi(4)

By applying the estimator of Equation (1) to Equation (4)we obtain the stochastic gradient estimate our method isbased on:(1)

where ϵ = (ϵ1 , .., ϵd ) is a user-defined perturbation magnitude vector s = (s1 , .., sd ) is a random sign vector,si ∈ {−1, +1}, where each sign has equal probability. Wenote s ⊙ ϵ the element-wise product of both vectors.

d∂f=
∂θi

X(w,h) impacted by θifdw,h.
∂θi(5)In Section 3, we show how to implement this equation witha rasterizer and compute shaders.

2.2. Per-Pixel Formulation
The stochastic gradient estimate of Equation (1) is noisy,
especially in a high-dimensional parameter space (Figure 2,first row). In our use case, the error of a pixel contributes
to every parameter, even if this parameter is never used tocompute the pixel’s value. Hence, every parameter receives
noisy gradients from every pixel, increasing the variance ofthe estimator. We propose to compute gradients per-pixel toalleviates this problem and makes the method scalable.

3. Turning a Rasterizer Differentiable
We turn a rasterizer based renderer (denoted R) to a differentiable one with 2 compute shaders: a P (perturbation) anda G (gradient) shaders which implements Equation (5) We
provide an overview of our pipeline in Figure 1 and detailits three steps next.
2

Differentiable Raterizer using Stochastic Gradient Estimation

3.1. Perturbation (Compute shader P)

4. Results

First, we launch a compute shader that execute Algorithm 1
over d threads (one thread per scene parameter). The shadercomputes the perturbed scene parameters θ + s ⊙ ϵ andθ − s ⊙ ϵ and store them in GPU memory. Its main ingredient is the generation of the random sign vector s via
randomsign(), which we implement with a random hashfunction [12].

We provide results of our method and compare it to a vanilla
stochastic difference approach. For all our example, we
modified Unity to apply our compute shaders and the Adam
optimizer (as a compute shader) once the gradients are computed. All our results are timed on an NVIDIA 4090 GPU.
Optimizing Triangle Meshes. In Figure 2 we illustrate
the difference between the full-image approach of Equation (1) and the per-pixel approach of Equation (5). In thisexperiment, each triangle is represented by 12 parameters (3vertices + 1 RGB color). Using 100K triangles, we obtain atotal of 1228800 parameters. Optimizing with the full-image
error is impractical with that numbers of parameters. The
per-pixel gradients permit a quick convergence.

Algorithm 1 Compute shader P (perturbation)
Require: thread ID iload θi , ϵisi = randomsign()
store si ϵi , θi + si ϵi , θi − si ϵi▷ load 2 float▷ hash function [12]▷ store 3 float

3.2. Rasterization (R)init.

full-image error
Eq. (1)

Then we utilize the perturbed parameter to rasterize thescene twice. First, using parameters θ + s ⊙ ϵ and thenusing parameters θ − s ⊙ ϵ. We thus obtain two images:I(θ + s ⊙ ϵ), and I(θ − s ⊙ ϵ). Those are used to evaluate
the gradient in the next shader.

20 iterations 100 iterations 200 iterations 1000 iterations(1.2 sec)(2.9 sec)(5.8 sec)(28.9 sec)

per-pixel error
Eq. (5)3.3. Gradients Estimate (Compute shader G)
Finaly, we launch a compute shader that execute Algorithm 2 over W × H threads (one thread per pixel). Theshader computes the pixel errors fw,h (θ + s ⊙ ϵ) andfw,h (θ − s ⊙ ϵ) between the perturbed-scene images
I(θ + s ⊙ ϵ) and I(θ − s ⊙ ϵ) and the target image I. Oncethese errors are available, they provide the gradient estimate
for each parameter i contributing to pixel (w, h) following
Equation (5). We add the result to the gradient estimate
using an AtomicAdd operation to avoid interferences between multiple threads (pixels) adding simultaneously their
gradient contribution to the same parameter. Note that thecritical point of this algorithm is the ability to loop over eachparameter i contributing to pixel (w, h): R has to output an
ID buffer (else, we get it from another rasterization step).

Figure 2: Validation of the per-pixel formulation. In thisexperiment, we optimize triangles soups to match a 2D image. The full-imagevariant implements Equation (1) wherethe error over the whole image contributes to every parameter and the per-pixel approach implements Equation (5).
Optimizing various primitives. Figures 3 showcase optimizing subdivision surfaces [10], PBR textures, volumes,
and Gaussian Splats [11]. For Catmull-Clark subdivision
surfaces [10], we optimize the control mesh and tessellate
it on the fly in the rasterizer [13]. We write the control
mesh’s triangle in the ID buffer. We additionaly optimize
the displacement and normal maps and further store UV
coordinates. For physically based shading, we optimize
roughness, metallicity, albedo, height and normal maps.
Similary to the subdivision surface case, we need to store
the UV coordinate in the ID buffer. For 3D Gaussian splats,
we rasterize transparent quads and use a front-to-back sorting to output a deep ID buffer. To improve the results, we
implement an additional resampling and a splat subdivisioncompute shader executed after each gradient descent, following Kerb et al. [11]. Lastly, for 3D volumes, we rely on anadditional ray marching phase in compute shader G to splat
the gradients and avoid to store all the voxels coordinates.

Algorithm 2 Compute shader G (gradient)
Require: thread ID (w, h)load Iw,h (θ + s ⊙ ϵ), Iw,h (θ − s ⊙ ϵ), Iw,h ▷ load 3
float3 (3× rgb)
2fw,h (θ + s ⊙ ϵ) = ∥Iw,h − Iw,h (θ + s ⊙ ϵ)∥
2fw,h (θ − s ⊙ ϵ) = ∥Iw,h − Iw,h (θ − s ⊙ ϵ)∥for each parameter θi contributing to pixel (w, h) do ▷implementation of Equation (5)load si ϵi ▷ load 1float
fw,h (θ+s⊙ϵ)−fw,h (θ−s⊙ϵ)∂fAtomicAdd ∂θi ←▷2 s i ϵi
atomic add 1 float
end for3

Differentiable Raterizer using Stochastic Gradient Estimation
our method
init., 24K vertices

our method
3 sec.

our method
30 sec.

referenceno timeour method
init., 50K triangles

our method
35 sec.

our method
200 sec.

referenceno timeour method
init., 1K splats

our method
3 sec., 2K splats

our method
53 sec., 128K splatsreferenceno timeour method
init., 1283 voxels

our method
5 sec.

our method
32 sec.

referenceno timeFigure 3: Optimizing 3D Assets. We showcase different application of our method. We can optimize either subdivision
surfaces, PBR materials, 3D Gaussian Splats, and voxels.

5. Conclusion

However, we do not position our method as a replacementfor other state-of-the-art differentiable rasterizers. We aim
to bring the benefits of differentiable rasterization to an audience that already possesses a rasterization engine and hasworkflow or platform constraints that prevent using existing
differentiable rasterizers. Our method makes it possible toenjoy the possibilities of differentiable rasterization within
the existing engine.

We have proposed a method to transform a nondifferentiable rasterizer into a differentiable one. Our experiments have shown that our transformed rasterizer supports
the same applications as state-of-the-art differentiable rasterizers without critical performance or qualitative penalty.
We successfully used it to optimize meshes, subdivision
surfaces, physically based materials, volumes, and 3DGS.
4Differentiable Raterizer using Stochastic Gradient Estimation

References
[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]
[10]

[11]

[12]

[13]

H. Rhodin, N. Robertini, C. Richardt, H.-P. Seidel, andC. Theobalt, “A versatile scene model with differentiablevisibility applied to generative pose estimation,” in Proceedings of the IEEE International Conference on Computer
Vision, 2015, pp. 765–773.
H. Kato and T. Harada, “Learning view priors for singleview 3d reconstruction,” in Proceedings of the IEEE/CVF
conference on computer vision and pattern recognition,
2019, pp. 9778–9787.
S. Wu, C. Rupprecht, and A. Vedaldi, “Unsupervised learning of probably symmetric deformable 3d objects fromimages in the wild (invited paper),” IEEE Transactions on
Pattern Analysis and Machine Intelligence, vol. 45, no. 4,
pp. 5268–5281, 2023.
J. Hasselgren, J. Munkberg, J. Lehtinen, M. Aittala, andS. Laine, “Appearance-driven automatic 3d model simplification.,” in EGSR (DL), 2021, pp. 85–97.
D. Azinovic, T.-M. Li, A. Kaplanyan, and M. Nießner, “Inverse path tracing for joint material and lighting estimation,”in Proceedings of the IEEE/CVF conference on computervision and pattern recognition, 2019, pp. 2447–2456.
W. Jakob, S. Speierer, N. Roussel, and D. Vicini, “Dr.jit: A
just-in-time compiler for differentiable rendering,” Transactions on Graphics (Proceedings of SIGGRAPH), vol. 41,
no. 4, 2022.
S. Bangaru, L. Wu, T.-M. Li, et al., “Slang.d: Fast, modular
and differentiable shader programming,” ACM Transactions on Graphics (SIGGRAPH Asia), vol. 42, no. 6, pp. 1–28, Dec. 2023.
S. Laine, J. Hellsten, T. Karras, Y. Seol, J. Lehtinen, andT. Aila, “Modular primitives for high-performance differentiable rendering,” ACM Transactions on Graphics, vol. 39,
no. 6, 2020.
M. Fu, “Stochastic gradient estimation,” Technical report,
2005.
E. Catmull and J. Clark, “Recursively generated b-spline
surfaces on arbitrary topological meshes,” Computer-Aided
Design, vol. 10, no. 6, pp. 350–355, 1978.
B. Kerbl, G. Kopanas, T. Leimkuehler, and G. Drettakis,“3d gaussian splatting for real-time radiance field rendering,”
ACM Trans. Graph., vol. 42, no. 4, 2023.
M. Jarzynski and M. Olano, “Hash functions for gpu rendering,” Journal of Computer Graphics Techniques (JCGT),
vol. 9, no. 3, pp. 20–38, Oct. 2020, ISSN: 2331-7418.
J. Dupuy and K. Vanhoey, “A halfedge refinement rulefor parallel catmull-clark subdivision,” Computer Graphics
Forum, vol. 40, no. 8, pp. 57–70, 2021.

5