Distributions for Compositionally Differentiating Parametric Discontinuities

Jesse Michel 1 Kevin Mu 2 Xuanda Yang 3 Sai Praveen Bangaru 1 Elias Rojas Collins 1 Gilbert Bernstein 2
Jonathan Ragan-Kelley 1 Michael Carbin 1 Tzu-Mao Li 3

Abstract
Computations in computer graphics, robotics, andprobabilistic inference often require differentiating integrals with discontinuous integrands. Popular differentiable programming languages do notsupport the differentiation of these integrals. To
address this problem, we extend distribution theory to provide semantic definitions for a broadclass of programs in a programming language,
Potto. Potto can differentiate parametric discontinuities under integration, and it also supports
first-order functions and compositional evaluation. We formalize the meaning of programs using denotational semantics and the evaluation ofprograms using operational semantics. We provecorrectness theorems and show that the operational semantics is compositional, enabling separate compilation and overcoming compile-timebottlenecks. Using Potto, we prototype a differentiable renderer with separately compiled shaders.

(a) Depth shader(b) Depth shader deriv.

(c) Thresh. Lambert shader(d) Thresh. Lambert deriv.

Figure 1. Potto can separately compile and swap between different
shaders, making the design process tractable. The sparsity patternof the derivative shaders are displayed—the derivative shader isdisplayed red, say, if the derivative is nonzero in the red channel.

Such discontinuous functions arise in computer graphics
and vision (Li et al., 2018), robotics (Hu et al., 2020; Bangaru et al., 2021), and probabilistic inference (Lee et al.,2018). In computer graphics, discontinuities arise fromobject boundaries, occlusion, and sharp changes in color.
Robot controllers and computer simulations often model
contact, which introduces discontinuities. In probabilistic
inference, the models often have discontinuities, e.g., a probabilistic simulation of a controller that regulates room temperature is discontinuous due to switching a heater on/off.

1. Introduction
The advent of deep learning frameworks supporting automatic differentiation—the automated computation of derivatives of a function given just the definition of the function
itself—within general-purpose programming languages has
opened up new avenues to build large-scale applications fordifferentiable programming, including in deep learning, optimization, and uncertainty quantification. However, thesetechniques have traditionally been limited to continuous
processes, excluding a variety of natural phenomena thatare typically modeled as discontinuous functions.

Consider a program that models contact, such as a simulation or animation of a robot walking (Stengel, 1994; Witkin& Kass, 1988). Such a program might compute the inteR1
gral 0 [x < θ] dx, where the Iverson bracket [P ] is one if
P is true and zero otherwise. The parameter θ represents
the time the ball hits the wall and x < θ is a parametricdiscontinuity. Although the derivative with respect to θ
R1
is Dθ 0 [x < θ] dx = [0 < t < 1], popular differentiable
programming languages return zero because they do not account for parametric discontinuities (Bradbury et al., 2018;
Paszke et al., 2019).

*

Equal contribution 1 CSAIL, MIT, 32 Vassar Street in Cambridge, Massachusetts 02139, USA 2 Computer Science and Engineering, University of Washington, University of Washington,
3800 E Stevens Way NE, Seattle, WA 98195, USA 3 Computer Science and Engineering, University of California San Diego, 9500
Gilman Drive, La Jolla, CA 92093, USA. Correspondence to: Jesse
Michel <jmmichel@csail.mit.edu>.
Published at the Differentiable Almost Everything Workshop of the
40 th International Conference on Machine Learning, Honolulu,
Hawaii, USA. July 2023. Copyright 2023 by the author(s).

Teg (Bangaru et al., 2021) automatically differentiates parametric discontinuities, providing an unbiased estimation of1

Distributions for Compositionally Differentiating Parametric Discontinuities(a) Scene(b) Shaders change how light attenuates(c) The derivative w.r.t. light position
Figure 2. We implement a differentiable renderer in Potto and render a scene depicting a single line of light pointing diagonally downward(a) with different light attenuations (b). Swapping between different shaders is an example of how designers modify a scene for an artistic
effect. In Potto, programs and their derivatives (c) can be separately compiled and composed to efficiently swap among shaders.

3. Case Studya program with integrals. However, their approach requires
global program transformations, leading to long compile
times. Our programming language, Potto, avoids these problems leading to an 81x and 441x speedup in compile timein an application to image stylization.

We motivate our language by implementing a differentiable
renderer (de La Gorce et al., 2011; Loper & Black, 2014; Liet al., 2018; Zhao et al., 2020; Li et al., 2020). Applications,
ranging from autonomous driving and robotics to CGI, usedifferentiable renderers to e.g. recognize the 3D shapes ofcars, signs, and pedestrians, reconstruct the 3D scene for the
robot to interact with, and motion capture actors’ faces.

Moreover, global program transformations prevent separate
compilation—the ability to compose compiled code snippets. For a renderer, this means that when an artist tweaks
the scene, such as changing the color of an object (swap
between shaders), the whole scene must be recompiled.

A renderer is a program which takes in the geometry andcolor of each object in the scene and outputs an image.
Renderers are built out of programs called shaders. A differentiable renderer computes the change in the color of a
pixel with respect to the change of a parameter, such as thelocation or color of an object.

Figure 1 shows the result of using a renderer implemented
in Potto to swap between two shaders applied to the same
triangle in 3D space. Both the triangle and shaders can beseparately compiled and composed in Potto, but not in Teg.

Figure 2 depicts images generated by a differentiable renderer implemented in Potto. The scene contains a single
object—a line of light shining diagonally downward. The
color of a single pixel is the average of light within the pixel
area:
Z 1Z 1s(x, y, c)[x + y + c ≥ 0] dxdy (1)f (c, s) =2. Related Work
Most automatic differentiation methods ignore discontinuities, such as if-else branches, during differentiation (Paszke
et al., 2019; Bradbury et al., 2018). Normally, doing so
is correct almost everywhere (Lee et al., 2020). However,when the problem specification involves integration, as incomputer graphics, robotics, and probabilistic inference,
ignoring the discontinuities produces incorrect gradients.
In practice, these incorrect gradients results in slower convergence or even divergence during optimization (Bangaru
et al., 2021; Li et al., 2018; Lee et al., 2018).

00

We use the convention that the origin is in the top left cornerand the y-axis points down. The half plane [x+y +c ≥ 0] is
the visibility shader for the light. The function s : R3 → R
is the color shader for the light that depends on the point(x, y) and parameter c.

Many works account for parametric discontinuities by handderiving application-specific derivatives (Loper & Black,2014; Li et al., 2018; Hu et al., 2020; Loubet et al., 2019;
Bangaru et al., 2020). Emerging research accounts for parametric discontinuities: Lee et al. (2018) proposes a solution
for affine discontinuities, while Yang et al. (2022); Liu et al.
(2019); Chen et al. (2019); Petersen et al. (2022) support awider class of discontinuities but introduces bias. In contrast,
Teg (Bangaru et al., 2021) directly samples discontinuous
surfaces that arise from differentiation, providing an unbiased, low-variance integral estimation.

The goal is to optimize the parameter c so that the renderer
generates a pixel with color a. We use gradient descent to2minimize the loss function L(c, s) = (f (c, s) − a) with
derivative Dc L(c, s) = 2(f (c, s) − a)Dc f (c, s), where Dc
represents the partial derivative with respect to c. A differentiable renderer computes the derivative Dc f (c, s).
In order to easily iterate on designs, a user should be able toefficiently replace the visibility shader and the color shader
to change the shape and color of the object, respectively.

Subsequent work uses distributional semantics to build
an equational theory, but lacks an operational semantics (Azevedo de Amorim & Lam, 2022).

Differentiating Parametric Discontinuities We give an
informal description of the differentiating parametric discontinuities and give a formal treatment in the following
2Distributions for Compositionally Differentiating Parametric Discontinuities

section. In Figure 2, the image using the constant shaders(x, y, c) = 1 shows a half plane and its derivative is a line
along the boundary of the half plane. Since the derivativeis the change resulting from perturbing the parameter c inf (c, s). The half-plane shifts diagonally downward, making
the boundary the only region with
√ a non-zero derivative.
−1
The linear shader s(x, y, c) = ( 2(x√ + y + c) + 2) andthe quadratic shader s(x, y, c) = ( 2(x + y + c) + 2)−2
have the same boundary contribution to the derivative, butalso have a non-zero interior derivative.

The following code snippet shows a differentiable renderer:
1
2
3
4
5
6
7

# renderer.po
from half_plane import cond
def renderer(c, shader)integral ([0,1],[0,1])((shader (x,y)) c)*(((cond (x, y)) c)?1:0) d(x,y)
drenderer = deriv(renderer)

Derivatives, integrals, and discontinuities Line 2 imports the invertible, differentiable function (diffeomorphism)\(x,y).\c.(x+y+c,x-y+c) from the half_plane.
diffeo file, where the \(x, y) notation declares ananonymous function with parameters x and y. For instance,in Python, we write lambda x, y:... to declare an
anonymous function. It takes in variables of integration x, yand variable c, and returns an affine combination of the three.
Line 4 declares the integral to be estimated (Equation 1).
The first argument specifies that x, y each range from zero toone, the second argument is the integrand, and the d(x,y)
declares that x and y are variables of integration. In Line5, the deriv operator specifies the dual number derivativeof drenderer. The derivative drenderer takes in pairsof an input and an infinitesimal perturbation to that input
and produces a pair of the outputs for the evaluation of therenderer and its derivative.

The derivative of the renderer decomposes into the interior
and boundary contributions:Z
Dc f (c, s) = Dc s(x, y, c)[x + y + c ≥ 0]{z}
S|interior(2)+ s(x, y, c)δ(x + y + c) d(x, y).
{z}|boundary

The Dirac delta distribution δ in the integrand can be thoughtof as zero everywhere and approaching infinity along theline x + y + c = 0. This shows up as the diagonal yellow
line in the derivative of all three shaders.
Automatic Differentiation of Parametric Discontinuities
A naı̈ve implementation would discretize the integral to a
sum and use automatic differentiation to compute the derivative of the discretization. The resulting program would
approximate the interior term of Equation 2, but ignore thesecond due to the Dirac delta distribution, producing an
incorrect result.

1
2
3
4
5
6

Recent work accounts for both terms by introducing an integral primitive to a differentiable programming language,
Teg (Bangaru et al., 2021). Teg performs a series of codetransformations such as distributing multiplication over addition, performing a global change of variables, and applying
a global symbolic rule to eliminate Dirac delta terms.

# color_shaders.po
dconst_shader = deriv(\(x,y).\c.1)
dlin_shader = deriv(\((x,y).\c.
1/(sqrt(2)*(x+y+c)+2))
dquad_shader = deriv(\(x,y).\c.
1/(sqrt(2)*(x+y+c)+2)ˆ2)

The three shaders are first-order functions that model how
quickly light attenuates. The const_shader corresponds
to no attenuation—intensity is invariant to the distance fromthe light. While in lin_shader and quad_shader, theattenuation is linear and quadratic, respectively.
1
2This global rewriting approach fundamentally relies upon
having the syntax of the whole expression. These rules are
not compositional and do not allow for separate compilation.
As a result, they are a barrier to performance.

34
5

For example, separately compiling shaders is criticalin video games, animation, photo-editing software, andcomputer-generated imagery, allowing users/designers to
see multiple variants of a scene or video without requiring
compiling the whole scene again. Likewise, users often run
a differentiable renderer with multiple shaders.

# main.po
from color_shaders import dconst_shader,
dlin_shader, dquad_shaderfrom renderer import drenderer
for dshader in [dconst_shader,
dlin_shader, dquad_shader]:print(drenderer(dc=(-2,1), dshader))A differentiable renderer In main.po, we compose thederivative of the renderer with the derivative of each of thecolor shaders to compute the color of a single pixel. We
evaluate the resulting expression at a base point −2 with
infinitesimal 1, producing a number representing the derivative. We can use the differentiable renderer in a derivativebased optimization procedure, such as gradient descent, tofind the value of c that results in a pixel that is most similar
to the pixel provided.

Our differentiable programming language, Potto, has anintegral primitive and can separately compile programs. We
implement the renderer specified in Equation 1 and demonstrate separate compilation by writing the renderer (andvisibility shader) and color shaders in separate files.
3Distributions for Compositionally Differentiating Parametric Discontinuities

Figure 3. A bar chart, where smaller is better, comparing Potto to Teg on a rendering task for image stylization. Potto is so much faster incompile time that the bars on not visible. Compile time was the bottleneck in Teg.

Separate compilation Potto separately compiles (calculates the derivative of) the renderer and color shaders. As aresult, it only compiles the renderer and each of the shadersonce. In contrast, Teg must compile the renderer threetimes—once for each of the shaders.

tics, providing unbiased estimates of the denoted program.
The operational semantics is compositional and therefore
supports separate compilation.

4. Potto programming language

Implementation and applications We implement a prototype system, Potto, and use it to build a renderer withmultiple shaders. We show that the renderer supports separate compilation, enabling interactive workflows that would
otherwise be computationally intractable.

We introduce a differentiable programming language, Potto,that is the first with an integration primitive, to have firstorder functions, and to support compositional evaluation. A
first-order function takes in values of base types and produces values of base types. For instance, in Potto, a user
can specify a function of type real to real, but not a functionthat that takes in another function as an argument. Compositional evaluation means that a compiler can evaluate
programs independently, passing values between modules.

5. Empirical Results
Figure 1 depicts a differentiable ray tracing renderer ofa triangle tilted in 3D space, colored using two different
shaders that can be separately compiled. These toon shaders
are 1) a z-depth shader that assigns color to objects based on
their distance from the camera, and 2) a thresholded Lambertshader that models the reflectance of a matte surface undera point light in front of the triangle (Lake et al., 2000).

We present a first-order functional language for programming with distributions. The language has constants c,variables x, sums and products of arbitrary terms t1 + t2
and t1 · t2 , conditionals if t1 then t2 else t3 , diffeomorphic conditionals if ⌊Ψ⌋(x1 , . . .) then t1 else t2 , integrals
int t d(x1 , . . . , xn ), pairs (t1 , t2 ), and applications of firstorder functions ⌊f ⌋(t). The function Ψ is a differentiable,
invertible function, taking in variables of integration andfree parameters written as (x1 , . . .) and f is a piecewise
differentiable function with piecewise invertible pieces. The
type system (not shown) prevents variables of integrationfrom occurring in the condition t1 of if t1 then t2 else t3 .

We compare the performance of Potto to previous work,
Teg (Bangaru et al., 2021), on a rendering benchmark forimage stylization. The results in Figure 3 show that Potto
outperforms Teg in all areas: compile time, evaluation time,
total time, and AST size. Most striking is the 88x and 441x
compile time speed up of Potto relative to Teg.

6. Conclusion
Denotational semantics Building on our extension of
distribution theory, we present a novel denotational semantics for programs and derivatives of programs. We provea soundness theorem that shows that the derivative of thedenotation is equivalent to the separately defined derivative
denotation under mild conditions.

In our work, we extend the scope of differentiable programming languages to handle integrals and parametric discontinuities. Potto supports compositional evaluation and as aresult, transformations that were once global can be madelocal. This enables separate compilation. We envision that
our theoretical approach and programming language design
will lead to more expressive differentiable programming
languages that better serve application domains including
graphics, robotics, and probabilistic inference.

Operational semantics We present a novel operational semantics that we prove accords with the denotational seman4Distributions for Compositionally Differentiating Parametric Discontinuities

References

Liu, S., Li, T., Chen, W., and Li, H. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. The IEEE
International Conference on Computer Vision (ICCV),Oct 2019.

Azevedo de Amorim, P. H. and Lam, C. Distribution Theoretic Semantics for Non-Smooth Differentiable Programming. arXiv e-prints, 2022.

Loper, M. M. and Black, M. J. OpenDR: An approximate
differentiable renderer. In European Conference on Computer Vision, 2014.

Bangaru, S., Michel, J., Mu, K., Bernstein, G., Li, T.-M.,
and Ragan-Kelley, J. Systematically differentiating parametric discontinuities. In Special Interest Group on Computer Graphics and Interactive Techniques, 2021.

Loubet, G., Holzschuch, N., and Jakob, W. Reparameterizing discontinuous integrands for differentiable rendering.
Special Interest Group on Computer Graphics and Interactive Techniques in Asia, 2019.

Bangaru, S. P., Li, T.-M., and Durand, F. Unbiased warpedarea sampling for differentiable rendering. Special Interest Group on Computer Graphics and Interactive Techniques in Asia, 2020.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J.,
Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga,
L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang,
L., Bai, J., and Chintala, S. PyTorch: An imperative
style, high-performance deep learning library. In Neural
Information Processing Systems. 2019.

Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary,C., Maclaurin, D., and Wanderman-Milne, S. JAX: composable transformations of Python+NumPy programs,
2018.
Chen, W., Gao, J., Ling, H., Smith, E., Lehtinen, J., Jacobson, A., and Fidler, S. Learning to predict 3d objects
with an interpolation-based differentiable renderer. In Advances In Neural Information Processing Systems, 2019.

Petersen, F., Goldluecke, B., Borgelt, C., and Deussen,
O. GenDR: A Generalized Differentiable Renderer. In
IEEE/CVF International Conference on Computer Vision
and Pattern Recognition (CVPR), 2022.

de La Gorce, M., Fleet, D. J., and Paragios, N. Model-based
3D hand pose estimation from monocular video. IEEE
Trans. Pattern Anal. Mach. Intell., 2011.

Stengel, R. F. Optimal control and estimation. Courier
Corporation, 1994.
Witkin, A. and Kass, M. Spacetime constraints. Special
Interest Group on Computer Graphics and Interactive
Techniques, 1988.

Hu, Y., Anderson, L., Li, T.-M., Sun, Q., Carr, N., RaganKelley, J., and Durand, F. DiffTaichi: Differentiable
programming for physical simulation. International Conference on Learning Representations, 2020.

Yang, Y., Barnes, C., Adams, A., and Finkelstein, A. Aδ:Autodiff for discontinuous programs - applied to shaders.
In Special Interest Group on Computer Graphics andInteractive Techniques, 2022.

Lake, A., Marshall, C., Harris, M., and Blackstein, M. Stylized rendering techniques for scalable real-time 3d animation. In International Symposium on Non-Photorealistic
Animation and Rendering, pp. 13–20, 2000.

Zhao, S., Jakob, W., and Li, T.-M. Physics-based differentiable rendering: From theory to implementation. In
SIGGRAPH Courses, 2020.

Lee, W., Yu, H., and Yang, H. Reparameterization gradientfor non-differentiable models. In Neural Information
Processing Systems, 2018.
Lee, W., Yu, H., Rival, X., and Yang, H. On correctness of
automatic differentiation for non-differentiable functions.
In Neural Information Processing Systems, 2020.
Li, T.-M., Aittala, M., Durand, F., and Lehtinen, J. Differentiable Monte Carlo ray tracing through edge sampling.
Special Interest Group on Computer Graphics and Interactive Techniques in Asia, 2018.
Li, T.-M., Lukáč, M., Michaël, G., and Ragan-Kelley, J. Differentiable vector graphics rasterization for editing andlearning. Special Interest Group on Computer Graphics
and Interactive Techniques in Asia, 2020.
5