Structural foundations for differentiable programming

This dissertation supports the broader thesis that categorical semantics is a powerful tool to study and design programming languages. It focuses on the foundational aspects of differentiable programming in a simply typed functional setting. Although most of the category theory used can be boiled down to a more elementary presentation, its influence was certainly key in obtaining the results presented in this dissertation. The conciseness of certain proofs and the compactness of certain definitions and insights were made easier thanks to my background in category theory.
Backpropagation is the key algorithm that allows fast learning on neural networks. It enabled some of the impressive recent advancements in machine learning. With models of increasing complexity, data structures equally complex are required, which calls for the ability to go beyond standard differentiability. This emerging generalization was coined as differentiable programming. The idea is to allow users to write expressive programs representing (a generalization of) differentiable functions, whose gradient computation can be automated using automatic differentiation. In this dissertation, I lay some foundations for differentiable programming. This is done in three ways.
Firstly, I present a simple higher-order functional language and define automatic differentiation as a structure-preserving program transformation. The language is given a denotational semantics using diffeological spaces, and it is shown that the transformation is correct, i.e. that AD produces programs that do compute gradients of the original programs, using a logical relations argument.
Secondly, I extend the language from the previously described chapter to introduce new expressive program constructs such as conditionals and recursion. In such a setting, even first-order programs may represent functions that need not be differentiable. I introduce better-fitted denotational semantics for such a language and show how to extend AD to such a setting and what guarantees about AD now hold. This extended language models the more realistic needs in expressiveness that can be found in the literature, e.g. in modern probabilistic programming languages.
Thirdly, I present detailed applications of the developed theory. I first show a general recipe for extending AD to non-trivial new types and new primitives. I then show how the guarantees about AD are sufficient for usage in certain applications, such as the change of variable formula of stochastic gradient descent, but how it may not be sufficient, for instance, in simple gradient descent. Finally, more applications in the specific context of probabilistic programming are explored. First, a denotational proof that the trace semantics of a probabilistic program is almost everywhere differentiable is given. Second, a characterization of posterior distributions of probabilistic programs valued in Euclidean spaces is obtained: they have densities with respect to (w.r.t.) some sum-of-Hausdorff measure on a countable union of smooth manifolds.
Overall, these contributions give us better insights into differentiable programming. They form a foundational setting to study the differentiability-like properties of realistic complex programs, beyond usual settings such as differentiability or convexity. They give general recipes to prove some properties of such programs and modularly extend automatic differentiation to richer contexts with new types and primitives.

Keywords: functional programming
semantics
differentiable programming
category theory
automatic differentiation
Advisor: Staton, S