Daphne: Multi-Pass Compilation of Probabilistic Programs into Graphical Models and Neural Networks
Authors: Christian Dietrich Weilbach
Frank Wood
Venue: Accepted by TMLR ()

Abstract: Daphne is a probabilistic programming system that provides an expressive syntax to denote a large, but restricted, class of probabilistic models. Programs written in the Daphne language can be compiled into a general graph data structure of a corresponding probabilistic graphical model with simple link functions that can easily be implemented in a wide range of programming environments. Alternatively Daphne can also further compile such a graphical model into understandable and vectorized PyTorch code that can be used to train neural networks for inference. The Daphne compiler is structured in a layered multi-pass compiler framework that allows independent and easy extension of the syntax by adding additional passes. It leverages extensive partial evaluation to reduce all syntax extensions to the graphical model at compile time.

Published in Transactions on Machine Learning Research (03/2025)
Daphne: Multi-Pass Compilation of Probabilistic Programs into Graphical Models and Neural Networks
Christian Weilbach
stttbach@csuub..co
Deportment of Computer Science University of British Columbia
Frank Wood
fwood sobod&@cs.uc.co.
Deportment of Computer Science University of British Columbia
Reviewed on OpenReview: https: // openreview. net/ formm? id= XXXX
Abstract
Daphne is a probabilistic programming system that provides an expressive syntax denote large, but restricted, class of probabilistic models. Programs written in the Daphne language can be compiled into general graph data structure of corresponding probabilistic graphical model with simple link functions that can easily be implemented in wide range of programming environments. Alternatively Daphne can also further compile such a graphical model into understandable and vectorized PyTorch code that can be used to train ncural nelworks for inference. The Daphne compiler is structured in a layered multi-pass compiler framework that allows independent and easy extension of the syntax by adding additional passes. It leverages extensive partial evaluation to reduce all syntax extensions to the graphical model at compile time.
1 Introduction
Probabilistic modeling is fundamental modern machine learning and statistics. The recent rise of gen- erative AT has placed many deep learning applications within approximate Bayesian modeling frame- work (Brown et al., 2020; Rombach et al., 2022). At the same time, structured probabilistic programming systems are rapidly gaining traction in statistics (Strumbelj et al., 2024). These systems allow users to specify models and inference problems in programming languages that balance expressivity and tractability, While furing-complets probabilistic programming languages offer universal expressivity, they can make inference intractable. In contrast, more restricted languages can still capture large classes of problems while enabling more efficient inference.
An important class of probabilistic programming languages those that compile to representations op- timized for powerful inference algorithms, such as Stan (Carpenter et al., 2017). These systems transform expressive input syntax into a lower-level model that can be evaluated efficiently using the numerical primitives of an inference engine. Additionally, an ideal probabilistic programming system should allow users to extend its syntax and should be embedded in a rich programming environment that includes both probabilistic modeling primitives and modern machine learning tools. This combination of properties makes such a system particularly well suited for tcaching and rescarch.
Following these design principles, introduce Daphne.' Among the probabilistic programming systems are aware of, Daphne stands out due to its expressive higher-order syntax and its extensive use of partial evaluation in its compiler (Section 4).
'https://github.. http://itttuub.ccmmls.aaapa wtt.:/cccppppppccee.eee hham:plllligroo//apheee

---

Published in Transactions on Machine Learning Research (03/2025)
2 Motivation for Daphne
To demonstrate the use cases of Daphne, highlight two applications in Figure 1 and Figure 2. The implementation details of these papers are not covered here, they are shown to highlight the benefit of having Daphne an inference problem specification interface to succinctly describe probabilistic graphical models. The graphical models are then used in both cases to sparsify neural network architectures for amortized inference learning tasks. The language is designed to facilitate extensions that extract rich, yet easy-to-process, compute graphs from probabilistic programs for such applications.
Zn L(S;1)
81 ~ ((-2:1)
(t)
2s ~ M(tanh(Ko
2.8)
~ N/m (xm.))))
~ (),
N(taa((s 0.)0
1x((t))
(()
n ~ M(z;0.1)
ba(t")
dx((t)
x(0)
Figure 1: Translation from Daphne program (here written in statistical syntax) into its compute graph, its faithful inverse, and then into a sparsity mask of multilayer perceptron (MLP) guiding a continuous normalizing flow (Grathwohl et al., 2018). The resulting sparsified MLP can be trained and exccuted more efficiently. Figure taken from (Weilbach et al., 2020).
graphical model
attention mask
diffusion process
R
A
REJ
A
Cak
E
permutation invariances for
Figure 2: Integration of a graphical model extracted from Daphne into a transformer-based diffusion model trained for amortized inference tasks. On the left, the graphical model extracted from Daphne (or manually specified), its adjacency matrix structures the sparse self-attention transformer layers guiding a diffusion process. Figure taken from (Weilbach et al., 2023b).
Daphne provides these graphical model translations in a modular translation process that cases adjustments to the specification language, and the properties that can be captured. As an example of an extension that would be easy add Daphne, the permutation invariances projected in Figure could also be extracted automatically, since they statically induced by independence of link functions from plate indices (Weilbach al., 2023b).² Both papers require the translation are trained with traditional parametric variational families. The ability of Daphne to reduce an expressive language to a simple data structure of graphical model is particularly useful for such integrations.
graphical model before they
2This was not done for this paper the neural network was directly implemented PyToch.
2

---

Published in Transactions on Machine Learning Research (03/2025)
3 Graphical Probabilistic Programming Language
A graphical probabilistic programming language (GPPL) a language in which all random variables can identified without evaluating the program's 's stochastic expressions- Stan (Carpenter al., 2017) being a prominent example. Such a language enables loops and control flow to be unrolled at compile time, explicitly representing all random variables in traditional probabilistic graphical model (PGM) (Koller & Friedman, 2009).
This contrasts with general probabilistic programming languages (PPLs), which support loops and recursion constructs that cannot. be simplified before execution e-g., when a loop or recursion termination condition depends on a random variable sample. van de Meent et al. (2018) distinguish these as first-order (FOPPL) vs. higher-order (HOPPL) languages. In their formulation, FOPPL enforees termination guarantees by requiring bounded loops and prohibiting higher-order functions and recursion avoid unbounded computation. We found these restrictions unnecessarily limiting, leading us to develop GPPLs. Nonetheless, FOPPL. remains valid subset of our GPPL (Section 5).
3.1 Syntax
The core language syntax supported by the compiler defined by the following grammar in Backus-Naur form (BNF):
= symbol (indicating variables)
constant value primitive operation (syntactic atoms)
f ::= procedure
I (let [s] 61 ... 8n enl eb) (if e e2 es) I (ej e ... ex) (sapple e) I (observe e e2) === (defn [s] ... s.] e)
Language 1: Daphne GPPL.
The syntax follows the Clojure programming language (Hickey, 2008; 2020). For readers unfamiliar with Lisp syntax, it can be understood as an extended form JSON, where 'code is data.' " Lisp expressions use list-based meta-syntax, explicitly grouped with parentheses expressions begin with "(" and end with ") As in the grammar rule for e, expressions can contain symbols, making them symbolic expressions (serps).
The language supports lexical binding through the let form and control flow via conditional expressions using if. Function applications follow prefer notation, where the function appears first, followed by its arguments, e.g., (+ 1 2). Here, f must evaluate to either previously defined procedure or primitive operation.
Functions can be defined using defn, enabling code reuse via named procedures. The last expression in a program serves the global entry point, providing access to all defined functions. Like Anglican (Tolpin et al., 2016), Daphne includes probabilistic primitives:
sample to draw from a random variable
observe to condition the variable on data.
3.2 Example: Bayesian Linear Regression
A Bayesian linear regression example in Daphne is shown in Program 2. A normal prior is assigned to slope and bias, and the function reduce itcrates over six (x. data pairs, applying the normal likelihood function observe-data before returning the posterior parameters. Unlike van de Meent al. (2018), Daphne allows recursion and higher-order functions, demonstrated in our simple implementation of reduce.
3

---

Published in Transactions on Machine Learning Research (03/2025)
(defn reduce acc s] (if (> (count s) 0)
(reduce (f acc (first s)) (rest s))
acc))
(defn observe-data [acc data]
(let [slope (first acc)
bias (second acc)
xn (first data)
yn
(second data)
zn (+ (* slope xn) bias)]
(observe (normal zn 1.0) yn)
[slope bias]))
global entry point (let [slope (sample (normal 0.0 10.0))
bias (sample (normal 0.0 10.0))
data [[1.0 2.1] [2.0 3.9] [3.0 5.3]
[4.0 7.7] [5.0 10.2] [6.0 12.9]]]
(reduce observe-data [slope bias] data) [slope bias])
Program 2: Daphne GPPL Example Linear regression
4 Compiler
The Daphne compiler runs ahead-of-time (AOT) before inference is conducted,
The compiler takes as input the full program syntax-includin input data and function definitions, seen in Program 2 and translates it into a dictionary describing a probabilistic graphical model. Daphne provides JSON export for its compiler passes via its command- interface.
4.1 Compiler passes
The compilation process happens in multiple passes, where each pass defines the translation of specific language features into simpler syntax before ultimately leaving only the graphical model representation (van de Meent et al., 2018). This approach facilitates the construction extensible compilers using a layered, modular architecture that processes individual language features separately (Keep & Dybvig, 2013).
The compiler includes the following standard passes:
desugar: Factorizes large let bindings into nested single bindings.
symbolic-simplity: Applies operations on syntactic objects whenever possible. For example, (firat [samplei]) simplifies sample1, since the argument is syntactically represented vector and can be evaluated symbolically.
partial-ewaluation Evaluates parts of the program ahcad of time, as described in Section 4.2.
substitute: An explicit substitution pass that replaces symbols with values while respecting the binding structure of the language.
Daphne can be understood two-level approach (Niclson & Niclson, 1992), generally, staged computation system. The first interpreter translates the GPPL into graphical model representation, and the second interpreter evaluates the Oth-order language expressions graphical model during inference. For clarity, we will refer the first interpreter the compiler and the second the inference runtime.
4

---

Published in Transactions on Machine Learning Research (03/2025)
4.2 Partial Evaluation
Partial evaluation (Jones et al., 1993) is a technique that interprets parts of program ahead of time, effectively implementing compilation by running the interpreter. It evaluates sub-expressions as soon as sufficient information is available. For example, (+ 1 2) can be evaluated to 3 and replacod in the program code since all necessary information is available.
The Daphne language, like Clojure (Hickey, 2020), is functional, stateless Lisp, naturally supporting substitution semantics, which particularly well-suited for partial evaluation (Jones et al., 1993).
Daphne applies partial evaluation using fined-point operalor alongside other compiler passes, continuing iteration until no further simplifications are possible. It follows Clojure's eval semantics, traversing sub- expressions in a top-down manner, using a properly scoped environment, until sub-expression can be fully evaluated. Code that does not depend on random variables can be fully reduced by the partial evaluator, while expressions involving random variables are simplified as much as possible (van de Meent et al., 2018), as shown in the compiled graph for linear regression in Program
Since fixed-point iteration is used, the compiler does not have constant number of passes, and it is non- trivial to determine in advance how many iterations will be required. Furthermore, functions defined using defn naturally allow bounded recursion during partial evaluation, provided that their inputs shrink at each recursive step.
4.3 Graphical model
After iteratively substitating, expanding and reducing the input synlax a graph data structure with vertices :V, adjacency :A, link functions :P and observed nodes :Y remain,
{:V #[sample1 sample2 observe3 observe4 observe5 observe6 observe7 observe8},
:
(sample2 #[observe3 observe4 observe5 observe6 observe7 observe8),
sample1 #(observe3 observe4 observe5 observe6 observe7 observe8)), :P
(sample1 (sample* (normal 0.0 10.0)).
sample2 (sample* (normal 0.0 10.0)),
observe3 (observe* (normal (+ (* sample1 1.0) sample2) 1.0) 2.1),
observe4 (observe* (normal (+ (+ sample1 2.0) sample2) 1.0) 3.9),
observe5 (observe* (normal (+ (+ sample1 3.0) sample2) 1.0) 6.3),
observe6 (observe* (normal (+ (* sample1 4.0) sample2) 1.0) 7.7),
observe7 (observe* (normal (+ (* sample1 5.0) sample2) 1.0) 10.2),
observe8 (observe* (normal (+ (* sample1 6.0) sample2) 1.0) 12.9)}, :Y
[observe3 2.1, observe4 3.9, observe5 5.3,
observe6 7.7, observe7 10.2, observe8 12.9}}
Program 3: Compiled Graphical Model Linear regression
The link functions of :P are expressed in a language that requires neither variable binding nor loop sup- port. For example, (observe* (normal (+ (* sample1 1.0) sample2) 1.0) 2.1) represents an observation statement in its simplified form. This ropresentation can be readily evaluated using NumPy or PyTorch arithmetic and probability primitives in Python, facilitating implementations such as ancestral sampling (Appendix A.1).
Many compilers transform code into A-normal form (ANF) single static assignment (SSA), which closely resemble this graphical representation. As result, Daphne map these computations efficiently to low- level languages that lack garbage collection, particularly CUDA.
*Thi condition is currently not euforced, meaning the compiler may not terminate if violated. Termination chocks could be implemented in the partial evaluator, similar runtime contracts in Racket (Nguyen et al., 2019), but would provide guarantees at translation time this context.
5

---

Published in Transactions on Machine Learning Research (03/2025)
In correspondence with the syntax primitives of the GPPL language, observe* and sample* refer to low-level implementations of sampling and log-probability evalualions, respectively, ensuring computational efficiency in probabilistic inference.
4.4 Inference runtime
Daphne also provides optional runtime support for inference. A differentiable subset of the GPPL, is sup- ported with source to source reverse-mode automatic differentiation (Baydin et al., 2017) and can be plugged into a simple Hamiltonian Monte Carlo (HMC) implementation. normal distributions, arithmetic and a set of scalar functions is provided, but might require additional def- initions of derivatives for wider classes of programs. There is also support for Metropolis within Gibbs sampling. Both of these inplementations have been used for testing and teaching and provide a starting point for further exploration, but should not yet be expected to perform competitively with mature proba- bilistic programming systems. Daphne builds on well tested Anglican (Tolpin et al., 2016) primitives though and and can be extended to a wide range of MCMC methods if needed."
core set of derivatives such as those for
Research with Daphne led to exploration variational inference methods for amortized inference (Weilbach et al., 2020). In this work the compiler furthermore provides a sparse inversion of the graphical model structure according to Webb additional compilation step provides human readable PyTorch implementation of sampling and log prob- ability evaluation of prior and likelihood for given graphical model. The code supports batching and can be used to sample synthetic data set for training a continuous normalizing flow (Grathwohl et al., 2018) as described in (Weilbach et al., 2020). Follow-up work has extended this approach and leveraged the diffusion model framework to yield reliable and more scalable amortized inference artifacts (Weilbach et al., 2023b;a). Daphne can provide aulomatic derivation of the attention masks from the GPPL language for the sparse transformer in this line of work (Weilbach et al., 2023b) as shown in Section 2.
(2018) and a translation of the graphical model Python code. This
5 Deep learning primitives
:===
(foreach ee [s] er ... so en] ... e (loop ea Cina Sace er c..)
Language 2: Loop extensions.
Daphne also optionally supports the two loop forms of van de Meent et al. (2018). These forms do not make the language more expressive, but have been originally used implement linear algebra library for (Weilbach et al., 2020), which includes matrix multiplication and 2d convolution. Both foreach and loop require loop counter ee expression evaluating to integer to determine the number of loop iterations. foreach binds $n, with each clement of the collection yiclding expressions 61...., in the body expressions fi....,en- loop iterates an accumulating function sinit starting with Cinit over et,..., en- To illustrate the use of the language we provide an excerpt of the library together with an example 2d convolution invocation at the end.
(defn dot-helper [t state b] (+ state
(* (get t) (get b t))))
(defn dot b] (loop (count a) 0 dot-helper b))
(defn row-mul It state v] (conj state (dot (get m t) v)))
öThere is also zero-copy access Python through http:///ttuubccc///jjj//ttttt/ee https:// github.coom/ con//oracieegraaappthonn

---

Published in Transactions on Machine Learning Research (03/2025)
(defn mmul [m vl
(loop (count m) [] row-mul m v))
(defn rov-helper [i sum a b]
(+ (dot (get 1)
(get 1))))
(defn squar [a b]
(loop (count a) 0 row-helper b))
(defn inner-cubic [a b] (apply (foreach (count a) [n (range (count a))] (inner-square (get n) (get n)))))
(defn slice-square [input size stride j] (foreach size [k (range (+ 1 stride)
size (* stride)))]
(subvec (get input k)
(* j stride)
(+ size (+ stride)))))
(defn slice-cubic [inputs size stride i j] (foreach (count inputs) [input inputs] (slice-square input size stride 1))
(defn conv-kernel [inputs kernel bias stride]
(let [ic (count (first inputs))
size (count (first kernel))
remainder (- size stride)
to-cover (- remainder)
iters (int (Math/floor (/ to-cover stride)))] (foreach iters [i (range iters)]
(foreach iters [j (range iters)]
(inner-cubic (slice-cubic inputs size stride j)
kernel)))))
(defn conv2d [inputs kernels bias stride]
(foreach (count kernels) [ksi (range (count kernels))]
(conv-kernel inputs (get kernels ksi) (get bias ksi) stride)))
(let [w1 [[[[o.8, 0.9],
[0.9, 0.6]],
[[o.0, 0.0],
[0.1, 0.5]]],
[[[o.2, 0.4].
[0.6, 0.1]],
[[o.4, 0.5],
[0.1, 0.1]]]]
b1 [0.1, 0.2]
[[[0.4, 0.5, 0.8, 0.8]
[0.5, 0.8 0.6, 0.1]
[0.9, 0.4 0.7, 0.2]
[0.5, 0.0 0.4, 0.2]],
[[o.0, 0.8, 0.2, 0.3]
[0.2, 0.2 0.8, 0.7] 0.7]
[0.1, 0.6, 0.6, 0.3]
[0.6, 0.7, 0.5, 0.2]111
(conv2d x x1 b1 2))

---

Published in Transactions on Machine Learning Research (03/2025)
In combination those primitives can be used to implement the deep learning applications in (Weilbach al., 2020), including a stochastic deconvolution layer and small convolutional network.
6 Related work
Table 1: Comparison of Probabilistic Programming Systems
Feature
Daphne
Stan
PyMC
Anglican
Pyro
Gen
X
/
Stochastic recursion
x
X
V
/
J
Graphical model focus
X
X
X
X
Higher-order functions
X
J
V
Recursion support
V
V
HMC inference
V
X
XXX
Python integration
X
Compilation to CUDA
X
X
X
X
X
Compared to corineee languages such Anglican (Tolpin et al., 2016), Pyro (Bingham et al., 2019), PyMC (Oriol et al., 2023), or Gen Missmanno-oownee el al., 2019), the Daphnee language only supports programs without stochastic recursion. This restriction prevents it from implementing certain non- parametric models and complex stochastic recursion schemes. However, inference in such models can be highly challenging and brittle, making the restriction to graphical models an acceptable trade-off for many applications.
While Daphne shares most of its syntax with Anglican, key difference is that Anglican programs cannot be translated into graphical model ahead of inference time and integrated into other inference systems, such as the deep learning pipelines shown in Section 2. Daphne, by contrast, is designed to first compile into a graphical model before inference.6 6 Anglican, on the other hand, can freely execute any Clojure code within its inference runtime and provides a set of powerful Monte Carlo sampling-basee inference methods.
Stan (Carpenter et al., 2017) is a well-ostablished GPPL, designed to work scamlessly with its Hamiltonian Monte Carlo (HMC) inference engine. Compared to Stan, Daphne offers a significantly more expressive functional probabilistic programming syntax, including support. for recursion and higher-order functions. BUGS, which has similar syntax to Stan, has been translated into the FOPPL language in (van de Meent et al., 2018). Likewise, the Daphne graphical model output could be translated back into Stan to leverage its inference engine while benefiting from Daphne's more expressive syntax.
Unlike Stan, which implemented C++, Daphne is built in Clojure a high-level, interactive programming environment naturally suited for meta-pres poot-proggamminn and rapid prototyping. Daphne's codebase small and modular, with compiler passes and core semantics implemented in most few hundred lines of self- contained code. This makes Daphne highly adaptable to ideas while providing seamless integration with the broader Clojure ecosystem, such Pyrohc through libpython-clj. Further exploration of this runtime integration is left for future work.
In terms of user experience, Stan provides more extensive documentation and a polished interface, making it more accessible to users who do not wish to modify the system. Daphne, by contrast, is designed for flexibility and extensibility, prioritizing expressivity and compiler-level transformations over case of use.
7 Conclusion and future work
Daphne is a small yet versatile probabilistic programming environment that represents new point in the design space of probabilistic programming languages and compilers. Its goal is to capture as much of higher-
6There also some exploratory interpreters included in the code base for tcaching purposcs, but they not the main focus of the system. 7 hhttps://gg /ttp://ggttthhteeeee com/clj/c/e/oeeeeocoo http://ljjyytton/lyyyeeee pytthon/1sbpython-cll
8

---

Published in Transactions on Machine Learning Research (03/2025)
order functional programming possible while remaining reducible to traditional probabilistic graphical models ahead of inference time. The representations produced at different compiler stages can be exported to provide fine-grained program information, such as dependencies between random variables, for downstream inference runtimes. These representations have been used in a line of research on structured neural networks for amortized inference and for teaching graduale courses in probabilistic programming.
Limitations For very large input programs, such as deep learning models, partial evaluation can lead expression swell, significantly slowing down compilation as all linear algebra operations need to be syntac- tically expanded and reduced. Scaling better to such programs requires further research, particularly in structuring the expansion and reduction of expressions during partial evaluation to ensure that reductions are always applied before expansions. Many other probabilistic programming systems (Bingham et al., 2019; Tolpin et al., 2016) treat tensors as single objects rather than explicitly modeling the full computational graph, including all scalars and intermediate computations, as Daphne docs. While this avoids the problem, it comes al the cost of not tracking fine-grained sparse structures.
Beyond these improvements, expression swell is inherently unavoidable in some cases, as Daphne unrolls all branches and loops into a graph before execution. Consider Markov Decision Process (MDP), such as agent navigating a maze. Each additional time step into the future requires expanding all branches with all possible subsequent branches, leading to an exponential blow-up in the representation. This makes GPPL-based systems like Daphne unsuitable for such problems.
Adding a module system, likely following Clojure's approach, is left. for future work. Currently, the lin- ear algebra functions in Section 5 must be manually concatenated with the program before compilation. Additionally, the Daphne language does not yet support anonymous function definitions, which would en- able closures, making it more in line with standard Clojure and more convenient for complex programs. Supporting closures is planned for a future iteration of the language.
References
Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Au- tomatic Differentiation in Machine Learning: A Survey. Journal of Machine Learning Research, 18: 153:1-153:43, 2017.
Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. Pyro: Deep universal probabilistic programming. The Journal of Machine Learning Research, 20(1):973 978, January 2019. ISSN 1532-4435.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vore Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. 2xXix:205.144660 [cs], July 2020.
Bob Carpenter, Andrew Gelman, Matthew Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A Probabilistic Programming Language. Journal of Statistical Software, Articles, 76(1):1 32, 2017. ISSN 1548-7660. doi: 10.18637/jss.v076 v0..
Marco F. Cusumano-Towee
Feras A. Saad, Alexander K. Lew, and Vikash K. Mansinghka. Gen: A purppal probabílistic programming system with programmable inforence. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2019, pp. 221-236 Phoenix, AZ, USA, June 2019. Association for Computing Machinery. ISBN 678-1-4500--612277 doi: 10/gf4jwn.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Πya Sutskever, and David Duvenand. FFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models. In International Conference on Learning Representations, September 2018.
9

---

Published in Transactions on Machine Learning Research (03/2025)
Rich Hickey.
The Clojure programming language. In Proceedings of the 2008 Sympostum Dynamic Languages, Dis '08, pp. 1:1-1:1, New York, NY, USA, 2008. ACM. ISBN 978-1-600558-77--2 doi: 10.1145/
1408681.1408682.
Rich Hickey. A history of Clojure. Proceedings of the ACM on Programming Languages, (HHPL)): 46, June 2020. ISSN 2475-1421. doi: 40.1145/3386322
Neil D. Jones, Carsten K. Gomard, and Peter Sestoft. Partial Evaluation and Automatic Program Generation. Prentice-Hall International Series in Computer Science. Prentice Hall, New York, 1993. ISBN 978-0-13- 020249-9.
Andrew W. Keep and R. Kent Dybvig. A nanopass framework for commercial compiler development. ACM SIGPLAN Notices, 48(9):343-350, September 2013. ISSN 0362-1340. doi:
Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, 2009. ISBN 978-0-262-01319-22
Phuc C. Nguyen, Thomas Gilray, Sam Tobin-Hochstadt, and David Van Horn. Size-Change Termination as a Contract. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation PLDI 2019, pp. 845-859, 2019. doi: 01444//331221..33
Flemming Niclson and Hanne Riis Niclson. TeveL Functional Languages. Number 34 in Cambridge Tracts in Theoretical Computer Science. Cambridge Universily Press, Cambridge; New York, NY, USA, 1992. ISBN 978-0-521-4038449
Abril-Pla Oriol, Andreani Virgile, Carroll Colin, Dong Larry, Fonnesbeck Christopher J., Kochurov Maxim, Kumar Ravin, Lao Jupeng, Luhunann Christian C., Martin Osvaldo A., Osthege Michael, Vieira Ricardo, Wiecki Thomas, and Zinkov Robert. PyMC: A modern and comprehensive probabilistic programming framework in python. PeerJ Computer Science, 9ce1516, 2023. doi: 10.7717//perrj-cc.556.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Rosolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVE CCFF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Erik Strumbelj, Alexandre Bouchard-Côtéé Jukka Corander, Andrew Gelman, Håvard Rue, Lawrence Mur- ray, Henri Pesonen, Martyn Plummer, and Aki Vehtari. Past, Present and Future of Software for Bayesian Inference. Statistical Science, (91)):66-6, February 2024. ISSN 0883-4237, 2168-8745. doi: 10.1214/23-STS907
David Tolpin, WallWellem van de Meent, Hongseok Yang, and Frank Wood. Design and Implementation of Probabilistic Programming Language Anglican. In Proceedings of the 28th Symposium on the Implementa- tion and Application of Functional Programming Languages, IFL 2016, pp. 1-12, New York, NY, USA, Au- gust 2016. Association for Computing Machinery. ISBN 978-1-4503-4767-99 doi: 00144///666899.306400
Jan- Willem van de Meent, Brooks Paige, Hongseok Yang, and Frank Wood. An Introduction to Probabilistic Programming. arxiv:1809. 10756, September 2018.
Stefan Webb, Adam Golinski, Rob Zinkov, Siddharth N, Tom Rainforth, Yee Whye Teh, and Frank Wood. Faithful Inversion Generative Models for Effectivee Amortized Inference. In Advances in Neural Infor- malion Processing Systems, volume 31. Curran Associates, Inc., 2018.
Christian Weilbach, Boyan Beronov, Frank Wood, and William Harvey. Structured Conditional Continuous Normalizing Flows for Efficient Amortized Inference in Graphical Models. In International Conference on Artificial Intelligence and Statistics, pp. 4441 4451. PMLR, June 2020.
Christian Dictrich Weilbach, William Harvey, Hamod Shirzad, and Frank Wood. Scaling Graphically Struc- tured Diffusion Models. In ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling, July 2023a.
10

---

Published in Transactions on Machine Learning Research (03/2025)
Christian Dietrich Weilbach, William Harvey, and Frank Wood. Graphically Structured Diffusion Models. In Proceedings of the 40th International Conference Mochine Learning, pp- 36887-36909 PMLR, July 2023b.
A Appendix
A.1
Python export
The following export is done with the help of hy-lang" which allows direct translation between lisps and then into Python syntax. This is the compiled code for Program 2.
import by
import torch
nath
from torch distrbbditiibsions import Normal, Bernoulli, Laplace, Uniform
class Model: dis latent
din_condition
faithful adjacency [[9. 13. [0. 21. [6. 11. 16. 4) [0. 51. [6, 10. 71, 21. [1, 31, (1.
5], [L, 6), [1. 70, 6),
'((defnia reducein [f acc slin
(count (reduce 1 (first
acc)2An (defein
daserve [ace dara]An (Tetin [stopein (first acc)An blaste (second acc)an xain (first data)
yn'n (second data)in znin ( (# slope xn) bias)]in (observe (nornal 1.6) yn)'n [slope bias])) (letin
[slope (sample (normal 9.9 10.83) biss (sample (nornal 6.8 18.0))3 (reduce\n observerdataln [slope bias]An
[[1.0 2.13 12.0 3.9] [3.8 5.3] [4.0 7.7] [5.0 18.2) [6.9 12.9]]1Aa [slope bias]))\n
sample
sample_1 Normal 10.0).saxpled.
sanple_0 Normsl (9.0, (0.0).saxpleC
observe_4 4 (sample_ * 3.0 sample_1, sampee)
observe_6 6 Normal (sample_0 * 5.0 sample_1, 1.4). sampleO
observe 5 Nornal (sample. 4.0 sample.1, 1.0). sample)
observe_ Hornal (sample_
sample_1, 1.0). sample()
observe (sample
sample 1, 1.0). sample()
ohserve_3 Nornal (sample_6 * sample_], sample()
[turch. tensor ([sample_0, sample.1)).
torch. tensor (Lobserve_2, 2, observe.3, observe.4, 4, observe. observe 6, observe_7]]
def tho sample, observe):
log_likeli torch. (sarple (6])
log likeli
saapee((slice(foae), 1]1, 1.0).log (bbo(bbbeerr(())scceeeeeee)) 1]1)
log_likels
Normal [aamaaaCaaaa[sssssceCCaaana
83] & 6.9 sample ((llce(()))))))me)), 1]], (0g.
* sanple[[slsceettno))
log_likeli Normal
5]])
1og_1ikels + Normal (armaa((sarne Carsaal(aanpAAA[sssseeeCCC 811 # 1.9 sample [[s]e((((Cc 111. (og.
811)
log. likeli +
8]] & 3.9
sample (ssic 1]]. 1.0). Lo0.).oo.pprobbbbbe
2]])
likelikels -- Normal [sapaaa(ccaalsssscceeeeeeoi 8]] 4.9
sample[[slice[[nn]] 131. 1.6). 1og_log Cobsere (sliaceeeeesss)eeeeee 3]])
return lkeelikeli
(ogeprior(self,, sample):
log_prior torch. www.sse sbappeeeeeesbpe
lng_prior -- Nornal (9.9, 19.8). fee (None), 111)
"https://hylang,
11

---

Published in Transactions on Machine Learning Research (03/2025)
log. prior (9.9. 19.8). pro. (sample (Xonc) B]l) log_prior
12


Reviews:
[{"review_id": "HlLA6D1LIV", "reviewer": "Reviewer_ADBH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The direction of designing probabilistic programs to represent neural networks and study inference methods is interesting.\n\nHowever, with the current form of the submission, it is hard to capture why it is beneficial to use the proposed Daphne system, and what are the unique advantages compared to other probabilistic programming systems.\n\nIn particular, I am not sure whether the topic of this paper is a good fit for TMLR. It is important to clarify what are the key takeaways for audience working on deep neural networks to learn from this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Daphne, a probabilistic programming system that supports the representation and compilation into graphical models and (simple and small) neural networks. The submission describes the language syntax, compiler design, and example code to represent linear regression and convolutional neural networks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1. The submission needs to clearly explain what are the compelling use cases of the proposed system, and provide more details on what are the new features that are not supported by other probabilistic programming systems.\n\n2. The authors also need to expand the related work section. From the current submission, it is unclear why designing another probabilistic programming system for representing neural networks is necessary.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UxJVT8DtMi", "reviewer": "Reviewer_cAHD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n[s1]: The manuscript is easy to follow if one is familiar with the literature, gives a good amount of context, and overall reads fairly well.\n\n[s2]: Daphne itself is fairly straightforward as a language, and the paper does a good job at presenting its overview.\n\n[s3]: the language design seems to hit a good balance between expressivity and tractability, and the manuscript includes literature that has apparently already successfully made use of the language.\n\n## Weaknesses\n[w1]: the manuscript lacks... purpose. It generally lacks a _clear_ articulation of what makes Daphne fundamentally different from existing PPLs. In particular, I think a 1:1 comparison with Anglican is necessary, given the clear (and stated) similarity in both design and (perhaps) capabilities.\n\n[w2]: the manuscript is at times confusing wrt. its origin, and what is specifically novel about the contribution. Is Daphne fundamentally a formalization of the language presented in de Meent et al., 2018? Is it the AOT symbolic compiler? Is it presenting some novel _and qualitatively better_ ergonomics point in design space from a user perspective?\n\n[w3]: while the limitation section is welcomed, it is still really limited for the critical learning and inference cases where the complexity of the model becomes non-trivial. Since the manuscript doesn't go into details in _how_ any of it scales (e.g. how long the AOT steps take as the graphical model gets bigger, or as more higher order functions are used), a reader may justifiably label the language as a useful teaching tool, rather than a powerful system to do bayesian inference research with.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript presents Daphne, a probabilistic clojure-like lisp that (AOT) compiles graphical models into a symbolic representation amenable for learning and inference using existing ML backends.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Comparative Analysis / Performance\n\nI would like to see the following:\n\n[c1] Some detailed comparisons (both qualitative and quantitative) with Anglican, Stan, and other relevant modern probprog frameworks (Pyro?).\n\n[c2] A few concrete examples where Daphne offers advantages over existing approaches.\n\n[c3] Some benchmarks on compilation time, and inference efficiency for the default backend (or all available / supported backends).\n\n### Case studies on ergonomics and features\n\n[c4] The manuscript includes a very few details on how the language is generally run and how programs managed by the user. Nowadays I believe it is reasonable to expect papers presenting these kinds of details at least as an FYI. Particularly, one of the exciting contributions is the ability to compile the lisp down to a python representation, which makes a pytorch / jax / $FRAMEWORK translation possible. I was disappointed when the authors showed the users having to use Hy as a middle step, which makes it seem as aftertought.\n\n[c5] The manuscript includes a couple of toy examples, which are great and clear. However, it also claims that some work used Daphne and its advantages to implement state of the are inference for interesting problems. Using these papers as case studies, particularly focusing on how Daphne enabled the implementation of the methods (which would otherwise have been challenging in other sysmtes), would greatly enhance the contribution of this manuscript.\n\n### Positioning\n\n[c6] The manuscript could use from committing to a specific target audience. Currently it reads well if one is faimilar with both the PL and ML literature, but some of the language seems to be too PL-coded in a way that might make the average TMLR reviewer puzzled at times. Note that it would be perfectly fine to keep it PL-targeted, but at that point it would not necessarily be a good manuscript for TMLR.\n\n[c7] I would look to significantly strengthen the motivation for why this system is needed. If the primary contribution is the ability to represent graphical models symbolically, I would double down on that and focus on providing motivation to the reader that it is an impactful constraint.\n\n### Assorted questions\n\n#### Intro\n\n>An interesting position in the language design space are languages that can be readily translated into a representation for which efficient inference algorithms exist, e.g. Stan (Carpenter et al., 2017). Such systems compile an expressive input syntax into a lower-level model that can be efficiently evaluated with the numerical primitives of the inference engine. Furthermore it should be possible for users to extend the language with new syntax if needed.\n\nThis paragraphs reads a little weirdly to me. It seems to miss an argument that justifies why translating down to graphical models is useful. The two other sentencea seem to read as a bucket list of properties that would make for a good PPL, so I wonder if the paragraph had additional language that was mistakenly hidden.\n\n>[...] particularly well suited for teaching and research\n\nAs opposed to what? And is the intention of the work to particularly overfit to these cases (which seem very broad, especially when including a generic \"research\").\n\n#### GPPL\n\n>The last expression is the global entry point into the program having access to all defined functions.\n\nThis is a fairly long-winded and SICP-like way to say that language has an implicit global entrypoint. A quick code listing with comments would probably be more useful (and be perfectly clear).\n\nTha language also doesn't seem to include a module / import / load system. Is that correct? If so, how are complex programs likely to be representable / reusable in a growing experimental codebase?\n\n>In contrast to van de Meent et al. (2018) Daphne does not rule out recursion or higher-order functions as can be seen in our simple implementation of reduce.\n\nIs this novel wrt. Anglican / Stan / et al.? I think it's a little strange (though still defensible) to employ a teaching-focused-- daresay toy? --language as the baseline for such things.\n\n#### Compiler\n\n>This approach allows to build extensible compilers with towers of languages that deal with single language features\n\nI suspect even good readers won't be familiar with this concept. Might be worth a quick explanation if this requirement greatly informed the design of Daphne.\n\n>The need of potentially inconsistent compilation semantics separate from the interpreter\n\nWould it be possible to include an example of failure modes of this kind? I couldn't think of one without going into fairly complex scenarios, but it would be good to understand what types of common patterns the authors are trying to mitigate away.\n\n>Daphne also provides optional runtime support for inference. A differentiable subset of the language is supported with source to source reverse-mode automatic differentiation (Baydin et al., 2017) and can be plugged into a simple Hamiltonian Monte Carlo (HMC) implementation.\n\nIt would be great to significantly expand this section to describe the variuos inference runtimes. The ability to autodifferentiate through the language is an extremely exciting feature, but without knowing the relevant subset it is impossible for a reader to understand whether Daphne might fit their research problems.\n\n#### Related work\n\n>Inference in such models is often very challenging and brittle though and this motivates our restriction to graphical models.\n\nIt's not clear whether this fits as a motivation. Are the authors arguing that PPL developers should be discouraged from supporting stochastic recursion?\n\nNote that I would understand _if_ that was indeed a reason, but it does need at least some kind of qualitative argument based on the literature or the authors' experience.\n\n>BUGS, which is similar to Stan in syntax, has been translated to the Daphne language in van de Meent et al. (2018)\n\nThis seems to imply that Daphne was designed in van de Meent et al. 2018. If that's the case, I would reframe this paper to characterize that accurately.\n\n#### Conclusions\n\n>Very large sized input programs, such as deep learning models, partial evaluation can lead to expression swell that significantly slows down compilation as all linear algebra operations need to be syntactically expanded and reduced.\n\nThis is an important limitation, and a critical one, given how strongly the manuscript focuses on one of the motivations being that the language enables good use of \"modern ML primitives\". I think more clarity on the limitations, and guidance on follow-up work that is going to remove these seems warranted.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7REYY3wA7K", "reviewer": "Reviewer_2Cmz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and discusses both the strengths and limitations of Daphne in substantial detail\n2. The paper is filled with helpful examples that do a wonderful job of explaining the basics of Daphne syntax and its usage in different contexts\n3. The Daphne language itself appears to be very expressive and would definitely be useful in the context of both teaching and research\n4. The compiled graphical model output by the model appears to be translatable to a wide variety of formats (including Stan and PyTorch), which is a huge plus\n\nWeaknesses:\nThe only real weakness I see here is a lack of discussion of related work. It is clear from the paper what Daphne is good at and how versatile and general it is. However, I am sure that readers such as myself who have dabbled in some of the more popular probabilistic programming frameworks such as PyMC would be interested to see a more detailed comparison of how Daphne compares to these. Some of these comparisons can already be inferred from the paper such as Daphne's expressive syntax that allows for higher-order functions and recursions. However, a more in-depth comparison would certainly be interesting to see, particularly with respect to the other higher-order languages such as Pyro, etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors introduce Daphne - a higher-order probabilistic programming language that can be used to define probabilistic graphical models with support for higher-order functions and an unbounded number of random variables. The contributions can be summarized as follows:\n1. Programs written in Daphne can be directly compiled into graphical models using their multi-pass compiler\n2. Unlike most traditional and well-known probabilistic programming languages, Daphne supports higher-order functions and recursions and can directly convert these into structured probabilistic models at the compilation step\n3. Daphne programs can also be compiled into deep-learning models for use with PyTorch\n4. Models defined using Daphne have limited support for inference but can be compiled to Stan and take advantage of the latter's mature inference pipeline", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper is in pretty good shape as it is. The only thing I would recommend is what I mentioned in the Weaknesses section. It would be great to see a more in-depth comparison with other well-known probabilistic programming languages (particularly the higher-order ones). A table would be best, but a couple of paragraphs highlighting the differences should also be sufficient.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce and provide a high-level overview of Daphne, a probabilistic programming system with a fairly expressive syntax which is still sufficiently restrictive to allow for effective sampling-based inference. This is an appealing combination which occupies an under-explored niche in the probabilistic programming space. All the reviewers recommended acceptance as the latest revision addressed most of their concerns.
Decision: Accept (poster)