Scaling Continuous Latent Variable Models as Probabilistic Integral Circuits
Authors: Gennaro Gala
Cassio de Campos
Antonio Vergari
Erik Quaeghebeur
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Latent variable models
probabilistic ML
tractable models
Primary area: probabilistic_methods

Abstract: Probabilistic integral circuits (PICs) have been recently introduced as probabilistic models enjoying the key ingredient behind expressive generative models: continuous latent variables (LVs). PICs are symbolic computational graphs defining continuous LV models as hierarchies of functions that are summed and multiplied together, or integrated over some LVs. They are tractable if LVs can be analytically integrated out, otherwise they can be approximated by tractable probabilistic circuits (PC) encoding a hierarchical numerical quadrature process, called QPCs.

So far, only tree-shaped PICs have been explored, and training them via numerical quadrature requires memory-intensive processing at scale. In this paper, we address these issues, and present: (i) a pipeline for building DAG-shaped PICs out of arbitrary variable decompositions, (ii) a procedure for training PICs using tensorized circuit architectures, and (iii) neural functional sharing techniques to allow scalable training. In extensive experiments, we showcase the effectiveness of functional sharing and the superiority of QPCs over traditional PCs.

Scaling Continuous Latent Variable Models as Probabilistic Integral Circuits
Gennaro Gala!.
Cassio Campos'
Antonio Vergari !
Erik Quaeghebeur"
Eindhoven University of Technology, NL School of Informatics, University of Edinburgh, UK
Abstract
Probabilistic integral circuits (PICs) have been recently introduced probabilistic models enjoying the key ingredient behind expressive generative models: contin- uous latent variables (LVs). PICs symbolic computational graphs defining continuous LV models as hierarchies of functions that are summed and multiplied together, integrated over some LVs. They tractable if LVs be analyti- cally integrated out, otherwise they can be approximated by tractable probabilistic circuits (PC) encoding hierarchical numerical quadrature process, called QPCs. So far, only tree-shaped PICs have been explored, and training them via numer- ical quadrature requires memory-intensive processing at scale. In this paper, we address these issues, and present: (i) pipeline for building DAG-shaped PICs out of arbitrary variable decompositions, (ii) procedure for training PICs using tensorized circuit architectures, and (iii) neural functional sharing techniques to allow scalable training. In extensive experiments, showcase the effectiveness functional sharing and the superiority of QPCs over traditional PCs.
1 Introduction
Continuous latent variables (LVs) are arguably the key ingredient behind many successful genera- tive models, from variational autoencoders [24] generative adversarial networks [20], and more recently diffusion models [56]. While these models allow to learn expressive distributions from data, they are limited sampling and require task-specific approximations when comes to per- form probabilistic reasoning. as even simple tasks such as computing marginals or conditionals are intractable for them. On the other hand, performing these tasks can be tractable for (hierarchical) discrete LV models [4, 3], but these prove to be more challenging to learn at scale [9, 10, 32, 33].
This inherent trade-off among tractability, ease of learning, and expressiveness and explored with probabilistic integral circuits (PICs) [18], a recently introduced class of deep generative models defining hierarchies of continuous LVs using symbolic functional circuits. PICs are tractable when their continuous LVs can be analytically integrated out. Intractable PICs can however be systematically approximated as (tensorized) probabilistic circuits (PCs) [53, 4], the representation language of discrete LV models. An instance of such PCs encodes a hierarchical numerical quadrature process of the PIC approximate, and as such is called quadrature PC (QPC).
be analyzed
Distilling QPCs from PICs has proven to be an effective alternative way to train PCs, but it has only been explored for tree-shaped PICs, building and scaling richer LV structures is open research question that requires new tools [18]. In this paper, we fill this gap by redefining the se- mantics of PICs and extending them DAG-shaped hierarchies of continuous L.s. Specifically,
Corresponding author: g- gawaaatu...... nl
Shared supervision
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

design PICs as a language to represent hierarchical quasi-tenson factorizations [49], parameterized by light- multi-layer perceptrons.
Contributions. (1) We present a systematic pipeline to build DAG-shaped PICs, starting from ar- bitrary variable decompositions (Section 3.1). (2) We show how learn and approximate PICs via a hierarchical quadrature process which we encode in tensorized QPCs that match certain circuit architectures proposed in different prior works [43, 42, 31, 36] (Section 3.2). (3) We present func- tional sharing techniques to scale the training of PICs, which lead us to parameterize them with multi-headed multi-layer perceptrons (MLPs) requiring comparable resources as PCs (Section 3.3). (4) In extensive experiments (Section 4), we show that (i) functional sharing proves remarkably effective for scaling and that (ii) QPCs outperform PCs commonly trained via EM or SGD, while being distilled from PICs with up to 99% less trainable parameters.
2 Probabilistic integral circuits
Notation. We denote input variables as X and latent variables (LVs) as Y and Z, with y,y and z as their realization respectively. We denote scalars with lower-case letters (e.g., w E R), vectors with boldface lower-case letters (e.g.. w E R. matrices with boldface upper-case letters (excluding X, , Z. e.g., W E RxM,, and tensors with boldface calligraphic letters (e.g.. W E R LxxMxN,,
A probabilistic integral circuit (PIC) с is a symbolic computational graph representing non- negative function c(X) / c(X, 2) dz, i.e. c(x) 2 O. over observed variables X and continuous latent variables Z. Similar to probabilistic circuits (PCs) [53, 4]. PICs have input, sum and product units.² 2 Different from PCs, however, PICs operate on functioons, not scalars, and make use of a new type of unit: the integral unit O which allows to represent continuous LVs. To satisfy non- negativity, we parameterize PICs with non-negative functions and positive sum weights, specifically:
An input unit u (depicted as 2 in our figures) represents a possibly non-normalized distribution fu(X,, u: Z,) + RR+ where X, C X and Z, C Z:
A sum unit (+) outputs a weighted sum of the functions it receives from its input units, i.e. gu 9X.,.., Zo.) = Seinn W: Z.), where in(u) is the set of units u takes as input, w; > O, X = Usgin(w) X, and Z, Usein(u) Z, Similarly, product unit (x) outputs the product of its incoming functions, i.c. gu Zw) = (e (X,, Z.);
Finally, an integral unit u ( encodes an "uncountable weighted sum" whose weights are compactly represented by function fz., Y.) - R+, where Ø 2 Y. CY are the LVs that are being integrated out by u. while Z, can potentially be empty, i.e. Z,= Ø. The unit receives function gi (X;, Y.) from its only input unit i and outputs the function (x,, Zw.) = S& (Z,,, Yu) Yu) dyy., where A = suupp(Y,, is the support of Y... For instance, an integral unit u with fu (12), {Y = 22 - y2 and sY)
1,, receiving function gi ((X), [Y x2 3XX + 4Y would output g, ((2) (x]) = 2/3 X 36 1).
Fig. 1(b) and Fig. 2(b) show example PICs. Note that we use f to indicate the functions attached to input and integral units which are essentially parameters of the model, while we use g to indicate functions being outputted by all type of units. The output of PIC e the output function returned by its root unit v. which is only defined on X, i.e. e(X) = gu (X) and Z,, = Ø. Similar to PCs, imposing structural constraints over PICs can unlock tractable inference [4]. As such, we assume that (i) all Units receive functions defined on the same input variables (aka smoothness), and (ii) all -nitt receive functions defined over disjoint sets of input variables (aka decompasability).
PICs are tractable when their LVs can be analytically integrated out, meaning that we can pass- through integral units computing the integration problem they define, eventually outputting a func- tion. Notably, this is possible when LVs are in Gaaasiann relationships [27] or when functions are polynomials. Intractable PICs can however be approximated via a hierarchical numerical quadra- ture process that can be encoded as a PC called quadrature PC (QPC). Intuitively, each PIC integral unit can be approximated by a set of sum units in QPC, each conditioning on some previously computed quadrature values, with large but finite number of input units [18]. Materializing a QPC allows to train PICs by approximate maximum likelihood: Given a PIC, gradients to its parameters
We refer the reader to Appendix A for an introductory overview of (probabilistic) circuits.

---

attached to input, sum and integral units can be backpropagated through the corresponding QPC [18]. This also provides an alternative way to train PCs that can rival traditional learners.
So far, the construction of PICs has been limited to a simple compilation process from probabilistic graphi- cal models (PGMs) [27] with continuous LVs [18]. In a nutshell, the LV nodes of PGM become integral units of a PIC model, and the PGM (conditional) distribu- tions become the functions fu attached to input and in- tegral units of the PIC, as we illustrate in Fig. 1. How- ever, the PGM structure needs to be limited to a tree, as to avoid that the hierarchical quadrature process would yield an exponentially large QPC, thus hindering learn- ing. This imposes semantics for current PICs sim- ple latent tree models [3], and clearly limits their expressiveness as more complex LV interactions are not possible. Building more expressive PICs requires reinterpreting this semanties and introduc- ing new tools, which we do next.
(a)
(b)
Dise. (21)
2
X
Z3
OHIZ 1.1Z
A(X,, Z)
f1(ss
7
OLL(Z (2+))
(1(22 2,) 3 (x(X,,
Figure 1: PGM (a) + tree PIC (b)
3 Building, learning and scaling PICs
In Section 3.1, we systematize the construction of DAG-shaped PICs, showing how to build them starting from arbitrary variable decompositions, going beyond the current state-of-the-art [18]. Then, in Section 3.2, we show how to learn and approximate such PICs with QPCs encoding a hierarchical quadrature process, retrieving PC architectures proposed in prior works [36]. Finally, in Section 3.3, we present (neural) functional sharing, a technique which we use to parameterize PICs as to make their QPC materialization fast and cheap, allowing scaling larger models and larger datasets.
3.1 Building PICs from arbitrary variable decompositions
Standard PCs can be built according to established pipelines that allow to flexibly represent arbitrary variable decompositions, as well as rich discrete LV interactions [36, 41]. In the following, we derive an analogous pipeline for PICs that allows to take care of continuous LVs, going beyond their current tree-shaped semanties (Section 2) yet allowing to perform hierarchical quadrature without blowing up the size of the materialized QPCs. To do so, we start by formalizing the notion of hierarchical variable decomposition, or region graph, out of which we will build our PIC structures.
Definition (Region Graph (RG) [15]). An RG over imput variables X is a bipartite and rooted directed acyelic graph (DAG) whose nodes are either regions, denoting subsets of X, or partitions, specifying how a region is partitioned into other regions (Fig. 2(a)).
RGs can be (i) compiled from PGMs [5, 27, 3, 31], (ii) randomly initialized [43, 16), (ii)) learned from data [15, 19, 40, 57], or (iv) built according to the data modality (c.g. images) [45, 42, 36]. If compile from a tree PGM, as in (Fig. 1), the resulting RG will be tree, thus yielding tree-like PIC [18]. Our pipeline, detailed in Algorithm 1, takes an arbitrary DAG-shaped RG as input, and can deliver DAG-like PICs. Without loss of generality, we assume to have an RG R which only allows for (i) binary partitionings of regions (i.e. all product units will have two input units) and (ii) univariate leaves, shown in Fig. 2(a). Our construction iteratively builds PIC in bottom-up fashion, associating regions to PIC units. For every leaf region X., E X in R, we instantiate an input unit u with function f.,(. {Zm)), where Z., Z an arbitrary continuous LV (Line 8, Algorithm 1). Such functions can be univariate conditional densities, i.e. P.. (X,l Zu), resembling small VAE-like decoders [24] amenable to be numerically integrated.
Once all leaf regions have been processed, we move to the inner ones. Let C X be an inner region partitioned in N > different ways {(X()) z(())
1 J= i.c. (X.") nx("") = Ø and (x(*) (X(")XX (X X for every n. For each partition (x("), x("),, we will merge the PIC units associated to regions x(1) and x/") using consecutive applications of product and integral units as we explain next- eventually associating a unit the partition itself. One can design such merging as desired, as long as smoothness and decomposability are not violated. Finally, in case N W=1 1, we associate to X the unit associated to its only partition, otherwise, in case N 1, we merge the N units associated to each n-th partition using a sum unit which we then associate to X (Line 6, Algorithm 1).
3

---

(a)
ad (a)
Xs) (c)
(d)
O O (a))
S (st(Zz)) (22.2.))
X1. X2
X Xa
(b)
X1
22)
[s( Ra)
Figure 2: The pipeline presented in this paper: RG + PIC + QPC + folded QPC. Starting from (fragment of) DAG-shaped region graph (a), we build DAG-like PIC via Algorithm using Tucker-merge (b). Then, we materialize a tensorized QPC encoding hierarchical quadrature process via Algorithm 3, using K 2 quadrature points, which we fold allow faster inference (d).
Algorithm 1 (G2PIC(R)
Algorithm 2 merge U2,
Input RG R over variables X
Input Units u1, U2. and boolean flag P
Output PIC c(X) = cXX, z)dz
Output 1 or X unit u with (u1, uz) as descendants
1: 21 map()
from regions to PIC units
procedure (Tucker-merge)
P Zuj # Zuz
2: for each region X E postOrder(R) do
2:
Z. + Ø if else {2} C Z (Znn Luz}
3:
x partitioned as {(x xx2 J then
return (18(,,, wal); : fu (Zu, Zun 1)
4:
P True if X else False
4: procedure (CP-merge)
D Zun = Zuz
5:
S - [merge " "..4_"]]..2 - [)p) p)]m
Zus 2 if else {Zuz} C Z {Zuz
6:
2 + pop () if N =1 else ((S))
6: da + D(u1, fun (Zugy { {un H))
7:
else
> X
Zu, t Ø if p else 1Zua C Z {Zuz}
8:
U[X]
Offu (X, 12. H))
8: u4 + D(uz, sus (Zus [Z }))
9:
return A PIC with 2Z X| as root unit
return Alus, ual)
Merging PIC units. Let i1 and Ui be candidate units to merge, each outputting functions with LV Zu, and Zu, respectively. We present two ways of merging units: Tucker-merge and CP-merge, which we detail in Algorithm and whose names will be clearer in the next section. Zu, + Zwz. use Jucker-merge: We merge ui and u2 with a product, which is then input to an integral unit u with function (1 }.{. [Zug, Zuz)), where Z, E Z \ (Zuy Zug
J- Otherwise, if Zu, = Zuz. CP-merge: We add two integral units, ug with input #1 and u4 with input uz. which finally merge with a product. We parameterize unit U3 (resp. u4) with fuz (1 J. {Z }) (resp. fus (f ( (2mz)) })), where Zug + Za, (resp. Zua + Zwz)) Note that whenever merging two units defined on X, we need to marginalize out the remaining LVs, without introducing new ones. We illustrate the application of Algorithm 1 in Fig. 2(a-b). Our pipeline generalizes the PICs used in prior work [18] (Fig. 1) as we can build them by just converting latent tree structures in tree RGs and using CP-merge as merging procedure. While we now do not need a PGM to build a complex PIC structure, one could try to geverse-engines our PICs to retrieve a PGM via decompilation [1], the result would very intricate hierarchy over continuous LVs [41].
3.2 Learning PICs via tensorized QPCs
Given DAG-shaped (intractable) PIC, we now show how to approximate it with tensorized PC en- coding a hierarchical quadrature process, namely a QPC. Intuitively, we interpret PICs as to encode a set of quasi-tensors [49], generalization of tensors with potentially infinite entries in each dimen- sion corresponding to a continuous L.V, which we materialize into classical tensors via quadrature. We begin with definition of tensorized circuits and brief refresher numerical quadrature.
Definition 2 (Tensorized Circuit [36, 431). A tensorized circuit is a parameterized computational graph encoding a function c(X) E R. and comprising of input Lg product 2 and sum + layers. Each layer consists of many computational units defined over the same variables, and every non- input layer receives vectors as input from one or more layers. Each input layer is defined on vari- ables X CX and computes collection of Ke parametric functions [fx: dom (Xom + R2 out- putting a Keddimensional vector. Each product layer l computes either an Hadamard product (O) or a Kronecker product (8) of the vectors it receives from its imputs layers. Specificully, the Hadamard product is an element-wiss product of vectors, and therefore applicable when these have sume size, while the outer product of two vectors E R and v TR w u @ v = u
u.y E R where || is the concatenation operator. Finally, a sum layer e with Se sum units receives inputs from

---

[((Z=5) (1 Aimá))
U 1(12=1122 (M x==E))
+
W
(17)))
BERK quad. poants
Z
W K KxKxX
RR quad. weights
Z
W
Tucker
X
KxK*
layer
Y22
wx /112=xx ) (K==x Y==Ex)
(a)
(b)
(c)
(d)
Figure 3: From functions to sum-product layers via multivariate numerical quadrature (See- tion 3.2). We illustrate how the 3-variate function f((Z), (Y2, ()) (a) can be seen as an infi- nite (quasi) tensor that we first materialize w.r.t. integration points z as a finite tensor W of size K x K x K (b, Equation (2)), then flatten as matrix accounting for integration weights w (c, Equation (3)), and finally use to parameterize Tucker layer (d, Equation (Tucker-layer).
N layers {lit and computes the matrix-vector product W where W RX K=E, - -e, are the sum layer parameters. When N = 1, then it simply computes (Xl().))
Numerical quadrature. A numerical quadrature rule is an approximation of the definite inte- gral of function as weighted sum of function evaluations at specified points [13]. Specif- ically, given some integrand f R + and interval ==
>==[a,)) ba.b). a quadrature rule consists of a
set of K integration points E A and weights w RRKK minimizing the integration error εк
I/a (z) dz - ΣK (f which goes to zero as K - co. To approximate an integral of an N- dimensional function f. we can phrase the multiple integral as repeated one-dimensional integrals by applying Fubini's theorem [17], aka tensor product rule, as follows.
/ Ax (Z)dz
[(El..,,W)daa)) ...dZN
Wii Σ []C[K] [NE(K]
Σ
Wiy (( my kty)
From PICs to QPCs. Given a candidate PIC, we will explore it in post order traversal, and itera- tively associate circuit layer (Definition 2) to each PIC unit, process that we call materialization. We detail such procedure in Algorithm 3, which essentially applies Eq. (1) hierarchically over the PIC units. Each unit encodes function over potentially continuous and discrete variables, hence representing a quasi-tensor, which we approximate by evaluating it over the quadrature points only, thus materializing classical tensor (Fig. 3 (a.b)). We facilitate quadrature by assuming that all PIC LVs have bounded domain A = [- This way, we can always use the same quadrature rule (ž, w) for each required (multivariate) approximation, and also simplify treatment and exposition.
We begin materializing every PIC input unit u with function (, (Zw}) w.r.t. integration points 2, ef- fectively creating an input layer L: dom(X..) - K as [fu(X,, Z, =x) _ (Line 4, Algorithm 3). The pa- rameters of such layer can be matcrialized as a matrix of shape K x P. where P is the number of parameters fu requires. For example, if fa is a univariate conditional Gaussian Pa (x,,, Zu.,,
Algorithm PIC2QPC ž, W)
Input PIC c, quadrature rule ž, E IR Output Tensorized QPC
1: L < map() > from PIC units to layers
2: for each unit E postOrder(c)
3: if is @ then
K x 2 matrix for pa- rameterizing the layer, where each row stores the mean and standard deviation each integration point dµ-
4: C(u) + [ff Z.=āx)) |
5: else if u O(u., fu) then
6: C[u] + W((") [[us] via Eq. (3)
Next, we address the most important part of this quadra- ture process, i.e. the materialization of PIC integral units as sum layers. Specifically, let u be an integral unit with N N-dimensional function fu Zu., Y.)) where
7: else if is @(uu, wi] N then
8: Clu] <- W( [lu.] via Eq. (4)
9: else if u ((uu, wa]) then
+ Z,, Zong else
IZ..| = =Nz, YY,| = Ny and = Nz + Ny. We mate- rialize fa w.r.t. integration points ž, effectively creating an N N-dimensional tensor W(1)) E R x-- x K" such that
11:
C[u) + L[u.) O Gluz]
12:
return QPC with L(c] as root layer
W
f fu((Zi=ă,,.
ZNz=ās 1Y 2 YN. = - }).
After materializing tensor W((), we flatten it w.r.t. variables Z, and Y.,. so as creating a matrix (()) of size KN2 x KN an operation aka matricization. As last step, we plug-in the quadrature
5

---

weights W E RK in W("), arriving to matrix W(") of K N2 x Ny,y whose i-th row is
[a)
w@Ny
(M)
(3)
where w is the vector of size K-NY resulting from the Ny-timess application of the Kronecker product @ over w (Line 6, Algorithm 3). We illustrate this process in Fig. 3(a-c). Similarly, we also materialize every PIC sum unit with weights {w; Ti=1 sum layer, but parameterized by
(4) where I the K K identity matrix and the concatenation operator (Line 8, Algorithm 3). Note that such sum layer can be seen as mixing layer [42, 36]. Finally, consider a PIC product unit with inputs μ and ug, each outputting functions with LVs Z., and Zn, respectively. We associate to an Hadamard product layer if Z,,- Zuy. Kronecker product layer if Zu, & Zuz. reflecting the fact that we are marginalizing out two different LVs. We summarize our PIC materialization illustrated in Fig. in Algorithm 3, where iteratively associate PIC unit to circuit layer, eventually delivering a tensorized QPC. We stress that being QPCs just standard PCs they enjoy their same properties (e.g. tractable marginalizaation. We will learn PICs via maximizing the likelihood of its QPC materialization.
w(
||ا= IK E R RKXNK
QPCs as existing tensorized architectures. Materializing PICs built via Algorithm 1 delivers ten- sorized PCs with alternating sum and product layers, aka sum-product layers [36]. An instance of such layers is the Tucker layer, used in architectures like RAT-SPNs [43] and EiNets [42]. Specifi- cally, a binary Tucker layer & [51] computes
((X+) W (G((e,) (Xx,))
(Tucker-layer)
where W R and L, l2 are input layers of l, each outputting a K K-dimensional vector. In contrast, the recent HCLT architectures [31] use the canonical polyadic (CP) layer é [2], i.e
e(X,) = (xx))=(wW)/p) (w())((X))) + ( (w(2) (2
(CP-layer)
where W()), W E RKK, We exactly recover Tucker (resp. CP) layers in our QPCs when these materialized from PICs built via Tucker-merge (resp. CP-merge) in Algorithm 2, and hence the name of the merging procedure. Therefore, some QPCs can exactly match existent tensorized ar- chitectures, and this certainly happens when these are materialized from PICs built via Algorithm 1. This gives a new point of view on traditional tensorized architectures, and new possibilities for rep- resentation learning [54]. Figure 3 illustrates how the materialization of 3-variate function leads to a Tucker layer. This 1-to-1 mapping between tensorized PC architectures and QPCs will allow for fair comparison in our experiments.
Folding tensorized circuits for faster inference. The layers of a tensorized circuit that (i) share the same functional form and that (ii) can be evaluated in parallel, can be stacked together as create which speeds up inference and learning on GPU by orders of magnitude. For instance, let {e.),= be F parallelizable Tucker layers each parameterized by matrix W of size K x K22 Such layers can be evaluated as a folded layer l parameterized by tensor W of size FxXxX* R2, which computes otherwise sequential-F tucker layers in parallel. We illustrate folding in Fig. 2(d), and later on in Fig. 4(c). Note that (i) the input layers sharing the same function form can always be folded and that (ii) although a tensorized circuit may have many types of sum-product layers, using one type only is common in practice, and promotes depth-wise folding.
a folded layer [36, 42]
3.3 Scaling PICs with neural functional sharing
Materializing QPCs can be memory intensive and time consuming, depending on: (i) the cost of evaluating the functions we need to materialize, (ii) the degree of parallclization of the required function evaluations, and (iii) the number of integration points K. To solve these issues, intro- duce neural functional sharing [47], i.e. we share multi-layer perceptrons as to parameterize multiple PIC units once. This allows to scale larger models and datasets, faster and more memory-effiiient than previous work [18].
make materialization
PIC functional sharing, Functional sharing is to PICs as parameter-sharing is to PCs. This type of sharing can be applied over a group of input/integral units grot-ee according to some criteria- whose functions have all the same number of input and output variables. Specifically, let y =

---

(4)
(6)
DLllIZ)) (2.1)
S
OolliZ2 (2
22
21
FF-MLP
DAltZE) {Z
(1112)), (2,)
+
W
hm7
(5)
A
A
fi(X1, Z1)
fx(2; Z1)
(a( Zi)
(b)
(c)
Figure 4: From neural C-sharing to folded CP-layer (Section 3.3). We sketch 4-headed MLP (a) which we use to parameterize a group of integral units (at the same depth) a a PIC (b), whose materialization leads to a folded CP-layer parameterized by a tensor W of size 2 x x K x K (c), with K being the number of integration point. Note that, during materialization, the FF-MLP block in (a) will be only evaluated K" times, and not 4K².
with
[= be a group of N input/integral units, each with function fi R - =R. The simplest form of functional sharing is to set all functions to be equal, i.c Vi.j [N] : = In this way, reduce the number of function evaluations from NK to K long as materialize each fi w.r.t. the same integration points & E R which the case for Algorithm 3. We call this type of per full-sharing. More interestingly, leveraging functional composition, may define fi = h42 o so as sharing an inner function f for all unit functions. Similarly as before, long as materialize each fa w.r.t. the same quadrature points K function evaluations for I instead of N K, as we can share them with all outer functions h; for further evaluation. We call this type of sharing C-sharing, as per composite-shuring. The original implementation of PICs [18] used neither F-sharing nor C-sharing.
sharing F-sharing,
R we would only need
Finally, we present and apply two different ways of grouping units. The first consists of grouping all input units, a technique which is only applicable when all input variables share the same domain. With this grouping, coupled with F-sharing, we would only need to materialize K x P parameters, and use them to parameterize every QPC input layer. The second consists of grouping all integral units at the same depth of the PIC structure, which folded sum-product layer. Despite grouping units that materialize into folded layer is natural and convenient choice, note that can also group units that do not materialize as such. Once all units in PIC have been grouped, materialization can be performed per-group.
couple with C-sharing and materialize
PIC functional sharing with (multi-headedt MLPs. Similar to [18], parameterize PIC input and integral units with light-weight multi-layer perceptrons (MLPs). However, instead of using a single MLP for each function, we will apply functional sharing
strive make the QPC mate- rialization faster and memory efficient. Specifically, consider a group of integral units 7 = (us N each with function fo R R, over which we want to apply functional sharing. For every group y, we would have an L + 1 layered MLP the form:
x67) R R = of o
(7) FF,
(5)
where FF IR
M is Fourier- layer [48], and each ((²²) RM RM is standard linear
layer followed by an element-wise non linearity th iee. ((Az + b), with A E MM b E RM, and M being the size of the MLP. Applying F-sharing over y would simply consist of setting
f R
+ R softplus(h ((() b(7)),
(neural F-sharing)
where h7) E M and ((7) E are group-dependent parameters, therefore making all functions in the group equal. Instead, to implement C-sharing, we parameterize each f; as
fs TR
RR: = softplus(h()) ((1)) + b()),
(neural C-sharing)
parameters, effectively creating a multi- headed MLP. As an example, consider a folded CP-layer with F - 500 and 51- 1511-hhichh we actually used in practice-resulting resulting
where (() E RM and ((0 R are function-depente
262M trainable parameters. Assuming no bias term, an MIP with I- and M = 256 would only instead require LM2 + 2FM as 387K trainable parameters to materialize the same tensor, resulting in more than 99% less trainable parameters. We illustrate such C-sharing in Fig. 4. In Appendix C.I provide more details about our MLPs.
2F
7

---

CP. QG-CP, QG-TK). To the right, Figure 5: Learning PICs using functional sharing requires (i) com- parable resources as PCs and (ii) up to 99% less trainable parameters. We compare the GPU memory (top- left) and time (bottom-left) required to perform an optimization step with PCs (0), PICs with functional sharing (.) and without (a), while consider- ing three different architectures (QT-
GPU Memory (GIB)
Num, trainable parameters
25
Km 16
32
64
125
236
K-
18-
Time (ms)
we report the number of trainable pa- rameters for (i) PCs (0) at different K, and (ii) for PICs (A, at differ- ent MLP sizes M. The isolated nodes refer to refer to PIC (F, N) with QG-TK which we could only run at 16 The benchmark is conducted using a batch of 128 RGB images of size 64x64 and Adam [23]. Extra de- tails in Appendix D.1.
10- 32
64
128
256
100
2M-
32
54
128
256
32
64
126
266
QNCP
QS-CP
OP-K
PE
MI IF.N)
PCIP.
Fast & memory-efficient QPC materialization. Combing PIC functional sharing and per-group materialization allows scaling the training of PICs via numerical quadrature, as we drastically reduce the number and the cost of function evaluations required for the QPC materialization. We can now materialize very large QPCs, matching the scale of recent over-parameterized PCs yet requiring up to 99% less trainable parameters when using large K. This was not possible in the original formulation of PIC [18] as (i) the entire QPC was materialized in one-shot, not per-group, and (ii) no functional sharing was implemented, as each input/integral function had its own MLP.
4 Experiments
In our experiments, we first benchmark the effectiveness of functional sharing for scaling the training of PICs via numerical quadrature, comparing with standard PCs and PICs w/o functional sharing [18]. Then, following prior work [10, 32, 33, 18], we compare QPCs and PCs as distribution esti- mators on several image datasets. We always train using the trapezoidal integration rule. We use an NVIDIA A100 40GB throughout our experiments. Our code is available at hethub..oo/ggngalaattenn
pics.
Thanks to our pipeline, we can now use two recently introduced RGs tailored for image data which deliver architectures that scale better than those built out of classical RGs [45, 43, 31]: quad-trees (QTs), tree-shaped RGs, and quad-graphe (QGs), DAG-shaped RGs [36]. These are perfectly bal- anced RGs, and therefore applying Algorithm over them would deliver balanced PIC structures amenable to depth-wise C-sharing of integral units. We report full details about QTs and QGs in Appendix B. We denote a tensorized architecture as [RG]-[sum-prooee 16, which can be trained as standard PC or materialized as QPC from PIC. We treat pixels as categorical variables, and, as such, our architectures model probability mass functions.
layer]-[K], e.g. QT-CP-
For each model type, (PC, PIC), we specify a pair (-.:) where the first (resp. second) argument specifies the sharing technique, {F. C. N), for the input (resp. inner) layers/groups, where N stands for no sharing. In Fig. 5, we report the time and GPU memory required to perform an Adam [23] optimization step using PCs (0), and PICs with (m) and without (a) functional C-sharing over the integral unit groups. We note that PICs using functional sharing proves very effective for scaling, requiring comparable resources as standard PCs (0), while those who do not (a)- like prior work [18]-are are orders of magnitude slower and quickly go Out-Of-Memory (OOM) for K > 64. Remarkably, some QG-TK configurations
Scaling PICs.
PICs (o), see Table D.2, require even less GPU memory than PCs, and this is because of the significant difference in the number of trainable parameters, since copies of these have to be stored by Adam during optimization. fact, the number of parameters for PCs and PICs w/o functional sharing is in the order of hundreds of millions
8

---

QPC
PC
Sp-PC
HCLT
RAT
IDF
BitS
BRans
McB
35
MNIST
1.11
1.17
1.14
1.21
1.67
1.90
1.27
1.39
1.98
F-MNIST
3.16
3.32
3.27
3.34
4.29
3,47
3.28
3.66
3.72
100
290
EMN-MN
1.55
1.64
1.52
1.70
2.56
2.07
1.88
2.04
2.19
8 122
EMN-LE
1.54
1.62
1.58
1.75
2.73
1.95
1.84
2.26
3.12
EMN-BA
1.59
1.66
1.60
1.78
2.78
2.15
1.96
2.23
2.88
EMN-BY
1.53
1.47
1.54
1.73
2.72
1.98
1.87
2.23
3.14
PC
0.00
bpd
Figure 6: QPCs improve over PCs and other DGM baselines in terms of test-set bpd for the MNIST-family datasets. We compare against SparsePC [10], HCLT [31], RAT-SPN [43], IDF [21], BitSwap [25]. BBans [50] and McBits [46]. HCLT results are taken from [18]. Columns QPC and PC report results from this paper, with QG-CP-512 being the best performing architecture for both. Scatter plot (right): bpd for QPCs (y-axis) and PCs (x-axis) paired by architecture and min-max normalized for the MNIST and F-MNIST datasets. A point below the diagonal is a win for QPCs.
QPC"
PC'
QPC
PC
HCLT
LVD'
LVD-PG
Table 1: QPCs improve over PCs. We mark results with for YCoCg-R and t for YCoCg. QG-CP-512 (resp. QG- CP-256) the best performing archi- tecture for QPCs and PCs on CIFAR & ImgNet32 (resp. ImgNet64 & CelebA).
CIFAR
5.09
5.50
4.48
4.85
4.61
4.37
3.87
Img
5.08
5.25
4.46
4.63
4.82
4.38
4.06
1et64
5.05
5.22
4.42
4.59
4.67
4.12
3.80
CelebA
4.73
4.78
4.11
4.16
(hitting 2B+), while PICs with functional sharing (m) scale much more gracefully, hitting only 6M+ parameters. We emphasize that the number of trainable parameters of PICs is independent of the K at which we materialize, but only dependent on the size of the MLPs M we use to parameterize them, which can also be thought as the cost of evaluating PIC functions. We report more (tabular) details in Appendix D.I.
Distribution estimation. Following prior work [10, 32, 33, 18], we extensively test QPCs and PCs as distribution estimators on standard image datasets. Our full results are in Appendix D.2, while we only report here the tis-pee--dimension (bpd) of the best performing models, which always belong to a QG-CP architecture, reflecting the additional expressiveness of DAG-shaped RGs. Our full results also highlights how the the more expressive yet expensive Tucker layers we introduced for PICs deliver the best performance for small K, but are hard to scale. All QPCs are materialized from PICS applying F-sharing over input units and C-sharing over groups of integral units, i.e. PIC (F, C). We begin with the MNIST-family, which includes datasets of gray-scale 28x28 images: MNIST [29], FASHIONMNIST [55]. and EMNIST with its 4 splits [7]. Fig. 6 shows that QPCs generally perform best, improving over standard PCs (5/6), complex heuristic-based PC learning schemes as pruning-and-gen and-growing (5/6) [10], and some deep generative models (DGMs) (6/6).
Then, we move to larger RGB image datasets as CIFAR [28], ImageNet32, ImageNet64 [14], and CelebA [34]. To compare against prior work [32, 33], we have to preprocess the datasets using the YCoCg transform, a lossy color-coding that consistently improves performance for PCs when ap- plied to RGB images. We also report results over datasets preprocessed with the lossless YCoCg-R transform [39], effectively doubling the number of datasets. We report details about these trans- forms in Appendix C.3. From Table 1,
that QPCs prove again very competitive, consistently outperforming standard PCs commonly trained with Adam, and the best performing PC from the lit- erature, HCLI, which is trained via EM schemes and patch-wiee wise methods [32]. Furthermore, QPCs are close to PCs trained via latent variable distillation (LVD, LVD-PG in Table 1) [32, 33], frame- work that requires extra supervision over their latent spaces by distilling information from existing deep generative models (DGMs). This technique requires pre-trained DGMs, several heuristics, and a final fine-tuning stage via EM or SGD, while PIC training method is instead end-to-end and self-contained.
'The use of the lossy YCoCg transform is undocumented in [32, 33J but confirmed via personal communi- cation with the authors. Note that columns in Table using it (f) are not directly comparable to the rest.

---

5 Discussion & Conclusion
With this work, we systematized the construction of PICs, extending them to DAG-like structures (Section 3.1), tensorizing (Section 3.2), and scaling their training with functional sharing (Sec- tion 3.3). In our experiments (Section 4), we showed how this pipeline is remarkably effective when the tractable approximations of PICs, QPCs, are used distribution estimators. This in turn be- comes new and effective tool to learn PCs at scale. In fact, prior work has shown that naively training large PCs via EM or gradient-ascent is challenging, and that PC performance plateau as their size increases [10, 8, 32, 33]. Our contributions go beyond these limitations, while offering simple, principled and fully-differentiable pipeline that delivers performance that rival more phisticated alternatives [10, 32] (Section 4). We conjecture that this happens as training PCs via PICs drastically reduces the search space while allowing (i) smoother training dynamics and (ii) the materialization of arbitrarily large tractable models.
The development of tractable models is an important task in machine learning they provide many inference routines, and can be used in many down-stream applications such as tabular data modeling [8], generative modeling [42], lossless compression [30], genetics [11], knowledge-grape
[37], constrained text generation [58], and more. Our work has also certain parallels with tensor networks and (quasi- ttnsor decompositions [51, 26, 22, 6, 49], as [36] recently showed how hierarchical tensor decompositions can be represented using the language of tensorized circuits. Furthermore, note that the recent non- monmonoto PCs [35] (i.c., PCs with negative sum parameters) can also be thought as the result of quadrature process from PICs whose function can return negative values.
Our work does not come without limitations. Although we showed that training PICs with function sharing requires comparable resources as standard PCs, traditional continuous LV models as VAEs, flows and diffusion models are more scalable. Also, sampling from PICs is currently not possible, as we cannot perform differentiable sampling from our (multi-headed) MLPs. Future work may include the investigation of more efficient ways of training PICs, possibly using techniques as LVD or variational inference [24] to directly maximize PIC lownddbounds, requiring numerical quadrature only fine-tuning step distill performant tractable model. We believe our work will foster new research in the field of generative modeling, and specifically in the realm of tractable models.
Author Contributions
GG led the project, proposed neural functional sharing, and ran all experiments. GG and AV devised the original idea of a pipeline to build PICs, and leverage tensorized folded circuits for training them. AV and EQ equally supervised all the phases of the project. CdC supervises the project, and critically read the manuscript and provided feedback.
Acknowledgments
The Eindhoven University of Technology authors thank the support from the Eindhoven Artificial Intelligence Systems Institute and the Department of Mathematics and Computer Science of TU Eindhoven. TUle authors also thank the support of EU European Defence Fund Project KOIOS ----0022-DIII--------
AV was supported by the "UNREAL: Unified Reasoning Layer for Trustworthy ML" project (EP/Y023838/I) selected by the ERC and funded by UKRI EPSRC. We thank Lorenzo Loconte for insightful discussions about (quasi-)tensor decompositions.
References
[1] Cory Butz, Jhonatan S Oliveira, and Robert Peharz. Sum-product network decompilation. In International Conference on Probabilistic Graphical Models, pages 53 64. PMLR, 2020.
[2] J. Douglas Carroll and Jih-Jie Chang. Analysis of individual differences in multidimensional scaling via an n-way generalization of "eckart-youngg decomposition. Psychometrika, 35:283- 319, 1970.
[3] Myung Jin Choi, Vincent F. Tan, Animashree Anandkumar, and Alan S Willssy. Learning latent tree graphical models. Journal of Machine Learning Research, 1714911171--181, 2011
10

---

[4] YooJung Choi, Antonio Vergari. and Guy Van den Broeck. Probabilistic circuits: A unifying framework for tractable probabilistic models. Technical report, UCLA, 2020.
[5] C. Chow and C. Liu. Approximating discrete probability distributions with dependence trees. IEEE Transactions on Information Theory, 14(3):462-467, 1968.
[6] Andrzej Cichocki and Anh Huy Phan. Fast local algorithms for large scale nonnegative ma- trix and tensor factorizations. IEICE Trans. Fundam. Electron. Commun. Comput. Sci., 92- A(3):708-71,, 2009.
[7] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: Extending MNIST to handwritten letters. In IJCN 20/7, pages 2921-2926, 2017.
[8] Alvaro Correia, Robert Peharz, and Cassio P de Campos. Joints in random forests. In Advances in Neural Information Processing Systems, volume 33, pages 11404-1145, 2020.
[9] Alvaro C. Correia, Gennaro Gala, Erik Quaeghebeur, Cassio de Campos, and Robert Peharz. Continuous mixtures of tractable probabilistic models. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6):7244-7255 2023.
[10] Meihua Dang, Anji Liu, and Guy Van den Broeck. Sparse probabilistic circuits via pruning and growing. In NeurlPS 2022, volume 35 Advances in Neural Information Processing Systems, 2022.
[11] Meihua Dang, Anji Liu, Xinzhu Wei, Sriram Sankararaman, and Guy Van den Broeck. Tractable and expressive generative models of genetic variation data. In Research in Com- putational Molecular Biology, pages 356-357, 2022.
[12] Adnan Darwiche. Modeling and reasoning with Bayesian networks. Cambridge University Press, 2009.
[13] Philip J. Davis and Philip Rabinowitz. Methods of numerical integration. Academic Press, 1984.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 245-255. leee, 2009.
[15] Aaron W. Dennis and Dan Ventura. Learning the architecture of sum-product networks using clustering on variables. In Advances in Neural Information Processing Systems 25 (NeurIPS), pages 2033-2041. Curran Associates, Inc., 2012.
[16] Nicola Di Mauro, Gennaro Gala, Marco lannotta, and Teresa Maria Altomare Basile. Ran- dom probabilistic circuits. In 37th Conference on Uncertainty in Artificial Intelligence (UAI), volume 161, pages 1682-1691. PMLR, 2021.
[17] Guido Fubini. Sugli integrali multipli. Rend. Acc. Naz. Lincei, 111608-614, 1907.
[18] Gennaro Gala, Cassio de Campos, Robert Peharz, Antonio Vergari, and Erik Quaeghebeur. Probabilistic integral circuits. In Proceedings of The 27th International Conference on Arti- ficial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 2143-2151 PMLR, 02-04 May 2024.
[19] Robert Gens and Pedro M. Domingos. Learning the structure of sum-product networks. In International Conference on Machine Learning, 2013.
[20] Ian Goodfellow, Jean Pouget-Abadin Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS 2014, volume 27 of Advances in Neural Information Processing Systems, 2014.
[21] Emiel Hoogeboom, Jorn Peters, Rianne van den Berg, and Max Welling, Integer discrete flows and lossless compression. NeurlPS 2019, volume 32 of Advances Neural Information Processing Systems, 2019.

---

[22] Yong- Bong-eeok Kim and Seungjin Choi. Nonnegative tucker decomposition. In CVPR. IEEE Computer Society, 2007.
[23] Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR 2015, 2015.
[24] Diederik P. Kingma and Max Welling, Auto-encoding variational Bayes. In ICLR 2014, 2014.
[25] Friso Kingma, Pieter Abbeel, and Jonathan Ho. Bit-swap: Recursive bits-back coding for loss- less compression with hierarchical latent variables. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3408-3417, 2019.
[26] Tamara Kolda. Multilinear operators for higher-order decompositions. Technical report, Sandia National Laboratories, 2006.
[27] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
[28] Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
[29] Yann LeCun, Corinna Cortes, and Christopher C. Burges. The MNIST database of hand- written digits, 2010.
[30] Anji Liu, Stephan Mandt, and Guy Van den Broeck. Lossless compression with probabilistic circuits. In ICLR 2022, 2022.
[31] Anji Liu and Guy Van den Broeck. Tractable regularization of probabilistic eircuits. In NeurlPS 2021, volume 34 of Advances in Neural Information Processing Systems, pages 3558- 3570, 2021.
[32] Anji Liu, Honghua Zhang, and Guy Van den Broeck. Scaling up probabilistic circuits by latent variable distillation. In ICLR 2023, 2023.
[33] Xuejie Liu, Anji Liu, Guy Van den Broeck, and Yitao Liang. Understanding the distillation process from deep generative models to tractable probabilistic circuits. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 21825 0025-21838, 2023.
[34] Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[35] Lorenzo Loconte, M. Sladek Aleksanteri, Stefan Mengel, Martin Trapp, Amo Solin, Nicolas Gillis, and Antonio Vergari. Subtractive mixture models via squaring: Representation and learning. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[36] Lorenzo Loconte, Antonio Mari, Gennaro Gala, Robert Peharz, Cassio de Campos, Erik Quaeghebeur, Gennaro Vessio, and Antonio Vergari. What is the relationship between ten- sor factorizations and circuits (and how can we exploit it)? arXiv preprint arXiv:2409.07953, 2024.
[37] Lorenzo Loconte, Nicola Di Mauro, Robert Peharz, and Antonio Vergari. How to turn your knowledge graph embeddings into generative models via probabilistic eircuits. Advances in Neural Information Processing Systems 37 (NeurIPS). Curran Associates, Inc., 2023.
[38] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR 2017, 2017.
[39] Henrique Malvar and Gary Sullivan. Ycoeg-r: color space with rgb reversibility and low dynamic range. ISO/EEC JTCI/S22//GI1 and SG16 Q. 6, 2003.
[40] Alejandro Molina, Antonio Vergari, Nicola Di Mauro, Sriraam Natarajan, Floriana Esposito, and Kristian Kersting. Mixed sum-product networks: A deep architecture for hybrid domains. In AAAI Conference on Artificial Intelligence, 2018.
12

---

[41] Robert Peharz, Robert Gens, Franz Pernkopf, and Pedro Domingos. On the latent variable interpretation in sum product networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 59(10):2030-2044 2017.
[42] Robert Peharz, Steven Lang, Antonio Vergari, Karl Stelzner, Alejandro Molina, Martin Trapp, Guy Van Den Broeck, Kristian Kersting, and Zoubin Ghahramani. Einsum networks: Fast and scalable learning of tractable probabilistic circuits. In 37th International Conference Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pages 7563-7574. PMLR, 2020.
[43] Robert Peharz, Antonio Vergari. Karl Stelzner, Alejandro Molina, Xiaoting Shao, Martin Trapp. Kristian Kersting, and Zoubin Ghahramani. Random sum umm-product networks: A simple and effective approach probabilistic deep learning. Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Re- search, pages 334-344,2 2020.
[44] Knot Pipatsrisawat and Adnan Darwiche. New compilation languages based on structured decomposability. In Proceedings of the 23rd National Conference on Artificial Intelligence (AAA/'08), volume 1, pages 517-522, 2008.
[45] Hoifung Poon and Pedro Domingos. Sum Sum-product networks: A new deep architecture. In IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pages 689-690. IEEE, 2011.
[46] Yangjun Ruan, Karen Ullrich, Daniel S. Severo, James Townsend, Ashish Khisti, Arnaud Doucet, Alireza Makhzami, and Chris Maddison. Improving lossless compression rates via monte carlo bits-back coding. In Proceedings of the 38th International Conference Ma- chine Learning, volume 139 Proceedings of Machine Learning Research, pages 9136-9147, 2021.
[47] Eran Segal, Dana Pe'er, Aviv Regev, Daphne Koller, Nir Friedman, and Tommi Jaakkola. Learning module networks. Journal of Machine Learning Research, 6(4), 2005.
[48] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let net- works learn high frequency functions in low dimensional domains. In NeurIPS 2020, vol- ume 33 of Advances in Neural Information Processing Systems, pages 7537-7547, 2020.
[49] Alex Townsend and Lloyd N Trefethen. Continuous analogues of matrix factorizations. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 471(2173):20140555 2015.
[50] James Townsend, Thomas Bird, and David Barber. Practical lossless compression with latent variables using bits back coding. In ICLR 2019, 2019.
[51] L. R. Tucker. The extension of factor analysis to direeeiimenional matrices. In Contributions mathematical psychology., pages 11-1177 Holt, Rinehart and Winston, 1964.
[52] Antonio Vergari, YooJung Choi, Anji Liu, Stefano Teso, and Guy Van den Broeck. A com- positional atlas of tractable circuit operations for probabilistic inference. In NeurIPS 2021, volume 36 of Advances in Neural Information Processing Systems, 2021.
[53] Antonio Vergari, Nicola Di Mauro, and Guy Van den Broeck. Tractable probabilistic models: Representations, algorithms, learning, and applications, 2019. Tutorial at the 35th Conference on Uncertainty in Artificial Intelligence (UAI 2019).
[54] Antonio Vergari, Robert Peharz, Nicola Di Mauro, Alejandro Molina, Kristian Kersting, and Floriana Esposito. Sum-product autoencoding: Encoding and decoding representations using sum-product networks. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018.
[55] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for benchmarking machine learning algorithms. arXiv, 2017.
13

---

[56] Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsune Hsuan Yang. Diffusion models: A comprehensive survey of meth- ods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
[57] Yang Yang, Gennaro Gala, and Robert Peharz. Bayesian structure scores for probabilistic circuits. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 563-575, 2023,
[58] Honghua Zhang, Meihua Dang, Nanyun Peng, and Guy Van den Broeck. Tractable control for autoregressive language generation. In 40th International Conference on Machine Learn- ing (ICML), volume 202 of Proceedings of Machine Learning Research, pages 40932-40945. PMLR, 2023.
14

---

A Background on Circuits
Definition 3 (Circuit [4, 52]). A circuit over variables X is parameterized computational graph encoding a function ef ), and comprising three kinds of computational units: input, product, and Each product or unit u, outputs scalar and receives imputs the output scalars of other units, denoted with the set in(u). Each unit u computes a function fu defined as: (i) (X.)
if an input unit, where fu is function over variables X.. X, called its scope, (ii) [ if u is product unit, and (iii) Lienn()) fX.) ) if is sum unit, with w, E R denoting the weighted sum parameters. The scope of a product or sum unit is the union of the scopes of its input units.
Definition 4 (Probabilistic Circuit). A PC over variables X is a circuit encoding (possibly non- normalized) distribution, e.g,, a function that is non-negative for all values of X:
c(x) 2 O, Vx E dom(X)
Definition (Smoothness). A circuit is smooth for each sum unit u, its inputs depend on the same variables: Vu1, U2 E in(u), Xu, Xu..
Definition 6 (Decomposability). A circuit is decomposable if the inputs of each product unit depend on disjoint sets of variables: Vui, U2 E in(u), Xu, & Xuz-
Definition 7 derruturedddecompsaabiiee
[44, 12]). A circuit is druccured-decoomosabbe (i) is smooth and decomposable, and (2) any pair of product units having the same scope decompose their scope at their input units the same way.
Although all tensorized architectures mentioned in this paper are smooth and decomposable, only PCs built from tree RGs are also pructured-decompontt and as such are potentially less expressive because they belong a restricted class.
B Region Graphs
In Algorithm B.1 we detail the construction of the Quad-Tres (QT) and Quad-Graph (QG) region graphs [36]. Specifically, QT's (resp. QGs) are built setting the input flag Tree to True (resp. False). Intuitively, these RGs recursively split an image into patches, until reaching regions associated to exactly one pixel. The splitting performed both horizontally and vertically, and subsequent patches can either be shared, thus yielding a RG that is not a tree (QGs), or not (QTs).
Algorithm B.1 buildQuadGGrahh(1. H, W, isree)
Algorithm B.2 merge (R, 22
Input: Image height H. image width W. and whether to
Input: A RG R. a set of four coordinates Ω and set of regions S
enforce the output RG to he a tree.
Output: A RG R over H- W variables
It merges the regions indexed by Ω in R by forming tree structure
Behavior:
I: S + (Xij } (i,j) E [H] [W]}
2: R <- a RG with leaf regions S
1: Let Xuv Yptu a+v S S.t.
3: h H:; w W
2:
(p u. v) S u, E {0, 1}
4: while > 1 V w > do
3: X + X Xx U x10 U X11
5:
/t < [h/2]; w < [w/2]; S' <
4: addPartition X, (Xoo. Xn1. X10, X11)
6:
i.j E [h] [w] do
7:
Q Qx- (1-i-1211,,2 2i (2j (2ii
8:
if [ then
Algorithm B.3 mergeDAG 22
9:
Let %pq E S st. (p.q) E addRegion RR, Xpp)
Input: A RG R, a set of four coordinates Ω, and set of regions S
10:
11:
else if [ = then Let Xpg X S.L.
It merges the regions indexed by Et in R by forming DAG structure
Behavior:
12:
13:
(p. q). (r, Ω p p<r,9<>
14:
addPartition Xpg U..., (Xpg Xxr))
1: Let Xuv V pt g+v S s.t.
15:
else > |2] =
2:
(p - u. + v) E 2 u, E {0, 1}
16:
if Tree then merge St, else mergeDAG(R, S)
3: x < Xno U Xo U X10 U
4: addPartition(R.. x, {%oo UXo X UX
17:
5: addPartition R, X, [Xoo Xm UXn })
18:
X,j Unaaen X S.l. X S
6: addPartition R, X Xo1, (Xoo, Xo1
19:
S' - S' S'U {X1}
7: addPartition (R, X10 U X (K X11
20:
S'
8: addPartition (R, X U X10, (%oo, X
21:
return R
9: addPartition (R, Xo1 X11, (xn,, X11))
15

---

C Implementation details
C.1 Multi-headed MLP details
A multi-headed MLP of size M parameterizing a group y= [= of N PIC units with functions of the form R + R consists of:
1. A Fourier-Features Layer (details below), i.e. a non-linear mapping E' + RA
2. Two linear layers followed by hyperbolic tangent activation function, i.c. two consecu- tive non-linear mappings R.M + RRM
3. N heads with Softplus non-linearity, i.e. N different non-linear mapping RM RR.
Note that, if y is group of CP (resp. Tucker) integral units, then 7 = 2 (resp. T= 3), while the output dimension O is always equal to Instead, if the group a group input units, the input dimension is always equal to 1, while the output dimension O is equal to the number of required parameters of the specific distribution, e.g. 2 for Gaussians.
Such multi-headed MLP is implemented using grouped ID convolutions, which allow one-shot materialization of all the layer parameters associated the group. We found that initializing all the heads to be equal improves convergence.
Fourier Feature Layer. Fourier Feature Layers (FFLs) important ingredient for the multi- headed MIPs. FFLs [48] enable MIPs to learn high-frequency functions in low-dimensional prob- lem domains and are usually used as first layers of coordinate-based MLPs. FFLs transform input z E IR
to
FFL(Z): RR' + M = [cos (2of(2 (2nf(2
21/2 (in h))
where M
and vectors f; E TR. are non-learnable, randomly initialized param- eters. FFLs have two main benefits: (i) They allow learning more expressive functions by avoiding over-smoothing behaviors, and (ii) they reduce the total number of trainable parameters when used instead of conventional linear layers as the initial layers in MLPs.
C.2 Training details
We train both PICs and PCs using the same training setup. Specifically, for each dataset, we perform training cycle of T optimization steps, after which we perform validation step and stop training if the validation log-likes likelihoodd did not improve by nats after 5 training cycles. Using >> can avoid long trainings with negligible improvements. We report these common training hyper-parameter in Table C.I. We use Adam [23] and a batch size of 256 for all experiments.
PIC training, After some preliminary runs, we found that a learning rate of 5 x 10-3 worked best, which we annealed towards 10-4 using cosine annealing with warm restarts across 500 optimization steps [38]. We also apply weight decay with X = 0.01.
PC training. After some preliminary runs, we found that constant learning rate of 0.01 worked best for all PC models, and for all datasets. We keep the PC parameters unnormalized, and, as such, clamp them small positive value (10-19) after each Adam update to keep them non-negative, and subtract the log normalization constant to normalize the log-likelihoods.
Table C.I: Common training hyper-parameet for PICs and PCs. max num epochs
dataset
7
0
MNIST-family excl. EMNIST-BY
200
250
EMNIST-BY
100
1000
0
CIFAR
200
250
ImageNet32
50
2000
10
ImageNet64
50
2000
30
CelebA
200
750
10
16

---

C.3 YCoCg color-coding transforms
In Fig. C.I and Fig. C.2 provide pytorch code for the lossless and lossy versions of the YCoCg transform that we used in our experiments (Section 4). In Fig. C.3, we show how apply them and that the lossy version is on average off less than a bit. Finally, in Fig. C.4 we show the significant visual difference of the two transforms when applied to an RGB image.
def @bb2ycc_losslesss rgb_img: torch. Tensor
def ycc2rgb_lossless ycc_img: torch. Tensor
assert rgb_ing. size s-1e ==
assert ycc. ycc_ing.sizer (-1) ==
def forward, lift(x, y): diff = (y x) %
def reverse wwweeellttteeee (average,, diff):
x (average (diff 1)) 256
average (x + (diff >> 1)) % 256 return average, diff
y = (x diff) % 256
return
red rgb_ing(.... green rgb img[ 1]
y ycc_img ...
co yec_ img[. 1]
blue = rgb_ ingl. [..., 2]
cg ycc img[....... 2]
temp, forward lift (red, blue)
green, temp reverse_lift(f cg) red, blue reverse_lift co)
y, cg forward_lift temp) ycc_img torch.stack [y, co, cgl, dim
rgb_img torch.stack( [red, green, bluel, dim
return yce_ing
return rgb_ing
Figure C.I: Lossless YCoCg transform (aka YCoCg-R (39]). We attach pytorch code for the RGB - YCoCg direction (left) and YCoCg - RGB direction (right), where one is the inverse of the other. The input to both functions a tensor with discrete values in [0, 255], so their output.
def rob2ycc_lossy( rgb_img: torch. Tensor
def ycc2rgb_lossy ycc_ing: torch. Tensor
assert rgb_img size (-1) 3
assert ycc_ing size (-1) =
deq_img (rgb_img img / 127.5
deq leq_img_ (ycc_imp img / 127. .)
red (deq [ ..., 0] 1)
y deq leg_imgl__. imgl.... ...
green (deq ingl. .., 1] 1) /
co deq img[..... 1]
blue - (deq img 2] + +) /
cg deq img ..... 2]
red blue tmp blue co
y (y 1)
tmp y cg / 2 green cg tmp blue tmp
cg green tmp
y tmp / 2
y
red blue
transformed, ing torch. (tack(
traneformed img torch.stack( [red, green, blue], dim=-1
[y, co, egl, dim
ycc_img torch. floor ((transformed__ng img 1) 2) 256
rgb_ime torch floor (transformed_ ling_iimmm--- * 255)
) (lngg() c),pp(p 255)
) cl.p(O) return rgb_ing
c1,p(0, 255)
return ycc_img
Figure C.2: Lossy YCoCg transform. We attach pytorch code for the RGB - YCoCg direction (left) and YCoCg - RGB direction (right). The two functions do not represent a bijection. The input to both functions is a tensor with discrete values in [0, 255], so their output.
17

---

batch_eize size 100
img_size = 32
agb_batch torch. randint (256, (batch_size, img_size img_size, 3))
ycc_ roc_batch__osslee.e 1oboycc. batch)
recon_ becon_rgb_bee rat.battchhhbbbeeeeeeeeeeeeeeeet lossless lossless pec2rgb_losslen (yceeesyyccc_eebceeee
print
=- rgb_batchi all()) True
ycc_batcch_lose gbbyycc__ batch)
rar.o__rg_baahhheeoo ycc2rgb_lossy (aapar/____aayycc..aat_____aaa batec_baahc_____yarp print
baan__rb_bbatcc_ooeee abs () ffloat() mean()) around 0. 66
Figure C.3: Application the YCoCg transforms. We show that YCoCg-R R is indeed a bijection, and that the lossy YCoCg is average off less than bit.
Figure C.4: Visual difference of YCoCg-R and YCoCg. Given the RGB image to the left, we show the application of YCoCg-R in the middle and that YCoCg the right.
D Additional results
D.1 Scaling experiments
We report the time and GPU memory required perform and Adamp optimization step for several model configurations in Table D.1, Fig. D.1 and Table D.2.
GPU Memory (GIB)
Time
QT-CP
QG-Cr CP
QG: QG-TK
250
2.2M
K
270K
800K
6M
150
K=32
IM
3M
SIM
K=61
K 128
17M
51.1M
K=2
69M
204M
K512
277M
817M
16
32
12.8
256
512
64
1.2
00-TK
Figure D.I: Training PICs using functional sharing requires comparable resources as PCs. We compare the average GPU memory (left) and time (right) required perform Adam optimization step with PCs (blue) and PICs (green), varying region graph type, parameterizations, and K. We pair the plots with table reporting the number of parameters of PCs different K and PICs, with the latter being independent of K and allowing up to 99% less trainable parameters (QG-CP-512), The benchmark conducted using a batch size of 256 gray-scale images of size 28x28, i.e. MNIST-like. Extra (tabular) details in Table D.I.
18

---

Table D.I: Training PICs using functional sharing requires comparable resources as PCs. We report the time (in milliseconds, top) and GPU memory (in GiB, bottom) re- quired to perform an Adam optimization step on 256 MNIST-like images by varying: architec- ture ((QT-CP, QG-CP, QG-TK)), size ((2)) )==4), model ((PC, PIC)), and sharing technique 19
F,C, For each model we attach pair (+:-) where the first (resp. second) argument speci- fies the sharing technique for the input (resp. inner) layer(s).
PC
PC
PIC
PIC
PIC
RG-layer
K
(F. N)
(N,
N)
(F. C)
(C, C)
(F, N)
16
16
16
18
18
67
32
17
19
64
19
19
21
21
89
QT-CP
128
24
25
26
OOM
256
38
40
43
47
OOM
512
82
116
128
OOM
16
48
43
43
193
32
52
52
45
46
189
64
58
59
52
52
OOM
QG-CP
128
74
74
66
67
OOM
256
119
121
122
126
OOM
512
257
264
326
337
OOM
16
57
156
QG-TK
32
102
102
89
89
OOM
64
221
221
288
289
OOM
PC
PC
PIC
PIC
PIC
RG-layer
K
(F,N)
(N.N)
(F, C)
(C,
(F, N)
16
0.08
0.14
0.15
1.00
2.68
32
0.16
0.28
0.19
1.05
6.97
64
0.36
0.60
0.40
1.27
20.1
QT-CP
128
0.83
1.31
0.97
1.93
OOM
256
2.17
3.12
2.68
3.83
OOM
512
6.86
8.39
8.75
10.09
OOM
16
0.18
0.24
0.22
1.03
7.85
32
0.38
0.49
0.42
1.24
13.95
64
0.83
1.07
0.92
1.79
OOM
QG-CP
128
2.03
2.50
2.29
3.25
OOM
256
5.54
6.50
6.66
7.66
OOM
512
17.14
19.05
21.70
23.04
OOM
16
0.68
0.73
0.80
1.55
18.60
QG-TK
32
2.94
3.04
3.75
4.37
OOM
64
14.72
14.92
21.04
21.71
OOM
19

---

Table D.2: PICs using functional sharing require comparable resources as PCs to be trained, and scale much more gracefully than PICs without sharing. We report the time (in milliseconds, top) and GPU memory (in GiB, bottom) required to perform an Adam optimization step on a batch of 128 RGB images of size 64x64 by varying: architecture ((QT-CP. QG-CP, QG-TK)), size K and also the MLP size M ({2') of the PIC models. For each model we attach pair (::-) where the first (resp. second) argument specifies the sharing technique for the input (resp. inner) layen(s).
((2")=4)) )) model (CC), sharing technique
PIC (F, C)
PIC (F, N)
K
PC (F, N)
M = 256
128
64
32
M
256
128
64
32
16
37
40
38
330
203
94
59
32
51
53
49
53
51
388
161
124
72
QT-CP
64
74
78
77
OOM
OOM
245
120
128
126
126
124
126
126
OOM
OOM
OOM
OOM
256
246
257
253
249
251
OOM
OOM
OOM
OOM
16
77
73
71
71
70
OOM
550
238
129
32
98
94
92
88
89
OOM
762
312
150
QG-CP
64
135
132
130
128
123
OOM
OOM
OOM
264
128
215
213
209
208
201
OOM
OOM
OOM
OOM
256
428
449
440
433
425
0OM OOM OOM
OOM
16
117
99
98
98
95
OOM OOM 429
164
QG-TK
32
211
204
202
199
183
0OM 0OM OOM
OOM
PIC (F,
PIC (F, N)
K
PC (F, N)
M = 256
128
64
32
M = 256
128
64
32
0.67
0.70
0.68
0.67
0.67
13.81
5.03
2.14
1.13
32
1.23
1.27
1.25
1.24
1.23
31.45
12.54
6.15
2.94
QT-CP
2.48
2.57
2.52
2.50
2.49
OOM
OOM
22.59
10.97
128
5.48
5.74
5.61
5.54
5.51
0OM
OOM
OOM
OOM
256
13.48
14.45
13.96
13.72
13.60
OOM
OOM
OOM
OOM
16
0.71
0.78
0.74
0.72
0.72
OOM
12.68
4.88
2.10
32
1.40
1.50
1.44
1.42
1.41
OOM
27.72
12.93
6.00
QG-CP
64
3.15
3.35
3.24
3.20
3.17
0OM
OOM
OOM
22.13
128
8.15
8.74
8.44
8.30
8.22
OOM
OOM
OOM
OOM
256
24.14
26.29
25.22
24.69
24.42
0OM OOM
OOM
OOM
16
2.24
2.35
2.26
2.22
2.20
OOM
OOM
22.46
11.10
QG-TK
10.77
11.35
10.81
10.54
10.40
OOM
OOM
OOM
OOM
20

---

D.2 Additional distribution estimation results
In this section, we report tabular results for all our experiments.
Note that, every input layer of a standard PC is parameterized by matrix K x P. where P is the number of categories, which is 256 for grey-scale image datasets and 768 =56 3 for RGB images. We found that sharing single input layer among all pixels results in (slightly) worse performance for grey-scale images (as detailed in Table D.3). In contrast, sharing considerably improves performance for RGB image datasets. Besides improving perfor- mance for RGB image datasets, such sharing considerably lower the number of trainable param- eters from D x K x P to only K x P where D is the number of pixels. For instance, parame- terizing all imput layers of a tensorized architecture with K 256 built for 64x64 images would require 805, 306, 368 = 256- 64 64-768 parameters, while only 3, 145, 728 if we apply the shar- ing. Therefore, without applying such sharing, we cannot even scale to big tensorized architectures (e.g. QG-CP-512) on our GPUs.
found that such
All QPCs materialized from PICs applying F-sharing over the group of input units, and C-sharing over the groups of integral units.
We extensively compare QPCs and PCs density estimators several image datasets and report test-set bits-per-dimension (bpd) in Table D.3, Table D.4, and Table D.5.
Table D.3: PCs with a shared input layer deliver comparable performance as PCs who do not on the MNIST-family datasets. We compare the bits-per-dimensoon of PCs with (w/) and without (w/o) a shared input layer considering three different tensorized architectures: QT-CP-512, QG-CP- 512 and QG-TK-64.
QT-CP-512
QG-CP-512
QG-TK-64
w/o
w/
w/o
w/
w/o
w/
MNIST
1.175 + 0.001
1.213 0.03000000 0.002
0.006 0.006
1.241 = 0.005
1.257 + 0.005
1.300 + 0.004
F-MNIST
3.381 + 0.001
3.381 0.002
0.005 3.317 0.005 3.375
3.499
0.006
3.560
0.007
EMN-MN
1.706 + 0.007
1.761
1.643=0.000
11711=0000
1.756 + 0.002
1 0.004
EMN-LE
1.698 + 0.006
1.735
1.626 + 0.004
1.656
0.001
1.725 + 0.003
1.728 + 0.003
EMN-BA
1.731 0.007
1.772
0.007
1.665 + 0.004
1.696
+ 0.003
1.751
0.002
1.749
+ 0.005
EMN-BY
1.542 + 0.008
1.548
1.474 + 0.009
1.481 0.007
1.665 0.007
1.679 + 0.006
Table D.4: QPCs consistently improve over PCs on MNIST and FASHION Test-set bits- per-dimension (bpd) on MNIST (top) and MNSHI (bottom) averaged over runs. All QPCs are materialized from PICs applying F-sharing over the group of input units, and C-sharing over the integral units groups, i.e. QPCs are materialized from PICs (F. C). PCs do not apply any form of parameter sharing, as these deliver the best performance for these datasets, as detailed in Table D.3.
QT-CP
QG-CP
QG-TK
K
QPC
PC
QPC
PC
QPC
PC
16
1.275
1 0.009
1.283 1 0.004
1.237 1 0.009
1.248 1 0.003
1.215 1 0.010
1.233 1 0.004
32
1.220
+ 0.003
1.242
1.189 0.004
+ 0.004
+ 0.008
1.212 + 0.003
1.168 = 0.002
1.222 = 0.004
64
+.002
1.217
0.002
1.185 1.002
1.144 = 0.006
1.257 = 0.005
1.161
0.003
1.196
0.004
1.144
1.171 +.000
OOM
256
1.135
1: 0.006
0.002
1.120
1.173 1: 0.009
OOM
512
1.126
+ 0.004
1.175
+ 0.001
1.115 + 0.005
1.177 + 0.006
0OM
16
3.547
+ 0.003
3.589 + 0.005
3.427 + 0.006
3.464 + 0.005
3.424 + 0.009
3.446 + 0.008
32
3.429
0.001
3.497
3.319 0.001
3.385 - 0.004
3.323
0.003
3.417 + 0.005
64
$.349+00.000
3.442
3.258 = 0.004
0.004
3.339 = 0.004
3.251
0.003
3.499 + 0.006
128
3.289
0.001
3.212 1 0.003
3.319 1 0.004
0OM
256
3.242
1.0001
3.392
1 3.174
3.317
1 0.002
OOM
512
3.209
0.003
3.381
3.154 1 0.004
0.004
3.317 0.005
00M
21

---

Table D.5: QPCs generally improve over PCs on distribution estimation. We report the average test-set bits-pes tiis-pe-diimesionss of QT-CP-512, QG-CP-512 and QG-TK-66 for datasets up to image size 32x32, and of QT-CP-256, QG-CP-256 and QG-TK-32 for datasets of image size 64x64. All architectures are trained both as QPCs and PCs. QPCs are materialized from PICs applying F- sharing over the group input units, and C-sharing over the groups of integral units. PCs do not apply any form of parameter sharing for MNIST-family datasets, these delivered the best performance for such datasets Table D.3, while they apply F-sharing at the input layer for RGB datasets. We mark with (resp. f) datasets preprocessed using YCoCg-R (resp. YCoCg). All results are averaged over 5 different runs.
QPC
PC
QPC
PC
QPC
PC
QT-CP-512
QG-CP-512
64-TK-64
1.126 + 0.004
1.175 0.001
1.115 0.005
1.177 = 0.006
1.144 + 0.006
1.257 + 0.005
F-MNIST
3.209 + 0.003
3.381
0.001
=.154
3.317 = 0.005
3.251 0.003
3.499 + 0.006
EMN-MN
1.706
0.007
1.556 0.006
1.643 = 0.007
1.699
1.756 + 0.002
EMN-LE
1.622 0.007
1.698
1.545
0.007
1.626 0.004
1.000
1.725 1: 0.003
EMN-BA
1.638 + 0.006
1.707
1
1.665
0.004
1.015
4:000
EMN-BY
1.597 0.006
1.542
0.008
1.537 0.004
1.474 1: 0.009
1.703 + 0.004
1.665 = 0.007
CIFAR
5.198 + 0.003
5.596
0.004
5.097 0.002
5.496 = 0.004
5.556 = 0.003
5.647 + 0.004
CIFAR
4.577 + 0.004
4.884
0.003
4,486
0.009
4.856 0.010
4.888 + 0.007
4.983 1: 0.004
ImgNet32"
5.196 = 0.007
5.286
0.001
5.085
0.001
5.255 = 0.001
5.544 = 0.001
5.700 + 0.002
ImgNer32
4.578 + 0.001
4.662 1 0.002
4,468 1 0.001
4.632 = 0.003
4.893 + 0.004
5.045 1 0.001
QT-CP-256
QG-CP-256
QG-TK-32
ColebA
4.810 + 0.004
4.851 + 0.002
4.7.9 + 0.002
4.781 0.002
5.352 + 0.002
5.364 + 0.002
CelebAt
4.159 + 0.003
4.215
0.003
4.114
0.003
4.155 = 0.006
4.720 = 0.003
5.718 = 0.001
ImgNet64*
5.143 + 0.003
5.221
0.003
5.051
0.002
5.220 = 0.005
5.657
+ 0.004
5.764 + 0.001
ImgNet64*
4.523 + 0.003
4.591 + 0.002
4.425 0.004
4.590 0.006
5.011 + 0.007
5.138 + 0.001
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and main text accurately and precisely state the actual claims of the research presented.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations mentioned in the last section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificattin, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All data used publicly available. All model and experimental code to fully reproduce our own experimental results is shared.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models. it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All data used publicly available. All model and experimental code fully reproduce our own experimental results is shared.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https: futtpss//nips... cc/
www.oeeeeubmisssooooooooooo onPolicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/niis. ccccccpp..ccc wccppbllcc///cccccccccccccccllcl belic//uuieeee gwwdeu/dde//e/ Coo///odeeSeeeeeeeeeeeeee weabbbmsssiinnii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings overview included in paper; key details included in appendix; full details in the included code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Statistical significance indicated for all experimental results, including the nature of the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Overview compute resources included in paper; key details discussed in ap- pendix; full details listed in structured template-description of corresponding assets.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps://neurs https.cc/publlli wwtppccccccpcttteeeeeeecc http:/ulll//ttissseteeeeeee http://thhiiiiuieeeeeeee csGuideliness
Answer: [Yes]
Justification: Problematic aspects of generative AI mentioned, used asset licenses men- tioned, produced assets (code, models) licensed and shared, reporducibility ensured.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Positive: better interpretability of this class of generative Al models men-
tioned; Negative: problematic aspects of generative Al models in general mentioned.
Guidelines:
26

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Original papers cited and/or URLs provided; license mentioned in reference.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcom www.hhheee..... com catasatsets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yss]
Justification: Code and models included (as zip file for submission, also URL for final); Structured templates used for details.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can cither create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects. and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "kM4TrLHDkp", "reviewer": "Reviewer_WQnV", "summary": "This work extends the probabilistic integral circuits (PICs) from tree-shaped region graphs (RGs) based ones to DAG-shaped ones. While constructing PICs from DAG RGs can lead to more expressive models, it comes with the concern of scalability since the circuit sizes might be increased a lot. To address this concern, this work proposes tensorized circuits to approximate the resulting PICs by a PIC to QPC reduction. This is further combined with functional sharing and materialization to achieve efficiency and scalability, which is validated by experimental results on several image datasets.", "questions": "- Can you provide some examples to help understand the integral units?\n- From the definition of g_u in Line 71, the integral is a function of both variables X_i and Z_u. Can you explain the claim \"In this way, the output LVs of any unit u will simply be Zu\" that directly follows the definition of g_u?\n- Can you characterize the circuit sizes of the resulting PICs and QPCs?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The extension from tree-shaped PICs to DAG-shaped ones is a solid contribution as it greatly improves the expressiveness of the resulting PICs and is an important step towards PICs constructed from more complex RGs for complicated tasks.\n- The functional sharing and materialization techniques proposed in this work are shown to be very useful by the empirical results. From Fig 5, learning PICs consumes similar resources to PCs which are efficient models and it requires much less trainable parameters than the ones without functional sharing, making the training of large-scale PICs feasible.", "weaknesses": "I appreciate that this work aims to solve an important problem and the proposed solution is shown to be novel and efficient. Still, it's heavy in technical details and thus I don't think I fully understand the algorithmic details. Specifically,\n- I'm confused by the definition of the integral unit from Line 58 to Line 62, especially by the inconsistency in the input of g_i between Line 61 and Line 62 (the g_i inside the integral). Also, the integral in Line 62 contains the function g_i while Figure 1(b) does not, which is also confusing. It might be helpful to put a numeric example there.\n- This also applies to the other technical sections of this work given that this work is heavy in techniques. It would be very helpful to provide a toy example to showcase the reduction of PICs to QPCs to improve accessibility and reproducibility for readers who are not well-versed in these areas.\n- I'm curious to see some theoretical analysis on the circuit sizes characterized. I understand that the bounds might be loose and not that informative. Still, it might be helpful to understand how the PIC to QPC reduction affects the circuit sizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for deeming our paper to be a solid contribution, and a novel and efficient solution to an important problem. Below we clarify the confusion about notation, and respond to their points.\n\n> I'm confused by the definition of the integral unit from Line 58 to Line 62, especially by the inconsistency in the input of g_i between Line 61 and Line 62\n\n$g_i(\\mathbf{X}_i, \\\\{ \\mathbf{Y}_i, \\mathbf{Z}_u \\\\} )$ and $g_i(\\mathbf{X}_i,\\mathbf{Y}_i,\\mathbf{Z}_u)$  are just equivalent notations, we will remove the brackets to avoid confusion. We used them trying to emphasize the set of $g_i$ LVs. Finally, note that, when writing an integral, the LVs that we integrate out are in lower-case (e.g. line 62)\n\n>  the integral in Line 62 contains the function g_i while Figure 1(b) does not, which is also confusing. It might be helpful to put a numeric example there.\n\nAn integral unit $u$, by definition, receives input from a single unit, $i$. Assume that $u$ is associated with function $f_u(Z, Y) = 2Z - Y^2$, and that $i$ outputs the function $g_i(X, Y) = X^2 -3X + 4Y$. The integral unit would then compute the integral $\\int_{-1}^{1} f_u(Z, y)g_i(X, y) dy$ (the integration domain is the support of $Y$, an assumption), therefore outputting the function $g_u(Z, X) = 2/3 (X - 3) X (6Z - 1)$, as we can compute the integral in closed-form in this example. (We will add this example to the camera-ready.)\nTherefore, in Figure 1, for instance, integral unit 5 (the red coloured one) computes $\\int f_5(Z_2, z_4) f_4(X_4, z_4) dz_4$, i.e. $g_u = f_5$ and $g_i = f_4$.\n\n> It would be very helpful to provide a toy example to showcase the reduction of PICs to QPCs\n\nThis is a great suggestion and in the camera-ready version we will describe more in detail in the main text the example we provided appears in Figure 2. There, we illustrate the application of PIC2QPC (Algorithm 3) from the PIC in (b) to the QPC in (c), and then to its tensorized folded version in (d). Zooming in, we also illustrated the details of the materialization of a 3-variate function to a Tucker Layer in Figure 3. We will further expand this explanation and mention all the functions involved and comment on how Algorithm 3 operates step by step. \n\n> I'm curious to see some theoretical analysis on the circuit sizes characterized\n\nWe are not entirely sure how to understand the reviewer's request. We interpreted it as an analysis on the number of trainable parameters of PCs and PICs. We are happy to discuss more in case we are off-topic.\n\nWe address the answer by analyzing 3 types of folded layers (CP-layer, Tucker-layer, and Categorical-Layer), as our tensorized architectures are nothing more than a sequence of them.\nWe use $F$ to denote the number of layers stacked together with folding, $K$ the number of integration points, $M$ the MLP size for PIC neural nets, and $L$ the number of MLP layers. We recall that the size of PIC neural nets is independent from $K$.\n- A folded CP-layer is parameterized by a tensor of shape $F\\times K\\times K$ (where F is always even). Using PIC composite sharing, the size of the multi-headed MLP (assuming no bias term for simplicity) to parameterize such a layer is $LM^2 + FM$ (see Figure 4 for an illustration).\n- A folded Tucker-layer is parameterized by a tensor of shape $F\\times K\\times K\\times K$. Parameterizing such a layer with composite sharing, again requires an MLP of size $LM^2 + FM$.\n- A folded Categorical-layer is parameterized by a tensor of shape $F\\times K\\times C$, where $C$ is the number of categories (256 of gray-scale images, and 768 for RGB ones). When using composite sharing, we can parameterize it with an MLP of size $LM^2 + FMC$. However, we found that PICs deliver better results when using full-sharing for the input layer, and this means using an MLP of size $LM^2 + MC$.\n\nAs an example, consider a folded CP-layer with $F=1000$, $K=512$, whose number of trainable parameters is $262,144,000$. Parameterizing it with an MLP with $L=2$ and $M=256$, delivers instead only $387,072$ trainable parameters, more than 99% less trainable parameters.\nWe refer the reader to Appendix C1 for details about our MLPs, and to the plots on the right of Fig. 5 for the total trainable parameter counts of PC and PIC models. We will add this analysis in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ktPIMqrQD", "reviewer": "Reviewer_6CM1", "summary": "This paper introduces a new approach for building probabilistic integral circuits (PICs), a recently introduced probabilistic model that extends probabilistic circuits (PC) with continuous functions of latent variables (in addition to discrete latent variables), with inference performed using a numerical quadrature procedure. In particular, while the previous implementation was restricted to tree-structured region graphs/variable decompositions, it is shown how to construct PICs for arbitrary region graphs. Additionally, sharing of functions across different regions is used to improve parameter efficiency. Empirical results show that the architectural improvements enable scaling to complex image datasets such as ImageNet64, outperforming comparable PCs trained using maximum likelihood or EM.", "questions": "- What numerical quadrature rule was used? \n- In the conclusion, it is remarked that differentiable sampling from PICs is not possible. What is meant by differentiable here, and is it possible to sample from the materialized PC if one is not interested in gradients (e.g. for visualization)? \n- To what extent do the authors consider their method a continuous latent variable model, as opposed to a means of effectively constructing PCs? For instance, if the same quadrature points are used throughout training and evaluation, can one expect the MLPs to learn meaningful function values between evaluation points?\n- Did the authors investigate learning PICs according to a HCLT region graph? This would give a better idea of whether a DAG (as opposed to tree-structured) region graph is necessary.\n\nTypos:\n- modes -> nodes 190\n- capitalization of 'functional' 234\n- 'pixels are categorical' -> 'pixels as categorical' 286", "limitations": "The limitations of the method are appropriately addressed in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper proposes a number of innovations on top of the recently proposed PICs, such as allowing for arbitrary region graphs and functional sharing. Taken together, this constitutes a well-engineered solution that significantly improves the scalability and flexibility of PICs. \n\nThe experiments are well-executed with strong results on a range of image datasets in comparison to comparable PCs learned directly. Many experimental details are reported, such as memory usage, time taken and parameter counts/hyperparameters, which should make the results reproducible.\n\nThe paper was a pleasure to read, with clear and consistent notation throughout explaining the method and accompanying pictorial illustrations that make the idea of the paper transparent and easy to understand.", "weaknesses": "The paper is arguably a little weak in terms of novelty as most of the new components are adaptations of existing ideas in the literature (e.g. tucker/cp layers, parameter sharing) to probabilistic integral circuits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for deeming our paper to be a well-engineered solution delivering strong results. We are also happy that the reviewer enjoyed the reading. Below we reply to their questions.\n\n> The paper is arguably a little weak in terms of novelty\n\nWe respectfully disagree as it is non-trivial to recover a compact quadrature PC (QPC) from a PIC with a DAG structure. First we needed to change the original semantics of PICs/QPCs, which were defined only for tree region graphs, second, parameterizing the newly found tensorized QPCs needs some careful engineering otherwise the models would never be able to scale to the number of integration points we used (up to K=512).  \n\n> What numerical quadrature rule was used?\n\nWe trained using the trapezoidal integration rule, we will document this in the revised version of the paper.\n\n> What is meant by differentiable [sampling] here, and is it possible to sample from the materialized PC if one is not interested in gradients (e.g. for visualization)?\n\nWhen we say that differentiable sampling is not currently possible, it is just because the functions we use are parameterized as simple unconstrained MLPs, which can be seen as energy-based models, for which exact sampling is not possible. While using MCMC would be possible to sample from these neural networks, obtaining meaningful gradients through the samples would require additional approximations. \nNevertheless, we can always materialize PICs as QPCs and sample from them as usually done with PCs.\n\n> To what extent do the authors consider their method a continuous latent variable model, as opposed to a means of effectively constructing PCs?\n\nPICs are **continuous** latent variable models by definition. It is true, indeed, that the way we currently use PICs seems like a parameter learning procedure for tensorized PCs. However, we stress that at test-time we can actually materialize an arbitrarily sized PCs based on the number of integration points we use. We plan to explore (and exploit) this capability further in future work. \n\n> Did the authors investigate learning PICs according to a HCLT region graph? \n\nLearning PICs using HCLT region graphs (RGs) has been already investigated in prior work [A]. In our work, we directly use RGs that, differently from HCLTs, are not learned from data but rather fixed for a given image size: QuadTree (QT) and QuadGraph (QG) RGs (see Appendix B for details). By comparing our results with the ones in [A], one can clearly see that QT and QG outperform HCLT, and highlight how learning an RG from images does not bring any advantage. QG outperforming QT brings evidence to support the claim that non-tree RGs can be more expressive.\n\n[A] Gennaro Gala, Cassio de Campos, Robert Peharz, Antonio Vergari, and Erik Quaeghebeur. Probabilistic integral circuits. AISTATS 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bsp0vdPYRG", "reviewer": "Reviewer_iaCH", "summary": "This paper presents a pipeline to build probabilistic integral circuits (PICs) more generally as directed acyclic graphs, as opposed to the tree-shaped structure they were limited to before. This significantly increases the expressiveness of the PICs, improving their representation of distributions. The authors present a procedure for turning a region graph into a PIC, into a QPC that approximates the PIC, and finally into a folded QPC, which are effectively tensorized structures that significantly speed up the inference and learning process. From the tensorized PIC the authors then make further improvements on QPC training, using neural functional sharing techniques over groupings of the input and integral units to result in greater materialization of QPCs and far fewer parameters.", "questions": "1. Among the techniques discussed in the paper, how significant is the tree-shape vs DAG-shape for empirical expressiveness of PICs? It appears that DAG-shaped PICs achieve slightly better bpds, but tree-shaped ones scale much better.\n\n2. Section 3.2 describes a numerical quadrature rule as consisting of integration points and weights that minimize the integration error, and the authors later say that the same quadrature rule is used for each approximation. Does this raise any issues in approximation?\n \n3. How crucial/limiting is the assumption that every integral unit integrates out all incoming latent variables?\n\n4. Do the QPCs approximating PICs still support tractable inference (e.g. marginal probabilities) similar to traditional PCs? It would be great to clarify the probabilistic reasoning capabilities mentioned in the introduction.\n\nMinor comments:\n- Lines 133-134: do you also need $Z_{u_3}\\neq Z_{u_4}$ for CP-merge?\n- Table 1: what does LVD-PG refer to?", "limitations": "The authors acknowledge the limitation of PICs that, despite being a continuous latent variable model like many generative models, they do not support sampling. This is left as future work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The proposed approaches significantly increase the expressivity and learning efficiency of PICs, as showcased by the experiments.\n\n+ The paper is technically sound with clear arguments presented for each step.\n\n+ The paper is overall well structured and easy to follow.\n\n+ The authors also address the current limitations of PICs with a clear goal for future improvements.", "weaknesses": "Not a major weakness, but as the authors also point out, it is currently not possible to sample from PICs, limiting their impact as a generative model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for deeming our paper to be technically sound, clearly argumented, well-structured and easy-to-follow. Below we reply to their points.\n\n> it is currently not possible to sample from PICs, limiting their impact as a generative model.\n\nIt is true that since we use simple unconstrained MLPs to parameterize PICs --- which can be seen as energy-based models --- (differentiable) sampling from them is not possible. However, this is not a limitation of PICs per se, but a consequence of the way we choose to parameterize them. For instance, parameterizing using simple mixtures of Gaussians or even normalizing flows, would allow sampling from PICs.\n\nNevertheless, it is always possible to materialize PICs as QPCs and sample from them as usually done with PCs.\n\n> how significant is the tree-shape vs DAG-shape for empirical expressiveness of PICs? It appears that DAG-shaped PICs achieve slightly better bpds, but tree-shaped ones scale much better.\n\n\nWe remark that dropping 0.1 bpd on MNIST-like datasets would correspond to an increase of ~54 nats in log-likelihood, and perhaps this would be considered to be more significant. On RGB 64x64 datasets, the gap would even be much larger, i.e. ~851 nats.\n\nIt is true that there is a trade-off in terms of accuracy/efficiency. We just started exploring how to scale DAG RGs in a more effective way and believe our paper is a stepping stone in this direction. E.g., note that using Tucker layers yields the best performance for lower values of K, but we do not know how to scale it properly yet.\n\n> Section 3.2 describes a numerical quadrature rule as consisting of integration points and weights that minimize the integration error, and the authors later say that the same quadrature rule is used for each approximation. Does this raise any issues in approximation?\n\nUsing the same integration technique for each integral unit is not a limitation nor raises approximation issues when assuming a *static* quadrature regime.  Assuming the same number of integration points, instead, allow us to tensorize the QPC in a regular way, thus promoting GPU parallelism via folding.\n\nWe conjecture that since we *learn* our functions through numerical approximation, these will stay nicely integrable w.r.t. to the points we used during training to fit them.\n\n> How crucial/limiting is the assumption that every integral unit integrates out all incoming latent variables\n\nThis is an interesting question! From a theoretical perspective, we would need to devise novel quadrature routines to materialize the QPC, as some latent variable would affect multiple integral units at once. In practice, it might be possible to “move” the non-integrated-out latent variables of the input function $g_i$ as to be part of variables $\\mathbf{Z}_u$ of function $f_u$, for certain parameterizations of $g_i$ and $f_u$.\n\nNevertheless, we believe that from a learning perspective, PICs/QPCs have enough capacity to learn complex distributions with our assumption.\n\n> Do the QPCs approximating PICs still support tractable inference (e.g. marginal probabilities) similar to traditional PCs? \n\nYes, QPCs --- being composed of only sum, product and input units --- are proper PCs that inherit the structural properties of their corresponding PIC , e.g., they are also  structured-decomposability if the underlying PIC is structured-decomposable. We will clarify this in the revised version of the manuscript.\n> Lines 133-134: do you also need Z_u3 != Z_u4 for CP-merge?\n\nIt does not need to be. If  $Z_{u_3} \\neq Z_{u_4}$, we would then merge u3 and u4 using Tucker-merge, whereas if $Z_{u_3} = Z_{u_4}$, we would then use again CP-merge. In this way, we allow for Tucker layers on top of CP-layers, and vice-versa. \n\n> Table 1: what does LVD-PG refer to?\n\nIt stands for Latent Variable Distillation - Progressive Growing, and it is a sophisticated learning scheme developed in [Liu, Xuejie, et al. \"Understanding the distillation process from deep generative models to tractable probabilistic circuits.\" ICML 2023] that employs several heuristics to learn the parameters and structure of a PC on a lossy color space. We will specify this in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "avo6krBMV8", "reviewer": "Reviewer_fmPL", "summary": "This paper extends the work of PIC with continuous LVs, proposes to tensorize and to use neural functional sharing to scale the model.\nThe results show that the proposed approach decreases the trainable parameters as well as the memory usage in training, and also reduces the running time. The approach also works well in density estimation tasks.", "questions": "- Could you elaborate why the trainable parameters in PIC (F, C) have reduced significantly compared with the PC (F, N), since there is no functional sharing in PCs as I understand. That is, why the PIC (F, C) can have much fewer trainable parameters than PC (F, N)? Is this comparison fare?\n\n- In the left most plots in Fig 5, I would infer that the single orange triangles in upper left are PIC(F,N) with QG-TK, but without a dotted line connected it is a bit unclear. \n\n- is the ```and``` before $||$ in line 156 a typo?", "limitations": "The limitations are well discussed in the end of the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper extends the work of PIC for better scalability and efficiency, and thus has its novelty.\n\nThe paper is well written and not difficult to understand. e.g., the illustration of Algorithm 1 in Fig.2 (a-b) is a very nice visualisation. \n\nWith the claimed contribution, the paper has a high potential to contribute to the community.", "weaknesses": "It is not clear to me what data set is used in the experiment of \"Scaling PICs\", and how many RVs are modelled. \n\nI compared the density estimation results in PIC[18] on the MNIST family and the results in this paper outperforms the ones in PIC[18].\nAs far as I understand, the PIC (F, C) in this paper has reduced its trainable parameters and therefore should in principle has smaller model capacity. I am wondering why the density estimation results of the proposed model outperform the original PIC? \n\nSimilarly, it would be nice if the drop (or increase, though unlikely) of density estimation performance can be shown between PIC (F, N) and PIC (F, C) so readers will know how much they can lose/gain w.r.t. performance when applying the functional sharing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for deeming our paper to be novel, well-written, and for expecting it to have a high impact in the community. Below we reply to their points.\n\n> It is not clear to me what data set is used in the experiment of \"Scaling PICs\", and how many RVs are modelled.\n\nIn the experiment “Scaling PICs” (in the main text), whose results are shown in Fig 5, we used a PC modeling ImageNet 64x64. Note that it is not important which likelihood the PC assigns to different images, but only the time and space requirements for a forward/backward pass. Therefore, any batch of 64x64  RGB images would reproduce the same performance as reported in the plot. As we write in the caption, “The benchmark is conducted using a batch of 128 RGB images of size 64x64”. Finally, all RVs are categorical (256 categories for MNIST-like images, and 768 categories for RGB images), as described in lines 286-287.\n\n> About PICs with and without functional sharing\n\nAn extensive comparison between PICs with and without function sharing *in terms of performance* is not present in the paper, and below we explain why. However, luckily, as the reviewer pointed out, such comparison can be retrieved --- to a certain extent --- by taking results from Fig. 5 in [A], which corresponds to much smaller models than ours. We will account for it in the revised version of the paper. \n\nAs we extensively tested in our benchmarks (Fig. 5, Table D1, Table D2), PICs without functional sharing cannot scale to large model sizes, and therefore, simply because of that, PICs with functional sharing can deliver (better) results for the computational resources we have available.  \n\nThis is in fact one of the main points of our paper: Scaling PICs with functional sharing, as we cannot without it. Therefore, given that PICs without functional sharing can unfortunately be run only for very small sizes (see Table D1 and D2), an **extensive comparison** in terms of performance w.r.t. PICs with functional sharing is not feasible on our GPUs.\n\nWe expect functional sharing to deliver better models as applying it drastically reduces the number of trainable parameters, and therefore possible overfitting. In Fig. 5 (top-right) we indeed showed that PICs without functional sharing hit 2B parameters, whereas with functional sharing only 6M.\n\n[A] Gennaro Gala, Cassio de Campos, Robert Peharz, Antonio Vergari, and Erik Quaeghebeur. Probabilistic integral circuits. AISTATS, 2024.\n\n> Could you elaborate why the trainable parameters in PIC (F, C) have reduced significantly compared with the PC (F, N)\n\nYes, the reason why there is a significant reduction in the number of trainable parameters between PIC (F, C) and PC (F, N) is because the former applies compositional sharing of MLPs, as explained in Section 3.3. However, once we materialize a QPC out of it, the number of its parameters match exactly the ones of PC (F, N), since these represent exactly the same architecture. With that in mind, we believe the comparison is fair as the PIC approach can be seen as a way of training PC (F, N).\n\nFor a detailed analysis, we refer the reviewer to our answer to reviewer WQnV about characterizing the size of PICs and PCs.\n\n> In the left most plots in Fig 5, I would infer that the single orange triangles in upper left are PIC(F,N) with QG-TK, but without a dotted line connected it is a bit unclear\n\nPrecisely, those isolated orange triangles refer to PIC (F, N) with QG-TK. The reason why they are isolated is because we could only run such configuration at K = 16, hence the lack of dotted lines. We refer the reviewer to Table D.2 for some tabular results of the plots, where this is evident. We will make sure to detail this better in the main text.\n\n> is the and before $||$ in line 156 a typo?\n\nYes, it should be and will be removed, thanks for spotting it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful feedback, questions, and kind words. We are glad that the paper seems to be (i) **very well-received** (“The paper was a pleasure to read, with clear and consistent notation”  - 6CM1, “transparent and easy to understand” - 6CM1, “The paper is technically sound with clear arguments presented for each step” - iaCH, “The paper is overall well structured and easy to follow” - iaCH, “The paper is well written and not difficult to understand'' - fmPL), and (ii) **deemed as a strong contribution** to the field (“The extension from tree-shaped PICs to DAG-shaped ones is a solid contribution” - WQnV, “the proposed solution is shown to be novel and efficient” - WQnV, “The paper proposes a number of innovations” - 6CM1, “this constitutes a well-engineered solution” - 6CM1, “The proposed approaches significantly increase the expressivity and learning efficiency of PICs” - iaCH, “the paper has a high potential to contribute to the community” - fmPL).


We answer all remaining questions below and will include feedback in the revised version of the paper.

Meta-review: This work presents a novel construction for probabilistic integral circuits (PICs) over general directed acyclic graphs.  Extending PICs beyond trees increases their representational power. Overall, the reviewers felt that the work was clearly presented and represented a significantly novel contribution to the field with only minor weaknesses.
Decision: Accept (spotlight)