Scaling Continuous Latent Variable Models as Probabilistic Integral Circuits

Authors: gennaro  gala
cassio  de campos
antonio  vergari
erik  quaeghebeur
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 22:05:47

Abstract: Probabilistic integral circuits (PICs) have been recently introduced as probabilistic models enjoying the key ingredient behind expressive generative models: continuous latent variables (LVs). PICs are symbolic computational graphs defining continuous LV models as hierarchies of functions that are summed and multiplied together, or integrated over some LVs. They are tractable if LVs can be analytically integrated out, otherwise they can be approximated by tractable probabilistic circuits (PC) encoding a hierarchical numerical quadrature process, called QPCs.

So far, only tree-shaped PICs have been explored, and training them via numerical quadrature requires memory-intensive processing at scale. In this paper, we address these issues, and present: (i) a pipeline for building DAG-shaped PICs out of arbitrary variable decompositions, (ii) a procedure for training PICs using tensorized circuit architectures, and (iii) neural functional sharing techniques to allow scalable training. In extensive experiments, we showcase the effectiveness of functional sharing and the superiority of QPCs over traditional PCs.

Main paper: 
 introduction: 
 Continuous latent variables (LVs) are arguably the key ingredient behind many successful generative models, from variational autoencoders [24] to generative adversarial networks [20] , and more recently diffusion models [56] . While these models allow to learn expressive distributions from data, they are limited to sampling and require task-specific approximations when it comes to perform probabilistic reasoning, as even simple tasks such as computing marginals or conditionals are intractable for them. On the other hand, performing these tasks can be tractable for (hierarchical) discrete LV models [4, 3] , but these prove to be more challenging to learn at scale [9, 10, 32, 33] . This inherent trade-off among tractability, ease of learning, and expressiveness can be analyzed and explored with probabilistic integral circuits (PICs) [18] , a recently introduced class of deep generative models defining hierarchies of continuous LVs using symbolic functional circuits. PICs are tractable when their continuous LVs can be analytically integrated out. Intractable PICs can however be systematically approximated as (tensorized) probabilistic circuits (PCs) [53, 4] , the representation language of discrete LV models. An instance of such PCs encodes a hierarchical numerical quadrature process of the PIC to approximate, and as such is called quadrature PC (QPC).Distilling QPCs from PICs has proven to be an effective alternative way to train PCs, but it has only been explored for tree-shaped PICs, as building and scaling to richer LV structures is an open research question that requires new tools [18] . In this paper, we fill this gap by redefining the semantics of PICs and extending them to DAG-shaped hierarchies of continuous LVs. Specifically, we design PICs as a language to represent hierarchical quasi-tensors factorizations [49] , parameterized by light-weight multi-layer perceptrons.Contributions. (1) We present a systematic pipeline to build DAG-shaped PICs, starting from arbitrary variable decompositions (Section 3.1). (2) We show how to learn and approximate PICs via a hierarchical quadrature process which we encode in tensorized QPCs that match certain circuit architectures proposed in different prior works [43, 42, 31, 36] (Section 3.2). (3) We present functional sharing techniques to scale the training of PICs, which lead us to parameterize them with multi-headed multi-layer perceptrons (MLPs) requiring comparable resources as PCs (Section 3.3). (4) In extensive experiments (Section 4), we show that (i) functional sharing proves remarkably effective for scaling and that (ii) QPCs outperform PCs commonly trained via EM or SGD, while being distilled from PICs with up to 99% less trainable parameters. 
probabilistic integral circuits: 
 Notation. We denote input variables as X and latent variables (LVs) as Y and Z, with x, y and z as their realization respectively. We denote scalars with lower-case letters (e.g., w ∈ R), vectors with boldface lower-case letters (e.g., w ∈ R N ), matrices with boldface upper-case letters (excluding X, Y, Z, e.g., W ∈ R M ×N ), and tensors with boldface calligraphic letters (e.g., W ∈ R L×M ×N ).A probabilistic integral circuit (PIC) c is a symbolic computational graph representing a nonnegative function c(X) = ∫ c(X, z) dz, i.e. c(x) ≥ 0, over observed variables X and continuous latent variables Z. Similar to probabilistic circuits (PCs) [53, 4] , PICs have input, sum and product units. 2 Different from PCs, however, PICs operate on functions, not scalars, and make use of a new type of unit: the integral unit ∫ , which allows to represent continuous LVs. To satisfy nonnegativity, we parameterize PICs with non-negative functions and positive sum weights, specifically:• An input unit u (depicted as in our figures) represents a possibly non-normalized distribution f u (X u , Z u ) → R + , where X u ⊆ X and Z u ⊆ Z;• A sum unit u ( + ) outputs a weighted sum of the functions it receives from its input units, i.e.g u (X u , Z u ) = Σ i∈in(u) w i g i (X i , Z i ), where in(u) is the set of units u takes as input, w i > 0, X u = ∪ i∈in(u) X i and Z u = ∪ i∈in(u) Z i . Similarly, a product unit u ( × ) outputs the product of its incoming functions, i.e. g u (X u , Z u ) = Π i∈in(u) g i (X i , Z i );• Finally, an integral unit u ( ∫ ) encodes an "uncountable weighted sum" whose weights are compactly represented by a function f u (Z u , Y u ) → R + , where ∅ ̸ = Y u ⊆ Z are the LVs that are being integrated out by u, while Z u can potentially be empty, i.e. Z u = ∅. The unit receives a function g i (X i , Y u ) from its only input unit i and outputs the function g u (X i , Z u ) = ∆ f u (Z u , y u ) g i (X i , y u ) dy u , where ∆ = supp(Y u ) is the support of Y u . For instance, an integral unit u with f u ({Z}, {Y }) = 2Z -Y 2 and supp(Y ) = [-1, 1] receiving function g i ({X}, {Y }) = X 2 -3X + 4Y would output g u ({Z}, {X}) = 2 /3 X(X -3)(6Z -1). The output of a PIC c is the output function returned by its root unit u, which is only defined on X, i.e. c(X) = g u (X) and Z u = ∅. Similar to PCs, imposing structural constraints over PICs can unlock tractable inference [4] . As such, we assume that (i) all + -units receive functions defined on the same input variables (aka smoothness), and (ii) all × -units receive functions defined over disjoint sets of input variables (aka decomposability).PICs are tractable when their LVs can be analytically integrated out, meaning that we can passthrough integral units computing the integration problem they define, eventually outputting a function. Notably, this is possible when LVs are in linear-Gaussian relationships [27] or when functions are polynomials. Intractable PICs can however be approximated via a hierarchical numerical quadrature process that can be encoded as a PC called quadrature PC (QPC). Intuitively, each PIC integral unit can be approximated by a set of sum units in a QPC, each conditioning on some previously computed quadrature values, with a large but finite number of input units [18] . Materializing a QPC allows to train PICs by approximate maximum likelihood: Given a PIC, gradients to its parameters attached to input, sum and integral units can be backpropagated through the corresponding QPC [18] . This also provides an alternative way to train PCs that can rival traditional learners. So far, the construction of PICs has been limited to a simple compilation process from probabilistic graphical models (PGMs) [27] with continuous LVs [18] . In a nutshell, the LV nodes of a PGM become integral units of a PIC model, and the PGM (conditional) distributions become the functions f u attached to input and integral units of the PIC, as we illustrate in Fig. 1 . However, the PGM structure needs to be limited to a tree, as to avoid that the hierarchical quadrature process would yield an exponentially large QPC, thus hindering learning. This imposes a semantics for current PICs as simple latent tree models [3] , and clearly limits their expressiveness as more complex LV interactions are not possible. Building more expressive PICs requires reinterpreting this semantics and introducing new tools, which we do next.3 Building, learning and scaling PICsIn Section 3.1, we systematize the construction of DAG-shaped PICs, showing how to build them starting from arbitrary variable decompositions, going beyond the current state-of-the-art [18] . Then, in Section 3.2, we show how to learn and approximate such PICs with QPCs encoding a hierarchical quadrature process, retrieving PC architectures proposed in prior works [36] . Finally, in Section 3.3, we present (neural) functional sharing, a technique which we use to parameterize PICs as to make their QPC materialization fast and cheap, allowing scaling to larger models and larger datasets. 
building pics from arbitrary variable decompositions: 
 Standard PCs can be built according to established pipelines that allow to flexibly represent arbitrary variable decompositions, as well as rich discrete LV interactions [36, 41] . In the following, we derive an analogous pipeline for PICs that allows to take care of continuous LVs, going beyond their current tree-shaped semantics (Section 2) yet allowing to perform hierarchical quadrature without blowing up the size of the materialized QPCs. To do so, we start by formalizing the notion of hierarchical variable decomposition, or region graph, out of which we will build our PIC structures. Definition 1 (Region Graph (RG) [15] ). An RG R over input variables X is a bipartite and rooted directed acyclic graph (DAG) whose nodes are either regions, denoting subsets of X, or partitions, specifying how a region is partitioned into other regions (Fig. 2(a) ).RGs can be (i) compiled from PGMs [5, 27, 3, 31] , (ii) randomly initialized [43, 16] , (iii) learned from data [15, 19, 40, 57] , or (iv) built according to the data modality (e.g. images) [45, 42, 36] . If we compile from a tree PGM, as in (Fig. 1 ), the resulting RG will be a tree, thus yielding a tree-like PIC [18] . Our pipeline, detailed in Algorithm 1, takes an arbitrary DAG-shaped RG as input, and can deliver DAG-like PICs. Without loss of generality, we assume to have an RG R which only allows for (i) binary partitionings of regions (i.e. all product units will have two input units) and (ii) univariate leaves, as shown in Fig. 2 (a). Our construction iteratively builds a PIC in a bottom-up fashion, associating regions to PIC units. For every leaf region X u ∈ X in R, we instantiate an input unit u with function f u ({X u }, {Z u }), where Z u ∈ Z is an arbitrary continuous LV (Line 8, Algorithm 1). Such functions can be univariate conditional densities, i.e. p u (X u |Z u ), resembling small VAE-like decoders [24] amenable to be numerically integrated.Once all leaf regions have been processed, we move to the inner ones. Let X ⊆ X be an inner region partitioned inN ≥ 1 different ways as {(X (n) 1 , X (n) 2 )} N n=1 , i.e. (X (n) 1 ∩X (n) 2 ) = ∅ and (X (n) 1 ∪X (n) 2 ) = X for every n. For each partition (X (n) 1 , X (n)2 ), we will merge the PIC units associated to regions X (n) 1 and X (n) 2 using consecutive applications of product and integral units-as we explain nexteventually associating a unit the partition itself. One can design such merging as desired, as long as smoothness and decomposability are not violated. Finally, in case N = 1, we associate to X the unit associated to its only partition, otherwise, in case N > 1, we merge the N units associated to each n-th partition using a sum unit which we then associate to X (Line 6, Algorithm 1).Figure 2 : The pipeline presented in this paper: RG → PIC → QPC → folded QPC. Starting from a (fragment of) a DAG-shaped region graph (a), we build a DAG-like PIC via Algorithm 1 using Tucker-merge (b). Then, we materialize a tensorized QPC encoding a hierarchical quadrature process via Algorithm 3, using K = 2 quadrature points, which we fold to allow faster inference (d).Algorithm 1 RG2PIC(R) Input RG R over variables X Output PIC c(X) = c(X, z)dz 1: U ← map()▷ from regions to PIC units 2: for each region X ∈ postOrder(R) do 3:if X partitioned as {(X (n) 1 , X (n) 2 )} N n=1 then 4: ρ ← True if X = X else False 5: S ← {merge(U[X (n) 1 ], U[X (n) 2 ], ρ)} N n=1 6: U[X] ← pop(S) if N =1 else + ([S]) 7: else ▷ |X| = 1 8: U[X] ← (fu(X, {Zu})) 9: return A PIC with U[X] as root unit Algorithm 2 merge(u 1 , u 2 , ρ)Input Units u1, u2, and boolean flag ρ Output ∫ or × unit u with (u1, u2) as descendants 1: procedure (Tucker-merge)▷ Zu 1 ̸ = Zu 2 2: Zu ← ∅ if ρ else {Z} ⊂ Z \ {Zu 1 , Zu 2 } 3: return ∫ ( × ([u1, u2]), fu(Zu, {Zu 1 , Zu 2 })) 4: procedure (CP-merge) ▷ Zu 1 = Zu 2 5: Zu 3 ← ∅ if ρ else {Zu 3 } ⊂ Z \ {Zu 1 } 6: u3 ← ∫ (u1, fu 3 (Zu 3 , {Zu 1 })) 7: Zu 4 ← ∅ if ρ else {Zu 4 } ⊂ Z \ {Zu 2 } 8: u4 ← ∫ (u2, fu 4 (Zu 4 , {Zu 2 })) 9: return × ([u3, u4])Merging PIC units. Let u 1 and u 2 be candidate units to merge, each outputting functions with LV Z u1 and Z u2 respectively. We present two ways of merging units: Tucker-merge and CP-merge, which we detail in Algorithm 2 and whose names will be clearer in the next section. If Z u1 ̸ = Z u2 , we use Tucker-merge: We merge u 1 and u 2 with a product, which is then input to an integral unit u with function f u ({Z u }, {Z u1 , Z u2 }), where Z u ∈ Z \ {Z u1 , Z u2 }. Otherwise, if Z u1 = Z u2 , we use CP-merge: We add two integral units, u 3 with input u 1 and u 4 with input u 2 , which we finally merge with a product. We parameterize unit u 3 (resp. u 4 ) with f u3 ({Z u3 }, {Z u1 }) (resp. f u4 ({Z u4 }, {Z u2 })), where Z u3 ̸ = Z u1 (resp. Z u4 ̸ = Z u2 ). Note that whenever merging two units defined on X, we need to marginalize out the remaining LVs, without introducing new ones. We illustrate the application of Algorithm 1 in Fig. 2(a-b ). Our pipeline generalizes the PICs used in prior work [18] (Fig. 1 ) as we can build them by just converting latent tree structures in tree RGs and using CP-merge as merging procedure. While we now do not need a PGM to build a complex PIC structure, one could try to reverse-engineer our PICs to retrieve a PGM via decompilation [1] , the result would be a very intricate hierarchy over continuous LVs [41] .Given a DAG-shaped (intractable) PIC, we now show how to approximate it with a tensorized PC encoding a hierarchical quadrature process, namely a QPC. Intuitively, we interpret PICs as to encode a set of quasi-tensors [49] , a generalization of tensors with potentially infinite entries in each dimension corresponding to a continuous LV, which we materialize into classical tensors via quadrature. We begin with a definition of tensorized circuits and a brief refresher on numerical quadrature. Definition 2 (Tensorized Circuit [36, 43] ). A tensorized circuit c is a parameterized computational graph encoding a function c(X) ∈ R, and comprising of input , product × and sum + layers. Each layer consists of many computational units defined over the same variables, and every noninput layer receives vectors as input from one or more layers. Each input layer ℓ is defined on variables X ℓ ⊆ X and computes a collection of K ℓ parametric functions [f k : dom(X ℓ ) → R] K ℓ k=1 , outputting a K ℓ -dimensional vector. Each product layer ℓ computes either an Hadamard product (⊙) or a Kronecker product (⊗) of the vectors it receives from its inputs layers. Specifically, the Hadamard product is an element-wise product of vectors, and therefore applicable when these have same size, while the outer product of two vectorsu ∈ R N and v ∈ R M is w = u ⊗ v = || N i=1 u i v ∈ R N M, where || is the concatenation operator. Finally, a sum layer ℓ with S ℓ sum units receives inputs from 3 : From functions to sum-product layers via multivariate numerical quadrature (Section 3.2). We illustrate how the 3-variate function f ({Z}, {Y 1 , Y 2 }) (a) can be seen as an infinite (quasi) tensor that we first materialize w.r.t. integration points z as a finite tensor W of size K × K × K (b, Equation ( 2)), then flatten as a matrix accounting for integration weights w (c, Equation ( 3)), and finally use to parameterize a Tucker layer (d, Equation (Tucker-layer)).N layers {ℓ i } N i=1 and computes the matrix-vector product W|| N i=1 ℓ i (X ℓi ), where W ∈ R S ℓ ×K , K=Σ N i=1 K ℓi , are the sum layer parameters. When N = 1, then it simply computes Wℓ 1 (X ℓ1 ). Numerical quadrature. A numerical quadrature rule is an approximation of the definite integral of a function as a weighted sum of function evaluations at specified points [13] . Specifically, given some integrand f : R → R and interval ∆ := [a, b], a quadrature rule consists of a set of K integration points z ∈ ∆ K and weights w ∈ R K minimizing the integration error εK = | ∫ ∆ f (z) dz -Σ K k=1 w k f (z k )|, which goes to zero as K→ ∞. To approximate an integral of an Ndimensional function f , we can phrase the multiple integral as repeated one-dimensional integrals by applying Fubini's theorem [17] , aka tensor product rule, as follows.∆ K f (z)dz = ∆ ... ∆ f (z 1 , ..., z N )dz 1 ...dz N ≈ i1∈[K] w i1 ... i N ∈[K] w i N f (z i1 , ..., zi N ). (1)From PICs to QPCs. Given a candidate PIC, we will explore it in post order traversal, and iteratively associate a circuit layer (Definition 2) to each PIC unit, a process that we call materialization. We detail such procedure in Algorithm 3, which essentially applies Eq. ( 1) hierarchically over the PIC units. Each unit encodes a function over potentially continuous and discrete variables, hence representing a quasi-tensor, which we approximate by evaluating it over the quadrature points only, thus materializing a classical tensor (Fig. 3 (a, b )). We facilitate quadrature by assuming that all PIC LVs have bounded domain ∆ := [-1, 1]. This way, we can always use the same quadrature rule (z, w) for each required (multivariate) approximation, and also simplify treatment and exposition.Algorithm 3 PIC2QPC(c, z, w) Input PIC c, quadrature rule z, w ∈ R K Output Tensorized QPC 1: L ← map() ▷ from PIC units to layers 2: for each unit u ∈ postOrder(c) do 3: if u is then 4: L[u] ← [ (fu(Xu, Zu=z k )) ] K k=1 5: else if u is ∫ (u1, fu) then 6: L[u] ← W (u) L[u1] via Eq. (3) 7: else if u is + ([ui, wi] N i=1 ) then 8: L[u] ← W (u) || N i=1 L[ui] via Eq. (4) 9: else if u is × ([u1, u2]) then 10: ← ⊙ if Zu 1 = Zu 2 else ⊗ 11: L[u] ← L[u1] L[u2] 12: return A QPC with L[c] as root layerWe begin materializing every PIC input unit u with function f u ({X u }, {Z u }) w.r.t. integration points z, effectively creating an input layer ℓ :dom(X u ) → R K as [ f u (X u , Z u =z k ) ] K k=1 (Line 4, Algorithm 3). The parameters of such layer can be materialized as a matrix of shape K × P , where P is the number of parameters f u requires. For example, if f u is a univariate conditional Gaussian p u (X u |Z u ), we use a K × 2 matrix for parameterizing the layer, where each row stores the mean and standard deviation at each integration point zk .Next, we address the most important part of this quadrature process, i.e. the materialization of PIC integral units as sum layers. Specifically, let u be an integral unit withEQUATIONAfter materializing tensor W (u) , we flatten it w.r.t. variables Z u and Y u , so as creating a matrix W (u) of size K N Z × K N Y , an operation aka matricization. As last step, we plug-in the quadratureEQUATIONwhere w ⊗N Y is the vector of size K N Y resulting from the N Y -times application of the Kronecker product ⊗ over w (Line 6, Algorithm 3). We illustrate this process in Fig. 3(a-c ). Similarly, we also materialize every PIC sum unit u with weights {w i } N i=1 as a sum layer, but parameterized byEQUATIONwhere I K is the K ×K identity matrix and || the concatenation operator (Line 8, Algorithm 3). Note that such sum layer can be seen as a mixing layer [42, 36] . Finally, consider a PIC product unit u with inputs u 1 and u 2 , each outputting functions with LVs Z u1 and Z u2 respectively. We associate to u an Hadamard product layer if Z u1 = Z u2 , or a Kronecker product layer if Z u1 ̸ = Z u2 , reflecting the fact that we are marginalizing out two different LVs. We summarize our PIC materializationillustrated in Fig. 2 -in Algorithm 3, where we iteratively associate a PIC unit to a circuit layer, eventually delivering a tensorized QPC. We stress that being QPCs just standard PCs they enjoy their same properties (e.g. tractable marginalization). We will learn PICs via maximizing the likelihood of its QPC materialization.QPCs as existing tensorized architectures. Materializing PICs built via Algorithm 1 delivers tensorized PCs with alternating sum and product layers, aka sum-product layers [36] . An instance of such layers is the Tucker layer, used in architectures like RAT-SPNs [43] and EiNets [42] . Specifically, a binary Tucker layer ℓ [51] computesEQUATIONwhere W ∈ R K×K 2 and ℓ 1 , ℓ 2 are input layers of ℓ, each outputting a K-dimensional vector. In contrast, the recent HCLT architectures [31] use the canonical polyadic (CP) layer ℓ[2], i.e. ℓ(X ℓ ) = W (1) ℓ 1 (X ℓ1 ) ⊙ W (2) ℓ 2 (X ℓ2 ) , (CP-layer)where W (1) , W (2) ∈ R K×K . We exactly recover Tucker (resp. CP) layers in our QPCs when these are materialized from PICs built via Tucker-merge (resp. CP-merge) in Algorithm 2, and hence the name of the merging procedure. Therefore, some QPCs can exactly match existent tensorized architectures, and this certainly happens when these are materialized from PICs built via Algorithm 1. This gives a new point of view on traditional tensorized architectures, and new possibilities for representation learning [54] . Figure 3 illustrates how the materialization of a 3-variate function leads to a Tucker layer. This 1-to-1 mapping between tensorized PC architectures and QPCs will allow for a fair comparison in our experiments.Folding tensorized circuits for faster inference. The layers of a tensorized circuit that (i) share the same functional form and that (ii) can be evaluated in parallel, can be stacked together as to create a folded layer [36, 42] which speeds up inference and learning on GPU by orders of magnitude. For instance, let {ℓ i } F i=1 be F parallelizable Tucker layers each parameterized by a matrix W (i) of size K×K 2 . Such layers can be evaluated as a folded layer ℓ parameterized by a tensor W of size F ×K×K 2 , which computes the-otherwise sequential-F tucker layers in parallel. We illustrate folding in Fig. 2(d) , and later on in Fig. 4(c) . Note that (i) the input layers sharing the same function form can always be folded and that (ii) although a tensorized circuit may have many types of sum-product layers, using one type only is common in practice, and promotes depth-wise folding.Materializing QPCs can be memory intensive and time consuming, depending on: (i) the cost of evaluating the functions we need to materialize, (ii) the degree of parallelization of the required function evaluations, and (iii) the number of integration points K. To solve these issues, we introduce neural functional sharing [47] , i.e. we share multi-layer perceptrons as to parameterize multiple PIC units at once. This allows us to scale to larger models and datasets, as we make materialization faster and more memory-efficient than previous work [18] . PIC functional sharing. Functional sharing is to PICs as parameter-sharing is to PCs. This type of sharing can be applied over a group of input/integral units-grouped according to some criteriawhose functions have all the same number of input and output variables. Specifically, let γ = Figure 4 : From neural C-sharing to folded CP-layer (Section 3.3). We sketch a 4-headed MLP with Fourier-Features (a) which we use to parameterize a group of 4 integral units (at the same depth) of a PIC (b), whose materialization leads to a folded CP-layer parameterized by a tensor W of size 2 × 2 × K × K (c), with K being the number of integration point. Note that, during materialization, the FF-MLP block in (a) will be only evaluated K 2 times, and not 4K 2 .{u i } Ni=1 be a group of N input/integral units, each with function f i : R I → R O . The simplest form of functional sharing is to set all functions to be equal, i.e. ∀i, j ∈ [N ] : f i = f j . In this way, we reduce the number of function evaluations from N K to K as long as we materialize each f i w.r.t. the same integration points z ∈ R K , which is the case for Algorithm 3. We call this type of sharing F-sharing, as per full-sharing. More interestingly, leveraging functional composition, we may define f i = h i • f , so as sharing an inner function f for all unit functions. Similarly as before, as long as we materialize each f i w.r.t. the same quadrature points z ∈ R K , we would only need K function evaluations for f instead of N K, as we can share them with all outer functions h i for further evaluation. We call this type of sharing C-sharing, as per composite-sharing. The original implementation of PICs [18] used neither F-sharing nor C-sharing.Finally, we present and apply two different ways of grouping units. The first consists of grouping all input units, a technique which is only applicable when all input variables share the same domain. With this grouping, coupled with F-sharing, we would only need to materialize K × P parameters, and use them to parameterize every QPC input layer. The second consists of grouping all integral units at the same depth of the PIC structure, which we couple with C-sharing and materialize as a folded sum-product layer. Despite grouping units that materialize into a folded layer is a natural and convenient choice, note that we can also group units that do not materialize as such. Once all units in a PIC have been grouped, materialization can be performed per-group.PIC functional sharing with (multi-headed) MLPs. Similar to [18] , we parameterize PIC input and integral units with light-weight multi-layer perceptrons (MLPs). However, instead of using a single MLP for each function, we will apply functional sharing as we strive to make the QPC materialization faster and memory efficient. Specifically, consider a group of integral units γ = {u i } N i=1 , each with function f i : R I → R, over which we want to apply functional sharing. For every group γ, we would have an L + 1 layered MLP of the form:EQUATIONwhere FF : R I → R M is a Fourier-feature layer [48] , and each ϕ (γ) i: R M → R M is a standard linear layer followed by an element-wise non linearity ψ, i.e. ψ(Az + b), with A ∈ R M ×M , b ∈ R M , and M being the size of the MLP. Applying F-sharing over γ would simply consist of settingf i : R M → R := softplus(h (γ) • ϕ (γ) + b (γ) ), (neural F-sharing)where h (γ) ∈ R M and b (γ) ∈ R are group-dependent parameters, therefore making all functions in the group equal. Instead, to implement C-sharing, we parameterize each f i asf i : R M → R := softplus(h (i) • ϕ (γ) + b (i) ), (neural C-sharing)where h (i) ∈ R M and b (i) ∈ R are function-dependent parameters, effectively creating a multiheaded MLP. As an example, consider a folded CP-layer with F = 500 and K = 512-which we actually used in practice-resulting in 2F K 2 ≈ 262M trainable parameters. Assuming no bias term, an MLP with L = 2 and M = 256 would only instead require LM 2 + 2F M ≈ 387K trainable parameters to materialize the same tensor, resulting in more than 99% less trainable parameters. We illustrate such C-sharing in Fig. 4 . In Appendix C.1 we provide more details about our MLPs. Fast & memory-efficient QPC materialization. Combing PIC functional sharing and per-group materialization allows scaling the training of PICs via numerical quadrature, as we drastically reduce the number and the cost of function evaluations required for the QPC materialization. We can now materialize very large QPCs, matching the scale of recent over-parameterized PCs yet requiring up to 99% less trainable parameters when using a large K. This was not possible in the original formulation of PIC [18] as (i) the entire QPC was materialized in one-shot, not per-group, and (ii) no functional sharing was implemented, as each input/integral function had its own MLP. 
experiments: 
 In our experiments, we first benchmark the effectiveness of functional sharing for scaling the training of PICs via numerical quadrature, comparing it with standard PCs and PICs w/o functional sharing [18] . Then, following prior work [10, 32, 33, 18] , we compare QPCs and PCs as distribution estimators on several image datasets. We always train using the trapezoidal integration rule. We use an NVIDIA A100 40GB throughout our experiments. Our code is available at github.com/gengala/tenpics.Thanks to our pipeline, we can now use two recently introduced RGs tailored for image data which deliver architectures that scale better than those built out of classical RGs [45, 43, 31] : quad-trees (QTs), tree-shaped RGs, and quad-graphs (QGs), DAG-shaped RGs [36] . These are perfectly balanced RGs, and therefore applying Algorithm 1 over them would deliver balanced PIC structures amenable to depth-wise C-sharing of integral units. We report full details about QTs and QGs in Appendix B. We denote a tensorized architecture as [RG]-[sum-product layer]-[K], e.g. QT-CP-16, which can be trained as a standard PC or materialized as QPC from a PIC. We treat pixels as categorical variables, and, as such, our architectures model probability mass functions.Scaling PICs. For each model type, {PC, PIC}, we specify a pair (•, •) where the first (resp. second) argument specifies the sharing technique, {F, C, N}, for the input (resp. inner) layers/groups, where N stands for no sharing. In Fig. 5 , we report the time and GPU memory required to perform an Adam [23] optimization step using PCs ( ), and PICs with ( ) and without ( ) functional C-sharing over the integral unit groups. We note that PICs using functional sharing ( ) proves very effective for scaling, requiring comparable resources as standard PCs ( ), while those who do not ( )like prior work [18] -are orders of magnitude slower and quickly go Out-Of-Memory (OOM) for K > 64. Remarkably, some QG-TK configurations of PICs ( ), see Table D .2, require even less GPU memory than PCs, and this is because of the significant difference in the number of trainable parameters, since copies of these have to be stored by Adam during optimization. In fact, the number of parameters for PCs and PICs w/o functional sharing ( , ) is in the order of hundreds of millions K = 16 K = 32 K = 64 K = 128 K = 256 K = 512Figure 6 : QPCs improve over PCs and other DGM baselines in terms of test-set bpd for the MNIST-family datasets. We compare against SparsePC [10] , HCLT [31] , RAT-SPN [43] , IDF [21] , BitSwap [25] , BBans [50] and McBits [46] . HCLT results are taken from [18] . Columns QPC and PC report results from this paper, with QG-CP-512 being the best performing architecture for both. Scatter plot (right): bpd for QPCs (y-axis) and PCs (x-axis) paired by architecture and min-max normalized for the MNIST and F-MNIST datasets. A point below the diagonal is a win for QPCs. (hitting 2B+), while PICs with functional sharing ( ) scale much more gracefully, hitting only 6M+ parameters. We emphasize that the number of trainable parameters of PICs is independent of the K at which we materialize, but only dependent on the size of the MLPs M we use to parameterize them, which can also be thought as the cost of evaluating PIC functions. We report more (tabular) details in Appendix D.1.Distribution estimation. Following prior work [10, 32, 33, 18] , we extensively test QPCs and PCs as distribution estimators on standard image datasets. Our full results are in Appendix D.2, while we only report here the bits-per-dimension (bpd) of the best performing models, which always belong to a QG-CP architecture, reflecting the additional expressiveness of DAG-shaped RGs. Our full results also highlights how the the more expressive yet expensive Tucker layers we introduced for PICs deliver the best performance for small K, but are hard to scale. All QPCs are materialized from PICs applying F-sharing over input units and C-sharing over groups of integral units, i.e. PIC (F, C). We begin with the MNIST-family, which includes 6 datasets of gray-scale 28x28 images: MNIST [29] , FASHIONMNIST [55] , and EMNIST with its 4 splits [7] . Fig. 6 shows that QPCs generally perform best, improving over standard PCs (5/6), complex heuristic-based PC learning schemes as pruning-and-growing (5/6) [10] , and some deep generative models (DGMs) (6/6).Then, we move to larger RGB image datasets as CIFAR [28] , ImageNet32, ImageNet64 [14] , and CelebA [34] . To compare against prior work [32, 33] , we have to preprocess the datasets using the YCoCg transform, a lossy color-coding that consistently improves performance for PCs when applied to RGB images. 3 We also report results over datasets preprocessed with the lossless YCoCg-R transform [39] , effectively doubling the number of datasets. We report details about these transforms in Appendix C.3. From Table 1 , we see that QPCs prove again very competitive, consistently outperforming standard PCs commonly trained with Adam, and the best performing PC from the literature, HCLT, which is trained via EM schemes and patch-wise methods [32] . Furthermore, QPCs are close to PCs trained via latent variable distillation (LVD, LVD-PG in Table 1 ) [32, 33] , a framework that requires extra supervision over their latent spaces by distilling information from existing deep generative models (DGMs). This technique requires pre-trained DGMs, several heuristics, and a final fine-tuning stage via EM or SGD, while PIC training method is instead end-to-end and self-contained. 
discussion & conclusion: 
 With this work, we systematized the construction of PICs, extending them to DAG-like structures (Section 3.1), tensorizing (Section 3.2), and scaling their training with functional sharing (Section 3.3). In our experiments (Section 4), we showed how this pipeline is remarkably effective when the tractable approximations of PICs, QPCs, are used as distribution estimators. This in turn becomes a new and effective tool to learn PCs at scale. In fact, prior work has shown that naively training large PCs via EM or gradient-ascent is challenging, and that PC performance plateau as their size increases [10, 8, 32, 33] . Our contributions go beyond these limitations, while offering a simple, principled and fully-differentiable pipeline that delivers performance that rival more sophisticated alternatives [10, 32] (Section 4). We conjecture that this happens as training PCs via PICs drastically reduces the search space while allowing (i) smoother training dynamics and (ii) the materialization of arbitrarily large tractable models.The development of tractable models is an important task in machine learning as they provide many inference routines, and can be used in many down-stream applications such as tabular data modeling [8] , generative modeling [42] , lossless compression [30] , genetics [11] , knowledge-graphs [37] , constrained text generation [58] , and more. Our work has also certain parallels with tensor networks and (quasi-)tensor decompositions [51, 26, 22, 6, 49] , as [36] recently showed how hierarchical tensor decompositions can be represented using the language of tensorized circuits. Furthermore, we note that the recent non-monotonic PCs [35] (i.e., PCs with negative sum parameters) can also be thought as the result of a quadrature process from PICs whose function can return negative values.Our work does not come without limitations. Although we showed that training PICs with function sharing requires comparable resources as standard PCs, traditional continuous LV models as VAEs, flows and diffusion models are more scalable. Also, sampling from PICs is currently not possible, as we cannot perform differentiable sampling from our (multi-headed) MLPs. Future work may include the investigation of more efficient ways of training PICs, possibly using techniques as LVD or variational inference [24] to directly maximize PIC lower-bounds, requiring numerical quadrature only as fine-tuning step to distill a performant tractable model. We believe our work will foster new research in the field of generative modeling, and specifically in the realm of tractable models.A Background on Circuits Definition 3 (Circuit [4, 52] ). A circuit c over variables X is a parameterized computational graph encoding a function c(X), and comprising three kinds of computational units: input, product, and sum. Each product or sum unit u outputs a scalar and receives as inputs the output scalars of other units, denoted with the set in(u). Each unit u computes a function f u defined as: (i) f u (X u ) → R if u is an input unit, where f u is a function over variables X u ⊆ X, called its scope, (ii) Π i∈in(u) f i (X i ) if u is a product unit, and (iii) Σ i∈in(u) f i (X i ) if u is a sum unit, with w i ∈ R denoting the weighted sum parameters. The scope of a product or sum unit is the union of the scopes of its input units. Definition 4 (Probabilistic Circuit). A PC over variables X is a circuit c encoding a (possibly nonnormalized) distribution, e.g., a function that is non-negative for all values of X: c(x) ≥ 0, ∀x ∈ dom(X) Definition 5 (Smoothness). A circuit is smooth if, for each sum unit u, its inputs depend on the same variables: ∀u 1 , u 2 ∈ in(u), X u1 = X u2 . Definition 6 (Decomposability). A circuit is decomposable if the inputs of each product unit u depend on disjoint sets of variables: ∀u 1 , u 2 ∈ in(u), X u1 ̸ = X u2 . Definition 7 (Structured-decomposability [44, 12] ). A circuit is structured-decomposable if (i) it is smooth and decomposable, and (2) any pair of product units having the same scope decompose their scope at their input units in the same way.Although all tensorized architectures mentioned in this paper are smooth and decomposable, only PCs built from tree RGs are also structured-decomposable, and as such are potentially less expressive because they belong to a restricted class. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Experimental settings overview included in paper; key details included in appendix; full details in the included code.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Statistical significance indicated for all experimental results, including the nature of the error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Overview compute resources included in paper; key details discussed in appendix; full details listed in structured template-description of corresponding assets.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Positive: better interpretability of this class of generative AI models mentioned; Negative: problematic aspects of generative AI models in general mentioned.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] . Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Original papers cited and/or URLs provided; license mentioned in reference. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In Algorithm B.1 we detail the construction of the Quad-Tree (QT) and Quad-Graph (QG) region graphs [36] . Specifically, QTs (resp. QGs) are built setting the input flag isTree to True (resp. False). Intuitively, these RGs recursively split an image into patches, until reaching regions associated to exactly one pixel. The splitting is performed both horizontally and vertically, and subsequent patches can either be shared, thus yielding a RG that is not a tree (QGs), or not (QTs).Input: Image height H, image width W , and whether to enforce the output RG to be a tree.Output: A RG R over H • W variables 1: S ← {Xij = {Xij} | (i, j) ∈ [H] × [W ]} 2: R ← a RG with leaf regions S 3: h ← H; w ← W 4: while h > 1 ∨ w > 1 do 5: h ← ⌈h/2⌉; w ← ⌈w/2⌉; S ′ ← ∅ 6: for i, j ∈ [h] × [w] do 7: Ω←({2i-1, 2i}×{2j-1, 2j})∩([H]×[W ]) 8: if |Ω| = 1 then 9:Let Xpq ∈ S s.t. (p, q) ∈ Ω 10:addRegion(R, Xpq) 11:else if |Ω| = 2 then 12:Let Xpq, Xrs ∈ S s.t. 13:(p, q), (r, s) ∈ Ω, p < r, q < s 14:addPartition(R, Xpq ∪ Xrs, {Xpq, Xrs}) 15:else ▷ |Ω| = 4 16:if isTree then mergeTree(R, Ω, S) 17:else mergeDAG(R, Ω, S) (p + u, q + v) ∈ Ω, u, v ∈ {0, 1} 3: X ← X00 ∪ X01 ∪ X10 ∪ X11 4: addPartition(R, X, {X00, X01, X10, X11}) Algorithm B.3 mergeDAG(R, Ω, S) Input: A RG R,(p + u, q + v) ∈ Ω, u, v ∈ {0, 1} 3: X ← X00 ∪ X01 ∪ X10 ∪ X11 4: addPartition(R, X, {X00 ∪X01, X10 ∪X11}) 5: addPartition(R, X, {X00 ∪X10, X01 ∪X11}) 6: addPartition(R, X00 ∪ X01, {X00, X01}) 7: addPartition(R, X10 ∪ X11, {X10, X11}) 8: addPartition(R, X00 ∪ X10, {X00, X10}) 9: addPartition(R, X01 ∪ X11, {X01, X11})A multi-headed MLP of size M parameterizing a group γ = {u i } N i=1 of N PIC units with functions of the form R I → R O consists of:1. A Fourier-Features Layer (details below), i.e. a non-linear mapping R I → R M ; 2. Two linear layers followed by hyperbolic tangent as activation function, i.e. two consecutive non-linear mappings R M → R M ; 3. N heads with Softplus non-linearity, i.e. N different non-linear mapping R M → R O .Note that, if γ is a group of CP (resp. Tucker) integral units, then I = 2 (resp. I = 3), while the output dimension O is always equal to 1. Instead, if the group γ is a group of input units, the input dimension I is always equal to 1, while the output dimension O is equal to the number of required parameters of the specific distribution, e.g. O = 2 for Gaussians.Such multi-headed MLP is implemented using grouped 1D convolutions, which allow a one-shot materialization of all the layer parameters associated to the group. We found that initializing all the heads to be equal improves convergence.Fourier Feature Layer. Fourier Feature Layers (FFLs) are an important ingredient for the multiheaded MLPs. FFLs [48] enable MLPs to learn high-frequency functions in low-dimensional problem domains and are usually used as first layers of coordinate-based MLPs. FFLs transform input z ∈ R I toFFL(z) : R I → R M := [cos(2πf ⊺ 1 z), sin(2πf ⊺ 1 z), . . . , cos(2πf ⊺ M/2 z), sin(2πf ⊺ M/2 z)],where M is a hyper-parameter and vectors f i ∈ R I are non-learnable, randomly initialized parameters. FFLs have two main benefits: (i) They allow learning more expressive functions by avoiding over-smoothing behaviors, and (ii) they reduce the total number of trainable parameters when used instead of conventional linear layers as the initial layers in MLPs.We train both PICs and PCs using the same training setup. Specifically, for each dataset, we perform a training cycle of T optimization steps, after which we perform a validation step and stop training if the validation log-likelihood did not improve by δ nats after 5 training cycles. Using δ > 0 can avoid long trainings with negligible improvements. We report these common training hyper-parameters in Table C .1. We use Adam [23] and a batch size of 256 for all experiments.PIC training. After some preliminary runs, we found that a learning rate of 5 × 10 -3 worked best, which we annealed towards 10 -4 using cosine annealing with warm restarts across 500 optimization steps [38] . We also apply weight decay with λ = 0.01.PC training. After some preliminary runs, we found that a constant learning rate of 0.01 worked best for all PC models, and for all datasets. We keep the PC parameters unnormalized, and, as such, we clamp them to a small positive value (10 -19 ) after each Adam update to keep them non-negative, and subtract the log normalization constant to normalize the log-likelihoods. QT-CP QG-CP QG-TK PIC 1.1M 2.2M 1.8M K=16 270K 800K 6M K=32 1M 3M 51M K=64 4M 13M 408M K=128 17M 51.1M - K=256 69M 204M - K=512 277M 817M - Figure D.1:Training PICs using functional sharing requires comparable resources as PCs. We compare the average GPU memory (left) and time (right) required to perform an Adam optimization step with PCs (blue) and PICs (green), varying region graph type, parameterizations, and K. We pair the plots with a table reporting the number of parameters of PCs at different K and PICs, with the latter being independent of K and allowing up to 99% less trainable parameters (QG-CP-512). The benchmark is conducted using a batch size of 256 gray-scale images of size 28x28, i.e. MNIST-like. Extra (tabular) details in Table D .1.Training PICs using functional sharing requires comparable resources as PCs. We report the time (in milliseconds, top) and GPU memory (in GiB, bottom) required to perform an Adam optimization step on 256 MNIST-like images by varying: architecture ({QT-CP, QG-CP, QG-TK}), size K ({2 i } 9 i=4 ), model ({PC, PIC}), and sharing technique ({C, F, N}). For each model we attach a pair (•, •) where the first (resp. second) argument specifies the sharing technique for the input (resp. inner) layer(s). EQUATIONIn this section, we report tabular results for all our experiments.Note that, every input layer of a standard PC is parameterized by a matrix K × P , where P is the number of categories, which is 256 for grey-scale image datasets and 768 =256 • 3 for RGB images. We found that sharing a single input layer among all pixels results in (slightly) worse performance for grey-scale images (as detailed in Table D 3 .381 ± 0.001 3.381 ± 0.002 3.317 ± 0.005 3.375 ± 0.005 3.499 ± 0.006 3.560 ± 0.007 EMN-MN 1.706 ± 0.007 1.761 ± 0.005 1.643 ± 0.007 1.711 ± 0.006 1.756 ± 0.002 1.772 ± 0.004 EMN-LE 1.698 ± 0.006 1.735 ± 0.007 1.626 ± 0.004 1.656 ± 0.004 1.725 ± 0.003 1.728 ± 0.003 EMN-BA 1.731 ± 0.007 1.772 ± 0.007 1.665 ± 0.004 1.696 ± 0.003 1.751 ± 0.002 1.749 ± 0.005 EMN-BY 1.542 ± 0.008 1.548 ± 0.007 1.474 ± 0.009 1.481 ± 0.007 1.665 ± 0.007 1.679 ± 0.006 Table D .4: QPCs consistently improve over PCs on MNIST and FASHIONMNIST. Test-set bitsper-dimension (bpd) on MNIST (top) and FASHIONMNIST (bottom) averaged over 5 runs. All QPCs are materialized from PICs applying F-sharing over the group of input units, and C-sharing over the integral units groups, i.e. QPCs are materialized from PICs (F, C). PCs do not apply any form of parameter sharing, as these deliver the best performance for these datasets, as detailed in Table D • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: Problematic aspects of generative AI mentioned, used asset licenses mentioned, produced assets (code, models) licensed and shared, reporducibility ensured.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).We refer the reader to Appendix A for an introductory overview of (probabilistic) circuits.The use of the lossy YCoCg transform is undocumented in[32,33] but confirmed via personal communication with the authors. Note that columns in Table1using it ( †) are not directly comparable to the rest. 


Decision: This work presents a novel construction for probabilistic integral circuits (PICs) over general directed acyclic graphs.  Extending PICs beyond trees increases their representational power. Overall, the reviewers felt that the work was clearly presented and represented a significantly novel contribution to the field with only minor weaknesses.

Reviews: review: paper_summary: paper_summary: This work extends the probabilistic integral circuits (PICs) from tree-shaped region graphs (RGs) based ones to DAG-shaped ones. While constructing PICs from DAG RGs can lead to more expressive models, it comes with the concern of scalability since the circuit sizes might be increased a lot. To address this concern, this work proposes tensorized circuits to approximate the resulting PICs by a PIC to QPC reduction. This is further combined with functional sharing and materialization to achieve efficiency and scalability, which is validated by experimental results on several image datasets.
strength_weakness: strength_weakness: I appreciate that this work aims to solve an important problem and the proposed solution is shown to be novel and efficient. Still, it's heavy in technical details and thus I don't think I fully understand the algorithmic details. Specifically,
- I'm confused by the definition of the integral unit from Line 58 to Line 62, especially by the inconsistency in the input of g_i between Line 61 and Line 62 (the g_i inside the integral). Also, the integral in Line 62 contains the function g_i while Figure 1(b) does not, which is also confusing. It might be helpful to put a numeric example there.
- This also applies to the other technical sections of this work given that this work is heavy in techniques. It would be very helpful to provide a toy example to showcase the reduction of PICs to QPCs to improve accessibility and reproducibility for readers who are not well-versed in these areas.
- I'm curious to see some theoretical analysis on the circuit sizes characterized. I understand that the bounds might be loose and not that informative. Still, it might be helpful to understand how the PIC to QPC reduction affects the circuit sizes.
questions: questions: - Can you provide some examples to help understand the integral units?
- From the definition of g_u in Line 71, the integral is a function of both variables X_i and Z_u. Can you explain the claim "In this way, the output LVs of any unit u will simply be Zu" that directly follows the definition of g_u?
- Can you characterize the circuit sizes of the resulting PICs and QPCs?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces a new approach for building probabilistic integral circuits (PICs), a recently introduced probabilistic model that extends probabilistic circuits (PC) with continuous functions of latent variables (in addition to discrete latent variables), with inference performed using a numerical quadrature procedure. In particular, while the previous implementation was restricted to tree-structured region graphs/variable decompositions, it is shown how to construct PICs for arbitrary region graphs. Additionally, sharing of functions across different regions is used to improve parameter efficiency. Empirical results show that the architectural improvements enable scaling to complex image datasets such as ImageNet64, outperforming comparable PCs trained using maximum likelihood or EM.
strength_weakness: strength_weakness: The paper is arguably a little weak in terms of novelty as most of the new components are adaptations of existing ideas in the literature (e.g. tucker/cp layers, parameter sharing) to probabilistic integral circuits.
questions: questions: - What numerical quadrature rule was used? 
- In the conclusion, it is remarked that differentiable sampling from PICs is not possible. What is meant by differentiable here, and is it possible to sample from the materialized PC if one is not interested in gradients (e.g. for visualization)? 
- To what extent do the authors consider their method a continuous latent variable model, as opposed to a means of effectively constructing PCs? For instance, if the same quadrature points are used throughout training and evaluation, can one expect the MLPs to learn meaningful function values between evaluation points?
- Did the authors investigate learning PICs according to a HCLT region graph? This would give a better idea of whether a DAG (as opposed to tree-structured) region graph is necessary.

Typos:
- modes -> nodes 190
- capitalization of 'functional' 234
- 'pixels are categorical' -> 'pixels as categorical' 286
limitations: limitations: The limitations of the method are appropriately addressed in the paper.
paper_summary: paper_summary: This paper presents a pipeline to build probabilistic integral circuits (PICs) more generally as directed acyclic graphs, as opposed to the tree-shaped structure they were limited to before. This significantly increases the expressiveness of the PICs, improving their representation of distributions. The authors present a procedure for turning a region graph into a PIC, into a QPC that approximates the PIC, and finally into a folded QPC, which are effectively tensorized structures that significantly speed up the inference and learning process. From the tensorized PIC the authors then make further improvements on QPC training, using neural functional sharing techniques over groupings of the input and integral units to result in greater materialization of QPCs and far fewer parameters.
strength_weakness: strength_weakness: Not a major weakness, but as the authors also point out, it is currently not possible to sample from PICs, limiting their impact as a generative model.
questions: questions: 1. Among the techniques discussed in the paper, how significant is the tree-shape vs DAG-shape for empirical expressiveness of PICs? It appears that DAG-shaped PICs achieve slightly better bpds, but tree-shaped ones scale much better.

2. Section 3.2 describes a numerical quadrature rule as consisting of integration points and weights that minimize the integration error, and the authors later say that the same quadrature rule is used for each approximation. Does this raise any issues in approximation?
 
3. How crucial/limiting is the assumption that every integral unit integrates out all incoming latent variables?

4. Do the QPCs approximating PICs still support tractable inference (e.g. marginal probabilities) similar to traditional PCs? It would be great to clarify the probabilistic reasoning capabilities mentioned in the introduction.

Minor comments:
- Lines 133-134: do you also need $Z_{u_3}\neq Z_{u_4}$ for CP-merge?
- Table 1: what does LVD-PG refer to?
limitations: limitations: The authors acknowledge the limitation of PICs that, despite being a continuous latent variable model like many generative models, they do not support sampling. This is left as future work.
paper_summary: paper_summary: This paper extends the work of PIC with continuous LVs, proposes to tensorize and to use neural functional sharing to scale the model.
The results show that the proposed approach decreases the trainable parameters as well as the memory usage in training, and also reduces the running time. The approach also works well in density estimation tasks.
strength_weakness: strength_weakness: It is not clear to me what data set is used in the experiment of "Scaling PICs", and how many RVs are modelled. 

I compared the density estimation results in PIC[18] on the MNIST family and the results in this paper outperforms the ones in PIC[18].
As far as I understand, the PIC (F, C) in this paper has reduced its trainable parameters and therefore should in principle has smaller model capacity. I am wondering why the density estimation results of the proposed model outperform the original PIC? 

Similarly, it would be nice if the drop (or increase, though unlikely) of density estimation performance can be shown between PIC (F, N) and PIC (F, C) so readers will know how much they can lose/gain w.r.t. performance when applying the functional sharing.
questions: questions: - Could you elaborate why the trainable parameters in PIC (F, C) have reduced significantly compared with the PC (F, N), since there is no functional sharing in PCs as I understand. That is, why the PIC (F, C) can have much fewer trainable parameters than PC (F, N)? Is this comparison fare?

- In the left most plots in Fig 5, I would infer that the single orange triangles in upper left are PIC(F,N) with QG-TK, but without a dotted line connected it is a bit unclear. 

- is the ```and``` before $||$ in line 156 a typo?
limitations: limitations: The limitations are well discussed in the end of the paper.

Comments: comment: Thank you for the response and clarifications. I will keep my positive score.
Thank you for the answer. My concerns have been addressed and I will keep my positive rating.
Thank you for the detailed response! I think clarifying the tractable inference and sampling (although differentiable sampling still seems to be a challenge at this point) supported by the learned PICs will make the contribution more impactful. I will keep my score of 8.