Title: Scaling Continuous Latent Variable Models as Probabilistic Integral Circuits
Venue: NeurIPS 2024
Accepted: yes
Keywords: Latent variable models

probabilistic ML

tractable models
Abstract: Probabilistic integral circuits (PICs) have been recently introduced as probabilistic models enjoying the key ingredient behind expressive generative models: continuous latent variables (LVs). PICs are symbolic computational graphs defining continuous LV models as hierarchies of functions that are summed and multiplied together, or integrated over some LVs. They are tractable if LVs can be analytically integrated out, otherwise they can be approximated by tractable probabilistic circuits (PC) encoding a hierarchical numerical quadrature process, called QPCs.

So far, only tree-shaped PICs have been explored, and training them via numerical quadrature requires memory-intensive processing at scale. In this paper, we address these issues, and present: (i) a pipeline for building DAG-shaped PICs out of arbitrary variable decompositions, (ii) a procedure for training PICs using tensorized circuit architectures, and (iii) neural functional sharing techniques to allow scalable training. In extensive experiments, we showcase the effectiveness of functional sharing and the superiority of QPCs over traditional PCs.

.

Abstract

Probabilistic integral circuits (PICs) have been recently introduced as probabilistic models enjoying the key ingredient behind expressive generative models: continuous latent variables (LVs). PICs are symbolic computational graphs defining continuous LV models as hierarchies of functions that are summed and multiplied together, or integrated over some LVs. They are tractable if LVs can be analytically integrated out, otherwise they can be approximated by tractable probabilistic circuits (PC) encoding a hierarchical numerical quadrature process, called QPCs. So far, only tree-shaped PICs have been explored, and training them via numerical quadrature requires memory-intensive processing at scale. In this paper, we address these issues, and present: (i) a pipeline for building DAG-shaped PICs out of arbitrary variable decompositions, (ii) a procedure for training PICs using tensorized circuit architectures, and (iii) neural functional sharing techniques to allow scalable training. In extensive experiments, we showcase the effectiveness of functional sharing and the superiority of QPCs over traditional PCs.

1 Introduction

Continuous latent variables (LVs) are arguably the key ingredient behind many successful generative models, from variational autoencoders [24] to generative adversarial networks [20], and more recently diffusion models [56]. While these models allow to learn expressive distributions from data, they are limited to sampling and require task-specific approximations when it comes to perform probabilistic reasoning , as even simple tasks such as computing marginals or conditionals are intractable for them. On the other hand, performing these tasks can be tractable for (hierarchical) discrete LV models [4, 3], but these prove to be more challenging to learn at scale [9, 10, 32, 33].

This inherent trade-off among tractability, ease of learning, and expressiveness can be analyzed and explored with probabilistic integral circuits (PICs) [18], a recently introduced class of deep generative models defining hierarchies of continuous LVs using symbolic functional circuits. PICs are tractable when their continuous LVs can be analytically integrated out. Intractable PICs can however be systematically approximated as (tensorized) probabilistic circuits (PCs) [53, 4], the representation language of discrete LV models. An instance of such PCs encodes a hierarchical numerical quadrature process of the PIC to approximate, and as such is called quadrature PC (QPC).

Distilling QPCs from PICs has proven to be an effective alternative way to train PCs, but it has only been explored for tree-shaped PICs, as building and scaling to richer LV structures is an open research question that requires new tools [18]. In this paper, we fill this gap by redefining the semantics of PICs and extending them to DAG-shaped hierarchies of continuous LVs. Specifically, we

,

∫

design PICs as a language to represent hierarchical quasi-tensors factorizations [49], parameterized by light-weight multi-layer perceptrons.

Contributions. (1) We present a systematic pipeline to build DAG-shaped PICs, starting from arbitrary variable decompositions (Section 3.1). (2) We show how to learn and approximate PICs via a hierarchical quadrature process which we encode in tensorized QPCs that match certain circuit architectures proposed in different prior works [43, 42, 31, 36] (Section 3.2). (3) We present functional sharing techniques to scale the training of PICs, which lead us to parameterize them with multi-headed multi-layer perceptrons (MLPs) requiring comparable resources as PCs (Section 3.3). (4) In extensive experiments (Section 4), we show that (i) functional sharing proves remarkably effective for scaling and that (ii) QPCs outperform PCs commonly trained via EM or SGD, while being distilled from PICs with up to 99% less trainable parameters .

2 Probabilistic integral circuits

Notation. We denote input variables as X and latent variables (LVs) as Y and Z , with x , y and z as their realization respectively. We denote scalars with lower-case letters (e.g., w ∈ R ), vectors with boldface lower-case letters (e.g., w ∈ R N ), matrices with boldface upper-case letters (excluding X , Y , Z , e.g., W ∈ R M × N ), and tensors with boldface calligraphic letters (e.g., W ∈ R L × M × N ).

A probabilistic integral circuit (PIC) c is a symbolic computational graph representing a nonnegative function c ( X ) = ∫ c ( X , z ) d z , i.e. c ( x ) ≥ 0 , over observed variables X and continuous latent variables Z . Similar to probabilistic circuits (PCs) [53, 4], PICs have input , sum and product units. 2 Different from PCs, however, PICs operate on functions, not scalars, and make use of a new type of unit: the integral unit ∫ , which allows to represent continuous LVs. To satisfy nonnegativity, we parameterize PICs with non-negative functions and positive sum weights, specifically:

An input unit u (depicted as in our figures) represents a possibly non-normalized distribution f u ( X u , Z u ) → R + , where X u ⊆ X and Z u ⊆ Z ;

A sum unit u ( + ) outputs a weighted sum of the functions it receives from its input units, i.e. g u ( X u , Z u ) = Σ i ∈ in ( u ) w i g i ( X i , Z i ) , where in ( u ) is the set of units u takes as input, w i > 0 , X u = ∪ i ∈ in ( u ) X i and Z u = ∪ i ∈ in ( u ) Z i . Similarly, a product unit u ( × ) outputs the product of its incoming functions, i.e. g u ( X u , Z u ) = Π i ∈ in ( u ) g i ( X i , Z i ) ;

̸

Finally, an integral unit u ( ∫ ) encodes an 'uncountable weighted sum' whose weights are compactly represented by a function f u ( Z u , Y u ) → R + , where ∅ = Y u ⊆ Z are the LVs that are being integrated out by u , while Z u can potentially be empty, i.e. Z u = ∅ . The unit receives a function g i ( X i , Y u ) from its only input unit i and outputs the function g u ( X i , Z u ) = ∫ ∆ f u ( Z u , y u ) g i ( X i , y u ) d y u , where ∆= supp ( Y u ) is the support of Y u . For instance, an integral unit u with f u ( { Z } , { Y } ) = 2 Z -Y 2 and supp ( Y ) = [ -1 , 1] receiving function g i ( { X } , { Y } ) = X 2 -3 X +4 Y would output g u ( { Z } , { X } ) = 2 / 3 X ( X -3)(6 Z -1) .

Fig. 1(b) and Fig. 2(b) show example PICs. Note that we use f to indicate the functions attached to input and integral units which are essentially parameters of the model, while we use g to indicate functions being outputted by all type of units. The output of a PIC c is the output function returned by its root unit u , which is only defined on X , i.e. c ( X ) = g u ( X ) and Z u = ∅ . Similar to PCs, imposing structural constraints over PICs can unlock tractable inference [4]. As such, we assume that (i) all + -units receive functions defined on the same input variables (aka smoothness ), and (ii) all × -units receive functions defined over disjoint sets of input variables (aka decomposability ).

PICs are tractable when their LVs can be analytically integrated out, meaning that we can passthrough integral units computing the integration problem they define, eventually outputting a function. Notably, this is possible when LVs are in linear-Gaussian relationships [27] or when functions are polynomials. Intractable PICs can however be approximated via a hierarchical numerical quadrature process that can be encoded as a PC called quadrature PC (QPC). Intuitively, each PIC integral unit can be approximated by a set of sum units in a QPC, each conditioning on some previously computed quadrature values, with a large but finite number of input units [18]. Materializing a QPC allows to train PICs by approximate maximum likelihood: Given a PIC, gradients to its parameters

attached to input, sum and integral units can be backpropagated through the corresponding QPC [18]. This also provides an alternative way to train PCs that can rival traditional learners.

So far, the construction of PICs has been limited to a simple compilation process from probabilistic graphical models (PGMs) [27] with continuous LVs [18]. In a nutshell, the LV nodes of a PGM become integral units of a PIC model, and the PGM (conditional) distributions become the functions f u attached to input and integral units of the PIC, as we illustrate in Fig. 1. However, the PGM structure needs to be limited to a tree , as to avoid that the hierarchical quadrature process would yield an exponentially large QPC, thus hindering learning. This imposes a semantics for current PICs as sim-

ple latent tree models [3], and clearly limits their expressiveness as more complex LV interactions are not possible. Building more expressive PICs requires reinterpreting this semantics and introducing new tools, which we do next.

3 Building, learning and scaling PICs

In Section 3.1, we systematize the construction of DAG-shaped PICs, showing how to build them starting from arbitrary variable decompositions, going beyond the current state-of-the-art [18]. Then, in Section 3.2, we show how to learn and approximate such PICs with QPCs encoding a hierarchical quadrature process, retrieving PC architectures proposed in prior works [36]. Finally, in Section 3.3, we present (neural) functional sharing, a technique which we use to parameterize PICs as to make their QPC materialization fast and cheap, allowing scaling to larger models and larger datasets.

3.1 Building PICs from arbitrary variable decompositions

Standard PCs can be built according to established pipelines that allow to flexibly represent arbitrary variable decompositions, as well as rich discrete LV interactions [36, 41]. In the following, we derive an analogous pipeline for PICs that allows to take care of continuous LVs, going beyond their current tree-shaped semantics (Section 2) yet allowing to perform hierarchical quadrature without blowing up the size of the materialized QPCs. To do so, we start by formalizing the notion of hierarchical variable decomposition, or region graph, out of which we will build our PIC structures.

Definition 1 (Region Graph (RG) [15]) . An RG R over input variables X is a bipartite and rooted directed acyclic graph (DAG) whose nodes are either regions , denoting subsets of X , or partitions , specifying how a region is partitioned into other regions (Fig. 2(a)).

RGs can be (i) compiled from PGMs [5, 27, 3, 31], (ii) randomly initialized [43, 16], (iii) learned from data [15, 19, 40, 57], or (iv) built according to the data modality (e.g. images) [45, 42, 36]. If we compile from a tree PGM, as in (Fig. 1), the resulting RG will be a tree, thus yielding a tree-like PIC [18]. Our pipeline, detailed in Algorithm 1, takes an arbitrary DAG-shaped RG as input, and can deliver DAG-like PICs. Without loss of generality, we assume to have an RG R which only allows for (i) binary partitionings of regions (i.e. all product units will have two input units) and (ii) univariate leaves, as shown in Fig. 2(a). Our construction iteratively builds a PIC in a bottom-up fashion, associating regions to PIC units. For every leaf region X u ∈ X in R , we instantiate an input unit u with function f u ( { X u } , { Z u } ) , where Z u ∈ Z is an arbitrary continuous LV (Line 8, Algorithm 1). Such functions can be univariate conditional densities, i.e. p u ( X u | Z u ) , resembling small VAE-like decoders [24] amenable to be numerically integrated.

Once all leaf regions have been processed, we move to the inner ones. Let X ⊆ X be an inner region partitioned in N ≥ 1 different ways as { ( X ( n ) 1 , X ( n ) 2 ) } N n =1 , i.e. ( X ( n ) 1 ∩ X ( n ) 2 ) = ∅ and ( X ( n ) 1 ∪ X ( n ) 2 ) = X for every n . For each partition ( X ( n ) 1 , X ( n ) 2 ) , we will merge the PIC units associated to regions X ( n ) 1 and X ( n ) 2 using consecutive applications of product and integral units-as we explain nexteventually associating a unit the partition itself. One can design such merging as desired, as long as smoothness and decomposability are not violated. Finally, in case N =1 , we associate to X the unit associated to its only partition, otherwise, in case N > 1 , we merge the N units associated to each n -th partition using a sum unit which we then associate to X (Line 6, Algorithm 1).

̸

Algorithm 1 RG2PIC ( R )

Algorithm 2 merge ( u 1 , u 2 , ρ )

Input Units u 1 , u 2 , and boolean flag ρ Output ∫ or × unit u with ( u 1 , u 2 ) as descendants

1: procedure ( Tucker-merge ) ▷ Z u 1 = Z u 2

2: Z u ← ∅ if ρ else { Z } ⊂ Z \ { Z u 1 , Z u 2 }

3: return ∫ ( × ([ u 1 , u 2 ]) , f u ( Z u , { Z u 1 , Z u 2 } ))

4: procedure ( CP-merge ) ▷ Z u 1 = Z u 2

5: Z u 3 ← ∅ if ρ else { Z u 3 } ⊂ Z \ { Z u 1 }

6: u 3 ← ∫ ( u 1 , f u 3 ( Z u 3 , { Z u 1 } ))

7: Z u 4 ← ∅ if ρ else { Z u 4 } ⊂ Z \ { Z u 2 }

8: u 4 ← ∫ ( u 2 , f u 4 ( Z u 4 , { Z u 2 } ))

9: return × ([ u 3 , u 4 ])

̸

Merging PIC units. Let u 1 and u 2 be candidate units to merge, each outputting functions with LV Z u 1 and Z u 2 respectively. We present two ways of merging units: Tucker-merge and CP-merge , which we detail in Algorithm 2 and whose names will be clearer in the next section. If Z u 1 = Z u 2 , we use Tucker-merge : We merge u 1 and u 2 with a product, which is then input to an integral unit u with function f u ( { Z u } , { Z u 1 , Z u 2 } ) , where Z u ∈ Z \ { Z u 1 , Z u 2 } . Otherwise, if Z u 1 = Z u 2 , we use CP-merge : We add two integral units, u 3 with input u 1 and u 4 with input u 2 , which we finally merge with a product. We parameterize unit u 3 (resp. u 4 ) with f u 3 ( { Z u 3 } , { Z u 1 } ) (resp. f u 4 ( { Z u 4 } , { Z u 2 } ) ), where Z u 3 = Z u 1 (resp. Z u 4 = Z u 2 ). Note that whenever merging two units defined on X , we need to marginalize out the remaining LVs, without introducing new ones. We illustrate the application of Algorithm 1 in Fig. 2(a-b). Our pipeline generalizes the PICs used in prior work [18] (Fig. 1) as we can build them by just converting latent tree structures in tree RGs and using CP-merge as merging procedure. While we now do not need a PGM to build a complex PIC structure, one could try to reverse-engineer our PICs to retrieve a PGM via decompilation [1], the result would be a very intricate hierarchy over continuous LVs [41].

̸

3.2 Learning PICs via tensorized QPCs

Given a DAG-shaped (intractable) PIC, we now show how to approximate it with a tensorized PC encoding a hierarchical quadrature process, namely a QPC. Intuitively, we interpret PICs as to encode a set of quasi-tensors [49], a generalization of tensors with potentially infinite entries in each dimension corresponding to a continuous LV, which we materialize into classical tensors via quadrature. We begin with a definition of tensorized circuits and a brief refresher on numerical quadrature.

Definition 2 (Tensorized Circuit [36, 43]) . A tensorized circuit c is a parameterized computational graph encoding a function c ( X ) ∈ R , and comprising of input , product × and sum + layers. Each layer consists of many computational units defined over the same variables, and every noninput layer receives vectors as input from one or more layers. Each input layer ℓ is defined on variables X ℓ ⊆ X and computes a collection of K ℓ parametric functions [ f k : dom ( X ℓ ) → R ] K ℓ k =1 , outputting a K ℓ -dimensional vector. Each product layer ℓ computes either an Hadamard product ( ⊙ ) or a Kronecker product ( ⊗ ) of the vectors it receives from its inputs layers. Specifically, the Hadamard product is an element-wise product of vectors, and therefore applicable when these have same size, while the outer product of two vectors u ∈ R N and v ∈ R M is w = u ⊗ v = || N i =1 u i v ∈ R NM , where || is the concatenation operator. Finally, a sum layer ℓ with S ℓ sum units receives inputs from

̸

N layers { ℓ i } N i =1 and computes the matrix-vector product W || N i =1 ℓ i ( X ℓ i ) , where W ∈ R S ℓ × K , K =Σ N i =1 K ℓ i , are the sum layer parameters. When N =1 , then it simply computes W ℓ 1 ( X ℓ 1 ) .

Numerical quadrature. A numerical quadrature rule is an approximation of the definite integral of a function as a weighted sum of function evaluations at specified points [13]. Specifically, given some integrand f : R → R and interval ∆:=[ a, b ] , a quadrature rule consists of a set of K integration points ˜ z ∈ ∆ K and weights ˜ w ∈ R K minimizing the integration error ε K = | ∫ ∆ f ( z ) d z -Σ K k =1 ˜ w k f (˜ z k ) | , which goes to zero as K →∞ . To approximate an integral of an N -dimensional function f , we can phrase the multiple integral as repeated one-dimensional integrals by applying Fubini's theorem [17], aka tensor product rule, as follows.

From PICs to QPCs. Given a candidate PIC, we will explore it in post order traversal, and iteratively associate a circuit layer (Definition 2) to each PIC unit, a process that we call materialization . We detail such procedure in Algorithm 3, which essentially applies Eq. (1) hierarchically over the PIC units. Each unit encodes a function over potentially continuous and discrete variables, hence representing a quasi-tensor, which we approximate by evaluating it over the quadrature points only, thus materializing a classical tensor (Fig. 3 (a,b)). We facilitate quadrature by assuming that all PIC LVs have bounded domain ∆:=[ -1 , 1] . This way, we can always use the same quadrature rule (˜ z , ˜ w ) for each required (multivariate) approximation, and also simplify treatment and exposition.

We begin materializing every PIC input unit u with function f u ( { X u } , { Z u } ) w.r.t. integration points ˜ z , effectively creating an input layer ℓ : dom ( X u ) → R K as [ f u ( X u , Z u =˜ z k ) ] K k =1 (Line 4, Algorithm 3). The parameters of such layer can be materialized as a matrix of shape K × P , where P is the number of parameters f u requires. For example, if f u is a univariate conditional Gaussian p u ( X u | Z u ) , we use a K × 2 matrix for parameterizing the layer, where each row stores the mean and standard deviation at each integration point ˜ z k .

Next, we address the most important part of this quadrature process, i.e. the materialization of PIC integral units as sum layers. Specifically, let u be an integral unit with N -dimensional function f u ( Z u , Y u ) , where | Z u | = N Z , | Y u | = N Y and N = N Z + N Y . We materialize f u w.r.t. integration points ˜ z , effectively creating an N -dimensional tensor W ( u ) ∈ R K ×···× K , such that

Algorithm 3 PIC2QPC ( c, ˜ z , ˜ w )

Input PIC c , quadrature rule ˜ z , ˜ w ∈ R K Output Tensorized QPC

1: L ← map() ▷ from PIC units to layers

2: for each unit u ∈ postOrder ( c ) do

3:

is

u

then if

4: L [ u ] ← [ ( f u ( X u , Z u =˜ z k )) ] K k =1

5: else if u is ∫ ( u 1 , f u ) then

6: L [ u ] ← ˜ W ( u ) L [ u 1 ] via Eq. (3)

7:

i

is

+([

u

, w

8:

9:

10:

11:

12:

N

i

=1

||

W

(

u

×

else if

L

[

u

u

]

else if

u

is

#

←⊙

L

[

([

if

u

1

u

u

, u

1

N

i

=1

i

]

L

[

2

=

Z

]

←L

return

1

[

u

]

A QPC with

)

then

u

i

])

Z

2

u

L

L

⊗

via Eq. (4)

]

then else

2

u

[

c

]

as root layer

[

]

#

After materializing tensor W ( u ) , we flatten it w.r.t. variables Z u and Y u , so as creating a matrix W ( u ) of size K N Z × K N Y , an operation aka matricization . As last step, we plug-in the quadrature

)

weights ˜ w ∈ R K in W ( u ) , arriving to matrix ˜ W ( u ) of K N Z × K N Y , whose i -th row is

where ˜ w ⊗ N Y is the vector of size K N Y resulting from the N Y -times application of the Kronecker product ⊗ over ˜ w (Line 6, Algorithm 3). We illustrate this process in Fig. 3(a-c). Similarly, we also materialize every PIC sum unit u with weights { w i } N i =1 as a sum layer, but parameterized by

̸

where I K is the K × K identity matrix and || the concatenation operator (Line 8, Algorithm 3). Note that such sum layer can be seen as a mixing layer [42, 36]. Finally, consider a PIC product unit u with inputs u 1 and u 2 , each outputting functions with LVs Z u 1 and Z u 2 respectively. We associate to u an Hadamard product layer if Z u 1 = Z u 2 , or a Kronecker product layer if Z u 1 = Z u 2 , reflecting the fact that we are marginalizing out two different LVs. We summarize our PIC materializationillustrated in Fig. 2-in Algorithm 3, where we iteratively associate a PIC unit to a circuit layer, eventually delivering a tensorized QPC. We stress that being QPCs just standard PCs they enjoy their same properties (e.g. tractable marginalization). We will learn PICs via maximizing the likelihood of its QPC materialization.

QPCs as existing tensorized architectures. Materializing PICs built via Algorithm 1 delivers tensorized PCs with alternating sum and product layers, aka sum-product layers [36]. An instance of such layers is the Tucker layer, used in architectures like RAT-SPNs [43] and EiNets [42]. Specifically, a binary Tucker layer ℓ [51] computes

where W ∈ R K × K 2 and ℓ 1 , ℓ 2 are input layers of ℓ , each outputting a K -dimensional vector. In contrast, the recent HCLT architectures [31] use the canonical polyadic (CP) layer ℓ [2], i.e.

where W (1) , W (2) ∈ R K × K . We exactly recover Tucker (resp. CP) layers in our QPCs when these are materialized from PICs built via Tucker-merge (resp. CP-merge ) in Algorithm 2, and hence the name of the merging procedure. Therefore, some QPCs can exactly match existent tensorized architectures, and this certainly happens when these are materialized from PICs built via Algorithm 1. This gives a new point of view on traditional tensorized architectures, and new possibilities for representation learning [54]. Figure 3 illustrates how the materialization of a 3-variate function leads to a Tucker layer. This 1-to-1 mapping between tensorized PC architectures and QPCs will allow for a fair comparison in our experiments.

Folding tensorized circuits for faster inference. The layers of a tensorized circuit that (i) share the same functional form and that (ii) can be evaluated in parallel, can be stacked together as to create a folded layer [36, 42] which speeds up inference and learning on GPU by orders of magnitude. For instance, let { ℓ i } F i =1 be F parallelizable Tucker layers each parameterized by a matrix W ( i ) of size K × K 2 . Such layers can be evaluated as a folded layer ℓ parameterized by a tensor W of size F × K × K 2 , which computes the-otherwise sequentialF tucker layers in parallel. We illustrate folding in Fig. 2(d), and later on in Fig. 4(c). Note that (i) the input layers sharing the same function form can always be folded and that (ii) although a tensorized circuit may have many types of sum-product layers, using one type only is common in practice, and promotes depth-wise folding.

3.3 Scaling PICs with neural functional sharing

Materializing QPCs can be memory intensive and time consuming, depending on: (i) the cost of evaluating the functions we need to materialize, (ii) the degree of parallelization of the required function evaluations, and (iii) the number of integration points K . To solve these issues, we introduce neural functional sharing [47], i.e. we share multi-layer perceptrons as to parameterize multiple PIC units at once. This allows us to scale to larger models and datasets, as we make materialization faster and more memory-efficient than previous work [18].

PIC functional sharing. Functional sharing is to PICs as parameter-sharing is to PCs. This type of sharing can be applied over a group of input/integral units-grouped according to some criteriawhose functions have all the same number of input and output variables. Specifically, let γ =

{ u i } N i =1 be a group of N input/integral units, each with function f i : R I → R O . The simplest form of functional sharing is to set all functions to be equal, i.e. ∀ i, j ∈ [ N ] : f i = f j . In this way, we reduce the number of function evaluations from NK to K as long as we materialize each f i w.r.t. the same integration points ˜ z ∈ R K , which is the case for Algorithm 3. We call this type of sharing F-sharing , as per full-sharing . More interestingly, leveraging functional composition, we may define f i = h i ◦ f , so as sharing an inner function f for all unit functions. Similarly as before, as long as we materialize each f i w.r.t. the same quadrature points ˜ z ∈ R K , we would only need K function evaluations for f instead of NK , as we can share them with all outer functions h i for further evaluation. We call this type of sharing C-sharing , as per composite-sharing . The original implementation of PICs [18] used neither F-sharing nor C-sharing.

Finally, we present and apply two different ways of grouping units. The first consists of grouping all input units, a technique which is only applicable when all input variables share the same domain. With this grouping, coupled with F-sharing, we would only need to materialize K × P parameters, and use them to parameterize every QPC input layer. The second consists of grouping all integral units at the same depth of the PIC structure, which we couple with C-sharing and materialize as a folded sum-product layer. Despite grouping units that materialize into a folded layer is a natural and convenient choice, note that we can also group units that do not materialize as such. Once all units in a PIC have been grouped, materialization can be performed per-group.

PIC functional sharing with (multi-headed) MLPs. Similar to [18], we parameterize PIC input and integral units with light-weight multi-layer perceptrons (MLPs). However, instead of using a single MLP for each function, we will apply functional sharing as we strive to make the QPC materialization faster and memory efficient. Specifically, consider a group of integral units γ = { u i } N i =1 , each with function f i : R I → R , over which we want to apply functional sharing. For every group γ , we would have an L +1 layered MLP of the form:

where FF : R I → R M is a Fourier-feature layer [48], and each ϕ ( γ ) i : R M → R M is a standard linear layer followed by an element-wise non linearity ψ , i.e. ψ ( Az + b ) , with A ∈ R M × M , b ∈ R M , and M being the size of the MLP. Applying F-sharing over γ would simply consist of setting

where h ( γ ) ∈ R M and b ( γ ) ∈ R are group-dependent parameters, therefore making all functions in the group equal. Instead, to implement C-sharing, we parameterize each f i as

where h ( i ) ∈ R M and b ( i ) ∈ R are function-dependent parameters, effectively creating a multiheaded MLP. As an example, consider a folded CP-layer with F =500 and K =512 -which we actually used in practice-resulting in 2 FK 2 ≈ 262 M trainable parameters. Assuming no bias term, an MLP with L =2 and M =256 would only instead require LM 2 +2 FM ≈ 387 Ktrainable parameters to materialize the same tensor, resulting in more than 99% less trainable parameters. We illustrate such C-sharing in Fig. 4. In Appendix C.1 we provide more details about our MLPs.

16

32

64

128

256

0

5

10

15

20

25

K

=

16

32

64

128

256

100

200

300

400

K

=

Time (ms)

16

32

64

128

256

1B

2B

K

=

Num. of trainable parameters

32

64

128

256

1B

2B

M

=

32

64

128

256

2M

5M

M

=

QT-CP

QG-CP

QG-TK

PC (F, N)

PIC (F, N)

PIC (F, C)

4 Experiments

In our experiments, we first benchmark the effectiveness of functional sharing for scaling the training of PICs via numerical quadrature, comparing it with standard PCs and PICs w/o functional sharing [18]. Then, following prior work [10, 32, 33, 18], we compare QPCs and PCs as distribution estimators on several image datasets. We always train using the trapezoidal integration rule. We use an NVIDIA A100 40GB throughout our experiments. Our code is available at github.com/gengala/tenpics.

Thanks to our pipeline, we can now use two recently introduced RGs tailored for image data which deliver architectures that scale better than those built out of classical RGs [45, 43, 31]: quad-trees (QTs), tree-shaped RGs, and quad-graphs (QGs), DAG-shaped RGs [36]. These are perfectly balanced RGs, and therefore applying Algorithm 1 over them would deliver balanced PIC structures amenable to depth-wise C-sharing of integral units. We report full details about QTs and QGs in Appendix B. We denote a tensorized architecture as [RG]-[sum-product layer]-[ K ], e.g. QT-CP16, which can be trained as a standard PC or materialized as QPC from a PIC. We treat pixels as categorical variables, and, as such, our architectures model probability mass functions.

Scaling PICs. For each model type, { PC , PIC } , we specify a pair ( · , · ) where the first (resp. second) argument specifies the sharing technique, { F, C, N } , for the input (resp. inner) layers/groups, where N stands for no sharing . In Fig. 5, we report the time and GPU memory required to perform an Adam [23] optimization step using PCs ( ), and PICs with ( ) and without ( ) functional C-sharing over the integral unit groups. We note that PICs using functional sharing ( ) proves very effective for scaling, requiring comparable resources as standard PCs ( ), while those who do not ( )like prior work [18]-are orders of magnitude slower and quickly go Out-Of-Memory (OOM) for K> 64 . Remarkably, some QG-TK configurations of PICs ( ), see Table D.2, require even less GPU memory than PCs, and this is because of the significant difference in the number of trainable parameters, since copies of these have to be stored by Adam during optimization. In fact, the number of parameters for PCs and PICs w/o functional sharing ( , ) is in the order of hundreds of millions

0.00

1.00

QPC

PC

min-max bpd

K

= 16

K

= 32

K

= 64

K

= 128

K

= 256

K

= 512

(hitting 2B+), while PICs with functional sharing ( ) scale much more gracefully, hitting only 6M+ parameters. We emphasize that the number of trainable parameters of PICs is independent of the K at which we materialize, but only dependent on the size of the MLPs M we use to parameterize them, which can also be thought as the cost of evaluating PIC functions. We report more (tabular) details in Appendix D.1.

Distribution estimation. Following prior work [10, 32, 33, 18], we extensively test QPCs and PCs as distribution estimators on standard image datasets. Our full results are in Appendix D.2, while we only report here the bits-per-dimension (bpd) of the best performing models, which always belong to a QG-CP architecture, reflecting the additional expressiveness of DAG-shaped RGs. Our full results also highlights how the the more expressive yet expensive Tucker layers we introduced for PICs deliver the best performance for small K , but are hard to scale. All QPCs are materialized from PICs applying F-sharing over input units and C-sharing over groups of integral units, i.e. PIC (F, C). We begin with the MNIST-family, which includes 6 datasets of gray-scale 28x28 images: MNIST [29], FASHIONMNIST [55], and EMNIST with its 4 splits [7]. Fig. 6 shows that QPCs generally perform best, improving over standard PCs (5/6), complex heuristic-based PC learning schemes as pruning-and-growing (5/6) [10], and some deep generative models (DGMs) (6/6).

Then, we move to larger RGB image datasets as CIFAR [28], ImageNet32, ImageNet64 [14], and CelebA [34]. To compare against prior work [32, 33], we have to preprocess the datasets using the YCoCg transform, a lossy color-coding that consistently improves performance for PCs when applied to RGB images. 3 We also report results over datasets preprocessed with the lossless YCoCg-R transform [39], effectively doubling the number of datasets. We report details about these transforms in Appendix C.3. From Table 1, we see that QPCs prove again very competitive, consistently outperforming standard PCs commonly trained with Adam, and the best performing PC from the literature, HCLT, which is trained via EM schemes and patch-wise methods [32]. Furthermore, QPCs are close to PCs trained via latent variable distillation (LVD, LVD-PG in Table 1) [32, 33], a framework that requires extra supervision over their latent spaces by distilling information from existing deep generative models (DGMs). This technique requires pre-trained DGMs, several heuristics, and a final fine-tuning stage via EM or SGD, while PIC training method is instead end-to-end and self-contained.

5 Discussion & Conclusion

With this work, we systematized the construction of PICs, extending them to DAG-like structures (Section 3.1), tensorizing (Section 3.2), and scaling their training with functional sharing (Section 3.3). In our experiments (Section 4), we showed how this pipeline is remarkably effective when the tractable approximations of PICs, QPCs, are used as distribution estimators. This in turn becomes a new and effective tool to learn PCs at scale. In fact, prior work has shown that naively training large PCs via EM or gradient-ascent is challenging, and that PC performance plateau as their size increases [10, 8, 32, 33]. Our contributions go beyond these limitations, while offering a simple, principled and fully-differentiable pipeline that delivers performance that rival more sophisticated alternatives [10, 32] (Section 4). We conjecture that this happens as training PCs via PICs drastically reduces the search space while allowing (i) smoother training dynamics and (ii) the materialization of arbitrarily large tractable models.

The development of tractable models is an important task in machine learning as they provide many inference routines, and can be used in many down-stream applications such as tabular data modeling [8], generative modeling [42], lossless compression [30], genetics [11], knowledge-graphs [37], constrained text generation [58], and more. Our work has also certain parallels with tensor networks and (quasi-)tensor decompositions [51, 26, 22, 6, 49], as [36] recently showed how hierarchical tensor decompositions can be represented using the language of tensorized circuits. Furthermore, we note that the recent non-monotonic PCs [35] (i.e., PCs with negative sum parameters) can also be thought as the result of a quadrature process from PICs whose function can return negative values.

Our work does not come without limitations. Although we showed that training PICs with function sharing requires comparable resources as standard PCs, traditional continuous LV models as VAEs, flows and diffusion models are more scalable. Also, sampling from PICs is currently not possible, as we cannot perform differentiable sampling from our (multi-headed) MLPs. Future work may include the investigation of more efficient ways of training PICs, possibly using techniques as LVD or variational inference [24] to directly maximize PIC lower-bounds, requiring numerical quadrature only as fine-tuning step to distill a performant tractable model. We believe our work will foster new research in the field of generative modeling, and specifically in the realm of tractable models.

Author Contributions

GGled the project, proposed neural functional sharing, and ran all experiments. GG and A V devised the original idea of a pipeline to build PICs, and leverage tensorized folded circuits for training them. AVandEQequally supervised all the phases of the project. CdC supervises the project, and critically read the manuscript and provided feedback.

Acknowledgments

The Eindhoven University of Technology authors thank the support from the Eindhoven Artificial Intelligence Systems Institute and the Department of Mathematics and Computer Science of TU Eindhoven. TU/e authors also thank the support of EU European Defence Fund Project KOIOS (EDF-2021-DIGIT-R-FL-KOIOS). AV was supported by the "UNREAL: Unified Reasoning Layer for Trustworthy ML" project (EP/Y023838/1) selected by the ERC and funded by UKRI EPSRC. We thank Lorenzo Loconte for insightful discussions about (quasi-)tensor decompositions.

References

Cory Butz, Jhonatan S Oliveira, and Robert Peharz. Sum-product network decompilation. In International Conference on Probabilistic Graphical Models , pages 53-64. PMLR, 2020.

J. Douglas Carroll and Jih-Jie Chang. Analysis of individual differences in multidimensional scaling via an n-way generalization of 'eckart-young' decomposition. Psychometrika , 35:283319, 1970.

Myung Jin Choi, Vincent Y. F. Tan, Animashree Anandkumar, and Alan S Willsky. Learning latent tree graphical models. Journal of Machine Learning Research , 12(49):1771-1812, 2011.

YooJung Choi, Antonio Vergari, and Guy Van den Broeck. Probabilistic circuits: A unifying framework for tractable probabilistic models. Technical report, UCLA, 2020.

C. Chow and C. Liu. Approximating discrete probability distributions with dependence trees. IEEE Transactions on Information Theory , 14(3):462-467, 1968.

Andrzej Cichocki and Anh Huy Phan. Fast local algorithms for large scale nonnegative matrix and tensor factorizations. IEICE Trans. Fundam. Electron. Commun. Comput. Sci. , 92A(3):708-721, 2009.

Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: Extending MNIST to handwritten letters. In IJCNN 2017 , pages 2921-2926, 2017.

Alvaro Correia, Robert Peharz, and Cassio P de Campos. Joints in random forests. In Advances in Neural Information Processing Systems , volume 33, pages 11404-11415, 2020.

Alvaro H. C. Correia, Gennaro Gala, Erik Quaeghebeur, Cassio de Campos, and Robert Peharz. Continuous mixtures of tractable probabilistic models. Proceedings of the AAAI Conference on Artificial Intelligence , 37(6):7244-7252, 2023.

Meihua Dang, Anji Liu, and Guy Van den Broeck. Sparse probabilistic circuits via pruning and growing. In NeurIPS 2022 , volume 35 of Advances in Neural Information Processing Systems , 2022.

Meihua Dang, Anji Liu, Xinzhu Wei, Sriram Sankararaman, and Guy Van den Broeck. Tractable and expressive generative models of genetic variation data. In Research in Computational Molecular Biology , pages 356-357, 2022.

Adnan Darwiche. Modeling and reasoning with Bayesian networks . Cambridge University Press, 2009.

Philip J. Davis and Philip Rabinowitz. Methods of numerical integration . Academic Press, 1984.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Aaron W. Dennis and Dan Ventura. Learning the architecture of sum-product networks using clustering on variables. In Advances in Neural Information Processing Systems 25 (NeurIPS) , pages 2033-2041. Curran Associates, Inc., 2012.

Nicola Di Mauro, Gennaro Gala, Marco Iannotta, and Teresa Maria Altomare Basile. Random probabilistic circuits. In 37th Conference on Uncertainty in Artificial Intelligence (UAI) , volume 161, pages 1682-1691. PMLR, 2021.

Guido Fubini. Sugli integrali multipli. Rend. Acc. Naz. Lincei , 16:608-614, 1907.

Gennaro Gala, Cassio de Campos, Robert Peharz, Antonio Vergari, and Erik Quaeghebeur. Probabilistic integral circuits. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , volume 238 of Proceedings of Machine Learning Research , pages 2143-2151. PMLR, 02-04 May 2024.

Robert Gens and Pedro M. Domingos. Learning the structure of sum-product networks. In International Conference on Machine Learning , 2013.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS 2014 , volume 27 of Advances in Neural Information Processing Systems , 2014.

Emiel Hoogeboom, Jorn Peters, Rianne van den Berg, and Max Welling. Integer discrete flows and lossless compression. In NeurIPS 2019 , volume 32 of Advances in Neural Information Processing Systems , 2019.

Yong-Deok Kim and Seungjin Choi. Nonnegative tucker decomposition. In CVPR . IEEE Computer Society, 2007.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR 2015 , 2015.

Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR 2014 , 2014.

Friso Kingma, Pieter Abbeel, and Jonathan Ho. Bit-swap: Recursive bits-back coding for lossless compression with hierarchical latent variables. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 3408-3417, 2019.

Tamara G. Kolda. Multilinear operators for higher-order decompositions. Technical report, Sandia National Laboratories, 2006.

Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques . MIT press, 2009.

Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.

Yann LeCun, Corinna Cortes, and Christopher J. C. Burges. The MNIST database of handwritten digits, 2010.

Anji Liu, Stephan Mandt, and Guy Van den Broeck. Lossless compression with probabilistic circuits. In ICLR 2022 , 2022.

Anji Liu and Guy Van den Broeck. Tractable regularization of probabilistic circuits. In NeurIPS 2021 , volume 34 of Advances in Neural Information Processing Systems , pages 35583570, 2021.

Anji Liu, Honghua Zhang, and Guy Van den Broeck. Scaling up probabilistic circuits by latent variable distillation. In ICLR 2023 , 2023.

Xuejie Liu, Anji Liu, Guy Van den Broeck, and Yitao Liang. Understanding the distillation process from deep generative models to tractable probabilistic circuits. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 21825-21838, 2023.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

Lorenzo Loconte, M. Sladek Aleksanteri, Stefan Mengel, Martin Trapp, Arno Solin, Nicolas Gillis, and Antonio Vergari. Subtractive mixture models via squaring: Representation and learning. In The Twelfth International Conference on Learning Representations (ICLR) , 2024.

Lorenzo Loconte, Antonio Mari, Gennaro Gala, Robert Peharz, Cassio de Campos, Erik Quaeghebeur, Gennaro Vessio, and Antonio Vergari. What is the relationship between tensor factorizations and circuits (and how can we exploit it)? arXiv preprint arXiv:2409.07953 , 2024.

Lorenzo Loconte, Nicola Di Mauro, Robert Peharz, and Antonio Vergari. How to turn your knowledge graph embeddings into generative models via probabilistic circuits. In Advances in Neural Information Processing Systems 37 (NeurIPS) . Curran Associates, Inc., 2023.

Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR 2017 , 2017.

Henrique Malvar and Gary Sullivan. Ycocg-r: A color space with rgb reversibility and low dynamic range. ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q , 6, 2003.

Alejandro Molina, Antonio Vergari, Nicola Di Mauro, Sriraam Natarajan, Floriana Esposito, and Kristian Kersting. Mixed sum-product networks: A deep architecture for hybrid domains. In AAAI Conference on Artificial Intelligence , 2018.

Robert Peharz, Robert Gens, Franz Pernkopf, and Pedro Domingos. On the latent variable interpretation in sum-product networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 39(10):2030-2044, 2017.

Robert Peharz, Steven Lang, Antonio Vergari, Karl Stelzner, Alejandro Molina, Martin Trapp, Guy Van Den Broeck, Kristian Kersting, and Zoubin Ghahramani. Einsum networks: Fast and scalable learning of tractable probabilistic circuits. In 37th International Conference on Machine Learning (ICML) , volume 119 of Proceedings of Machine Learning Research , pages 7563-7574. PMLR, 2020.

Robert Peharz, Antonio Vergari, Karl Stelzner, Alejandro Molina, Xiaoting Shao, Martin Trapp, Kristian Kersting, and Zoubin Ghahramani. Random sum-product networks: A simple and effective approach to probabilistic deep learning. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference , volume 115 of Proceedings of Machine Learning Research , pages 334-344, 2020.

Knot Pipatsrisawat and Adnan Darwiche. New compilation languages based on structured decomposability. In Proceedings of the 23rd National Conference on Artificial Intelligence (AAAI'08) , volume 1, pages 517-522, 2008.

Hoifung Poon and Pedro Domingos. Sum-product networks: A new deep architecture. In IEEE International Conference on Computer Vision Workshops (ICCV Workshops) , pages 689-690. IEEE, 2011.

Yangjun Ruan, Karen Ullrich, Daniel S. Severo, James Townsend, Ashish Khisti, Arnaud Doucet, Alireza Makhzani, and Chris Maddison. Improving lossless compression rates via monte carlo bits-back coding. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 9136-9147, 2021.

Eran Segal, Dana Pe'er, Aviv Regev, Daphne Koller, Nir Friedman, and Tommi Jaakkola. Learning module networks. Journal of Machine Learning Research , 6(4), 2005.

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS 2020 , volume 33 of Advances in Neural Information Processing Systems , pages 7537-7547, 2020.

Alex Townsend and Lloyd N Trefethen. Continuous analogues of matrix factorizations. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 471(2173):20140585, 2015.

James Townsend, Thomas Bird, and David Barber. Practical lossless compression with latent variables using bits back coding. In ICLR 2019 , 2019.

L. R. Tucker. The extension of factor analysis to three-dimensional matrices. In Contributions to mathematical psychology. , pages 110-127. Holt, Rinehart and Winston, 1964.

Antonio Vergari, YooJung Choi, Anji Liu, Stefano Teso, and Guy Van den Broeck. A compositional atlas of tractable circuit operations for probabilistic inference. In NeurIPS 2021 , volume 36 of Advances in Neural Information Processing Systems , 2021.

Antonio Vergari, Nicola Di Mauro, and Guy Van den Broeck. Tractable probabilistic models: Representations, algorithms, learning, and applications, 2019. Tutorial at the 35th Conference on Uncertainty in Artificial Intelligence (UAI 2019).

Antonio Vergari, Robert Peharz, Nicola Di Mauro, Alejandro Molina, Kristian Kersting, and Floriana Esposito. Sum-product autoencoding: Encoding and decoding representations using sum-product networks. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), 2018.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv , 2017.

Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys , 56(4):1-39, 2023.

Yang Yang, Gennaro Gala, and Robert Peharz. Bayesian structure scores for probabilistic circuits. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 563-575, 2023.

Honghua Zhang, Meihua Dang, Nanyun Peng, and Guy Van den Broeck. Tractable control for autoregressive language generation. In 40th International Conference on Machine Learning (ICML) , volume 202 of Proceedings of Machine Learning Research , pages 40932-40945. PMLR, 2023.

A Background on Circuits

Definition 3 (Circuit [4, 52]) . A circuit c over variables X is a parameterized computational graph encoding a function c ( X ) , and comprising three kinds of computational units: input, product, and sum. Each product or sum unit u outputs a scalar and receives as inputs the output scalars of other units, denoted with the set in ( u ) . Each unit u computes a function f u defined as: (i) f u ( X u ) → R if u is an input unit, where f u is a function over variables X u ⊆ X , called its scope, (ii) Π i ∈ in ( u ) f i ( X i ) if u is a product unit, and (iii) Σ i ∈ in ( u ) f i ( X i ) if u is a sum unit, with w i ∈ R denoting the weighted sum parameters. The scope of a product or sum unit is the union of the scopes of its input units.

Definition 4 (Probabilistic Circuit) . A PC over variables X is a circuit c encoding a (possibly nonnormalized) distribution, e.g., a function that is non-negative for all values of X :

Definition 5 (Smoothness) . A circuit is smooth if, for each sum unit u , its inputs depend on the same variables: ∀ u 1 , u 2 ∈ in ( u ) , X u 1 = X u 2 .

̸

Definition 6 (Decomposability) . A circuit is decomposable if the inputs of each product unit u depend on disjoint sets of variables: ∀ u 1 , u 2 ∈ in ( u ) , X u 1 = X u 2 .

Definition 7 (Structured-decomposability [44, 12]) . A circuit is structured-decomposable if (i) it is smooth and decomposable, and (2) any pair of product units having the same scope decompose their scope at their input units in the same way.

Although all tensorized architectures mentioned in this paper are smooth and decomposable, only PCs built from tree RGs are also structured-decomposable, and as such are potentially less expressive because they belong to a restricted class.

B Region Graphs

In Algorithm B.1 we detail the construction of the Quad-Tree (QT) and Quad-Graph (QG) region graphs [36]. Specifically, QTs (resp. QGs) are built setting the input flag isTree to True (resp. False). Intuitively, these RGs recursively split an image into patches, until reaching regions associated to exactly one pixel. The splitting is performed both horizontally and vertically, and subsequent patches can either be shared, thus yielding a RG that is not a tree (QGs), or not (QTs).

Algorithm B.1 buildQuadGraph ( H,W, isTree )

Algorithm B.2 mergeTree ( R , Ω , S )

Input: A RG R , a set of four coordinates Ω , and a set of regions S

Behavior: It merges the regions indexed by Ω in R by forming a tree structure

Algorithm B.3 mergeDAG ( R , Ω , S )

Input: A RG R , a set of four coordinates Ω , and a set of regions S

Behavior: It merges the regions indexed by Ω in R by forming a DAG structure

C Implementation details

C.1 Multi-headed MLP details

A multi-headed MLP of size M parameterizing a group γ = { u i } N i =1 of N PIC units with functions of the form R I → R O consists of:

A Fourier-Features Layer (details below), i.e. a non-linear mapping R I → R M ;

Two linear layers followed by hyperbolic tangent as activation function, i.e. two consecutive non-linear mappings R M → R M ;

N heads with Softplus non-linearity, i.e. N different non-linear mapping R M → R O .

Note that, if γ is a group of CP (resp. Tucker ) integral units, then I = 2 (resp. I = 3 ), while the output dimension O is always equal to 1. Instead, if the group γ is a group of input units, the input dimension I is always equal to 1, while the output dimension O is equal to the number of required parameters of the specific distribution, e.g. O =2 for Gaussians.

Such multi-headed MLP is implemented using grouped 1D convolutions, which allow a one-shot materialization of all the layer parameters associated to the group. We found that initializing all the heads to be equal improves convergence.

Fourier Feature Layer. Fourier Feature Layers (FFLs) are an important ingredient for the multiheaded MLPs. FFLs [48] enable MLPs to learn high-frequency functions in low-dimensional problem domains and are usually used as first layers of coordinate-based MLPs. FFLs transform input z ∈ R I to

where M is a hyper-parameter and vectors f i ∈ R I are non-learnable, randomly initialized parameters. FFLs have two main benefits: (i) They allow learning more expressive functions by avoiding over-smoothing behaviors, and (ii) they reduce the total number of trainable parameters when used instead of conventional linear layers as the initial layers in MLPs.

C.2 Training details

We train both PICs and PCs using the same training setup. Specifically, for each dataset, we perform a training cycle of T optimization steps, after which we perform a validation step and stop training if the validation log-likelihood did not improve by δ nats after 5 training cycles. Using δ > 0 can avoid long trainings with negligible improvements. We report these common training hyper-parameters in Table C.1. We use Adam [23] and a batch size of 256 for all experiments.

PIC training. After some preliminary runs, we found that a learning rate of 5 × 10 -3 worked best, which we annealed towards 10 -4 using cosine annealing with warm restarts across 500 optimization steps [38]. We also apply weight decay with λ = 0 . 01 .

PC training. After some preliminary runs, we found that a constant learning rate of 0.01 worked best for all PC models, and for all datasets. We keep the PC parameters unnormalized, and, as such, we clamp them to a small positive value ( 10 -19 ) after each Adam update to keep them non-negative, and subtract the log normalization constant to normalize the log-likelihoods.

C.3 YCoCg color-coding transforms

In Fig. C.1 and Fig. C.2 we provide pytorch code for the lossless and lossy versions of the YCoCg transform that we used in our experiments (Section 4). In Fig. C.3, we show how to apply them and that the lossy version is on average off less than a bit. Finally, in Fig. C.4 we show the significant visual difference of the two transforms when applied to an RGB image.

D Additional results

D.1 Scaling experiments

We report the time and GPU memory required to perform and Adamp optimization step for several model configurations in Table D.1, Fig. D.1 and Table D.2.

16

32

64

128

256

512

0

5

10

15

20

K

=

16

32

64

128

256

512

50

100

150

200

250

300

K

=

Time (ms)

QT-CP

QG-CP

QG-TK

PC (F, N)

PIC (F, C)

QT-CP

QG-CP

QG-TK

PIC

1.1M

2.2M

1.8M

K

=16

270K

800K

6M

K

=32

1M

3M

51M

K

=64

4M

13M

408M

K

=128

17M

51.1M

-

K

=256

69M

204M

-

K

=512

277M

817M

-

D.2 Additional distribution estimation results

In this section, we report tabular results for all our experiments.

Note that, every input layer of a standard PC is parameterized by a matrix K × P , where P is the number of categories, which is 256 for grey-scale image datasets and 768 =256 · 3 for RGB images. We found that sharing a single input layer among all pixels results in (slightly) worse performance for grey-scale images (as detailed in Table D.3). In contrast, we found that such sharing considerably improves performance for RGB image datasets. Besides improving performance for RGB image datasets, such sharing considerably lower the number of trainable parameters from D × K × P to only K × P where D is the number of pixels. For instance, parameterizing all input layers of a tensorized architecture with K =256 built for 64x64 images would require 805 , 306 , 368 = 256 · 64 · 64 · 768 parameters, while only 3 , 145 , 728 if we apply the sharing. Therefore, without applying such sharing, we cannot even scale to big tensorized architectures (e.g. QG-CP-512) on our GPUs.

All QPCs are materialized from PICs applying F-sharing over the group of input units, and C-sharing over the groups of integral units.

We extensively compare QPCs and PCs as density estimators on several image datasets and report test-set bits-per-dimension (bpd) in Table D.3, Table D.4, and Table D.5.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract and main text accurately and precisely state the actual claims of the research presented.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations mentioned in the last section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All data used is publicly available. All model and experimental code to fully reproduce our own experimental results is shared.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All data used is publicly available. All model and experimental code to fully reproduce our own experimental results is shared.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental settings overview included in paper; key details included in appendix; full details in the included code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Statistical significance indicated for all experimental results, including the nature of the error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Overview compute resources included in paper; key details discussed in appendix; full details listed in structured template-description of corresponding assets.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Problematic aspects of generative AI mentioned, used asset licenses mentioned, produced assets (code, models) licensed and shared, reporducibility ensured.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Positive: better interpretability of this class of generative AI models mentioned; Negative: problematic aspects of generative AI models in general mentioned.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Original papers cited and/or URLs provided; license mentioned in reference.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Code and models included (as zip file for submission, also as URL for final); Structured templates used for details.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the work of PIC with continuous LVs, proposes to tensorize and to use neural functional sharing to scale the model.\nThe results show that the proposed approach decreases the trainable parameters as well as the memory usage in training, and also reduces the running time. The approach also works well in density estimation tasks.\n\nSTRENGTHS:\nThis paper extends the work of PIC for better scalability and efficiency, and thus has its novelty.\n\nThe paper is well written and not difficult to understand. e.g., the illustration of Algorithm 1 in Fig.2 (a-b) is a very nice visualisation. \n\nWith the claimed contribution, the paper has a high potential to contribute to the community.\n\nWEAKNESSES:\nIt is not clear to me what data set is used in the experiment of \"Scaling PICs\", and how many RVs are modelled. \n\nI compared the density estimation results in PIC[18] on the MNIST family and the results in this paper outperforms the ones in PIC[18].\nAs far as I understand, the PIC (F, C) in this paper has reduced its trainable parameters and therefore should in principle has smaller model capacity. I am wondering why the density estimation results of the proposed model outperform the original PIC? \n\nSimilarly, it would be nice if the drop (or increase, though unlikely) of density estimation performance can be shown between PIC (F, N) and PIC (F, C) so readers will know how much they can lose/gain w.r.t. performance when applying the functional sharing.\n\nQUESTIONS:\n- Could you elaborate why the trainable parameters in PIC (F, C) have reduced significantly compared with the PC (F, N), since there is no functional sharing in PCs as I understand. That is, why the PIC (F, C) can have much fewer trainable parameters than PC (F, N)? Is this comparison fare?\n\n- In the left most plots in Fig 5, I would infer that the single orange triangles in upper left are PIC(F,N) with QG-TK, but without a dotted line connected it is a bit unclear. \n\n- is the ```and``` before $||$ in line 156 a typo?\n\nLIMITATIONS:\nThe limitations are well discussed in the end of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a pipeline to build probabilistic integral circuits (PICs) more generally as directed acyclic graphs, as opposed to the tree-shaped structure they were limited to before. This significantly increases the expressiveness of the PICs, improving their representation of distributions. The authors present a procedure for turning a region graph into a PIC, into a QPC that approximates the PIC, and finally into a folded QPC, which are effectively tensorized structures that significantly speed up the inference and learning process. From the tensorized PIC the authors then make further improvements on QPC training, using neural functional sharing techniques over groupings of the input and integral units to result in greater materialization of QPCs and far fewer parameters.\n\nSTRENGTHS:\n+ The proposed approaches significantly increase the expressivity and learning efficiency of PICs, as showcased by the experiments.\n\n+ The paper is technically sound with clear arguments presented for each step.\n\n+ The paper is overall well structured and easy to follow.\n\n+ The authors also address the current limitations of PICs with a clear goal for future improvements.\n\nWEAKNESSES:\nNot a major weakness, but as the authors also point out, it is currently not possible to sample from PICs, limiting their impact as a generative model.\n\nQUESTIONS:\n1. Among the techniques discussed in the paper, how significant is the tree-shape vs DAG-shape for empirical expressiveness of PICs? It appears that DAG-shaped PICs achieve slightly better bpds, but tree-shaped ones scale much better.\n\n2. Section 3.2 describes a numerical quadrature rule as consisting of integration points and weights that minimize the integration error, and the authors later say that the same quadrature rule is used for each approximation. Does this raise any issues in approximation?\n \n3. How crucial/limiting is the assumption that every integral unit integrates out all incoming latent variables?\n\n4. Do the QPCs approximating PICs still support tractable inference (e.g. marginal probabilities) similar to traditional PCs? It would be great to clarify the probabilistic reasoning capabilities mentioned in the introduction.\n\nMinor comments:\n- Lines 133-134: do you also need $Z_{u_3}\\neq Z_{u_4}$ for CP-merge?\n- Table 1: what does LVD-PG refer to?\n\nLIMITATIONS:\nThe authors acknowledge the limitation of PICs that, despite being a continuous latent variable model like many generative models, they do not support sampling. This is left as future work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new approach for building probabilistic integral circuits (PICs), a recently introduced probabilistic model that extends probabilistic circuits (PC) with continuous functions of latent variables (in addition to discrete latent variables), with inference performed using a numerical quadrature procedure. In particular, while the previous implementation was restricted to tree-structured region graphs/variable decompositions, it is shown how to construct PICs for arbitrary region graphs. Additionally, sharing of functions across different regions is used to improve parameter efficiency. Empirical results show that the architectural improvements enable scaling to complex image datasets such as ImageNet64, outperforming comparable PCs trained using maximum likelihood or EM.\n\nSTRENGTHS:\nThe paper proposes a number of innovations on top of the recently proposed PICs, such as allowing for arbitrary region graphs and functional sharing. Taken together, this constitutes a well-engineered solution that significantly improves the scalability and flexibility of PICs. \n\nThe experiments are well-executed with strong results on a range of image datasets in comparison to comparable PCs learned directly. Many experimental details are reported, such as memory usage, time taken and parameter counts/hyperparameters, which should make the results reproducible.\n\nThe paper was a pleasure to read, with clear and consistent notation throughout explaining the method and accompanying pictorial illustrations that make the idea of the paper transparent and easy to understand.\n\nWEAKNESSES:\nThe paper is arguably a little weak in terms of novelty as most of the new components are adaptations of existing ideas in the literature (e.g. tucker/cp layers, parameter sharing) to probabilistic integral circuits.\n\nQUESTIONS:\n- What numerical quadrature rule was used? \n- In the conclusion, it is remarked that differentiable sampling from PICs is not possible. What is meant by differentiable here, and is it possible to sample from the materialized PC if one is not interested in gradients (e.g. for visualization)? \n- To what extent do the authors consider their method a continuous latent variable model, as opposed to a means of effectively constructing PCs? For instance, if the same quadrature points are used throughout training and evaluation, can one expect the MLPs to learn meaningful function values between evaluation points?\n- Did the authors investigate learning PICs according to a HCLT region graph? This would give a better idea of whether a DAG (as opposed to tree-structured) region graph is necessary.\n\nTypos:\n- modes -> nodes 190\n- capitalization of 'functional' 234\n- 'pixels are categorical' -> 'pixels as categorical' 286\n\nLIMITATIONS:\nThe limitations of the method are appropriately addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work extends the probabilistic integral circuits (PICs) from tree-shaped region graphs (RGs) based ones to DAG-shaped ones. While constructing PICs from DAG RGs can lead to more expressive models, it comes with the concern of scalability since the circuit sizes might be increased a lot. To address this concern, this work proposes tensorized circuits to approximate the resulting PICs by a PIC to QPC reduction. This is further combined with functional sharing and materialization to achieve efficiency and scalability, which is validated by experimental results on several image datasets.\n\nSTRENGTHS:\n- The extension from tree-shaped PICs to DAG-shaped ones is a solid contribution as it greatly improves the expressiveness of the resulting PICs and is an important step towards PICs constructed from more complex RGs for complicated tasks.\n- The functional sharing and materialization techniques proposed in this work are shown to be very useful by the empirical results. From Fig 5, learning PICs consumes similar resources to PCs which are efficient models and it requires much less trainable parameters than the ones without functional sharing, making the training of large-scale PICs feasible.\n\nWEAKNESSES:\nI appreciate that this work aims to solve an important problem and the proposed solution is shown to be novel and efficient. Still, it's heavy in technical details and thus I don't think I fully understand the algorithmic details. Specifically,\n- I'm confused by the definition of the integral unit from Line 58 to Line 62, especially by the inconsistency in the input of g_i between Line 61 and Line 62 (the g_i inside the integral). Also, the integral in Line 62 contains the function g_i while Figure 1(b) does not, which is also confusing. It might be helpful to put a numeric example there.\n- This also applies to the other technical sections of this work given that this work is heavy in techniques. It would be very helpful to provide a toy example to showcase the reduction of PICs to QPCs to improve accessibility and reproducibility for readers who are not well-versed in these areas.\n- I'm curious to see some theoretical analysis on the circuit sizes characterized. I understand that the bounds might be loose and not that informative. Still, it might be helpful to understand how the PIC to QPC reduction affects the circuit sizes.\n\nQUESTIONS:\n- Can you provide some examples to help understand the integral units?\n- From the definition of g_u in Line 71, the integral is a function of both variables X_i and Z_u. Can you explain the claim \"In this way, the output LVs of any unit u will simply be Zu\" that directly follows the definition of g_u?\n- Can you characterize the circuit sizes of the resulting PICs and QPCs?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-20"}