A Compositional Atlas of Tractable Circuit Operations for Probabilistic Inference

1 Introduction . Many core computational tasks in machine learning ( ML ) and AI involve solving complex integrals , such as expectations appearing in training losses or in information-theoretic quantities including entropies or divergences . A fundamental question naturally arises : under which conditions do these quantities admit tractable computation ? Or equivalently , when can we compute them reliably and efficiently without resorting to approximations or heuristics ? If we are able to find model classes to tractably compute these quantities of interest—henceforth called queries—we can then design efficient algorithms with important applications in learning , approximate inference [ 44 ] , model compression [ 28 ] , explainable AI [ 24 , 47 , 52 ] and algorithmic bias detection [ 23 , 7 , 6 ] . This “ quest ” for tracing the tractability of different queries has been carried out several times , often independently for different model classes in ML and AI and crucially , for each query in isolation . For example , the computation of the Kullback-Leibler divergence ( KLD ) is known to have a closed form for Gaussians , but only recently has an exact algorithm been derived for a more complex tractable model class such as probabilistic sentential decision diagrams ( PSDDs ) [ 28 ] . On the other hand , tractable computation of the entropy , despite being a sub-routine for the KLD , has only been derived for a different tractable model class—selective sum-product networks ( SPNs ) [ 36 ] —by Shih and Ermon [ 44 ] . In the current paradigm , if one were to trace the tractability of a query that has not ∗This work was done while AV was at the CS Department at UCLA . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . p q / r log s × t ∫ function kld ( p , q ) r = quotient ( p , q ) s = log ( r ) t = product ( p , s ) return integrate ( t ) end function xent ( p , q ) r = log ( q ) s = product ( p , r ) return -integrate ( s ) end 1 p q log r × s ∫ function kld ( p , q ) r = quotient ( p , q ) s = log ( r ) t = product ( p , s ) return integrate ( t ) end function xent ( p , q ) r = log ( q ) s = product ( p , r ) return -integrate ( s ) end 1 Figure 1 : Computational pipelines of the KLD ( left ) and cross entropy ( right ) over two distributions p and q encoded as circuits , with the intermediate computations ( r , s and t ) also represented as circuits . Their corresponding implementations i a few lin s of Julia code are shown on their right . yet been investigated but still involves the same “ building blocks ” such as logarithms , integrals and products over distributions , for instance Rényi ’ s alpha divergence [ 40 ] , they would need to derive a novel custom algorithm for each model class and prove its tractability from scratch . In this paper , we take a different path and introduce a general framework under which the tractability of complex queries can be traced in a unified and effortless manner over model classes and query classes . To abstract from the different model formalisms , we carry our analysis over circuit representations [ 8 ] as they subsume many tractable generative models—probabilistic circuits such as Chow-Liu trees [ 9 ] , hidden Markov models ( HMMs ) [ 39 ] , sum-product networks ( SPNs ) [ 38 ] , and other deep mixture models—as well as discriminative ones , including decision trees [ 25 , 10 ] and deep regressors [ 23 ] , thus enabling a unified treatment across model classes . To generalize our analysis across queries , we propose to represent a single query as a circuit pipeline : a computational graph whose intermediate operations transform and combine the input circuits into other circuits . We can first build a set of simple tractable circuit transformations—sums , products , powers , logarithms , and exponentials—and then i ) analyze the tractability of a single query by propagating the sufficient conditions for tractability of the intermediate operators in the pipeline ; and ii ) automatically distill a tractable inference algorithm by composing the operators used . For instance , Fig . 1 shows the pipeline for computing the KLD of p and q , two distributions encoded by circuits . We can identify a general class of models that supports its tractable computation : by tracing the conditions for tractable quotient , logarithm , and product over circuits such that the output circuit ( i.e. , t ) admits tractable integration , we can derive a set of sufficient conditions for the input circuits . Moreover , we can reuse the logarithm and product operations in the KLD pipeline to reason about the tractability of cross entropy , in the very same way we can reuse the corresponding subroutines we provide in Julia to quickly implement algorithms for the two queries in a couple lines of code as shown in Fig . 1 . This compositionality greatly speeds up the design of novel tractable algorithms . We make the following contributions : ( 1 ) a systematic way to compositionally answer many complex queries using simple circuit transformations ( Sec . 3 ) , proving sufficient conditions for their tractability and computational hardness when these conditions are unmet ( Tab . 1 ) ; ( 2 ) a unification and generalization of many inference algorithms proposed in the literature so far for specific representations ( Sec . 4 ) ; ( 3 ) novel tractability and hardness results of complex information-theoretic queries including several widely used entropies and divergences ( Tab . 2 ) ; and ( 4 ) a publicly available implementation of these operators in the Juice circuit library [ 12 ] . We now start by introducing the circuit language . 2 Circuit Representations . Circuits represent functions as parameterized computational graphs . By imposing certain structural constraints on these graphs , we can guarantee the tractability of certain operations over the encoded functions . Moreover , these constraints help understand how circuits unify several classical tractable model classes , such as mixture models , bounded-treewidth probabilistic graphical models ( PGMs ) , decision trees , and compact logical function representations [ 8 , 51 ] . As such , circuits provide a language for building and reasoning about tractable representations , and it follows that all our results in the following sections automatically translate to these model classes . We introduce the basic rules of this language by distinguishing between general circuits and those encoding probability distributions , as some operators in Sec . 3 may be restricted to the latter . Then , we will review the structural constraints we need to characterize different inference scenarios , also known as classes of queries . We denote random variables by uppercase letters ( X ) and their assignments by lowercase ones ( x ) . Sets of variables and their assignments are denoted by bold uppercase ( X ) and bold lowercase ( x ) letters , respectively , and the set of all their values as val ( X ) . Definition 2.1 ( Circuit ) . A circuit p over variables X is a parameterized computational graph encoding a function p ( X ) and comprising three kinds of computational units : input , product , and sum . Each inner unit n ( i.e. , product or sum unit ) receives inputs from other units , denoted in ( n ) . If n is an input unit , it encodes a parameterized function pn ( φ ( n ) ) over variables φ ( n ) ⊆ X , also called its scope . Instead , if n is a sum unit , it encodes ∑ c∈in ( n ) θcpc ( φ ( c ) ) where θc ∈ R are the sum parameters ; while if it is a product unit , it encodes ∏ c∈in ( n ) pc ( φ ( c ) ) . The scope of an inner unit is the union of the scopes of its inputs : φ ( n ) = ⋃ c∈in ( n ) φ ( c ) . The output unit of the circuit is the last unit ( i.e. , with out-degree 0 ) in the graph , encoding p ( X ) . The support of p is the set of all complete states for X for which the output of p is non-zero : supp ( p ) = { x ∈ val ( X ) | p ( x ) 6= 0 } . Circuits can be understood as compact representations of polynomials with exponentially many terms , whose indeterminates are the functions encoded by the input units . These functions are assumed to be simple enough to allow tractable computations of the operations discussed in this paper . Fig . 2 shows some examples of circuits . A probabilistic circuit ( PC ) [ 8 ] represents a ( possibly unnormalized ) probability distribution by encoding its probability mass , density , or a combination thereof . Definition 2.2 ( Probabilistic circuit ) . A PC over variables X is a circuit encoding a function p that is non-negative for all values of X ; i.e. , ∀x ∈ val ( X ) : p ( x ) ≥ 0 . From here on , we will assume that a PC has positive sum parameters and input units that model valid ( unnormalized ) distributions , which is a sufficient condition to satisfy the above definition . Moreover , w.l.o.g . we will assume that each layer of a circuit alternates between sum and product units and that every product unit n receives only two inputs c1 , c2 , i.e. , pn ( X ) = pc1 ( X ) ·pc2 ( X ) . These conditions can easily be enforced on a circuit in exchange for only a polynomial increase in its size [ 49 , 50 ] . Computing ( functions of ) p ( X ) , or in other words performing inference , can be done by evaluating its computational graph . Hence , the computational cost of inference on a circuit is a function of its size , defined as the number of edges in it and denoted as |p| . For instance , querying the value of p for a complete assignment x equals its feedforward evaluation—inputs before outputs—and therefore is linear in |p| . Other common inference scenarios such as function integration—which translate to marginal inference in the context of probability distributions—can be tackled in linear time with circuits that exhibit certain structural properties , as discussed next . Structural Properties of Circuits . Structural constraints on the computational graph of a circuit in terms of its scope or support provide sufficient and/or necessary conditions for certain queries to be tractably computed . We now define the structural properties needed for the query classes that this work will focus on , referring to Choi et al . [ 8 ] for more details . Definition 2.3 ( Smoothness ) . A circuit is smooth if for every sum unit n , its inputs depend on the same variables : ∀ c1 , c2 ∈ in ( n ) , φ ( c1 ) = φ ( c2 ) . Smooth PCs generalize shallow mixture models [ 30 ] to deep and hierarchical models . For instance , a Gaussian mixture model ( GMM ) can be represented as a smooth PC with a single sum unit over as many input units as mixture components , each encoding a ( multivariate ) Gaussian density . Definition 2.4 ( Decomposability ) . A circuit is decomposable if the inputs of every product unit n depend on disjoint sets of variables : in ( n ) = { c1 , c2 } , φ ( c1 ) ∩ φ ( c2 ) = ∅ . Decomposable product units encode local factorizations . That is , a decomposable product unit n over variables X encodes pn ( X ) = p1 ( X1 ) · p2 ( X2 ) where X1 and X2 form a partition of X . Taken together , decomposability and smoothness are a sufficient and necessary condition for performing tractable integration over arbitrary sets of variables in a single feedforward pass , as they enable larger integrals to be efficiently decomposed into smaller ones [ 15 , 8 ] . Next proposition formalizes it . Proposition 2.1 ( Tractable integration , Choi et al . [ 8 ] ) . Let p be a smooth and decomposable circuit over X with input functions that can be tractably integrated . Then for any variables Y ⊆ X and their assignment y , the integral ∫ z∈val ( Z ) p ( y , z ) dZ can be computed exactly in Θ ( |p| ) time , where Z denotes X \Y . As the complex queries we focus on in this work involve integration as the last step , it is therefore needed that any intermediate operation preserves at least decomposability ; smoothness is less of an issue , as it can be enforced in polytime [ 45 ] . A key additional constraint over scope decompositions is compatibility . Intuitively , two decomposable circuits are compatible if they can be rearranged in polynomial time2 such that their respective product units , once matched by scope , decompose in the same way . We formalize this with the following inductive definition . Definition 2.5 ( Compatibility ) . Two circuits p and q over variables X are compatible if ( 1 ) they are smooth and decomposable and ( 2 ) any pair of product units n∈ p and m∈ q with the same scope can be rearranged into binary products that are mutually compatible and decompose in the same way : ( φ ( n ) =φ ( m ) ) =⇒ ( φ ( ni ) =φ ( mi ) , ni and mi are compatible ) for some rearrangement of the inputs of n ( resp . m ) into n1 , n2 ( resp . m1 , m2 ) . We can derive from compatibility the following properties pertaining to a single circuit , which will be useful in our analysis later . Definition 2.6 ( Special types of compatibility ) . A circuit is structured-decomposable if it is compatible with itself . A decomposable circuit p over X is omni-compatible if it is compatible with any smooth and decomposable circuit over X . Not all decomposable circuits are structured-decomposable ( see Figs . 2a and 2b ) , but some can be rearranged to be compatible with any decomposable circuit . For instance , in Fig . 2c , the fully factorized product unit p ( X ) = p1 ( X1 ) · p2 ( X2 ) · p3 ( X3 ) can be rearranged into p1 ( X1 ) · ( p2 ( X2 ) · p3 ( X3 ) ) and p2 ( X2 ) · ( p1 ( X1 ) · p3 ( X3 ) ) to match the yellow and pink products in Fig . 2a . We can easily see that omni-compatible circuits must assume the form of mixtures of fully-factorized models ; i.e. , ∑ i θi ∏ j pi , j ( Xj ) . For example , an additive ensemble of decision trees over variables X can be represented as an omni-compatible circuit ( see Ex . D.1 in the Appendix ) . Also note that if two circuits are compatible and neither is omni-compatible , then both must be structured decomposable . Definition 2.7 ( Determinism ) . A circuit is deterministic if the inputs of every sum unit n have disjoint supports : ∀ c1 , c2 ∈ in ( n ) , c1 6= c2 =⇒ supp ( c1 ) ∩ supp ( c2 ) = ∅ . Analogously to decomposability , determinism induces a recursive partitioning , but this time over the support of a circuit . For a deterministic sum unit n , the partitioning of its support can be made explicit by introducing an indicator function per each of its inputs , i.e. , ∑ c∈in ( n ) θcpc ( x ) =∑ c∈in ( n ) θcpc ( x ) Jx ∈ supp ( pc ) K. Determinism allows for tractable maximization of circuits [ 13 , 8 ] . While we do not consider maximization queries in this work , determinism will still play a crucial role in the next sections . Moreover , bounded-treewidth PGMs , such as Chow-Liu trees [ 9 ] and thin junction trees [ 1 ] , can efficiently be represented as smooth , deterministic , and decomposable PCs via compilation [ 13 , 11 ] . Probabilistic sentential decision diagrams ( PSDDs ) [ 26 ] are deterministic and structured-decomposable PCs that can be efficiently learned from data [ 11 ] .

Summary: This work suggests a general framework for deriving complex information-theoretic queries using probabilistic circuits (PCs). This framework represents a complex query as a pipeline of operations. The manuscript also provides algorithms for applying these operations to PCs as well.