A Compositional Atlas of Tractable Circuit Operations for Probabilistic Inference

1 Introduction . Many core computational tasks in machine learning ( ML ) and AI involve solving complex integrals , such as expectations appearing in training losses or in information-theoretic quantities including entropies or divergences . A fundamental question naturally arises : under which conditions do these quantities admit tractable computation ? Or equivalently , when can we compute them reliably and efficiently without resorting to approximations or heuristics ? If we are able to find model classes to tractably compute these quantities of interest—henceforth called queries—we can then design efficient algorithms with important applications in learning , approximate inference [ 44 ] , model compression [ 28 ] , explainable AI [ 24 , 47 , 52 ] and algorithmic bias detection [ 23 , 7 , 6 ] . This “ quest ” for tracing the tractability of different queries has been carried out several times , often independently for different model classes in ML and AI and crucially , for each query in isolation . For example , the computation of the Kullback-Leibler divergence ( KLD ) is known to have a closed form for Gaussians , but only recently has an exact algorithm been derived for a more complex tractable model class such as probabilistic sentential decision diagrams ( PSDDs ) [ 28 ] . On the other hand , tractable computation of the entropy , despite being a sub-routine for the KLD , has only been derived for a different tractable model class—selective sum-product networks ( SPNs ) [ 36 ] —by Shih and Ermon [ 44 ] . In the current paradigm , if one were to trace the tractability of a query that has not ∗This work was done while AV was at the CS Department at UCLA . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . p q / r log s × t ∫ function kld ( p , q ) r = quotient ( p , q ) s = log ( r ) t = product ( p , s ) return integrate ( t ) end function xent ( p , q ) r = log ( q ) s = product ( p , r ) return -integrate ( s ) end 1 p q log r × s ∫ function kld ( p , q ) r = quotient ( p , q ) s = log ( r ) t = product ( p , s ) return integrate ( t ) end function xent ( p , q ) r = log ( q ) s = product ( p , r ) return -integrate ( s ) end 1 Figure 1 : Computational pipelines of the KLD ( left ) and cross entropy ( right ) over two distributions p and q encoded as circuits , with the intermediate computations ( r , s and t ) also represented as circuits . Their corresponding implementations i a few lin s of Julia code are shown on their right . yet been investigated but still involves the same “ building blocks ” such as logarithms , integrals and products over distributions , for instance Rényi ’ s alpha divergence [ 40 ] , they would need to derive a novel custom algorithm for each model class and prove its tractability from scratch . In this paper , we take a different path and introduce a general framework under which the tractability of complex queries can be traced in a unified and effortless manner over model classes and query classes . To abstract from the different model formalisms , we carry our analysis over circuit representations [ 8 ] as they subsume many tractable generative models—probabilistic circuits such as Chow-Liu trees [ 9 ] , hidden Markov models ( HMMs ) [ 39 ] , sum-product networks ( SPNs ) [ 38 ] , and other deep mixture models—as well as discriminative ones , including decision trees [ 25 , 10 ] and deep regressors [ 23 ] , thus enabling a unified treatment across model classes . To generalize our analysis across queries , we propose to represent a single query as a circuit pipeline : a computational graph whose intermediate operations transform and combine the input circuits into other circuits . We can first build a set of simple tractable circuit transformations—sums , products , powers , logarithms , and exponentials—and then i ) analyze the tractability of a single query by propagating the sufficient conditions for tractability of the intermediate operators in the pipeline ; and ii ) automatically distill a tractable inference algorithm by composing the operators used . For instance , Fig . 1 shows the pipeline for computing the KLD of p and q , two distributions encoded by circuits . We can identify a general class of models that supports its tractable computation : by tracing the conditions for tractable quotient , logarithm , and product over circuits such that the output circuit ( i.e. , t ) admits tractable integration , we can derive a set of sufficient conditions for the input circuits . Moreover , we can reuse the logarithm and product operations in the KLD pipeline to reason about the tractability of cross entropy , in the very same way we can reuse the corresponding subroutines we provide in Julia to quickly implement algorithms for the two queries in a couple lines of code as shown in Fig . 1 . This compositionality greatly speeds up the design of novel tractable algorithms . We make the following contributions : ( 1 ) a systematic way to compositionally answer many complex queries using simple circuit transformations ( Sec . 3 ) , proving sufficient conditions for their tractability and computational hardness when these conditions are unmet ( Tab . 1 ) ; ( 2 ) a unification and generalization of many inference algorithms proposed in the literature so far for specific representations ( Sec . 4 ) ; ( 3 ) novel tractability and hardness results of complex information-theoretic queries including several widely used entropies and divergences ( Tab . 2 ) ; and ( 4 ) a publicly available implementation of these operators in the Juice circuit library [ 12 ] . We now start by introducing the circuit language . 2 Circuit Representations . Circuits represent functions as parameterized computational graphs . By imposing certain structural constraints on these graphs , we can guarantee the tractability of certain operations over the encoded functions . Moreover , these constraints help understand how circuits unify several classical tractable model classes , such as mixture models , bounded-treewidth probabilistic graphical models ( PGMs ) , decision trees , and compact logical function representations [ 8 , 51 ] . As such , circuits provide a language for building and reasoning about tractable representations , and it follows that all our results in the following sections automatically translate to these model classes . We introduce the basic rules of this language by distinguishing between general circuits and those encoding probability distributions , as some operators in Sec . 3 may be restricted to the latter . Then , we will review the structural constraints we need to characterize different inference scenarios , also known as classes of queries . We denote random variables by uppercase letters ( X ) and their assignments by lowercase ones ( x ) . Sets of variables and their assignments are denoted by bold uppercase ( X ) and bold lowercase ( x ) letters , respectively , and the set of all their values as val ( X ) . Definition 2.1 ( Circuit ) . A circuit p over variables X is a parameterized computational graph encoding a function p ( X ) and comprising three kinds of computational units : input , product , and sum . Each inner unit n ( i.e. , product or sum unit ) receives inputs from other units , denoted in ( n ) . If n is an input unit , it encodes a parameterized function pn ( φ ( n ) ) over variables φ ( n ) ⊆ X , also called its scope . Instead , if n is a sum unit , it encodes ∑ c∈in ( n ) θcpc ( φ ( c ) ) where θc ∈ R are the sum parameters ; while if it is a product unit , it encodes ∏ c∈in ( n ) pc ( φ ( c ) ) . The scope of an inner unit is the union of the scopes of its inputs : φ ( n ) = ⋃ c∈in ( n ) φ ( c ) . The output unit of the circuit is the last unit ( i.e. , with out-degree 0 ) in the graph , encoding p ( X ) . The support of p is the set of all complete states for X for which the output of p is non-zero : supp ( p ) = { x ∈ val ( X ) | p ( x ) 6= 0 } . Circuits can be understood as compact representations of polynomials with exponentially many terms , whose indeterminates are the functions encoded by the input units . These functions are assumed to be simple enough to allow tractable computations of the operations discussed in this paper . Fig . 2 shows some examples of circuits . A probabilistic circuit ( PC ) [ 8 ] represents a ( possibly unnormalized ) probability distribution by encoding its probability mass , density , or a combination thereof . Definition 2.2 ( Probabilistic circuit ) . A PC over variables X is a circuit encoding a function p that is non-negative for all values of X ; i.e. , ∀x ∈ val ( X ) : p ( x ) ≥ 0 . From here on , we will assume that a PC has positive sum parameters and input units that model valid ( unnormalized ) distributions , which is a sufficient condition to satisfy the above definition . Moreover , w.l.o.g . we will assume that each layer of a circuit alternates between sum and product units and that every product unit n receives only two inputs c1 , c2 , i.e. , pn ( X ) = pc1 ( X ) ·pc2 ( X ) . These conditions can easily be enforced on a circuit in exchange for only a polynomial increase in its size [ 49 , 50 ] . Computing ( functions of ) p ( X ) , or in other words performing inference , can be done by evaluating its computational graph . Hence , the computational cost of inference on a circuit is a function of its size , defined as the number of edges in it and denoted as |p| . For instance , querying the value of p for a complete assignment x equals its feedforward evaluation—inputs before outputs—and therefore is linear in |p| . Other common inference scenarios such as function integration—which translate to marginal inference in the context of probability distributions—can be tackled in linear time with circuits that exhibit certain structural properties , as discussed next . Structural Properties of Circuits . Structural constraints on the computational graph of a circuit in terms of its scope or support provide sufficient and/or necessary conditions for certain queries to be tractably computed . We now define the structural properties needed for the query classes that this work will focus on , referring to Choi et al . [ 8 ] for more details . Definition 2.3 ( Smoothness ) . A circuit is smooth if for every sum unit n , its inputs depend on the same variables : ∀ c1 , c2 ∈ in ( n ) , φ ( c1 ) = φ ( c2 ) . Smooth PCs generalize shallow mixture models [ 30 ] to deep and hierarchical models . For instance , a Gaussian mixture model ( GMM ) can be represented as a smooth PC with a single sum unit over as many input units as mixture components , each encoding a ( multivariate ) Gaussian density . Definition 2.4 ( Decomposability ) . A circuit is decomposable if the inputs of every product unit n depend on disjoint sets of variables : in ( n ) = { c1 , c2 } , φ ( c1 ) ∩ φ ( c2 ) = ∅ . Decomposable product units encode local factorizations . That is , a decomposable product unit n over variables X encodes pn ( X ) = p1 ( X1 ) · p2 ( X2 ) where X1 and X2 form a partition of X . Taken together , decomposability and smoothness are a sufficient and necessary condition for performing tractable integration over arbitrary sets of variables in a single feedforward pass , as they enable larger integrals to be efficiently decomposed into smaller ones [ 15 , 8 ] . Next proposition formalizes it . Proposition 2.1 ( Tractable integration , Choi et al . [ 8 ] ) . Let p be a smooth and decomposable circuit over X with input functions that can be tractably integrated . Then for any variables Y ⊆ X and their assignment y , the integral ∫ z∈val ( Z ) p ( y , z ) dZ can be computed exactly in Θ ( |p| ) time , where Z denotes X \Y . As the complex queries we focus on in this work involve integration as the last step , it is therefore needed that any intermediate operation preserves at least decomposability ; smoothness is less of an issue , as it can be enforced in polytime [ 45 ] . A key additional constraint over scope decompositions is compatibility . Intuitively , two decomposable circuits are compatible if they can be rearranged in polynomial time2 such that their respective product units , once matched by scope , decompose in the same way . We formalize this with the following inductive definition . Definition 2.5 ( Compatibility ) . Two circuits p and q over variables X are compatible if ( 1 ) they are smooth and decomposable and ( 2 ) any pair of product units n∈ p and m∈ q with the same scope can be rearranged into binary products that are mutually compatible and decompose in the same way : ( φ ( n ) =φ ( m ) ) =⇒ ( φ ( ni ) =φ ( mi ) , ni and mi are compatible ) for some rearrangement of the inputs of n ( resp . m ) into n1 , n2 ( resp . m1 , m2 ) . We can derive from compatibility the following properties pertaining to a single circuit , which will be useful in our analysis later . Definition 2.6 ( Special types of compatibility ) . A circuit is structured-decomposable if it is compatible with itself . A decomposable circuit p over X is omni-compatible if it is compatible with any smooth and decomposable circuit over X . Not all decomposable circuits are structured-decomposable ( see Figs . 2a and 2b ) , but some can be rearranged to be compatible with any decomposable circuit . For instance , in Fig . 2c , the fully factorized product unit p ( X ) = p1 ( X1 ) · p2 ( X2 ) · p3 ( X3 ) can be rearranged into p1 ( X1 ) · ( p2 ( X2 ) · p3 ( X3 ) ) and p2 ( X2 ) · ( p1 ( X1 ) · p3 ( X3 ) ) to match the yellow and pink products in Fig . 2a . We can easily see that omni-compatible circuits must assume the form of mixtures of fully-factorized models ; i.e. , ∑ i θi ∏ j pi , j ( Xj ) . For example , an additive ensemble of decision trees over variables X can be represented as an omni-compatible circuit ( see Ex . D.1 in the Appendix ) . Also note that if two circuits are compatible and neither is omni-compatible , then both must be structured decomposable . Definition 2.7 ( Determinism ) . A circuit is deterministic if the inputs of every sum unit n have disjoint supports : ∀ c1 , c2 ∈ in ( n ) , c1 6= c2 =⇒ supp ( c1 ) ∩ supp ( c2 ) = ∅ . Analogously to decomposability , determinism induces a recursive partitioning , but this time over the support of a circuit . For a deterministic sum unit n , the partitioning of its support can be made explicit by introducing an indicator function per each of its inputs , i.e. , ∑ c∈in ( n ) θcpc ( x ) =∑ c∈in ( n ) θcpc ( x ) Jx ∈ supp ( pc ) K. Determinism allows for tractable maximization of circuits [ 13 , 8 ] . While we do not consider maximization queries in this work , determinism will still play a crucial role in the next sections . Moreover , bounded-treewidth PGMs , such as Chow-Liu trees [ 9 ] and thin junction trees [ 1 ] , can efficiently be represented as smooth , deterministic , and decomposable PCs via compilation [ 13 , 11 ] . Probabilistic sentential decision diagrams ( PSDDs ) [ 26 ] are deterministic and structured-decomposable PCs that can be efficiently learned from data [ 11 ] .

Summary: This paper investigate the tractability of arithmetic operations on circuits for probabilistic inference.  There is considerable similar work when it comes to operators on logical circuits, and this paper generalises this to operators such as powers, logarithm and exponentials. The paper also considers these operations in a computational graph, which means that the results of this paper can be used to prove tractability conditions for many queries, in particular divergences and entropy calculations. The paper discusses various examples of such tractability results.