Tractable Regularization of Probabilistic Circuits

Probabilistic Circuits ( PCs ) are a promising avenue for probabilistic modeling . They combine advantages of probabilistic graphical models ( PGMs ) with those of neural networks ( NNs ) . Crucially , however , they are tractable probabilistic models , supporting efficient and exact computation of many probabilistic inference queries , such as marginals and MAP . Further , since PCs are structured computation graphs , they can take advantage of deep-learning-style parameter updates , which greatly improves their scalability . However , this innovation also makes PCs prone to overfitting , which has been observed in many standard benchmarks . Despite the existence of abundant regularization techniques for both PGMs and NNs , they are not effective enough when applied to PCs . Instead , we re-think regularization for PCs and propose two intuitive techniques , data softening and entropy regularization , that both take advantage of PCs ’ tractability and still have an efficient implementation as a computation graph . Specifically , data softening provides a principled way to add uncertainty in datasets in closed form , which implicitly regularizes PC parameters . To learn parameters from a softened dataset , PCs only need linear time by virtue of their tractability . In entropy regularization , the exact entropy of the distribution encoded by a PC can be regularized directly , which is again infeasible for most other density estimation models . We show that both methods consistently improve the generalization performance of a wide variety of PCs . Moreover , when paired with a simple PC structure , we achieved state-of-the-art results on 10 out of 20 standard discrete density estimation benchmarks . Open-source code and experiments are available at https : //github.com/UCLA-StarAI/Tractable-PC-Regularization . 1 Introduction . Probabilistic Circuits ( PCs ) [ 1 , 2 ] are considered to be the lingua franca for Tractable Probabilistic Models ( TPMs ) as they offer a unified framework to abstract from a wide variety of TPM circuit representations , such as arithmetic circuits ( ACs ) [ 3 ] , sum-product networks ( SPNs ) [ 4 ] , and probabilistic sentential decision diagrams ( PSDDs ) [ 5 ] . PCs are a successful combination of classic probabilistic graphical models ( PGMs ) and neural networks ( NNs ) . Moreover , by enforcing various structural properties , PCs permit efficient and exact computation of a large family of probabilistic inference queries [ 6 , 7 , 8 ] . The ability to answer these queries leads to successful applications in areas such as model compression [ 9 ] and model bias detection [ 10 , 11 ] . At the same time , PCs are analogous to NNs since their evaluation is also carried out using computation graphs . By exploiting the parallel computation power of GPUs , dedicated implementations [ 2 , 12 ] can train a complex PC with millions of parameters in minutes . These innovations have made PCs much more expressive and scalable to richer datasets that are beyond the reach of “ older ” TPMs [ 13 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , such advances make PCs more prone to overfitting . Although parameter regularization has been extensively studied in both the PGM and NN communities [ 14 , 15 ] , we find that existing regularization techniques for PGMs and NNs are either not suitable or not effective enough when applied to PCs . For example , parameter priors or Laplace smoothing typically used in PGMs , and often used in PC learning as well [ 16 , 17 , 18 ] , incur unwanted bias when learning PC parameters – we will illustrate this point in Sec . 3 . Classic NN methods such as L1 and L2 regularization are not always suitable since PCs often use either closed-form or EM-based parameter updates . This paper designs parameter regularization methods that are directly tailored for PCs . We propose two regularization techniques , data softening and entropy regularization . Both formulate the regularization objective in terms of distributions , regardless of their representation and parameterization . Yet , both leverage the tractability and structural properties of PCs . Specifically , data softening injects noise into the dataset by turning hard evidence in the samples into soft evidence [ 19 , 20 ] . While learning with such softened datasets is infeasible even for simple machine learning models , with their tractability , a class of PCs ( i.e. , deterministic PCs ) can learn the maximum-likelihood estimation ( MLE ) parameters given a softened dataset in O ( |p| · |D| ) time , where |p| is the size of the PC and |D| is the size of the ( original ) dataset . For PCs that are not deterministic , every parameter update step can be done in O ( |p| · |D| ) time , still allowing efficient parameter learning . Additionally , the entropy of the distribution encoded by a PC can be tractably regularized . Although the entropy regularization objective for PC is multi-modal and a global optimum can not be found in general , we propose an algorithm that is guaranteed to converge monotonically towards a stationary point . We show that both proposed approaches consistently improve the test set performance over standard density estimation benchmarks . Furthermore , we observe that when data softening and entropy regularization are properly combined , even better generalization performance can be achieved . Specifically , when paired with a simple PC structure , this combined regularization method achieves state-of-the-art results on 10 out of 20 standard discrete density estimation benchmarks . Notation We denote random variables by uppercase letters ( e.g. , X ) and their assignments by lowercase letters ( e.g. , x ) . Analogously , we use bold uppercase letters ( e.g. , X ) and bold lowercase letters ( e.g. , x ) for sets of variables and their joint assignments , respectively . 2 Two Intuitive Ideas for Regularizing Distributions . A common way to prevent overfitting in machine learning models is to regularize the syntactic representation of the distribution . For example , L1 and L2 losses add mutually independent priors to all parameters of a model ; other approaches such as Dropout [ 14 ] , Bayesian Neural Networks ( BNNs ) [ 21 ] , and Bayesian parameter smoothing [ 22 ] incorporate more complex and structured priors into the model [ 23 ] . In this section , we ask the question : how would we regularize an arbitrary distribution , regardless of the model at hand , and the way it is parameterized ? Such global , model-agnostic regularizers appear to be under-explored . Next , we introduce two intuitive ideas for regularizing distributions , and study how they can be practically realized in the context of probabilistic circuits in the remainder of this paper . Data softening Data augmentation is a common technique to improve the generalization performance of machine learning models [ 24 , 25 ] . A simple yet effective type of data augmentation is to inject noise into the samples , for example by randomly corrupting bits or pixels [ 26 ] . This can greatly improve generalization as it renders the model more robust to such noise . While current noise injection methods are implemented as a sequence of sampled transformations , we stress that some noise injection can be done in closed form : we will be considering all possible corruptions , each with their own probability , as a function of how similar they are to a training data point . Consider boolean variables1 as an example : after noise injection , a sample X=1 is represented as a distribution over all possible assignments ( i.e. , X=1 and X=0 ) , where the instance X=1 , which is “ similar ” to the original sample , gets a higher probability : P ( X= 1 ) =β . Here β∈ ( 0.5 , 1 ] is a hyperparameter that specifies the regularization strength — if β=1 , no regularization is added ; if β approaches 0.5 , the regularized sample represents an ( almost ) uniform distribution . For a sample x with K variables X : = { Xi } Ki=1 , where the kth variable takes value xk , we can similarly ‘ soften ’ x 1We postpone the discussion on regularizing samples with non-boolean variables in Appendix B.1 . by independently injecting noise into each variable , resulting in a softened distribution Px , β : ∀x′∈val ( X ) , Px , β ( X=x′ ) : = K∏ i=1 Px , β ( Xi=x ′ i ) = K∏ i=1 ( β ·1 [ x′i=xi ] + ( 1−β ) ·1 [ x′i 6=xi ] ) . For a full dataset D : = { x ( i ) } Ni=1 , this softening of the data can also be represented through a new , softened dataset Dβ . Its empirical distribution is the average softened distribution of its data . It is a weighted dataset , where weight ( Dβ , x ) denotes the weight of sample x in Dβ : Dβ : = { x |x∈val ( X ) } and weight ( Dβ , x ) = 1 N N∑ i=1 Px ( i ) , β ( X = x ) . ( 1 ) This softened dataset ensures that each possible assignment has a small but non-zero weight in the training data . Consequently , any distribution learned on the softened data must assign a small probability everywhere as well . Of course , materializing this dataset , which contains all possible training example , is not practical . Regardless , we will think of data softening as implicitly operating on this softened dataset . We remark that data softening is related to soft evidence [ 27 ] and virtual evidence [ 28 ] , which both define a framework to incorporate uncertain evidence into a distribution . Entropy regularization Shannon entropy is an effective indicator for overfitting . For a dataset D with N distinct samples , a perfectly overfitting model that learns the exact empirical distribution has entropy log ( N ) . A distribution that generalizes well should have a much larger entropy , since it assigns positive probability to exponentially more assignments near the training samples . Concretely , for the protein sequence density estimation task [ 29 ] that we will experiment with in Sec . 4.3 , the perfectly overfitting empirical distribution has entropy 3 , a severely overfitting learned model has entropy 92 , yet a model that generalizes well has entropy 177 . Therefore , directly controlling the entropy of the learned distribution will help mitigate overfitting . Given a model Pθ parametrized by θ and a dataset D : = { x ( i ) } Ni=1 , we define the following entropy regularization objective : LLent ( θ ; D , τ ) : = 1 N N∑ i=1 logPθ ( x ( i ) ) + τ · ENT ( Pθ ) , ( 2 ) where ENT ( Pθ ) : =− ∑ x∈val ( X ) Pθ ( x ) logPθ ( x ) denotes the entropy of distribution Pθ , and τ is a hyperparameter that controls the regularization strength . Various forms of entropy regularization have been used in the training process of deep learning models . Different from Eq . ( 2 ) , these methods regularize the entropy of a parametric [ 30 , 31 ] or non-parametric [ 32 ] output space of the model . Although both ideas for regularizing distributions are rather intuitive , it is surprisingly hard to implement them in practice since they are intractable even for the simplest machine learning models . Theorem 1 . Computing the likelihood of a distribution represented as a exponentiated logistic regression ( or equivalently , a single neuron ) given softened data is # P-hard . Theorem 2 . Computing the Shannon entropy of a normalized logistic regression model is # P-hard . Proof of Thm . 1 and 2 are provided in Appendices A.3 and A.4 . Although data softening and entropy regularization are infeasible for many models , we will show in the following sections that they are tractable to use when applied to Probabilistic Circuits ( PCs ) [ 1 ] , a class of expressive TPMs .

Summary: In the context of Probabilistic Circuits (PCs), recent developments of large-scale models point to the increasing need of regularization.  Having large models with millions of parameters is inherently a challenge, but more so for models with structural restrictions that favor tractability. In this paper, the authors present regularization techniques for probabilistic circuits including non-deterministic models. The regularization is based on data softening and on regularization via entropy, and the authors show that this is possible for PCs while intractable for other models. And demonstrate empirically that the approach is beneficial.