HyperSPNs: Compact and Expressive Probabilistic Circuits
Authors: Andy Shih
Dorsa Sadigh
Stefano Ermon
Venue: NeurIPS 2021 Poster (2021)
Keywords: probabilistic circuits
sum product networks
discrete density estimation
generalization

Abstract: Probabilistic circuits (PCs) are a family of generative models which allows for the computation of exact likelihoods and marginals of its probability distributions. PCs are both expressive and tractable, and serve as popular choices for discrete density estimation tasks. However, large PCs are susceptible to overfitting, and only a few regularization strategies (e.g., dropout, weight-decay) have been explored. We propose HyperSPNs: a new paradigm of generating the mixture weights of large PCs using a small-scale neural network. Our framework can be viewed as a soft weight-sharing strategy, which combines the greater expressiveness of large models with the better generalization and memory-footprint properties of small models.  We show the merits of our regularization strategy on two state-of-the-art PC families introduced in recent literature -- RAT-SPNs and EiNETs -- and demonstrate generalization improvements in both models on a suite of density estimation benchmarks in both discrete and continuous domains.

HyperSPNs:
Compact and Expressive Probabilistic Circuits
Andy Shih
Dorsa Sadigh Stanford University dorsa@cs. stanford. edu
Stefano Ermon Stanford University
Stanford University andyshih@cs stanford.
ermon0cs, stanford edu
Abstract
Prohabilistic circuits (PCs) family of generative models which allows for the computation of exact likelihoods and marginals of its probability distributions. PCs are both expressive and tractable, and serve popular choices for discrete density estimation tasks. However, large PCs are susceptible to overfitting, and only a few regularization strategies (e.g., dropout, weight-decay) have been explored. We propose HyperSPNs: a new paradigm of generating the mixture weights of large PCs using a small-scale neural network. Our framework can be viewed as a soft bright-sharing strategy, which combines the greater expressiveness of large models with the better generalization and memory-footprin properties small models. We show the merits of our regularization strategy on two state-of-the- art PC families introduced recent literature RAT-SPNs and EiNETs and demonstrate generalization improvements in both models on suite of density estimation benchmarks in both discrete and continuous domains.
1 Introduction
One of the core motivations for building models probability distributions from data is to reason about the data distribution. For example, given model of the distribution we may like to reason about the probability that driver will take a certain route given that there congestion
driving routes,
certain street. Or, given a distribution over weather conditions, may like to reason about the probability of it raining times in the next year. Traditional probabilistic models such Categorical/Gaussian Mixture Models Hidden Markov Models (HMMs) well- equipped to answer these types of queries with exact probabilities that are consistent with the model distribution [1]. However, their learned model distribution might not accurately approximate the data distribution due to their simplicity.
On the other end of the spectrum, the modern wave of deep generative models has largely focused on learning accurate approximations of the true data distribution, at the cost of tractability. High capacity models such autoregressive flow models can mimic the true data distribution with great fidelity [33, 13, 22]. However, they are not designed for reasoning about the probability distribution beyond computation of likelihoods. More illustrative of this trend GANs and EBMs, which are expressive enough to sample high nigh-quality images, but give up even the ability to compute exact likelihoods [9, 14].
To maintain the ability to reason about the distribution induced by our model, we need explore within tractable probabilistic models. Of these model families, probabilistic circuits (e.g. Sum Product Networks) are one of the most expressive and general, subsuming shallow mixture models, and HMMs while maintaining their tractable properties [6, 26, 34]. Their expressive power comes from their hierarchical deep architecture, allowing them to express large number of modes in their distribution. Their tractability comes from global constraints imposed in their network structure, enabling efficient and exact computation of likelihoods, marginals, and more. Probabilistic circuits
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(PC) are popular choice for density estimation [35, 18] and approximate inference in discrete settings [30].
Given the status of probabilistic circuits as one of the most expressive models among tractable probabilistic model families, recent works have looked into pushing the limits of expressivity of probabilistic circuits [15, 25, 24). Naturally, the bigger and deeper the probabilistic circuit, the greater the expressiveness of the model. This has led to a trend of building large probabilistic circuits via design choices such as tensorizing model parameters, forgoing structure learning, and more. For example, [24] report the training of probabilistic circuits with 9.4M parameters.
However, larger probabilistic circuits are more susceptible to overfitting. In addition, the so-called double-descent phenomenon [19] (wherein highly overparameterizee models exhibit low generaliza- tion gap) has yet to be observed in probabilisticc circuits. This puts even more importance on effective regularization strategies for training large probabilistic circuits. Recent works have used dropout [25] and weight decay [37], but there has not been much exploration of other regularization strategies.
In this work, we propose HyperSPNs, which regularize by aggressively limiting the degrees of freedom of the model. Drawing inspiration from HyperNetworks [11]. we generate the weights of the probabilistic circuit via a small-scale external neural network. More precisely, we partition the parameters (mixture weights) of the PC into several sectors. For each sector, we learn a low- dimensional embedding, then map that to the parameters of the sector using the neural network. The generated parameters still structurally form a PC, so we retain the same ability to reason about the probability distribution induced by the HyperSPN.
HyperSPNs combine the greater expressiveness of large probabilistic cireuits with the better gener- alization of models with fewer degrees of freedom. The external neural network has much fewer parameters than the original PC, effectively regularizing the PC through a soft weight-sharing strat- egy. As a bonus, the memory requirement for storing the model is much smaller, and the memory requirement for evaluating the model can also be drastically reduced by materializing sectors of the PC "on the fly". We verify the empirical performance of HyperSPNs on density estimation tasks for the Twenty Datasets benchmark [12], the Amazon Baby Registries benchmark [8], and the Street View House Numbers (SVHN) [20] dataset, showing generalization improvements over competing regularization strategies in all three settings. Our results suggest that HyperSPNs are promising framework for training probabilistic circuits.
2 Background
Probabilistic circuits (PCs) are a family of generative models known for their tractability proper- ties [6, 26, 34). They support efficient and exact inference routines for computing likelihoods and marginals/conditionaas of their probability distribution. The internals of a PC consist of a network weightwe Directed Acyelic Graph) of sum and product nodes, stacked in deep/hierarchical manner and evaluated from leaves to root. PCs gain their tractability by enforcing various structural constraints on their network, such decomposability. which restricts children of the same product node to have disjoint scopes [5, 6]. Different combination of constraints leading to different types of PCs, including Arithmetic Circuits [6], and most notably Sum Product Networks (SPNs) [26].
The probability of an input æ on the PC is defined to be the value groo( obtained at the root of the node
PC after evaluating the network topologically bottom-up. Internally, edges leading into are weighted with mixture probabilities a (that sum to 1), and edges leading into a product node are unweighted. Leaf nodes the base of the network contain leaf distributions! I. The nodes of the PC are evaluated as follows, with ci( denoting the children of node i:
4(x)
i is Ieaf node i is sum node
(x))
Σ Ejech()) 04g9 (x))
[ijeohi)) 9j (x)
i is product node
As long as the leaf distributions are tractable and the network structure satisfies the constraints of decomposability and smoothness, the output at front is guaranteed to be normalized, and corresponding marginals/condittonaal can be evaluated with linear-time exact inference algorithms (5, 6).
The network structure of a PC refers to the set of nodes and edge connections in the DAG, excluding the edge weights. There are various ways of constructing the network structure of PCs, e.g., through

---

1 X
x
k X
" 4
k: X
x
x
(n 20x2 (2n 1)s
(b) Counting the # of pa- rameters, given the ber of variables and layer size k.
variables using layer size k = 3. Solid edges are weighted with trainable parameters, dashed edges are unweighted. Each fully connected layer solid edges can be treated as a sector.
(a) We show the structure of RAT-SPN
Figure Given RAT-SPN, parameters (solid edges) in Figure la can be grouped into fully-connected layers of size x k, x k, and oriented in binary-tree structure. Figure lb, we visualize the sectors blocks, counting the number of blocks in each layer of the binary tree to arrive at the total parameter count. The sector abstraction a key ingredient in implementing the HyperSPN's soft weight-sharing strategy.
partition its of parameters into sectors. The trainable
the use of domain knowledge or through search-based methods [6. 17, 7, 28]. Recently, there has been trend of abstracting away individual sum and product nodes, and instead dealing directly with layers that satisfy the structural constraints necessary for tractability [25, 24, 30). Prescribing network structure through stacking of these layers leads memory and hardware efficiencies that allow us to scale up the training of PCs to millions of edges.
Given PC with fixed network structure, we can learn its edge weights best fit the probability distribution of data. We will refer to the full set of mixture probabilities leading into sum nodes in the PC as its parameters. These parameters can be learned using methods such as Expectation- Maximization (EM) [23], signomial program formulations [38]. or simply via (projected) gradient descent. EM has been well-studied choice for optimizing PCs [23, 24], although requires hand- derived updates that can be tedious and sometimes even incorrect [26]. In this work, we rely on gradient descent, which straightforward and works tit-of-the-bbee on new families of models [37].
Learning a PC with a large number of parameters can be prone to overfitting. One regularization strategy that has been proposed is "probabilistic dropout" [25], which takes inspiration from the standard dropout technique in neural networks [31] and randomly ignores a subset of edges leading into sum nodes. Weight decay has also been commonly used another form of regularization [37]. Unfortunately, there has been little exploration done around regularization strategies, and some current approaches only work in limited settings (e.g., dropout has been limited settings that use PCs as discriminative classifier [25]).
2.1 RAT-SPNs
Two particular PC structures have been recently proposed, namely RAT-SPNs [25] and EiNETs [24], both of which scale to millions of parameters. They are conceptually similar, the main difference being that EiNETs are designed for better GPU performance. To ground our discussion around regularizing large PCs, we will describe our methods just on RAT-SPNs, although we also apply our methods on EiNETs later experiments too. In RAT-SPNs, the input variables are organized into a binary tree, with one Ieaf in the binary tree corresponding to each variable. We start from the base

---

external neural network
underlying SPN
x k
&² outputs
2n empica
embedding
X
inputs
dimension It
xx2
Figure 2: HyperSPN: we introduce small-scale external neural network that generates the parameters of the underlying SPN. Using the sector abstraction from Figure we learn an embedding of dimension h for each of the 2n 1 sectors. The neural network maps each embedding to the parameters of its corresponding sector, which materializes the underlying SPN.
of the SPN and construct its full structure by stacking sum layers and product layers. For each leaf node in the binary tree, we create one fully connected sum layer on top of the SPN leaves. For each internal node in the binary tree, create product layer that merges its two children layers in the SPN, and then stack one fully connected sum layer on top. We illustrate this construction Figure 1. Finally, RAT-SPNs create so-called replicas of this structure, by repeating the construction but with permuted orderings of input variables in the binary tree leaves. The replicas are then combined under top-level mixture distribution (for readability, Figure does not show any replicas).
The size each sum layer in the RAT-SPN is determined by a single constant k. For convenience, also let the size of each product layer be k. This fully specifies the structure of our SPN. For the rest of the paper, it is helpful to abstractly group the parameters of the RAT-SPN into disjoint sectors. Corresponding to the root of the binary tree, we have sector with x l parameters. Corresponding to each internal node in the binary tree, we have a sector with k x parameters. Finally, corresponding to each leaf in the binary tree, we have sector with & x parameters, where is the number of SPN leaf distributions per variable (we use / 2 unless otherwise stated).
As shown in Figure 1, for variables (where is power of 2) and replicas, this corresponds to 2n - 1 sectors totaling O(rnk?) parameters. In particular, for each replica there are (n -2) + (2n 1) parameters. For example, in problem domain with 1000 variable, using layer size k 10 and replica r 10 leads to a RAT-SPN of around 1M parameters.
3 HyperSPNs
We propose HyperSPNs, an approach of regularizing proba- bilistic circuits with large number of parameters by limiting the degrees of freedom of the model using an external neural network. We describe our method concretely on RAT-SPNs, which we will refer to as simply SPNs.
x
1
embeddings
neural network
SPN
To regularize the SPN, introduce smaller-scale external neural network to generate the parameters of the SPN (see Figure 2). We associate each the sectors the SPN with a sector embedding. For each sector, we map the embedding the parameters the sector via the neural network. Then, we use these mapped parameters directly as the parameters of the SPN, re-nommalize such that parameters leading into sum node add up to 1, and evaluate the SPN usual. We visualize the computation graph of this process Figure 3. We materialize the SPN in the forward pass, and propagate the gradients through the SPN in the backward pass. We optimize the neural network and embeddings jointly using gradient descent.
1
leadwarrd poss
frewsed pass
Figure We depict the computation graph when training HyperSPN. The red modules have trainable parame- ters, whereas the gray module does not. Red arrows show the flow of gradients during backpropagation.

---

train
test
Large SPN has much flexi- bility in fitting its clusters.
(b) Small SPN is limited in the flexibility and number clusters.
(c) HyperSPN has many clusters, with shared degree of freedom.
Figure 4: We visualize the soft weight-sharing effect in HyperSPNs, with training data in blue and testing data in orange. HyperSPNS use a large underlying SPN structure, allowing for a large number of clusters (3 in each quadrant) to fit the data well. Since the SPN parameters correspond to directly mixture weights, the weight-sharing strategy in HyperSPNs prevent overfitting by limiting the degrees of freedom across clusters. We depict this fixing the shape of the cluster-triples across quadrants.
HyperSPNs can aggressively reduce the degrees of freedom of the original SPN. Using embeddings of size h, the number of learnable parameters is reduced to O((rn + 12). For the same above example with n 1000, 10, r 10, choosing embeddings of dimension h reduces the number of learnable parameters 20 fold, from around 1M down to around 50k parameters.
The lower degrees of freedom mitigates overfitting of the model. At the SPN has a large number of mixture weights, and thus can express distribution families with a greater number of clusters compared to small SPN with the same degrees of freedom. We visualize this with a motivating example in Figure 4, where we represent a dataset using a grid that is split recursively for each variable (we only draw the splits for the first variables; the rest implied). Intuitively, the number of parameters of the SPN is tied closely to the number of clusters in the distribution. As result, training large SPN can produce many clusters that overfit the training points in blue (Figure 4a), and training small SPN can lead to coarse factorizations and inflexible clusters (Figure 4b). Both cases can lead to poor generalization on the testing points in orange.
time, the materialized
On the other hand, the HyperSPN uses a soft weight-sharing strategy by storing its parameters as low-dimensional embeddings and decoding them through small-scale shared neural network. This allows the model to learn a high number of clusters without suffering from too high of a degree of freedom. In Figure 4c, we visualize the soft weight-sharing as fixing the shape of the cluster-triples within each quadrant, while still giving the model control over the center position of each cluster-triple In this way, HyperSPNs can improve generalization compared to large SPN with the same number of mixture weights and small SPN with the same degrees of freedom. We verify this phenomenon on hand-crafted dataset in the Appendix, and further observe similar improvements on real world datasets in our experiments.
3.1 Memory-Efficient Evaluation
Another merit HyperSPNs the ability for low-memory storage of the model, since we only need to store the embeddings and the neural network, which together have much fewer parameters than the SPN. Importantly, our abstraction of the SPN structure into sectors allows us to forgo storage the nodes and edges in the SPN network. Morcover, not only can we be memory-efficiee during storage but we can also be memory-efficient during evaluation. To do so, we matcrialize sectors of the SPN on the fly, and erase them from memory when they become unnecessary.
The sectors of the SPN are structured binary tree, so to evaluate each sector we need all the input values leading into the sector. These input values correspond to the output values (each vector of size k) of its children sectors. Evaluating the binary tree bottom-up depth-wise is possible, but is also costly since the leaf layer has sectors leading to (nkk) cost for storing the output values of children sectors. Rather, we can proceed greedily, evaluating leaf sectors left-to-right and propagating values up sectors soon all of its children sectors have been evaluated. A simple recursive argument shows that this procedure requires storing only ( log n) intermediate values at a time, and is provably optimal.
5

---

Proposition 1. Evaluating complete binary-tree computation graph requires (llogn) memory, where n is the number of leaves the complete binary-tree, and the memory cost of storing each intermediate value and performing each computation is assumed to be constant.
Proof. Perform the sequence of computations based on post-order traversal of the binary tree, erasing children values from memory when their parent value has been computed. This process requires storing at most log n+ intermediate values at any time. To show optimality, let C(i) denote the optimal memory cost needed to compute the value for a node i. For non-leaf nodes i, Let ji be the first child of that we begin to process, and ja be the second. Since we processed ji first (which requires at least 1 unit of memory throughout), we have C() 2 1 + C2)2)) As the base case, for leaf nodes i, we have (ii) 1. Thus, the optimal memory cost for the whole computation is least (root) = log + 1.
Corollary 1. Evaluating HyperSPN with an underlying RAT-SPN structure requires O log n + h(k² + rn) memory.
The cost in Corollary 1 can be broken down into two parts, where O log memory is required for storing intermediate values in the binary-tree computation graph, and (h k2)) memory is required for storing the HyperSPN parameters and materializing the SPN one sector at a time.
The takeaway is that both storing and evaluating a HyperSPN require less memory than storing and evaluating the underlying SPN, which require (rrnk²) memory. This memory efficiency arises from our key insight of using sector abstractions to enable 1) encoding the SPN parameters embeddings, 2) forgoing storage of the nodes and edges in the SPN network, and 3) mapping the SPN to a binary-tree computation graph that allows for low- oowmmemory evaluation order.
Experiments
We experiment with HyperSPN on three sets of density estimation tasks: the Twenty Datasets benchmark, the Amazon Baby Registries benchmark, and the Street View House Numbers (SVHN) dataset. All runs were done using a single GPU, We primarily compare with weight decay as the competing regularization method, We also include comparisons with smaller SPNs that have the same degrees of freedom as our HyperSPN, and with results reported from other works in literature. We omit results on dropout because in our experimentation (and as suggested by [25]), dropout on SPNs worked poorly for these generative density estimation tasks. In all three sets of tasks, we find that HyperSPNs give better generalization performance than weight decay, when comparing using the same underlying SPN structure and optimizer. Furthermore, HyperSPNs outperforms results reported in literature on EiNETs trained using the same optimization method (Adam), and is competitive against results reported on stochastic EM (sEM).
4.1 Twenty Datasets
The Twenty Datasets benchmark [12] is standard benchmark used in comparing PC models. Our experiment aims to compare the use of HyperSPN against the baseline of regularization via weight decay, using the same underlying SPN structure for both. We use the RAT-SPN structure described in Section 2. choosing layer size parameter k = 5 and replicas r = 50, randomizing the variable orders for each replica. The number of learnable parameters in this SPN structure is shown in the # Params column under Weight Decay in Table 1. For Weight Decay, we vary the weight decay value between 1c-3, 1c-4, and Ic-5.
The HyperSPN uses the exact same underlying SPN structure, along with an external neural network (a 2 layer MLP of width 20) and embeddings of dimension ranging between h 5,, 10, 20. The number of learnable parameters in the HyperSPN is around 5 times smaller, as shown in the # Params column under HyperSPN. We describe more details of our training setup in the Appendix.
In Table 1, we show bold the better of the results between Weight Decay and HyperSPN. We see that HyperSPN outperforms Weight Decay on all but two of the datasets. On top of that, storing and evaluating the HyperSPN can be much more memory-efficient, as shown by the reduced number of trainable parameters and as described by the techniques in Section 3.

---

Table 1: Twenty Datasets. We plot the test log- log-likelihood and the number of trainable parameters. Bold values indicate the best results between the two regularization strategies of Weight Decay and HyperSPN, using the same SPN structure and optimizer. Underlined values indicate the best results when compared also to reported values in literature using EiNETs/Adam and RAT-SPN/SEM.
Adam
Adam
Adam
sEM
Name
Variables
Weight Decay # Params
HyperSPN
i
Log-I.H
Log-LH
Params
Log-LH
Log-LH
NLTCS
16
-6.02
40050
-6.01
9115
-6.04
-6.01
MSNBC
17
-6.04
42550
-6.05
9615
-6.03
-6.04
KDDCup2k
64
-2.14
160050
-2.13
33115
-2.15
-2.13
Plants
69
-13.1
172550
-13.27
35615
-13.74
13.44
Audio
100
-40.14
250050
-39.74
51115
-40.22
339.96
Jester
100
-52.99
250050
-52.74
51115
-53.10
-2,977
Netflix
100
-57.18
250050
-56.62
51115
-57.10
-6.85
Accidents
111
-35.55
277550
-35.40
56615
-37.45
-35.49
Retail
135
10.90
337550
-10899
68615
-1.997
10.91
Pumsb-star
163
-31.08
407550
-31.07
82615
-39.23
-32.53
DNA
180
99..22
450050
-98.79
91115
-97.68
-97.23
Kosarek
190
10.88
475050
-10.90
96115
-10992
10.89
MSWeb
294
-10.44
735050
-9.90
148115
-10.26
10.12
Book
500
-3.844
1250050
-34.86
251115
-35.15
-34.68
EachMovie
500
-2.85
1250050
-51.22
251115
-55,49
-53.63
WebKB
839
-159.68
2097550
-157.69
420615
-160.51
157.53
Reuters-52
889
-90.55
2222550
-86.12
445615
-92.76
-87.37
20Newsgrp
910
-14.366
2275050
-152.49
456115
-15.44
152.06
BBC
1058
226..77
2645050
-254.44
530115
-267.86
-252.14
Ad
1556
-54.82
3890050
-2.588
779115
-63.82
-48.47
On the two right-most columns, we directly copy over results reported in literature. We compare with training EiNETs Adam (the optimizer similar network structure as our setup) [25]. We see that HyperSPN also compares favorably to these reported results (the best values underlined).
setup) [24], and training RAT-SPNs sEM (a
To illustrate the regularization effects of HyperSPN, we plot training curves on the training and validation data for the Plants and for Pumsb-star datasets. We the same HyperSPN model described above. For clarity, let M denote the size of its underlying SPN and m denote its true number of learnable parameters (those in its external neural network and embeddings). We then construct two SPNs one with size M (SPN-Large) and one with size m (SPN-Small) and overlay their training curves in Figure 5. For both plots in Figure 5a and 5b, SPN-Large suffers from overfitting on the training data (presumably due to the large degrees of freedom), and SPN-Small fits the data poorly (presumably limited by the size of the SPN). Instead, HyperSPN strikes the balance between the expressiveness of large underlying SPN, and regularization properties from the compact neural network with low degrees of freedom. This translates better generalization on the validation data shown in the plots of Figure 5, and on the testing data shown in Table 1.
4.2 Amazon Baby Registries
We conduct similar experiments the Amazon Baby Registries dataset [8]. We repeat the same experimental setup of maximizing the log-likelihood of data, and use the same model structure for HyperSPN and Weight Decay as their respective structures described in the previous section.
In Table 2, we see that HyperSPN clearly outperforms Weight Decay as a regularization technique, giving better test log-likelssoods (in bold) while using around 5 times fewer trainable parameters. On the right-most column we include results as reported literature [37] on training EiNETs using sEM. Although the comparison with this right-most column is hard to interpret due to the differences in model architecture, optimization procedure, and regularization, we still note that HyperSPNs compare favorably on the majority of the datasets (best values underlined).

---

-20
-2.5
hyperspa (train)
Ing-like Ing-llkellhoood
-30
hyperspa (valid)
-13.0
spn-large (train)
-31
spn-large (valid)
log
-13.5
-32
spn-small (train)
spn-smail (valid)
2500
5000
7500
10000 12500 15000 17500
20000
Iteration
(a) Plants
(b) Pumsb-star
Figure 5: We plot the training curves on Plants and Pumsb-star from the Twenty Datasets benchmark. The HyperSPN has the same underlying SPN SPN-Large, but limits its degrees of freedom the same as those of SPN-Small. In both plots, HyperSPN fits the data better than SPN-Small, and is better regularized than SPN-Large, leading the best results the validation set (solid orange line).
The experimental results on the Amazon Baby Registries benchmark confirm our findings in the previous subsection. On both tasks, HyperSPNs generalizes much better than Weight Decay when compared on the same underlying SPN structure, while enabling more memory-efficient storage and evaluation of the model.
Table 2: Amazon Baby Registries. We plot the test log-likelihood and the number trainable parameters. Bold values indicate the best results between the two regularization strategies of Weight Decay and HyperSPN, using the same SPN structure and optimizer. Underlined values indicate the best results when compared also reported values in literature using EiNETs/SEM".
Adam
Adam
sEM
Name
Variables
Weight Decay
HyperSPN
*
Log-LH
# Params
Log-LH
# Params
Log-LH
Apparel
100
-9.32
250050
-9.28
50655
-9.24
Bath
100
-8.54
250050
-8.52
50655
-8.49
Bedding
100
-8.59
250050
-8.57
50655
-8.55
Carseats
34
-4.72
85050
-4.65
17655
-4.72
Diaper
100
-9.99
250050
.9.91
50655
-9.86
Feeding
100
-11.35
250050
-1.330
50655
11.27
Furniture
-4.54
80050
-4.32
16655
-4.38
Gear
100
-9.21
250050
99.21
50655
-9.18
Gifts
-3.43
40050
-3.40
8655
-3.42
Health
62
-7.49
155050
-7.41
31655
-7,47
Media
-7.86
145050
-7.83
29655
-7.82
Moms
16
-3.48
40050
-3.47
8655
-3,48
Safety
-4.48
90050
.4.34
18655
-4.39
Strollers
40
-5.25
100050
-5.00
20655
-5.07
Toys
62
-7.83
155050
-7.79
31655
-7844
Sample Quality In addition to generalization performance on log-likelihood, we can examine the sample quality HyperSPNs in comparison to standard SPNs. To estimate sample quality, take the approach of applying kernel density estimation / Parzen windows [3] using the test dataset. Out of the 35 datasets from the above Twenty Datasets and Amazon Baby Registries, we observe that HyperSPNS report better sample quality for 29 of them, compared to SPNs trained with weight decay. We report the full results of this experiment the Appendix.
4.3 Street View House Numbers (SVHN)
Lastly, we experiment on the SVHN dataset [20]. We use the infrastructure provided by [24], which groups the train/valid/tes. data into 100 clusters, then learns a separate PC for each cluster. The final density estimator is combination of all the individual PCs under one mixture distribution. Here,

---

75
log LH improvement
25
0
-25
0
20
40
60
80
100
cluster index (sorted by y)
Figure 6: We plot the improvement in test log-likelihood on the Street View House Numbers (SVHN) dataset when using HyperSPN instead of weight decay. Following the setup from [24), the dataset split into 100 clusters and a separate HyperSPN is learned for each cluster. We sort the cluster index by improvement for readability. The dashed blue line corresponds to the average (weighted by cluster size) log-likeliibe ligeliil improvement of 9.
study the performance each individual PC on each cluster separately, comparing between weight decay and HyperSPN when trained using Adam.
The underlying PC structure is an EiNET with leaves that hold continuous distributions [24]. To introduce soft weight-sharing of the HyperSPN on the EiNET, we partition the parameters into sectors of size 40, which is the number of sum nodes used in each layer in the setup of [24]. We use an embedding size of h 8, which reduces the number of learnable edge-paree parameters in the EiNET by about fold. In Figure 6, we plot the difference between HyperSPN and weight decay in log-likelihood the test set, sorted in increasing order for readability. The higher the value, the more improvement we observe when using HyperSPN as the regularization method. In around 70% of the clusters, the HyperSPN achieves an improvement in log- likelihoooo of 9 = 4.
improvement in generalization, and on average (weighted by cluster size)
Our experiments suggest that HyperSPNs give superior regularization strategy for PCs compared to the primary alternative of weight decay. HyperSPNs also lead to better performance than other reported approaches in literature that use the same optimizer (Adam), well as other reported methods that use sEM. Our experimental results are general, showing that HyperSPNs are applicable across three standard benchmarks, across PCs with discrete and continuous leaf distributions, and across different state-of-the-an structures such as RAT-SPNs and EiNETs.
Related Work
Many existing works focus structure learning for probabilistic circuits, covering wide range of PC types [7, 28, 27, 15]. These approaches often regularize by limiting the complexity of the structure [7, 35). However, structure learning in general can be tedious and difficult to tune [25].
Recent works have embraced a more deep-learning-like framework by prescribing the structure the PC and learning only the model parameters [25, 24]. This framework leads to the training of large PCs with millions parameters. However, little exploration has been done around regularization this new parameter-centric framework. For generative settings, dropout was suggested unsuitable [25], leaving only weight decay as the primary parameter regularization method. Concurrently, other works [16] look into data softening and entropy regularization techniques. In work, propose the new regularization strategy of HyperSPNs which show promising improvements in generalization.
Finally, the idea of generating parameters with an external neural network comes from HyperNet- works [11], which were introduced as an effective architecture recurrent models. Soft weight- sharing strategies in general [21] have also shown good empirical results [2] and proven useful for neural network compression [32]. There have also been recent work on using similar techniques to parameterize conditional SPNs [29, 36).

---

Conclusion
We propose HyperSPNs, a new paradigm of training large probabilistic circuits using a small external neural network to generate the parameters of the probabilistic circuit. HyperSPNs provide regularization by aggressively reducing the degrees of freedom through a soft weight-sharing strategy. We show empirical improvements in the generalization performance of PCs compared to competing regularization methods, across both discrete and continuous density estimation tasks, and across different state-of-ther aff PC structures. As bonus, HyperSPNs enable memory-efficiem storage and evaluation, combining the expressiveness of large PCs with the compactness of small models.
Limitations and Future Work Our analysis of the memory efficiency HyperSPNs is limited their storage and evaluation, since training HyperSPNs involves manipulating of the gradients through the underlying SPN. Future work can look into combining HyperSPNs with techniques from memory- efficient back propagation [4, 10] to train very large SPNs that do not fit in memory.
7
Acknowledgments
This research supported by NSF (#1651565, #1522054, #1733686), ONR (N00014-19-1-2145) AFOSR 17A9550-19-1-00244 ARO 199-112--111-01222 and Sloan Fellowship.
References
[1] Leonard E Baum and Ted Petrie. Statistical inference for probabilistic functions of finite state markov chains. The annals of mathematical statistics, 37(6))1554- 1563, 1966.
[2] Gabriel Bender, Hanxiao Liu, Bo Chen, Grace Chu, Shuyang Cheng, Pieter-Jan Kindermans, and Quoc V. Le. Can weight sharing outperform random architecture search? an investigation with tunas. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[3] Yoshua Bengio, Grégoire Mesnil, Yann N. Dauphin, and Salah Rifai. Better mixing via deep representations. In Proceedings of the 30th International Conference on Machine Learning, ICML, 2013.
[4] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. CoRR, abs/1604.06 01t///64.06674 74, 2016.
[5] Adnan Darwiche. Decomposable negation normal form. J. ACM, 48(4) 647, 2001.
[6] Adnan Darwiche. A differential approach inference in bayesian networks. J. ACM, 50(3):280-
305, 2003.
[7] Robert Gens and Pedro M. Domingos. Learning the structure of sum-product networks. In Proceedings of the 30th International Conference on Machine Learning, ICML, 2013.
[8] Jennifer Gillenwater, Alex Kulesza, Emily B. Fox, and Benjamin Taskar. Expectation- maximization for learning determinantal point processes. In Advances in Neural Information Processing Systems 27, NIPS, 2014.
[9] Ian J. Goodfellow, Jean Pouget-Abbas Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27 NIPS, 2014.
[10] Audrunas Gruslys, Rémi Munos, Ivo Danihelka. Mare Lanctot, and Alex Graves. Memory- efficient backpropagation through time. In Advances in Neural Information Processing Systems 29, NIPS, 2016.
[11] David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In 5th International Conference on Learning Representations, ICLR, 2017.
10

---

[12] Jan Van Haaren and Jesse Davis. Markov network structure learning: A randomized feature gen- eration approach. Proceedings of the Twenty-Sizen Sixth AAAI Conference on Artificial Intelligence. AAAI, 2012.
[13] Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron C. Courville. Neural autore- gressive flows. In Proceedings of the 35th International Conference on Machine Learning, ICMI., 2018.
[14] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting struciured data, 1(0), 2006.
[15] Yitao Liang and Guy Van den Broeck. Learning logistic circuits. In The Thirty-Thine AAAI Conference on Artificial Intelligence. AAAI, 2019.
[16] Anji Liu and Guy Van den Broeck. Tractable regularization of probabilistic circuits. In UAI Workshop on Tractable Probabilistic Modeling (TPM), 2021.
[17] Daniel Lowd and Pedro M. Domingos. Learning arithmetic circuits. In Proceedings of the 24th Conference in Uncertainty Artificial Intelligence, UAI, 2008
[18] Nicola Di Mauro, Antonio Vergari, Teresa Maria Altomare Basile, and Floriana Esposito. Fast and accurate density estimation with extremely randomized cutset networks. In Machine Learming and Knowledge Discovery in Databases European Conference, ECML PKDD, 2017.
[19] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang. Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In 8th International Conference on Learning Representations, ICLR, 2020.
[20] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[21] Steven J. Nowlan and Geoffrey F. Hinton. Simplifying neural networks by soft weight-sharing. Neural Comput., 4(4):473 1992.
[22] George Papamakarios, Eric T. Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. CoRR, abs/1912.02762, 2019.
[23] Robert Peharz. Foundations of sum-product networks for probabilistic modeling. PhD thesis, PhD thesis, Medical University of Graz, 2015.
[24] Robert Peharz, Steven Lang. Antonio Vergari, Karl Stelzner, Alejandro Molina, Martin Trapp, Guy Van den Broeck, Kristian Kersting, and Zoubin Ghahramani. Einsum networks: Fast and scalable learning of tractable probabilistic circuits. In Proceedings of the 37th International Conference Machine Learning, ICMI., 2020.
[25] Robert Peharz, Antonio Vergari, Karl Stelzner, Alejandro Molina, Martin Trapp, Xiaoting Shao, Kristian Kersting, and Zoubin Ghahramani. Random sum- sum-product networks: A simple and effective approach to probabilistic deep learning. In Proceedings of the Thirty-Fift Conference on Uncertainty in Artificial Intelligence, UAI, 2019.
[26] Hoifung Poon and Pedro M. Domingos. Sum-product networks: new deep architecture. In UAI 2011, Proceedings of the Swenth- Conference on Uncertainty in Artificial Intelligence, 2011.
[27] Tahrima Rahman, Prasanna Kothalkar, and Vibhav Gogate. Cutset networks: A simple, tractable, and scalable approach for improving the accuracy of chow-liu trees. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD) 2014.
[28] Amirmohammad Rooshenas and Daniel Lowd. Learning sum-product networks with direct and indirect variable interactions. In Proceedings of the 31th International Conference on Machine Learning, ICML, 2014.
11

---

[29] Xiaoting Shao, Alejandro Molina, Antonio Vergari, Karl Stelzner, Robert Peharz, Thomas Liebig, and Kristian Kersting, Conditional sum sum-produn cum-product networks: Imposing structure on deep probabilisticc architectures. In International Conference on Probabilistic Graphical Models, PGM, pages 401-412. PMLR, 2020.
[30] Andy Shih and Stefano Ermon. Probabilistic circuits for variational inference in discrete graphical models. In Advances in Neural Information Processing Systems 33, NeurIPS, 2020.
[31] Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: a simple way to prevent neural networks from overfitting. Mach. Learn. Res., 2014.
[32] Karen Ullrich, Edward Meeds, and Max Welling, Soft weight-sharing for neural network compression. In 5th International Conference on Learning Representations, ICLR, 2017.
[33] Benigno Uria, Mara-Alexandre Côté, Karol Gregor, lain Murray, and Hugo Larochelle. Neural autoregressive distribution estimation. Mach. Learn. Res., 1-205:37, 2016.
[34] Antonio Vergari, YooJung Choi, Robert Peharz, and Guy Van den Broeck. Probabilistic circuits: Representations, inference, learning and applications. In Tutorial at the The 34th AAAI Conference on Artificial Intelligence, 2020.
[35] Antonio Vergari, Nicola Di Mauro, and Floriana Esposito. Simplifying, regularizing and strengthening sum-product network structure learning. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD, 2015.
[36] Matej Zečevíć, Devendra Singh Dhami, Athresh Karanam, Sriraam Natarajan, and Kristian Kersting. Interventional sum-product networks: Causal inference with tractable probabilistic models. In Advances in Neural Information Processing Systems 34, NeurlPS, 2021.
[37] Honghua Zhang, Brendan Juba, and Guy Van den Broeck. Probabilistic generating circuits. In Proceedings of the 38th International Conference on Machine Learning, ICML, 2021.
[38] Han Zhao, Pascal Poupart, and Geoffrey J. Gordon. unified approach for learning the parameters of sum-product networks. In Advances in Neural Information Processing Systems 29, NIPS, 2016.
12


Reviews:
[{"review_id": "jUsmyNNSh60", "reviewer": "Reviewer_UiY3", "summary": "The paper presents HyperSPN, a way to reguralize the learning of Sum-Product networks (SPNs).", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "One advantage of the proposed approach is its memory efficiency.\n\nThe main problem of the paper regards the clarity of the proposed approach. \n- It is not clear how the embeddings are defined. The relation between the embeddings and the SPN is not defined. \n- The structure of the neural network used to learn the parameters is not defined\n- The loss used to optimize the parameters is not reported\n- How the embeddings are used by the neural network is not discussed\n\nThe experimental results prove an improvement of the proposed method when compared to weight decay. However, state of the art results on Twenty Datasets are not reported for a comparison.\n\nConcluding, the idea seems to be interesting but the paper does not prove its validity. The approach should be better explained and the experiments should be extended.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation regrds the clarity of the proposed method", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "j5DPworMATJ", "reviewer": "Reviewer_moxD", "summary": "This paper presents a novel method to regularize very large probabilistic circuits, mainly large sum-product networks (SPNs). The idea is to train a small external neural network to generate the parameters (mixture weights) of a large SPN. The neural network itself is defined by a conservative number of parameters. Given the structure of an SPN, the parameter generation process first learns low dimensional embeddings that are representative of groups of parameters (called sectors) in the SPN. These embeddings are then fed into the external neural network which outputs the actual parameters for the SPN. Both the embeddings and the external neural network can be jointly trained using gradient descent. The neural network imposes soft weight-sharing over the SPN parameters which results in limiting the degrees of freedom of the model, therefore reducing potential overfitting. On a variety of benchmark datasets, both discrete and continuous, the authors empirically evaluated the proposed technique to regularize large SPNs and found that their method outperformed the well-known weight decay regularizer in terms of accuracy (log-likelihood scores) with significantly fewer parameters.   \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper makes an interesting novel contribution. The main goal is to reduce overfitting of large SPNs by limiting their degrees of freedom without limiting their expressivity. Learning smaller SPNs can reduce degrees of freedom at the cost of their expressivity. \nHere the authors propose an alternative method where the degrees of freedom of the model is indirectly controlled by a small-scale external neural network which offers to impose a soft weight-sharing strategy over the parameters. This soft weight-sharing does not necessarily reduce the total number of parameters (clusters) of the model but restricts their structure and thus restrictis their degrees of freedom. The proposed technique is not only an effective regularizer but also an efficient means to compactly represent SPNs. One can easily opt to store only the neural network parameters and the embeddings which together have much fewer parameters than the underlying SPN itself. As the authors described this has the added benefit of reducing the memory footprint of SPNs during their evaluation as well. \n \nThe experiments seem quite comprehensive with a few exceptions. The authors tested their method on both discrete and continuous datasets and on two different structures (RAT-SPNs and EiNets). The results are promising. In majority of the cases the method reduced overfitting compared to the weight-decay strategy. The paper is written clearly and with quality. \n\nQuestions/Comments to authors:\n1) An important piece of study that is missing is the effect of the length of 'h'. Two values were used in the experiments (5 and 8).   \n2) The values in tables 1 and 2 present average test log-likelihood?\n3) The plot in figure 5 was quite informative. It will be useful if the scores for other datasets are also presented. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "azq-BLFLX46", "reviewer": "Reviewer_ohxd", "summary": "The paper proposes a novel method for regularizing large scale probabilistic circuits: The PC's parameters are partitioned into evenly sized sectors, and each sector is represented via a low dimensional embedding. A shared neural network is used to map the embeddings in to the PC's original parameter space. Experimental results show that this reduction in the number of trainable parameters yields improved generalization performance when compared to previous methods, mainly weight decay.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality / Significance\nThe proposed method is novel to my knowledge. It addresses a relevant problem, as the fast growth in the number of parameters is one of the main issues preventing RAT-SPNs and EiNets from scaling further.\n\nThe proposed method is intuitively plausible, and the theoretical analysis is correct.\n\n### Evaluation\nThe proposed method is evaluated on a large variety of datasets, and the results are overall convincing.\n\nGiven that weight decay is the main point of comparison, it would be good run a hyperparameter search over its value, to verify that the chosen setting is sensible (and ideally optimal).\n\nThe robustness of the results in Tables 1 and 2 could also be improved by running the experiments multiple times and testing for significance.\n\nFor the SVHN experiment, it would be good to report the change in performance of the overall model, especially given the fact that the clusters are not uniform in size (i.e., given the reported data, it is not clear whether overall performance went up or down).\n\n### Clarity\nThe paper is very well written and easy to follow.\n\n### Summary\nOverall, this paper makes a simple but useful contribution that is well presented, theoretically analyzed, and empirically evaluated.\nI have therefore set my rating to accept, with the expectation that the authors will address the above comments on the empirical evaluation during the response phase.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impact are adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "El9DX-EMjQn", "reviewer": "Reviewer_BMBc", "summary": "This work proposes a deep learning architecture, called HyperSPN, with built-in regularization. The architecture comprises two components: a probabilistic circuit (PC) and a neural network (NN). The main idea here is that the NN generates the parameters of the PC. The NN structure provides the regularization technique: parts of the NN embeddings are shared when generating the PC parameters.\nThe paper's theoretical contribution is the architecture involving the PC and NN for efficient training and inference tasks. Practical contributions include experiments in three different problem domains.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The originality of this work is not clear from the manuscript. The idea of conditioning a PC with weights from a NN has been explored in the PC literature. Some examples are [1] and, more recently, [2]. This paper can benefit from including a proper discussion on how the proposed method differs from those published. It would be interesting to discuss also how it compares to not recent but similar techniques, such as in [3].\nThe idea of regularizing the PC by sharing some of the NN architecture is smart and with promising results.\n\nThe paper's theoretical contributions are sound, to the best of my knowledge. There are only two derivations, namely Proposition 1 and Corollary 1, and both are well presented.\n\nThe manuscript is well written, and it has a good presentation. The architecture's description is supported by informative Figures 1, 2, and 3, and the experiments are also well described. This description is essential for the work's reproducibility.\n\nThe motivation of the manuscript is significant for the research community. Large models are prominent now in the literature and, thus, efficient regularization techniques are welcome. However, the manuscript should clarify this work's main contributions and how the work adds to previous similar papers. This clarification will strengthen this work's significance.\nAnother suggestion for improving significance is to show results on specific tasks. Improvements on log-likelihood, at times, do not reflect on improvements of particular tasks, such as classification or sampling. It would be interesting to see results on such tasks, besides the log-likelihood numbers.\n\nReference\n* [1] \"Conditional Sum-Product Networks: Imposing Structure on Deep Probabilistic Architectures.\" PDF: https://arxiv.org/abs/1905.08550\n* [2] \"RECOWNs: Probabilistic Circuits for Trustworthy Time Series Forecasting.\" PDF: https://arxiv.org/abs/2106.04148\n* [3] \"Hierarchical mixtures of experts and the EM algorithm.\" PDF: https://www.cs.toronto.edu/~hinton/absps/hme.pdf\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper briefly discusses one limitation in the Conclusion section. Future works include combining HyperSPNs with techniques from memory-efficient back-propagation.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for submitting your work to NeurIPS. The paper targets training large scale sum-product networks with a given structure. To this end, it pushes the idea of generating the mixture weights of
large PCs using a small-scale neural network. While there is a connection to Conditional SPNs as presented at PGM 2020, it actually covers the different aspect of learning SPNs and not conditional ones. It is very interesting to see that a shared neural network can be used during training to map the embeddings in to the PC's original parameter space. Scaling PCs (and not only conditional ones) is a really important question. Simple but highly effective and nice idea. 
Decision: Accept (Poster)