Scaling Up Probabilistic Circuits by Latent Variable Distillation
Authors: Anji Liu
Honghua Zhang
Guy Van den Broeck
Venue: ICLR 2023 notable top 5% (2023)

Abstract: Probabilistic Circuits (PCs) are a unified framework for tractable probabilistic models that support efficient computation of various probabilistic queries (e.g., marginal probabilities). One key challenge is to scale PCs to model large and high-dimensional real-world datasets: we observe that as the number of parameters in PCs increases, their performance immediately plateaus. This phenomenon suggests that the existing optimizers fail to exploit the full expressive power of large PCs. We propose to overcome such bottleneck by latent variable distillation: we leverage the less tractable but more expressive deep generative models to provide extra supervision over the latent variables of PCs. Specifically, we extract information from Transformer-based generative models to assign values to latent variables of PCs, providing guidance to PC optimizers. Experiments on both image and language modeling benchmarks (e.g., ImageNet and WikiText-2) show that latent variable distillation substantially boosts the performance of large PCs compared to their counterparts without latent variable distillation. In particular, on the image modeling benchmarks, PCs achieve competitive performance against some of the widely-used deep generative models, including variational autoencoders and flow-based models, opening up new avenues for tractable generative modeling. Our code can be found at https://github.com/UCLA-StarAI/LVD.

Published as a conference paper at ICLR 2023
SCALING UP PROBABILISTIC CIRCUITS BY LATENT VARIABLE DISTILLATION
Anji Liu;" Honghua Zhang' & Guy Van den Brveck
Department of Computer Science University of California, Los Angeles {livanji. i,, hzhang1 9, guyvdb}ēcs. ecs. cucllleesssllleel..e.. edu
ABSTRACT
Probabilistic Circuits (PCs) are unified framework for tractable probabilistic models that support efficient computation of various probabilistic queries (e.g., marginal probabilities). One key challenge is to scale PCs to model large and high- dimensional real-world datasets: we observe that as the number of parameters PCs increases, their performance immediately plateaus. This phenomenon suggests that the existing optimizers fail to exploit the full expressive power of large PCs. We propose to overcome such bottleneck by latent variable distillation: the less tractable but more expressive deep generative models to provide extra supervision over the latent variables of PCs. Specifically, we extract information from Transformer-based generative models to assign values to latent variables of PCs, providing guidance to PC optimizers. Experiments on both image and language modeling benchmarks (e.g.. ImageNet and WikiText-2) show that latent variable distillation substantially boosts the performance of large PCs compared their counterparts without latent variable distillation. In particular, on the image modeling benchmarks, PCs achieve competitive performance against some of the widely-used deep generative models, including variational autoencoders and flow- based models, opening up new avenues for tructable generative modeling. Our code can be found at https: /ttp::///hhha/ com UCLA- Satttttttrrrriaaaa
leverage
INTRODUCTION
The development of tractable proba- bilistic models (TPMs) is impor- tant task in machine learning: they allow various tractable probabilistic inference (e.g., computing marginal probabilities), enabling wide range of down-stream applications such as
ImageNet32
WikiText-2
5.4
280
Bits-por-ding dimdensiionn (bpd)
260
Perplexity
240
4.8
4.6
lossless compression (Liu et al. 2022)
4.4
200
and constrainedd/cnnditonall genera- tion (Peharz et al. 22020a). Probabilis-
10
10"
10a
10"
10a
10
tic circuit (PC) (Choi et al. 2020) is
# parameters
parameters
unified framework for a wide range of families of TPMs, examples in- clude bounded tree-width graphical models (Meila & Jordan 2000), And- Or search spaces (Marinescu & Dechter 2005), hidden Markov models (Rabiner & Juang 1986).
Figure 1: Latent variable (LV) distillation significantly boosts PC performance on challenging image (ImageNet32) and language (WikiText-2) modeling datasets. Lower is better.
Probabilistic Sentential Decision Diagrams (Kisa al.) 2014)) and sum- nun-product networks (Poon & Domingos, 201T). Yet, despite the tractability of PCs, scaling them up for generative modeling on large and digh-dimensional vision/language dataset has been a key challenge.
By leveraging the computation power of modern GPUs, recently developed PC learning frameworks (Peharz et 2020a; Molina al. 2019 Dang et al. 202T) have made possible to train PCs with
Authors contrihuted equally.

---

Published as a conference paper at ICLR 2023
over 100M parameters (e.g., Correia et al. (2022) Yet these computational breakthroughs not leading to the expected large-scale learning breakthroughs: as we scale up PCs, their performance immediately plateaus (dashed curves in Fig. D even though their actual expressive power should increase monotonically with respect to the number of parameters. Such a phenomenon suggests that the existing optimizers fail utilize the expressive power provided by large PCs. PCs be viewed as latent variable models with a deep hierarchy of latent variables. As we scale them up, size of their latent space increases significantly, rendering the landscale of the marginal likelihood over observed variables highly complex. We propose to ease this optimization bottleneck by latent variable distillation (LVD): we provide extra supervision to PC optimizers by leveraging tess-tractable yet more expressive deep generative models to induce semantics-aware assignments to the latent variables of PCs, in addition to the observed variables.
The LVD pipeline consists of two major components: (i) inducing assignments to a subset of (or all) latent variables in PC by information obtained from deep generative models and (ii) estimating PC parameters given the latent variable assignments. For (i), we focus clustering-based approach throughout this paper: we cluster training examples based on their neural embeddings and assign the same values to latent variables for examples in the same cluster; yet, note that there is no constraint on how we should assign values to latent variables and the methodology may be engineered depending on the nature of the dataset and the architecture of PC and deep generative model. For (ii), to leverage the supervision provided by the latent variable assignments obtained in (i), instead of directly optimizing the maximum- meniiumm-ikkelhoodd estimation objective for PC training, we estimate PC parameters by optimizing the its lower- bound shown on the right-hand side:
Σ log logp(z()) = Σ
.
log p(z())
log ((x (0))
(I)
where (ax(1) Ti=1 is the training set and (+) is the induced assignments to the latent variables for (1), After LVD, we continue to finetune PC on the training examples to optimize the actual MLE
objective, i.e., Σ, ())
As shown in Figure [ with LVD, PCs successfully escape the plateau: their performance improves progressively as the number of parameters increases. Throughout the paper, we highlight two key advantages of LVD: first, it makes much better use of the extra capacity provided by large PCs; second, by leveraging the supervision from distilled LV assignments, we can significantly speed up the training pipeline, opening up possibilities further scale up PCs.
We start by presenting simple example where we apply LVD on hidden Markov models to improve their performance on language modeling benchmarks (Sec. 2 Then we introduce the basics for PCs (Sec. (.1) and present the general framework of L.VD for PCs (Sec. (32)). The general framework is then elaborated in further details, focusing on techniques to speed up the training pipeline (Sec. 4 In Section 5 we demonstrate how this general algorithm specializes to train patch-based PCs for image modeling. Empirical results show that LVD outperforms SOTA TPM baselines by a large margin challenging image modeling tasks. Besides, PCs with LVD also achieve competitive results against various widely-used deep generative models, including flow-based models (Kingma &
Dhariwal 2018 Dinh al. 2016) and variational autoencoders (Maaløe et al. 2019) (Sec. 6
LATENT VARIABLE DISTILLATION FOR HIDDEN MARKOV MODEL
In this section, we consider the task of language modeling by hidden Markov models (HMM) as an illustrating example for LVD. In particular, we demonstrate how we can use the BERT model (Devlin) ctt al. 2019) to induce semantics-aware assignments to the latent variables of HMMs. Experiments on the Wikiiexxt2 (Merity al. 2016) dataset show that our approach effectively boosts the performance of HMMs compared to their counterpart trained with only random initialization.
Dataset & Model. The WikiText-2 dataset consists of roughly million tokens extracted from Wikipedia, with vocabulary size of 33278. Following prior works on autoregressive language modeling (Radford et al. 2019), we fix the size of the context window to be 32: that is, the HMM model will only be trained subsequences of length 32 and whenever predicting the next token, the model is only conditioned on the previous 31 tokens. In particular, we adopt a non-homogeneous HMM model, that is, its transition and emission probabilities at each position share no parameters;
2

---

Published as a conference paper at ICLR 2023
contextualized embeddings for suffixes X Ky) X12
X
Xso X31 X32
Z
1
Z
Zzz
given to the
-1.63
-0.05
2
day
sunny and warm BERT
0.43
-1.61
K-means
Alice
playing video games
-0.97
-1.85
1
X12
4
night
cold
and stormy
-0.18
-0.82
5
really
playing
the piano
-1.07
1.30
2
token sequences lengthh
768 dimensional vectors
(a) Graphical model represen- tation of an HMM modeling token sequences of length 32. X, are the observed variables and Zi the latent variables.
(b) Pipeline inferring values for one latent variable Zzn. feed token sequences to the BERT model to obtain contextualized embeddings for their suffxces X Xa2; then we cluster all suffx embeddings into clusters; here h is the number of hidden states and the value for Zuo is set to the cluster id. We repeat this procedure independently infer values for all Zıs.
Figure 2: Latent variable distillation pipeline for hidden Markov models.
Figure 2ap shows its representation as graphical model, where X,s are the observed variables and 7,8 are the latent variables. To facilitate training and evaluation, we pre-process the tokens from WikiText-2 by concatenating them into one giant token sequence and collect all subsequences of length 32 construct the train, validation and test sets, respectively.
Latent Variable Distillation. Let (x J. be the training set; Figure shows an example on how to induce, for each training example x its corresponding assignment to the latent variable Z3o- We first feed all training examples to the BERT model to compute the contextualized embeddings for their suffixes X31 We cluster all suffix embeddings into h clusters by the K-means algorithm [Lloyd] 1982), where h is the number of hidden states; then, we set Z30 to be the cluster id of their corresponding suffixes, that is, suffixes in the same cluster get the same latent variable value: the intuition is that if the BERT embeddings of some suffixes are close to each other then the suffixes should be relatively similar, suggesting that they should be "generated" by the same hidden state. We repeat this procedure for 32 times to infer the values for all Z,s. Now we obtain an "augmented"
training set Daug = (x ((1)),, where 2+) are the corresponding assignments to the latent vari- ables Z: then, as suggested by Equation]] 1 we maximize the lower-bound E, log p(x((), (000) for the true MIE objective Σ; p(x((() The parameters of the HMM that maximize E, log p(xx()) (()), denoted by 0%, can solved closed-form. Finally, using 0* starting point, we finetune the HMM model via EM to maximize the true MLE objective Σ, log p(x)()).
Experiments. We apply LVD to HMMs with varying number of hidden states h 128, 256, 512, 750, 1024 and 1250; for comparison, we also train HMMs with random initialization. Please refer to Appx. C for details about training. The plot on the right of Figure T shows the test perplexity of HMMs (w/ and w/o LVD) on WikiTeext2:: as the number of parameters HMM increases, the performance of the HMMs trained with random parameter initialization immediately plateaus, while the performance of the HMMs trained with LVD progressively improves, suggesting that LVD effectively exploits the express power of the larger models.
LATENT VARIABLE DISTILLATION FOR PROBABILISTIC CIRCUITS
The previous section uses HMM as specific TPM elaborate key steps in LVD. In order to generalize LVD to broader TPMs, this section introduces Probabilistic Circuit (PC), which is unifying framework for large collection of tractable probabilistic models.
3.1
PROBABILISTIC CIRCUITS: A GENERAL TPM FRAMEWORK
PCs (Choi al. 2020) umbrella term for large variety of TPMs. Their syntax and semantics are defined as follows.
Definition 1 (Probabilistic Circuits). A PC p(X) that encodes a distribution over variables X is defined by parameterized directed acyelic computation graph (DAG) with single root node ny-
3

---

Published as a conference paper at ICLR 2023
59
84-
0.35
0.15
02-
P1
P2
Ps
X
X
Z=
(a) A mixture of three Gaussians.
(b) A PC that encodes the distribution.
(c) An equivalent deterministic PC.
Figure 3: A
distribution (a) and two PCs (b-c) that encode the distribution.
Every node in the DAG corresponds to computational unit. Specifically, every leaf node is defined by an input unit and every inner node n represents either a sum or product unit that receives inputs from its children, termed in(n). Each PC unit encodes distribution Pn:
if v an input unit,
Pn (z) = Exci((n) Ocl Po( [ogin(n) (c(
if is a sum unit,
if a product unit,
where (2) is a univariate probability distribution (e.g., Gaussian, Categorical) defined on a variable in X and 6ejn is the parameter corresponds to edge (n, c). For every sum unit n, we assume all its edge parameters {0ein (cin((n) are non-negative and sum up one. Intuitively, a product unit encodes a factorized distribution over its inputs, and sum unit models a weighted mixture of its children's ' distributions. A PC represents the distribution encoded by its root unit n,- We further assume w.l.o.g. that PCs alternate between sum and product layers before reaching an input layer.
A key property that separates PCs from many other generative models is their tractability, i.e., the ability to answer various queries exactly while efficiently. Such queries include common ones like marginals and conditional probabilities as well as task-specific ones such as structured prediction (Shao al. 2022) and variational inference (Shih & Ermon 2020). The tractability of PCs is governed by structural constraints on their DAG structure. For example, smoothness and decomposability together guarantee linear time (w.r.t. size of the PC) computation of arbitrary marginal probabilities.
Definition (Smoothness and Decomposability). Define the (variable) scope Φ(n) of a PC unit n as the set of variables defined by all its descendent input units. A PC is smooth for every sum unit n, all its children have the same scope: Vei, 02 E in(n), d(ci) = d(c2). PC is decomposable if the children of every product unit have disjoint scopes: Ve,, C2 (c)) #c2), Ø(c)) 6(c2)= =0
3.2 MATERIALIZING AND DISTILLING LATENT VARIABLES IN PROBABILISTIC CIRCUITS
PCs can be viewed as latent variable models with discrete latent spaces (Peharz et al. 2016). Specifi- cally, since sum unit in a PC can be viewed a mixture over its input distributions, can also be interpreted as a simple latent variable model Σx p(z|2)p(( 2))(2), where decides which input to choose from and the summation enumerates over all inputs. Figure shows such an example, where the sum unit in Figure (b) represents the mixture over Gaussians in Figure 3 (a).
In general, the latent space for large PCs hierarchical and deeply nested; as we scale them up, we are in effect scaling up the size/complexity of their latent spaces, making it difficult for optimizers to find good local optima. To overcome such bottleneck, we generalize the idea presented in Section 2 and propose latent variable distillation (LVD). The key intuition for LVD is to provide extra supervision on the latent variables of PCs by leveraging existing deep generative models: given a PC p(X); view it as latent variable model E(x, Z = z) over some set of latents Z and assume that for each training example (+))) deep generative model can always induce some semantics-aware assignment Z = 20)); then, instead of directly optimizing the MLE objective Σ, log p(x:()), can optimize its lower-bound Σ, log pl ((), 2()), thus incorporating the guidance provided by the deep generative model. The LVD pipeline consists of three major steps, elaborated in the following:
Step 1:
Materializing Latent Variables. The first step of LVD is to materialize some/all la- tent variables in PCs. By materializing latent variables, we can obtain a new PC representing the joint distribution Pr(X, Z), where the latent variables Z are explicit and its marginal distribu- tion Pr(X) corresponds to the original PC. Although we can assign every sum unit in a PC an unique LV, the semanties of such materialized LVs depend heavily on PC structure and parameters,
4

---

Published as a conference paper at ICLR 2023
Algorithm 1 Materializing a LV in a PC
1: Input: A PC p( X) and a variable scope W for some sum unit in p(X)
2: Output: An augmented PC p defined over (X. Z). where Z is the materialized LV corresponding to W
3: Sw f- {n n E p s.t. n is a product unit and ((n) = W
D Created as an ordered set
for j =)= 1 to [Sw] do
5: n,t be the jth unit Sw
{
J. otherwise
Add an input unit c over Z, with distribution pc(xi)
0
as a new child of ny
which makes it extremely hard to obtain super- vision. Instead, we choose to materialize LVs based subsets of observed variables defined by PC. That is, each materialized LV corre- sponds to all PC units with partieular variable scope (cf. Def. 2 For example, we can mate- rialize the latent variable Z, which corresponds to the scope ((n.) (vi E [3), to construct the PC
Ri
2-1 2-3
2-4
Figure 4: Materializing LVs in a PC.
in Figure [c) that explicitly represents p(X, 2). whose marginal distribution p(X) corresponds to the PC in Figure (b). Algorithm| I (Peharz et al.] 2016) provides one general way to materialize latent variables PCs, where Figure shows an example where the four product units C1, C4 are augmented with input units 2-1,.... Z= 4, respectively.
Continuing with our example in Figure 3 note that after materialization, the sum unit representing p(X, Z) in Figure c) is no longer latent variable distribution: each assignment to X, Z uniquely determines the input distribution to choose, where the other inputs give zero probability under this assignment; say that this sum unit is deterministic (Darwiche (003).
Definition 3 (Determinism). Define supp(n) as the set of complete assignments Eval(X) such that (r( >> 0. A sum unit is deterministic if its children have disjoint supports: Vei, 02 E (in(n) (2), (epp(c)) ) n supp(ca)== 20
Determinism characterizes whether a sum unit introduces latent variables: by materializing some sum units with the scope, we enforce them become deterministic. Intuitively, more deterministic sum units in PCs implies smaller latent spaces, which implies casier optimization; in fact. if all sum units in PC are deterministic then the MLE solution can be computed in closed-form (Kisa et al. 2014). By materializing more latent variables, we make PCs "more deterministic", pushing the optimization procedure towards a closed-form estimation.
Step 2: Inducing Latent Variable Assignments. Latent variable materialization itself cannot provide any extra supervision to the PC training pipeline; addition, we also need to leverage some existing deep generative models to induce semantics-aware assignments for the materialized latent variables. Though there no general guideline on how the assignments should be induced, we focus on clustering- based approach throughout this paper. Recall from Section 2 where cluster the suffix embeddings generated by the BERT model and for each training example, we assign the latents the cluster id that its suffixes belong to. Similarly, for image modeling, in Section we will show how to induce latent variable assignments by clustering the embeddings for patches of images. The main take-away is that the method for inducing latent variable assignments should be engineered depending on the nature of the dataset and the architecture of PC and deep generative model.
Step 3: PC Parameter Learning. Given PC p(X; 0) with parameters 0 and training set D {a ( in Step by materializing some set of latent variables Z, we obtain an augmented PC Pang X, Z) whose marginal distribution on X corresponds to p(X; 0); in Step 2, by leveraging some deep generative model G, we obtain augmented training set D ing
{(xr (t) 2²))) Note that
since Paug and p share the same parameter space, we can optimize Σ log Paug Pang( (z, 201 0) as
lower-bound for Σ/ log (1)
N
log (1: E,"1 log P.Pan
.V 1
Pang ( (1) 2) ;= log Pang (ar 2(000);
N
i=1
we denote the parameters for Paug after optimization by 0-. Finally, we initialize with 0* and optimize the true MILE objective with respect to the original dataset D, Σi"n log p()
5

---

Published as a conference paper at ICLR 2023
Summary. Here we summarize the general pipeline for latent variable distillation. Assume that are given: PC p(x; 0) over observed variables X with parameter 0, a training set 'D {xx(*) and deep generative model G:
1. Construct PC Paug [X Z: 0) by materializing subset of latent variables Z in p(X: 0); note that and Pang share the same parameter space.
2. Use G to induce semantics-awan latent variable assignments 2(1) for each training exam- ple (x()); denote the augmented dataset as Dang = (xx(1) ,0)
3. Optimize the log-likelihood of Paug respect to Daug. i.e. E, log Pang (x) Pang(a the parameters for Pang after optimization as 0"
20;; 0); denote
4. Initialize pl X 0) with 0* and then optimize the log-likelihood of p w/ respect to the original dataset D, i.c., 2; log (()(())))))))))))))))
4 EFFICIENT PARAMETER LEARNING
Another major obstacle for scaling up PCs is training efficiency. Specifically, despite recently developed packages (Dang et al., 2021 Molina et al., 2019) and training pipelines (Peharz al.) 2020a) that leverage the computation power of modern GPUs, training large PCs is still extremely time-consuming. For example, in our experiments. training a PC with 5500M parameters on CIFAR (using existing optimizers) would take around one GPU day to converge. With the efficient parameter learning algorithm detailed in the following, training such PC takes around 10 GPU hours.
The most computationally expensive part in LVD is to optimize the MLE lower bound (Eq. with regard to the observed data and inferred LVs, which requires feeding all training samples through the whole PC. By exploiting the additional conditional independence assumptions introduced by the materialized LVs, we show that the computation cost of this optimization process can be significantly reduced. To gain some intuition, consider applying LVD to the PC in Figure 3(c) with materialized LV Z. For a sample æ whose latent assignment 2 is 1, since the Gaussian dis- tributions P2 and p3 are independent with this sample, only need to feed it to the input unit corresponds to Pı in order to estimate its parameters. To formalize this efficient LVD algorithm, we start by introducing the conditional independence assumptions provided by the materialized LVs.
Lemma 1. For a PC p(X), denote W as the scope of some units in p. Assume the variable scope of every PC unit is either a subset of or disjoint with W. Let Z be the LV corresponds W created by Algorithm ! Then variables W are conditional independent of X X W given Z.
plat.
4(n) XX, ofrul Kimm olnail Mozer
Proof of the above lemma is provided in Appx. A.1 Take Figure 4 as an example. Define the scope of
[n 1 and [c,)1_ as W and the corresponding LV as Z: denote the scope of the full PC as X. Lemma D iimplies that variables W and X W are conditional independent given Z.
M,
2
2g M₁
pias 2
2-
pixa 22
M6.
Figure 5: Distribution decomposition of an
example PC with materialized LVs Z1, Zz.
We consider a simple yet effective strategy for ma- terializing LVs: the set of observed variables X is partitioned into disjoint subsets {X,]/= then for each X,, we use Algorithm| [ to construct corresponding LV, termed Z,. As a di- rect corollary of Lemma I the joint probability over X and Z can be decomposed as follows: p(æ, 2) = =(
The key to speed up LVD is the observation that the MLE lower bound objective (Eq. can be factored into independent components following the decomposition of p(æ, z):
N
LL(p, Daug)=
log p( (1) (0
log 11
Σ
log logp(z ()),
(3)
1=1

---

Published as a conference paper at ICLR 2023
Cluster #1
Cluster #2
Cluster #3
Cluster #4
Eucoder
Decoder
Latent features
(a) Illustration Masked Autoencoder model.
(b) Example of image patches belonging the cluster.
Figure 6: Extracting LVs for image data. The MAE model (a) is used to extract categorical LVs [2: Mi that correspond to image patches {Xr) respectively. (b) provides example patches from the training set that belong to four randomly chosen clusters of the LV Z1.
where Daug. := ((=1), ,000 is the training set augmented with LV assignments. According to Equation (3). optimize LL(V, Daug) is equivalent to performing MLE on the factorized distributions separately. Specifically, we decompose the optimization process into the following independent steps: (i) for each cluster i and category j. optimizing PC parameters w.r.t. the distribution p(X. [Z, = =j) using the subset of training samples whose LV 24 is assigned to category j. and (ii) optimizing the sub-PC corresponds
p(z) using the set of all LV assignments. Consider the example PC shown in Figure 5 The subset PC surrounded by every blue box encodes the distribution labeled on its edge. To maximize LL(p, Pang), we can separately train the sub-PCs correspond to the decomposed distributions, respectively. Compared to feeding training samples to the whole PC, the above procedure trains every cotent-conditioned distribution pl (X, =j) using only samples that have the corresponding LV assignment (i.e., 2;- = =),, which significantly reduces computation cost.
Recall from Section 3.2 that in the LVD pipeline, after training the PC parameters by maximizing [L(p, Daug). we still need to finetune the model on the original dataset D. However, this finetuning step often suffers from slow convergence speed, which significantly slows down the learning process. To mitigate this problem, we add an additional latent distribution training step where we only finetune parameters correspond to p(Z). In this way, we only need to propagate training samples through the sub-PCs correspond to the latent-conditioned distributions once. After this step converges, we move on to finetune the whole model, which then takes much fewer epochs to converge.
5 EXTRACTING LATENT VARIABLES FOR IMAGE MODELING
This section discusses how to induce assignments to LVs using expressive generative models. While the answer is specific to individual data types, we proposes preliminary answers of the question in the context of image data. We highlight that there are many possible LV selection strategies and target generative model; the following method is only an example that shows the effectiveness of LVD.
Motivated by recent advances of image-based deep generative models (Dosovitskiy et al. 2020 Liu et al. 2021). we model images by two levels of hierarchy - the low-level models independently encode distribution of every image patch, and the top-level model represents the correlation between different patches. Formally, we define X, the variables in the ith M x M patch of an Hl x W image (w.loo.g. assume H and W are both divisible by M). Therefore, the image X is divided into A = H W/MM subsets (X, 1 Every Z; is defined the LV corresponds to patch X,.
Recall that our goal is to obtain the assignment of [ each as concise representation of {X respectively. Despite various possible model choices, we choose to use Masked Autoen- coders (MAEs) (He al. 2022) as they produce good features for image patches. Specifically, as shown in Figure 6 a), MAE consists of an encoder and a decoder. During training, a randomly se- lected subset of patches are fed to the encoder to generate a latent representation for every patch. The features are then fed to the decoder to reconstruct the full image. The simplest way to compute latent features for every patch is to feed them into the encoder independently, and extract the corresponding features. However, we find that it is beneficial to also input other patches as context. Specifically, first compute the latent features without context. We then compute the correlation between features of all pair of patches and construct the Maximum Spanning Tree (MST) using the pairwise correlations. Finally, to compute the feature each patch X,, we additionally input patches correspond to its ancestors in the MST. Further details are given in Appx. B.2
As shown in Section 3.2 LVs (2 are required to be categorical. To achieve this, we run the K-means algorithm on the latent features (of all training examples) and use the resultant cluster
7

---

Published as a conference paper at ICLR 2023
Table 1: Density estimation performance of Tractable Probabilistic Models (TPMs) and Deep Generative Models (DGMs) on three natural image datasets. Reported numbers are test set bit-per- dimension (bpd). Bold indicates best bpd (smaller is better) among all four TPMs.
TPMs
DGMs
Dataset
LVD (ours)
HCLT
EiNet
RAT-SPN
Glow
RealNVP
BIVA
ImageNet32
4.39
4.82
5.63
6.90
4.09
4.28
3,96
ImageNet64
4.12:0.00
4.67
5.69
6.82
3.81
3.98
CIFAR
4.38
4.61
5.81
6.95
3.35
3.49
3.08
ImageNet332
ImageNet64
CIFAR
Bits per-per--die diddmmnniio (bpd)
Hit--per-al Site-per-diment dimension (bpd)
Sits-ger-dimension (bpd)
LVD (rmm)
LVD ICLT RAT-SPN EiNes
LVD
6.5
63
RAT-SPN
6.5
RAT-SPN
6-
6
5.5
5.3
5.5
5-
4.5
5
10
10
10²
10
108
parameters
parameters
#
parameters
Figure 7: Generative modeling performance of four TPMs on three natural image datasets. For each method, we report the test set bits-per- bits-per-dimee per-dimension (y-axis) in terms of the number of parameters (x-axis) for different numbers of latent states.
indices as the LV assignments. Figure bbb) shows some example image patches æ1 belonging to four latent clusters (i.e., Zi 1.....4) 4). Clearly, the LVs capture semantics of different image patches.
To illustrate the effectiveness of LVD, we make minimum structural changes compared to Hidden Chow-Liu Trees (HCLTs) (Liu & Van den Broeck (02T), a competitive PC structure. Specifically, we use the HCLT structure for all sub-PCs {p(x,] Z, = ji and p(z). This allows us to materialize patch-based LVs while keeping the model architecture similar to HCLTs.
EXPERIMENTS
In this section, we evaluate the proposed latent variable distillation (LVD) technique on three natural image benchmarks, i..., CIFAR (Krizhevsky et al. 2009) and two versions of down-sampled ImageNet (ImageNet32 and ImageNet64) (Deng et al.) 2009). On all benchmarks, we demonstrate the effectiveness of LVD from two perspectives. First, compared PCs trained by existing EM-based optimizers, the proposed technique offers a significant performance gain especially on large PCs. Second, PCs trained by LVD achieve competitive performance against some of the less tractable deep generative models, including variational autoencoders and flow-based models.
Baselines We compare the proposed method against three TPM baselines: Hidden Chow-Liu Tree (HCLT) (Liu & Van den Broeck 2021), Einsum Network (EiNet) (Pcharz et al. 2020a). and Random -un-Prrduct Network (RAT-SPN) (Peharz et al. 2020b). Though not exhausive, this baseline suite embodies many of the recent advancement in tractable probabilistic modeling, and can be deemed as the existing SoTA. To evaluate the performance gap with less tractable deep generative models. we additionally compare LVD with the following dow-based and VAE models: Glow (Kingma & Dhariwal 2018), RealNVP (Dinh et al. 2016), and BIVA (Maaløe et al. (019).
To facilitate fair comparison with the chosen TPM baselines, we implement both HCLT and RAT- SPN using the Julia package Juice.jl (Dang et al. 2021) and tune hyperparameters such as batch size, learning rate and its schedule. We use the original Py Torch implementation of EiNet and similarly tune their hyperparameters. For all TPMs, we train various models with number of parameters ranging from 1 100M, and report the number of the model with the best performance. For deep generative model baselines, we adopt the numbers reported in the respective original papers. Please refer Appx. C for more details of the experiment setup.
Empirical Insights We first compare the performance of the four TPM approaches. As shown in Figure 1. for all three benchmarks, PCs trained by LVD are consistently better than the competitors
8

---

Published as a conference paper at ICLR 2023
by large margin. In particular, ImageNet32, 25M PC trained by LVD is better than HCLT with r~400M parameters. Next, looking at individual curves, we observe that with LVD, the test set bpd keeps decreasing as the model size increases. This indicates that LVD is able to take advantage of the extra capacity offered by large PCs. In contrast, PCs trained by HM immediately suffer from a performance bottleneck the model size increases. Additionally, the efficient LVD learning pipeline described in Section Dection[allooss allows us to train PCs with 500M parameters in 10 hours with single NVIDIA A5000 GPU, while existing optimizers need over day to train baseline PCs with similar sizes. Please refer to Appx. for detailed analysis of the computation efficiency of LVD.
We move on to compare the performance of LVD with the three adopted DGM baselines. As shown in Table ! although the performance gap is relatively large on CIFAR, the performance of LVD is highly competitive on ImageNet32 and ImageNet64, with bpd gap ranging from ~0.1 to ~0.3. We hypothesize that the relatively large performance gap on CIFAR is caused by insufficient training samples. Specifically, for the PC structures specified in Section [5] the sub-PCs correspond to the latent-conditioned distributions {p(æ; I2, j)lis constructed independently. and thus every training sample D; can only be used to train its corresponding latent-conditioned distribution, making the model extremely data-hungry. However, we note that this is not an inherent problem of LVD. For example, by performing parameter tying of sub-PCs correspond to different image patches, we can significantly improve sample complexity of the model. This is left to future work.
7 RELATED WORKS
There has been various recent endeavors to improve the performance of PCs on modeling complex and high-dimensional datasets. An extensivelyeepplored direction is to construct or learn the structure of PCs that is tailored to the target dataset. For example, Gens & Pedro (2013) Dang et al. (2022) seek to progressively improve the PC structure during the optimization process. Many other work aim to construct good PC structures given the dataset in one shot, and then move on to parameter optimization (Rahman al.) 2014 Adel et al. (015).. Model agnostic PC structures such as RAT-SPN (Peharz et al., 2020b) and extremely randomized PCs (XPCs) (Di Mauro et al. 202T) are also shown effective various density estimation benchmarks. There are also papers that focus exclusively on scaling up a particular TPM. For example, scaling HMM (Chiu & Rush 2020) uses techniques such as learning blocked emission probabilitics to boost the performance of HMMs.
Another line of work seek to scale up PCs by learning hybrid models with neural networks (NNs). Specifically, Shao et al. (022) leverages the expressive power of NNs to learn expressive yet tractable conditional distributions; HyperSPN (Shih et al.) 102 uses NN to regularize PC parameters, which prevents large PCs from overfitting. Such hybrid models are able to leverage the expressiveness of NNs the cost of losing tractability certain queries.
8 CONCLUSION
Scaling probabilistic circuits to large and high-dimensional real-world datasets has been a key challenge: as the number of parameters increases, their performance gain diminishes immediately. In this paper, we propose to tackle this problem by latent variable distillation: general framework for training probabilistic cireuit that provides extra supervision over their latent spaces by distilling information from existing deep generative models. The proposed framework significantly boosts the performance of large probabilistic circuits on challenging benchmarks for both image and language probabilistic circuit
modeling. In particular, with latent variable distillation, a image-pater mmage-patc--structure achieves competitive performance against Bow-based models and variational autoencoders. Despite its empirical success on scaling up probabilistic circuits, at high-level, latent variable distillation also implies new way to organically combine probabilistic circuits and neural models, opening up new avenues for tractable generative modeling.
Reproducibility statement To facilitate reproducibility, we provide detailed description of the models and training details in both the main text and the appendix. Specifically, the last paragraph in Section 5 elaborates the PC structure used for image modeling tasks; training details of both the proposed method and the baselines are provided in Section ① and Appx. C For baselines, we always use the official GitHub implementation if possible. For our method, we provide detailed explanation of all hyperparameters used in the experiment (Appx. C.

---

Published as a conference paper at ICLR 2023
REFERENCES
Tameem Adel, David Balduzzi, and Ali Ghodsi. Learning the structure of sum sum-product networks via an svd-based algorithm. In UAI, pp. 41, 2015.
Justin Chiu and Alexander Rush. Scaling hidden Markov language models. In Proceedings of the
2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 1341- 1349, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/11/200
emnlp-main.1033 URL https: aattpss//aclaatthoooog.oo htt.:yy.rrg/20000 emnlp main. 103
YooJung Choi, Antonio Vergari, and Guy Van den Broeck. Probabilistic circuits: A unifying framework for tractable probabilistic models. oct 2020.
YooJung Choi, Meihua Dang, and Guy Van den Broeck. Group fairness by probabilistic modeling with latent fair decisions. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, Feb 2021.
Alvaro HC Correia, Gennaro Gala, Erik Quaegheheur, Cassio de Campos, and Robert Peharz. Continuous mixtures of tractable probabilistic models. arXiv preprint arXiv:2209. 10584, 2022.
Meihua Dang, Pasha Khosravi, Yitao Liang, Antonio Vergari, and Guy Van den Broeck. Juice: A julia package for logic and probabilistic circuits. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (Demo Track), Feb 2021.
Meihua Dang, Anji Liu, and Guy Van den Broeck. Sparse probabilistic circuits via pruning and growing. In The 5th Workshop on Tractable Probabilistic Modeling, 2022.
Adnan Darwiche. A differential approach inference in bayesian networks. Journal of the ACM (JACM), 50(3):280-305, 2003.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-258 Icee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (I). Association for Computational Linguisties, 2019.
Nicola Di Mauro, Gennaro Gala, Marco lannotta, and Teresa MA Basile. Random probabilistic circuits. In Uncertainty in Artificial Intelligence, pp. 1682-1691. PMLR, 2021.
Laurent Dinh, Jascha Sohl-Dicksted Diccksteik, and Samy Bengio. Density estimation using real nvp. In International Conference Learning Representations, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arX1v:2010.11929 2020.
Robert Gens and Domingos Pedro. Learning the structure of sum-product networks. International conference on machine learning, pp. 873-880. PMLR, 2013.
Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Diederik P Kingma and Prafulla Dhariwal. Glow; generative flow with invertible 1 convolutions. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 10236-10245, 2018
Doga Kisa, Guy Van den Broeck, Arthur Choi, and Adnan Darwiche. Probabilistic sentential decision diagrams. In Fourteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2014.
10

---

Published as a conference paper at ICLR 2023
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Anji Liu and Guy Van den Broeck. Tractable regularization of probabilistic eircuits. Advances in Neural Information Processing Systems 35 (NeurIPS), dec 2021.
Anji Liu, Stephan Mandt, and Guy Van den Broeck. Lossless compression with probabilistic circuits. In Proceedings of the International Conference on Learning Representations (ICLR), apr 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2021.
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2): 129-137, 1982.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: very deep hierarchy of latent variables for generative modeling. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 6551-6562, 2019.
Radu Marinescu and Rina Dechter. And/or graachhandd-oout for graphical models. In IJCAI, pp- 224 229. Citeseer, 2005.
Marina Meila and Michael Jordan. Learning with mixtures trees. Journal of Machine Learning Research, 1(Oct):1-48, 2000.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.074 2016.
Alejandro Molina, Antonio Vergari, Karl Stelzner, Robert Peharz, Pranav Subramani, Nicola Di Mauro, Pascal Poupart, and Kristian Kersting. Spflow: An easy and extensible library for deep probabilistic learning using sum-product networks. arXiv preprint arXiv: 1901. 03704, 2019.
Robert Peharz, Robert Gens, Franz Pernkopf, and Pedro Domingos. the latent variable interpreta- tion in sum-product networks. IEEE transactions patlern analysis and machine intelligence, 39 (10):2030-2044, 2016.
Robert Peharz, Steven Lang, Antonio Vergari, Karl Stelzner, Alejandro Molina, Martin Trapp, Guy Van den Broeck. Kristian Kersting, and Zoubin Ghahramani. Einsum networks: Fast and scalable learning of tractable probabilistic circuits. In International Conference on Machine Learning, pp. 7563-7574. PMLR, 2020a.
Robert Peharz, Antonio Vergari, Karl Stelzner, Alejandro Molina, Xiaoting Shao, Martin Trapp, Kristian Kersting, and Zoubin Ghahramani. Random sum-product networks: simple and effective approach to probabilistic deep learning. In Uncertainty in Artificial Intelligence, pp. 334-344.
PMLR, 2020b.
Hoifung Poon and Pedro Domingos. Sum-product networks: new deep architecture. In 201 IEEE International Conference Computer Vision Workshops (ICCV Workshops), pp. 689-690. IEEE, 2011.
Lawrence Rabiner and Biinhhwang Juang. An introduction hidden markov models. ieee assp magazine, 3(1):4-16, 1986.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Tahrima Rahman, Prasanna Kothalkar, and Vibhav Gogate, Cutset networks: A simple, tractable, and scalable approach for improving the accuracy of chow-liu trees. In Joint European conference on machine learning and knowledge discovery in databases, pp. 630-645, Springer, 2014.
Xiaoting Shao, Alejandro Molina, Antonio Vergari, Karl Stelzner, Robert Peharz, Thomas Liebig, sum- product networks: Modular probabilistic circuits via gate functions. International Journal Approximate Reasoning, 140:298-313, 2022.
and Kristian Kersting. Conditional sum

---

Published as a conference paper at ICLR 2023
Andy Shih and Stefano Ermon. Probabilistic circuits for variational inference in discrete graphical models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 4635-46444 2020.
Andy Shih, Dorsa Sadigh, and Stefano Ermon. Hyperspns: compact and expressive probabilistic circuits. Advances in Neural Information Processing Systems, 34:8571-8582, 2021.
12

---

Published as a conference paper at ICLR 2023
A
PROOFS
In this section, we provide detailed proofs of Lemma
A.1 PROOF OF LEMMA
Proof. Define Y W. To prove that variables W is conditional independent with given Z, sufficient show that Vww E val((W) Eval Z), E vval( have p( ((w//) p( y).
Define p.w and Sprod "pW as the set of sum and product units with scope W. respectively. Vn P. W since the scope of n does not contain variables Y, we can immediately conclude that pa (w/z) = p(w|z, y). In order to show that this equation also holds for the root PC unit, we only need to show that for each PC unit n p. all its children (whose scope contains W) satisfy this equation, then n also does. The reason is that the root unit n, of p must be ancestor unit of every unit in P.W
We start with the case that n is a product unit. Since the PC is assumed to be decomposable, only one child, denoted m, satisfies W S 6(m). Therefore, the distribution of n can be written
Pa (x)) = pm (w)
II
Pe(ar)
(a) Pm( )
II
Po(y),
cEin(n)).o/mm
= Therefore, we have pr. (w(2)) = Pm (w]2) and p,. [w]2 y) Pm (w|z, y). Taking the two equations together and use the assumption from the induction step: Pm ( ()) Pm [w/= y), we conclude that p.. (w([W2)) = (w|z,y).
where (a) holds because (n)in(n).......... cf m, we have d(c) ==
Define n, as the product unit in Sp. that is augmented with input unit Z Before proving the main result, we highlight that Yn whose scope contains W, Pn. (x) can be written as Pn. (w) 9n (1). where gn (y) is independent with w. This is because Vm Sprod and m n2. p.
by Algorithm
Pm Pm(w,22 2) = 0('' val(W)).
Next, assume is a sum unit whose scope contains W. Using the above result, we know that every child of satisfies the following: Pa(x)) = Pn. (w) (y))). Thus, we have
p=(x)= = Σ
O.l Pn, (w) ge(y) = Pn, pn.(p)))) (w) ( Σ
Onj sc(v)).
cEin(n)
cEin(n)
Since Pn, (w|z)=Prn. (w|z, y), we have Pa (w|z) Pm (ww,,)/)....)))
Taking the above two inductive cases (i.e., for sum and product units, respectively), we can conclude that for the root unit n2,, Pn, Pn.(w (w]z)==pa) (n/,.,/la (wl2.y).
B DETAILS FOR LATENT VARIABLE DISTILLATION
This section provides additional details for latent variable distillation (LVD), including description of the adopted EM algorithm and details of the LV extraction step.
B. PARAMETER ESTIMATION
We adopt stochastic mini-batch version of the Expectation -axittiioo-aaimizzttiio algorithm. Specifically, mini-batch of samples are drown from the dataset, and the EM algorithm for PCs (Choi et al.) 2021 Dang al. 2021) is used to compute of new parameters gnew which is updated with learning rate ac 0z+1 < a. nuew + (1 a)-02.
B.2 DETAILS OF THE MAE-BASED LV EXTRACTION STEP
We use the official code (https://githii) htwphuu..oomeeee.. com/ models the adopted datasets (i.e., CIFAR, ImageNet32, and ImageNet64). At each training step, the percentage of masked patches is chosen uniformly from 10% to 90%. After training, the LV extraction step follows the description in Section
butrss//dthbbb.co/ocaooeeeeeeeee cccm://acebookeeeerrr///aaaaaaa to train MAE
13

---

Published as a conference paper at ICLR 2023
C EXPERIMENT DETAILS
In this section, we describe experiment details of all four TPMs adopted in Section Section 6
Hardware specification All experiments are servers/workstations with the following configuration:
32 CPUs, 128G Mem, 4 x NVIDIA A5000 GPU;
32 CPUs, 64G Mem, X NVIDIA GeForce RTX 3090 GPU;
64 CPUs, 128G Mem, 3 x NVIDIA A100 GPU.
HMM The HMM models are trained with varying hidden states = 128, 256, 512, 750, 1024 and 1250, with and without LVD. All HMM models are trained with mini- batch EM (Appx. [B.1) for two phases: in phase I, the model is trained with learning rate 0.1 for 20 epochs; in phase 2, the model is trained with learning rate 0.01 for 5 epochs. Note that for HMM models with hidden states 2 750, we train for 30 epochs in phase 1. The number of epochs are selected such that all model converges before training stops.
LVD For every subset X,, the number of hidden categories, i.e., {M: are set to values in {8, 16, 32, 64, 128, 256]. For the latent-conditioned distribution {p(X, = we adopt HCLTs with hidden size 16, and for the latent distribution p(Z), HCLT with hidden size M, is adopted. When optimizing the model with the MLE lower bound, we adopt mini-batch EM (Appx. B.I) with learning rate annealed linearly from 0.1 to 0.01. In the latent distribution training step (Sec. Section we anneal learning rate from 0.1 to 0.001.
We use the publicly available implementation of HCLT at https://githus htt.r//iiirctttt...o//oaaaaaaaee httpb//bmmastee/////oo/pb/ /steeer/rrreeeeeeeee structures hclts. The hidden size is chosen from {16, 32, 64, 128, 256, 512, 1024). We anneal the EM learning rate from 0.1 to 0.01 and train for 100 epochs, and then anneal the learning rate from 0.01 to 0.001 and train for another 100 epochs.
HCLT
com Juice wttprobabilisticiiiiiiccccaaaail
We adopt the publicly available implementation at Juice- (aaee-11//robbbblii htr.roobabiist.... cci rcuits.
RAT-SPN
ji num num_nodes region, num features, and num som_nodes_.o leaf are set to the same value, which is chosen from {16, 32, 64, 128, 256, 512, 1024). Learning rate schedule same with HCLTs.
EiNet We use the official implementation on GitHub: https: //git thub. com/
cembridge-min g nsumNetworka We use the PD structure provided in the codebase. We select hyperparameter delta from {4, 6, 8} and select num-sums from {16, 32, 64, 128, 256}. Learn- ing rate is set 0.001.
D EFFICIENCY ANALYSIS
This section provides the breakdown of the runtime for each stage of the LVD algorithm for the PC that achieves 4.38 bpd on ImageNet32. The PC has 836M parameters. All experiments are done on single NVIDIA A5000 GPU.
For this PC, training all latent conditioned distributions {p(x; ,)} take 8 hours, and training the latent distribution p(z) takes 0.5 hours. Finally, the fine-tuning stage takes ~ hour.
As shown in the above computation time breakdown, the most time-consuming part is to train the latent conditioned distributions. However, we note that this is not fundamental problem of LVD: we are training every latent conditioned distributions independently, while there could be massive structure/parameter sharing among such distributions. We left this future work.
14


Reviews:
[{"review_id": "uWUiWwA5aA", "reviewer": "Reviewer_tcWu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach to scale tractable probabilistic models via latent variable extensions. In particular, the authors extend the naive factorizations of models such as an SPN to introduce a new latent variable that comes from the embedding of a more complex neural network.\n\nThis extended model retains the tractability properties while reducing the bits-per-dimension for both image and text datasets compared to other state-of-the-art tractable probabilistic models.\n\nThe authors present a distributed learning approach that uses the independencies induced by the conditioning to accelerate the training, scaling the capabilities of TPMs.\n", "summary_of_the_review": "I find the paper very interesting and it opens the door to integrate other more complex models while retaining some of the tractability properties. \n\nThis is a significant contribution, although I'm wondering about downside of this approach or some comments on the questions raised regarding the use of the model when we don't have access or can't compute the LVs.\n\nThe bpd results are very impressive. I'm also wondering about how this model differs from the one presented by Shao 2022, in the case you would set the latent variables and the conditional part. \n\n", "strength_and_weaknesses": "The authors present a well-written and technically sound paper. \n\nThe empirical evaluation is outstanding for TPMs, both in terms of scale and model performance.\n\nIt would be nice to have some more comments on the impact of LVD with regards to the rest of the model. How would you do imputation? How is the performance of your model when you marginalize the LVs?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ifmSa-gDDb", "reviewer": "Reviewer_7gAx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Probabilistic circuits (PCs) are a unified framework that encompasses a number of tractable probabilistic models, such as arithmetic circuits and sum-product networks. While recent work has been able to scale them by means of parallelization and vectorization, their performance does not scale with the number of parameters. This rather unintuitive results suggest that the problem comes from the optimization process, which cannot fully utilize the model parameters and gets stuck in local optima. This work proposes to leverage existing deep generative models to assist the training of PCs through a novel method called latent variable distillation. In short, the PC is extended to include a set of latent variables, which are instantiated via the latent representation from deep generative models. This alone permits to better exploit the model. Additionally, said latent extension permits to divide the PCs into conditionally independent modules which can be trained simultaneously, speeding up the training process. The authors demonstrate the advantages of the proposed method in the text and image domains by leveraging Bert and MAE models.", "summary_of_the_review": "The paper is novel and well-written. The empirical results clearly demonstrate the advantages of the proposed method with respect to existing approaches. However, I feel the paper falls short when it comes to understanding the influence of the different factors of the proposed method in the final performance.", "strength_and_weaknesses": "**Strengths:**\n- S1. This work clearly demonstrates a novel method to exploit all the capacity of PCs using auxiliary and non-tractable methods from Deep Learning. This is extremely relevant for the PC community to close the gap with existing DL approaches.\n- S2. The practical speed-up obtained by their method (section 4) is quite relevant as well.\n- S3. Experimental results are strong.\n\n**Weaknesses:**\n- W1. While the idea is novel, I felt a bit let down when I read that the approach need to be \"engineered\" depending on the dataset, architecture, and deep generative model. The two instantiations of the approach look really ad-hoc. I would've hoped to see some sort of general guidelines, or desiredata for the method to be successful.\n- W2. On a similar note, I feel a deeper ablation study would help a lot. There are too many questions open:\n   - How important is the deep generative architecture to obtain good results?\n   - Do other PCs architectures benefit from the proposed approach? So far, it is only tried in a modified version of HCLT.\n   - K-means is also prone to fall into local optima. How much do the results vary with different K-means initializations?\n   - How does the algorithm perform if we condition (Fig. 5) on deeper layers?\n- W3. I might have missed it, but it is not clear to me how many times each experiment was repeated. In any case, standard deviations are not reported.\n- W4. No samples were reported, so one cannot assess the qualitative improvements. Are the new generated samples better as well?\n\n**Questions:**\n- Q1. Any intuition on why the order to generate the latent samples is reversed with respect to the HMM in the example of section 2?\n- Q2. Could you clarify why is the upper part of Fig. 5 supposed to be $p(z_1, z_2)$. While I agree on the bottom modules, I do not see how the top part represents the marginal of $z$.\n- Q3. I understand LVD is HCLT using the proposed training approach. Why is there so much difference in number of parameters between both approaches in Figure 7? Is it because of the \"minimum modifications required\" described in the text? Why not comparing with that modified architecture without LVD?\n-  Q4. What do you mean by \"semantics\" in the manuscript? I really struggle to understand what \"semantic-aware\" means.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "GWy0Alf2OtT", "reviewer": "Reviewer_o5Cj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about improving the expressivity of large scale probabilistic circuits (PCs). Finding a good starting point for EM based learning of these large latent variable models is problematic and the authors propose one such solution to this problem.  The main idea is to obtain semantic-aware assignments (called supervision) to the latent variables from less tractable deep generative models and then perform maximum likelihood learning over the data combined with these newly assigned latent variables. The variables that receive these assignments are said to be materialized and the assignments themselves are generated by a deep generative model by clustering over the latent embeddings of the observed (sub)space(s). When all the latent variables have been assigned values, the optimization (MLE) can be performed in closed form. The result of MLE serves as a starting point for optimizing the data likelihood in the following steps. The authors propose a couple of techniques to efficiently compute the MLE parameters which include exploiting conditional independency achieved by materialized latent variables and fine tuning the latent distributions only while keeping the parameters learned over the observed space fixed. On CIFAR and ImageNet datasets, the proposed method has shown superior performance compared to other SoTA TPMs learners and were comparable to less tractable but expressive flow-based models and VAEs.   \n", "summary_of_the_review": "The authors have addressed an important practical issue with large scale probabilistic circuits. The expressivity of these models tend to plateau once a certain capacity is reached typically in the order of millions of parameters. With such large scale circuits of deeply nested latent variables, the optimization landscape becomes very complex and finding a local minima becomes hard. The main idea in the paper is to make latent variables observed by assigning them values and performing an optimization step that works on less number of latent variables. This will give the EM step a good starting point. I found the idea to obtain latent variable assignments using a deep generative model to be interesting. The method seems to be effective according to the empirical results presented in the paper. \nQuestions:\na) In the introduction it is stated that the expressive power of PCs should monotonically increase with respect to the number of parameters. I am curious if these models don't suffer from overfitting issues. Maybe the authors could comment on this aspect.  \nb) Did all the models have the same structure?\nc) Could the method be useful for smaller scale PCs? It seems that the clustering in the latent embedding is the key reason behind the performance boost of LVDs.\nd) The significant speed up in training (for all the datasets) should probably be presented in the paper since it is mentioned in the introduction. \ne) Have you done any analysis on the number of LVs that were materialized and the performance of the PCs?\n", "strength_and_weaknesses": "Strengths:\n1) a novel method to improve the performance of large scale probabilistic circuits.\n2) motivating empirical evaluations on three image datasets to show the performance improvements over very large circuits. \n3) clear writeup with good examples. \nWeaknesses:\nI didn't find major weaknesses in the technical aspects of the paper. Please see some questions and comments below. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%