Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is about improving the expressivity of large scale probabilistic circuits (PCs). Finding a good starting point for EM based learning of these large latent variable models is problematic and the authors propose one such solution to this problem.  The main idea is to obtain semantic-aware assignments (called supervision) to the latent variables from less tractable deep generative models and then perform maximum likelihood learning over the data combined with these newly assigned latent variables. The variables that receive these assignments are said to be materialized and the assignments themselves are generated by a deep generative model by clustering over the latent embeddings of the observed (sub)space(s). When all the latent variables have been assigned values, the optimization (MLE) can be performed in closed form. The result of MLE serves as a starting point for optimizing the data likelihood in the following steps. The authors propose a couple of techniques to efficiently compute the MLE parameters which include exploiting conditional independency achieved by materialized latent variables and fine tuning the latent distributions only while keeping the parameters learned over the observed space fixed. On CIFAR and ImageNet datasets, the proposed method has shown superior performance compared to other SoTA TPMs learners and were comparable to less tractable but expressive flow-based models and VAEs.   

strength_and_weaknesses: Strengths:
1) a novel method to improve the performance of large scale probabilistic circuits.
2) motivating empirical evaluations on three image datasets to show the performance improvements over very large circuits. 
3) clear writeup with good examples. 
Weaknesses:
I didn't find major weaknesses in the technical aspects of the paper. Please see some questions and comments below. 

clarity,_quality,_novelty_and_reproducibility: The paper is well written with illustrative examples. I found the idea to be novel and the authors have detailed their experimental setups for reproducibility. 
summary_of_the_review: The authors have addressed an important practical issue with large scale probabilistic circuits. The expressivity of these models tend to plateau once a certain capacity is reached typically in the order of millions of parameters. With such large scale circuits of deeply nested latent variables, the optimization landscape becomes very complex and finding a local minima becomes hard. The main idea in the paper is to make latent variables observed by assigning them values and performing an optimization step that works on less number of latent variables. This will give the EM step a good starting point. I found the idea to obtain latent variable assignments using a deep generative model to be interesting. The method seems to be effective according to the empirical results presented in the paper. 
Questions:
a) In the introduction it is stated that the expressive power of PCs should monotonically increase with respect to the number of parameters. I am curious if these models don't suffer from overfitting issues. Maybe the authors could comment on this aspect.  
b) Did all the models have the same structure?
c) Could the method be useful for smaller scale PCs? It seems that the clustering in the latent embedding is the key reason behind the performance boost of LVDs.
d) The significant speed up in training (for all the datasets) should probably be presented in the paper since it is mentioned in the introduction. 
e) Have you done any analysis on the number of LVs that were materialized and the performance of the PCs?
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents an approach to scale tractable probabilistic models via latent variable extensions. In particular, the authors extend the naive factorizations of models such as an SPN to introduce a new latent variable that comes from the embedding of a more complex neural network.

This extended model retains the tractability properties while reducing the bits-per-dimension for both image and text datasets compared to other state-of-the-art tractable probabilistic models.

The authors present a distributed learning approach that uses the independencies induced by the conditioning to accelerate the training, scaling the capabilities of TPMs.

strength_and_weaknesses: The authors present a well-written and technically sound paper. 

The empirical evaluation is outstanding for TPMs, both in terms of scale and model performance.

It would be nice to have some more comments on the impact of LVD with regards to the rest of the model. How would you do imputation? How is the performance of your model when you marginalize the LVs?


clarity,_quality,_novelty_and_reproducibility: The paper is easy to read, and introduces the idea clearly. It is a novel approach that makes TPMs even more scalable.

The algorithms presented are well defined and allows for ease of reproducibility.
summary_of_the_review: I find the paper very interesting and it opens the door to integrate other more complex models while retaining some of the tractability properties. 

This is a significant contribution, although I'm wondering about downside of this approach or some comments on the questions raised regarding the use of the model when we don't have access or can't compute the LVs.

The bpd results are very impressive. I'm also wondering about how this model differs from the one presented by Shao 2022, in the case you would set the latent variables and the conditional part.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Probabilistic circuits (PCs) are a unified framework that encompasses a number of tractable probabilistic models, such as arithmetic circuits and sum-product networks. While recent work has been able to scale them by means of parallelization and vectorization, their performance does not scale with the number of parameters. This rather unintuitive results suggest that the problem comes from the optimization process, which cannot fully utilize the model parameters and gets stuck in local optima. This work proposes to leverage existing deep generative models to assist the training of PCs through a novel method called latent variable distillation. In short, the PC is extended to include a set of latent variables, which are instantiated via the latent representation from deep generative models. This alone permits to better exploit the model. Additionally, said latent extension permits to divide the PCs into conditionally independent modules which can be trained simultaneously, speeding up the training process. The authors demonstrate the advantages of the proposed method in the text and image domains by leveraging Bert and MAE models.
strength_and_weaknesses: **Strengths:**
- S1. This work clearly demonstrates a novel method to exploit all the capacity of PCs using auxiliary and non-tractable methods from Deep Learning. This is extremely relevant for the PC community to close the gap with existing DL approaches.
- S2. The practical speed-up obtained by their method (section 4) is quite relevant as well.
- S3. Experimental results are strong.

**Weaknesses:**
- W1. While the idea is novel, I felt a bit let down when I read that the approach need to be "engineered" depending on the dataset, architecture, and deep generative model. The two instantiations of the approach look really ad-hoc. I would've hoped to see some sort of general guidelines, or desiredata for the method to be successful.
- W2. On a similar note, I feel a deeper ablation study would help a lot. There are too many questions open:
   - How important is the deep generative architecture to obtain good results?
   - Do other PCs architectures benefit from the proposed approach? So far, it is only tried in a modified version of HCLT.
   - K-means is also prone to fall into local optima. How much do the results vary with different K-means initializations?
   - How does the algorithm perform if we condition (Fig. 5) on deeper layers?
- W3. I might have missed it, but it is not clear to me how many times each experiment was repeated. In any case, standard deviations are not reported.
- W4. No samples were reported, so one cannot assess the qualitative improvements. Are the new generated samples better as well?

**Questions:**
- Q1. Any intuition on why the order to generate the latent samples is reversed with respect to the HMM in the example of section 2?
- Q2. Could you clarify why is the upper part of Fig. 5 supposed to be $p(z_1, z_2)$. While I agree on the bottom modules, I do not see how the top part represents the marginal of $z$.
- Q3. I understand LVD is HCLT using the proposed training approach. Why is there so much difference in number of parameters between both approaches in Figure 7? Is it because of the "minimum modifications required" described in the text? Why not comparing with that modified architecture without LVD?
-  Q4. What do you mean by "semantics" in the manuscript? I really struggle to understand what "semantic-aware" means.

clarity,_quality,_novelty_and_reproducibility: **Clarity and quality.** I find the paper very well written and of high quality. There are a number of typos that should be corrected for the next revision. I also find the paper to reiterate the same statements too many times, but this is personal taste.

**Novelty.** The method seems novel to my eyes, and the advantages are clear. However, I find _really_ surprising that there are no citations at all regarding Probabilistic Circuits. They were first introduced in a UAI tutorial, and afterwards [a technical report](https://web.cs.ucla.edu/~yjchoi/publications/ProbCirc20/) was written by the authors which can be cited.

**Reproducibility.** The experimental set-up should be reproducible.
summary_of_the_review: The paper is novel and well-written. The empirical results clearly demonstrate the advantages of the proposed method with respect to existing approaches. However, I feel the paper falls short when it comes to understanding the influence of the different factors of the proposed method in the final performance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper describes an approach (using latent variables) to over come (most likely) optimisation issues in large-scale probabilistic circuits.  The work seems novel and the empirical results are compelling. One downside was a limited understanding of what factors make the method successful. This was addressed quite in-depth by the authors during the discussion phase and the reviewers were satisfied with the conclusions reached by the authors.

This is a well written high quality paper with a nice methodological contribution supported by extensive experiments. 
justification_for_why_not_lower_score: The reviewer scores are consistently high throughout all reviewing stages, with also extensive additional input from the authors to address thoroughly the reviewer's issues.
Decision: Accept: notable-top-5%